Learning to Iteratively Solve Routing Problems with Dual-Aspect Collaborative Transformer

Abstract: Recently, Transformer has become a prevailing deep architecture for solving vehicle routing problems (VRPs). However, it is less effective in learning improvement models for VRP because its positional encoding (PE) method is not suitable in representing VRP solutions. This paper presents a novel Dual-Aspect Collaborative Transformer (DACT) to learn embeddings for the node and positional features separately, instead of fusing them together as done in existing ones, so as to avoid potential noises and incompatible correlations. Moreover, the positional features are embedded through a novel cyclic positional encoding (CPE) method to allow Transformer to effectively capture the circularity and symmetry of VRP solutions (i.e., cyclic sequences). We train DACT using Proximal Policy Optimization and design a curriculum learning strategy for better sample efficiency. We apply DACT to solve the traveling salesman problem (TSP) and capacitated vehicle routing problem (CVRP). Results show that our DACT outperforms existing Transformer based improvement models, and exhibits much better generalization performance across different problem sizes on synthetic and benchmark instances, respectively.


Reviews and author responses:

 Thanks for raising the concern. Regarding the example of VRPTW, the [1, 2, 3, 4] and [2, 3, 4, 1] are indeed two solutions because of the orders of delivery, however, the two solutions should be [0,1,2,3,4,0] and [0,2,3,4,1,0] (0 is the depot) which forms the cyclic tour(s). Our CPE "works" (having additional benefit) better in the sense that we make the node embeddings of nodes 0, 2, 3, 4, and 1 cyclicly distributed in the high-dimensional spaces (node embedding 0 is near to node embedding 1 and 4 for  [0,1,2,3,4,0]; and node embedding 0 is near to node embedding 2 and 1 for [0,2,3,4,1,0]). In this sense, our CPE does treat them as two different solutions, and such design helps the generalization performance over different problem sizes as verified in the ablation study. In other words, our PFEs are introduced for encoding the ring topology of the VRP solution and the different orders of delivery mentioned by the reviewer can be captured by the Node Feature Embeddings (NFEs) rather than the Positional Feature Embeddings (PFEs). For example, we can add the cumulative time consumed before and after servicing each node (customer) to its corresponding NFEs as a feature to cope with the visiting orders. Further empowered by our dual-aspect representation and designed DAC-Att, the two types of heterogeneous information (PFE and NFE) can be synthesized to make decisions.

Regarding using CPE in Transformer for other tasks, we will investigate it in our future work.
***
Moreover, we also tested current SOTA POMO (8 augment) on those instances, which achieved average Gaps of 10.6% and 6.1%, respectively. Those results are inferior to the ones achieved by our DACT in Table 4 and 5, i.e., 2.1% and 3.4%, respectively, which further justified the desirable generalization performance of our model induced by the proposed CPE. We will add the new results to the paper.

 Dear Reviewer #pCD7,

Thanks for taking the time to review our paper. May we know if you still have additional concerns?

 We thank the reviewer for the acknowledgment and response. 
***
**Regarding the generalization**, please refer more to the results on the instances of TSPLIB and CVRPLIB (Table 4 and 5 in Appendix). ***Moreover, we also tested current SOTA POMO ($\times$8 augment) on those instances***, which achieved average Gaps of 10.6% and 6.1%, respectively. Those results are inferior to the ones achieved by our DACT in Table 4 and 5, i.e., 2.1% and 3.4%, respectively, which further justified the desirable generalization performance of our model. We will add the new results to the paper. 
***
**Regarding the extremely large instances**, we do acknowledge that it is meaningful for solving routing problems. However, it may need a lot of effort to specifically design for a learning-based method, which alone is worth a publication and also out of the scope of this paper, if the reviewer reads through our motivation again while taking into account other related works in this area. 
***
**Regarding the 'limited technical novelty'**, we really disagree. To our knowledge, although quite a number of learning-based methods have been proposed for solving routing problems, this paper is the first one to identify that the cyclic or symmetric properties are useful to improve performance. Accordingly, we developed DACT including the CPE, which we believe is sufficiently novel. Moreover, this paper is also the first one to discover that the CL strategy is perfectly helpful for a learning-based method to solve the combinatorial optimization problems; it is also justified by the experiments. 
***
**Regarding the speed-gap curve**, we do appreciate the suggestion, and we will emphasize it more in the revised paper. 

 Thanks to the authors for the responses. After reading the responses and other reviewers' concerns, I think most of my concerns are addressed, but indeed the contributions need a better justification. In my opinion, the better generalization on large sizes introduced by CPE and the speed-gap comparisons are definitely valuable. However, I agree with other reviewers that the authors should either emphasize more on the speed-gap curve comparison or explore a significantly large instance setting to better position this paper, given the limited technical novelty. Also, whether the proposed framework is applicable to other tasks remains unsupported. In this regard, I will slightly lower my scores.

 We deeply appreciate the acknowledgement and support, and we will continue to polish our work.

 Based on the clarifications added here and the expectation of further community contributions (code, etc) I feel my concerns have been met. I see significant advance in the state of the art with the contributions of the DACT and CPE architectures. The cyclic disposition of the input encoder and the curriculum strategy for the PPO component make this approach a flexible solution with good potential for scalability. 

Furthermore, I did initially share my concerns with some of the other reviewers about generalization, advance on the SOTA and some inconsistencies related to the cyclic nature of some VRPs which have been clarified by some further analysis of the high dimensional context. But due to the extension and complexity of the current approach I see those concerns more as potential extensions to the current research.

I stand convinced that this work is a valuable contribution to the field.

 Thank you very much for the feedback and suggestion. Solving significantly large instances (e.g. CVRP1000) is indeed an interesting topic that may require special designs such as divide-and-conquer, even the contribution in that aspect alone would deserve an extra publication in our opinion. And we will definitely explore that direction in the future. Meanwhile, we would like to seek the reviewers' understanding that it is still an open question whether there exists an efficient solver for significantly larger **NP-hard** problems including VRPs. 

On the other hand, as explicitly stated in the paper (including the title), we are not targeting other generic combinatorial optimization problems like the knapsack, but focused on improving neural architectures and the positional encoding method for solving VRPs. We are glad to see that the reviewer acknowledged that our work pushes the boundary for learning improvement-based heuristics for VRPs. Our DACT has achieved smaller optimality gaps and better generalization performance over different problem sizes due to the original design. The effectiveness is verified on relatively larger cases in the appendix (up to 200 nodes).

 I don't agree that a VRPTW tour is cyclic. For TSP, [1, 2, 3, 4] and [2, 3, 4, 1] represent the same tour. For a problem with time windows, [1, 2, 3, 4] and [2, 3, 4, 1] represent different orders of delivery. While it may not hurt to have cyclic encoding, I am not convinced that it helps either and I consider the claims that it 'works' for other VRP variants unsupported (where I interpret 'works' as 'having additional benefit' rather than 'won't hurt').

Note that I do think the cyclic encoding technique is sensible and indeed it may be interesting to see if this cyclic property would allow better performance using the Transformer for other tasks such as encoding DNA, which would add to the generality of the method. However, this really needs to be supported experimentally before it can be used as a claim to support this method.



 I appreciate the extensive clarifications but they do not change significantly my opinion of the paper. I encourage the authors to investigate if their improvements can be used to push the boundaries further, i.e. solve (significantly) larger instances.

 Thanks for the valuable feedback. We are gratified that you found our paper technically correct and well-motivated. Following the suggestions, we conducted additional experiments, and also addressed the remaining concerns.
***
**[Contribution]**
- Regarding *“it can be seen as making certain specific improvements to an existing setup (learning to solve routing problems by an improvement heuristic using transformers)”*, we would like to clarify that there are two major research lines in the field of *neural combinatorial optimization (NCO)* for solving routing problems, which include learning *construction* and *improvement* heuristics, respectively. Most existing works focused only on one because the MDPs for the two lines are considerably different. For either learning construction or improvement heuristics, the Transformer-style architectures (or as the backbone) are state-of-the-art. However, we found that the original Transformer is not efficient for learning VRPs especially for improvement heuristics and thus proposed the DACT. In our work, we not only propose the dual-aspect attention (DAC-Att) module in the encoder (line 185) but also the feed-forward aggregation (line 222) in the decoder. Our DACT has advanced the current SOTA on learning neural improvement solvers for VRPs in terms of the optimality gaps and more importantly the generalization performance.
- Besides, the CPE is also a novel design for properly encoding any cyclic sequences, which hold for many routing problems. Please see the response below on how to adapt CPE to other problems and construction type heuristics, and kindly refer to the general response to all reviewers on why CPE is novel and necessary to improve the generalization performance. 
- Regarding the curriculum learning strategy, we agree with the reviewer that our CL strategy might be simple. But we strongly disagree that it is of no contribution. Though the CL is an existing idea, we are the first to exploit it for NCO and proposed new designs/equations for it in line 5 of algorithm 1 (appendix D). It increases the probability for the agent to observe higher-quality solutions and thus reduces the value estimation variance of the reinforcement learning. As depicted in Figure 6, CL significantly stabilized the training and saved at least 5 times of the training time to achieve the same performance even on the relatively simple problem CVRP20. For larger problems like CVRP100 or more complicated VRP variants, the gains of CL are even more based on our observations. We believe that our CL strategy can significantly benefit the succeeding researchers in the field of NCO, because it allows for efficient training of RL, and avoids long-time exploration in stabilizing and improving the sample efficiency of PPO for solving VRPs, which we have experienced.
***
**[Is DACT general for other routing problems]**
Regarding *“it is unclear how these generalize to other problems (e.g. when adding time windows, the cyclic property seems to break) or larger instances”*, we would like to clarify that our DACT is a principled neural improvement solver for most VRP variants, including CVRPTW. Please kindly refer to the general response to all reviewers on why DACT is a general framework for VRPs for more details.
***
**[CPE for construction heuristics or other problems]**
In our opinion, we can introduce CPE to encode history (best) solutions generated by the constructive models. We believe this may provide additional useful features for the decoder of constructive models to balance the exploration and exploitation of the sampling or beam searching process. We will further investigate this interesting idea in the future. For other problems where solutions are not naturally represented as ordered sequences (e.g. knapsacks), we would like to clarify that the scope (also the title) of this paper is for routing problems and our DACT can generalize to many other routing problems as discussed above. Nevertheless, we believe our CPE has favorable potential to address other tasks that have cyclic inputs, e.g., encoding circular DNA/RNA structure in computational biology [39][40].

References:

[39] Liu, Chengyu, et al. "Biogenesis mechanisms of circular RNA can be categorized through feature extraction of a machine learning model." Bioinformatics 35.23 (2019): 4867-4870.

[40] Yu C Y, Li T C, Wu Y Y, et al. The circular RNA circBIRC6 participates in the molecular circuitry controlling human pluripotency[J]. Nature communications, 2017, 8(1): 1-15.
***
**[Performance on larger-scale instances]**
Please kindly refer to the general response to all reviewers for this point.
***
**[Regarding the Clarify]**
We apologize for the inconvenience that the reviewer cannot find crucial details of the training algorithm and 2-opt operator in the main paper. We seek the understanding of the reviewer that due to the page limit, including algorithm pseudocode in the main paper is hard to fulfill, as it occupies more than half a page. In lines 234 to 249, we included the motivation and the reason why we introduce CL to PPO, and most of the details of PPO are described in Appendix D. However, we will try to follow the suggestion, and move details from Appendix as much as possible and reduce the space for the 'standard parts' of the architecture (multi-head attention and feed-forward layers).
***
**[Adding the convergence curve]**
We will follow the suggestion and add the convergence curve of DACT against POMO in the revised paper. 
***
**[Minor comments/questions]**
[-] We will simplify Figure 2. 
[-] Figure 5 used the attention in the first layer as the behavior is similar for successors. 
[-]We will change the resolution of Figure5.

 Thanks for the acknowledgment of no obvious flaw in our work and the valuable comments. We understand that major concern is regarding the necessity of our CPE and the significance of the results, for which we respond as follows.
***
**[Whether the method is overly complicated?]**
We understand the concern about complexity. However, please kindly note that although we separate the encoding stream (NFEs and PFEs), we reduced the dimension of the network, where we use 64-dimensional embeddings for both NFEs and PFEs, while the original Transformer in [10] uses 128-dimensional embeddings. In doing so, we even have fewer parameters in total than the original Transformer (2 $\times$ 64 $\times$ 64 < 128 $\times$ 128). In particular, DACT has around 0.29M parameters while the original Transformer [10] (in ablation study) has around 0.37M parameters. Finally, disabling dual aspect representation will only marginally make inference faster (e.g., for solving TSP 100, DACT consumes 2.5h while single-aspect Transformer is about 20 mins faster). We will add time/parameter comparison in Table 2.
***
**[Are our insights very specific?]**
We would like to clarify that our **DACT is a principled neural improvement solver for most VRP variants**. The TSP and CVRP, standing as the most generic and representative ones, are chosen for study in our paper, which is similar to the latest works in this field, i.e., POMO [9] (NeurIPS2020) and CVAE-opt[12] (ICLR2021). Please kindly refer to the general response to all reviewers on why our DACT is a generic framework for VRPs. Finally, we would like to note that the insights of CPE actually enable Transformer to be better applied to other tasks besides VRP where cyclic property holds, such as encoding circular DNA/RNA structure in computational biology [39][40].

References:

[39] Liu, Chengyu, et al. "Biogenesis mechanisms of circular RNA can be categorized through feature extraction of a machine learning model." Bioinformatics 35.23 (2019): 4867-4870.

[40] Yu, Chun-Ying, et al. "The circular RNA circBIRC6 participates in the molecular circuitry controlling human pluripotency." Nature communications 8.1 (2017): 1-15.

***
**[Necessity of CPE; Why not choose existing learnable PE?]**
Regarding *“If the authors just use learnable positional embeddings, they would not need to introduce the Gray codes”*, please kindly refer to our general response to all reviewers. The absolute PE method can indeed capture the cyclic symmetry properties of VRP; however, **the ability to capture such properties significantly drops when generalizing over different problem sizes**. This is a serious drawback since it hinders the practical application of the neural VRP solver. The relative PE method seems able to help, however, it uses trainable biases and needs to fuse the positional and node features together which is found not working for VRPs in [10] (footnote 2 of page 2). As far as we know, there is no existing work proposing other types of positional encoding for Transformer which is able to accurately encode the cyclic sequences like the routes of VRPs. In contrast, our proposed CPE method enables the Transformer model to encode cyclic sequences, which also helps to significantly improve the generalization performance of neural improvement solvers for VRP (as depicted in Table 3, Figure 5, and discussed in lines 321 to 336). Moreover, this generalization performance is also significantly better than other neural solvers, e.g., AM (which is also the underlying model of POMO) as depicted in Table 4 and Table 5. 
***
**[Do Cyclical behavior exist in existing PE?]**
Regarding *“if the authors train with learnable embeddings, and the resulting learned embeddings do not exhibit the cyclic behavior, then maybe the cyclical behavior is not favored in the Transformer”*, it is not correct. As illustrated in the green box of Figure 5 and discussed in lines 331 to 336, the results showed that learned embeddings indeed exhibit the cyclic behavior for solving VRP. As mentioned above, the issue is that the ability to capture such properties significantly drops without CPE, when generalizing over different problem sizes.
***
**[Is inference time comparable?]**
- **When compared with neural improvement solvers**, the computational time of our DACT is highly comparable with them. Please note that we report the time for solving **10,000** instances (to compare with most of the baselines) but some neural improvement solvers such as CVAE-opt, NLNS, and NeuRewritter only reported time for solving **2,000** instances in their original papers. Therefore, we also report the time for solving a small batch in Table 1 marked with {}. 
- **When compared with neural construction solvers**, our methods deliver smaller gaps on both inference (Table 1) and generalization (results on TSPLIB and CVRPLIB in Appendix F.3). We acknowledge that neural construction solvers are in general faster than neural improvement solvers (which is also true for the latest improvement-type work VAE-opt at ICLR2021) since neural construction solvers pursue fast route construction in an incremental fashion. However, improvement methods usually have the potential to deliver better performance when the time budget is sufficient. For example, increasing the inference time of AM [9] from 1h to 3h on TSP100 (by adding its sampling width from 1280 to 5000) will only decrease the average cost from 7.94 to 7.93; however, our DACT achieves 7.77 within 2.5h. Regarding the most efficient baseline POMO [9], we can adopt a similar scheme (i.e., force diverse rollouts) in our DACT to find better solutions earlier. As discussed in lines 331 to 334, since the focus of this paper is to ameliorate the Transformer in the improvement models, we will investigate these interesting extensions in the future.
- **When compared with conventional solvers**, please kindly note that DACT uses **4.5h** while LKH uses **8h** on CVRP100. Meanwhile, the implementation of LKH is highly optimized and written in C while ours is written in Python. Our model could be further sped up using more efficient implementation and advanced techniques such as TensorRT. We will add more results regarding this in Table 1 in the revised paper.
***
**[Performance on larger-scale instances]**
Please kindly refer to the general response to all reviewers for this point.

 Thanks for taking a long time to review our paper and offer valuable comments. We appreciate that the reviewer finds our results promising. We hope that the following response will clear the remaining concerns.
***
**[Parameter comparison with baseline methods]**
Please kindly note that although we separate the encoding stream (NFEs and PFEs), we reduced the dimension of the network, where we use 64-dimensional embeddings for both NFEs and PFEs, while [10] uses 128-dimensional embeddings (2 $\times$ 64 $\times$ 64 < 128 $\times$ 128). In doing so, we even have fewer parameters in total than [10]. DACT has around 0.29M parameters while the network of [10] in the ablation study has around 0.37M parameters. We will add the parameter comparison in Table 2 in the revised paper.
***
**[Generalize to random initial solutions]**
The reviewer is correct that DACT could infer any randomly initialized (feasible) solutions. We choose the one obtained by nearest insertion for fair comparison because the baseline [10] also adopted it to generate initial solutions. Based on our experiments, we usually need additional 100-500 steps to improve the totally random solutions to be as good as the ones generated by nearest insertion, but this is marginal compared with the 10k steps used in Table 1. Meanwhile, please note that the solution generated by the nearest insertion is still far from optimality, e.g., the average objective value of 20.80 on CVRP100 compared with 15.64 obtained by LKH.
***
**[What if testing DACT with L > 3]**
We believe that DACT with L > 3 may produce further lower objective values, but it would become computationally inefficient considering the increase of parameters and required memory. Moreover, we selected L=3 mainly following AM [6] and Wu et al. [10] for a fair comparison, which also adopted Transformer-style encoders with L=3.

 Thanks for taking a long time to review our paper and offer constructive comments. We appreciate that the reviewer finds our work valuable and interesting. In the following, we will respond to other concerns.
***
**[Originality and Significance]**
* Regarding *“the work is clearly focused on a very particular problem”*, we would like to note that our DACT is a principled neural improvement solver for most of the VRP variants as mentioned in the response to all reviewers. Following POMO [9], CVAE [12], AM [6], Wu et. al. [10], etc, we focus on the most classic ones in the experiments, including TSP and CVRP. 
* Regarding *“It's difficult to evaluate if the work presented here advances the state of the art”*, we would like to clarify that our DACT has advanced the current SOTA on learning neural improvement solvers for VRPs in terms of the optimality gaps and more importantly the generalization performance.
***
**[Releasing code]**
We will release our code in PyTorch together with pre-trained models on GitHub upon the acceptance of this paper.
***
**[Introductions about the combination of technologies and the target problem]**
We appreciate the suggestion. However, we would like to clarify that our work is not a simple combination of existing technologies. The architecture of DACT including the dual-aspect attention (DAC-Att) module in the encoder (line 185), and the feed-forward aggregation (line 222) in the decoder were all originally designed by us. Meanwhile, the CPE is also a novel design that enables the Transformer to encode cyclic input sequences (kindly refer to our general response to all reviewers for this point). Though curriculum learning (CL) is an existing idea, we are the first to exploit it in neural combinatorial optimization (NCO), and also design new equations (line 5 of algorithm 1 in appendix D), for which we find it fairly suitable in the context of gradually learning to solve NP-hard problems like VRPs. Moreover, our targeted problem is VRP (which has hundreds of variants) for which we will add more declarations in the revised paper.
***
**[Training details on the PPO learner]**
The used PPO was detailed in Appendix D, including the pseudocode (algorithm 1), return estimation method (line 526), PPO mini-epoch update rules (line 528), PPO clipping trick (lines 529 to 530, Eq.12), RL objective function (Eq. 11) and the loss function of the critic (Eq. 13).
***
**[Adding reference paper]**
We thank the reviewer for offering those references. 
* Regarding the first one, its performance on TSP is inferior to the OR-tools (as shown in its original paper), while the baselines chosen in Table 1 all surpassed OR-tools on TSP.
* Regarding the second one, it is particularly focused on a specific VRP variant, i.e., EVRPTW (electric VRP with time window), for which their approach is not as generic as our baselines in table 1 (e.g., AM[6], POMO[9], Wu et. al.[10]).

We will include them and other relevant ones in the related works.

 First of all, we thank all the reviewers for their valuable comments. We are glad to see that all reviewers found the claims in our work well supported, and reviewers #VmAG and #vqcX acknowledged that our contributions are solid and novel. In the following, we respond to three commonly raised concerns.
***
**[Our DACT is a general framework for VRPs]**
While reviewers #VmAG and #25wv mentioned that our framework is focused on a very particular problem, we would like to clarify that our proposed *DACT is a principled neural improvement solver for most VRP variants.* We mainly study TSP and CVRP (same as POMO [9], CVAE-opt [12], Wu et. al. [10], etc.), because they are considered as the most fundamental ones in this field. For more complicated VRP variants, there should be no issue regarding our contributions including DACT architecture and CL strategy, since there are no underlying designs for a specified problem. For the proposed CPE, it works as long as the solution (routes) of the targeted VRP is a sequence of nodes that starts and ends at the same node, which holds for many other types of routing problems, including SDVRP, PCTSP, PDP, CVRPTW, etc. In particular to CVRPTW mentioned by reviewer #25wv, cyclic tour(s) are still there. We could add the starting and ending time of the windows for serving each customer to each node feature $x_i$, similar to how we add the demand of each customer for CVRP (lines 572 to 578 of Appendix). Other components can just remain unchanged. We will add more instructions on adapting our framework to other VRP variants. 
***
**[Performance on larger-scale instances]**
Firstly, we would like to note that CVRP100 is already hard to solve to optimality. In fact, the Gurobi solver (general SOTA for mixed-integer programming solver) will consume more than 1 hour to optimally solve CVRP50 instances. Therefore, instances with <= 100 customers are still used to evaluate SOTA models, e.g., such sizes (20, 50, 100) are also adopted to evaluate models in the latest work like POMO[9] or CVAE-opt[12]. Regarding larger instances, please kindly refer to Table 4 and Table 5 in the Appendix where we have already evaluated the generalization performance for instances with up to 200 customers on benchmark datasets CVRPLIB and TSPLIB.  Our DACT demonstrates significantly better generalization performance than neural *improvement* solver [10] and neural *construction* solver AM [6]. We are even better than the traditional generic solver OR-tools (e.g., 8% v.s. 3% on CVRPLIB). 
***
**[Necessity and Novelty of CPE]**
One major concern raised by reviewer #pCD7 is the necessity of CPE. In a nutshell, our CPE not only enables the Transformer to encode cyclic sequences for VRP but more importantly also ensures *a much better generalization performance of the learned solver across different instance sizes*. 
* Regarding the latter, the generalization performance for the original absolute PE is relatively poor, e.g., a gap of 16.37% in the ablation study (Table 3) and 15.56% on TSPLIB (Table 4) while the corresponding gaps of our CPE are 3.87% and 3.9%, respectively. In the visualization (Figure 5) and the corresponding analysis (lines 331 to 336), the rationale behind this was presented. We found that even though the original PE does not consider the cyclic property, the learned embeddings after training still capture this pattern (as highlighted in the green box of Figure 5(a)). This is fine when the model is trained and tested on the same sizes. However, when tested on different sizes (say TSP115), the encoders with original PE might get confused and generate random attention scores (see Figure 5(a)), thus, inferior performance. 
* The relative PE seems able to exhibit cyclic property, however, it uses trainable biases and requires the fusion of the positional and node features together, which was found not working for VRPs in [10] (footnote 2 of page 2). As far as we know, there is no existing work enabling Transformer to explicitly tackle the cyclic input sequences. Although RNNs have the potential to achieve this, they are usually less efficient in terms of both representation and computation (line 79). 
* In contrast, our CPE method could capture the cyclic property more accurately and exhibit better generalization across different sizes. When compared with the construction method, e.g., AM (which is also the underlying model of POMO), DACT also delivers superior generalization performance. In particular, the average gaps of AM on TSPLIB and CVRPLIB are 79% and 31% while ours are 2% and 3% (Appendix F.3). 
* Finally, we would like to note that CPE enables applying Transformer to other tasks besides VRP where the cyclic property is important, such as encoding circular DNA/RNA structure in computational biology [39][40].

References:

[39] Liu, Chengyu, et al. "Biogenesis mechanisms of circular RNA can be categorized through feature extraction of a machine learning model." Bioinformatics 35.23 (2019): 4867-4870.

[40] Yu, Chun-Ying, et al. "The circular RNA circBIRC6 participates in the molecular circuitry controlling human pluripotency." Nature communications 8.1 (2017): 1-15.

## Summary
The paper proposes an improvement for the Travelling Salesman problem using Transformers to compute positional information. The paper introduces a new architecture to learn separate groups of embeddings for the node and positional features following the encoder-decoder structure; this new architecture is introduced as a Dual Aspect Collaborative Transformer. By separating the node and position embeddings this transformer structure allows for the non deterministic nature of routing tasks. The paper also proposes a cyclic positional encoding method to address the non-linearity of VRPs. The DACT learner is trained using Proximal Policy Optimisation following a curriculum learning strategy in order to address stability issues.  ## Originality
I would classify the originality presented here here in the mid-range. Because even though using transformers for VRPs has been done in the past, this paper splits the information streams, usually encoded together as a key feature for the original use cases, in order to gain flexibility and therefore improving exploration.

## Quality
* The paper presents clear explanations of any new contributions made; it reads like a complete piece of work but it leaves the reader with a feeling of having expected to see some of the work left for future efforts.
* I would have liked to see more effort on adding clear introductions about the combination of technologies used and the target problem, aiding readers unfamiliar with them to get setup with constructive knowledge.
* The paper does have well supported claims.

## Clarity
* The paper presents the concepts in a clear and well organised manner.
* More reproducibility details would have been well received by the reader; for instance, a repository containing the code and clear steps for running training and inference.
* More information needs to be presented on the implementation and training aspects of the PPO learner.

## Significance
The results and ideas used here are interesting and valuable. The paper presents an interesting line of work where some more ground can be made to create solutions for TSPs. It's difficult to evaluate if the work presented here advances the state of the art, but it does present a very interesting and novel combination of methods to address a well known and difficult problem.

## References
A quick google search provided a couple of papers not listed that seem relevant: 
* https://hanalog.polymtl.ca/wp-content/uploads/2018/11/cpaior-learning-heuristics-6.pdf
* https://arxiv.org/pdf/2010.02068.pdf \
It may be worth scanning for more. \
Please provide details of how this paper relates to those. ## Limitations
The work is clearly focused on a very particular problem. Whether that focus is enough to establish an assertion of its limitations is perhaps a bit subjective.

## Societal impact
I do not foresee any negative societal impact sourced from this work. It addresses a pretty fundamental problem already present in our daily lives.

This paper addresses the vehicle routing problem. It proposes to learn the embeddings for the nodes and the positional encodings separately, where the positional encoding is learned in a novel cyclic manner. The results show that the proposed dual transformers outperform previous works.  I do not see apparent flaws in this paper. The paper presentation is detailed, the figures are nice, and the results are promising. The ablation studies properly evaluate the contribution of each component.

Building on top of [10], this paper makes several solid contributions, yet none of them can be considered groundbreaking.

There are some minor details not clear to me.
How does the proposed model compare to baseline methods in terms of parameters?
How does this model generalize to random initial solutions?
What if testing the model with L > 3?
I believe the g_n^(l) in Figure 2 should be g_N^(l)
 I do not see the potential negative societal impact.

This paper proposes a few variants to the Transformer model to learn to find solutions for routing problems. There are two main architectural changes: (1) using cyclic Gray codes instead of sinusoidal positional embeddings, and (2) using a two-stream attention function (named Dual-Aspect Collaborative Attention (DAC-Att)) instead of the multihead QKV attention.

The resulting model is trained with Proximal Policy Optimization (PPO), and is shown to find good solutions to the Traveling Salesman Problem (TSP) and the Capacitated Vehicle Routing Problem (CVRP).
  I have two main concerns with this paper.

First, the contributions seem overly complicated and whether it is necessary to go for such complexity is questionable. For instance, the use of cyclic Gray codes is motivated by the hypothesis that it better encodes the cyclical similarity which is needed when searching for tours, in the case of TSP and CVRP. However, these insights are very specific for these problems. Also, learnable positional embeddings (which is done in many modern Transformer language models), is capable of learning this cyclic behavior as well. If the authors just use learnable positional embeddings, they would not need to introduce the Gray codes. If the authors train with learnable embeddings, and the resulting learned embeddings do not exhibit the cyclic behavior, then maybe the cyclical behavior is not favored in the Transformer. Either way, I feel like the use of Gray codes is not justified.

Second, while the Dual-aspect Attention (DAC-Att) mechanism is somewhat more justified than the Gray code embeddings, DAC-Att would significantly increase the computational complexity. Perhaps this is the reason why the time stamps in Table 1 for DACT are bad. Other than a slower training and inference time, I think that DACT should compare against methods that have the same computational complexity. If no such method exists (they do), then DACT should still compare to heuristic methods that are allowed to run for more randomized trials in order to account for the complexity differences. Under such circumstances, it is unlikely that DACT will deliver any benefits, which leads to my next point.

Third, the experimental results in the paper are not convincing. In Table 1, DACT lost to many other baselines *under the same time constraint*. Understandably, the authors focus on the gap to optimal solutions, which fafor their method DACT, but the tradeoff for the smallest gaps in DACT is the much longer operating time. Furthermore, all the instances considered in the experiments are relatively small (i.e., with 20, 50, and 100 instances). More convincing experiments on larger situations, preferably the situations where previous methods fail to solve, are needed in order to prove the effectiveness of DACT.

Therefore, while the paper has no obvious flaw, I think the authors will need more work in order to justify the effectiveness of DACT. I recommend rejecting the paper.
 Very limited or none.

This paper proposes an improved Transformer-like architecture to learn an improvement heuristic for the Travelling Salesman Problem (TSP) and Vehicle Routing Problem (VRP). It proposes a special form of positional encoding that represents the cyclic structure of a current solution and proposes to learn separate embeddings for 'node' and 'positional' features (in each layer) which only exchange information through an attention matrix. It uses a PPO algorithm with a curriculum learning strategy. Experiments on TSP and VRP with up to 100 nodes show reasonable performance compared to existing alternatives.

While the paper presents an improvement to the results for TSP and VRP, in my opinion it is too incremental to be accepted in its current form, as it largely reuses existing architecture and experimental setup, but with improved positional encodings/embeddings and a simple curriculum learning strategy. It is unclear how the proposed improvements (which share some similarity with United Positional Encoding) can generalize to other problems or how this will affect the ability to solve larger instances, which I think should be the main goal of this line of research, as the current SOTA is not yet competitive with alternative methods. Given the quality of the work I encourage the authors to continue but I think the paper should be improved to show this potential before it can be accepted.  Originality

The paper is somewhat incremental as it can be seen as making certain specific improvements to an existing setup (learning to solve routing problems by an improvement heuristic using transformers). Specifically, the improvements made are the 1) using cyclic positional features, 2) separate embeddings for positional and node features (somewhat related to United Positional Encoding) and 3) using PPO with a simple curriculum learning strategy.

Clarity

The paper is quite clear and well written in general, although I find that it should balance the amount of space dedicated to different topics. Much space is used to describe the cyclic positional encoding and equations relating to the 'standard parts' of the transformer architecture (multi-head attention and feed forward layers), whereas to understand the proposed method I would find it much more helpful if the pseudocode in the appendix D was included in the main paper, as well as the description of the '2-opt' operator (reversing the segment between selected nodes i and j), which are crucial details of the used methods.

The Figures are a bit scary to look at at first sight and could be simplified possibly, but are very helpful in understanding the architecture, positional encoding, etc.

Quality

The work is technically correct and well-motivated. The experiments are well-conducted and carefully reported to deal reasonably with differences between evaluations of different methods. They support the claim that the architecture outperforms existing improvement models, although this should not be mistaken for 'outperforming VRP' or neural VRP, in which case POMO is a reasonably simple and strong baseline. To give a better picture of where it stands, it would be good to report full performance vs time curves against some of the strongest baselines, which can be easily obtained by running with a different number of iterations.

Significance

The results push the boundary for learning improvement based heuristics for VRP which is promising. However, I consider the main contribution the architectural improvement as using PPO with a curriculum learning strategy is rather straightforward. While promising, it is unclear how these generalize to other problems (e.g. when adding time windows, the cyclic property seems to break) or larger instances, which would ultimately be the goal as the method is not yet competitive to VRP. DACT mainly exploits the fact that the input is a (circular) sequence, so I do think it would be good to at least hint at how this could extend to construction heuristics or other problems where solutions are not naturally represented as ordered sequences (e.g. knapsacks).

Minor comments/questions

- Please check spelling (e.g. matmal -> matmul Fig 2.)?
- Line 288 -> multiple GPU in (.) and single GPU in {.}, I think this should be other way around?
- Which attention exactly is shown in Fig. 5? Is it averaged over layers?
- Scrolling past Fig. 5 lags on my PC. If others experience this as well it may be worth saving the images differently for convenience of your e-readers. Limitations are discussed in the discussion. I do not see any direct negative societal impact.

Meta-review: Based on reviews, rebuttal and discussion, I recommend acceptance of this paper. Based on reviews:

* There is no fundamental flaws
* The overall paper is clear
* The experiments are well conducted
* The rebuttal addresses the main concerns

While the significance of the contributions are borderline, it is sufficient to inspire new works in the same direction and is likely to have positive impact on the sub-community.

Decision: accept