Boosting Neural Combinatorial Optimization for Large-Scale Vehicle Routing Problems
Authors: Fu Luo
Xi Lin
Yaoxin Wu
Zhenkun Wang
Tong Xialiang
Mingxuan Yuan
Qingfu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Neural Combinatorial Optimization
Large-Scale Vehicle Routing Problem
Primary area: applications to robotics, autonomy, planning

Abstract: Neural Combinatorial Optimization (NCO) methods have exhibited promising performance in solving Vehicle Routing Problems (VRPs). However, most NCO methods rely on the conventional self-attention mechanism that induces excessive computational complexity, thereby struggling to contend with large-scale VRPs and hindering their practical applicability. In this paper, we propose a lightweight cross-attention mechanism with linear complexity, by which a Transformer network is developed to learn efficient and favorable solutions for large-scale VRPs. We also propose a Self-Improved Training (SIT) algorithm that enables direct model training on large-scale VRP instances, bypassing extensive computational overhead for attaining labels. By iterating solution reconstruction, the Transformer network itself can generate improved partial solutions as pseudo-labels to guide the model training. Experimental results on the Travelling Salesman Problem (TSP) and the Capacitated Vehicle Routing Problem (CVRP) with up to 100K nodes indicate that our method consistently achieves superior performance for synthetic and real-world benchmarks, significantly boosting the scalability of NCO methods.

Published as a conference paper at ICLR 2025
BOOSTING NEURAL COMBINATORIAL OPTIMIZATION FOR LARGE-SCALE VEHICLE ROUTING PROBLEMS
Fu Luo', Xi Lin", Yaoxin Wu", Zhenkun Wang Tong Xialiang", .Minxuuan Yuan", Qingfu Zhang²
Southern University of Science and Technology
PCity University of Hong Kong
EEnnhooven University of Technology
*Huawei Noah's s Ark Lab
luof 1 sustech edu. cr, xi ctt.mmm..citt.eeeeee y. wu2@tue .nee.nn nn, sangahenkun90@@mmal. com, tongxialiangghuuwei.... com, yuan. mingxuan@huawee com, qingfu. zhang@cityu.edu.
ABSTRACT
Neural Combinatorial Optimization (NCO) methods have exhibited promising performance in solving Vehicle Routing Problems (VRPs). However, most NCO methods rely on the conventional self-attention mechanism that induces excessive computational complexity, thereby struggling to contend with large-scale vRPs and hindering their practical applicability. In this paper, we propose a lightweight cross-attention mechanism with linear complexity, by which Transformer network is developed to learn efficient and favorable solutions for large-scale VRPs. We also propose Self- Improved Training (SIT) algorithm that enables direct model training on large-scale VRP instances, bypassing extensive computational overhead for attaining labels. By iterating solution reconstruction, the Transformer network itself can generate improved partial solutions as pseudo-labels to guide the model training. Experimental results on the Travelling Salesman Problem (TSP) and the Capacitated Vehicle Routing Problem (CVRP) with up to 100K nodes indicate that our method consistently achieves superior performance for synthetic and world benchmarks, significantly boosting the scalability of NCO methods. The code is available http:://ggttbbccom/MAMMMGoooooo
INTRODUCTION
The vehicle routing problem (VRP) a typical type of combinatorial optimization problem (COP) and is often encountered in numerous real-world applications (Garaix et al. 2010 Brophy & Voigt 2014, Elgarej et al., 2021). Due to the NP-hard nature, solving VRPs remains extremely challeng- ing (Ausiello et al. 2012). Traditional methods are generally hindered by their heavy reliance domain expertise and tuning work in algorithm design. Meanwhile, they often suffer from low computational efficiency that hampers their applicability large-scale VRP instances.
Recently, the neural combinatorial optimization (NCO) methods for solving VRPs in an end-to-end manner have attracted considerable attention (Bengio al. 2021]. These methods build deep neural models to automatically learn problem-solving policies from data, significantly mitigating the need for costly manual effort algorithm design. The learned policy can efficiently generate approximate solutions for VRP instances. NCO methods have gained comparable or even superior performance to traditional methods on amall-scale problem instances with no more than 100 nodes (Kool et 2019 Kwon et al., 2020; Hottung et al., 2022). especially on Traveling Salesman Problem (TSP) and Capacitated Vehicle Routing Problem (CVRP) instances.
Nevertheless, existing NCO methods often struggle when applied to large-scale VRPs. Some efforts have been devoted to training neural models on larger VRPs with up to 500 nodes (Jin et al. 2023 Zhou et al. 2023), aiming to enhance their generalization for solving large-scale VRPs. However, the
"corresponding author

---

Published as a conference paper at ICLR 2025
difficulty of training will increase drastically as the size of the problem grows, resulting in the inability to obtain sufficient generalization capabilities. Consequently, some methods resort to simplifying
large-scale VRPs via decomposition or learning local policies (Pan et al. 2023 Ye et al. 2024
Gao et al. 2024 Fang et al. 2024). The decomposition- baaddd subproblem solver can be trained by learning construct either complete solution of a small-scale VRP or partial solutions (e.g..
some segments of TSP solution) of a large-scale one (Kim et al., [021] Cheng et al., 20233 Luo al. 2023). The method based on local policy reduces the decision space into the current node's
neighborhood in each construction step.
Despite the above efforts in solving large-scale VRPs, current NCO methods still suffer from two obstacles in terms of scalability. First, they usually rely on the conventional self-attention mechanism with high computational complexity (Vaswani et al., 2017). which severely restricts the model's efficiency in constructing a complete solution or multiple partial solutions with large sizes for a large-scale VRP. Second, their models are often trained in supervised learning (SL) or reinforcement learning (RL) manner, both of which grapple with effective training large-scale VRPs. On the one hand, the SL-based NCO has difficulty in obtaining sufficient (near)-optimal solutions as labels; on the other hand, the RL-based model training suffers from severe sparse rewards well high GPU memory usage.
In this paper, we propose lightweight cross-attentin mechanism with linear computational com- plexity, which can significantly improve the efficiency of the NCO model in solving large-scale VRPs. Unlike the conventional self-attention that makes each node attend to all the other nodes in instance, the cross-attention reforms the computational process through representative nodes. In particular, the representative nodes first attend to each node of an instance to update their own embeddings, and then the instance nodes' embeddings are updated via attending to the representative nodes. With a fixed number of representative nodes, the computational complexity is greatly reduced in comparison to the conventional self-attention while maintaining effective attention computation between nodes. Based on the proposed cross-attention mechanism, we develop a novel Transformer network for solving large-scale VRPs more efficiently. In addition, we propose an innovative Self- self-Improved Training (SIT) algorithm that empowers our model to be successfully trained on large-scale instances. The SIT employs the Transformer network itself to refine the solution via iterative reconstruction. The improved solutions, in turn, serve as pseudo-labels to train the network. By iterating solution reconstruction and network training, the SIT enables our NCO method to effectively solve large-scall scale VRP instances without any labeled data.
We conduct comprehensive experiments on both synthetic and real-world TSP and CVRP benchmarks. The results demonstrate that our NCO method achieves state-of-the-an performance on large-scale VRPs with up to 100K nodes. Our ablation study reveals the effect of the cross-attention and SIT algorithm in improving computational efficiency and solving performance for large-scale VRPs.
2
RELATED WORK
2.1
GENERALIZATION-I BASED METHOD
The generalization-base methods usually train the neural models on small-scale instances and then test them on the same-scale or larger-scale instances. It generally refers to the construction- based method that learns model to construct approximate solutions for given problem instances in an
autoregressive manner. Pioneering works (Vinyals et al.) 20155: Bello et al. 2016 Nazari et al.) 2018)
show that neural models trained with supervised learning (SL) or reinforcement learning (RL) can
achieve promising results on small-scale CO problems. Kool et al. (201) and Deudon et al. (0018)
leverage the Transformer structure Vaswani et al. (2017) to develop powerful iitention-basee models to solve small-scale VRPs. Since then, various Transformer-basee
methods have been proposed with different improvements (Xin et al. 2021a; 2020 Kwon et al. 20200) Hottung et al. 2022 Kim et al.. Choo 2021,CChoo et al., 2022. Manchanda et al. (202).. Subsequently. many studies attempt to enhance the performance of neural models on large-scale VRPs (Son et al. 2023: Zhou et al., 2023 Drakulic et al. 2023 Luo et al. 2023). Among them, Drakulic et al. (2023) :: Luo et al. (2023) employ SL to train the model on 100-node and equip the model with good generalization ability on VRPs with up to 1K nodes. BQ reformulates the Markov Decision Process (MDP) of solution construction to effectively leverage common symmetries of COPs, while LEHD proposes a light encoder and heavy decoder structure to achieve the same goal. However, since the distribution of 100-node instances
2

---

Published as a conference paper at ICLR 2025
differs drastically from that of instances with more than 10K nodes, the features learned from such scall-scale instances are not applicable to very large-scale instances, resulting in poor performance on instances with more than 10K nodes. Recent attempts propose train models on larger-scale instances with up to 500 nodes (Jin et al. 2023: Zhou et al. 2024 Wang et al.; 2024; Zhhou et al. 2023). However, the training diffieulty increases dramatically as the problem sizer grows, resulting in the inability to obtain sufficient generalization capabilities.
2.2 SIMPLIFICATION BASSDD METHOD
Some methods resort to simplifying large-scale VRPs via decomposition or learning local policies. On the one hand, decomposition-basee methods generally transform a surge-scale problem into multiple simpler small-scale subproblems, solve them, and then merge their solutions to construct the complete solution of the original large-scale problem. These methods propose different strategies to learn individual models for problem decomposition and subproblem-solving respectively (Li et al.
2021 Zong et al., 2022 Hou et al. 2023 |ann et al.) 2023 Ye et al., 2024). The decomposition- based subproblem solver can be trained by learning to construct either a complete solution of a small-scale VRP partial solutions (e.g., some segments of a TSP solution) of large-scale one (Kim et al. T
Cheng et al. 2023 Luo et al., 2023). On the other hand, the local policy-bass baaicy-based method reduces the deciston space into the current node's neighborhood in each construction step. While al. (2024) adopt an auxiliary policy to bias the model to make decisions within the current node's 's neighborhood,
Fang et al. 20244) directly restricts the decisions the neighborhood.
2.3 HEATMAP-BASED METHOD
In addition to the above works, some heatmap-basee methods are proposed to address large-scale TSP instances (Fu al. | Qiu et al. 2022: Li al. 2023 Min al. 2023 Sun Yang 2023). This kind of method first builds graph neural network (GNN) model to predict a heatmap that measures the probability for each edge to be in the optimal solution, and then iteratively searches for an approximate solution using the heatmap (Joshi et al.) 2019). Since they rely on the search strategy (e.g., MCTS (Fu et al., 202T)) specifically designed for TSP, they cannot be applied to solve other complicated CO problems such as CVRP. In this work, we primarily focus on the construction-basel method without requiring expert knowledge.
3 PRELIMINARIES
VRP Definition. A VRP instance S can be represented by graph G (V,E), where V = (u; T denotes the node set and 3 = {(,,, uy)|us, lu. Uj E V, & vÿ} denotes the edge set. In particular, vo denotes the depot in some problems (e.g., CVRP).
On the VRP graph, each node is featured by vector E Rd:, with the elements including the node coordinates and other problem-specific attributes (e.g., demands { in CVRP). A solution to a VRP instance is a tour æ, which is a permutation of the nodes. Given a cost function e(-), solving the VRP instance is to search the tour with minimal cost, i.c., π = arg min c(TlG), from the feasible tour set Ω. Specifically, the cost function of TSP and CVRP is defined as the Euclidean length of the tour in this work.
A VRP solution is feasible if it adheres to problem-specifii constraints, c.g., feasible solution to a TSP instance is a tour that visits each node in the graph exactly once. Constraints in the CVRP further entail the limited capacity of vehicle, which are detailed in Appendix
Solution Construction for VRP. Most NCO methods adopt encoder-deecoder-based neural networks to learn the solution construction. The encoder produces node embedding h; for each node With node embeddings (h, o the decoder sequentially constructs the solution by appending one node to the partial solution at each step. In particular, if none of the nodes have been visited, the partial solution is empty. At each construction step, one node is selected from the unvisited nodes to be added to the partial solution and marked as visited. For example, partial solution at step can be represented by (T) T22 -t-1), where TT - E V are the first and last visited node. The process continues until all nodes are visited and the complete solution is returned.
3

---

Published as a conference paper at ICLR 2025
2 g g
R E g
Selection Probabiltties
Finst Nade
Last Nade
Reprosentative Node
P1
Unvisited Node
Sefmax
Linear Projection
CrosssAlteriin Modale
Deaxler
Embedking Laver
Self-Attemio: O(n²)
Cross-Anestio:: O(2n)
20-1 good
f ged
(a) Idea of the Proposed Cross-Attention.
(b) Cross-Attention Module.
(c) Transformer Network.
Figure I: in (a), we employ it to design the cross-attention module (b), which utilizes a certain number of representative nodes lighten the attention in the Transformer network in (c). The resultant Transformer network significantly reduces the amount of computations between nodes, potentially achieving more efficiency in solving large-scale VRPs than current self-attention-basee Transformers.
Transformer network. Given less complexity of the cross-attention
(Self-)Attentiion Mechanism. Given the embedding matrices X Raxd and Rixad where is the dimension of the embeddings, the scaled dot-product attention can be formulated as:
XWQ XWo(CWR)) CWv,
x = Attn(X, C) softmax
(1)
Vd
where Wq. WK, Wy Rddd learnable matrices. The attention Attn(-.)) aggregate the information from C to X. The commonly used Multi-Head Attention (MHA) further performs attention computations in parallel. We omit h for simplicity throughout this paper. In general, the most complexity of the conventional Transformer architectures originates from calculating the attention in Eq with (nm) computational and memory complexities. Especially if the embedding matrices are identical (i.e., X = C), Eq. defines the self-attention with (n complexities.
METHODOLOGY
In this section, we first introduce the cross-attention mechanism tailored for solving VRPs. Then, provide the cross-attention-baell Transformer network and the SIT algorithm. Without loss of generality, we present our method by taking TSP as an example. The implementation details for CVRP are described in Appendix
4.1 LIGHTWEIGHT CROSS PROSS-ATTENTION FOR VRPs
As shown in Figure aa), the self-attention (stacked in the encoder or decoder) enforces each input node interact comprchensively with all the other nodes of VRP instance for updating their embeddings (won et al. 2020 Pirnay & Grimm 2024 Luo et al. 2023). However, it inevitably the number of nodes in an
results in (nn)) computational and memory complexities, where instance, i.e., the problem size. As the size increases, these complexities increase drastically, making the model can be hardly trained on large-scale VRP instances. Our empirical results on representative NCO methods verify their quadratic complexities, as shown in Table in the experiments.
To address this issue, we design lightweight cross-attention mechanism that significantly reduces the computational and memory complexities. As shown in Figure Ia), we use two representative nodes the attention calculation. The two representative nodes are first updating their embeddings based on the attention between them and all nodes of the instance. Subsequently, the node embeddings of the instance are updated by conducting attention calculations on the representative nodes. Compared to self-attention, this cross-attention owns O(nm) complexity, where m is the number of representative nodes. Thanks to propagating node embeddings through representative nodes, the cross-attention mechanism maintains effective interactions between nodes while achieving low complexity.
4

---

Published as a conference paper at ICLR 2025
4.2 TRANSFORMER NETWORK
We develop coss-attention-ooe based Transformer network for solving large-scale VRPs, following the heavy decoder paradigm in (Drakulic al., [023] Luo et al. (023). As shown in Figure [T].. the
network consists of single embedding layer and decoder with 7. stacked cross-attention modules, which are delineated in the following.
Given VRP instance S with n nodes, the embedding layer transforms node features {x,}² = E Rgxx to initial node embeddings (he E RRaxd by linear projection such that h, ( x; ((0,, vi E mn. where W Rda and (()) E Id are learnable.
Embedding Layer.
Decoder with Cross-Attention Modules. At the t-th decoding step. select the first and last nodes M1, -t-1 from the current partial solution ( Rt-1) as the representative nodes. These representative nodes reflect the dynamics in the solution construction process, which are commonly used as the context nodes in the literature (Kwon et al., 2020; Luo et al.) 2023] Kool et al.) 2019). We also explore other configurations of representative nodes and assess their impact performance. More details can be found in Appendix C
Given the representative nodes, we design the cross-attention module advance the node embeddings, as depicted in Figure [[b). At the t-th decoding step, the embeddings of the first and last nodes denoted by hs, and hx,-1 respectively. The embeddings of unvisited nodes are denoted by H² - (h.li (I....,
,.....,,7 n M [ At the first decoding step, one node æi randomly selected to be the partial solution, and we view it as both the first and last nodes for the next decoding. All the other nodes remain unvisited, i.c., H (h,li E (1...., n) \ Accordingly, we define the initial representative node embeddings (0) and graph node embeddings (0) at the f-th decoding step as
7 [h,, W1: h W2).
(0)) [h., Wi, h W2, Hy)
where [: means the vertical concatenation operator, and W1, W2 Raddd are learnable matrices. Next, Z E 2 and (0) E Nxd are processed by cross-attention modules. In the 1-th cross-attention module, first apply representative nodes attend the graph node embeddings (1-1)) to update the representative node embeddings, i.c.,
2 Attn (l (1(l1))) + ((1-1)
( (F()")) + 200),
where FF denotes the ferd-forward layer that is formulated as FF(X) max XW + by2, where Wf1 E Rdxdd byi E Rd4 W/2 E xd,xd, byz E RR are learnable matrices, and d is the hidden dimension of the layer. Subsequently, the graph node embeddings (1-1) attend the representative nodes for updating their embeddings, i.e.,
O Attn ( (1-1), 1),2()))))))))))))))))) + (1-11)
(4)
H ( () ())
After L cross-attention modules, the output (4) include the advanced embeddings of the first, last, and unvisited nodes. A linear projection and softmax function are applied to embeddings of unvisited nodes (6(2)) li node, i.c..
} in order to produce probabilities of selecting each unvisited
U4 Wo. i {T
otherwise
(5)
p softmax(u),
where Wo Rdx1 is learnable matrix. Each p; P corresponds to the probability of selecting the unvisited node i. We sample the node by the probabilities and add it to the partial solution. A complete VRP solution ( n is constructed with decoding steps.
5

---

Published as a conference paper at ICLR 2025
Complete Solution
Partial Solution
Enhanced Partial Solution
M
Local Reconstruction
Sample
Reconstruction Performed by the Model
Remove Solution Information
Subgraph
Selection Probability
Pseudo-Labbl
Training
Model
Loss
Optimine
First Node
Last Node
Unvisited Node
Selection Probability
Figure 2: The self- self-improved training process. In each iteration, the neural model performs multiple local reconstructions (in parallel) improve the solution quality, and then, the enhanced partial solutions as pseudo-labels are used to train the model to improve its performance.
Complexity Analysis. According to Eq. B). the dimensions of input to cross-attention modules are R2dd for (t--1) and R for ( where ñ < (n + 1) is the number node embeddings input the decoder. Except for the constant (d 128 for most Transformers in NCO methods), the cross-altention between node embeddings z and ( yields a complexity ((ñî). Similarly, 4 exhibits the same linear complexity. By setting fixed number of representative nodes, the cross-attention significantly reduces the computations between all nodes, which more lightweight than prevailing self-attention- Transformers in NCO methods.
4.3 SEPF-VE TRAINING
Construction-basse NCO models exhibit bias in decoding, where variations in starting nodes, desti- nation nodes, and directions result in vastly different solutions (Kwon al. 2020). Benefiting from this bias, the model can gradually improve the solution quality by performing iterative local reconstruction until convergence (Luo al. 2023 Ye et al.) 2024). This paradigm holds significant potential to efficiently discover superior solutions without the need to explore complete solutions. However, current local reconstruction techniques still rely SL RL, which hinders their applica- bility to large-scale VRPs due to the scarcity of labels or the sparsity of rewards. Instead, we propose Self- lelf-improved Training (SIT) algorithm to specialize in local reconstruction for more effective solution exploration on large-scale VRPs. As illustrated in Figure 2 SIT involves iterative local reconstruction and model training, which are elaborated the following.
Local Reconstruction. The local reconstruction comprises two steps. In the first step, partial solution T?" of random size < w S. Imax is sampled from ग, where lmar refers to the maximum size of the partial solution. Since can be expressed or counterclockwis. sampling direction. In the second step, the neural model reconstructs the partial solution node by node from its first node to the last node, i.e., the order of nodes between the first and the last nodes is rearranged. This generated partial solution nh is compared with лτ, so that the better one (e.g., the one with shorter length) is adopted in the complete solution π. In other words, better partial solution can result in better complete solution. Through iterative local reconstructions, the quality of the solution can be significantly improved.
circle of nodes, we allow either clockwise
To improve the reconstruction efficiency, M non-overlapping partial solutions are sampled and reconstructed in parallel. The overlap is avoided by evenly dividing the solution into M consecutive segments of equal length w, in which the sampling and reconstruction can be parallelly performed. In this paper, we set M to [n/w] with n denoting the problem size. Throughout the SIT process, we maintain dataset D containing VRP instances and their solutions updated by local reconstructions. In each SIT iteration, the local reconstructions progressively enhance the quality solutions in the dataset, which avoids revisiting previously explored solutions and thus improves the efficiency.

---

Published as a conference paper at ICLR 2025
Model Training. The enhanced solutions from local reconstructions serve pseudo-labels for training the model in a supervised manner. For a large-scale VRP instance (e.g.. TSP instance with 100K nodes), the learning for constructing its complete solution can be difficult due to the massive GPU memory usage. To relieve the issue, we restrict the model's learning scope to local parts of the solution. Specifically, a random size 4 w S Iaar is adopted to sample partial solutions from the dataset D. Let 1i = (#t, -g..... =g) be a sampled partial solution. Taking it as a pseudo-label, the model learns predict the order from #p to # using the proposed Transformer network parameterized by 0. The loss function can be formulated as
((0) = EEs=-D- log SS, * (f: #5)] VL (2. (2....,. -};
(6)
where #i and #g denote the first and last nodes of the partial solution, and their embeddings constitute the representative node embeddings and graph node embeddings in Eq. - represents the sequence of visited nodes till the t-th decoding step. By training with the partial solutions, the model can be more efficient in enhancing solutions during the local reconstruction.
The SIT process alternates between the local reconstruction and the model training, until predefined time budget is reached. The detailed training process and pseudocodes are provided in Appendix D
EXPERIMENTS
We empirically evaluate the proposed method from two perspectives. Firstly, we compare the proposed method with diverse bascline methods to demonstrate its performance on synthetic and world large-scale TSP and CVRP instances. We then analyze the impact of the proposed method's 's critical components to verify our method's capability in reducing computational and memory complexities and analyze key hyperparameters.
Dataset. Following the common approach in literature Kool et al. (2019); ; Kwon et al. (2020); Luo et al. (2023), we generate five synthetic datasets with instances of scales TK, 5K, 10K, 50K, and 100K, respectively. We denote TSP and CVRP instances of these scales as TSP/CVRPIK, 5K, 10K, 50K. and 100K, respectively. According to Fu et al. (202T), we set the number of instances for the TSPIK test dataset to 128. For datasets with larger instances, each contains 16 instances. Similarly, the CVRP test dataset includes the same number of instances, with capacities set to 250 for CVRPIK. 500 for CVRP5K, 1, 0,0 for CVRPIOK, and 2,000 for CVRP50K/100K. The optimal solutions of TSP instances are computed using LKH3 (Helsgaun) 2017), while CVRP instances are solved via HGS [Vidal) 2022). To evaluate our method on real- woal-world large-scall instances, we also extract all symmetrie instances with EUC 2D features and more than IK nodes from TSPLIB Reinelt 1191I) and CVRPLIB Uchoa et al. (0177), a total of 33 TSP instances and 14 CVRP instances.
Model Training. For our Transformer network, we set the embedding dimension d = 128. The decoder employs L = 6 stacked cross cross-attention modules, with each attention layer including 8 attention heads and feed-forware layer with hidden dimension of 512. The model initially undergoes a warm-up training process on instances of scale 1K using the pseudo-labels generated by random insertion (see Appendix DD. After that, we continue the self-improved training on instances of scale IK and then leverage the trained model to conduct separate training on larger scales, including 5K, 10K, 50K, and 100K.
The size D of the training dataset for scales IK, 5K/10K, 50K/100K are 20K, 200, and 100, respec- tively. Each iteration in our SIT algorithm comprises 100 times of local reconstruction and 20 epochs of model training. The Adam optimizer (Kingma & Ba, 2015) is utilized for training the models, with an initial learning rate lc-4 and decay rate 0.97 per epoch. Throughout the SIT process, the maximum length of partial solutions Imaz is 1, 1,000 to balance efficiency and effectiveness. In all experiments, we use single NVIDIA GeForce RTX 3090 GPU with 24GB memory for both training and testing. We provide more detailed training settings in Appendix E
Baselines. We compare our method with 1) Classical Solvers:
Concorde (Applegate et al. 2006).
LKH3 (Helsgaun) 2017). and HGS (Vidal 2022); 2) Insertion Heuristie: Random Insertion:
3) Construction based NCO Methods: POMO (K won et al. 2020). BQ (Drakulic et al. 2023).
LEHD (Luo al. 2023), INVIT (Fang et al.
2024) and SIGD (Pirnay & Grimm 2024); 4) Heatmap-
based Methods: DIFUSCO (Sun & Yang
2023): 5) Decomposition-baser Method: GLOP (Ye
al. 2024), and H-TSP (Pan al. 2023
Local Policy-based Method: EL. (Gao et al. 2024)
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparative results on synthetic TSP and CVRP instances. *; results are cited directly from original publications. N/A: the method exceeds the time limit (e.g., seven days) or produces infeasible solutions. OOM: the method exceeded memory limits.
TSP5K
TSPIOK
TSPSOK
TOP
Method
Obj. (Kāap)
Time
Oby (GGup)
Time
(Bg (Caap)
Time
Objy (Giap)
Time
UBy. (Caap)
LKH3
23.12 (0.005))
1.7m
(OLDWHE)
12m
71.78 (8.008))
159.93 (0.00))
10h
225.99 (0.00R5)
Concorde
23:12 (0.00%)
1m
50.95 (-0.059)
31m
72.00 (B. 15%)
1
N/A
NIA
NIA
N/A
Randocn Insertion
26.11 (12.9%)
<3s
58.06 (33.9%)
<1s
$1.82 (13.9%%
<3s
182.65 (34.216)
15.45
(14.28)
1,7m
DIPUSCO*
(3.19%)
13.5s
73.02 (2. (2.585))
3,0m
H-TSP
24.66 666))
48c
55.16 (8.21 5%)
1.2m
77.75 (8. 38%)
2.5m
00M
DOM
GLOP
23.78 (2.85%)
10.2s
53.15
(4.26%)
1.0m
75.04
(4.39%
1.9m
168.09 (5.101))
1.5m
237.61
(5.14%)
3.9m
POMO ang
32,51 (40.65)
4.1%
87.72
(72.1
8.6m
0OM
OBM
00M
ELG a
(1..38
0.8s
60.19
(18.08
21s
0OM (12.58)
00M
0OM
LERD RRC1,000
23.29 (0.72%)
3.3m
(6.79%))
8.6m
18.6m
OUM
00M
bi16 SIGD
23.43 37%)
24c
00M 00M
00M
DOM
(1.06))
17.3s
(9:42%))
30.5m
00M
00M
IY greedy
26.66 (6.06%)
9.5s
(6.901))
1.2m
76.85 (7.07%)
3:7mm
171.42 (7.1886)
1.3h
242.26 (7.20%))
gready
(3.19)
0.8
58.85
(15.466))
1.5m
91. 03.33/2 (27.84%)
11
OOM
DOM
BQ greedy
23.6512.30%)
0%
58.27
(14.31%)
22.5s
89. 79.73 (25 (2%)
1.0m
DOM
DOM
SIGD greedy
23.573 (3. 96%)
1.26
57.39 (12.20%)
1.8m
93.800 (30.009))
15.5m
00M
0OM
Ours greedy
23.509 (1 95%)
0.26
52.59
(3.17%)
5:2s
74.69 (4.055%)
20:14
168 15.3666 )
3.7m
239,84 (6.1558)
33,dm
Ours PRC10
23.306 (1 20%%
0.5%
52.36 (2. 73%%
5.1s
99 (3 .088))
10.04
166 69 (4 (4.22%)
1.93m
255.38 (4. (4.169))
3.0m
Ours PRC50
23.279 (0.691%)
4.66
51.92 (1.85%)
23.45
72.41
(2.29%)
49,0s
165.00 (3.1716)
4.9m
233.13(3.1665
9.2m
Ouss PRCT00
23.254 (0 (
9.46
51.82
82 (1.67%)
$2.0%
73.29
(2.1196)
1.7m
164.3 (2.91%)
5.6m
232.55
(2.90%)
17m
Ours PRC350
23,217 (0.43%)
46%
51.70
(1.43%
1.6m
73,12 (1 (1.87%)
8.5m
164 (2.60%)
42.2m
231.75 (2.55%)
1,4h
Ours PRC11000
23,207 (0.38%)
(1.36))
9.4m
(1.81%)
1.38h
251.52
CVRPIK
CVRPSK
CVRPIOK
CVRPSIK
CVRP100K
Method
Obj. (Gxg)
Time
Obj (Gap)
Time
Obj. (Gap)
Time
Obj (Gap)
Time
Obj. (Gup)
Time
HGS
36.29 (0.00%)
2.5m
89.74 (0.00%)
2.0h
107.40 (0.00%)
5.0b
267. (0.0016)
8.1h
476.11 (0.00%
24h
LKH3
37,00 (1))
3,3m
93.71 (5.19%))
1.33h
118. 76 (10 (96)
1.741
399 12 (49.15)
15.8h
N/A
N/A
Randem Insertion
(58.2155.29) 2%)
<ls
154.38 (72.0%)
<Is
191.80 (78.6%) (8.6
490.56 (83.2%)
<Is
943.87 (98.3%)
GLOP-G
1.36
38.90/15 (10.25%)
6.56
116.28
(8.27%)
11.24
00M
0OM
POMO ang
84,50
4.8
(3385))
GOM
O0M
DOM
ELG a
56%)
109
(22
30s
00M
00M
0OM
RRC1,000
(3.151)))
3.4m
(12.6%)
31m
138.73 (29.28)
0UM
OOM
he16
38.17 17%)
14c
(16.55 36)) 103.46
2.6m
GOM (22.4%)
00M
DOM
SIGD bs16
(7.19%7)
17.3s
(15.3%)
1.91m
131.48
3.97m
477.43
(78.3%5)
25.9m
00M
INVI 3V groedy
42.75 (17.8%%
11.4s
109.85
(22.415)
1.4m
141.41
(31.60%)
4.2m
402.08
(50,17%)
2.hh
688.80
(48.57%))
8.3b
LEHD groedy
38.91 (7.23%)
0.8
105
(17 6998)
1 1.56m
146.24 (36. 1665)
11 15.55m
D0M
DOM
BQ gredy
(8.23%)
1.00s
108
09.009900
(20, 48%)
8.1s
196, 64 (82 9%
1.2m
00M
DOM
SIGD gready
(10.7%)
1.25
196.14
(18.5%)
7.5%
135.12 (25,8%)
45s
493.64 (54.015)
4.3m
0OM
Ours gready
(5.055)
0.2s
92:44 (3.01
5.4%
109 02 (1.50%)
20.624
269 34 (0.60%)
8.06m
475
22%%
33,
Ours PRC1O
37,93 (4.32%)
0.7s
(4.65%))
3.5%
(4.43%)
285 (6.3216)
28s
498 (4,2396)
59%
Oura PRC:5
37.57 (3.5444)))
3.56
19.0%
108, 76
271.37 (1.51%)
3.3m
476.71
(0.
Ours
(3.31))
91.58
(2.05%%
46c
(0.59%
1.3m
(0. 11%)
5.49m
3.00%)
11.5m
Ours
37.33 (296)
44.64
91.00 (3.41%))
4.4m
106.85 (40:5316)
7.000
56 561))
31.1m
(-2.3016)
Oues PRC1 .oon
(2.725))
1.5m
90.81 (1.19%)
8.8m
106.69
(-0.66 ()
15:2m
262.82 (-1.83%)
1.04h
463.95 (-2.55%)
2.17h
For comparison, we provide the average objective value (Obj.), optimality gap (Gap), and inference time (Time) of each method. Obj. indicates the length of the VRP solution, with shorter values indicating better performance. Gap measures the solution difference from the ground truth (i.e., results produced by LKH for TSP and HGS for CVRP). Time, recorded in seconds (s), minutes (m), or hours (h), reflects the efficiency in generating solutions for test instances.
Metries&Inferrnee
For our method, we present the results of the greedy search and Parallel local ReConstruction (PRC) under different numbers of iterations. For PRC, we adopt random insertion to generate initial solutions. We refer to Appendix F for the impact of different initialization methods on PRC performance.
5.1 COMPARATIVE RESULTS
We present the results on synthetic TSP and CVRP instances in Table I From the results, we can observe that our method consistently demonstrates superior performance. For both TSP and CVRP, when all baseline methods use the greedy search for inference, our method significantly outperforms the representative construction-basee NCO methods with much smaller gaps and runtime. Our method, with only 50 PRC iterations, can beat the other learning-bast solution quality and solving efficiency across all scales except for CVRPIK. On CVRPIK, our method needs 500 or more PRC iterations to achieve the best among all the competitors. Overall, our method shows good scalability and can achieve outstanding performance even for very large-scale problem instances with up to 100K nodes. Remarkably, our method outperforms the classical solver HGS CVRP10K. CVRP50K, and CVRP100K. As we know, this is the first time that learning- based method gains a significant advantage over the specialized solver on scrge-scale VRP instances, manifesting notable achievement for the NCO methods.
methods in terms of both
8

---

Published as a conference paper at ICLR 2025
Table Results on TSPLIB and CVRPLIB. OOM: The method inapplicable due to the memory
limit. There exist instances that are not solvable by the NCO method due to the OOM issue.
TSPLIB
CVRPLIB
Method
S 5K
5K
All
Solved
IK
7K
Solved
GLOP
5.017%
6.870%1
5.495%
31/33
15.335%
21.317%
17.898%
14/14
ELG a
11.34%
OOM
11.34%
23/33
10.57%1
OOM
10.57%
6/14
BQ bs16
10.648%
30.5799:
12.948%
26/33
13.918%
0OM
13.918%
K/14
LEHD RRC1,000
3.996%
18.458%
7.371%
30/33
8.423%
21.525%
11.043%
10/14
SIGD greedy
12.369%
152.879%
48 63.630%
31/33
14.733%
49.491%
29.6 29.629%
14/14
BQ greedy
11.640%
162. 1
64.649%
32/33
16.923%
52.267%
32.071%
14/14
INVIT greedy
11.492%
9.996%
11.038%
33/33
15.873%
26.637%
20 486%
14/14
LEHD greedy
11.139%
39.343%
17.720%
30/33
15,203%
32.797%
18. 18.722%
10/14
Ours greedy
6.767%
10.697%
8.244%
33/33
15.806%
15.504%
67.6777
14/14
Ours PRC1,000
1.576%
4.043%
2.556
33/33
8.347%
11.209%
9.574%
14/14
Table Effects of self-improved training and cross-attention. w/o SIT: Our model is trained by SL rather than SIT. w/o Cross-Attention: Our model uses the conventional self-attention rather than cross-attention. Ours: The model is equipped with both cross-attention and SIT. The trained models tested by using greedy search. Gap (%), Time (s), and Memory (MB) are averaged over instances.
Scule
1K
5K
10K
Gap
Time
Memory
Gap
Time
Memory
Gap
Time
Memory
w/o SIT
12.69
0.21
9.65
24.98
5.41
47.94
37.63
19.83
94.41
w/o Frnss-Attentin
5.20
0.75
96.94
6.04
82,22
2317.50
12.03
705.90
9219.09
5.01
9.65
5.37
5.41
47.94
7.58
19.83
94.41
The experimental results real-world large-scale TSPLIB and CVRPLIB instances are provided in T When all methods perform the greedy search for inference, our method achieves smaller gaps than the other construction-based NCO methods 2 (out of 4) groups of instances. Using the PRC method, our method consistently outperforms the other baseline methods, achieving the smallest gaps for both TSP and CVRP. While some NCO methods struggle to solve large-scale instances due to heavy seff-attention-basee Transformers, our method can solve all TSP and CVRP instances with up to 85, 900 nodes.
5.2 ABLATION STUDY
We conduct ablation study to demonstrate the impact of the key components (i.e., cross-attention and SIT) our method. First, we train the proposed cross-attention-aasse scale VRPs without the SIT algorithm (i.e., w/o SIT in Table B). Due to the scarce labeled data on larger VRP instances, we train the model on CVRPI00 following (Luo et al., 2023). Then, we replace the cross-attention in Transformer network with the self-attention, and the resultant model (i.e., w/o Cross-Attention in Table 3) is trained by our SIT algorithm. We test the above two models on CVRPI K/5K/10K. From Table 3, we observe that our model trained with SIT significantly outperforms the model trained without SIT, which indicates SIT enables more effective model training on large-scale problems. On the other hand, the results verify that the proposed cross-attention more efficient than the self-attention in terms of both time and memory usage (c.g., over 30x speedup and x x memory saving on CVRP10K).
Transformer on large-
5.3 ADDITIONAL ANALYSIS
Efficiency analysis. We test our method and self-attention aateention-based Transformers, including POMO, LEHD, SIGD Kwon et al. (202)): Luo et al. (002)); Pirnay & Grimm (0024), on TSP instances of different scales, and compare them in terms of inference time and memory usage. The results in Table 4 demonstrate the superior time and space efficiency of our Transformer network across all problem scales. For example, our model runs 4x faster and takes up 10x smaller memory than LEHD on instances with 1K nodes. Moreover, the time and space efficiency significantly enlarge the problem scale grows. Especially for instances with 50K 100K nodes, self-attention-bert based

---

Published as a conference paper at ICLR 2025
Table The time and memory usage of our model and representative NCO methods. All methods are tested by using greedy search. Time (s) and Memory (MB) are averaged over instances.
Scale
5K
10K
50K
100K
Time
Memory
Time
Memory
Time
Memory
Time
Memory
Time
Memory
POMO
0.5
107.5
192.6
2599.1
553.2
10352.3
OOM
OOM
SIGD
1.2
41.2
108.0
971.0
930.6
3846.8
OOM
OOM OOM
LEHD
0.8
97.4
90.0
2323.2
730.8
9224.3
OOM
Ours
0.2
9.3
5.5
45,4
20.6
91.9
483.4
459.4
1983.3
918.6
Table 5: Impact of maz in both training and inference.
Amaz (Inference)
100
1,000
10,000
Ama (Training) 4
Gap (Time)
Gap (Time)
Gap (Time)
100
4.568% (18s)
2.647% (1.7m)
2.731 % (6.0m)
1,000
4.513% (18s)
2.319% (1.7m)
2.279% (6.0m)
10,000
4.566% (18s)
2,490% (1.7m)
2.365% % (6.0m)
models encounter out-of-memory (OOM) issues, highlighting their limitations in tackling large- scale problems. In contrast, our model is consistently efficient on large-scale VRPs, indicating its advantageous scalability. Furthermore, the time and memory usage in the methods empirically verify the linear complexity of our cross-attention-bue -aaed Transformer and the quadratic complexities of self-attention-bar based models.
Sensitivity analysis. We evaluate the impact of the key hyperparameter maz model performance. To this end, we train and test (with PRC100) our model with different values (ex on TSP10K. As shown in Table the model trained to construct solutions with lmaz 10, 000 delivers inferior policy compared to the one trained with maa 1, 000. Intuitively, learning to construct overly long solutions overwhelms the model's capability, which makes hard to learn favorable representations by a limited number of cross-attention computations. On the other hand, setting bmax to small values leads excessively local policies and sacrifices global performance. Therefore, we set maa to an intermediate value of 1,000 for balancing the training and inference performance.
6 CONCLUSION
In this paper, we have proposed a lightweight cross-attention mechanism with linear complexity to improve the efficiency of the NCO model solving large-scale VRPs. Benefiting from propagating node embeddings through representative nodes, the cross-attention mechanism maintains effective interactions between nodes while achieving low complexity. In addition, we have developed novel Transformer network to learn efficient and favorable solutions, in which the cross-attention iteratively used to advance the node embeddings. Moreover, we have proposed an innovative Self-Improved Training (SIT) algorithm for direct model training large-scale VRP instances without the need for labeled data. Extensive experimental results on TSP and CVRP with up to 100K nodes in both synthetic and real-world distributions fully demonstrate the superior performance of our method. Since a specific range of random sizes is predefined when sampling the partial solutions, potential improvement to develop an adaptive strategy for setting the sampling size of partial solutions, thereby enabling more efficient model training and PRC. In addition, we will extend our method to other types of combinatorial optimization problems, such scheduling, bin packing, and knapsack problems.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. 624761 18), the Natural Science Foundation of Guangdong Province (Grant No. 2024A 11241195501155), the Na- tional Natural Science Foundation of Shenzhen (Grant N 3033301), the Research Grants Council of the Hong Kong Special Administrative Region, China (CityU11215622 and CityU11215223..
REFERENCES
David Applegate, Ribert Bixby, Vasek Chvatal, and William Cook. Concorde tsp solver, 2006.
Giorgio Ausiello, Pierluigi Crescenzi, Giorgio Gambosi, Viggo Kann, Alberto Marchetti-Spaacamelle and Marco Protasi. Complexity and approximation: Combinatorial optimization problems and their approximability properties. Springer Science & Business Media, 2012.
Irwan Bello, Hieu Pham, Quoc Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning, arXiv preprint 1eXiv::611.99900 2016.
Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: methodological tour d' d'horizon. European Journal of Operational Research, 290(2):405-421, 2021.
Jennifer AN Brophy and Christopher A Voigt. Principles of genetic circuit design. Nature methods, 11(5):508-53 520, 2014.
Hanni Cheng, Haosi Zheng, Ya Cong, Weihao Jiang, and Shiliang Pu. Select and optimize: Learning to aolve large-scale tsp instances. In International Conference on Artificial Intelligence and Statistics, pp. 1219-1231. PMLR, 2023.
Jinho Choo, Ycong-Dae Kwon, Jihoon Kim, Jeongwoo Jac, André Hottung, Kevin Tiemney, and Youngjune Gwon. Simulation--uidder beam search for neural combinatorial optimization. In Advances in Neural Information Processing Systems, volume 35, pp. 8760-8772, 2022.
Michel Deudon, Pierre Cournut, Alexandre Lacoste, Yossiri Adulyasak, and Maris-Mattin Rousseau. Learning heuristies for the tsp by policy gradient. International conference on the integration of constraint programming, artificial intelligence, and operations research, pp. 170-181. Springer, 2018.
Darko Drakulic, Sofia Michel, Florian Mai, Arnaud Sors, and Jean-Marc Andreoli. BQ-NCO: Bisimulation quotienting for efficient neural combinatorial optimization. In Advances in Neural Information Processing Systems, 2023.
Mouhcine Flgarej. Mansouri Khalifa, and Mohamed Youssfi. Optimized path planning for electric vehicle routing and charging station navigation systems. In Research Anthology on Architectures, Frameworks, and Integration Strategies for Distributed and Cloud Computing, pp. 1945-1967. IGI Global, 2021.
Han Fang, Zhihao Song, Paul Weng, and Yutong Ban. INVit: A generalizable routing problem solver with invariant nested view transformer. In International Conference on Machine Learning, 2024.
Zhang-Huu Hua Fu, Kai-Bin Qiu, and Hongyuan Zha. Generalize a small pre-trained model to arbitrarily large tsp instances. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 7474-7482, 2021.
Chengrui Gao, Haopu Shang, Ke Xue, Dong Li, and Chao Qian. Towards generalizable neural solvers for vehicle routing problems via ensemble with transferrable local policy. In International Joint Conference on Artificial Intelligence, 2024.
Thierry Garaix, Christian Artigues, Dominique Feillet, and Didier Josselin. Vehicle routing prob- lems with alternative paths: An application to on-demand transportation. European Journal of Operational Research, 204() 1::2-75, 2010.

---

Published as a conference paper at ICLR 2025
Keld Helsgaun. An extension of the eekkeiighanhhelgggaea tsp solver for constrained traveling salesman and vehicle routing problems. Roskilde: Roskilde University, 12, 2017.
André Hottung, Yeong-Dae Kwon, and Kevin Tierney. Efficient active search for combinatorial optimization problems. In International Conference on Learning Representations, 2022.
Qingchun Hou, Jingwei Yang, Yiqiang Su, Xiaoqing Wang, and Yuming Deng. Generalize learned heuristics to solve large-scale vehicle routing problems in real-time. In International Conference on Learning Representations, 2023.
Yan Jin, Yuandong Ding, Xuanhao Pan, Kun He, Zhao, Tao Qin, Lei Song, and Jiang Bian. Pointerformer: deep reinforced multi-pointer transformer for the traveling salesman problem. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Chaitanya Joshi, Thomas Laurent, and Xavier Bresson. An efficient graph convolutional network technique for the travelling salesman problem. arXiv preprint arXiv: 0et::0/0660022 2019.
Minsu Kim, Jinkyoo Park, al. Learning collaborative policies solve np-hard routing problems. In Advances in Neural Information Processing Systems, volume 34, pp. (0418-10436) 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, San Diega, CA, USA, 2015.
Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In International Conference Learning Representations, 2019,
Wouter Kool, Herke Hoof, Joaquim Gromicho, and Max Welling. Deep policy dynamic pro- gramming for vehicle routing problems. In Integration of Constraint Programming, Artificial Intelligence, and Operations Research: 19th International Conference, CPAIOR 2022, Los Angeles, CA, USA, June 20-23, 2022, Proceedings, pp. 190-213. Springer, 2022.
Yeong- Dae Kwon, Jinho Choo, Byoungüip Kim, Iljoo Yoon. Youngjune Gwon, and Seungjai Min. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances Neural Information Processing Systems, volume 33, pp. 211 111888-1111 211-211999 198, 2020.
Sirui Li, Zhongxia Yan, and Cathy Wu. Learning to delegate for large-scale vehicle routing. In Advances in Neural Information Processing Systems, volume 34, pp. 20198-262222 26221, 2021.
Yang Li, Jinpei Guo, Runzhong Wang, and Junchi Yan. From distribution learning in training to gradient search in testing for combinatorial optimization. In Advances in Neural Information Processing Systems, 2023.
Fu Luo, Xi Lin, Fei Liu, Qingfu Zhang, and Zhenkun Wang. Neural combinatorial optimization with heavy decoder: Toward large scale generalization. In Advances Neural Information Processing Systems, 2023.
Sahil Manchanda, Sofia Michel, Darko Drakulic, and Jean-Marc Andreoli. On the generalization of neural combinatorial optimization heuristics. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 426 442. Springer, 2022.
Yimeng Min, Yiwei Bai, and Carla P Gomes. Unsupervised learning for solving the travelling salesman problem. In Advances in Neural Information Processing Systems, 2023.
Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takác. Reinforcement learning for solving the vehicle routing problem. In Advances in Neural Information Processing Systems, volume 31, 2018.
Xuanhao Pan, Yan Jin, Yuandong Ding, Mingxiao Feng, Li Zhao, Lei Song, and Jiang Bian. H-tsp: Hierarchically solving the large-scale traveling salesman problem. Proceedings of the AAAI Conference Artificial Intelligence, February 2023.
Jonathan Pirmay and Dominik G. Grimm. Seff-improvement for neural combinatorial optimization: Sample without replacement, but improvement. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. Featured Certification.
12

---

Published as a conference paper at ICLR 2025
Ruizhong Qiu, Zhiqing Sun, and Yiming Yang. DIMES: A differentiable meta solver for combinatorial optimization problems. In Advances in Neural Information Processing Systems, 2022.
Gerhard Reinelt. TSPLIB-a traveling salesman problem library. ORSA Journal on Computing, 3(4): 376-384, 1991.
Jiwoo Son, Minsu Kim, Hyeonah Kim, and Jinkyoo Park. Meta-sage: Scale meta-learning scheduled adaptation with guided exploration for mitigating scale shift on combinatorial optimization. In International Conference on Machine Learning, pp. 32194 32210. PMLR, 2023.
Zhiqing Sun and Yiming Yang. DIFUSCO: Graph- based diffusion solvers for combinatorial opti- mization. In Advances in Neural Information Processing Systems, 2023.
Eduardo Uchoa, Diego Pecin, Artur Pessoa, Marcus Poggi, Thibaut Vidal, and Anand Subramanian. New benchmark instances for the capacitated vehicle routing problem. European Journal of Operational Research, 257(3):845-88) 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Thibaut Vidal. Hybrid genetic search for the evrp: Open-source implementation and swap* neighbor- hood. Computers & Operations Research, 140: 105643, 2022.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems, volume 28, 2015.
Yang Wang, Ya-Hui Jia, Wei-Neng Chen, and Yi Mei. Distance-aware attention reshaping: En- hance generalization of neural solver for large-scale vehicle routing problems. arXiv preprint arXiv:2401.06999, 2024.
Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Step-wise deep learning models for solving routing problems. IEEE Transactions on Industrial Informatics, ((7))4861-4877 2020.
Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang, Multi-decode attention model with embedding glimpse for solving vehicle routing problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12042-12049, 2021a.
Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Neurolkh: Combining deep learning model with waateernighha--eelggaaan heuristic for solving the traveling salesman problem. Advances in Neural Information Processing Systems, volume 34, pp. 7472-7483, 2021b.
Haoran Ye, Jiarui Wang, Helan Liang, Zhiguang Cao, Yong Li, and Fanzhang Li. Glop: Learning global partition and local construction for solving large-scale routing problems in real-time. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Zhi Zheng, Changliang Zhou, Tong Xialiang, Mingxuan Yuan, and Zhenkun Wang. UDC: unified neural divide-and- divide-and-conquer framework for large-scale combinatorial optimization problems. In Advances in Neural Information Processing Systems, 2024.
Changliang Zhou, Xi Lin, Zhenkun Wang, Xialiang Tong, Mingxuan Yuan, and Qingfu Zhang adaptation for large-scale generalization of neural combinatorial optimization. arXiv preprint 2rXiv:2405.01900 01900, 2024.
Jianan Zhou, Yaoxin Wu, Wen Song, Zhiguang Cao, and Jie Zhang. Towards omni-generalizable neural methods for vehicle routing problems. In International Conference on Machine Learning, PMLR, 2023.
pp.
Zefang Zong, Hansen Wang, Jingwei Wang, Meng Zheng, and Yong Li. Rbg: Hierarchically solving large-scale routing problems in logistic systems via reinforcement learning. Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 4648-4658, 2022.
13

---

Published as a conference paper at ICLR 2025
A COMPLEXITY ANALYSIS OF ITERATIVE RECONSTRUCTTIN
We would like to clarify that the complexity analysis in the main text (Section 4.2) to demonstrate the complexity advantage of our model rather than the entire algorithm. Our model has linear complexity (i.c., ((n)) and is more suitable for handling large-scale problems than existing NCO models, with significant computational efficiency compared to the quadratic complexity (i.e., O(n²) of self-attention-base models in the literature.
For the iterative reconstruction process, the computational complexity is (k and space com- plexity is (n)) where k is the number of iterations and 4moz is the maximum length of the partial solutions during reconstruction progress. Specifically, a single reconstruction has O ((maz) com- putational complexity, since 1) it consists of (maz sequential model predictions and 2) one time of linear model prediction has (bmaz) complexity. Therefore, performing h reconstructions yields an overall computational complexity of O Okkl As the space complexity is not cumulative, the space required for performing A reconstructions remains (lmazz) this paper, Imax
reconstructions are I -Lmaz) = (
executed parallelly in each iteration. Therefore, the space complexity is O
n
while the overall computational complexity remains Ok1 mem))
It is worth noting that many NCO methods (including ours) employ post-search or iterative inference to trade more computation time for better accuracy. To comprehensively illustrate our method's advantages computational efficiency, we compare it with three representative NCO methods (i.e.. POMO (Kwon et al., 2020), LEHD (Luo et al. 2023), and SIGD (Pirnay & Grimm 2024)) under single-round (greedy search) and multi-round inference, respectively. Under the multi-round inference, augmentation (aug) for POMO, random reconstruction (RRC) for LEHD, and beam search (bs) for SIGD as suggested in their original papers, and each of them is operated with 20 rounds.
Table 6: Complexity comparison between our method and three representative ones with single-round inference
TSPIK
TSPSK
TSP10K
Method
Gap
Time
Memory
Gap
Time
Memory
Gap
Time
Memory
POMO greedy
42.3%
0.5s
108MB
72.9%
3.2m
2599MB
86.48%
9,2m
10352MB
SIGD greedy
1.96%
1.2s
41MB
12.20%
1.8m
971MB
30.68%
15.5m
3847MB
LEHD greedy
3.11%
0.8s
97MB
15.46%
1.5m
2323MB
27.24%
11.7m
9224MB
Ours greedy
1.95%
0.2s
9,3MB
3.17%
5.2s
45MB
4.05%
20s
92MB
Table 7: Complexity comparison between our method and three representative ones with multi-round
inference
TSP1K
TSPSK
TSP10K
Method
Gap
Time
Memory
Gap
Time
Memory
Gap
Time
Memory
POMO 20
40.4%
10.6s
880MB
71.8%
21.4m
21185MB
OOM
SIGD bs20
1.01%
21.7s
750MB
9.15%
36.8m
17564MB
OOM
LEHD RRC20
1.70%
3.7s
97MB
12.09%
1.6m
2323MB
21.78%
12.
9224MB
Ours PRC20
0.92%
1.6s
12MB
2.27%
8.6s
85MB
2.59%
20.3s
129MB
The experimental results with the single-round inference are given in Table These results indicate that our method achieves better optimality gaps with less running time and memory usage all three problem sizes. Our method yields at least 2.5x, 17.3x, and 27.6x speedups and 4.4x, 21.5x, and 41.8x memory reductions TSPIK, TSP 5K, and TSP10K, respectively.
Table 7 provides the experimental results under the mouti- inference setting. These results show that our method also significantly outperforms the other three competitors regarding the optimality gap, running time, and memory usage. Our method achieves at least 2.3x, 11.1x. x, and 35.7x speedups and 8x, 27.3x, and 71.5x memory reductions TSP1K, TSP5K, and TSP10K, respectively.
14

---

Published as a conference paper at ICLR 2025
In short, the advantage of computational complexity enables method to have higher solving efficiency in either greedy or iterative manner, especially when dealing with problem instances with sizes larger than IK.
B IMPLEMENTATION DETAILS FOR CVRP
In this section, we introduce the problem setup, Transformer network details, complexity analysis, and PRC implementation details for CVRP.
B. PROBLEM SETUP
A CVRP instance comprises one depot node and n customer nodes, where each customer node i has a demand 6, fulfill. Our goal find a set of sub-tours that begin and end at the depot, ensuring the sum of demands in each sub-tour adheres to the vehicle's capacity constraint C. The objective is to minimize the total distance across these sub-tours while maintaining the capacity constraint C. Our CVRP instances are generated in a manner similar to that described in (Kool et al. (209)) featuring customer and depot node coordinates uniformly sampled from a unit square [0, 1j² Demands 6, are uniformly sampled from 1,...
In line with (Kool et al. 2022 Drakulic et al. 2023 Luo et al. 2023), we establish a feasible CVRP solution formation. Instead of isolating a depot visit as a distinet step, we employ binary variables to signify whether a customer node is accessed via the depot or another customer node, In feasible solution, node is assigned 1 if accessed through the depot and 0 if accessed through another customer node. For example, a viable CVRP solution (0, 1, 2, 3, 0, 4, 5, O, 6, 7, 0, 8,8,99 10) with 0 representing the depot can be represented as shown below:
1
1
2
3
8
9
10
1
0
where the first row displays the visited node sequence, while the second row signifies if each node is accessed through the depot another customer node.
This notation aims to maintain solution consistency, In CVRP cases, solutions with equal customer node counts might have differing sub-tour quantities, causing potential misalignment. This notation prevents such problems.
B.2 TRANSFORMER NETWORK
Embedding Layer. In CVRP, the node feature x; is 3D vector, combining 2D coordinates and the demand of node i, where and {x:]== are depot node and customer node features, respectively. The depot's demand is set O. We normalize the vehicle capacity C' to Ò and the demand to ô; & for simplicity. Given the node features {x:] the encoder produces each node's embedding (h; J by a linear projection such that h, = x, (0) + b(0), Vi E nn.. where W(0) 3xd and (0) E Rd are learnable matrices.
Decoder with Cross-Attention Modules. Similar to K et al. (2020), we add the dynamically changing remaining capacity to the first and last node embeddings in the decoder. The remaining capacity can be denoted as C, E RR and the first and last node embeddings are h and ha.-1 respectively. We fuse the remaining capacity to the first node and last nodes' embeddings via
h', h br_, [br,11 C,JW22 + b2.
[ba C, + b1
where Wi, W2 E R(d+1)xd and b, b2 E Rd. Then the representative nodes' initial embeddings ((0) and graph node embeddings (0) are calculated
z)) = [h', h' 7 = [h',, h H'
where W1, W2 dd are learnable matrices, H is the set of unvisited customer nodes' embeddings at the t-th step. the first decoding step. the depot node म, is selected to be the initial partial solution, and we treat it as both the first and last nodes for the second decoding step.
15

---

Published as a conference paper at ICLR 2025
Similar to TSP, the output of our model's L-th linear attention module is Z and H where H (.. Then, linear projection and softmax function are applied to it, producing the selected probability of each unvisited node. The first and last nodes are masked before the softmax calculation, i.e.,
h Wo:
+ or otherwise
u;
(10)
is learnable matrix. Each u, R corresponds two actions: access customer node via the depot or another customer node. It corresponds to the notation in Eq.(7) Finally, a softmax function is applied to all the u, to produce the selected probability of cach action
where Wo E Idxx
Complexity Analysis. According to Eq. 3). the dimensions of input to cross- cross-attention modules 22xd for Z(-1) and R for (T-11), where ñ < (n 2) is the number of node embeddings input to the decoder. Except for the constant d (d = 128 for most Transformers of NCO methods), the cross-attention between node embeddings ( and (1-1) yields complexity ((2n). Therefore, 4 exhibits the same linear complexity in solving CVRP.
B.3 IMPLEMENTATION DETAILS OF PRC FÜR SOLVING CVRP
During the PRC process, we need to ensure that the sampled partial solutions have no overlap, and that no illegal solutions are generated after merging the partial solutions. These details can be described below.
Guarantee Non-overlapping. We always sample a contiguous subset of the complete solution as partial solution. All partial solutions have the same number of customer nodes. We sample these partial solutions sequentially from the beginning of the solution to the end, following its order. Each sampled partial solution is unique.
For example, consider CVRP instance with ten customer nodes, represented by the solution (0, 1, 3, 4, 5, 0, 7, O, 8, 9, 10, 0), where 0 represents the depot. If each partial solution contains four customer nodes, then the partial solutions can be (0, 1, 2, 3, 4) and (5, O, 6, 7, O, 8). The remain- ing segment, (9, 10, 0), with only customer nodes, will not participate in the reconstruction process.
Avoid Violating the Capacity Constraint. For CVRP, the sum of customer demands in each sub-tour must not exceed the vehicle's capacity. This constraint may make the solution infeasible after merging reconstructed partial solutions. To ensure the solutions remain feasible, we take two measures during the reconstruction:
(a) Exclude Tail Subtour. If the subtour at the end of a partial solution does not conclude at the depot, it will not participate in the reconstruction process.
For example, consider a CVRP solution (0, 2, (5, 6, 7, 0, 8), the subtour (0, 8) does not conclude at the depot, and it will not participate in the reconstruction.
3, 4, 5, 0, 6, 7, 0, 8, 9, 10, 0). for its partial solution
If the tail subtour (0, 8) participates in the reconstruction process, the result might be (5, 0, 6, 8). which is then merged with (9, 10, 0) to form the partial solution (5, 0, 6, 8, 9, 10, 0). However, within it, (0, 6, 7, 8, 9, 10, 0) might violate the capacity constraint. This occurs when the sum of the demands of node {8, 9, 10) is very close to equal to the capacity. Therefore, we need to exclude the reconstruction of the tail subtour.
(b) Initial Capacity Calculation. When the head node of the partial solution is not the depot, the vehicle's s initial capacity is determined by the full capacity minus the total demand nodes between this head node and its preceding depot in the original complete solution.
For example, consider the CVRP solution (0, 2, 3, 4, 5, O, 6, 7, O, 8, 90, 0) again, for the partial solution (5, 0, 6, 7, O, 8), its head node is not the depot. The nodes between this head node and the preceding depot are {1, 2, 3, 43. Therefore, the initial capacity for reconstructing the partial solution (5, 0, 7, O, 8) is the full capacity minus the total demand of nodes 2,,2,3, 4).
16

---

Published as a conference paper at ICLR 2025
If this constraint restriction is removed (i.e., using full capacity), the reconstructed partial solu- tion (5,0, 6, 7, 8)) may become (5, 6, 7, 8). When it merges with (0, 2,3,, 4), forming the route (0, 1, 2, 3, 4, 5, 6, 7, it has high probability of exceeding the vehicle's capacity limit. To avoid this, we need to implement this initial capacity calculation.
C CHOICE OF REPRESENTATIVE NODES
In this paper, we use the first and last visited nodes as the representative node. We also try other options for the representative nodes, such the pooling nodes (i.e., summing up the embeddings of all nodes and then averaging them) and learnable nodes (i.e., learning two embeddings). However, we experimentally find that models with these types of representative nodes suffer from numerical instability during training, where the loss becomes NaN in the early stage of training.
In solving VRPs. the last node is dynamically updated to the one selected in the previous step, and the number of unvisited nodes decreases. This adjustment continuously alters the relationship among the first, last, and unvisited nodes. The representative nodes must accurately reflect these changes at every step. However, pooling nodes or learnable nodes cannot achieve it. Instead, the first and last nodes inherently represent dynamically changing information and, thus, are more suitable as representative nodes.
Inspired by Deudon et al. (2018), we explore strategies to enhance the model's perception of last node's information to improve the model's learning capabilities. We repeat the last node several times in the representative nodes, and conduct the model training on TSPIK instances with two SIT iterations. The optimality gap of corresponding models versus the number of repeated last nodes is plotted in Figure 3 We can find that there a positive correlation between the number of repetitions and performance gains. Moreover, the number of repetitions increases, the incremental benefits become smaller and smaller, especially when the number is larger than 15, In addition, increasing the number of representative nodes results extra computational overhead. Considering these factors, we use 15 in the experiments to balance between performance improvement and resource efficiency.
4.0
Gap (%)
3.8
3.6
34
Number of Repetitions
Figure 3: Comparion results with different numbers of repetitions the last node.
17

---

Published as a conference paper at ICLR 2025
D PSEUDOCODE
This section provides the pseudocode- explanation of SIT in Section 4.3 to enhance clarity and understanding. Algorithm iis the pseudocode-based description of SIT. Algorithm for generating the initial pseudo-labels. Algorithm 3 is for the parallel local reconstruction for a single instance. Algorithm is for the model training process.
Algorithm 1 Self-Improved Training
Input: Problem size N, dataset size D, batch size B, maximum length of partial solution Imaz-
2: Output: The trained model with parameters 6",
3: Randomly initialize the model with parameters A;
4: S; ~ Samplelnstance(( N) vi E (1, ..... D};
5: и, f Genecreelntiallootttion////////
Vi (I,...,D}
b Serve as initial pseudo-labels.
6: for each iteration do
7: Ti; + LocalReconsstuctionnn n, St, N. Imaz)
Vi (L..... D]: vi (1.....Dp:
8: 0 +- ModelTraining(O, {T,, { D, B, lmax)
9: end for
Algorithm 2 GeneratelInitiaaSllttiin
Input:
Instance S.
The procedure is essentially conducting random insertion.
2: Output: Complete solution π.
3: Calculate the distance matrix [day]:
4: Pick random node to form the initial partial solution
5: while there are unvisited nodes do
6: Select random unvisited node i
7: Find the best position (j. k) to insert i that minimizes d3s + dis - dsk
8:
Insert between nodes and in
9: end while
Algorithm 3 LocalRaconsstruction
Input:
Model parameter 0, solution æ, instance S, problem size maximum length of the partial solution Imaz-
2: Output: The improved solution æ.
3: for each iteration do
4: w Uniform|4, Imaz]:
5:
{TT"" [.S.) +- SammleeartialSllttinnske S, Vi [1..... [N/w]]:
6:
{T) f- ReconstruccByyModdll.. {n, SP)) vie [1, [N/w]];
7:
T! SaveBetterOne(mT, x²") Vi (1...., [N/w]]:
8:
π + Combine( (TT)) {r/²)):
9:
end for
18

---

Published as a conference paper at ICLR 2025
Algorithm 4 ModelTraining
1: Input: Model parameter 0, pseudo-labels {T.), instances {S.). dataset size D, training batch
size B, maximum length of the partial solution (max-
2: Output: The trained model parameter
3: for each epoch do
4:
for step = 1...., D/B] do Uniform[4, mear:; 5 4- SamplePariaaSSlltiiiiffs Si,w) Vi {I,..... B); for t 2 do
5:
6:
7:
VC(0) - B E Vo log
8:
9:
<- ADAM(O, VL(O)):
end for
11: end for
end for
E TRAINING SETTING
Our Transformer networks are trained on instances with different scales separately. We summarize the training settings in Table
Dataset Size. refers the number of instances in the dataset, which varies depending on the problem scale. When training larger scale (e.g. 100K), fewer instances are required because complete solution of this scale can generate enough partial solutions to facilitate efficient training.
Selrove Iteration. Each self-improved training iteration involves number of training epochs and a number of reconstruction iterations.
Training Epoch. When all training instances have been used for training the model once, one epoch is completed.
Training Batch Size. It refers to the number training data used each time the model parameters are updated. In one training epoch, the dataset is divided into multiple smaller batches, each containing a certain number of instances. During training, larger-scale datasets like TSP100K and CVRP100K constraints, while smaller-scale datasets like TSPIK and CVRPIK use larger batch size (i.e., 256).
smaller batch size (i.e., 16) due to device memory
Reconstruction Iteration. One iteration refers the process in which all training instances are reconstructed once.
Reconstruction Batch Size. one local reconstruction iteration, the entire dataset is reconstructed in batches. Larger-scale datasets like TSPI00K and CVRPI00K use smaller batch size due to device memory constraints, while smaller-scale datasets like TSPIK and CVRP1K use a larger batch size.
Learning Rate. The initial learning rate for training the model.
Decay per Training Epoch. The rate at which the learning rate decays with each epoch.
Maximum Length of Partial Solution. This represents the maximum length for partial solutions during training and reconstruction.
In each self-improved iteration, we save the model that with the best greedy performance on the validation dataset to use next training iteration.
19

---

Published as a conference paper at ICLR 2025
Table Training settings.
TSPIK
TSPSK
TSPIOK
TSP50K
TSPI00K
Dataset Size Sell-impruved Iteration
20,000
200
200
100
100
7
Training Epoch Training Batch Size
20
256
32
32
32
16
Reconstruction Iteration
100
100
100
100
100
Reconstruction Batch Size
256
32
32
16
8
Leaminng Rate
1c-4
le-4
1c-4
lc-4
1c-4
Decay per Training Epoch
0.97
0.97
0.97
0.97
0.97
Partial Solution's Maximum Length
1,000
1,000
1,000
1,000
1,000
CVRPIK
CVRP5K
CVRPIOK
CVRP50K
CVRP100K
Dutaset size
20,000
200
200
100
100
Self-improved Iteration
Truining Epoch
20
20
20
Training Batch Size
256
32
32
32
16
Reconstruction Iteration
100
100
100
100
100
Reconstruction Batch Size
256
32
32
16
8
Leariing Rate
le-4
le-4
le-4
le-4
le-4
Decay per Training Epoch
0.97
0.97
0.97
0.97
0.97
Partial Solution's Maximum Length
1,000
1,000
1,000
1,000
1,000
F IMPACT OF SOLUTION INITIALIZATION METHODS ON PRC PERFORMANCE
We empirically study the impact of different initialization methods on the PRC performance. We test three simplest initialization methods, including greedy search performed by the model, heuristic nearest neighbor, and heuristic random insertion. We refer to (Kool et al. 2019) for their details. We use PRC100 to improve the solutions generated by these initialization methods and obtain the results, which are shown (able These results demonstrate that random insertion+PRC produces solutions with lower objective values than the other methods on 6 (out of 10) groups of instances. At the same time, random insertion is generally faster than the model's greedy search in initializing the solution (see Table Tp. Considering these factors, we choose the random insertion method to generate the initial solution for PRC.
Table 9: Performance of PRC100 (in terms of Obj.) with different solution initialization methods.
TSPIK
TSP5K
TSPIOK
TSP50K
TSP100K
Model's greedy
23,251
51.698
73.368
165.407
234.651
Nearest neighbor
23.254
54.954
79.
179.886
255.367
Random insertion
23.254
51.824
73.293
164.591
232.550
CVRPIK
CVRP5K
CVRPIOK
CVRP50K
CVRP100K
Model's greedy
37.460
91.857
108.229
268.416
464.325
Nearest neighbor
37.493
91.725
111.832
316.338
529.006
Random insertion
37.491
91.580
108.038
268.024
471.348
G COMPARISON WITH TAM
We compare the proposed method with the decomposition- -bassd method TAM Hou et al. (2023). Since TAM does not release source code, we test our method on the test dataset introduced in their paper. TAM's results are from the original paper. For method, the model is trained on CVRPIK instances and subsequently tested across larger-scale instances. The results are shown in Table I0 We can observe that with the simple greedy search, our method can achieve better objective values than these methods, indicating excellent performance and efficiency of our method in solving large-scale CVRP instances.
20

---

Published as a conference paper at ICLR 2025
Table 10: Experimental results compared with the decomposition- methods.
CVRPIK Obj. (Time)
CVRP2K Obj. (Time)
CVRP5K Obj. (Time)
CVRP7K Obj. (Time)
Method
HGS
41.2 (5m)
57.2 (5m)
126.2 (5m)
172.1 (5m)
LKH3
46.4 (6.2s)
64.9 (20s)
175.7 (2.5m)
245.0 (8.4m)
Random Insertion
66.3 (<1s)
95.3 (<ls)
225.4 (<1s)
309.2 (<1s)
TAM-LKH3
46.3 (1.8s)
64.8 (5.6s)
144.6 (178)
196.9 (33s)
TAM-HGS
142.8 (30s)
193.6 (52s)
Ours greedy
44.1 (0.2s)
59.6 (0.8s)
131.0 (4.76)
176.4 (8.9s)
H RESULTS ON SMALL SCALE INSTANCES
We empirically evaluate our method on small-scale TSP and CVRP (C=100 for all scales) instances and present the results in Table From the results, we can observe that our method consistently demonstrates outstanding performance. For both TSP and CVRP, our method with PRC outperforms the other tearning-base methods in terms of solution quality, while maintaining shorter equal inference time the majority instances except for TSP300. Although slightly more time is required for the TSP300 instance, this is accompanied by notable improvement in solution quality. These findings highlight the effectiveness and efficiency of our method.
Table 11: Comparative results on small-scale TSP and CVRP instances.
TSP300
TSP500
TSP700
Method
Obj.
Gap
Total Time
Obj.
Gap
Total Time
Obj.
Gup
Total Time
Concorde
12,973
0.000%
5.9m
16.522
0.000%
32m
19.442
0.000%
188h
H-TSP
13.537
4.346%
15.8s
17.549
6.219%
23s
20.694
6.444%
29.1s
GLOP
13.137
1.2639
1 1.4m
16,883
2.186%
1.6m
19.963
2.683%
3.0m
POMO augx8
13.834
6.636%
16.8s
20.187
22,18995
1.1m
25.576
31.555%
3.7m
INVIT-3V greedy
13.725
5.797%
4.8m
17.327
4.877%
20.712
6.536%
12.2m
LEHD greedy
13.117
1.104%
4.8s
16,780
1.560%
16s
19.848
2.088
37s
BQ greedy
13.071
0.752%
19.3s
16.717
1.180%
19.752
1.597%
1.2m
SIGD greedy
13.080
0.822%
11s
16.714
1.166%
29s
19.692
1.290%
59x
Ours
13,063
0.695%
13.9s
16,700
1.083%
16s
19.678
1.216%
36s
CVRP300
CVRP500
CVRP700
Method
Obj.
Total Time
Obj.
Gap
Total Time
Obj.
Gap
Total Time
HGS
23.978
0.0005
4.2h
36.561
0.000%
4h
49.836
0.000%
4.2h
POMO a
26.985
12.538%
18s
44.638
22.091%
1.2m
73.560
47 60.603
3.4m
LEHD greedy
25.182
5.020%
5s
38.413
5.064%
17s
50.930
2.194%
37s
INVIT-3V greedy
26.873
12.070%
6.1m
41.133
12.505%
10.4m
54.838
10.036%
15.4m
BQ gredd
25.155
4.906%
26.9s
38.438
5.134%
47s
$1.020
2.375%
1.4m
SIGD greedy
25.148
4.878%
22s
38.671
5.770%
42s
$1.656
3.651 %
1.6m
Ours
24.961
4.099%
5s
37.903
3.671 %
16s
50.221
0.771
32s
21

---

Published as a conference paper at ICLR 2025
I EXPANSIBILITY
We evaluate the expansibility of our method by applying it to another NCO method BQ (Drakulic al. 2023). Specifically, we replace the self-attention within BQ with the proposed cross-attention, and then train it by SIT on CVRPIK and CVRP5K directly. We then use the greedy search with the default setting to test the model performance. The results displayed proposed method can significantly boost the performance of BQ on large-scale problems in terms of gap, inference time, and memory usage.
that the
Table 12: Expansibility of our method to BQ (Drkkulic et al. 2023 Gap (%), Time (s), and Memory (MB) are averaged over instances.
CVRPIK
CVRP5K
Gap
Time
Memory
Gap
Time
Memory
BQ origin
8.23
1.0
20.6
20.48
8.1
479.1
7.46
0.3
3.1
8.19
3.2
5.0
J EXPERIMENTS AND DISCUSSION ON THE BASELINES RETRAINED ON LARGE-SCALE INSTANCES
We did not train baseline NCO models on problem instances large as those our model uses for the following reasons: 1) For NCO methods that use supervised learning for model training, it is very difficult obtain sufficient labeled data (i.e., optimal solutions of problem instances with such large sizes). 2) For NCO methods based on reinforcement learning, excessive GPU memory overhead prevents them from using such large-scale problem instances for model training.
We agree that training the model with larger problem instances can increase its performance on large-scale problems. Herein, we train two representative models (i.c., POMO and LEHD) using instances with problem sizes as large as possible. For POMO, considering the GPU memory limit, we use TSP500 to perform reinforcement learning- bbased model training. For LEHD, we use LKH3 to get the labels (i.e., optimal solutions) for TSPIK instances, and conduct its supervised learning- based model training. The two models are compared with ours on in-domain data (the same size as the training data) and out-of-domain data (larger than the training data size), and the results provided in Table 13 and Table 14] These results reveal that our method significantly outperforms the two baseline models on both in-domain and -ft-ma-domain data.
We would like to emphasize that the main contribution of this work precisely lies in overcoming the limitations of reinforcement learning and supervised learning methods for NCO, allowing the model to be directly trained with problem instances whose sizes are larger than 1K. This innovation is key enabling the NCO method solve large-scale problems effectively.
Table 13: Comparison on in-domain data (the same size as the training data)
TSP500
TSPIK
Gap
Time
Gap
Time
POMO. augx8
4.88%
0.5s
LEHD, greody
2.37%
0.8s
Ours, greedy
2.19%
0.1s
Ours, greedy
1.95%
0.2s
K STUDY ON THE GENERALIZABILITY OF SIT AND THE EFFECTIVENESS OF THE CROSS-ATTENTION MODEL
To illustrate the generalizability of SIT, we apply it to two other models: POMO wwon al., 2020) and LEHD (Luo et al.) 2023). The two models are trained via SIT using the dataset of TSPIK and tested datasets TSPTK, TSP5K, and TSP10K. From the results in Table 15
22

---

Published as a conference paper at ICLR 2025
Table 14: Comparison on out-of-domain data (larger than the training data size)
TSPSK
TSPIOK
Gap
Time
Gap
Time
POMO trained on TSP500, aug x
56.05%
8.6m
00M
LEHD trained on TSP1K, greedy
5.84%
1.5m
12.45%
11.7m
Ours, greedy
3.84 %
5.28
4.62%
20.18
that POMO+SIT outperforms the original POMO by 70.5% and 61. 61.7% on TSPIK and TSPSK, respectively. Meanwhile, LEHD+SIT improves the original LEHD by 36.3%. 58.6%, and 49.3% on three kinds of problem instances. These results suggest that SIT can be generalized to other NCO models that enable them to be trained on larger problem instances, thereby improving their ability to solve large-scale problems.
Table 15: Generalizability of SIT POMO and LEHD
TSPIK
TSP5K
TSPI0K
Gap
Time
Gap
Time
Gap
Time
POMO. x
40.6%
4.1s
87.72%
8.6m
POMO SIT, a
11.96%
4.1s
25.99%
8.6m
OOM
LEHD, greedy
3.11%
0.8s
15.46%
1.5m
27.24%
11.7m
LEHD SIT. greedy
1.98%
0.8s
6.40%
1.5m
13.81%
11.7m
To demonstrate the effectiveness of our cross-attention model, we compare it with LEHD (Luo al. 2023) under the same SIT training setting (i.e., trained using TSPIK and tested on TSPIK, TSP5K, and TSPIOK). The experimental results in Table 16 indicate that our model significantly outperforms LEHD in all three aspects. Our model achieves 0.03%, 2.56%, and 9.19% optimality gap improvement over LEHD. In addition, its inference speed is 4x, 17x, and 34x that of LEHD, and its memory usage is 9.9%, 2.1%, and 1.02% of LEHD.
Table 16: Comparison between our model and LEHD
TSPIK
TSPSK
TSPIOK
Gap
Time
Memory
Gap
Time
Memory
Gap
Time
Memory
LEHD SIT, greedy
1.98%
0.8s 0.2s
97.4MB
6.40%
1.5m 5.2s
2323.2MB 47,94MB
13.81% 4.62% %
11.7m
9224.3MB
Our model + SIT, greedy
1.95%
9.65MB
3884%
20.1s
94.41MB
L COMPARISON WITH NEUROLKH
We additionally include the experimental results of NeuroLKH (Xin et al. 2021b) for comparison in Table [7 and Table 18 The results show that NeuroL obtains optimal or netr-optimal solutions on TSPTK, TSP5K, and TSPIOK. However, it fails to solve TSP50K, and TSP100K due to singular values or the out-of-memory issue. In contrast, our model can achieve high-quality solutions for problem instances of all five sizes. Moreover, NeuroLKH is significantly worse than our model on CVRP instances. This is because NeuroLKH relies on heuristic solver LKH3, which is designed for TSP and is less efficient in solving CVRP.
23

---

Published as a conference paper at ICLR 2025
Table 17: Comparion results with NeuroLKH on uniformly sampled TSP and CVRP instances
TSPIK
TSPSK
TSPIOK
TSP50K
TSPI00K
Method
Gap
Time
Gap
Time
Gup
Time
Gap
Time
Gap
Time
KeuroLKH (1000 trials)
0.00
27s
0.003%
3.7m
0.0069
8.9m
N/A
N/A
Ours PRC100
0.58%
9.48
1.67%
52.0%
2.11%
1.7m
2.91 %
8.6m
2.90%
17m
CVRPIK
CVRP5K
CVRPIOK
CVRPS0K
CVRP 100K
Method
Gap
Time
Gup
Time
Gup
Time
Gap
Time
Gap
Time
NeuroLKH (1000 trials)
4.20%
15.6s
19.65%
3.6m
41.52%
14.1m
NIA
N/A
Ours PRC100
3.31 %
8.0s
2.05%
46s
0.59%
1.3m
0.11%
5.49m
-1.00%
11.5m
Table 18: Comparion results with Ke TSPLIB and CVRPLIB. t: the method encounters problems solving one or more problem instances
TSPLIB
CVRPLIB
Method
1K < n a
5K
n..> 5K
All
Solved #
IK < n
7K
n.> 7K
All
Solved #
NeuroL (1000 trials)
0.22%
0.05%
0.20%
27/33
11.58%
41.91%
21.70%
12/14
Ours PRC1,000
1.576%
4.043%
2.556%
33/33
8.347%
11.209%
9.574%
14/14
M EFFECT OF AUGMENTATION METHOD
We include the experimental results of our model with data augmentation (i.c., augx8 (Kwon et al. 2020b) in Table 19 These results indicate that the aug 8 improves the model's accuracy each kind of TSP or CVRP instance, but also consumes more inference time. Specifically, the optimality gap is improved by 0.73% average, but the inference time increases by average of 7.1 times.
Table 19: Comparative results on synthetic TSP and CVRP instances.
TSPIK Obj. (Gap)
TSP5K
TSPIOK Obj. (Gap)
Method
Time
Obj. (Gap)
Time
Time
Ours greedy
23.569 (1.95%)
0.2s
52.59 (3.17%)
5.2s
74.69 (4.05%)
20.
Ours greedy augx8
23.391 (1.18%)
1.5s
52.11 (2.22%%
36s
73.99 (3.08%)
2.3m
Ours PRC50
23.279 (0.69%)
4.6s
51.92 (1.85%)
23.4s
73.41 (2.27%%
49.0s
Ours PRC50 aug 8
23.22 (0.45%)
32.5s
51.82 (1.66%)
2.6m
73.29 (2. 10.1))
6.4m
CVRPIK Obj. (Gap)
CVRP5K Obj. (Gap)
CVRP10K
Method
Time
Time
Obj. (Gap)
Time
Ours greedy Ours greedy aug 8
38.11 (5.01%) ())
0.2s
92.44 (3.01%)
5.5s
109.02 (1 (1.50%)
20.6s
37.74 (3.99%)
1.5s
91.86 (2.37%)
37s
107.96 (0.52%)
2.4m
Ours PRC50
37.57 (3.54%)
3.5s
92.06 (2.58%)
19.9s
108.79 (1 (1.29%)
34s
Ours PRC50 augx8
37.35 (2.92%)
24.7s
91.36 (1.80%)
2.2m
107.28 (-0.11%%
4.0m
24

---

Published as a conference paper at ICLR 2025
N EFFECT OF MODEL SIZE
The sizes of all compared constructive models are summarized in Table Our model size is 2.63M, which is larger than POMO, LEHD, and SIGD, but smaller than INViT and BQ
Table 20: Comparion results TSP instances.
OOM: the method exceeded memory limits
TSPIK
TSPSK
TSP10K
TSP50K
TSP100K
Method
Model Size
Gap
Time
Gap
Time
Gap
Time
Gap
Time
Gap
Time
POMO a
40.6%
4.1%
72.1%
OOM
OOM
OOM
INVIT-3V greedy
2,64M
6.66%
9.0s
6.90%
1.2m
7.07%
3.7m
7.18%
1.3h
7.20%
5.0h
LEHD greedy
143M
3.11%
0.8s
15.46%
1.5m
27.24%
11.7m
OOM
OOM
BQ greedy
3.11M
2.30%
0.9s
14.31%
22.58
25.02%
1.0m
OOM
SIGD greedy
1.78M
1.96%
1.25
12.20%
1.8m
30.68%
15.5m
OOM
OOM
Ours greedy
2.63M
1.95%
0.2s
3.17%
5.2s
4.05%
20.1s
5.36%
7.7m
6.13%
33.0m
We agree that more parameters may produce better results. However, the increase in model size cannot lead to performance improvement as remarkable as the innovation of model structure and training schemes. To demonstrate it, we increase the model sizes of SIGD and LEHD 3. 1M and 3.01M and retrain them for comparison. Since the performance of POMO is significantly worse than other methods, we do not consider retraining and comparing it. The experimental results of the two retrained models along with INVIT (2.64M parameters) and BQ (3.1 IM parameters) are listed in Table 21 These results reveal that the four state-of-the-an methods are significantly worse than our method, despite that their model sizes are larger than ours.
believe that our method's good performance solving large-scale problems benefits a from the SIT training strategy that enables our model to be trained directly using large-scale instances. The proposed SIT opens a door for scalable NCO methods and we believe it can inspire more follow-up works solving large-scale VRPs.
Table 21:
Comparion results on TSP instances. Models are tested using greedy search. OOM: the method exceeded memory limits
TSPIK
TSPSK
TSP10K
TSP50K
TSP100K
Model
Gap
Time
Gap
Time
Gap
Time
Gap
Time
Gup
Time
BQ (3. 11M
2.30%
0.9s
14.31%
22.5s
25.02%
1.0m
OOM
OOM
INViT-3V (2.64M)
6.66%
9.0s
6.90%
1.2m
7.07%
3.7m
7.18%
1.3h
7.20%
5.0h
SIGD (3.11M)
2.69%
1.8s
9.68%
2.9m
17.84%
23.2m
OOM
0OM
LEHD (3.01M)
2.10%
1.78
9.58%
3.7m
18.30%
27.2m
OOM
00M
Ours (2.63M)
1.95%
0.2s
3.17%
5.2s
4.05%
20.1s
5.36%
7.7m
6.13%
33.0m
25

---

Published as a conference paper at ICLR 2025
O COMPARISON WITH RECENTLY PUBLISHED RL-BASED METHODS
In the main text, we already compared several recently published RL-baaed methods, including GLOP (AAAI 2024) (Ye et al. 2024) and INViT (ICML 2024) (Fang et al. 2024). Beyond that, We
herein include two additional RL-based methods ELG (ICCAI 2024) Goo el al. 2024) and UDC
2024 for comparison, and the results are presented Table 22 From (NeurIPS 2024) (Zheng al.
the results, we can observe that our method achieves the best performance among all competitors on all scales of problem instances. Specifically, our method with only 10 PRC iterations can beat these RL--based methods in terms of both solution quality and solving efficiency on all scales of TSP instances. For CVRP, our method using the basic greedy search already achieves better performance than the other methods concerning optimality gap and running time.
The advantages of our method can be summarized in the following two aspects. Firstly, the proposed cross-attention model owns linear complexity, enabling method to have much better inference efficiency, especially when dealing with problem instances with sizes larger than 1K. Secondly, the proposed Self-Improved Training (SIT) allows the model to be directly trained using large-scale (even larger than IK) instances. Using large-scale instances for model training equips the model with better ability to solve large-scale problems.
disadvantage of our method is that it requires multiple iterations to obtain high-quality pseudo- labels for self-improved training. An interesting future work to address this issue is to develop more effective schemes to generate high-quality pseudo-labels.
Table 22: Comparion results on synthetic TSP and CVRP instances. OOM: the method exceeded memory limits
TSPIK
TSPSK
TSPPOK
TSPS0K
TSP100K
Method
Gap
Time
Gap
Time
Gap
Time
Gap
Time
Gap
Time
LKH3
0.00%
1.7m
0.005:
12m
0.005:
33m
0.00%
10h
0.00%
25h
GLOP (AAAI 2024)
2.85%
10.2s
4.26%
1.0m
4.39%
1.9m
5.10%
1.5m
5.14%
3.9m
ELG augx8 (UCAI 2024)
11.33%
0.8s
18.08%
21s
0OM
OOM
OOM
INViT-3V greedy (ICML 2024)
6.66%
6.90%
1.2m
7.07%
3.7m
7.18%
1.3h
7.20%
5.0h
UDC (z=1000) (NeurIPS 2024)
1.70%
1.4m
OOM
00M
00M
00M
Ours greedy
1.95%
0.2s
3.17%
5.2s
4.05%
20.
5.36%
7.7m
6.13%
33. 0m
Ours PRC10
1.20%
2.73%
3.08%
10.0s
4.22%
1.33m
4.16%
3.0m
Ours PRC100
0.58%
9.4s
1.67%
52.0s
2.11%
1.7m
2.91%
8.6m
2.90%
17m
CVRPIK
CVRPSK
CVRP1OK
CVRPSOK
CVRP100K
Method
Gup
Time
Gap
Time
Gap
Time
Gap
Time
Gup
Time
HGS
0.00%
2.5m
0.00%
2.0h
0.00%
5.0h
0.00%
8.1h
0.00%
24h
GLOP-- (LKH3) (AAAI 2024)
8.83%
1.3s
10.2%
6.8s
8.27%
11.24
OOM
OOM
ELG x (UCAL 2024)
14.56%
22.06%
00M
OOM
OOM
INViT-3V greedy (ICML. 2024)
17.8%
11.4s
22.41%
1.4m
31.66%
4.2m
50.17%
2.9h
44.67%
8.3h
UDC (z=1000) (NeurIPS 2024)
5.29%
6.2m
OOM
DOM
OOM
OOM
Ours greedy
5.01%
3.01%
5.49%
1.50%
20.62s
0.60%
8.06m
-..2%%
Ours PRC10
4.52%
0.7s
4.65%
3.9s
4.43%
6.8s
6.52%
28s
4.23%
59s
Ours PRC100
3.31%
8.0s
2.05%
46s
0.59%
1.3m
0.11%
5.49m
-1.00%
11.5m
P EXPERIMENTS ON LARGE-SCALE VRP INSTANCES BEYOND 100K NODES
Our method can scale VRP instances beyond 100K nodes in zero-shot manner. To demonstrate this, we use the model trained on TSP100K to solve TSP200K and TSP500K instances. The results in Table 23 indicate that our method can successfully solve these extremely large-scale instances with relatively low optimality gaps.
The model's performance can be further improved if it is fine-trained using larger problem instances (e.g., on TSP500K). However, the time required to generate pseudo-labels increases with the size of the training data. Thereby, training models on very large-scale VRP instances can result in prohibitively long training times. A possible coping strategy is to adopt multi-GPU parallel training. which will be studied in our future work.
26

---

Published as a conference paper at ICLR 2025
Table 23: Results on TSP200K and TSP500K instances
Scale
TSP200K
TSP500K
Gap
Time
Memory
Gap
Time
Memory
Ours PRC100
3.57%
33m
3166MB
6.03%
1.1h
7916MB
Q LICENSES
The licenses for the codes and the datasets used in this work are listed in Table 24
Table 24: List of licenses for the codes and datasets we used in this work.
Resouree
Type
Link
License
(Heligun
hrp:://ebbsotel44 keld/rescarctvl KH-V
Available
research use
HGS
VI3RI 2024
Code
https///ithbb...
http://ltbuub.co/hhhooe.e. cow/chhwos/yyHHyy..... Hyygos
MIT License
Conedrua
al. 2006
Code
http://github.com/ wowssoeeoooooooooooo.o..
BSD 3-Clause License
hmps http:///hhhh..oo/l..//////
LEHD
Ld0 01 2025
Code
http: Mgghh/uttbboooo.
MIT License
single objective/LEHID
BQ Drakulic
hrpp://jthah.com/nvee///
BY-NC-SA
GLOPP af 2074
Code
hipps wtt.h//w.woo//hcyyyyylliti.....
MIT License
H-TSP
2043
Code
Optimizationn
S HUST/H-TSP
MIT License
DIFUSCO
18m
2023
Code
hitp wtt.....o//dwwdd-S/wienn SonnEElwaddSS///DFFUSSOO
MIT License
INVI
Fung al A034)
Code
htts://gitbub http://iibub.oom. wom/Kasmmggacaa Unah/INVIT
Available for academic research use
AI-GCNTIMUTS
2011
Code
hrps fiithahhh.hh.omo/... Sumn/Sael.yYX/SS
Ant-CCRN--MCSS
MIT License
(aa (114)
Code
htps flg. Sowp.///////////LL.
MIT License
NKI Xin 2
2021b
Code
htip://gitbe http://itbbboooooooe wwt.://iaagggiccc//eeee.........
Available for acudemic research use
UDC
Zheng
Code
http https//gitta..com. www.hhhow///AMM.....o CrunCN/NNC. code/treemmiin
MIT License
single objectivv/ LD--aaage-cc---ee---CCCC arge-scall-COmmmsstr
SIGD
Pirnay Grimm 2024
Code
https//githube comt htwp//riammla//gambeeeee
Available for academic research use
TSPLI
Dutaset
http:/comoppt uni heidelberg
Available for any commercial
htp://vpp galges laggo.i.i wttp:/.w....ue.eeee..e.eee.ee pac-ria beyndex, phopen/
Available
27


Reviews:
[{"review_id": "2upleIn5FK", "reviewer": "Reviewer_R2Zn", "summary": "This paper presents a lightweight cross-attention mechanism with linear complexity to improve the efficiency of the NCO model in solving large-scale VRPs. By propagating node embeddings through representative nodes, the cross-attention mechanism maintains effective interactions between nodes while achieveing low complexity. The experimental results on TSP and CVRP with scaling up to 100K show that the proposed method is effective", "questions": "Pls refer to the weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The complexity of the proposed method is lower than the traditional methods.", "weaknesses": "The complexity analysis seems to be unfair to compare with the traditional method in that the proposed method should consider not only the computational complexity in single round, but also accumulate all computation in iterative reconstruction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mk7UOGcJ9E", "reviewer": "Reviewer_vFLE", "summary": "In their paper, the authors present a deep learning-based, lightweight model and training scheme for solving VRPs. Specifically, the propose replacing the expensive self-attention mechanism in transformer-based solvers with cross-attention based on the first and last node of the currently constructed route. Additionally, they rely on Self-improved training (SIT) as a training mechanism instead of reinforcement or supervised learning. SIT decomposes the total solution into multiple partial solutions and resolves them to find the optimal subsolution.", "questions": "Are the baselines retrained in this scenario? \nHave you tried using augmentation methods similar to POMO on your own cross-attention transformer?\nHow generalizable to other models do you think this SIT method is?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper presents a novel learning method for deep-learning-based approaches in VRPs. SIT seems to improve performance significantly with the cross-attention making previously prohibitive problem sizes now possible. The performance of the model also seems to be very good, consistently beating the baselines.", "weaknesses": "I have some concerns about the evaluation procedure. Specifically, current transformer-based methods such as POMO are trained on small-scale problems (around 100 nodes) and are known to generalize poorly to bigger problems. Comparing to pre-trained models on the larger problems is a bit misleading as in this scenario the authors' method is trained and tested on the same size problem while the other baselines are forced to generalize. \n\nMoreover, and especially considering the SIT training scheme, it might have been good to see a replacement where these deep learning methods are applied on the partial problems with a size of 100 etc. that is closer to their training setting. This would help both see the effectiveness of SIT in other models and better compare with the introduced cross-attention. Alternatively, a direct comparison on smaller problems (of the same training size as the baselines) would have been useful. \n\n Finally, an important baseline would have been NeuroLKH [1] which combines deep learning methods with the LKH algorithm. It was shown to consistently beat LKH with much smaller runtimes which would be interesting to see since, in some cases, the margin between LKH and the authors' methods is not so large.\n\n[1] Xin, Liang, et al. \"Neurolkh: Combining deep learning model with lin-kernighan-helsgaun heuristic for solving the traveling salesman problem.\" Advances in Neural Information Processing Systems 34 (2021): 7472-7483.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z6MjNlzljS", "reviewer": "Reviewer_S6Ze", "summary": "The paper proposes an efficient cross attention mechanism and a self-improved training (SIT) procedure for scaling up NCO solvers. The paper introduces representative embeddings that serve as an information bottleneck to reduce the complexity of attention. SIT algorithm iteratively improves the solution by using parallel local reconstructions. The resulting model achieves low complexity and is scalable to large VRP instances.", "questions": "Please address the weaknesses mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The efficient cross attention mechanism is a novel way to reduce the complexity of transformer based NCO solvers. While the idea is simple, the experiments show that cross attention is really effective for large scale VRP. \n- The SIT algorithm is simple yet effective although local self improvement is not exactly a novel thing.\n- The paper is well written.", "weaknesses": "- The Related work section is pushed to the appendix. I know there is a lot to say about the proposed method, but the related work should be in the main text. The related work is not very well written. To improve readability, please divide the text into subsections, each discussing a different approach to the problem.\n- The paper did not compare with strong baselines such as ELG (Gao et al., 2024), GLOP (more revisions) (Ye et al., 2024) although the authors mentioned them in the related work section.\n- The paper lacks a comparison with other self improvement methods. Some examples of self improvement approach to NCO are [1]\n- The paper did not compare the sizes of the models. More parameters often lead to better results. To ensure a fair comparison, the experiment should be conducted on models of the same size.\n\n[1] Self-Improvement for Neural Combinatorial Optimization: Sample without Replacement, but Improvement. Jonathan Pirnay, Dominik G. Grimm", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Up9PgamBJC", "reviewer": "Reviewer_w834", "summary": "The paper addresses the challenge of solving large-scale vehicle routing problems (VRPs) using neural combinatorial optimization (NCO). The authors propose a novel approach that combines a lightweight cross-attention mechanism with a self-improved training (SIT) algorithm. The method is evaluated on multiple benchmark datasets, demonstrating significant improvements over existing methods in terms of solution quality and computational efficiency.", "questions": "1) How does the proposed method scale to even larger problem instances beyond 100K nodes? Are there any known limitations or bottlenecks in the current implementation?\n\n2) How does the proposed method compare to recent reinforcement learning-based approaches for VRPs in terms of solution quality and computational efficiency? Are there any specific advantages or disadvantages?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper introduces a lightweight cross-attention mechanism that significantly reduces the computational complexity of NCO models, addressing a key limitation in handling large-scale VRPs.\n\n2) The experimental setup is robust, using a variety of benchmark datasets and comparing against multiple state-of-the-art methods.\n\n3) The paper is well-structured and clearly written, making it easy to follow the methodology and understand the contributions.", "weaknesses": "1) Although the paper compares against several state-of-the-art methods, a more detailed comparison with recent advancements in reinforcement learning-based approaches for VRPs could provide a more comprehensive evaluation.\n\n2) An exploration of the limitations of the cross-attention mechanism and potential areas for improvement would enhance the paper's depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)