SlowGuess/NeurIPS_2023_OCR huggingface dataset:
normalization, which leads to more stable training and better performance in our experiments. Finally, a linear layer projects the output of the last attention block into a vector of size  $N$ , from which unfeasible actions, corresponding to the origin and destination nodes, are masked out, before applying a softmax operator so as to interpret the scalar node values for all allowed nodes as action probabilities. Note the absence of autoregressive decoder layers. In the asymmetric TSP, node coordinates are not given, but the travelling cost between nodes is provided through a (possibly asymmetric) matrix. The policy model for path-ATSP (Fig. 2 right) differs slightly from that for path-TSP. Initially, a randomly generated identifier is assigned to each node, which is embedded by a linear layer. The cost matrix is then incorporated by using a weighted graph convolution operation in each layer of the model before applying the attention mechanism. The edge weights of the graph are obtained by some standard normalization of the cost matrix. The use of random node identifiers as model input is justified in ATSP by the absence of node specific features. In problems where such features are available, a random identifier can still be added as extra node feature, as that often improves performance.
Trajectory generation. We train our model by imitation of expert trajectories, using a cross-entropy loss. Such trajectories are extracted from pre-computed (near) optimal solutions for instances of a relatively small and fixed size. Even for hard problems, we leverage the fact that it is generally possible to efficiently solve small instances (e.g. using MILP solvers). Of course the quality of the learned policy will be assessed on larger and therefore more challenging instances. Note that optimal solutions are not directly in the form of trajectories, i.e. sequences of construction steps. While Prop. 2 guarantees that a trajectory exists for any solution, it is usually not unique. In the TSP, an optimal tour corresponds to two possible trajectories (one being the reverse of the other). In the CVRP, each subtour similarly corresponds to two possible trajectories, and in addition, the different orders of the subtours lead to different trajectories. We noticed that this final order has an impact on the performance (see Appendix E.3). On the other hand, any sub-sequence from a fixed trajectory solution can be interpreted as the solution of a sub-instance. Note that these sub-instances will vary both in size and node distribution, therefore by training on them, we implicitly encourage the model to work well across sizes and node distributions, and generalize better than if such variations were not seen during the training.
Complexity. Because of the quadratic complexity of self-attention, and the fact that we call our policy at each construction step, the total complexity of our model, for an instance of size  $N$ , is  $\mathcal{O}(N^3)$ , whereas closely related transformer-based models such as the TransformerTSP [7] and the Attention Model [28] have a total complexity of  $\mathcal{O}(N^2)$ . We experimentally show in Sec. 6 that our model still provides better-quality solutions faster than these baselines. This is because most neural models are called many times per instance in practice, typically for sampling from the learned policy or within a beam search. In contrast, we observe that doing a single rollout of our policy yields excellent results. Intuitively, given the complexity (NP-hardness) of the targeted COPs, it makes sense to spend as much computation as possible at each time step, provided the whole rollout is fast enough. On the other hand, the quadratic attention bottleneck can be addressed by using some linear attention models. In our experiments, we show that replacing the Transformer by the linear PerceiverIO [19] significantly accelerates inference while still providing a competitive performance.
Generic frameworks for CO. Generic frameworks have been crucial in the development and adoption of CO since they allow to formulate a COP in a certain format and then give it as an input to a (black box) solver. Among the most widespread frameworks are Mixed-Integer Programming [11] and Constraint Programming [39]. On the other hand, Markov Decision Processes are a powerful and ubiquitous tool to model sequential decision making problems and make them amenable to Machine Learning. While using MDPs to solve CO is not new, there are very few attempts to make it a general framework. Khalil et al. [23] makes a first attempt to define a generic MDP for greedy heuristics on graph problems. Drori et al. [13] also focuses on graph problems and provides a flexible and efficient GNN-based architecture to train with Reinforcement Learning (RL). Unlike these methods, our approach does not assume a graph or any structure on the COPs to define the (direct and BQ) MDP and we are the first to prove the equivalence between our proposed MDPs and solving the COP.
Neural-based constructive heuristics. Many NCO approaches construct solutions sequentially, via auto-regressive models. Starting with the seminal work by Vinyals et al. [45], which proposed the Pointer network trained in a supervised way, Bello et al. [4] trained the same model by RL for the TSP and Nazari et al. [34] adapted it for the CVRP. Kool et al. [28] introduced an Attention-based encoder-decoder Model (AM) trained with RL to solve several routing problems. This architecture was reused, with a few extensions, in POMO [29] and further combined with efficient search procedures in [10]. TransformerTSP [7] uses a similar architecture with a different decoder for the TSP. While most of these auto-regressive constructive approaches used RL, another line of work focuses on learning, in a supervised way, a heatmap of solution segments: Nowak et al. [35] trained a Graph Neural Network to output an adjacency matrix, which is converted into a feasible solution using beam search; Joshi et al. [21] followed a similar framework and trained a Graph Convolutional Network instead, that was used by [14] as well. Recently Sun and Yang [42] proposed DIFUSCO, a graph-based diffusion model to output the heatmap. Although such non-autoregressive methods are promising for large instances since they avoid the sequential decoding, they generally rely on sophisticated search methods (such as Monte Carlo Tree Search) in order to get a high-quality solution from the heatmap. Our paper explores the less common combination of supervised (imitation) learning to train a policy (not a heatmap) and achieves excellent performance without any search. In the context of planning, a similar strategy was successfully used in [16] to learn generalized reactive policies on small planning problems instances while generalizing well to larger ones.
Step-wise approaches. Instead of encoding an instance once and decoding sequentially as in the AM [28], Peng et al. [36] proposed to update the encoding after each subtour completion for CVRP while Xin et al. [47] updated it after each node selection in TSP. Xin et al. [46] extended on this idea by introducing the Multi-Decoder Attention Model (MDAM) which contains a special layer to efficiently approximate the re-embedding process. Since MDAM constitutes the most advanced version, we employ it as a baseline in our experiments.
Generalizable NCO. Generalization to different instance distributions, and esp. larger instances, is regarded as one of the major challenges for current NCO approaches [22, 33]. Fu et al. [14] trained a Graph Convolution model in a supervised manner on small graphs and used it to solve large TSP instances, by applying the model on sampled subgraphs and using an expensive MCTS search to improve the final solution (Att-GCN+MCTS). While this method achieves excellent generalization on TSP instances, MCTS requires a lot of computing resources and is essentially a post-learning search strategy. Geisler et al. [15] investigate the robustness of NCO solvers through adversarial attacks and find that existing neural solvers are highly non-robust to out-of-distribution examples and conclude that one way to address this issue is through adversarial training. In particular, Xin et al. [48] trains a GAN to generate instances that are difficult to solve for the current model. Manchanda et al. [32], Son et al. [41], Zhou et al. [50] take a different approach and leverage meta-learning to learn a model that is easily adaptable to new distributions. In DIMES, Qiu et al. [38] combines meta-learning with a continuous parametrization of the candidate solutions that allows to solve large-scale TSP instances. Accounting for symmetries in a given COP is a powerful idea to boost the generalization performance of neural solvers. For example, for problems on Euclidean graphs, Sym-NCO [24] make use of solution symmetries, to enforce robust representations while [29] exploits it as part of their loss function during training and at inference time to augment the set of solutions. Note that the above adversarial or meta-learning strategies and data augmentation are orthogonal to our approach of providing an efficient MDP which has beneficial effects irrespective of the training method. Finally,
some hybrid methods have been successful in solving large-scale routing problems [30, 51, 18]; these are based on the divide-and-conquer principle: they learn how to split large instances into smaller ones, that are then efficiently solved by external specialized solvers.
To evaluate the effectiveness of our method, we test it on four routing problems - Euclidean TSP, CVRP, OP and Asymmetric TSP and one non-routing problem - KP (details and results for the latter are in Appendix C). For all routing problems, we train our model and all baselines on synthetic instances of size 100, following the same procedure of data generation as in [28]. For asymmetric TSP, we use Euclidean distance matrices and randomly break symmetries. We choose graphs of size 100 because it is the largest size for which (near) optimal solutions are still reasonably fast to obtain, and such training datasets are commonly used in the literature. Then, we test the trained models on synthetic instances of size 100, 200, 500 and 1K generated from the same distribution, as well as the standard TSPLib and CVRPLib datasets. Except for perceiver-based models and OP (dist), we limit the subgraphs to the 250 nearest neighbours of the origin node at test time. This reduces inference time while not significantly impacting the solution quality (see Appendix D).
Hyperparameters and training. We use the same hyperparameters for all problems. The model has 9 layers, each built with 12 attention heads with embedding size of 192 and dimension of feed-forward layer of 512. Our model is trained by imitation of expert trajectories, using a cross-entropy loss. Solutions of these problems are obtained by using the Concorde solver [1] for TSP, LKH heuristic [17] for ATSP and CVRP, and EA4OP heuristic [26] for OP. We use a dataset of 1 million solutions. To sample trajectories out of this dataset, we note that in the case of TSP, any sub-path of the optimal tour is also an optimal solution to the associated path-TSP sub-problem, hence amenable to our path-TSP model. We therefore form minibatches by first sampling a number  $n$  between 4 and  $N$  (path-TSP problems with less than 4 nodes are trivial), then sampling sub-paths of length  $n$  from the initial solution set. At each epoch, we sample a sub-path from each solution. By sampling subsequences among all possible inflexes of the optimal solutions, we automatically get an augmented dataset, which some previous models had to explicitly design their model for [29]. We use a similar sampling strategy for CVRP, OP and KP (see Appendix .A, B and C). Batches of size 1024 are formed, and the model is trained for 500 epochs. We use Adam [25] as optimizer with an initial learning rate of  $7.5\mathrm{e} - 4$  and decay of 0.98 every 50 epochs.
Baselines and test datasets. We compare our model with the following methods: OR-Tools [37], LKH [17], and Hybrid Genetic Search (HGS) for the CVRP [44] as SOTA non-neural methods; DIFUSCO+2opt, Att-GCN+MCTS and SGBS as hybrid methods; and AM, TransformerTSP, MDAM, POMO, DIMES and Sym-NCO as deep learning-based constructive methods. Description of all baseline models are provided in Sec. 5. For all deep learning baselines we use the corresponding model trained on graphs of size 100 and the best decoding strategy<sup>2</sup>. Following the same procedure as in [14], we generate four test datasets with graphs of sizes 100, 200, 500 and 1000. For CVRP, we use capacities of 50, 80, 100 and 250, respectively. In addition, we report the results on TSPLib instances with up to 4461 nodes and all CVRPLib instances with node coordinates in the Euclidean space. For all models, we report the optimality gap and the inference time. The optimality gap for TSP is based on the optimal solutions obtained with Concorde. For CVRP, although HGS gives better results than LKH, we use the LKH solutions as a reference to compute the "optimality" gap, in order to be consistent (and easily comparable) with previous works. For the OP task, we train and test the model using a "distance" variant of the problem, as described in [28]. For fixed maximum lengths, we chose  $T^{100} = 4$ ,  $T^{200} = 5$ ,  $T^{500} = 6$ , and  $T^{1000} = 7$ . While the optimality gap is easy to compute and compare, running times are harder to compare since they strongly depend on the implementation platforms (Python, C++, hardware (GPU, CPU), parallelization, batch size, etc. In our experiments, we run all deep learning models on a single Nvidia Tesla V100-S GPU with 24GB memory, and other solvers on Intel(R) Xeon(R) CPU E5-2670 with 256GB memory, in one thread.
Results. Table 1 summarizes our results for Euclidean TSP, CVRP and OP as well as asymmetric TSP. For all problems, our model shows superior generalization on larger graphs, even with a
$^{2}$ Att-GCN+MCTS is originally trained on graph sizes of 50. For DIMES and DIFUSCO authors provide generalization tables just for two types of decoding (greedy+2opt for DIFUSCO and RL+sampling for DIMES) and on some test datasets, so we report them in our results
greedy decoding strategy, which generates a single solution while all other baselines generate several hundreds (and select the best among them). In terms of running time, with greedy decoding, our model is slightly slower than Sym-NCO baseline, competitive with POMO, and significantly faster than other models. Beam search further improves the optimality gaps, but as expected, it takes more time. On the other hand, our model can be accelerated by replacing the quadratic attention blocks by the PerceiverIO architecture [19] (more details in Appendix E.1). This results in a considerable reduction of inference time for larger graphs, at the cost of some performance drop. Even with this trade-off, our model achieves remarkable performance compared with other NCO baselines. Figure 3 shows optimality gap versus running time for our models and other baseline models. Our models clearly outperform other models in terms of generalization to larger instances. The only models that are competitive with ours are Att-GCN+MCTS and DIFUSCO+2opt, but both are hybrid methods with an expensive search on top of the output of the neural network, and they are 2 to 15 times slower while being designed for TSP only. Finally, in addition to the synthetic datasets, we test our models on TSPLib and VRPLib instances, which are of varying graph sizes, node distributions, demand distributions and vehicle capacities. Table 1 shows that our model significantly outperforms the end-to-end baseline policies even with the greedy decoding strategy.
Figure 3: Generalization results on different graph sizes for TSP (left) and CVRP (right). Lower and further left is better.
We proposed a principled framework to formulate COPs as efficient symmetry-invariant MDPs. First, we presented how to derive a direct MDP for any COP that is amenable to a constructive approach, and proved the exact correspondence between the optimal policies of this direct MDP and the optimal solutions of the COP it stems from. Second, for the widespread class of recursive COPs, we introduced bisimulation quotienting to leverage the symmetries of the partial solutions. We showed that the resulting BQ-MDP has a significantly reduced state space while its optimal policies still exactly solve the original COP. We then designed a simple Transformer-based architecture as an exemplification of a possible policy network for the BQ-MDPs, and applied it to five classical COP problems: (A)TSP, CVRP, OP and KP. The resulting policy generalizes particularly well, and significantly outperforms not only similar end-to-end neural-based constructive solvers but also hybrid methods that combine learning with search. A limitation of our approach, shared with all constructive methods, is that it can only handle problems for which feasibility is easy to ensure throughout the construction process. In order to address problems with more challenging global constraints, approaches which combine learning and constraint programming (e.g. [8]) may be a promising direction. While training on relatively small instances allowed us to use imitation learning, our proposed MDPs and the associated model could also be trained with Reinforcement Learning. This would be particularly interesting to extend our framework from deterministic to stochastic COPs.
Test (10k inst.)	Generalization (128 instances)
N=100	N=200	N=500	N=1000
opt. gap	time	opt. gap	time	opt. gap	time	opt. gap	time
TSP	Concorde	0.00%	38m	0.00%	2m	0.00%	40m	0.00%	2.5h
OR-Tools	3.76%	1.1h	4.52%	4m	4.89%	31m	5.02%	2.4h
Att-GCN+MCTS*	0.04%	15m	0.88%	2m	2.54%	6m	3.22%	13m
DIFUSCO G+2opt*	0.24%	-	-	-	2.40%	4m	3.40%	12m
SGBS (10,10)	0.06%	15m	0.67%	1m	11.42%	20m	25.25%	2.9h
AM bs1024	2.51%	20m	6.18%	1m	17.98%	8m	29.75%	31m
TransTSP bs1024	0.46%	51m	5.12%	1m	36.14%	9m	76.21%	37m
MDAM bs50	0.39%	45m	2.04%	3m	9.88%	13m	19.96%	1.1h
POMO augx8	0.13%	1m	1.57%	5s	20.18%	1m	40.60%	10m
DIMES RL+S*	-	-	-	-	16.07%	1m	17.69%	2m
Sym-NCO s100	0.64%	13s	2.28%	2s	21.64%	13s	37.51%	2m
BQ-perceiver G	0.97%	1m	2.09%	2s	5.22%	8s	8.97%	22s
BQ-transformer G	0.35%	2m	0.54%	9s	1.18%	55s	2.29%	2m
BQ-transformer bs16	0.01%	32m	0.09%	3m	0.55%	15m	1.38%	38m
CVRP	LKH	0.00%	15.3h	0.00%	30m	0.00%	1.3h	0.00%	2.8h
HGS	-0.51%	15.3h	-1.02%	30m	-1.25%	1.3h	-1.10%	2.8h
OR-Tools	9.62%	1h	10.70%	3m	11.40%	18m	13.56%	43m
AM bs1024	4.18%	24m	7.79%	1m	16.96%	8m	86.41%	31m
MDAM bs50	2.21%	56m	4.33%	3m	9.99%	14m	28.01%	1.4h
POMO augx8	0.69%	1m	4.77%	5s	20.57%	1m	141.06%	10m
Sym-NCO s100	1.46%	15s	4.84%	3s	18.64%	13s	119.60%	9m
SGBS (4,4)	0.08%	15m	2.47%	1m	14.98%	20m	151.07%	2.9h
BQ-perceiver G	5.63%	1m	5.49%	3s	6.39%	8s	10.21%	24s
BQ-transformer G	2.79%	2m	2.81%	9s	3.64%	55s	5.88%	2m
BQ-transformer bs16	0.95%	32m	0.77%	3m	1.04%	15m	2.55%	38m
OP (DIST.)	EA4OP	0.00%	1.4h	0.00%	3m	0.00%	11m	0.00%	45m
AM bs1024	2.19%	17m	8.04%	1m	20.73%	4m	29.96%	15m
MDAM bs50	0.69%	32m	2.98%	49s	15.36%	4m	26.55%	15m
SymNCO s200	0.45%	4m	3.39%	2s	15.23%	37s	26.95%	2m
BQ-transformer G	-0.15%	2m	0.62%	8s	3.26%	42s	8.40%	8.9m
BQ-transformer bs16	-1.19%	35m	-0.84%	2.1m	1.04%	21m	4.81%	2.2h
ATSP	LKH	0.00%	18m	0.00%	29s	0.00%	2.3m	0.00%	9.4m
MatNet s128†	0.93%	45m	124.20%	4m	-	-	-
BQ-GCN-transformer G	1.27%	1m	1.56%	4s	4.22%	12s	11.59%	14s
BQ-GCN-transformer bs16	0.96%	19m	1.41%	1m	2.43%	3m	8.26%	7m
Table 1: BQ-models with greedy rollouts (G) or Beam Search (bs) versus classic and neural baselines. The results for models* are taken from the original papers. For a fair comparison, model † are evaluated on instances generated from the same distribution as the training datasets for this model, which may differ from our training and/or test distribution.
Size	MDAM bs50	POMO x8	BQ (ours)
greedy	bs16
<100	3.06%	0.42%	0.34%	0.06%
100-200	5.14%	2.31%	1.99%	1.21%
200-500	11.32%	13.32%	2.23%	0.92%
500-1K	20.40%	31.58%	2.61%	1.91%
>1K	40.81%	62.61%	6.42%	5.90%
All	19.01%	26.30%	3.30%	2.55%
Set (size)	MDAM bs50	POMO augx8	BQ (ours)
greedy	bs16
A (32-80)	6.17%	4.86%	4.52%	1.18%
B (30-77)	8.77%	5.13%	6.36%	2.48%
F (44-134)	16.96%	15.49%	6.63%	5.94%
M (100-200)	5.92%	4.99%	3.58%	2.66%
P (15-100)	8.44%	14.69%	3.61%	1.54%
X (100-1K)	34.17%	21.62%	9.94%	7.22%
All (15-1K)	22.36%	15.58%	7.62%	4.83%
Table 2: Experimental results on TSPLib (left) and CVRPLib (right).
[49] Cong Zhang, Wen Song, Zhiguang Cao, Jie Zhang, Puay Siew Tan, and Chi Xu. Learning to dispatch for job shop scheduling via deep reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, pages 1621-1632, Red Hook, NY, USA, December 2020. Curran Associates Inc. ISBN 978-1-71382-954-6.
[50] Jianan Zhou, Yaoxin Wu, Wen Song, Zhiguang Cao, and Jie Zhang. Towards Omni-generalizable Neural Methods for Vehicle Routing Problems, June 2023.
[51] Zefang Zong, Hansen Wang, Jingwei Wang, Meng Zheng, and Yong Li. RBG: Hierarchically Solving Large-Scale Routing Problems in Logistic Systems via Reinforcement Learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4648-4658, Washington DC USA, August 2022. ACM. ISBN 978-1-4503-9385-0.