SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 2: Performance of COMPASS and the main baselines aggregated across several tasks over three problems (TSP, CVRP, and JSSP). For each task (problem type, instance size, mutation power), we normalize values between 0 and 1 (corresponding to the worst and best performing RL method, respectively). Hence, all tasks have the same impact on the aggregated metrics. COMPASS surpasses the baselines on all of them, showing its versatility for all types of tasks and in particular, its generalization capacity.
In Distribution Instances drawn from the training distribution (see first cols. in Tables 1a to 1c).
Larger Instances Instance with a larger size than the training data. See generalization in Tables 1a to 1c.
Small/Medium/Large Shift Instances that have been procedurally mutated to be out of the training distribution. Detailed results in Fig. 3.
Few-Shot The standard benchmark with a lower budget (typically  $10\%$  of the usual). Numerical results in Appendix I.
Table 1: Results of COMPASS against the baseline algorithms for (a) TSP, (b) CVRP, and (c) JSSP problems. The methods are evaluated on instances from training distribution as well as on larger instance sizes to test generalization.
(a) TSP
Method	Training distr.	n = 125	n = 150	n = 200
n = 100	n = 125	n = 150	n = 200
Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time
Concorde	7.765	0.000%	82M	8.583	0.000%	12M	9.346	0.000%	17M	10.687	0.000%	31M
LKH3	7.765	0.000%	8H	8.583	0.000%	73M	9.346	0.000%	99M	10.687	0.000%	3H
POMO (greedy)	7.796	0.404%	37S	8.635	0.607%	6S	9.440	1.001%	10S	10.933	2.300%	21S
POMO (sampling)	7.779	0.185%	2H	8.609	0.299%	20M	9.401	0.585%	32M	10.956	2.513%	70M
Poppy 16	7.766	0.013%	2H	8.587	0.050%	20M	9.359	0.141%	32M	10.795	1.007%	70M
EAS	7.779	0.176%	7H	8.601	0.252%	62M	9.382	0.381%	2H	10.758	0.660%	210M
COMPASS (ours)	7.765	0.002%	2H	8.586	0.036%	20M	9.354	0.083%	32M	10.724	0.348%	70M
(b) CVRP
Method	Training distr.	n = 125	Generalization	n = 200
n = 100	Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time
LKH3	15.65	0.000%	-	17.50	0.000%	-	19.22	0.000%	-	22.00	0.000%	-
POMO (greedy)	15.874	1.430%	2M	17.818	1.818%	<1M	19.750	2.757%	1M	23.318	5.992%	2M
POMO (sampling)	15.713	0.399%	4H	17.612	0.642%	43M	19.488	1.393%	1H	23.378	6.264%	100M
Poppy 32	15.663	0.084%	4H	17.548	0.276%	42M	19.421	1.044%	1H	23.352	6.144%	100M
EAS	15.661	0.068%	13H	17.517	0.094%	2H	19.285	0.341%	4H	22.264	1.120%	7H
COMPASS (ours)	15.594	-0.361%	4H	17.511	0.064%	42M	19.313	0.485%	1H	22.462	2.098%	100M
(c) JSSP
Method	Training distr.	Generalization
10 × 10	Time	15 × 15	Time	20 × 15	Time
Obj.	Gap	Obj.	Gap	Obj.	Gap
OR-Tools	807.6	0.0%	37S	1188.0	0.0%	3H	1345.5	0.0%	80H
L2D (sampling)	871.7	8.0%	8H	1378.3	16.0%	25H	1624.6	20.8%	40H
Single	862.1	6.7%	3H	1302.6	9.6%	5H	1503.0	11.7%	8H
Poppy 16	849.7	5.2%	3H	1290.4	8.6%	5H	1495.7	11.2%	8H
EAS	858.4	6.3%	5H	1295.2	9.0%	9H	1498.0	11.3%	11H
COMPASS (ours)	845.5	4.7%	3H	1282.8	8.0%	5H	1485.6	10.4%	8H
Code availability We release the code<sup>1</sup> used to train our method and to run all baselines. We also make our checkpoints available for all three problems, along with the datasets necessary to reproduce the results. To ensure fair comparison and extend our evaluation to new settings, we implemented all baselines within the same codebase. For the three problems, we used the JAX (Bradbury et al., 2018) implementations from Jumanji (Bonnet et al., 2023) to leverage hardware accelerators (e.g. TPU). Our code is optimized for TPU v3-8, which is the hardware used for our experiments.
4.1 Standard benchmarking on TSP, CVRP, and JSSP
We evaluate our method on benchmark sets frequently used in the literature (Kool et al., 2019; Kwon et al., 2020; Grinsztajn et al., 2022; Hottung et al., 2022). Specifically, for TSP and CVRP, we use datasets of 10 000 instances drawn from the training distribution, with the positions of 100 cities/customers uniformly sampled within the unit square, and three datasets not seen during training, each containing 1000 problem instances but with larger sizes: 125, 150 and 200, also generated from a uniform distribution over the unit square. We use the exact same datasets as in the literature.
Results The average performance of each method across all problem settings are presented in Table 1. We find that COMPASS demonstrates superior performance on 9 out of the 11 test sets considered. Moreover, the degree of improvement is significant across all problem types. On TSP and JSSP, COMPASS reduces the optimality gap on the training distribution by a factor of 6.5 and 1.3, respectively. On CVRP, COMPASS is the only RL method able to outperform the industrial solver LKH. Finally, COMPASS is also found to generalize well to larger problem instances unseen during training. COMPASS obtains the best solutions in all TSP and JSSP sets and is only outperformed by
$^{1}$ Code, checkpoints and evaluation sets are available at https://github.com/instadeepai/compass
Figure 3: Relative difference between COMPASS and baselines as a function of mutation power. COMPASS outperforms the baselines on all 18 evaluation sets. Most methods have a decreasing performance ratio, showing that COMPASS generalizes better: its evolution strategy is able to find areas of its latent space that are high-performing, even on instances that are out-of-distribution.
EAS on two instance sizes of CVRP. Nevertheless, EAS is  $50\%$  slower and more computationally expensive as it requires updating an entire subset of its network's weights (see Appendix A), as opposed to simply navigating the 16-dimensional latent space of policies as is done in COMPASS.
The same benchmark is also reported with the "augmentation trick" in Table 2 for TSP and CVRP. This trick can only be used for the routing problem and is not applicable for JSSP. Interestingly, COMPASS is the only method that performs on par or better without the "augmentation trick", showing its ability to adapt and find diversity in its latent space rather than through a problem-specific trick. Other conclusions drawn above remain unchanged in this setting.
4.2 Robustness to generalization: solving mutated instances
To further study the generalization ability of our method, we consider the mutation operators introduced by Bossek et al. (2019) to procedurally transform instances drawn from the training distribution. By progressively increasing the power of the applied mutations we construct new datasets that are increasingly far from the training distribution whilst not modifying the overall size of the problem.
We use 9 different mutation operators (explosion, implosion, cluster, rotation, linear projection, axis projection, expansion, compression and grid). One can find an illustration of the entire set of mutations along with their mathematical definition in Appendix C. Interestingly, it enables us to evaluate the methods on instances that look closer to real-life situations. For instance, the operator that gathers nodes in a cluster can mimic a dense city surrounded by its nearby suburbs. In practice, each mutation operator is parameterized by a factor that controls the probability of mutating each node of the instance - referred to as mutation power - this factor directly impacts the shift between the training distribution and the new distribution. We use 10 values, going from 0 (no change) to 0.9 (highly mutated instances).
Results We plot the relative performance of the baselines compared to COMPASS in Fig. 3. A negative performance ratio indicates that a method does not provide as good of a solution as COMPASS, and we observe that this is the case for all baseline methods, at all mutation strengths, on both TSP and CVRP. Moreover, COMPASS is seen to generalize significantly better than the methods that only rely on stochastic sampling for their search, namely POMO and Poppy. This validates our intuition that adaptive policies are especially important for handling out-of-distribution data, where the optimal policy may be significantly different to that needed during pre-training. Even compared to EAS, which fine-tunes the policy to the target problem instance, we find that COMPASS maintains a significant performance gap across all mutation strengths. This result is particularly noteworthy as our approach only modifies 16 parameters (the conditioning vector sampled from our latent space), compared to EAS, which updates more than  $10^{4}$  parameters (the embeddings of the instance's nodes).
It is interesting to note that the relative generalization performance of COMPASS compared to EAS is stronger on these mutated instances than the larger instances considered in Section 4.1. We
Figure 4: Evolution of the overall performance and last performance obtained by the methods during their search on TSP150 - averaged on 1000 instances. The right plot reports mean and standard deviations of the most recent shots tried by methods during the search. It illustrates how COMPASS efficiently explores its latent space to search for high-performing solutions.
hypothesize that this is because EAS actively fine-tunes the embeddings of every location in a given problem instance. Therefore, as the problem size increases, so does the number of free parameters to adapt the policy (albeit with commensurately increasing computational overhead). This suggests that further improvements to COMPASS could be possible by increasing the number of adapted parameters (i.e. the latent space dimension), however, we defer further investigation to future works.
4.3 Analysis of the search strategies
In this section, we analyze the structure of the latent space and the behavior of the search procedure both empirically and visually.
Figure 4 details the performance of our considered methods as a function of the overall search budget. The left panel reports the quality of the best solution found so far (i.e. the cumulative performance), whereas the right plot reports the mean and standard deviations of the latest batch of solutions (i.e. the current performance) during the search process. From this, we would highlight three main conclusions. (i) Adaptive methods (COMPASS, EAS) perform well as they are able to improve the mean performance of their solution over time, in contrast to stochastic sampling methods (Poppy, POMO). This also highlights that the latent space of COMPASS has been able to diversify and can be exploited. (ii) Highly-focused (low-variance) search does not always outperform stochastic exploration. Concretely, whilst EAS quickly adapts a policy with better average performance than Poppy (right panel), the additional variance of Poppy's multiple diverse policies means it produces better overall solutions (left panel). (iii) COMPASS is able to combine both of the pre
Figure 5: Contour plot of COMPASS's latent space, reflecting performance on a problem instance. White crosses show the successive means of a CMA-ES component during the search. The width of the path is proportional to the search's variance.
Previously discussed aspects for a highly performant search procedure. By using an adaptive covariance mechanism as well as its multiple components to navigate several regions of the latent policy space, it focuses its search on promising strategies (better average performance) whilst maintaining a broad beam (higher variance).
To better understand how COMPASS's latent space is structured and explored, Fig. 5 presents the trajectory of a single CMA-ES component during the search of a 2D latent space on a randomly chosen problem instance. We can first observe that even for a specific problem instance, there are several high-performing areas of interest which highlights the advantage of having multiple search components. Furthermore, it shows how the evolution strategy explores the space. The search variance is initially high to improve exploration until the search center moves into a high-performing area, whereupon the variance is gradually decreased to better exploit these promising strategies. We
provide additional plots and explanation in Appendix E.2 for other problem instances, demonstrating the spread of the specialised areas depending on the problem instance.
Lastly, it is worth noting that the adaptation mechanism of COMPASS (CMA-ES search) comes with negligible time cost (e.g. three orders of magnitude smaller than the time needed for the environment rollout), which is a strength compared to the costly backpropagation-based updates performed in EAS. We provide additional time analysis in Appendix L.
Table 2: Results of COMPASS and the baseline algorithms with instance augmentation for (a) TSP and (b) CVRP. We also report COMPASS with no augmentation (no aug.).
(a) TSP
Method	Training distr.	Generalization
n = 100	n = 125	n = 150	n = 200
Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap
CVAE-Opt	-	0.343%	6D	8.646	0.736%	21H	9.482	1.45%	30H	-	-
SGBS	7.769	0.058%	9M	-	-	-	9.367	0.220%	8M	10.753	0.619%
SGBS+EAS-Lay	7.769	0.058%	3H	-	-	-	9.359	0.174%	1H	10.727	0.40%
POMO (sampling)	7.767	0.026%	2H	8.594	0.128%	20M	9.376	0.321%	32M	10.916	2.14%
Poppy 16	7.765	0.002%	2H	8.584	0.009%	20M	9.351	0.141%	32M	10.802	1.08%
EAS	7.768	0.039%	7H	8.590	0.082%	66M	9.360	0.175%	138M	10.724	0.350%
COMPASS (aug.)	7.765	0.002%	2H	8.584	0.009%	20M	9.350	0.043%	32M	10.723	0.337%
COMPASS (no aug.)	7.765	0.002%	2H	8.586	0.036%	20M	9.354	0.083%	32M	10.724	0.348%
(b) CVRP
Method	Training distr.	n = 125	Generalization	n = 200
Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time	Obj.	Gap	Time
CVAE-Opt	-	1.36%	11D	17.87	2.08%	36H	19.84	3.24 %	46H	-	-	-
SGBS	15.66	0.01%	10M	-	-	-	19.43	1.08%	4M	22.57	2.59%	9M
SGBS+EAS-Lay	15.594	-0.36%	6H	-	-	-	19.168	-0.27%	2H	21.988	-0.05%	5H
POMO (sampling)	15.67	0.18%	4H	17.56	0.33%	43M	19.43	1.08%	1H	23.24	5.64%	100M
Poppy 32	15.62	-0.14%	4H	17.49	-0.10%	42M	19.32	0.50%	1H	22.94	4.27%	100M
EAS	15.62	-0.21%	13H	17.462	-0.22%	2H	19.213	-0.037%	5H	22.162	0.73%	7H
COMPASS (aug.)	15.65	-0.00%	4H	17.52	0.09%	42M	19.33	0.56%	1H	22.55	2.49%	100M
COMPASS (no aug.)	15.594	-0.36%	4H	17.511	0.06%	42M	19.313	0.49%	1H	22.462	2.10%	100M
We present COMPASS, a novel approach to solving CO problems using RL. Our approach is motivated by the observation that active search is a key component to finding high-quality solutions to NP-hard problems. Finding one-shot solutions that are near-optimal is believed to be impossible. Instead, COMPASS is trained to create a distribution of diverse and specialized policies, conditioned on a structured latent space. This diversification is achieved by using an objective that specializes areas of the space on sub-distributions of problem instances. By navigating this latent space at inference time COMPASS is able to find the most performant policy for a given instance. Empirical results show that COMPASS achieves state-of-the-art performance on 9 out of 11 standard benchmark tasks across three distinct CO problems, TSP, CVRP and JSSP, outperforming prior RL methods based on either stochastic sampling or fine-tuning. We extend the canonical evaluation sets with instances that are procedurally transformed using mutation operators introduced in prior work. This additional set of tasks enables us to assess the generalization ability of COMPASS. We show that COMPASS is particularly robust to out-of-distribution instances, achieving superior performance in all 18 tasks considered. To better understand the benefits of our search procedure, we provide an empirical analysis of the latent space's structure, along with evidence of how it is explored at inference time. We show that, despite having no explicit regularization during training, the latent space exhibits clear regions of interest, and our search procedure is able to explore this space using an evolution strategy to produce high-performing policies. Overall, COMPASS proves to be performant, robust, and versatile on many types of CO problems and is able to provide solutions quickly at a reasonable computational cost.
Limitations and future work. The diversity of the policies contained in the latent space is closely linked to the specialisation that can be obtained from the training distribution, and hence potentially limited. We would like to inspect whether a broader range of policies could be obtained by using an additional unsupervised diversity reward, or by procedurally diversifying the distribution used. Another limitation of our method is the lack of structure in our latent space. Although we proved that it was enough to be successfully explored by an evolution strategy, we hypothesize that a better defined space could be searched through faster. We would like to inspect the use of regularization terms during the training phase to achieve this.
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). We thank anonymous reviewers for comments and helpful discussions that helped improve the paper.
B. Lim, M. Allard, L. Grillotti, and A. Cully. Accelerated quality-diversity for robotics through massive parallelism. arXiv preprint arXiv:2202.01258, 2022.
Y. Ma, J. Li, Z. Cao, W. Song, L. Zhang, Z. Chen, and J. Tang. Learning to iteratively solve routing problems with dual-aspect collaborative transformer. In Advances in Neural Information Processing Systems, volume 34, pages 11096-11107, 2021.
N. Mazyavkina, S. Sviridov, S. Ivanov, and E. Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 134:105400, 2021. ISSN 0305-0548. doi: https://doi.org/10.1016/j.cor.2021.105400. URL https://www.sciencedirect.com/science/article/pii/S0305054821001660.
L. Perron and V. Furnon. OR-Tools, 2019. URL https://developers.google.com/optimization/.
A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657, 2019.
V. Steinbiss, B.-H. Tran, and H. Ney. Improvements in beam search. In Third international conference on spoken language processing, 1994.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
O. Vinyals, M. Fortunato, and N. Jaitly. Pointer networks. In Advances in Neural Information Processing Systems, 2015.
Y. Wu, W. Song, Z. Cao, J. Zhang, and A. Lim. Learning improvement heuristics for solving routing problems. IEEE Transactions on Neural Networks and Learning Systems, 33(9):5057-5069, 2022.
L. Xin, W. Song, Z. Cao, and J. Zhang. Multi-decoder attention model with embedding glimpse for solving vehicle routing problems. In In Proceedings of the 35th National Conference on Artificial Intelligence, AAAI, 2021.
C. Zhang, W. Song, Z. Cao, J. Zhang, P. S. Tan, and C. Xu. Learning to dispatch for job shop scheduling via deep reinforcement learning. In Advances in Neural Information Processing Systems, 2020.