Title: Ensemble-based Deep Reinforcement Learning for Vehicle Routing Problems under Distribution Shift
Venue: NeurIPS 2023
Accepted: yes
Keywords: Vehicle Routing Problem

Distribution shift

Deep Reinforcement Learning

Ensemble Learning
Abstract: While performing favourably on the independent and identically distributed (i.i.d.) instances, most of the existing neural methods for vehicle routing problems (VRPs) struggle to generalize in the presence of a distribution shift. To tackle this issue, we propose an ensemble-based deep reinforcement learning method for VRPs, which learns a group of diverse sub-policies to cope with various instance distributions. In particular, to prevent convergence of the parameters to the same one, we enforce diversity across sub-policies by leveraging Bootstrap with random initialization. Moreover, we also explicitly pursue inequality between sub-policies by exploiting regularization terms during training to further enhance diversity. Experimental results show that our method is able to outperform the state-of-the-art neural baselines on randomly generated instances of various distributions, and also generalizes favourably on the benchmark instances from TSPLib and CVRPLib, which confirmed the effectiveness of the whole method and the respective designs.

Ensemble-based Deep Reinforcement Learning for Vehicle Routing Problems under Distribution Shift

Abstract

While performing favourably on the independent and identically distributed (i.i.d.) instances, most of the existing neural methods for vehicle routing problems (VRPs) struggle to generalize in the presence of a distribution shift. To tackle this issue, we propose an ensemble-based deep reinforcement learning method for VRPs, which learns a group of diverse sub-policies to cope with various instance distributions. In particular, to prevent convergence of the parameters to the same one, we enforce diversity across sub-policies by leveraging Bootstrap with random initialization. Moreover, we also explicitly pursue inequality between sub-policies by exploiting regularization terms during training to further enhance diversity. Experimental results show that our method is able to outperform the state-of-the-art neural baselines on randomly generated instances of various distributions, and also generalizes favourably on the benchmark instances from TSPLib and CVRPLib, which confirmed the effectiveness of the whole method and the respective designs.

1 Introduction

The vehicle routing problem (VRP) is a type of combinatorial optimization problems (COPs) with significant real-world applications, especially in the field of logistics [1]. The goal is to optimally route a fleet of vehicles, originating at a depot, to serve a set of clients. Given the NP-hard nature, heuristic methods are usually preferred to solve VRPs in practice. However, traditional heuristics always treat each VRP instance independently, resulting in limited computational efficiency or solution quality [2]. Hence, there is growing interest in developing neural heuristics based on deep (reinforcement) learning to exploit the underlying patterns across VRP instances and then generalize to unseen ones [3].

Early neural heuristics for VRPs were primarily supervised [4, 5], whose performance depends heavily on the ground-truth labels, making them less desirable due to the expensive computation for

Zhiguang Cao

Wen Song ∗

obtaining optimal solutions. By contrast, deep reinforcement learning (DRL) relies on feedback from the environment rather than the ground-truth optimal solution as a reward, allowing neural heuristics to perform favorably in solving VRPs [6]. Typically, DRL can be used to train networks for selecting the next client (or node) to visit in VRPs [7, 8, 9, 10], which is more capable of generalizing to different problem sizes in comparison with the supervised counterparts [11, 12].

While DRL-based neural heuristics have achieved a series of success for VRPs, most of them perform reasonably well only when the underlying deep models are trained and tested on the independent and identically distributed (i.i.d.) instances with respect to the client locations, e.g., the ones generated using the uniform distribution. In the presence of distribution shifts, they struggle to deliver desirable performance, which greatly hinders the applications of neural heuristics into practice. Especially in real-world VRP instances, the client locations may vary dramatically according to weekday or weekend, sunny or rainy weather, etc, leading to various or even unknown distributions. Therefore, how to strengthen the cross-distribution generalization capability is of great importance to the neural heuristics for VRPs. Although a number of preliminary studies have been conducted to investigate this issue, including the distributionally robust optimization (DRO, [13]), curriculum-learning-based hardness-adaptive generator (HAC, [14]), and adaptive multi-distribution knowledge distillation (AMDKD, [15]), they are still far from optimal due to their inferior performance or less practical settings. Hence, it remains challenging to comprehensively and effectively expand the limited generalization capability of the neural heuristics against the distribution shifts for VRPs.

Motivated by the recognition of the limited capability of an inflexible model to learn from multiple distributions [16], and inspired by the fact that an ensemble of varied models can leverage their respective strengths to collaboratively handle different distributions [17, 18], we propose an endto-end ensemble-based deep reinforcement learning (EL-DRL) to solve VRPs in the presence of distribution shifts, which ensures diverse and complementary learning across different sub-models (or sub-policies), thereby fostering the cross-distribution generalization capability. Specifically, we first extend the REINFORCE algorithm [19] and policy gradient [20] in DRL to the ensemble setting by adaptively training a set of sub-policies on instances of mixed distributions to collaboratively solve a VRP instance. Then, to encourage diversity among the sub-policies, we harness Bootstrap with random initialization to enforce that each sub-policy is updated according to different loss signals. Additionally, we also exploit different regularization terms to prevent the collapse of sub-policies by explicitly pursuing inequality among them.

Accordingly, our contributions are outlined as follows. (i) We empirically demonstrate that a simple ensemble of similar sub-policies leads to limited performance in solving VRPs with distribution shifts. (ii) We propose an ensemble-based deep reinforcement learning (EL-DRL), and its ensemble-based policy gradient allows multiple sub-policies to learn to collaboratively leverage their strengths on respective distributions. We foster diversity by assigning different loss signals to sub-policies and applying regularization based on inequality metrics. (iii) Our EL-DRL adaptively trains sub-policies on instances of various distributions and is flexible without manually specifying instance class or group [13, 15] to train each sub-policy. (iv) We deploy POMO [9] as the sub-model in EL-DRL for solving the traveling salesman problem (TSP) and capacitated vehicle routing problem (CVRP) on both synthetic and benchmark instances of various distributions. The results verified the superior cross-distribution generalization of our EL-DRL.

2 Related Works

2.1 Deep Learning for Vehicle Routing Problems

In recent years, there has been extensive exploration of the use of deep (reinforcement) learning to solve vehicle routing problems. The Pointer Network (PtrNet) [4], originated from the encoderdecoder-structured sequence-to-sequence network in natural language processing (NLP), was employed to solve VRPs using supervised learning. To curtail the computational expense of obtaining ground-truth labels for VRPs, a follow-up study [21] trains the deep architecture using reinforcement learning. Performance was further improved by introducing a self-attention encoder [22] and an attention decoder, leading to the prominent Attention Model (AM). The POMO [9] was constructed upon the Attention Model, which augmented input instances and started inference from multiple cities. Three distinct strategies were presented for integrating active search into POMO in the inference phase [23], resulting in significantly improved performance. Graph neural networks (GNNs) and their

variants, such as graph convolutional networks (GCNs) and graph attention networks (GATs), have been applied to the graphs of VRP instances [24, 25, 26, 27, 5]. A concurrent line of improvement methods [28, 29, 30, 31, 10] focuses on refining an initial solution through iterative local rewriting. Neural methods based on deep learning have shown impressive results when training and testing instances share the same or similar distribution with respect to node locations, such as a uniform distribution [9, 32]. However, applying the pre-trained models of these methods to instances with non-uniform or complex distributions can result in significant gaps to optimal solutions [33, 14]. To enable the generalization beyond a single uniform distribution during training, group distributionally robust optimization (DRO, [13]) has been used to train neural models such as POMO and GCN on different distributions, which requires manual labeling of typical and atypical instances and specifying distribution groups. A curriculum-learning-based AM (HAC, [14]) has also been explored, which is trained on instances of varying hardness produced by an adaptive generator but limited to the mixedGaussian distribution. More recently, a study (AMDKD, [15]) distills the knowledge from various pre-trained teacher models, which manually assigns specific distribution to train each teacher model. Multi-view graph contrastive learning (MVGCL, [34]) is also applied to incorporate generalizable representations in policy learning for better knowledge transferability across distributions.

2.2 Deep Ensemble Learning

Deep ensemble learning models combine the strengths of both deep learning and ensemble learning to achieve superior generalization performance [35, 36]. Ensemble learning is recognized as a promising solution for adaptation to various tasks [37, 35, 36], due to the unique advantage that respective sub-models hold in solving different tasks. A few works have (implicitly) borrowed ideas from ensemble learning to solve VRPs [38, 29]. For example, MDAM [38] leverages the AM equipped with multiple decoders to solve VRPs, and each decoder corresponds to a sub-model in ensemble learning. However, due to the demanding computation, no mechanism is in place to ensure the diversity of constructed solutions beyond the first node. L2I [29] requires manually assigning modification rules to train different policies separately, which also takes prohibitively long runtime to search for better solutions iteratively. In summary, these works focus more on combining multiple models rather than promoting the automatic acquisition of diverse knowledge. On the other hand, diversity is crucial in ensemble learning for improving prediction performance and robustness [39, 40]. Ensemble diversity measures, algorithms for creating diverse ensembles, and ensemble consensus have been investigated to produce high-accuracy ensemble outputs [41, 42]. Our work differs from existing DRL-based methods for VRPs in that we explicitly consider and enforce diversity among sub-models, allowing them to capture different patterns from various distributions without much manual interference.

3 Methodology

In this section, we present how our ensemble learning paradigm trains a set of diverse DRL-based sub-policies, i.e., EL-DRL (see Figure 1), to solve VRP instances of various distributions effectively.

3.1 Preliminary

The traveling salesman problem (TSP) and the capacitated vehicle routing problem (CVRP) are the two most classic VRP variants. The TSP involves finding the optimal route to visit a set of N cities (or nodes) { v 1 , v 2 , . . . , v N } , with the objective to minimize the total distance traveled. The CVRP considers a fleet of (homogeneous) vehicles with limited capacity, where the vehicle must only visit a client once and also satisfy its demand. The objective is to minimize the total distance traveled by all vehicles while complying with the capacity constraint.

Our EL-DRL originates from the conjecture that high similarity in parameters of the neural network in the ensemble-based learning paradigm correlates to poor performance, as demonstrated in Section 4.4. Recent studies on ensembles with reinforcement learning [43, 44, 45] have primarily focused on utilizing deep Q-learning [46] to play games with a limited number of actions determined by the game controller. However, estimating Q-values for VRPs with hundreds of potential actions (i.e., the next node to visit) is challenging [26, 47], and the performance of these methods is often inferior to that of policy gradient [48]. We bridge this gap by designing the ensemble-based DRL (EL-DRL) to solve VRPs with policy gradient. While we build EL-DRL on top of the policy optimization with multiple optima (POMO) [9], our method is generic and can also be applied to other neural heuristics.

For clarity, we begin by explaining the process of using a single policy to solve a TSP instance through reinforcement learning. In specific, the policy gradient aims to learn a policy π θ represented by a neural network and parameterized by θ , which constructs a solution for the input instance s . The (partial) solution τ t is defined by the sequence of selected actions a 1 , a 2 , . . . , a N . Typically, the neural network employs an encoder to embed the input features (e.g. coordinates of nodes) and a decoder to sample the next action a [8, 10]. To solve TSP, the decoder autoregressively outputs the probability distribution π θ ( · | s, τ t ) for selecting a valid node (action) to visit at each step t ∈ { 1 , . . . , N } such that π θ ( τ ) = ∏ t π θ ( a t +1 | s, τ t ) . Once a complete solution τ has been constructed, the negative length of the trajectory can be computed and used as the reward R ( τ ) .

3.2 Ensemble-based Policy Gradient

Sub-Policy

1

Sub-Policy

2

Sub-Policy

4

Sub-Policy

3

Reward

Signal

Training

Instances

Mask

Ensemble Models

Sub-Policy

1

Sub-Policy

2

Sub-Policy

4

Sub-Policy

3

Sub-Policy

1

Sub-Policy

2

Sub-Policy

4

Sub-Policy

3

Sub-Policy

1

Sub-Policy

2

Sub-Policy

4

Sub-Policy

3

Unseen

Instance 1

Ensemble Learning

Ensemble Inference on various distributions

Mask

Mask

Mask

Inequality

Measurement

Solving VRP on Distribution A

Solving VRP on Distribution C

Unseen

Instance 2

Unseen

Instance 3

Solution

Solution

Solution

Maximize

Diversity

Solving VRP on Distribution B

Next

Node

1

一

Our architecture employs a shared attention encoder from AM [8], followed by M respective attention decoders, as illustrated in Figure 1. Each sub-policy, denoted by π m θ with parameters θ m , corresponds to a unique decoder plus an encoder that shares parameters with other sub-policies. In this way, the overall representation of an instance is only calculated once through the (computationally intensive) encoder. The subsequent lightweight decoders reuse this representation and collaboratively select nodes autoregressively, reducing the computational overhead for the ensemble training.

In our EL-DRL, the action a t at each step is determined by gathering output from all sub-policies, i.e, π θ ( a t +1 | s, τ t ) = 1 M ∑ M 1 π m θ ( a t +1 | s, τ t ) . This is inspired by the real-world practice where open-source communities combine the strengths of different individuals to achieve various common goals [49]. Similarly, our ensemble policies leverage the strengths of individual sub-models, e.g., some of them may perform well on Uniform distribution, while others excel on Gaussian distribution. Consequently, EL-DRL is not dominated by any individual sub-policy. Instead, it averages the probabilities produced by all sub-policies to determine an action at each step.

Using complex RL algorithms may not guarantee significant improvement. A better practice is to tailor learning algorithms for routing problems. We train the sub-policy with the REINFORCE algorithm [19] based on the POMO model[9], which samples a set of solution trajectories { τ 1 , τ 2 , . . . , τ N } that start from each of all nodes ( i ∈ { 1 , 2 , . . . , N } ), and gather each return R ( τ i ) (i.e., the negative of route length). The goal is to maximize the reinforcement learning objective L = E τ 1 ∼ π θ R ( τ 1 ) E τ 2 ∼ π θ R ( τ 2 ) . . . E τ N ∼ π θ R ( τ N ) by updating θ to increase the likelihood of sampling desirable trajectories in the future. We extend this objective to multiple sub-policies, which maximizes the expected return L RL with gradients computed as below,

where b ( s ) is a baseline to reduce the gradient variance, π m θ ( τ i m | s ) means the probability produced by the m -th sub-policy for the solution τ i m given the instance s . Our sub-policies synergize to generate a single trajectory for each starting node i , thus we have τ i m = τ i and R ( τ i m ) = R ( τ i ) for all m .

While it is feasible to train all the sub-policies together with Eq. (1), directly applying it to them may lead to the identical convergence of parameters, because all sub-policies receive the same reward which is used to compute the reinforcement loss signal. To alleviate this issue, we harness the Bootstrap with random initialization [50] for training the ensemble of sub-policies, which fosters diversity across the sub-policies via two mechanisms. Firstly, we initialize all decoders with distinct random parameters, inducing an initial diversity in the ensemble. Secondly, we leverage distinct reinforcement loss signals to train each sub-policy. In particular, for a sub-policy π m θ , we generate binary masks w i m from a Bernoulli distribution with parameter β ∈ (0 , 1] and then apply them to the rewards. Integrating those designs into Eq. (1), we arrive at:

When updating the decoder parameters of sub-policies, we also apply the bootstrap mask to each objective function, similar to the above one in Eq. (2). Additionally, we utilize a shared baseline b ( s ) averaged from all sampled trajectories:

The probabilities of producing the actions of τ i are reinforced through other N -1 unique trajectories. This shared baseline contributes to reducing gradient variance, and also potentially helps restrain the identical convergence with respect to the sub-policies [9].

3.3 Diversity Enhancement via Regularization

To further promote the diversity of the ensemble of sub-policies, we exploit the regularization to encourage explicit diversity across the parameters θ 1 , . . . , θ M of sub-policies during training. In order to prevent sub-policies from converging to the same parameters or simply duplicating each other, we need to push them away from each other in the parameter space.

This motivates us to measure and pursue the inequality between sub-policies. Therefore, we utilize a statistical measurement of inequality within a population (ensembles) in economics, i.e., the Theil index [51]. From the perspective of information theory, it is also a measurement of redundancy, which indicates the difference between the maximum possible entropy of the data and the observed entropy [42]. With the Theil index, we could measure whether our ensembles have learned their respective knowledge instead of imitating each other. Concretely, the first regularization term based on the Theil index in our loss function is expressed as L T = 1 M ∑ M m =1 x m µ ln x m µ , where x m is the ℓ 2 -norm of θ m and µ is the mean of all ℓ 2 -norms.

Furthermore, since we rely on an ensemble of sub-policies to make the decision, we could also explicitly exert a penalty that discourages all the individual sub-policies from converging toward a common value. This inspires us to utilize the second regularization term from the principles of consensus optimization [52], where multiple models are independently optimized with their own task-specific parameters. Formally, this regularization term is defined as L P = ∑ M m =1 ∥ ¯ θ -θ m ∥ 2 , where ¯ θ is the mean of the parameters for all sub-policies.

Notice that we intend to minimize the objective function for training our EL-DRL, thereby we integrate the regularization terms with negative signs to maximize the ensemble diversity (inequality). The whole loss function, inclusive of regularization terms with their coefficients ( α 1 and α 2 ) for our EL-DRL is expressed as:

At last, the parameters of policies are updated using gradient ascent based on Eq. (4). In the inference phase, EL-DRL collaboratively and greedily produces multiple trajectories by augmenting each input instance with rotations [9] and starting the trajectory from each of all nodes. The final solution is specified as the best one among all the sampled trajectories (solutions).

4 Experiments

We propose EL-DRL to train multiple diverse policies for enhancing cross-distribution generalization performance. To evaluate the robustness of our method in handling distribution shifts, EL-DRL and other baselines are tested on synthetic TSP and CVRP instances from various distributions, together with benchmark instances from TSPLib [53] and CVRPLib [54]. Additionally, we perform ablation experiments to demonstrate the importance of diversity and the key designs of our EL-DRL.

Baselines. For TSP experiments, we evaluate against the Concorde exact solver [55] and representative end-to-end neural approaches: Attention Model (AM) [8], Policy Optimization with Multiple Optima (POMO) [9], Multi-Decoder Attention Model (MDAM)[38], Distributionally Robust Optimization with POMO (DROP) [13], Hardness-Adaptive Curriculum (HAC) [14], and Adaptive Multi-Distribution Knowledge Distillation (AMDKD) [15]. Of these methods, DROP, HAC and AMDKD are recent approaches tackling the cross-distribution generalization issue for solving VRPs. As HAC was initially intended only for TSP, we did not take it into account for our CVRP experiments.

Implementation. EL-DRL employs an architecture similar to AM[8] but with one encoder and four decoders as the backbone. Preliminary experiments showed that the use of four decoders allows for a good balance between computational overhead and performance. Each sub-policy was trained using the same hyperparameters as the original POMO [9]. We use Adam optimizer with a learning rate of 1e-4. The values of α 1 and α 2 are tested and set according to testing sets. All the experiments of neural baselines are conducted in parallel on NVIDIA GTX 2080Ti GPUs for both training and inference. The training time for EL-DRL varied with the problem size. Taking CVRP100 as an example, one epoch takes about 5 minutes for EL-DRL. We applied early stopping during training when the reduction in the gap was not significant.

The optimal solution for TSP is calculated using the Concorde solver. 2 For CVRP, as existing solvers are difficult to obtain optimal solutions within an acceptable time frame, we employ the strong heuristic HGS 3 [56] as the traditional baseline instead, which has reported better performance [57] than LKH3 [58]. Regarding neural baselines, we reuse their open-sourced code from GitHub. For AM-based baselines, we sample 1,280 solutions following [8]; for MDAM we set beam search size to 50; for POMO-based baselines, we adopt the greedy rollout with augments following [9], which produces multiple greedy trajectories by rotating each input instance to generate 8 × augmented instances with 8 different angles but the same optimal solution. For each generated instance, it starts the trajectory from each of the N nodes. In this way, POMO-based baselines sample 8 × N solutions in total for each input instance.

Dataset. To ensure the hardness and variety of training distributions and to guarantee that the training instances are not seen during testing, we generate 7.2 million training instances from mixed distributions. Data generation is done by first sampling a uniform instance and then non-repetitively applying three randomly mutation operators 4 of TSPGEN to the same instance. So a single training instance is of mixed distributions and does not belong to any single distributions in the testing. During the inference, unlike most existing neural heuristics that are only tested on a uniform distribution, we evaluated all methods on various distributions, including Explosion, Compression, Cluster, Expansion, and Rotation, to assess the generalization performance against distribution shift. These distributions are more diverse in terms of both visual and quantitative measures compared to the uniform distribution [59], making it fairly challenging for neural heuristics to generalize. We test baselines on 10,000 instances in total, with each of the five testing distributions containing 2,000 instances. Therefore, the training instances do not fall into any type of testing distribution and all the test instances are sampled from distribution not seen during training. HAC is trained on instances created by the built-in data generator as its design relies on this process. The training of DROP requires class labels of the input instance and we specify the class of the last applied mutation operation for it. AMDKD requires manually assigning a single distribution for training each teacher model, which is less applicable to our setting where each single training instance is of mixed distributions. Thus, we follow its original setting for the best performance, and directly utilize the pre-trained AMDKD to solve instances from TSPLib and CVRPLib (Table 5 ).

4.1 Cross-distribution Generalization on TSP

We evaluated the cross-distribution generalization of our EL-DRL on TSP50 and TSP100. Tables 1 and 2 gathered the averaged tour length and gaps of all baselines compared to the exact solver Concorde on unseen instances from five different distributions. Although all baselines were trained on mixed distributions, they did not generalize well to other specific distributions (especially for AM), resulting in drastically deteriorated performance (about 2.5% to 4% gaps) compared to the i.i.d testing results they reported in the original paper. For example, POMO reports only 0.03% and 0.14% gaps compared to LKH3 [58] on TSP50 and TSP100, respectively, when trained and tested on a uniform distribution in [9]. However, its gaps increase up to 2.69% in our cross-distribution experiments. This phenomenon implies that these neural baselines are relatively incapable of learning from the underlying common patterns on complex training distributions. In contrast, our EL-DRL significantly improved generalization performance on the five respective testing distributions, achieving the smallest gap among neural heuristic baselines. For example, our EL-DRL reduced the gaps by 2.34% (2.07% vs. 4.32%) compared to AM and achieved a 0.55% (2.07% vs 2.62%) gap reduction on the TSP100 with Cluster distribution compared to the best neural baseline POMO. Notice that while multiple decoders in MDAM slightly improved its performance compared to AM, it failed to generalize to various distributions with gaps exceeding 3.2% on TSP100. This indicates that simply composing multiple models (even with KL divergence) does not necessarily result in good generalization performance. Meanwhile, EL-DRL also consistently beats the recent generalizationspecialized DROP and HAC in terms of both gaps and time efficiency. Especially, DROP exhibits unstable generalization performance and HAC does not generalize well on TSP100 (4.21-5.32% gaps), possibly due to special data processing requirements for training, which hinder their application to our scenario. In short, our EL-DRL outperforms these two state-of-the-art approaches in generalizing to cross-distribution.

4.2 Cross-distribution Generalization on CVRP

Pertaining to CVRP, we present the averaged tour lengths and gaps to the best solutions (obtained from the HGS with a 30s runtime), along with the average time to inference on unseen instances of CVRP50 and CVRP100 from five distributions in Tables 3 and 4, respectively. Overall, the HGS solver performs the best regarding tour length, as this heuristic is highly specialized for CVRP. Notably, we observe greater advantages of our method against neural baselines on this harder problem than on TSPs. Our EL-DRL achieves superior results on unseen instances of various distributions, with significantly small gaps (0.69-0.87%) on CVRP50, and especially outperforms POMO by around 0.7% on Compression distribution. Table 4 shows that for neural baselines, generalizing to various distributions on CVRP100 is also challenging, with considerable gaps ranging from 2.36% to 3.64%. In contrast, our method consistently delivers high-quality solutions with shorter lengths and is less sensitive to distribution shifts, which attains gaps of close to 2% for cross-distribution generalization on CVRP100. This superiority verifies the effectiveness of our idea of leveraging the strength of handling different distributions through cooperation between sub-policies. Regarding efficiency, all neural baselines can solve an instance in less than one second. Owing to our design of lightweight ensemble architecture and parallel inference, EL-DRL does not incur much inference time than the counterparts using a single model while yielding better performance and even exhibiting faster inference than AM and MDAM.

4.3 Real World benchmarks

We continue to evaluate our EL-DRL by taking public benchmark datasets TSPLib [53] and CVRPLib [54] as the testbed, to prove our approach is also efficient in solving more realistic and varied distributions. TSPLib is a library of TSP instances from various sources and of various types, making it desirable yet challenging for assessing generalization. For TSPLib, results are collected from 25 instances which problem sizes ranging from 50 to 300. For CVRPLib, the dataset is XML100, which

contains 10,000 CVRP100 instances from heterogeneous distributions. We normalize the coordinates in each instance to a range of [0, 1] for all neural methods, which also include the neural baseline AMDKD [15].

The upper half of Table 5 illustrates the averaged results on TSPLib instances. These results demonstrate that EL-DRL generalizes well to realistic distributions and varied sizes, in a small gap (4.85%), and it is much lower than the gaps of neural heuristics (i.e., around 5%-17%). In particular, the advantage of our EL-DRL over MDAM implies learning similar ensembles may hold back the generalization performance. Compared to the poor results of HAC, which learns only from Gaussian distribution, we can conclude that training the models on various heterogeneous distributions could be more beneficial. In comparison to the most recent generalization-specialized AMDKD, our EL-DRL brings a reduction in the gap by 0.51%. The lower half of Table 5 shows the averaged lengths and gaps on (harder) CVRPLib instances. Our EL-DRL again outperforms the top two baselines, AMDKD and POMO, by margins of 0.20% and 0.40%, respectively. The results on the realistic benchmarks underscore the remarkable capability of our EL-DRL in solving miscellaneous instances that were not present during the training phase, highlighting its robustness in coping with distribution shifts.

4.4 Ablation Studies

We also carry out ablation studies to demonstrate the effectiveness of key designs in our EL-DRL, by generating 10,000 CVRP100 instances as an exemplary case and presenting the results in Table 6. To investigate the disadvantage of high similarity between ensemble policies, we deploy a single POMO and an ensemble of four POMOs (Multi-POMO), respectively, to compare with our EL-DRL. Additionally, we ablate Bootstrap with Random Initialization (BRI), the Theil index regularization ( L T ) and parameter dissimilarity regularization ( L P ), then report the averaged results for the 10,000 instances of five distributions on CVRP100. First, the single POMO model relies on its own learning ability, increasing the susceptibility to over-fitting and leading to poor generalization performance. However, simply stacking POMO to multiple copies (Multi-POMO) cannot overcome this limitation and only brings slight improvement compared to the single POMO, which is primarily attributed to the larger amount of model parameters and in accordance with the finding in [60]. By contrast, our EL-DRL attains a 0.37% reduction of the overall gaps compared to the single POMO by collaboratively solving instances on various distributions. Excluding the Bootstrap with random initialization, sub-policies cannot receive distinct loss signals and converge to similar ones, and thus the gap of the variant without BRI increases significantly to 2.201% compared to that of EL-DRL (2.055%). Moreover, removing two regularization terms for maximizing diversity results in redundant parameters for sub-policies and hinders their expressive ability, as evidenced by the drop in generalization performance shown in the last two columns of Table 6. In summary, the designs of our EL-DRL are largely orthogonal and can be effectively integrated to maintain ensembles of diverse sub-models, which are capable of capturing different aspects of various distributions.

5 Conclusion and Limitation

This paper presents an ensemble-based DRL approach for solving vehicle routing problems, which fosters the capability of existing neural heuristics in generalizing against distribution shifts. By enforcing distinct reinforcement loss signals on each sub-policy and integrating regularization to ensure diversity across sub-policies, our EL-DRL learns to leverage the strengths of individuals in solving different instances with various distributions. In our experiments, we demonstrate that our proposed method delivers high-quality solutions for both synthetic instances of various distributions and heterogeneous instances from real-world benchmarks, confirming the potential of the ensemble idea in contributing to more generalizable and robust neural heuristics.

While it is not the focus of this paper, our method might be less effective in handling large instances. A major reason is that the scalability of our EL-DRL highly depends on the sub-model, where POMO (the used sub-model) performs inferior on large ones. Moreover, since our EL-DRL follows the ensemble paradigm, the training needs more computational resources than the single one. In the future, we would like to, 1) foster the scalability of our method by exploring decomposition schemes such as the divide-and-conquer [61], so as to allow it to solve large instances more efficiently; 2) enhance the computational efficiency of our method by introducing sparsity in ensemble learning [62], so as to allow it to train more sub-models while reducing the training overhead; 3) enable the interpretability [63] and further strengthen the collaboration among the sub-models of our method; and 4) extend our method to non-routing combinatorial optimization problems.

Acknowledgments and Disclosure of Funding

This research was supported by the National Natural Science Foundation of China under Grant 62102228, the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant, and the RIE2025 MTC IAF-PP funding (M23L4a0001).

References

KORTE Bernhard and JENS Vygen. Combinatorial optimization: Theory and algorithms. Springer, 5th Edition. , 2012.

Natalia Vesselinova, Rebecca Steinert, Daniel F Perez-Ramirez, and Magnus Boman. Learning combinatorial optimization on graphs: A survey with applications to networking. IEEE Access , 8:120388-120416, 2020.

Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021.

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems , pages 2692-2700, 2015.

Chaitanya K Joshi, Thomas Laurent, and Xavier Bresson. An efficient graph convolutional network technique for the travelling salesman problem. In INFORMS Annual Meeting , 2019.

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866 , 2017.

Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. In International Conference on Learning Representations , 2017.

Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In the 7th International Conference on Learning Representations , 2019.

Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. Pomo: Policy optimization with multiple optima for reinforcement learning. In Advances in Neural Information Processing Systems , 2020.

Minsu Kim, Jinkyoo Park, et al. Learning collaborative policies to solve np-hard routing problems. Advances in Neural Information Processing Systems , 34, 2021.

Chaitanya K Joshi, Quentin Cappart, Louis-Martin Rousseau, and Thomas Laurent. Learning the travelling salesperson problem requires rethinking generalization. Constraints , pages 1-29, 2022.

Jianan Zhou, Yaoxin Wu, Wen Song, Zhiguang Cao, and Jie Zhang. Towards omni-generalizable neural methods for vehicle routing problems. In International Conference on Machine Learning , 2023.

Yuan Jiang, Yaoxin Wu, Zhiguang Cao, and Jie Zhang. Learning to solve routing problems via distributionally robust optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9786-9794, 2022.

Zeyang Zhang, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning to solve travelling salesman problem with hardness-adaptive curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9136-9144, 2022.

Jieyi Bi, Yining Ma, Jiahai Wang, Zhiguang Cao, Jinbiao Chen, Yuan Sun, and Yeow Meng Chee. Learning generalizable models for vehicle routing problems via knowledge distillation. In Advances in Neural Information Processing Systems , 2022.

Alan Pham, Eunice Chan, Vikranth Srivatsa, Dhruba Ghosh, Yaoqing Yang, Yaodong Yu, Ruiqi Zhong, Joseph E Gonzalez, and Jacob Steinhardt. The effect of model size on worst-group generalization. arXiv preprint arXiv:2112.04094 , 2021.

Guilherme Perin, Łukasz Chmielewski, and Stjepan Picek. Strength in numbers: Improving generalization with ensembles in machine learning-based profiled side-channel analysis. IACR Transactions on Cryptographic Hardware and Embedded Systems , pages 337-364, 2020.

Jieting Wang, Yuhua Qian, Feijiang Li, Jiye Liang, and Qingfu Zhang. Generalization performance of pure accuracy and its application in selective ensemble learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.

Jan Peters. Policy gradient methods. Scholarpedia , 5(11):3698, 2010.

Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takác. Reinforcement learning for solving the vehicle routing problem. Advances in neural information processing systems , 31, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

André Hottung, Yeong-Dae Kwon, and Kevin Tierney. Efficient active search for combinatorial optimization problems. In International Conference on Learning Representations , 2022.

Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems , 30, 2017.

Michel Deudon, Pierre Cournut, Alexandre Lacoste, Yossiri Adulyasak, and Louis-Martin Rousseau. Learning heuristics for the tsp by policy gradient. In International conference on the integration of constraint programming, artificial intelligence, and operations research , pages 170-181. Springer, 2018.

Zhuwen Li, Qifeng Chen, and Vladlen Koltun. Combinatorial optimization with graph convolutional networks and guided tree search. Advances in neural information processing systems , 31, 2018.

Zhang-Hua Fu, Kai-Bin Qiu, and Hongyuan Zha. Generalize a small pre-trained model to arbitrarily large tsp instances. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7474-7482, 2021.

Xinyun Chen and Yuandong Tian. Learning to perform local rewriting for combinatorial optimization. Advances in Neural Information Processing Systems , 32, 2019.

Hao Lu, Xingwen Zhang, and Shuang Yang. A learning-based iterative method for solving vehicle routing problems. In International Conference on Learning Representations , 2019.

Paulo R d O Costa, Jason Rhuggenaath, Yingqian Zhang, and Alp Akcay. Learning 2-opt heuristics for the traveling salesman problem via deep reinforcement learning. In Asian Conference on Machine Learning , pages 465-480. PMLR, 2020.

Yaoxin Wu, Wen Song, Zhiguang Cao, Jie Zhang, and Andrew Lim. Learning improvement heuristics for solving routing problems.. IEEE transactions on neural networks and learning systems , 2021.

Wouter Kool, Herke van Hoof, Joaquim Gromicho, and Max Welling. Deep policy dynamic programming for vehicle routing problems. In Integration of Constraint Programming, Artificial Intelligence, and Operations Research: 19th International Conference , pages 190-213. Springer, 2022.

Simon Geisler, Johanna Sommer, Jan Schuchardt, Aleksandar Bojchevski, and Stephan Günnemann. Generalization of neural combinatorial solvers through the lens of adversarial robustness. In International Conference on Learning Representations , 2022.

Yuan Jiang, Zhiguang Cao, Yaoxin Wu, and Jie Zhang. Multi-view graph contrastive learning for solving vehicle routing problems. In Uncertainty in Artificial Intelligence , pages 984-994. PMLR, 2023.

Mudasir A Ganaie, Minghui Hu, AK Malik, M Tanveer, and PN Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence , 115:105151, 2022.

Yongquan Yang, Haijun Lv, and Ning Chen. A survey on ensemble learning under the era of deep learning. Artificial Intelligence Review , 56(6):5545-5589, 2023.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. IEEE Transactions on Image Processing , 30:8008-8018, 2021.

Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Multi-decoder attention model with embedding glimpse for solving vehicle routing problems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, 2021.

E Ke Tang, Ponnuthurai N Suganthan, and Xin Yao. An analysis of diversity measures. Machine learning , 65:247-271, 2006.

Ammar Mohammed and Rania Kora. A comprehensive review on ensemble deep learning: Opportunities and challenges. Journal of King Saud University-Computer and Information Sciences , 2023.

Ling Liu, Wenqi Wei, Ka-Ho Chow, Margaret Loper, Emre Gursoy, Stacey Truex, and Yanzhao Wu. Deep neural network ensembles against deception: Ensemble diversity, accuracy and robustness. In 2019 IEEE 16th international conference on mobile ad hoc and sensor systems (MASS) , pages 274-282. IEEE, 2019.

Hassam Sheikh, Mariano Phielipp, and Ladislau Boloni. Maximizing ensemble diversity in deep reinforcement learning. In International Conference on Learning Representations , 2022.

Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502 , 2017.

Oren Peer, Chen Tessler, Nadav Merlis, and Ron Meir. Ensemble bootstrapping for q-learning. In International Conference on Machine Learning , pages 8454-8463. PMLR, 2021.

Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, and Roy Fox. Reducing variance in temporal-difference value estimation via ensemble of deep networks. In International Conference on Machine Learning , pages 13285-13301. PMLR, 2022.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Thomas Barrett, William Clements, Jakob Foerster, and Alex Lvovsky. Exploratory combinatorial optimization with reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3243-3250, 2020.

Bingjie Li, Guohua Wu, Yongming He, Mingfeng Fan, and Witold Pedrycz. An overview and experimental study of learning-based optimization algorithms for the vehicle routing problem. IEEE/CAA Journal of Automatica Sinica , 9(7):1115-1138, 2022.

Siobhán O'mahony and Fabrizio Ferraro. The emergence of governance in an open source community. Academy of Management Journal , 50(5):1079-1106, 2007.

Bradley Efron. The jackknife, the bootstrap and other resampling plans . SIAM, 1982.

J Johnston. H. theil. economics and information theory, 1969.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122, 2011.

Gerhard Reinelt. Tsplib-a traveling salesman problem library. ORSA journal on computing , 3(4):376-384, 1991.

Eduardo Queiroga, Ruslan Sadykov, Eduardo Uchoa, and Thibaut Vidal. 10,000 optimal cvrp solutions for testing machine learning based heuristics. In AAAI-22 Workshop on Machine Learning for Operations Research (ML4OR) , 2021.

David Applegate, Robert Bixby, Vasek Chvatal, and William Cook. Concorde tsp solver. https://www.math.uwaterloo.ca/tsp/concorde.html , 2006.

Thibaut Vidal. Hybrid genetic search for the cvrp: Open-source implementation and swap neighborhood. Computers & Operations Research , 140:105643, 2022.

André Hottung and Kevin Tierney. Neural large neighborhood search for the capacitated vehicle routing problem. In Proceedings of the 24th European Conference on Artificial Intelligence (ECAI) , 2020.

Keld Helsgaun. An extension of the lin-kernighan-helsgaun tsp solver for constrained traveling salesman and vehicle routing problems. Roskilde: Roskilde University , pages 24-50, 2017.

Jakob Bossek, Pascal Kerschke, Aneta Neumann, Markus Wagner, Frank Neumann, and Heike Trautmann. Evolving diverse tsp instances by means of novel and creative mutation operators. In Proceedings of the 15th ACM/SIGEVO Conference on Foundations of Genetic Algorithms , pages 58-71, 2019.

G Brown. Diversity in neural network ensembles,[ph. d. thesis]. University of Birmingham , 2004.

Sirui Li, Zhongxia Yan, and Cathy Wu. Learning to delegate for large-scale vehicle routing. Advances in Neural Information Processing Systems , 34, 2021.

Li Zhang and Wei-Da Zhou. Sparse ensembles using weighted combination methods based on linear programming. Pattern Recognition , 44(1):97-106, 2011.

Chia-Hsiu Chen, Kenichi Tanaka, Masaaki Kotera, and Kimito Funatsu. Comparison and improvement of the predictability and interpretability with ensemble learning models in qspr applications. Journal of cheminformatics , 12:1-16, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggests a novel method to mitigate the distributional shift of combinatorial optimization problems. The idea is to bootstrap and aggregate multiple diverse policies which are similar to a classical ensemble method called bagging. The idea is simple and effective having promising empirical performances. I clearly vote to accept this paper.\n\nSTRENGTHS:\nThey targeted one of the significant research topics of combinatorial optimization. The idea is simple and straightforward. The empirical results seem to be effective.\n\nWEAKNESSES:\nOne of the other major topics of the distributional shift of combinatorial optimization is scale shift (e.g., trained on $N=50$ and tested on $N=500). This paper did not discuss those topics.\n\nQUESTIONS:\n1.  I'm curious about the robustness of this algorithm on large-scale problems. \n\n2.  There were several pieces of literature to mitigate distributional shift; for example:\n- Sym-NCO (NeurIPS'22) [1]: leveraged symmetricity for better generalization\n- ROCO (ICLR'23) [2]: robustness training of neural combinatorial optimization solver \n- Meta-SAGE (ICML'23) [3]: leveraged meta-learning for better adaptation\nIn my humble opinion, this paper should include the recent rapid progress of related research more carefully. \n\n3. Can this method be applied to non-routing problems?\n\n[1] Kim, Minsu, Junyoung Park, and Jinkyoo Park. \"Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization.\" Advances in Neural Information Processing Systems.\n\n[2] Lu, Han, et al. \"ROCO: A General Framework for Evaluating Robustness of Combinatorial Optimization Solvers on Graphs.\" The Eleventh International Conference on Learning Representations. 2023.\n\n[3] Son, Jiwoo, et al. \"Meta-SAGE: Scale Meta-Learning Scheduled Adaptation with Guided Exploration for Mitigating Scale Shift on Combinatorial Optimization.\" arXiv preprint arXiv:2306.02688 (2023).\n\nLIMITATIONS:\nThey already stated their limitation regarding scalability.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigated the classical vehicle routing problems and proposed an ensemble-based deep reinforcement learning algorithm. Moreover, two regularization terms (Theil index and L2-dist) were designed to improve the generalization ability of learned policies. The motivation is convincing and the proposed methodology is easy to understand.\n\nSTRENGTHS:\n1、The classical vehicle routing problem is important to explore and the proposed ensemble-based deep reinforcement learning algorithm has been proven to handle such a critical problem;\n2、The motivation is convincing and the paper writing is easy to understand.\n\nWEAKNESSES:\nThe main concern of mine mainly focuses on the limited discussion and comparison between the existing/previous works and this work:\n(1): Since the authors break down the optimization problem into sub-policies learning, why not just use multi-agent reinforcement learning to solve the problem? I didn't quite understand the necessity of ensemble learning. What is the relationship between ensemble learning and multi-agent policy in RL?\n\n(2): The EL-DRL is built on the top of REINFORCE algorithm that is very old. Can EL-DRL use better baselines like PPO or PPG? Or this setting is indispensable?\n\n(3): Some existing work in RL proposes to use data augmentation (e.g., observation augmentation and intrinsic reward) to improve generalization ability, which is very efficient and cheap [1][2]. Can the authors compare EL-DRL with these approaches?\n\nQUESTIONS:\nQ1: Can the authors explain the relevance between their work and \"MAXIMIZING ENSEMBLE DIVERSITY IN DEEP REINFORCEMENT LEARNING\" published at ICLR2022?\n\nQ2: Is there any comparison between EL-DRL and traditional algorithms like genetic algorithms?\n\nQ3: More detailed literature review of DRL-based methods for VRP is needed.\n\n[1] Reinforcement Learning with Augmented Data\n[2] State Entropy Maximization with Random Encoders for Efficient Exploration\n\nLIMITATIONS:\nPlease see the weakness and question section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, an ensemble-based deep reinforcement learning algorithm is proposed for vehicle routing problems. The goal of the method introduced in this work is to improve handling of distribution shifts at test time. The approach, EL-DRL, combines multiple ideas: i) and ensemble of sub-policies, ii) distinct losses for each sub-policy, iii) explicit regularization, and iv) distinct random sub-policy weight initialization. The main contribution of the work is a demonstration that sub-policy diversity improves out-of-distribution generalization for ensemble DRL on the VRP. The EL-DRL method is evaluated on a range of TSP and CVRP instances sampled from multiple distributions. It is compared against a significant number of advanced neural baselines and demonstrates improved performance.\n\nSTRENGTHS:\n#### Originality\n\n- The main contribution of this paper appears to be a demonstration that diversity-enhanced Deep RL [1] can be applied to VRPs. I think this paper can be useful to those working on solving VRPs with Deep RL.\n\n#### Quality\n\n- The experiments are conducted on a wide range of benchmarks and out-of-distribution generalization is evaluation on a large number of synthetic and real VRP distributions.\n- The authors did a good job placing this work in the context of the relevant literature on deep reinforcement learning for VRPs.\n\n#### Clarity\n\n- The paper is written well and easy to follow.\n\n#### Significance\n\n- The proposed approach EL-DRL, appears to achieve state-of-the-art results on a large number of experiments. However, there are some concerns about the interpretation of the results, detailed below.\n\n#### References\n\n[1] Sheikh, Hassam, Mariano Phielipp, and Ladislau Boloni. \"Maximizing ensemble diversity in deep reinforcement learning.\" In International Conference on Learning Representations. 2021.\n\nWEAKNESSES:\n#### Originality\n\n- It is not clear to me that the significance of this work extends beyond the applied research area of Deep RL for VRPs.\n- I would assess the overall theoretical and technical contributions to the machine learning community to be incremental.\n\n#### Soundness\n\n- I struggled to interpret the quantitative results with any amount of confidence. While EL-DRL appears to \"win\" against the other methods (lots of bold), it was not clear to me whether the improvements due to the diversity-enhancing mechanisms are significant or marginal. \n- I would highly recommend using rliable (https://github.com/google-research/rliable) to calculate boostrapped confidence intervals for summary statistics across multiple (5+, ideally) random seeds and to show performance profiles. The Probability of Improvement metric $P( X > Y)$  implemented in `rliable` is a great way to compare these methods against each other. \n- On the majority of the tables in the results, POMO already beats the other baselines, including other advanced strategies for dealing with out-of-distribution generalization. This makes it difficult to interpret comparisons between EL-DRL and the other advanced strategies, as the strength of POMO alone appears to already explain the majority of the performance gains. \n- For many settings, EL-DRL only achieves a slight improvement over POMO. For example, in Table 5 Real-World TSPLib, POMO gets 5.16% gap and EL-DRL gets 4.79% (a length difference of only 7.44 vs. 7.39).\n- Moreover, I believe a random baseline and the Multi-POMO baseline should be included in each Table. A random baseline can help place into context what the difference in gap percentage means. Likewise, Multi-POMO will help provide context, as it has a similar number of parameters to EL-DRL. In Table 6, the Multi-POMO approach does improve over POMO (2.311% vs. 2.423%). \n\n#### A minor issue\n\n- L43-44: The authors write, \"Motivated by the recognition of the limited capability for a single model to learn from multiple distributions, [...], we propose...\" citing [1]. This statement is incorrect and should be revised. It is fairly clear by now that large pre-trained models are certainly capable of learning from multiple distributions. In support of this, the conclusions made in [1] are in fact that \"increasing pre-trained model size consistently improves performance on Waterbirds and MultiNLI. We advise practitioners to use larger pre-trained models when subgroup labels are unknown\".\n\n#### References\n\n[1] Pham, Alan, Eunice Chan, Vikranth Srivatsa, Dhruba Ghosh, Yaoqing Yang, Yaodong Yu, Ruiqi Zhong, Joseph E. Gonzalez, and Jacob Steinhardt. \"The effect of model size on worst-group generalization.\" arXiv preprint arXiv:2112.04094 (2021).\n\nQUESTIONS:\n- How sensitive is the model to hyperparameters---e.g., the learning rate, $\\alpha_1$ and $\\alpha_2$?\n- How does EL-DRL fare against random and Multi-POMO baselines on the other benchmarks beyond CVRP100? \n\n---\n\nUpdate after rebuttal: The authors have made an effort to respond in a satisfactory way to my concerns and questions. Under the assumption that the new experimental results (especially demonstrating the statistical significance of the improvement over baselines) and promised revisions make it into the final version, I have increased my score.\n\nLIMITATIONS:\nThe authors mention two key limitations: EL-DRL inherits limitations of the sub-model, and the increase in computation required to use ensembles of multiple models. They mention that POMO performs inferior on large instances, although I did not see where this is demonstrated in the experiments?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a deep reinforcement learning approach for vehicle routing problems under distribution shift that trains an ensemble of models. During solution construction, all ensemble models are evaluated at each decision step and the actions are selected based on the combined output probabilities. The ensemble is trained and evaluated on instance sets that contain instances sampled from different distributions. Ideally, the individual models have different strength and weaknesses (e.g., with some performing well on uniform instances, while others are better at Gaussian instances) which could result in an improved generalization performance. In order to improve diversity between the ensemble models, the authors propose 3 different components: 1) Bootstrap with random initialization: All models are initialized with different parameter values and during training each model only sees a subset of the training data. 2) Theil index regularization: Aims at increasing the inequality between sub-policies. 3)  Parameter dissimilarity regularization: Aims at increasing the difference between the model parameters. The results indicate that the proposed ensemble approach performs better than comparable approaches.\n\nSTRENGTHS:\n- The paper addresses an important problem (designing a neural combinatorial optimization approach that generalizes to instances from different distributions).\n- The results show that the method outperforms other methods in the considered generalization setting.\n- The authors perform some ablation experiments to evaluate the effectiveness of the proposed diversity-increasing components.\n\nWEAKNESSES:\n- Overall, I am not fully convinced that the concept of combining outputs of multiple models to create a single solution is ideal for the neural combinatorial optimization setting where we usually generate multiple solutions per instance. In the classical machine learning prediction setting, model outputs are combined because users are interested in a single prediction. In the considered neural combinatorial setting there is no direct need to combine model outputs, instead one solution could be generated per ensemble model.\n- While the authors performed some ablation experiments, I would have wished for even more experiments that provide insights into the method:\n     - Some experiments that directly evaluate how different the policies of the ensemble models are would provide interesting insights. For example, how often do the models disagree on the next actions (if argmax is used to select an action separately for each model).\n    - What is the impact of the weights for the two regularization terms (which are set to 0.5 and 0.5 in the experiments)? A parameter sweep that finds the sweet spot between too much and too little diversity would have been nice.\n    - What is the impact of the ensemble size on the performance?\n- The description of the proposed regularization terms and their theoretical motivation is difficult to understand (or lacking). I do not understand the Theil index regularization and I am skeptical that the parameter dissimilarity regularization meaningfully increases the difference between models. The experimental results do not fully convince me that they work as intended because it is unclear if the reported improvements are statistically significant. The authors should expand the description of the regularization terms, provide a more mathematically-grounded motivation, and more experiments.\n- The experimental setup is a bit unclear. More specifically, the generation of the training instances is not explained in detail and it is unclear from how many different distributions instances are sampled. Furthermore, it is unclear if all test instances are sampled from distribution not seen during training or if there is some overlap.\n- The runtime of the proposed approach is roughly two times slower than that of the POMO model. This makes a direct comparison of the cost difficult. What happens if the number of generated solutions for POMO is increased (e.g., doubled)?\n\nQUESTIONS:\nHow many solutions do you generate per test instance?\n\nLIMITATIONS:\nThe authors briefly discuss limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}