SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuan Jiang
School of Computer Science and Engineering Nanyang Technological University yuan005@e.ntu.edu.sg
Zhiguang Cao
School of Computing and Information Systems  
Singapore Management University  
zgcao@smu.edu.sg
Yaoxin Wu
Department of Industrial Engineering & Innovation Sciences  
Eindhoven University of Technology  
y.wu2@tue.nl
Wen Song*
Marine Science and Technology Shandong University wensong@email.sdu.edu.cn
Jie Zhang
School of Computer Science and Engineering Nanyang Technological University zhangj@ntu.edu.sg
While performing favourably on the independent and identically distributed (i.i.d.) instances, most of the existing neural methods for vehicle routing problems (VRPs) struggle to generalize in the presence of a distribution shift. To tackle this issue, we propose an ensemble-based deep reinforcement learning method for VRPs, which learns a group of diverse sub-policies to cope with various instance distributions. In particular, to prevent convergence of the parameters to the same one, we enforce diversity across sub-policies by leveraging Bootstrap with random initialization. Moreover, we also explicitly pursue inequality between sub-policies by exploiting regularization terms during training to further enhance diversity. Experimental results show that our method is able to outperform the state-of-the-art neural baselines on randomly generated instances of various distributions, and also generalizes favourably on the benchmark instances from TSPLib and CVRPLib, which confirmed the effectiveness of the whole method and the respective designs.
The vehicle routing problem (VRP) is a type of combinatorial optimization problems (COPs) with significant real-world applications, especially in the field of logistics [1]. The goal is to optimally route a fleet of vehicles, originating at a depot, to serve a set of clients. Given the NP-hard nature, heuristic methods are usually preferred to solve VRPs in practice. However, traditional heuristics always treat each VRP instance independently, resulting in limited computational efficiency or solution quality [2]. Hence, there is growing interest in developing neural heuristics based on deep (reinforcement) learning to exploit the underlying patterns across VRP instances and then generalize to unseen ones [3].
Early neural heuristics for VRPs were primarily supervised [4, 5], whose performance depends heavily on the ground-truth labels, making them less desirable due to the expensive computation for
obtaining optimal solutions. By contrast, deep reinforcement learning (DRL) relies on feedback from the environment rather than the ground-truth optimal solution as a reward, allowing neural heuristics to perform favorably in solving VRPs [6]. Typically, DRL can be used to train networks for selecting the next client (or node) to visit in VRPs [7, 8, 9, 10], which is more capable of generalizing to different problem sizes in comparison with the supervised counterparts [11, 12].
While DRL-based neural heuristics have achieved a series of success for VRPs, most of them perform reasonably well only when the underlying deep models are trained and tested on the independent and identically distributed (i.i.d.) instances with respect to the client locations, e.g., the ones generated using the uniform distribution. In the presence of distribution shifts, they struggle to deliver desirable performance, which greatly hinders the applications of neural heuristics into practice. Especially in real-world VRP instances, the client locations may vary dramatically according to weekday or weekend, sunny or rainy weather, etc, leading to various or even unknown distributions. Therefore, how to strengthen the cross-distribution generalization capability is of great importance to the neural heuristics for VRPs. Although a number of preliminary studies have been conducted to investigate this issue, including the distributionally robust optimization (DRO, [13]), curriculum-learning-based hardness-adaptive generator (HAC, [14]), and adaptive multi-distribution knowledge distillation (AMDKD, [15]), they are still far from optimal due to their inferior performance or less practical settings. Hence, it remains challenging to comprehensively and effectively expand the limited generalization capability of the neural heuristics against the distribution shifts for VRPs.
Motivated by the recognition of the limited capability of an inflexible model to learn from multiple distributions [16], and inspired by the fact that an ensemble of varied models can leverage their respective strengths to collaboratively handle different distributions [17, 18], we propose an end-to-end ensemble-based deep reinforcement learning (EL-DRL) to solve VRPs in the presence of distribution shifts, which ensures diverse and complementary learning across different sub-models (or sub-policies), thereby fostering the cross-distribution generalization capability. Specifically, we first extend the REINFORCE algorithm [19] and policy gradient [20] in DRL to the ensemble setting by adaptively training a set of sub-policies on instances of mixed distributions to collaboratively solve a VRP instance. Then, to encourage diversity among the sub-policies, we harness Bootstrap with random initialization to enforce that each sub-policy is updated according to different loss signals. Additionally, we also exploit different regularization terms to prevent the collapse of sub-policies by explicitly pursuing inequality among them.
Accordingly, our contributions are outlined as follows. (i) We empirically demonstrate that a simple ensemble of similar sub-policies leads to limited performance in solving VRPs with distribution shifts. (ii) We propose an ensemble-based deep reinforcement learning (EL-DRL), and its ensemble-based policy gradient allows multiple sub-policies to learn to collaboratively leverage their strengths on respective distributions. We foster diversity by assigning different loss signals to sub-policies and applying regularization based on inequality metrics. (iii) Our EL-DRL adaptively trains sub-policies on instances of various distributions and is flexible without manually specifying instance class or group [13, 15] to train each sub-policy. (iv) We deploy POMO [9] as the sub-model in EL-DRL for solving the traveling salesman problem (TSP) and capacitated vehicle routing problem (CVRP) on both synthetic and benchmark instances of various distributions. The results verified the superior cross-distribution generalization of our EL-DRL.
2.1 Deep Learning for Vehicle Routing Problems
In recent years, there has been extensive exploration of the use of deep (reinforcement) learning to solve vehicle routing problems. The Pointer Network (PtrNet) [4], originated from the encoder-decoder-structured sequence-to-sequence network in natural language processing (NLP), was employed to solve VRPs using supervised learning. To curtail the computational expense of obtaining ground-truth labels for VRPs, a follow-up study [21] trains the deep architecture using reinforcement learning. Performance was further improved by introducing a self-attention encoder [22] and an attention decoder, leading to the prominent Attention Model (AM). The POMO [9] was constructed upon the Attention Model, which augmented input instances and started inference from multiple cities. Three distinct strategies were presented for integrating active search into POMO in the inference phase [23], resulting in significantly improved performance. Graph neural networks (GNNs) and their
variants, such as graph convolutional networks (GCNs) and graph attention networks (GATs), have been applied to the graphs of VRP instances [24, 25, 26, 27, 5]. A concurrent line of improvement methods [28, 29, 30, 31, 10] focuses on refining an initial solution through iterative local rewriting. Neural methods based on deep learning have shown impressive results when training and testing instances share the same or similar distribution with respect to node locations, such as a uniform distribution [9, 32]. However, applying the pre-trained models of these methods to instances with non-uniform or complex distributions can result in significant gaps to optimal solutions [33, 14]. To enable the generalization beyond a single uniform distribution during training, group distributionally robust optimization (DRO, [13]) has been used to train neural models such as POMO and GCN on different distributions, which requires manual labeling of typical and atypical instances and specifying distribution groups. A curriculum-learning-based AM (HAC, [14]) has also been explored, which is trained on instances of varying hardness produced by an adaptive generator but limited to the mixed-Gaussian distribution. More recently, a study (AMDKD, [15]) distills the knowledge from various pre-trained teacher models, which manually assigns specific distribution to train each teacher model. Multi-view graph contrastive learning (MVGCL, [34]) is also applied to incorporate generalizable representations in policy learning for better knowledge transferability across distributions.
2.2 Deep Ensemble Learning
Deep ensemble learning models combine the strengths of both deep learning and ensemble learning to achieve superior generalization performance [35, 36]. Ensemble learning is recognized as a promising solution for adaptation to various tasks [37, 35, 36], due to the unique advantage that respective sub-models hold in solving different tasks. A few works have (implicitly) borrowed ideas from ensemble learning to solve VRPs [38, 29]. For example, MDAM [38] leverages the AM equipped with multiple decoders to solve VRPs, and each decoder corresponds to a sub-model in ensemble learning. However, due to the demanding computation, no mechanism is in place to ensure the diversity of constructed solutions beyond the first node. L2I [29] requires manually assigning modification rules to train different policies separately, which also takes prohibitively long runtime to search for better solutions iteratively. In summary, these works focus more on combining multiple models rather than promoting the automatic acquisition of diverse knowledge. On the other hand, diversity is crucial in ensemble learning for improving prediction performance and robustness [39, 40]. Ensemble diversity measures, algorithms for creating diverse ensembles, and ensemble consensus have been investigated to produce high-accuracy ensemble outputs [41, 42]. Our work differs from existing DRL-based methods for VRPs in that we explicitly consider and enforce diversity among sub-models, allowing them to capture different patterns from various distributions without much manual interference.
In this section, we present how our ensemble learning paradigm trains a set of diverse DRL-based sub-policies, i.e., EL-DRL (see Figure 1), to solve VRP instances of various distributions effectively.
The traveling salesman problem (TSP) and the capacitated vehicle routing problem (CVRP) are the two most classic VRP variants. The TSP involves finding the optimal route to visit a set of  $N$  cities (or nodes)  $\{v_{1}, v_{2}, \ldots, v_{N}\}$ , with the objective to minimize the total distance traveled. The CVRP considers a fleet of (homogeneous) vehicles with limited capacity, where the vehicle must only visit a client once and also satisfy its demand. The objective is to minimize the total distance traveled by all vehicles while complying with the capacity constraint.
Our EL-DRL originates from the conjecture that high similarity in parameters of the neural network in the ensemble-based learning paradigm correlates to poor performance, as demonstrated in Section 4.4. Recent studies on ensembles with reinforcement learning [43, 44, 45] have primarily focused on utilizing deep Q-learning [46] to play games with a limited number of actions determined by the game controller. However, estimating Q-values for VRPs with hundreds of potential actions (i.e., the next node to visit) is challenging [26, 47], and the performance of these methods is often inferior to that of policy gradient [48]. We bridge this gap by designing the ensemble-based DRL (EL-DRL) to solve VRPs with policy gradient. While we build EL-DRL on top of the policy optimization with multiple optima (POMO) [9], our method is generic and can also be applied to other neural heuristics.
For clarity, we begin by explaining the process of using a single policy to solve a TSP instance through reinforcement learning. In specific, the policy gradient aims to learn a policy  $\pi_{\theta}$  represented by a neural network and parameterized by  $\theta$ , which constructs a solution for the input instance  $s$ . The (partial) solution  $\tau_{t}$  is defined by the sequence of selected actions  $a_1, a_2, \ldots, a_N$ . Typically, the neural network employs an encoder to embed the input features (e.g. coordinates of nodes) and a decoder to sample the next action  $a$  [8, 10]. To solve TSP, the decoder autoregressively outputs the probability distribution  $\pi_{\theta}(\cdot | s, \tau_{t})$  for selecting a valid node (action) to visit at each step  $t \in \{1, \ldots, N\}$  such that  $\pi_{\theta}(\tau) = \prod_{t} \pi_{\theta}(a_{t+1} | s, \tau_{t})$ . Once a complete solution  $\tau$  has been constructed, the negative length of the trajectory can be computed and used as the reward  $R(\tau)$ .
3.2 Ensemble-based Policy Gradient
Figure 1: Illustration of Our EL-DRL. It trains sub-policies using respective masked reward (reinforcement loss) signals and inequality regularization to maximize the diversity. During inference, sub-policies synergize by leveraging their strengths on various distributions. Different levels of transparency indicate varied contributions of each sub-policy in deriving the solutions.
Our architecture employs a shared attention encoder from AM [8], followed by  $M$  respective attention decoders, as illustrated in Figure 1. Each sub-policy, denoted by  $\pi_{\theta}^{m}$  with parameters  $\theta_{m}$ , corresponds to a unique decoder plus an encoder that shares parameters with other sub-policies. In this way, the overall representation of an instance is only calculated once through the (computationally intensive) encoder. The subsequent lightweight decoders reuse this representation and collaboratively select nodes autoregressively, reducing the computational overhead for the ensemble training.
In our EL-DRL, the action  $a_{t}$  at each step is determined by gathering output from all sub-policies, i.e.,  $\pi_{\theta}(a_{t + 1}\mid s,\tau_t) = \frac{1}{M}\sum_1^M\pi_\theta^m (a_{t + 1}\mid s,\tau_t)$ . This is inspired by the real-world practice where open-source communities combine the strengths of different individuals to achieve various common goals [49]. Similarly, our ensemble policies leverage the strengths of individual sub-models, e.g., some of them may perform well on Uniform distribution, while others excel on Gaussian distribution. Consequently, EL-DRL is not dominated by any individual sub-policy. Instead, it averages the probabilities produced by all sub-policies to determine an action at each step.
Using complex RL algorithms may not guarantee significant improvement. A better practice is to tailor learning algorithms for routing problems. We train the sub-policy with the REINFORCE algorithm [19] based on the POMO model[9], which samples a set of solution trajectories  $\{\tau^1, \tau^2, \dots, \tau^N\}$  that start from each of all nodes ( $i \in \{1, 2, \dots, N\}$ ), and gather each return  $R(\tau^i)$  (i.e., the negative of route length). The goal is to maximize the reinforcement learning objective  $\mathcal{L} = \mathbb{E}_{\tau^{1} \sim \pi_{\theta}} R(\tau^{1}) \mathbb{E}_{\tau^{2} \sim \pi_{\theta}} R(\tau^{2}) \dots \mathbb{E}_{\tau^{N} \sim \pi_{\theta}} R(\tau^{N})$  by updating  $\theta$  to increase the likelihood of sampling desirable trajectories in the future. We extend this objective to multiple sub-policies, which maximizes the expected return  $\mathcal{L}_{RL}$  with gradients computed as below,
$$
\nabla \mathcal {L} (\theta \mid s) \approx \frac {1}{N M} \sum_ {i = 1} ^ {N} \sum_ {m = 1} ^ {M} \left(R \left(\tau_ {m} ^ {i}\right) - b (s)\right) \nabla \log \pi_ {\theta} ^ {m} \left(\tau_ {m} ^ {i} \mid s\right), \tag {1}
$$
where  $b(s)$  is a baseline to reduce the gradient variance,  $\pi_{\theta}^{m}\left(\tau_{m}^{i} \mid s\right)$  means the probability produced by the  $m$ -th sub-policy for the solution  $\tau_{m}^{i}$  given the instance  $s$ . Our sub-policies synergize to generate a single trajectory for each starting node  $i$ , thus we have  $\tau_{m}^{i} = \tau^{i}$  and  $R(\tau_{m}^{i}) = R(\tau^{i})$  for all  $m$ .
While it is feasible to train all the sub-policies together with Eq. (1), directly applying it to them may lead to the identical convergence of parameters, because all sub-policies receive the same reward which is used to compute the reinforcement loss signal. To alleviate this issue, we harness the Bootstrap with random initialization [50] for training the ensemble of sub-policies, which fosters diversity across the sub-policies via two mechanisms. Firstly, we initialize all decoders with distinct random parameters, inducing an initial diversity in the ensemble. Secondly, we leverage distinct reinforcement loss signals to train each sub-policy. In particular, for a sub-policy  $\pi_{\theta}^{m}$ , we generate binary masks  $w_{m}^{i}$  from a Bernoulli distribution with parameter  $\beta \in (0,1]$  and then apply them to the rewards. Integrating those designs into Eq. (1), we arrive at:
$$
\nabla \mathcal {L} _ {R L} (\theta \mid s) \approx \frac {1}{N M} \sum_ {i = 1} ^ {N} \sum_ {m = 1} ^ {M} w _ {m} ^ {i} \left(R \left(\tau^ {i}\right) - b (s)\right) \nabla \log \pi_ {\theta} ^ {m} \left(\tau^ {i} \mid s\right). \tag {2}
$$
When updating the decoder parameters of sub-policies, we also apply the bootstrap mask to each objective function, similar to the above one in Eq. (2). Additionally, we utilize a shared baseline  $b(s)$  averaged from all sampled trajectories:
$$
b (s) = \frac {1}{N M} \sum_ {i = 1} ^ {N} \sum_ {m = 1} ^ {M} R \left(\tau_ {m} ^ {i}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} R \left(\tau^ {i}\right). \tag {3}
$$
The probabilities of producing the actions of  $\tau^i$  are reinforced through other  $N - 1$  unique trajectories. This shared baseline contributes to reducing gradient variance, and also potentially helps restrain the identical convergence with respect to the sub-policies [9].
3.3 Diversity Enhancement via Regularization
To further promote the diversity of the ensemble of sub-policies, we exploit the regularization to encourage explicit diversity across the parameters  $\theta^1,\dots ,\theta^M$  of sub-policies during training. In order to prevent sub-policies from converging to the same parameters or simply duplicating each other, we need to push them away from each other in the parameter space.
This motivates us to measure and pursue the inequality between sub-policies. Therefore, we utilize a statistical measurement of inequality within a population (ensembles) in economics, i.e., the Theil index [51]. From the perspective of information theory, it is also a measurement of redundancy, which indicates the difference between the maximum possible entropy of the data and the observed entropy [42]. With the Theil index, we could measure whether our ensembles have learned their respective knowledge instead of imitating each other. Concretely, the first regularization term based on the Theil index in our loss function is expressed as  $\mathcal{L}_T = \frac{1}{M}\sum_{m = 1}^{M}\frac{x_m}{\mu}\ln \frac{x_m}{\mu}$ , where  $x_{m}$  is the  $\ell^2$ -norm of  $\theta_{m}$  and  $\mu$  is the mean of all  $\ell^2$ -norms.
Furthermore, since we rely on an ensemble of sub-policies to make the decision, we could also explicitly exert a penalty that discourages all the individual sub-policies from converging toward a common value. This inspires us to utilize the second regularization term from the principles of consensus optimization [52], where multiple models are independently optimized with their own task-specific parameters. Formally, this regularization term is defined as  $\mathcal{L}_P = \sum_{m=1}^{M} \|\bar{\theta} - \theta^m\|^2$ , where  $\bar{\theta}$  is the mean of the parameters for all sub-policies.
Notice that we intend to minimize the objective function for training our EL-DRL, thereby we integrate the regularization terms with negative signs to maximize the ensemble diversity (inequality). The whole loss function, inclusive of regularization terms with their coefficients  $(\alpha_{1}$  and  $\alpha_{2})$  for our EL-DRL is expressed as:
$$
\nabla \mathcal {L} \approx \mathcal {L} _ {R L} (\theta | s) - \alpha_ {1} \mathcal {L} _ {T} - \alpha_ {2} \mathcal {L} _ {P}. \tag {4}
$$
At last, the parameters of policies are updated using gradient ascent based on Eq. (4). In the inference phase, EL-DRL collaboratively and greedily produces multiple trajectories by augmenting each input instance with rotations [9] and starting the trajectory from each of all nodes. The final solution is specified as the best one among all the sampled trajectories (solutions).
We propose EL-DRL to train multiple diverse policies for enhancing cross-distribution generalization performance. To evaluate the robustness of our method in handling distribution shifts, EL-DRL and other baselines are tested on synthetic TSP and CVRP instances from various distributions, together with benchmark instances from TSPLib [53] and CVRPLib [54]. Additionally, we perform ablation experiments to demonstrate the importance of diversity and the key designs of our EL-DRL.
Baselines. For TSP experiments, we evaluate against the Concorde exact solver [55] and representative end-to-end neural approaches: Attention Model (AM) [8], Policy Optimization with Multiple Optima (POMO) [9], Multi-Decoder Attention Model (MDAM)[38], Distributionally Robust Optimization with POMO (DROP) [13], Hardness-Adaptive Curriculum (HAC) [14], and Adaptive Multi-Distribution Knowledge Distillation (AMDKD) [15]. Of these methods, DROP, HAC and AMDKD are recent approaches tackling the cross-distribution generalization issue for solving VRPs. As HAC was initially intended only for TSP, we did not take it into account for our CVRP experiments.
Implementation. EL-DRL employs an architecture similar to AM[8] but with one encoder and four decoders as the backbone. Preliminary experiments showed that the use of four decoders allows for a good balance between computational overhead and performance. Each sub-policy was trained using the same hyperparameters as the original POMO [9]. We use Adam optimizer with a learning rate of 1e-4. The values of  $\alpha_{1}$  and  $\alpha_{2}$  are tested and set according to testing sets. All the experiments of neural baselines are conducted in parallel on NVIDIA GTX 2080Ti GPUs for both training and inference. The training time for EL-DRL varied with the problem size. Taking CVRP100 as an example, one epoch takes about 5 minutes for EL-DRL. We applied early stopping during training when the reduction in the gap was not significant.
The optimal solution for TSP is calculated using the Concorde solver. For CVRP, as existing solvers are difficult to obtain optimal solutions within an acceptable time frame, we employ the strong heuristic HGS [56] as the traditional baseline instead, which has reported better performance [57] than LKH3 [58]. Regarding neural baselines, we reuse their open-sourced code from GitHub. For AM-based baselines, we sample 1,280 solutions following [8]; for MDAM we set beam search size to 50; for POMO-based baselines, we adopt the greedy rollout with augments following [9], which produces multiple greedy trajectories by rotating each input instance to generate  $8 \times$  augmented instances with 8 different angles but the same optimal solution. For each generated instance, it starts the trajectory from each of the  $N$  nodes. In this way, POMO-based baselines sample  $8 \times N$  solutions in total for each input instance.