SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yining Ma
yiningma@u.nus.edu
Zhiguang Cao*
zgcao@smu.edu.sg
Yeow Meng Chee
ymchee@nus.edu.sg
In this paper, we present Neural k-Opt (NeuOpt), a novel learning-to-search (L2S) solver for routing problems. It learns to perform flexible k-opt exchanges based on a tailored action factorization method and a customized recurrent dual-stream decoder. As a pioneering work to circumvent the pure feasibility masking scheme and enable the autonomous exploration of both feasible and infeasible regions, we then propose the Guided Infeasible Region Exploration (GIRE) scheme, which supplements the NeuOpt policy network with feasibility-related features and leverages reward shaping to steer reinforcement learning more effectively. Additionally, we equip NeuOpt with Dynamic Data Augmentation (D2A) for more diverse searches during inference. Extensive experiments on the Traveling Salesman Problem (TSP) and Capacitated Vehicle Routing Problem (CVRP) demonstrate that our NeuOpt not only significantly outstrips existing (masking-based) L2S solvers, but also showcases superiority over the learning-to-construct (L2C) and learning-to-predict (L2P) solvers. Notably, we offer fresh perspectives on how neural solvers can handle VRP constraints. Our code is available: https://github.com/yining043/NeuOpt.
Vehicle Routing Problems (VRPs), prevalent in various real-world applications, present NP-hard combinatorial challenges that necessitate efficient search algorithms [1, 2]. In recent years, significant progress has been made in developing deep (reinforcement) learning-based solvers (e.g., [3-11]), which automates the tedious algorithm design with minimal human intervention in a data-driven fashion. Impressively, these neural solvers developed over the past five years have closed the gap to or even surpassed some traditional hand-crafted solvers that have evolved for several decades [5, 12].
In general, neural methods for VRPs can be categorized into learning-to-construct (L2C), learning-to-search (L2S), and learning-to-predict (L2P) solvers, each offering unique advantages while suffering respective drawbacks. L2C solvers (e.g., [4, 13]) are recognised for their fast solution construction but may struggle to escape local optima. L2P solvers excel at predicting crucial information (e.g., edge heatmaps [14, 15]), thereby simplifying the search especially for large-scale instances, but may lack the generality to efficiently handle VRP constraints beyond the Traveling Salesman Problem (TSP). L2S solvers (e.g., [8, 9]) are designed to learn exploration in the search space directly; however, their search efficiency is still limited and lags behind the state-of-the-art L2C and L2P solvers. In this paper, we delve into the limitations of existing L2S solvers, and aim to unleash their full potential.
* Zhiguang Cao is the corresponding author.
One potential issue of L2S solvers lies in their simplistic action space designs. Current L2S solvers that learn to control k-opt exchanges mostly leverage smaller  $k$  values (2-opt [9, 16] or 3-opt [17]), partly because their models struggle to efficiently deal with larger  $k$ . To address this issue, we introduce Neural k-Opt (NeuOpt), a flexible L2S solver capable of handling k-opt for any  $k \geq 2$ . Specifically, it employs a tailored action factorization method that simplifies and decomposes a complex k-opt exchange into a sequence of basis moves (S-move, I-move, and E-move) with the number of I-moves determining the  $k$  of a specifically executed k-opt action<sup>1</sup>. Such design allows k-opt exchanges to be easily constructed step-by-step, which more importantly, provides the deep model with the flexibility to explicitly and automatically determine an appropriate  $k$ . This further enables varying  $k$  values to be combined across different search steps, striking a balance between coarse-grained (larger  $k$ ) and fine-grained (smaller  $k$ ) searches. Correspondingly, we design a Recurrent Dual-Stream (RDS) decoder to decode such action factorization, which consists of recurrent networks and two complementary decoding streams for contextual modeling and attention computation, thereby capturing the strong correlations and dependencies between removed and added edges.
Besides, existing L2S solvers confine the search space to feasible regions based on feasibility masking. By contrast, we introduce a novel Guided Infeasible Region Exploration (GIRE) scheme that promotes the exploration of both feasible and infeasible regions. GIRE enriches the policy network with additional features that indicate constraint violations in the current solution and the exploration behaviour statistics in the search space. It also includes reward shaping to regulate extreme exploration behaviours and incentivize exploration at the boundaries of feasible and infeasible regions. Our GIRE offers four advantages: 1) it circumvents the non-trivial calculation of ground-truth action masks, particularly beneficial for constrained VRPs or broader action space as in NeuOpt, 2) it fosters searches at the more promising feasibility boundaries, similar to traditional solvers [18-21], 3) it bridges (possibly isolated) feasible regions, helping escape local optima and discover shortcuts to better solutions (See Figure 2), and 4) it forces explicit awareness of the VRP constraints, facilitating the deep model to understand the problem landscapes. In this paper, we apply GIRE to the Capacitated Vehicle Routing Problem (CVRP), though we note that it is generic to most VRP constraints.
Moreover, our NeuOpt leverages a Dynamic Data Augmentation (D2A) method during inference to enhance the search diversity and escape local optima. Our NeuOpt is trained via the reinforcement learning (RL) algorithm tailored in our previous work [12]. Extensive experiments on classic VRP variants (TSP and CVRP) validate our designs and demonstrate the superiority of NeuOpt and GIRE over existing approaches. Our contributions are four-fold: 1) we present NeuOpt, the first L2S solver that is flexible to handle k-opt with any  $k \geq 2$  based on a tailored formulation and a customized RDS decoder, 2) we introduce GIRE, the first scheme that extends beyond feasibility masking, enabling exploration of both feasible and infeasible regions in the search space, thereby bringing multiple benefits and offering fresh perspectives on handling VRP constraints, 3) we propose a simple yet effective D2A inference method for L2S solvers, and 4) we unleash the potential of L2S solvers and allow it to surpass L2C, L2P solvers, as well as the strong LKH-3 solver [20] on CVRP.
2 Literature review
We categorize recently developed neural methods for solving vehicle routing problems (VRPs) into learning-to-construct (L2C), learning-to-search (L2S), and learning-to-predict (L2P) solvers.
L2C solvers. They learn to construct solutions by iteratively adding nodes to the partial solution. The first modern L2C solver is  $\mathrm{Ptr}$ -Net [22] based on a Recurrent Neural Network (RNN) and supervised learning (extended to RL in [23] and CVRP in [24]). The Graph Neural Networks (GNN) were then leveraged for graph embedding [25] and faster encoding [26]. Later, the Attention Model (AM) was proposed by Kool et al. [3], inspiring many subsequent works (e.g., [4, 27-31]), where we highlight Policy Optimization with Multiple Optima (POMO) [4] which significantly improved AM with diverse rollouts and data augmentations. The L2C solvers can produce high-quality solutions within seconds using greedy rollouts; however, they are prone to get trapped in local optima, even when equipped with post-hoc methods (e.g., sampling [3], beam search [28], etc), or, advanced strategies (e.g., invariant representation [13, 32], learning collaborative policies [33], etc). Recently, the Efficient Active Search (EAS) [5] addressed such issues by updating a small subset of pre-trained
${}^{1}$  Broadly,in  $\mathrm{k}$  -opt,added edges may coincide with removed ones. Thus,2-opt may be viewed as degenerated 8-opt. To avoid ambiguity,unless specified,we refer to k-opt as an exchange introducing  $k$  entirely new edges.
model parameters on each test instance, which could be further boosted if coupled with Simulation Guided Beam Search (SGBS) [34], achieving the current state-of-the-art performance for L2C solvers.
L2S solvers. They learn to iteratively refine a solution to a new one, featuring a search process. Early attempts, e.g., NeuRewriter [35] and L2I [36], relied heavily on traditional local search algorithms and long run time. The NLNS solver [8] improved upon them by controlling a ruin-and-repair process that destroys parts of the solution using handcrafted operators and then fixes them using a learned deep model. Besides, the crossover exchanges between solutions were also learned in [37]. Recently, several L2S solvers focused on controlling the k-opt heuristic that is more suitable for VRPs [20, 38]. Wu et al. [39] made an early attempt to guide 2-opt, showing superior performance than the L2C solver AM [3]. Ma et al. [9] improved Wu et al. [39] by replacing vanilla attention with Dual-Aspect Collaborative Attention (DAC-Att) and a cyclic positional encoding method. The DAC-Att was then upgraded to Synthesis Attention (Synth-Att) [12] to reduce computational costs. Besides, Costa et al. [16] proposed an RNN-based policy to govern 2-opt, which was extended to 3-opt in [17] with higher efficiency. However, these neural k-opt solvers are limited by a small and fixed  $k$ . Besides, although L2S solvers strive to surpass L2C solvers by directly learning to search, they are still inferior to those state-of-the-art L2C solvers (e.g., POMO [4] and EAS [5]) even when given prolonged run time.
L2P solvers. They learn to guide the search by predicting critical information. Joshi et al. [14] proposed using GNN models to predict heatmaps that indicate probabilities of the presence of an edge in the optimal solution, which then uses beam search to solve TSP. It was leveraged for larger-scale TSP instances in [6] based on divide-and-conque, heatmap merging, and Monte Carlo Tree Search. In the GLS solver [40], a similar GNN was used to guide the traditional local search heuristics. More recently, the DIFUSCO solver [15] proposed to replace those GNN models with diffusion models [41]. Compared to L2C or L2S solvers, L2P solvers exhibit better scalability for large instances; however, they are mostly limited to supervised learning and TSP only, due to challenges in preparing training data and the ineffectiveness of heatmaps in handling VRP constraints. Though L2P solver DPDP [42] managed to solve CVRP with dynamic programming, it was outstripped by L2C solver EAS[5]. Recently, L2P solvers also explored predicting a latent continuous space for the underlying discrete solution space, where the latent space is then searched using differential evolution in [43] or gradient optimizer in [7]. Still, they can be either time-consuming or ineffective in tackling VRP constraints.
Feasibility satisfaction. Most neural solvers handle VRP constraints using the masking scheme that filters out invalid actions (e.g., [12, 44]). However, few works considered better ways of constraint handling. Although the works [45, 46] attempted to use mask prediction loss to enhance constraint awareness, they overlooked the benefits of the temporary constraint violation applied in many traditional solvers [18-21]. Lastly, we note that constraint handling in VRPs largely differs from safe RL tasks that focus on fully avoiding risky actions in uncertain environments [47, 48].
3 Preliminaries and notations
VRP notations. VRP aims to minimize the total travel cost (tour length) while serving a group of customers subject to certain constraints. It is defined on a complete directed graph  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$  where  $x_{i}\in \mathcal{V}$  are nodes (customers) and  $e(x_i\rightarrow x_j)\in \mathcal{E}$  are possible edges (route) weighted by the Euclidean distance between  $x_{i}$  and  $x_{j}$ . In the Traveling Salesman Problem (TSP), the solution is a Hamiltonian cycle that visits each node exactly once and returns to the starting one. In the Capacitated Vehicle Routing Problem (CVRP), a depot  $x_0$  is added to  $\nu$ , and each customer node  $x_{i}(i\geq 1)$  is assigned a demand  $\delta_{i}$ . A CVRP solution consists of multiple sub-tours, each representing a vehicle departing from the depot, serving a subset of customers, and returning to the depot, where each  $x_{i}(i\geq 1)$  is visited exactly once and the total demand of a sub-tour must not exceed the vehicle capacity  $\Delta$ . For instance,  $\tau = \{x_0\to x_2\to x_1\to x_0\to x_3\to x_0\}$  is a CVRP-3 solution with  $\delta = [5,5,9]$  and  $\Delta = 10$ .
Traditional k-opt heuristic. The k-opt heuristic iteratively refines a given solution by exchanging  $k$  existing edges with  $k$  (entirely) new ones. The Lin-Kernighan (LK) algorithm [49] narrowed the search with several criteria, where we underscore the sequential exchange criterion. It requires: for each  $i = 1,\dots ,k$ , the removed edge  $e_i^{\mathrm{out}}$  and added edge  $e_i^{\mathrm{in}}$  must share an endpoint, and so must  $e_i^{\mathrm{in}}$  and  $e_{i + 1}^{\mathrm{out}}$ . This allows a simplified sequential search process, alternating between removing and adding edges. Moreover, the LK algorithm considers scheduling varying  $k$  values in a repeated ascending order, so as to escape local optima by varying search neighbourhoods. Inspired by them, our paper proposes a powerful L2S solver that performs flexible k-opt exchanges with automatically chosen  $k$ .
Figure 1: Illustration of using our RDS decoder to determine a 3-opt exchange on TSP-9 given  $K = 4$  steps (with the E-move chosen at the final decoding step). The upper portion depicts a visual representation of how the dual-stream attentions (move stream  $\mu$  and edge stream  $\lambda$ ) are computed. The lower portion demonstrates how the inferred basis moves lead to the modification of the current solution. At step  $\kappa$ , RDS computes dual-stream attention from representations of historical decisions  $q_{\mu}^{\kappa}$ ,  $q_{\lambda}^{\kappa}$  to node embeddings  $h_i$ , thereby deciding a basis move  $\Phi_{\kappa}(x_{\kappa})$  by selecting  $x_{\kappa}$ . Ghost marks indicate the same location of a cyclic solution when viewed in a flat perspective as in this figure.
based on our action factorization method and the customized decoder. Recently, the LK algorithm has been implemented by Helsgaun [50] in the open-source LKH solver, with additional non-sequential exchanges and an edge candidate set. It was then upgraded to LKH-2 [51], leveraging more general k-opt exchanges, divide-and-conquer strategies, etc. The latest release, LKH-3 [20], further tackled constrained VRPs by penalizing constraint violations, making it a generic and powerful solver that serves as a benchmark for neural methods. Finally, we note that k-opt has been a foundation for various solvers, including the state-of-the-art Hybrid Genetic Search (HGS) solver [21] for CVRP.
4 Neural k-opt (NeuOpt)
Designing an effective neural k-opt solver necessitates addressing challenges potentially overlooked in prior works. Firstly, the solver should be generic for any given  $k \geq 2$ , using a unified formulation and architecture. Secondly, it should coherently parameterize the complex action space while accounting for the strong correlations and dependencies between removed and added edges. Finally, it should dynamically adjust  $k$  to balance coarse-grained (larger  $k$ ) and fine-grained (smaller  $k$ ) search steps.
In light of them, we introduce Neural k-Opt (NeuOpt) in this section. We first present our action factorization method for flexible k-opt exchanges, and then demonstrate our decoder to parameterize such actions, followed by the dynamic data augmentation method for inference. Note that this section focuses on TSP only, and we extend our NeuOpt to handle other VRP constraints in Section 5.
4.1 Formulations
We introduce a new factorization method that constructs a k-opt exchange using a combination of three basis moves, namely the starting move, the intermediate move, and the ending move. Concretely, the sequential k-opt can be simplified as performing one S-move, several (possibly none) I-moves, and finally one E-move, where the choice of the  $k$  corresponds to determining the number of I-moves.
S-move. The starting move removes an edge  $e^{\mathrm{out}}(x_a \to x_b)$ , converting a TSP solution  $\tau$  (a Hamiltonian cycle) into an open Hamiltonian path with two endpoints  $x_a$  and  $x_b$ . It is executed only at the beginning of action construction. We denote S-move as  $S(x_a)$  since  $x_b$  can be uniquely determined if  $x_a$  is specified. We term the source node  $x_a$  as anchor node to compute the node rank.
Definition 1 (Node rank w.r.t. anchor node) Given an instance  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  and a solution  $\tau$ , let  $x_{a}$  be the anchor node, as specified in an S-move. The node rank of  $x_{u}$ $(x_{u}\in \mathcal{V})$  w.r.t.  $x_{a}$ , denoted by  $\Gamma [x_a,x_u]$  or  $\Gamma [a,u]$ , is defined as the minimal number of edges in  $\tau$  needed to reach  $x_{u}$  from  $x_{a}$ .
I-move. The intermediate move adds a new edge, removes an existing edge, and fixes edge directions, transforming the open Hamiltonian path into a new one. Let  $x_{i}, x_{j}$  be the endpoints of the Hamiltonian
path before the I-move (with  $\Gamma [a,i] < \Gamma [a,j]$ ), and let  $e^{\mathrm{in}}(x_u\to x_v)$  be the introduced new edge. To avoid conflicts between consecutive I-moves, we impose sequential conditions on  $e^{\mathrm{in}}$ : (1) its source node  $x_{u}$  must be the endpoint of the current Hamiltonian path with a lower node rank, i.e.,  $x_{u} = x_{i}$ , and (2) the node rank of its target node  $x_{v}$ , i.e.,  $\Gamma [a,v]$ , must be higher than the node ranks of the two current endpoints  $x_{i},x_{j}$ , i.e.,  $\Gamma [a,i] < \Gamma [a,j] < \Gamma [a,v]$ . The removal of edge  $e^{out}(x_v\rightarrow x_w)$  followed when  $x_{v}$  is chosen, and directions of the edges between  $x_{j}$  and  $x_{v}$  are reversed to yield a valid Hamiltonian path. Since an I-move can be uniquely determined by  $x_{v}$ , we denote it as  $I(x_{v})$ .
E-move. The ending move adds a new edge connecting the two endpoints of the current Hamiltonian path, converting it into a Hamiltonian cycle, i.e., a new TSP solution  $\tau'$ . It is executed only at the end of action construction. Since it is uniquely determined without specifying any node, we denote it as  $E(x_{\mathrm{null}})$ . Note that if we relax the condition (2) of I-move to  $\Gamma[a,j] \leq \Gamma[a,v]$ , an E-move can be treated as a general I-move that selects  $x_v = x_j$ , denoted as  $I'(x_j)$  or  $E(x_j)$ .
MDP formulations. The examples of using the above basis moves to factorize 1-opt (void action), 2-opt, 3-opt, and 4-opt are depicted in Appendix A. Next, we present the Markov Decision Process (MDP) formulation  $\mathcal{M} = (\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\gamma < 1)$  for our NeuOpt. At step  $t$ , the state  $s_t = \{\mathcal{G},\tau_t,\tau_t^{\mathrm{bsf}}\}$  describes the current instance  $\mathcal{G}$ , the current solution  $\tau_{t}$ , and the best-so-far solution  $\tau_{t}^{\mathrm{bsf}}$  found before step  $t$ . Given a maximum number of allowed basis moves  $K(K\geq 2)$ , an action consists of  $K$  basis moves  $\Phi_{\kappa}(x_{\kappa})$ , i.e.,  $a_{t} = \{\Phi_{\kappa}(x_{\kappa}),\kappa = 1,\dots ,K\}$ , where the first move  $\Phi_1(x_1) = S(x_1)$  is an S-move, and the rest is either an I-move or an E-move (in the form of general I-move). Note that 1) we permit the I-move as the last move, adding an E-move during state transition if so, and 2) to ensure the action length is always  $K$ , we include null actions if E-move early terminates the action. Our state transition rule  $\mathcal{T}$  is deterministic and it updates  $s_t$  to  $s_{t + 1}$  based on the above action factorization method. We use reward  $r_t = f(\tau_t^{\mathrm{bsf}}) - min\left[f(\tau_{t + 1}),f(\tau_t^{\mathrm{bsf}})\right]$  following [9, 12].
4.2 Recurrent Dual-Stream (RDS) decoder
Our NeuOpt adopts an encoder-decoder-styled policy network. We use the encoder from our previous work [12] but upgrade the linear projection to an MLP for embedding generation (details are presented in Appendix B). Here, we introduce the proposed recurrent dual-stream (RDS) decoder to effectively parameterize the aforementioned k-opt action factorization (see Figure 1 for a concrete example).
GRUs for action factorization. An action  $a = \{\Phi_{\kappa}(x_{\kappa}),\kappa = 1,\dots ,K\}$  is constructed sequentially by  $K$  steps, where each decoding step  $\kappa$  specifies a basis move type  $\Phi_{\kappa}$  and a node  $x_{\kappa}$  to instantiate the move. Such decoding can be further simplified to a node selection process, with move types inferred based on: (1) for  $\kappa = 1$ , it is an S-move; (2) for  $\kappa >1$ , it is an I-move if  $\Gamma [a,j_{\kappa}] < \Gamma [a,\kappa]$  (assuming  $x_{i_\kappa},x_{j_\kappa}$  are the Hamiltonian path endpoints before the  $\kappa$ -th move with  $\Gamma [a,i_{\kappa}] < \Gamma [a,j_{\kappa}]$ ), otherwise, if  $x_{\kappa} = x_{j_{\kappa}}$ , it is an E-move that early stops the decoding. Formally, we use factorization:
$$
\pi_ {\theta} (a | s) = P _ {\theta} \left(\Phi_ {1} \left(x _ {1}\right), \Phi_ {2} \left(x _ {2}\right), \dots , \Phi_ {K} \left(x _ {K}\right) \mid s\right) = \prod_ {\kappa = 1} ^ {K} P _ {\theta} ^ {\kappa} \left(\Phi_ {\kappa} \mid \Phi_ {1}, \dots , \Phi_ {\kappa - 1}, s\right) \tag {1}
$$
where  $P_{\theta}^{\kappa}$  is a categorical distribution over  $N$  nodes for node selection. Our decoder leverages the Gated Recurrent Units (GRUs) [52] to help parameterize the conditional probabilities  $P_{\theta}^{\kappa}$ . Given node embeddings  $h \in \mathbb{R}^{N \times d}$  from the encoders ( $h_i$  is a  $d$ -dimensional vector), the decoder first computes hidden representations  $q^{\kappa}$  to model the historical move decisions  $\{\Phi_1, \dots, \Phi_{k-1}\}$  (the conditions of  $P_{\theta}^{\kappa}$ ) using Eq. (2), where  $q^{\kappa}$  (hidden state of GRU) is derived from  $q^{\kappa-1}$ , based on an input  $o^{\kappa}$ . For better contextual modeling, we consider two streams,  $\mu$  and  $\lambda$ , which differ from each other by learning independent parameters and taking different inputs  $o^{\kappa}$  at each  $\kappa$ . The  $q_{\mu}^{0} = q_{\lambda}^{0} = \frac{1}{N} \sum_{i=1}^{N} h_i$  are initialized by the mean pooling of node embeddings (i.e., a graph embedding).
$$
q _ {\mu} ^ {\kappa} = \operatorname {G R U} \left(o _ {\mu} ^ {\kappa}, q _ {\mu} ^ {\kappa - 1}\right), q _ {\lambda} ^ {\kappa} = \operatorname {G R U} \left(o _ {\lambda} ^ {\kappa}, q _ {\lambda} ^ {\kappa - 1}\right) \tag {2}
$$
Dual-stream contextual modeling. We employ a move stream  $\mu$  and an edge stream  $\lambda$  during contextual modeling and subsequent attention computation. On the one hand, to provide a relatively overall view, the move stream  $\mu$  considers modeling historical move decisions by taking the node embedding of the last selected node  $x_{\kappa -1}$ , which is a representation of the past selected move  $\Phi_{\kappa -1}(x_{\kappa -1})$ , as its GRU input, i.e.,  $o_{\mu}^{\kappa} = h_{\kappa -1}$ . On the other hand, to offer a relatively detailed view, the edge stream  $\lambda$  focuses on edge proposals in each step  $\kappa$  by taking the node embedding of  $x_{i_{\kappa}}$  which is stipulated to be the source node of the edge to be introduced in step  $\kappa$ , as its GRU input,