Title: Learning to Search Feasible and Infeasible Regions of Routing Problems with Flexible Neural k-Opt
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning to optimize

vehicle routing problem

combinatorial optimization
Abstract: In this paper, we present Neural k-Opt (NeuOpt), a novel learning-to-search (L2S) solver for routing problems. It learns to perform flexible k-opt exchanges based on a tailored action factorization method and a customized recurrent dual-stream decoder. As a pioneering work to circumvent the pure feasibility masking scheme and enable the autonomous exploration of both feasible and infeasible regions, we then propose the Guided Infeasible Region Exploration (GIRE) scheme, which supplements the NeuOpt policy network with feasibility-related features and leverages reward shaping to steer reinforcement learning more effectively. Additionally, we equip NeuOpt with Dynamic Data Augmentation (D2A) for more diverse searches during inference. Extensive experiments on the Traveling Salesman Problem (TSP) and Capacitated Vehicle Routing Problem (CVRP) demonstrate that our NeuOpt not only significantly outstrips existing (masking-based) L2S solvers, but also showcases superiority over the learning-to-construct (L2C) and learning-to-predict (L2P) solvers. Notably, we offer fresh perspectives on how neural solvers can handle VRP constraints. Our code is available: https://github.com/yining043/NeuOpt.

Learning to Search Feasible and Infeasible Regions of Routing Problems with Flexible Neural k-Opt

Abstract

In this paper, we present Neural k-Opt (NeuOpt), a novel learning-to-search (L2S) solver for routing problems. It learns to perform flexible k-opt exchanges based on a tailored action factorization method and a customized recurrent dual-stream decoder. As a pioneering work to circumvent the pure feasibility masking scheme and enable the autonomous exploration of both feasible and infeasible regions, we then propose the Guided Infeasible Region Exploration (GIRE) scheme, which supplements the NeuOpt policy network with feasibility-related features and leverages reward shaping to steer reinforcement learning more effectively. Additionally, we equip NeuOpt with Dynamic Data Augmentation (D2A) for more diverse searches during inference. Extensive experiments on the Traveling Salesman Problem (TSP) and Capacitated Vehicle Routing Problem (CVRP) demonstrate that our NeuOpt not only significantly outstrips existing (masking-based) L2S solvers, but also showcases superiority over the learning-to-construct (L2C) and learning-to-predict (L2P) solvers. Notably, we offer fresh perspectives on how neural solvers can handle VRP constraints. Our code is available: https://github.com/yining043/NeuOpt .

1 Introduction

Vehicle Routing Problems (VRPs), prevalent in various real-world applications, present NP-hard combinatorial challenges that necessitate efficient search algorithms [1, 2]. In recent years, significant progress has been made in developing deep (reinforcement) learning-based solvers (e.g., [3-11]), which automates the tedious algorithm design with minimal human intervention in a data-driven fashion. Impressively, these neural solvers developed over the past five years have closed the gap to or even surpassed some traditional hand-crafted solvers that have evolved for several decades [5, 12].

In general, neural methods for VRPs can be categorized into learning-to-construct (L2C), learning-tosearch (L2S), and learning-to-predict (L2P) solvers, each offering unique advantages while suffering respective drawbacks. L2C solvers (e.g., [4, 13]) are recognised for their fast solution construction but may struggle to escape local optima. L2P solvers excel at predicting crucial information (e.g., edge heatmaps [14, 15]), thereby simplifying the search especially for large-scale instances, but may lack the generality to efficiently handle VRP constraints beyond the Traveling Salesman Problem (TSP). L2S solvers (e.g., [8, 9]) are designed to learn exploration in the search space directly; however, their search efficiency is still limited and lags behind the state-of-the-art L2C and L2P solvers. In this paper, we delve into the limitations of existing L2S solvers, and aim to unleash their full potential.

One potential issue of L2S solvers lies in their simplistic action space designs. Current L2S solvers that learn to control k-opt exchanges mostly leverage smaller k values (2-opt [9, 16] or 3-opt [17]), partly because their models struggle to efficiently deal with larger k . To address this issue, we introduce Neural k-Opt (NeuOpt), a flexible L2S solver capable of handling k-opt for any k ≥ 2 . Specifically, it employs a tailored action factorization method that simplifies and decomposes a complex k-opt exchange into a sequence of basis moves (S-move, I-move, and E-move) with the number of I-moves determining the k of a specifically executed k-opt action 1 . Such design allows kopt exchanges to be easily constructed step-by-step, which more importantly, provides the deep model with the flexibility to explicitly and automatically determine an appropriate k . This further enables varying k values to be combined across different search steps, striking a balance between coarsegrained (larger k ) and fine-grained (smaller k ) searches. Correspondingly, we design a Recurrent Dual-Stream (RDS) decoder to decode such action factorization, which consists of recurrent networks and two complementary decoding streams for contextual modeling and attention computation, thereby capturing the strong correlations and dependencies between removed and added edges.

Besides, existing L2S solvers confine the search space to feasible regions based on feasibility masking. By contrast, we introduce a novel Guided Infeasible Region Exploration (GIRE) scheme that promotes the exploration of both feasible and infeasible regions. GIRE enriches the policy network with additional features that indicate constraint violations in the current solution and the exploration behaviour statistics in the search space. It also includes reward shaping to regulate extreme exploration behaviours and incentivize exploration at the boundaries of feasible and infeasible regions. Our GIRE offers four advantages: 1) it circumvents the non-trivial calculation of ground-truth action masks, particularly beneficial for constrained VRPs or broader action space as in NeuOpt, 2) it fosters searches at the more promising feasibility boundaries, similar to traditional solvers [18-21], 3) it bridges (possibly isolated) feasible regions, helping escape local optima and discover shortcuts to better solutions (See Figure 2), and 4) it forces explicit awareness of the VRP constraints, facilitating the deep model to understand the problem landscapes. In this paper, we apply GIRE to the Capacitated Vehicle Routing Problem (CVRP), though we note that it is generic to most VRP constraints.

Moreover, our NeuOpt leverages a Dynamic Data Augmentation (D2A) method during inference to enhance the search diversity and escape local optima. Our NeuOpt is trained via the reinforcement learning (RL) algorithm tailored in our previous work [12]. Extensive experiments on classic VRP variants (TSP and CVRP) validate our designs and demonstrate the superiority of NeuOpt and GIRE over existing approaches. Our contributions are four-fold: 1) we present NeuOpt, the first L2S solver that is flexible to handle k-opt with any k ≥ 2 based on a tailored formulation and a customized RDS decoder, 2) we introduce GIRE, the first scheme that extends beyond feasibility masking, enabling exploration of both feasible and infeasible regions in the search space, thereby bringing multiple benefits and offering fresh perspectives on handling VRP constraints, 3) we propose a simple yet effective D2A inference method for L2S solvers, and 4) we unleash the potential of L2S solvers and allow it to surpass L2C, L2P solvers, as well as the strong LKH-3 solver [20] on CVRP.

2 Literature review

We categorize recently developed neural methods for solving vehicle routing problems (VRPs) into learning-to-construct (L2C), learning-to-search (L2S), and learning-to-predict (L2P) solvers.

L2C solvers. They learn to construct solutions by iteratively adding nodes to the partial solution. The first modern L2C solver is Ptr-Net [22] based on a Recurrent Neural Network (RNN) and supervised learning (extended to RL in [23] and CVRP in [24]). The Graph Neural Networks (GNN) were then leveraged for graph embedding [25] and faster encoding [26]. Later, the Attention Model (AM) was proposed by Kool et al. [3], inspiring many subsequent works (e.g., [4, 27-31]), where we highlight Policy Optimization with Multiple Optima (POMO) [4] which significantly improved AM with diverse rollouts and data augmentations. The L2C solvers can produce high-quality solutions within seconds using greedy rollouts; however, they are prone to get trapped in local optima, even when equipped with post-hoc methods (e.g., sampling [3], beam search [28], etc), or, advanced strategies (e.g., invariant representation [13, 32], learning collaborative policies [33], etc). Recently, the Efficient Active Search (EAS) [5] addressed such issues by updating a small subset of pre-trained

model parameters on each test instance, which could be further boosted if coupled with Simulation Guided Beam Search (SGBS) [34], achieving the current state-of-the-art performance for L2C solvers.

L2S solvers. They learn to iteratively refine a solution to a new one, featuring a search process. Early attempts, e.g., NeuRewriter [35] and L2I [36], relied heavily on traditional local search algorithms and long run time. The NLNS solver [8] improved upon them by controlling a ruin-and-repair process that destroys parts of the solution using handcrafted operators and then fixes them using a learned deep model. Besides, the crossover exchanges between solutions were also learned in [37]. Recently, several L2S solvers focused on controlling the k-opt heuristic that is more suitable for VRPs [20, 38]. Wu et al. [39] made an early attempt to guide 2-opt, showing superior performance than the L2C solver AM [3]. Ma et al. [9] improved Wu et al. [39] by replacing vanilla attention with Dual-Aspect Collaborative Attention (DAC-Att) and a cyclic positional encoding method. The DAC-Att was then upgraded to Synthesis Attention (Synth-Att) [12] to reduce computational costs. Besides, Costa et al. [16] proposed an RNN-based policy to govern 2-opt, which was extended to 3-opt in [17] with higher efficiency. However, these neural k-opt solvers are limited by a small and fixed k . Besides, although L2S solvers strive to surpass L2C solvers by directly learning to search, they are still inferior to those state-of-the-art L2C solvers (e.g., POMO [4] and EAS [5]) even when given prolonged run time.

L2P solvers. They learn to guide the search by predicting critical information. Joshi et al. [14] proposed using GNN models to predict heatmaps that indicate probabilities of the presence of an edge in the optimal solution, which then uses beam search to solve TSP. It was leveraged for larger-scale TSP instances in [6] based on divide-and-conque, heatmap merging, and Monte Carlo Tree Search. In the GLS solver [40], a similar GNN was used to guide the traditional local search heuristics. More recently, the DIFUSCO solver [15] proposed to replace those GNN models with diffusion models [41]. Compared to L2C or L2S solvers, L2P solvers exhibit better scalability for large instances; however, they are mostly limited to supervised learning and TSP only, due to challenges in preparing training data and the ineffectiveness of heatmaps in handling VRP constraints. Though L2P solver DPDP [42] managed to solve CVRP with dynamic programming, it was outstripped by L2C solver EAS[5]. Recently, L2P solvers also explored predicting a latent continuous space for the underlying discrete solution space, where the latent space is then searched using differential evolution in [43] or gradient optimizer in [7]. Still, they can be either time-consuming or ineffective in tackling VRP constraints.

Feasibility satisfaction. Most neural solvers handle VRP constraints using the masking scheme that filters out invalid actions (e.g., [12, 44]). However, few works considered better ways of constraint handling. Although the works [45, 46] attempted to use mask prediction loss to enhance constraint awareness, they overlooked the benefits of the temporary constraint violation applied in many traditional solvers [18-21]. Lastly, we note that constraint handling in VRPs largely differs from safe RL tasks that focus on fully avoiding risky actions in uncertain environments [47, 48].

3 Preliminaries and notations

VRP notations. VRP aims to minimize the total travel cost (tour length) while serving a group of customers subject to certain constraints. It is defined on a complete directed graph G = {V , E} where x i ∈V are nodes (customers) and e ( x i → x j ) ∈E are possible edges (route) weighted by the Euclidean distance between x i and x j . In the Traveling Salesman Problem (TSP), the solution is a Hamiltonian cycle that visits each node exactly once and returns to the starting one. In the Capacitated Vehicle Routing Problem (CVRP), a depot x 0 is added to V , and each customer node x i ( i ≥ 1) is assigned a demand δ i . A CVRP solution consists of multiple sub-tours, each representing a vehicle departing from the depot, serving a subset of customers, and returning to the depot, where each x i ( i ≥ 1) is visited exactly once and the total demand of a sub-tour must not exceed the vehicle capacity ∆ . For instance, τ = { x 0 → x 2 → x 1 → x 0 → x 3 → x 0 } is a CVRP-3 solution with δ =[5 , 5 , 9] and ∆=10 .

Traditional k-opt heuristic. The k-opt heuristic iteratively refines a given solution by exchanging k existing edges with k (entirely) new ones. The Lin-Kernighan (LK) algorithm [49] narrowed the search with several criteria, where we underscore the sequential exchange criterion . It requires: for each i = 1 , . . . , k , the removed edge e out i and added edge e in i must share an endpoint, and so must e in i and e out i +1 . This allows a simplified sequential search process, alternating between removing and adding edges. Moreover, the LK algorithm considers scheduling varying k values in a repeated ascending order, so as to escape local optima by varying search neighbourhoods. Inspired by them, our paper proposes a powerful L2S solver that performs flexible k-opt exchanges with automatically chosen k

𝒉𝟒

Anchor node

Node to select

GRU

𝒒𝝁

𝟎

𝒒𝝁

𝟏

𝒐𝝁

𝟏

GRU

𝒒𝝀

𝟎

𝒒𝝀

𝟏

𝒉𝟏

𝒉𝟐

𝒉𝟑

𝒉𝟒

𝒉𝟓

𝒉𝟔

𝒉𝟕

𝒉𝟖

𝒉𝟗

𝒐𝝀

𝟏

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

→

𝒙𝟑)]

Remove

Add

[ ]

Input/Output

Action:

{𝑺 𝒙𝟐

}

GRU

𝒒𝝁

𝟐

𝒐𝝁

𝟐

= 𝒉𝟐

GRU

𝒒𝝀

𝟐

𝒉𝟏

𝒉𝟑

𝒉𝟓

𝒉𝟔

𝒉𝟕

𝒉𝟖

𝒉𝟗

𝒐𝝀

𝟐

= 𝒉𝟐

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

→

𝒙𝟑

, (𝒙 𝟒

→

𝒙𝟓)]

Remove

Add

→

𝒙𝟒)]

GRU

𝒒𝝁

𝟑

GRU

𝒒𝝀

𝟑

𝒉𝟏

𝒉𝟐

𝒉𝟒

𝒉𝟓

𝒉𝟔

𝒉𝟕

𝒉𝟖

𝒉𝟗

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟕

𝒙𝟖

𝒙𝟗

𝒙𝟔

→

𝒙𝟑

, (𝒙 𝟒

→

𝒙𝟓), (𝒙 𝟕

→

𝒙𝟖)]

Remove

Add

→

𝒙𝟒

, (𝒙 𝟑

→

𝒙𝟕)]

GRU

𝒒𝝁

𝟒

GRU

𝒒𝝀

𝟒

𝒉𝟏

𝒉𝟐

𝒉𝟑

𝒉𝟒

𝒉𝟔

𝒉𝟕

𝒉𝟖

𝒉𝟗

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟖

𝒙𝟗

𝒙𝟕

𝒙𝟔

→

𝒙𝟑

,

𝒙𝟒

→

𝒙𝟓

,

𝒙𝟕

→

𝒙𝟖

]

Remove

Add

→

𝒙𝟒

, (𝒙 𝟑

→

𝒙𝟕), (𝒙 𝟓

→

𝒙𝟖)]

Action:

{𝑺 𝒙𝟐

, 𝑰 𝒙𝟒

, 𝑰 𝒙𝟕

, 𝑬 𝒙𝟖

}

Last selected node

Current source node

Action:

{𝑺 𝒙𝟐

, 𝑰 𝒙𝟒

}

Removed edge

Endpoint

𝜞[

2

,

3]=1

𝜞[𝒂, 𝒖]

The rank of

𝑥𝑢

w.r.t anchor

𝑥𝑎

Move stream

𝜇

Attention

Direction change

𝜞[

2

,

5]=3

𝒉𝟐

𝒉𝟑

𝜞[

2

,

5]=3

𝜞[

2

,

8]=6

𝒉𝟓

Edge stream

𝜆

Reference

Ghost mark

Masked attention score

Attention score

Added edge

Decoder

𝜿

= 1

Action:

{𝑺 𝒙𝟐

, 𝑰 𝒙𝟒

, 𝑰 𝒙𝟕

}

Decoder

𝜿

= 2

Decoder

𝜿

=3

Decoder

𝜿

=4

𝒐𝝀

𝟑

= 𝒉𝟑

𝒐𝝁

𝟑

= 𝒉𝟒

𝒐𝝀

𝟒

= 𝒉𝟓

𝒐𝝁

𝟒

= 𝒉𝟕

Original solution

𝝉

New solution

𝝉 ′

based on our action factorization method and the customized decoder. Recently, the LK algorithm has been implemented by Helsgaun [50] in the open-source LKH solver, with additional non-sequential exchanges and an edge candidate set. It was then upgraded to LKH-2 [51], leveraging more general k-opt exchanges, divide-and-conquer strategies, etc. The latest release, LKH-3 [20], further tackled constrained VRPs by penalizing constraint violations, making it a generic and powerful solver that serves as a benchmark for neural methods. Finally, we note that k-opt has been a foundation for various solvers, including the state-of-the-art Hybrid Genetic Search (HGS) solver [21] for CVRP.

4 Neural k-opt (NeuOpt)

Designing an effective neural k-opt solver necessitates addressing challenges potentially overlooked in prior works. Firstly, the solver should be generic for any given k ≥ 2 , using a unified formulation and architecture. Secondly, it should coherently parameterize the complex action space while accounting for the strong correlations and dependencies between removed and added edges. Finally, it should dynamically adjust k to balance coarse-grained (larger k ) and fine-grained (smaller k ) search steps.

In light of them, we introduce Neural k-Opt (NeuOpt) in this section. We first present our action factorization method for flexible k-opt exchanges, and then demonstrate our decoder to parameterize such actions, followed by the dynamic data augmentation method for inference. Note that this section focuses on TSP only, and we extend our NeuOpt to handle other VRP constraints in Section 5.

4.1 Formulations

We introduce a new factorization method that constructs a k-opt exchange using a combination of three basis moves , namely the starting move , the intermediate move , and the ending move . Concretely, the sequential k-opt can be simplified as performing one S-move, several (possibly none) I-moves, and finally one E-move, where the choice of the k corresponds to determining the number of I-moves.

S-move. The starting move removes an edge e out ( x a → x b ) , converting a TSP solution τ (a Hamiltonian cycle) into an open Hamiltonian path with two endpoints x a and x b . It is executed only at the beginning of action construction. We denote S-move as S ( x a ) since x b can be uniquely determined if x a is specified. We term the source node x a as anchor node to compute the node rank .

Definition 1 (Node rank w.r.t. anchor node) Given an instance G =( V , E ) and a solution τ , let x a be the anchor node, as specified in an S-move. The node rank of x u ( x u ∈ V ) w.r.t. x a , denoted by Γ[ x a , x u ] or Γ[ a, u ] , is defined as the minimal number of edges in τ needed to reach x u from x a .

I-move. The intermediate move adds a new edge, removes an existing edge, and fixes edge directions, transforming the open Hamiltonian path into a new one. Let x i , x j be the endpoints of the Hamiltonian

path before the I-move (with Γ[ a, i ] < Γ[ a, j ] ), and let e in ( x u → x v ) be the introduced new edge. To avoid conflicts between consecutive I-moves, we impose sequential conditions on e in : (1) its source node x u must be the endpoint of the current Hamiltonian path with a lower node rank, i.e., x u = x i , and (2) the node rank of its target node x v , i.e., Γ[ a, v ] , must be higher than the node ranks of the two current endpoints x i , x j , i.e., Γ[ a, i ] < Γ[ a, j ] < Γ[ a, v ] . The removal of edge e out ( x v → x w ) followed when x v is chosen, and directions of the edges between x j and x v are reversed to yield a valid Hamiltonian path. Since an I-move can be uniquely determined by x v , we denote it as I ( x v ) .

E-move. The ending move adds a new edge connecting the two endpoints of the current Hamiltonian path, converting it into a Hamiltonian cycle, i.e., a new TSP solution τ ′ . It is executed only at the end of action construction. Since it is uniquely determined without specifying any node, we denote it as E ( x null ) . Note that if we relax the condition (2) of I-move to Γ[ a, j ] ≤ Γ[ a, v ] , an E-move can be treated as a general I-move that selects x v = x j , denoted as I ′ ( x j ) or E ( x j ) .

MDPformulations. The examples of using the above basis moves to factorize 1-opt (void action), 2-opt, 3-opt, and 4-opt are depicted in Appendix A. Next, we present the Markov Decision Process (MDP) formulation M = ( S , A , T , R , γ < 1) for our NeuOpt. At step t , the state s t = { G , τ t , τ bsf t } describes the current instance G , the current solution τ t , and the best-so-far solution τ bsf t found before step t . Given a maximum number of allowed basis moves K ( K ≥ 2) , an action consists of K basis moves Φ κ ( x κ ) , i.e., a t = { Φ κ ( x κ ) , κ =1 , . . . , K } , where the first move Φ 1 ( x 1 ) = S ( x 1 ) is an S-move, and the rest is either an I-move or an E-move (in the form of general I-move). Note that 1) we permit the I-move as the last move, adding an E-move during state transition if so, and 2) to ensure the action length is always K , we include null actions if E-move early terminates the action. Our state transition rule T is deterministic and it updates s t to s t +1 based on the above action factorization method. We use reward r t = f ( τ bsf t ) -min [ f ( τ t +1 ) , f ( τ bsf t ) ] following [9, 12].

4.2 Recurrent Dual-Stream (RDS) decoder

Our NeuOpt adopts an encoder-decoder-styled policy network. We use the encoder from our previous work [12] but upgrade the linear projection to an MLP for embedding generation (details are presented in Appendix B). Here, we introduce the proposed recurrent dual-stream (RDS) decoder to effectively parameterize the aforementioned k-opt action factorization (see Figure 1 for a concrete example).

GRUs for action factorization. An action a = { Φ κ ( x κ ) , κ =1 , . . . , K } is constructed sequentially by K steps, where each decoding step κ specifies a basis move type Φ κ and a node x κ to instantiate the move. Such decoding can be further simplified to a node selection process , with move types inferred based on: (1) for κ =1 , it is an S-move; (2) for κ> 1 , it is an I-move if Γ[ a, j κ ] < Γ[ a, κ ] (assuming x i κ , x j κ are the Hamiltonian path endpoints before the κ -th move with Γ[ a, i κ ] < Γ[ a, j κ ] ), otherwise, if x κ = x j κ , it is an E-move that early stops the decoding. Formally, we use factorization:

where P κ θ is a categorical distribution over N nodes for node selection. Our decoder leverages the Gated Recurrent Units (GRUs) [52] to help parameterize the conditional probabilities P κ θ . Given node embeddings h ∈ R N × d from the encoders ( h i is a d -dimensional vector), the decoder first computes hidden representations q κ to model the historical move decisions { Φ 1 , . . . , Φ k -1 } (the conditions of P κ θ ) using Eq. (2), where q κ (hidden state of GRU) is derived from q κ -1 , based on an input o κ . For better contextual modeling, we consider two streams, µ and λ , which differ from each other by learning independent parameters and taking different inputs o κ at each κ . The q 0 µ = q 0 λ = 1 N ∑ N i =1 h i are initialized by the mean pooling of node embeddings (i.e., a graph embedding).

Dual-stream contextual modeling. We employ a move stream µ and an edge stream λ during contextual modeling and subsequent attention computation. On the one hand, to provide a relatively overall view, the move stream µ considers modeling historical move decisions by taking the node embedding of the last selected node x κ -1 , which is a representation of the past selected move Φ κ -1 ( x κ -1 ) , as its GRU input, i.e., o κ µ = h κ -1 . On the other hand, to offer a relatively detailed view, the edge stream λ focuses on edge proposals in each step κ by taking the node embedding of x i κ , which is stipulated to be the source node of the edge to be introduced in step κ , as its GRU input,

i.e., o κ λ = h i κ . We intend the two streams to complement each other, coherently capturing complex correlations and dependencies among historical decisions. For the initialization, we use learnable parameters o 1 µ =ˆ o , o 1 λ = ˜ o . Based on the attention between q µ , q λ and node embeddings h , the node selection scores suggested by both streams can then be calculated:

where all matrices W ∈ R d × d are trainable parameters and we consider both summation and Hadamard products. Afterwards, the final categorical distribution P κ θ is induced by: Softmax ( C · tanh ( µ κ + λ κ )) where C =6 and invalid choices that do not satisfy Γ[ a, j κ ] ≤ Γ[ a, κ ] are masked as -∞ (an E-move is enforced if all nodes are masked, which happens when the highest-ranked node was chosen in the previous I-move). A node x κ is then sampled from P κ θ , after which its basis move type Φ κ can be inferred. The decoding process continues until it early terminates or reaches the maximum limit K .

4.3 Inference with the dynamic data augmentation (D2A)

Our NeuOpt is trained using the RL algorithm tailored in our previous work [12]. During inference, our previous work [12] employed a data augmentation method that applies optimal-solution-invariant transformations (e.g., flipping coordinates) on an instance, G , so as to generate a set of instances {G 1 , G 2 , ..., } for the trained model to solve. However, it was performed only once at the beginning of the inference. In this paper, we propose Dynamic Data Augmentation (D2A), an enhancement that generates new augmented instances each time the solver fails to find a better solution (i.e., getting trapped in local optima) within a consecutive maximum of T D2A steps. This allows for more diverse searches. Further details regarding training and inference algorithms are provided in Appendix C.

5 Guided infeasible region exploration (GIRE)

As demonstrated in Figure 2, the feasible region is usually fragmented with isolated islands. Masking in neural solvers typically restricts the search to feasible regions, which may lead to inefficient search trajectories or failure to find global optima. Our GIRE explores both feasible and infeasible regions, fostering shortcut discovery, more promising boundary searches, and the identification of possibly isolated regions. We now illustrate our GIRE by handling the capacity constraint in CVRP.

Global Optima

Trajectory (our GIRE scheme)

Trajectory (masking scheme)

Local Optima

Feasible (

𝓕

)

𝝐

-Feasible (

𝝐

-

𝓕

)

Infeasible (

𝓤

)

𝓕

𝓤

𝝐

-

𝓕

Definition 2 (Search space of CVRP) : We define the feasible search space F as the set of solutions that satisfy both TSP and the (CVRP-specific) capacity constraint; the infeasible search space U as the set of solutions that satisfy the TSP constraint but not the capacity constraint; and the ϵ -feasible search space ϵ -F ⊆ U as the set of solutions with a capacity violation percentage not exceeding ϵ .

Feature supplement. GIRE suggests supplementing Violation Indicator (VI) features and Exploration Statistics (ES) features into the policy network to identify specific constraint violations and understand its ongoing exploration behaviour. The VI features flag the specific infeasible portions of the current solution. For CVRP, we use two binary variables indicating if the cumulative demands exceed the capacity before or after visiting a particular node, respectively, which are treated as node features during the node embedding generation. Such an idea could work with most VRP constraints, e.g., indicating the nodes that violate their time windows (TW) for CVRP-TW. The ES features provide the network with ongoing exploration behaviour statistics to guide future exploration behaviour. We define H t := { ( τ t ′ → τ t ′ +1 ) }| t -1 t ′ = t -T his as the collection of the most recent T his steps of solution transition record (if they exist). The ES features J t consist of estimated feasibility transition probabilities derived from H t , including P ( τ ∈F , τ ′ ∈U ) , P ( τ ∈U , τ ′ ∈F ) , P ( τ ∈F , τ ′ ∈F ) , P ( τ ∈U , τ ′ ∈U ) , P ( τ ′ ∈F| τ ∈U ) , P ( τ ′ ∈U| τ ∈F ) , P ( τ ′ ∈F| τ ∈F ) , and P ( τ ′ ∈U| τ ∈U ) , along with a binary indicator w.r.t the feasibility of the current solution τ t . In order to make the policy network decisions dependent on these ES features, we introduce two hypernetworks [53], namely, MLP µ and MLP λ , which take ES features J t as inputs and generate the parameters of the last decoder layer, i.e., W O µ , W O λ of Eq. (3), respectively. We adopt the structure ( 9 × 8 × d ), and share the first hidden layer between W O µ and W O λ to reduce computational costs.

Reward shaping. Moreover, GIRE employs reward shaping:

to guide the reinforcement learning, where r t is the original reward, r reg t regulates extreme exploration behaviours; r bonus t encourages exploration in the ϵ -feasible regions; and α and β are reward shaping weights (we use 0.05 for both). Here, the regulation r reg t is determined by an entropy measure H [ P ] of the estimated conditional transforming probabilities P t ( U|U )= P ( τ ′ ∈U| τ ∈U ) and P t ( F|F )= P ( τ ′ ∈F| τ ∈F ) :

0.00

0.25

0.50

0.75

1.00

Probability

0.0

0.5

1.0

Entropy measure

where expectation E [ r t ] , that suggests the magnitude of r reg t , is estimated during training; the entropy measure H [ P ] , as shown in Figure 3, imposes larger penalties when P is either too high or too low (indicating extreme exploration behaviour). The bonus r bonus t utilizes a similar reward function as the regular reward r t ; however, it only considers an infeasible but ϵ -feasible solution as a potential new best-so-far solution. More illustrations and discussions on GIRE designs are detailed in Appendix D.

6 Experiments

We conduct experiments on TSP and CVRP, with sizes N = 20, 50, 100 following the conventions [9, 13, 37]. Training and test instances are uniformly generated following [3]. For NeuOpt, we use K =4 ; the initial solutions are sequentially constructed in a random fashion for both training and inference. Results were collected using a machine equipped with NVIDIA 2080TI GPU cards and an Intel E5-2680 CPU at 2.40GHz. More hyper-parameter details, discussions, and additional results are available in Appendix E. Our PyTorch code and pre-trained models are publicly available 2 .

6.1 Comparison studies

Setup. In Table 1, we benchmark our NeuOpt (TSP) and NeuOpt-GIRE (CVRP) against a variety of neural solvers, namely, 1) L2P solvers : GCN+BS [14] (TSP only), Att-GCN+MCTS [6] (TSP only), GNN+GLS [40] (TSP only), CVAE-Opt-DE [43], DPDP [42] (state-of-the-art), DIMES [7] (TSP only), DIFUSCO [15] (TSP only, state-of-the-art); 2) L2C solvers : AM+LCP [33], POMO [4], Pointerformer [32] (TSP only), Sym-NCO [13], POMO+EAS [5], POMO+EAS+SGBS [34] (state-ofthe-art); and 3) L2S solvers : Costa et al. [16] (TSP only), Sui et al. [17] (TSP only), Wu et al. [39], NLNS [8] (CVRP only), NCE [37] (CVRP only), DACT [9] (state-of-the-art). To ensure fairness, we test their publicly available pre-trained models on our hardware and test datasets. Those marked with ‡ are sourced from their original papers due to difficulties in reproducing, among which we find potential issues marked with # . More implementation details are listed in Appendix E. Following the conventions [5, 9, 34], we report the metrics of objective values and optimality gaps averaged on a test dataset with 10k instances, where the total run time is measured under the premise of using one GPU for neural methods and one CPU for traditional ones. The gaps are computed w.r.t. the exact solver Concorde [54] for TSP and the state-of-the-art traditional solver HGS [21] for CVRP. We also include the LKH [20, 51] as baselines. However, we note that it is hard to be absolutely fair when comparing the run time between those CPU-based traditional solvers and GPU-based neural solvers. The baselines are grouped, where the last group comprises variations of our NeuOpt, differentiated by the number of augments (marked as 'D2A=') and the number of inference steps (marked as 'T=').

TSP results. Compared to L2P solvers , NeuOpt (D2A=1, T=1k) surpasses GCN+BS, CVAE-OptDE, and GNN+GLS in all problem sizes with shorter run time. With increased T, NeuOpt continues reducing the gaps, and outshines the state-of-the-art DIFUSCO solver with less time at T=10k steps. The NeuOpt (D2A=5, T=1k), with more augmentations, shows lower gaps than NeuOpt (D2A=1, T=5k) in the same solution search count, where it achieves the lowest gap of 0.00% at (D2A=5, T=5k) on all sizes. Despite the longer run time compared to DPDP and Att-GCN+MCTS, their high efficiency is limited to TSP, and our NeuOpt could be potentially boosted by leveraging heatmaps

# We found issues with their code, causing an underestimation of their reported gaps by roughly 0.02% (TSP-50) and 0.04% (TSP-100). Thus our reproduced gaps may vary from those in their papers. For not reproduced ones, we use ( · ) to indicate the underestimated values sourced directly from their papers.

‡ The results with ' ≈ ' are based on their original papers since their code is either i) not publicly available (e.g., AM+CLP for CVRP), ii) raises (incompatible) errors that are difficult to fix on our hardware (e.g., Att-GCN+MCTS), or iii) requires long computation time to reproduce (e.g., CV AE-Opt-DE).

Note : The abbreviations refer to: A - Augmentation, D2A - Dynamic Data Augmentation, BS - Beam Search, H - Heuristics, MCTS - Monte Carlo Tree Search, DE - Differential Evolution, DP - Dynamic Programming, GLS - Guided Local Search, AS - Active Search, AS+M+M - AS+MCTS+Meta-Learning.

similar to theirs. As for L2C solvers , NeuOpt (D2A=5, T=3k) beats all baselines on TSP-100 with less time, including the state-of-the-art POMO+EAS+SGBS solver. When pitted against L2S solvers which ours also belongs to, NeuOpt is able to at least halve their gaps using a much shorter run time, demonstrating a substantial improvement. Lastly, our NeuOpt (D2A=5, T=5k) is one of the neural solvers (including DPDP) that can solve TSP-100 to near-optimal (i.e., 0.00%) in a reasonable time.

CVRP results. Most L2P solvers fail to handle CVRP. The remaining ones, CVAE-Opt-DE and DPDP, are significantly outstripped by our solver (D2A=5, T=6k) with less run time. For L2C solvers , our NeuOpt-GIRE (D2A=1, T=10k) could exceed the previous state-of-the-art POMO solver. Furthermore, with (D2A=5, T=6k) and (D2A=5, T=40k), our method surpasses POMO+EAS and POMO+EAS+SGBS (long) with less run time, respectively, even though they equip with extra posthoc per-instance processing boosters. We note that such post-processing strategies (e.g., per-instance active search) could potentially be integrated into ours for further performance gains. Compared to L2S solvers , due to the advances of our GIRE, even NeuOpt-GIRE (D2A=1, T=5k) could reduce the gaps of those (masking-based) L2S solvers by an order of magnitude in much less time on CVRP-100. Notably, we are the first L2S solver to surpass the strong LKH-3 solver on CVRP.

Base

+VI

+ES

VI+ES

6.1625

6.1650

6.1675

6.1700

6.1725

Obj. Value

RS

No

Yes

Effectiveness analysis. To further reveal the effectiveness of our RDS decoder and GIRE, we conduct a comparison with the state-of-the-art L2S solver, DACT [9]. We upgraded it to DACT-U for fairness, which uses identical embeddings and encoders as our NeuOpt. The evaluation criteria include the model size, the best objective value (averaged over four training runs with T=1k inference steps on 1k validation instances), the gaps w.r.t. the best solver, and the run time. As evidenced in Table 2, our GIRE scheme consistently boosts both NeuOpt and DACT-U on CVRP-20, reducing the mask computation time while marginally increasing model size, which reveals the generality of our GIRE. Moreover, when all are used for 2-opt, NeuOpt ( K =2 , for TSP) and NeuOpt-GIRE ( K =2 , for CVRP) surpass DACT-U (for TSP) and DACT-U-GIRE (for CVRP), respectively, with less run time. This is further amplified by increasing K to 4 for NeuOpt at the cost of slightly increased run time.

6.2 Ablation studies

Ablation on RDS decoder. In Table 3, we remove the key components of our RDS decoder including the GRUs, the move stream µ , and edge stream λ , respectively, and report the model size and the best objective value averaged over four training runs. As depicted, all components largely contribute to effective contextual modeling for parameterizing the k-opt decoder on both TSP-100 and CVRP-20.

Ablation on D2A inference. In Table 4, we gather the gaps (similar to Table 1) with and without our D2A design when a single data augmentation is leveraged during inference with T=5K and T=10K. The results confirm that our D2A consistently enhances the inference on both TSP-100 and CVRP-100.

Ablation on GIRE designs. We examine scenarios, considering the inclusion or exclusion of Violation Indicator (VI), Exploration Statistics (ES), and Reward Shaping (RS), using eight combinations of them during the training of NeuOpt on CVRP-20. We draw the box plots of the best objective values achieved across eight training runs for each scenario in Figure 4. We can conclude that: 1) the supplement of VI consistently enhances training outcomes, irrespective of the presence of RS; 2) the supplement of ES may bring improvement only when RS is present; and 3) our complete GIRE design (VI+ES+RS) yields the lowest objective values and demonstrates superior stability.

Ablation on basis moves. In Figure 5 (a), we depict the training curves of NeuOpt on TSP-100 with and without the learnable S-move. When removing the learnable S-move and opting for

0

20

40

60

80

100

7.75

8.00

8.25

8.50

Obj. Value

w/o-S-move

Ours

(a) Effects of S-move.

K=2

K=4

K=5

K=6

7.80

7.83

7.85

7.88

Obj. Value

w/o-E-move

Ours

(b) Effects of E-move.

a random selection of the anchor node x a , a significant degradation in the final performance can be observed, underscoring the necessity of the learnable S-move design. Meanwhile, Figure 5 (b)

0.01

0.05

0.1

6.1625

6.1650

6.1675

Obj. Value

Varying Beta (Alpha=0.05)

presents the pointplots with confidence intervals that show the performance of our NeuOpt on TSP100 with and without E-move during decoding across varying preset K . When E-move is absent (dotted blue line), the model, downgraded to performing fixed K -opt only, exhibits diminished performance on larger K . Conversely, our NeuOpt (solid orange line) could further benefit from a larger K , due to its flexibility in determining and combining different k across search steps.

6.3 Generalization and scalability studies

In Table 5 and Table 6, we further evaluate the generalization capability of our NeuOpt models on more complex instances from TSPLIB [55] and CVRPLIB [56], respectively. Our NeuOpt achieves lower gaps than DACT [9] and L2C solvers (AM [3] and POMO [4]), and even shows superiority over the AMDKD method [11] that is explicitly designed to boost the generalization of POMO through knowledge distillation. Beyond generalization, our NeuOpt also exhibits notable scalability. As shown in Table 7, when trained directly for size 200, NeuOpt finds close-to-optimal solutions and still surpasses the strong LKH-3 solver [20] on CVRP-200. Note that existing L2S solvers, e.g., DACT [9] may struggle with training on such scales. More details and discussions are available in Appendix E.

6.4 Hyper-parameter studies

Influence of preset K . In Table 8, we display the performance of NeuOpt on TSP-100, as well as the corresponding inference time (T=1k) and the training time (per epoch) for varying K values. The results highlight trade-offs between better performance and increased computational costs.

Influence of GIRE hyper-parameters. Figure 6 depicts the influence of α and β in Eq. (4) on CVRP-20, where we fix one while varying the other, investigating both extremely smaller (0.01) and larger (0.1) values. The results suggest that more effective reward shaping occurs when the weights are moderate. Please refer to Appendix D for discussions on more GIRE hyper-parameters.

7 Conclusions and limitations

In this paper, we introduce NeuOpt, a novel L2S solver for VRPs, that performs flexible k-opt exchanges with a tailored formulation and a designed RDS decoder. We also present GIRE, the first scheme to transcend masking for constraint handling based on feature supplement and reward shaping, enabling autonomous exploration in both feasible and infeasible regions. Moreover, we devise a D2A augmentation method to boost inference diversity. Despite delivering state-of-the-art results, our work still has limitations . While NeuOpt exhibits better scalability than existing L2S solvers, it falls short against some L2P solvers (e.g., [6, 7]) for larger-scale TSPs. Possible solutions in future works include: 1) integrating divide-and-conquer strategies as per [6, 57-60], 2) reducing search space via heatmaps as predicted in [6, 15], 3) adopting more scalable encoders [61, 62], and 4) refactoring our code with highly-optimized CUDA libraries as did in [7, 15]. Besides enhancing scalability, future works can also focus on: 1) applying our GIRE to more VRP constraints and even beyond L2S solvers, 2) integrating our method with post-hoc per-instance processing boosters (e.g., EAS [5]) for better performance, and 3) enhancing the generalization capability of our NeuOpt on instances with different sizes/distributions (e.g., by leveraging the frameworks in [11, 63]).

Acknowledgments and Disclosure of Funding

This research was supported by the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant.

References

Paolo Toth and Daniele Vigo. Vehicle routing: problems, methods, and applications . SIAM press, 2014.

Cong Zhang, Yaoxin Wu, Yining Ma, Wen Song, Zhang Le, Zhiguang Cao, and Jie Zhang. A review on learning to solve combinatorial optimisation problems in manufacturing. IET Collaborative Intelligent Manufacturing , 5(1):e12072, 2023.

Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In International Conference on Learning Representations , 2018.

Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. POMO: Policy optimization with multiple optima for reinforcement learning. In Advances in Neural Information Processing Systems , volume 33, pages 21188-21198, 2020.

André Hottung, Yeong-Dae Kwon, and Kevin Tierney. Efficient active search for combinatorial optimization problems. In International Conference on Learning Representations , 2022.

Zhang-Hua Fu, Kai-Bin Qiu, and Hongyuan Zha. Generalize a small pre-trained model to arbitrarily large TSP instances. In AAAI Conference on Artificial Intelligence , 2021.

Ruizhong Qiu, Zhiqing Sun, and Yiming Yang. DIMES: A differentiable meta solver for combinatorial optimization problems. In Advances in Neural Information Processing Systems , volume 35, pages 2553125546, 2022.

André Hottung and Kevin Tierney. Neural large neighborhood search for routing problems. Artificial Intelligence , page 103786, 2022.

Yining Ma, Jingwen Li, Zhiguang Cao, Wen Song, Le Zhang, Zhenghua Chen, and Jing Tang. Learning to iteratively solve routing problems with dual-aspect collaborative transformer. In Advances in Neural Information Processing Systems , volume 34, pages 11096-11107, 2021.

Qingfu Zhang Xi Lin, Zhiyuan Yang. Pareto set learning for neural multi-objective combinatorial optimization. In International Conference on Learning Representations , 2022.

Jieyi Bi, Yining Ma, Jiahai Wang, Zhiguang Cao, Jinbiao Chen, Yuan Sun, and Yeow Meng Chee. Learning generalizable models for vehicle routing problems via knowledge distillation. In Advances in Neural Information Processing Systems , pages 31226-31238, 2022.

Yining Ma, Jingwen Li, Zhiguang Cao, Wen Song, Hongliang Guo, Yuejiao Gong, and Yeow Meng Chee. Efficient neural neighborhood search for pickup and delivery problems. In International Joint Conference on Artificial Intelligence , pages 4776-4784, 2022.

Minsu Kim, Junyoung Park, and Jinkyoo Park. Sym-nco: Leveraging symmetricity for neural combinatorial optimization. In Advances in Neural Information Processing Systems , volume 35, pages 1936-1949, 2022.

Chaitanya K Joshi, Thomas Laurent, and Xavier Bresson. An efficient graph convolutional network technique for the travelling salesman problem. arxiv preprint arxiv:1906.01227, ArXiV, 2019.

Zhiqing Sun and Yiming Yang. Difusco: Graph-based diffusion solvers for combinatorial optimization. arxiv preprint arxiv:2302.08224, ArXiV, 2023.

Paulo da Costa, Jason Rhuggenaath, Yingqian Zhang, and Alp Eren Akçay. Learning 2-opt heuristics for the traveling salesman problem via deep reinforcement learning. In Asian Conference on Machine Learning , pages 465-480, 2020.

Jingyan Sui, Shizhe Ding, Ruizhi Liu, Liming Xu, and Dongbo Bu. Learning 3-opt heuristics for traveling salesman problem via deep reinforcement learning. In Asian Conference on Machine Learning , volume 157, pages 1301-1316, 2021.

Zbigniew Michalewicz et al. Do not kill unfeasible individuals. In Proceedings of the Fourth Intelligent Information Systems Workshop , pages 110-123, 1995.

Fred Glover and Jin-Kao Hao. The case for strategic oscillation. Annals of Operations Research , 183: 163-173, 2011.

Keld Helsgaun. LKH-3 (3.0.7), 2017. URL http://webhotel4.ruc.dk/~keld/research/LKH-3/ .

Thibaut Vidal. Hybrid genetic search for the cvrp: Open-source implementation and swap* neighborhood. Computers & Operations Research , 140:105643, 2022.

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems , volume 28, pages 2692-2700, 2015.

Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. In International Conference on Machine Learning (Workshop) , 2017.

Mohammadreza Nazari, Afshin Oroojlooy, Martin Takáˇ c, and Lawrence V Snyder. Reinforcement learning for solving the vehicle routing problem. In Advances in Neural Information Processing Systems , pages 9861-9871, 2018.

Hanjun Dai, Elias B Khalil, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. In Advances in Neural Information Processing Systems , pages 6351-6361, 2017.

Iddo Drori, Anant Kharkar, William R Sickinger, Brandon Kates, Qiang Ma, Suwen Ge, Eden Dolev, Brenda Dietrich, David P Williamson, and Madeleine Udell. Learning to solve combinatorial optimization problems on real-world graphs in linear time. In International Conference on Machine Learning and Applications (ICMLA) , pages 19-24, 2020.

Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Step-wise deep learning models for solving routing problems. IEEE Transactions on Industrial Informatics , 17(7):4861-4871, 2020.

Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Multi-decoder attention model with embedding glimpse for solving vehicle routing problems. In AAAI Conference on Artificial Intelligence , pages 12042-12049, 2021.

Yunqiu Xu, Meng Fang, Ling Chen, Gangyan Xu, Yali Du, and Chengqi Zhang. Reinforcement learning with multiple relational attention for solving vehicle routing problems. IEEE Transactions on Cybernetics , 52(10):11107 - 11120, 2022.

Jingwen Li, Liang Xin, Zhiguang Cao, Andrew Lim, Wen Song, and Jie Zhang. Heterogeneous attentions for solving pickup and delivery problem via deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems , 23(3):2306-2315, 2022.

Jingwen Li, Yining Ma, Ruize Gao, Zhiguang Cao, Andrew Lim, Wen Song, and Jie Zhang. Deep reinforcement learning for solving the heterogeneous capacitated vehicle routing problem. IEEE Transactions on Cybernetics , 52(12):13572-13585, 2022.

Yan Jin, Yuandong Ding, Xuanhao Pan, Kun He, Li Zhao, Tao Qin, Lei Song, and Jiang Bian. Pointerformer: Deep reinforced multi-pointer transformer for the traveling salesman problem. In AAAI Conference on Artificial Intelligence , 2023.

Minsu Kim, Jinkyoo Park, and Joungho kim. Learning collaborative policies to solve np-hard routing problems. In Advances in Neural Information Processing Systems , volume 34, pages 10418-10430, 2021.

Jinho Choo, Yeong-Dae Kwon, Jihoon Kim, Jeongwoo Jae, André Hottung, Kevin Tierney, and Youngjune Gwon. Simulation-guided beam search for neural combinatorial optimization. In Advances in Neural Information Processing Systems , volume 35, pages 8760-8772, 2022.

Xinyun Chen and Yuandong Tian. Learning to perform local rewriting for combinatorial optimization. In Advances in Neural Information Processing Systems , volume 32, pages 6281-6292, 2019.

Hao Lu, Xingwen Zhang, and Shuang Yang. A learning-based iterative method for solving vehicle routing problems. In International Conference on Learning Representations , 2019.

Minjun Kim, Junyoung Park, and Jinkyoo Park. Learning to CROSS exchange to solve min-max vehicle routing problems. In International Conference on Learning Representations , 2023.

Grigorios D Konstantakopoulos, Sotiris P Gayialis, and Evripidis P Kechagias. Vehicle routing problem and related algorithms for logistics distribution: A literature review and classification. Operational Research , pages 2033-2062, 2020.

Yaoxin Wu, Wen Song, Zhiguang Cao, Jie Zhang, and Andrew Lim. Learning improvement heuristics for solving routing problems. IEEE Transactions on Neural Networks and Learning Systems , 33(9):5057-5069, 2021.

Benjamin Hudson, Qingbiao Li, Matthew Malencia, and Amanda Prorok. Graph neural network guided local search for the traveling salesperson problem. In International Conference on Learning Representations , volume 35, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265, 2015.

Wouter Kool, Herke van Hoof, Joaquim Gromicho, and Max Welling. Deep policy dynamic programming for vehicle routing problems. In International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research , pages 190-213, 2022.

André Hottung, Bhanu Bhandari, and Kevin Tierney. Learning a latent search space for routing problems using variational autoencoders. In International Conference on Learning Representations , 2021.

Jiuxia Zhao, Minjia Mao, Xi Zhao, and Jianhua Zou. A hybrid of deep reinforcement learning and local search for the vehicle routing problems. IEEE Transactions on Intelligent Transportation Systems , 22(11): 7208-7218, 2021.

Renchi Zhang, Runsheng Yu, and Wei Xia. Constraint-aware policy optimization to solve the vehicle routing problem with time windows. Information Technology and Control , 51(1):126-138, 2022.

Hang Zhao, Qijin She, Chenyang Zhu, Yin Yang, and Kai Xu. Online 3d bin packing with constrained deep reinforcement learning. In AAAI Conference on Artificial Intelligence , pages 741-749, 2021.

Ashish K Jayant and Shalabh Bhatnagar. Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems , 35:2443224445, 2022.

Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Xueqian Wang, Bo Yuan, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. In International Joint Conference on Artificial Intelligence , pages 3744-3750, 2022.

Shen Lin and Brian W Kernighan. An effective heuristic algorithm for the traveling-salesman problem. Operations research , 21(2):498-516, 1973.

Keld Helsgaun. An effective implementation of the lin-kernighan traveling salesman heuristic. European journal of operational research , 126(1):106-130, 2000.

Keld Helsgaun. LKH-2 (2.0.9), 2018. URL http://webhotel4.ruc.dk/~keld/research/LKH/ .

Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1724-1734, 2014.

David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations , 2017.

David L Applegate, Robert E Bixby, Vašek Chvátal, and William J Cook. Concorde TSP Solver, 2020. URL http://www.math.uwaterloo.ca/tsp/concorde/ .

Gerhard Reinelt. TSPLIB-A traveling salesman problem library. ORSA journal on computing , 3(4): 376-384, 1991.

Eduardo Uchoa, Diego Pecin, Artur Pessoa, Marcus Poggi, Thibaut Vidal, and Anand Subramanian. New benchmark instances for the capacitated vehicle routing problem. European Journal of Operational Research , 257(3):845-858, 2017.

Qingchun Hou, Jingwei Yang, Yiqiang Su, Xiaoqing Wang, and Yuming Deng. Generalize learned heuristics to solve large-scale vehicle routing problems in real-time. In International Conference on Learning Representations , 2023.

Hanni Cheng, Haosi Zheng, Ya Cong, Weihao Jiang, and Shiliang Pu. Select and optimize: Learning to solve large-scale tsp instances. In International Conference on Artificial Intelligence and Statistics , pages 1219-1231, 2023.

Xuanhao Pan, Yan Jin, Yuandong Ding, Mingxiao Feng, Li Zhao, Lei Song, and Jiang Bian. H-TSP: Hierarchically solving the large-scale travelling salesman problem. In AAAI Conference on Artificial Intelligence , 2023.

Sirui Li, Zhongxia Yan, and Cathy Wu. Learning to delegate for large-scale vehicle routing. Advances in Neural Information Processing Systems , 34:26198-26211, 2021.

Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. In Advances in Neural Information Processing Systems , volume 35, pages 14501-14515, 2022.

Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, LUKASZ KAISER, Wojciech Gajewski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers. In Advances in Neural Information Processing Systems , volume 34, pages 9895-9907, 2021.

Jianan Zhou, Yaoxin Wu, Wen Song, Zhiguang Cao, and Jie Zhang. Towards omni-generalizable neural methods for vehicle routing problems. In International Conference on Machine Learning , 2023.

Learning to Search Feasible and Infeasible Regions of Routing Problems with Flexible Neural k-Opt (Appendix)

A Action factorization examples

Figure 7 depicts examples of our factorization method using combinations of basis moves (S-move, Imove, and E-move) to represent different k-opt exchanges. Initiated from a TSP-9 instance (leftmost), we list examples from 1-opt to 4-opt, where the number of I-move corresponds to varying k values, leading to distinct new solutions (rightmost). This demonstrates the flexibility of our factorization.

𝒙𝟖

𝒙𝟕

𝒙𝟔

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟗

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

1-opt:

[𝑺 𝒙𝟐

, 𝑬

𝒙𝐧𝐮𝐥𝐥

]

𝑺 𝒙𝟐

:

𝒆 𝒐𝒖𝒕 (𝒙𝟐

→

𝒙𝟑)

𝑬 𝒙𝐧𝐮𝐥𝐥

:

𝒆 𝒊𝒏 (𝒙𝟐

→

𝒙𝟑)

2-opt:

[𝑺 𝒙𝟐

, 𝑰 𝒙𝟒

, 𝑬

𝒙𝐧𝐮𝐥𝐥

]

𝑺 𝒙𝟐

:

𝒆 𝒐𝒖𝒕 (𝒙𝟐

→

𝒙𝟑)

𝑰 𝒙𝟒

:

𝒆 𝒊𝒏 (𝒙𝟐

→

𝒙𝟒)

𝑰 𝒙𝟒

:

𝒆 𝒐𝒖𝒕 (𝒙𝟒

→

𝒙𝟓)

𝑬 𝒙𝐧𝐮𝐥𝐥

:

𝒆 𝒊𝒏 (𝒙𝟑

→

𝒙𝟓)

𝒙𝟖

𝒙𝟕

𝒙𝟔

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟗

3-opt:

[𝑺 𝒙𝟐

, 𝑰 𝒙𝟒

, 𝑰 𝒙𝟕

, 𝑬

𝒙𝐧𝐮𝐥𝐥

]

𝑺 𝒙𝟐

:

𝒆 𝒐𝒖𝒕 (𝒙𝟐

→

𝒙𝟑)

𝑰 𝒙𝟒

:

𝒆 𝒊𝒏 (𝒙𝟐

→

𝒙𝟒)

𝑰 𝒙𝟒

:

𝒆 𝒐𝒖𝒕 (𝒙𝟒

→

𝒙𝟓)

𝑰 𝒙𝟕

:

𝒆 𝒊𝒏 (𝒙𝟑

→

𝒙𝟕)

𝑰 𝒙𝟕

:

𝒆 𝒐𝒖𝒕 (𝒙𝟕

→

𝒙𝟖)

𝑬 𝒙𝐧𝐮𝐥𝐥

:

𝒆 𝒊𝒏 (𝒙𝟓

→

𝒙𝟖)

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝟔

𝒙𝟕

𝒙𝟖

𝒙𝟗

𝑺 𝒙𝟐

:

𝒆 𝒐𝒖𝒕 (𝒙𝟐

→

𝒙𝟑)

𝑰 𝒙𝟒

:

𝒆 𝒊𝒏 (𝒙𝟐

→

𝒙𝟒)

𝑰 𝒙𝟒

:

𝒆 𝒐𝒖𝒕 (𝒙𝟒

→

𝒙𝟓)

𝑰 𝒙𝟕

:

𝒆 𝒊𝒏 (𝒙𝟑

→

𝒙𝟕)

𝑰 𝒙𝟗

:

𝒆 𝒐𝒖𝒕 (𝒙𝟗

→

𝒙𝟏)

𝑬 𝒙𝐧𝐮𝐥𝐥

:

𝒆 𝒊𝒏 (𝒙𝟖

→

𝒙𝟏)

𝑰 𝒙𝟕

:

𝒆 𝒐𝒖𝒕 (𝒙𝟕

→

𝒙𝟖)

𝑰 𝒙𝟗

:

𝒆 𝒊𝒏 (𝒙𝟓

→

𝒙𝟗)

[𝑺 𝒙𝟐

, 𝑰 𝒙𝟒

, 𝑰 𝒙𝟕

, 𝑰 𝒙𝟗

, 𝑬

𝒙𝐧𝐮𝐥𝐥

]

4-opt:

B NeuOpt encoder

Given a state s t = { G , τ t , τ bsf t } , where G is the current instance, τ t is the current solution, and τ bsf t is the best-so-far solution before step t , the encoding process first translates the raw features of state into node embeddings. These embeddings are subsequently refined through L = 3 stacked encoders.

Fearure Embedding. Building upon the dual-aspect representation design from [12], we embed state features 3 , denoted as ψ [ G , τ t ] = {{ φ t i } x i ∈V , { p t i } x i ∈V } into two separate sets of node embeddings: Node Feature Embeddings (NFEs) to encode ψ ( G )= { φ t i } and Positional Feature Embeddings (PFEs) to encode ψ ( τ t )= { p t i } . The NFEs , denoted as { h init i } x i ∈V , are a set of d -dimensional vectors, each of which embeds d h -dimensional problem-specific raw features φ t i of node x i at step t . To obtain h init i , we upgrade linear projection used in [12] into the MLP with structure of ( d h × d 2 × d ) for enhanced representation. The PFEs , denoted as { g init i } x i ∈V , are a set of d -dimensional vectors, each of which embeds positional features p t i of x i at step t , derived from the position of x i in the current solution τ t . Following [9], we employ the Cyclic Positional Encoding (CPE) to generate a series of cyclic embeddings in a d -dimensional space. These CPE embeddings are then used to initialize g init i correspondingly, so as to capture the topological structure of the nodes in the current solution τ t .

Problem-specific raw node features φ t i . For TSP , node features φ t i of x i contains its twodimensional coordinates (i.e., d h = 2 ); For CVRP , φ t i contains six features (i.e., d h =6 ) including 1-2) its two-dimensional coordinates, 3) the demand of node x i , 4) the sum of demand of the corresponding sub-tour before node x i (inclusive), 5) the sum of demand of the corresponding sub-tour after x i (exclusive), and 6) an indicator function to signify whether node x i is a customer node. When GIRE is applied, we enrich φ t i with two Violation Indicator (VI) features, i.e., d h =8 .

Stacked encoders. Following the encoders in [9, 12], we use Transformer-styled encoders with Synthesis Attention (Synth-Att) to refine the embeddings { h init i } x i ∈ V and { g init i } x i ∈ V , where PFEs serve as auxiliary embeddings that bolster the representation learning of NFEs. After the encoding, a unified set of embeddings { h i } x i ∈V is obtained, which is then inputted in our recurrent dual-stream (RDS) decoder for k-opt action decoding (see Section 4).

C Training and inference algorithms

C.1 Training algorithm

Algorithm 1 Reinforcement learning algorithms for NeuOpt

Input : policy network π θ , critic network v ϕ , PPO objective clipping threshold ϑ , learning rate η θ , η ϕ , learning rate decay rate ς , number of PPO inner loops Ω , training steps T train, number of epochs E , B ξ

number of batches per epoch , curriculum learning (CL) scalar 1: for epoch = 1 to E do 2: for batch = 1 to B do 3: Randomly generate a batch of training instances D = {G i } and their initial solutions { τ i } ; 4: CL : Improve { τ i } to { τ ′ i } by runing the current policy π θ for T = epoch/ξ steps; 5: Get initial state s 0 based on { τ ′ i } for each instance in the current batch; 6: t ← 0 ; 7: while t < T train do 8: Run policy π θ on each instance and get { ( s t ′ ,a t ′ ,r t ′ ) } t + n -1 t ′ = t where a t ′ ∼ π θ ( a t ′ | s t ′ ) ; 9: t ← t + n , π old ← π θ , v old ← v ϕ ; 10: for j = 1 to Ω do 11: ˆ R t = v ϕ ( s t ) ; 12: for t ′ ∈ { t -1 , ..., t -n } do 13: ˆ R t ′ ← r t ′ + γ ˆ R t ′ +1 ; 14: ˆ A t ′ ← ˆ R t ′ -v ϕ ( s t ′ ) ; 15: end for 16: Compute RL loss L PPO θ using Eq. (6) and critic loss L Critic ϕ using Eq. (7); 17: θ ← θ + η θ ∇L PPO θ ; ϕ ← ϕ -η ϕ ∇L Critic ϕ ; 18: end for 19: end while 20: end for 21: η θ ← ςη θ , η ϕ ← ςη ϕ ; 22: end for

As detailed in Algorithm 1, we adapt the n -step proximal policy optimization (PPO) with curriculum learning (CL) strategy used in [9, 12] to train our NeuOpt. For our GIRE scheme, we consider learning separate critics v origin ϕ , v reg ϕ , and v bonus ϕ to fit the respective reward shaping terms in Eq.(4), so as to better estimate the state values. In light of this, when GIRE is applied, we update Algorithm 1 by: 1) duplicating lines 11, 13-14 to compute ˆ R origin t ′ , ˆ R reg t ′ , and ˆ R bonus t ′ as well as ˆ A origin t ′ , ˆ A reg t ′ , and ˆ A bonus t ′ at the same time; 2) updating the line 16 to replace Eq. (6) and Eq. (7) into Eq. (8) and Eq. (9), respectively; and update line 17 accordingly where all the critics share the same learning rate η ϕ .

C.2 Inference algorithm

In our previous work [12], the data augmentation was incorporated during inference to boost the search diversity. The rationale is that a specific instance, G , can be transformed to different ones, yet still retain the identical optimal solution. These augmented instances can be solved differently by the trained model in parallel, thereby enhancing the diversity of the search for better performance. Specifically, each augmented instance is generated by consecutively executing four predetermined invariant augmentation transformations as listed in Table 9, where the execution order and configurations used for each transformation are randomly determined on the fly following Algorithm 2.

However, such augmentation is performed only once at the start of the inference, making the augmentation fixed across the search process. We thus propose Dynamic Data Augmentation (D2A). It suggests generating new augmented instances with different augmentation configurations each time when the solver fails to find a better solution within a consecutive maximum of T D2A steps (i.e., we consider the search to be trapped in local optima). This allows the model to explicitly solve instances differently once it gets trapped in the local optima, thus promoting an even more diverse search process. Note that the proposed D2A is generic to boost most L2S solvers, and even has the potential to boost L2C solvers when equipped with post-hoc per-instance processing boosters such as EAS [5]. In Algorothm 3, we summarize the D2A procedures where we note that the loops in line 1 and line 7 can be run in parallel during practical implementations. For example, when we use 'D2A=5" as per in Table 1, it means that for each instance G , there are 5 augmented instances {G 1 , G 2 , G 3 , G 4 , G 5 } being solved simultaneously. Each of these instances has its own counter T stall i and the best-so-far solution τ bsf i to track whether the search (for the particular instance G i ) has fallen into local optima.

D Additional discussions on GIRE scheme

Separate critic networks. As per Appendix C, we use separate critics in GIRE. We now introduce their detailed architectures. During the critic value estimation, we first upgrade embeddings { h i }

Algorithm 3 Dynamic Data Augmentation (D2A)

Input : Instance G , policy network π θ , inference step T , number of augments D 2 A , maximum number of consecutive steps allowed before considering the search trapped in local optima T D2A Output : Best solution found during solving all the augmented instances G i

(from the encoders of π θ ) to { ˆ h i } x i ∈V using a vanilla multi-head attention layer. The { ˆ h i } are then fed into a mean-pooling layer [39], yielding compressed representations as follows,

where W ∈ R d × d 2 are trainable matrices, and the mean and max are element-wise operators. All critics v origin ϕ , v reg ϕ , and v bonus ϕ then share these representations { ˆ y i } as parts of their inputs. Specifically, v origin ϕ is computed via a four-layer MLP in Eq. (11) with structure ( d +1 , d , d 2 , 1) which leverages f ( τ bsf t ) as additional features; v reg ϕ is computed via a four-layer MLP in Eq. (12) with structure ( d +10 , d , d 2 , 1) which leverages f ( τ bsf t ) and ES features J t as additional features; and v bonus ϕ is computed via a four-layer MLP in Eq. (13) with structure ( d +1 , d , d 2 , 1) which leverages the best-so-far cost w.r.t. ϵ -F regions f ( τ bsf-wrtϵ t ) as additional features 4 . All MLPs use the ReLU activation function.

Illustrations of extreme search behaviour and our GIRE efficacy. Recall that our GIRE considers reward-shaping terms to encourage search within more promising feasibility boundaries and regulate extreme search behaviours using an entropy measure of the estimated conditional transforming probabilities P t ( U|U ) = P ( τ ′ ∈U| τ ∈U ) and P t ( F|F ) = P ( τ ′ ∈F| τ ∈F ) . Figure 8 depicts the persistence of extreme search behaviour when RS is absent while validating the efficacy of our reward shaping (RS) in GIRE. We conduct training with and without RS, recording the convergence curves of the objective values (with mean and confidence intervals) in Figure 8(a), the detailed objective values per run in Figure 8(b), the estimated probability P t ( F|F ) in Figure 8(c), and the estimated probability P t ( U|U ) in Figure 8(d). As revealed in Figure 8(a), without RS, the runs show unstable convergence and poorer final objective values. This can be attributed to higher P t ( F|F ) and lower P t ( U|U ) as shown in Figure 8(c) and Figure 8(d), indicating biased search preference towards feasible regions

and inefficient exploration in infeasible regions. In the worst case, training may fail to converge to lower objective values due to extremely low P t ( U|U ) (below 0.1 as shown in Figure 8(d)), indicating extreme search behaviour and entrapment in local optima. Conversely, GIRE fosters moderate search behaviours and promotes exploration in infeasible regions (especially for the boundaries), leading to significantly improved training curves with reduced variance and lower objective values.

0

25

50

75

100

6.17

6.18

6.19

6.20

Obj. Value

Ours

Ours-w/o-RS

(a) Objective values.

0

25

50

75

100

6.17

6.18

6.19

6.20

Obj. Value

(b) Objective values.

0

100000

200000

300000

Gradient Step

0.00

0.25

0.50

0.75

1.00

P (feas. to feas.)

(c)

P

t

(

F|F

)

.

0

100000

200000

300000

Gradient Step

0.00

0.25

0.50

0.75

1.00

P (ifeas. to ifeas.)

(d)

P

t

(

U|U

)

.

Influence of ϵ . We employ a deterministic rule to decide ϵ , i.e., ϵ = ζ × N customer × ( ¯ δ/ ∆) , where ζ is a coefficient within [0 , 1] , ∆ is the vehicle capacity, and ¯ δ represents the average customer demands. This formula implies that the total violation corresponds to scenarios where ζ -ratio of customers, each with the average demand, breaches the constraint. Empirically, we set ζ =0 . 1 . Figure 9 illustrates the impact of varying ζ .

0.01

0.05

0.1

0.15

0.2

6.1620

6.1640

6.1660

Obj. Value

Figure 9: Impact of varying

ζ

.

Influence of c 1 and c 2 . Recall that in the entropy measure defined by Eq. (5), we introduce two hyper-parameters c 1 and c 2 to shape the entropy measure pattern H [ P ] , where the measure H [ P ] is used to penalize extreme search behaviours, particularly when values of P approach 0 or 1. Figure 10 illustrates how hyper-parameters c 1 and c 2 influence the shape of H [ P ] . When c 2 is fixed, a smaller c 1 expands the penalty range, while a larger c 1 constricts the penalty range. Similarly, when c 1 is fixed, a smaller or a larger c 2 will also control the shape of the patterns. Empirically, we set the values of c 1 = 0 . 5 , c 2 = 2 . 5 so as to only penalize extreme search behaviour if the feasibility transition probability is outside the [0 . 25 , 0 . 75] range. In Figure 11, we investigate the influence of c 1 , c 2 on the training stability of our NeuOpt-GIRE approach on CVRP-20. Results show that they may not affect training stability (thus no need for extensive tuning in practical usage).

0.00

0.25

0.50

0.75

1.00

Probability

0.8

0.9

1.0

Entropy measure

(a)

c

1

= 0

.

1

,

c

2

= 2

.

5

0.00

0.25

0.50

0.75

1.00

Probability

0.0

0.5

1.0

Entropy measure

(b)

c

1

= 0

.

5

,

c

2

= 2

.

5

0.00

0.25

0.50

0.75

1.00

Probability

0.0

0.5

1.0

Entropy measure

(c)

c

1

= 1

.

0

,

c

2

= 2

.

5

0.00

0.25

0.50

0.75

1.00

Probability

0.5

1.0

Entropy measure

(d)

c

1

= 0

.

5

,

c

2

= 1

.

5

0.00

0.25

0.50

0.75

1.00

Probability

0.0

0.5

1.0

Entropy measure

(e)

c

1

= 0

.

5

,

c

2

= 2

.

5

0.00

0.25

0.50

0.75

1.00

Probability

0.0

0.5

1.0

Entropy measure

(f)

c

1

= 0

.

5

,

c

2

= 3

.

5

0

20

40

60

80

100

6.16

6.18

6.20

Obj. Value

c1=0.5, c2=1.5 (expanded)

c1=0.5, c2=2.5 (original)

c1=0.5, c2=3.5 (constricted)

Applying GIRE to other constraints. Our GIRE can be viewed as an important early attempt that moves beyond the pure feasibility masking to autonomously explore both feasible and infeasible regions during the search. To adapt GIRE for a particular constraint, we suggest the following:

For Feature Supplement , simply binary indicator functions can be employed to discern specific constraint violations of each node in the solution, thereby forming the Violation Indicator (VI) features. For example, VI can indicate nodes (customers) in the solution that breach their time window or the pickup/delivery precedence constraints. For the Exploration Statistics (ES) features, they can be retained as originally conceptualized since they are based on historical exploration records, rather than specific constraints.

For Reward Shaping , it can be retained as originally conceptualized. We suggest characterizing the ϵ -feasible regions by a fraction of nodes that do not adhere to the constraints, i.e., ζ =0 . 1 .

E Additional experimental results

E.1 Details of implementation

Hyper-parameter details. We use d =128 , T D2A =10 , T his =25 for NeuOpt, and employ 4 attention heads in both the encoders and critics. The ReLU activation function is utilized for MLPs within the network. In line with the training algorithms in [9, 12], we retain their hyper-parameters to ensure an equivalent training amount. Training involves E =200 epochs and B =20 batches per epoch, with batch sizes of 512 (TSP) and 600 (CVRP). For TSP, we use n =4 , T train =200 ; and for CVRP, we use n =5 , T train =250 . The PPO inner loops is Ω=3 and the clip threshold is ϑ =0 . 1 . The Adam optimizer is employed with learning rates η θ =8 × 10 -5 for π θ , η ϕ =2 × 10 -5 for v ϕ , as well as a decay rate ς =0 . 985 . The reward discount factor γ is set to 0.999 for both problems. Besides, we empirically set approach-specific parameters: the gradient norm of NeuOpt is clipped at 0.05 in all cases, and the curriculum learning scalar ξ is set as 1, 0.5, and 0.25 for sizes 20, 50, and 100, respectively. The training time varies depending on the specific problem and size, e.g., CVRP requires about 4 days for size 20 (1 GPU), 5 days for size 50 (2 GPUs), and 8 days for size 100 (4 GPUs), which are around half the time taken as reported in DACT [9] and POMO [4] on CVRP100.

Setup details. We follow Kool et al. [3] to sample all training and test instance coordinates within the unit square [0 , 1] × [0 , 1] uniformly. For CVRP, the demands of customers are uniformly sampled from the set {1, 2,..., 9} and the vehicle capacity is set to 30, 40, and 50 for sizes 20, 50, and 100 respectively (we thus estimate ¯ δ as 5/30, 5/40, 5/50 in our GIRE, respectively). To facilitate GPU parallelization given the varying lengths of CVRP solutions, we employ the dummy depots design in [9, 39] for length padding, where depots are duplicated during both training and inference. Same as DACT [9], we include 10, 20, and 20 dummy depots for CVRP sizes of 20, 50, and 100 respectively.

Baseline details. We provide details on the compared neural baselines. For the ones with # , we found certain issues in their code, which may lead to the underestimation of reported gaps by around 0.02% (TSP-50) and 0.04% (TSP-100). We observed that they used 'GEO' (Geographical distance) instead of 'EUC' (Euclidean distance) as the type of edge weights while running the Concorde to determine the optimal solutions. This results in the optimal solutions (optimal in the 'GEO' setting) no longer being optimal in the 'EUC' settings, while the objectives derived by neural solvers are based on Euclidean distances. This discrepancy might be the reason for the reported negative optimality gaps on TSP, which seems anomalous as Concorde is guaranteed to produce optimal solutions.

GCN+BS [14] # : a classic L2P solver that predicts heatmaps for TSP based on GCN. We run its star version that considers both beam search (1,280 beams) and shortest tour heuristics.

Att-GCN+MCTS [6] # : an efficient L2P solver that generalizes heatmap-based L2P solvers to larger-scale TSP instances through divide-and-conquer. However, we encountered issues running their GPU-version code, and the CPU-version code appeared to be time-consuming, taking around 28h (10s/instance) for 10k TSP100. We thus opt to use their reported results directly.

GNN+GLS [40]: an insightful L2P solver that uses GNN to predict regrets, thereby directing the traditional local search (relocate and 2-opt) heuristics for TSP. Regrettably, the code is not optimized for batch inference and we encountered some issues loading our datasets when following their instructions. As such, we opt to use their reported results directly.

CVAE-Opt-DE [43]: a generic L2P solver that learns to predict a latent search space for TSP and CVRP. We use their reported DE version results due to the long runtime to reproduce results.

DPDP [42]: a state-of-the-art L2P solver that combines heatmaps with dynamic programming to efficiently solve TSP and CVRP. We run it for 100k iterations (TSP) and 1,000k iterations (CVRP).

DIMES [7] # : a novel L2P solver that learns to predict a latent search space and exploits differentiable optimizers to find TSP solutions. Due to incompatible issues when installing the required Pytorch extension library packages, we opt to directly use their reported results obtained by the settings of using REINFORCE, active search, Monte Carlo tree search, and meta-learning.

DIFUSCO [15] # : a state-of-the-art L2P solver, that learns a diffusion model to predict heatmaps for TSP. We run their used setting: T=50 diffusion steps, S=16 samplings, and 2-opt post-processing.

AM+LCP ⋆ [33]: a two-stage L2C solver that iteratively constructs (seeder) and re-constructs (reviser) parts of the solutions based on the AM [3]. For TSP, we run their optimized version with 1,280 samplings and two revisers - one revises a length of 10 nodes for 25 iterations, and the other revises a length of 20 nodes for 20 iterations. However, their code for CVRP is not publicly available. Hence for CVRP, we opt to directly use their reported results.

Pointformer [32]: a latest L2C solver that enhances POMO with a multi-pointer Transformer and feature augmentation for TSP instances. We run it with 200 samplings and × 8 augmentations.

Sym-NCO [13]: an effective L2C solver that enhances POMO with auxiliary losses for TSP and CVRP, aiming at better symmetry handling. We run it with 200 samplings and × 8 augmentations.

POMO [4]: a renowned L2C solver that enhances the AM [3] with diverse rollouts and data augmentations for TSP and CVRP. We run it with 200 samplings and × 8 augmentations.

POMO+EAS [5]: a leading L2C solver that adjusts a small subset of the pre-trained POMO model parameters on test instances for efficient active search. We run the EAS-lay version with × 8 augmentations for T=200 iterations (both TSP and CVRP) which is the best setting for CVRP.

POMO+EAS+SGBS [34]: a state-of-the-art L2C solver that bolsters EAS by incorporating simulation-guided beam search, executing beam search and efficient active search phases alternately. We run 5 rounds (TSP) and 50 rounds (CVRP) for the short version, and 20 rounds (TSP) and 200 rounds (CVRP) for the long version. The beam search width is set as per their suggested values.

Costa et al. [16]: a compelling L2S solver that employs RNN-based policy to control the 2-opt for solving TSP. We run it for T=2k iterations.

Sui et al. [17]: a recent L2S solver improves upon [16] by performing 3-opt, where the policy first removes three edges and then chooses a re-connection type from all candidate options. Given that their code is not available, we opt to directly use their reported results.

Wu et al. [39]: a classic L2S solver that employs Transformer-based policy to control the 2-opt for solving TSP. We run it for T=5k iterations (both TSP and CVRP).

DACT [9]: a state-of-the-art L2S solver that enhances Wu et al. [39] via the cyclic positional encoding and dual-aspect representations. Consistent with their original best settings, we run it for 10k iterations with × 4 augmentations for TSP and × 6 augmentations for CVRP.

NLNS [8]: a strong L2S solver that learns to control ruin-and-repair operators. Following their default settings, we run their code on 10 CPUs in parallel, executing T=5k iterations.

NCE [37]: a distinctive L2S solver that is designed to execute CROSSOVER exchanges between two CVRP solutions. As their code is not available, we opt to use their reported results directly.

E.2 Generalization on real-world datasets

We further evaluate the generalization performance of our NeuOpt models, which were trained on TSP100 and CVRP100, by testing them on real-world TSPLIB and CVRPLIB instances containing up to 200 nodes (customers). These real-world instances differ significantly from our training ones

with much larger sizes and different node (customer, depot) distributions. We compare our NeuOpt with the state-of-the-art L2S solver, DACT [9], reporting both the best and average gaps across 10 independent runs. We also compare with the results from a recent work [11] that focused on enhancing the generalization capabilities of existing L2C solvers, where we include AM-mix, POMOmix, and AMDKD (POMO) as baselines. According to Bi et al. [11], AM-mix and POMO-mix are enhanced AM and POMO models, respectively, which were trained on a combination of uniform, cluster, and a mix of uniform and cluster instances to bolster their generalization performance. The AMDKDapproach leverages adaptive multi-distribution knowledge distillation to glean various forms of knowledge from multiple teachers trained on exemplar distributions, so as to yield a lightweight yet highly adaptable student POMO model. Since our primary focus is on the generalization capability of the deep model itself, we do not consider post-hoc per-instance processing boosters (e.g., EAS [5]). The results for TSPLIB and CVRPLIB are presented in Table 10 and Table 11, respectively.

TSPLIB results. As depicted in Table 10, our NeuOpt model consistently yields lower average gaps than the state-of-the-art DACT solver when both are allowed to explore 10k solutions. Meanwhile, our NeuOpt exploring 25k solutions could even surpass the DACT exploring 40k solutions. When compared to L2C solvers, NeuOpt yields lower gaps than AM-mix and POMO-mix, and even the AMDKD (POMO) model that is explicitly designed to enhance cross-distribution generalization performance. These results indicate the desired generalization capabilities of our NeuOpt model.

CVRPLIB results. As depicted in Table 11, our NeuOpt could still deliver the lowest average gaps on CVRPLIB instances compared to other baselines. Despite the promising results, we acknowledge that there is still room for improvement in enhancing the generalization of L2S solvers (including our NeuOpt), which is akin to the efforts made by AMDKD in advancing L2C solvers. Meanwhile, as discussed in Section 7, the integration of post-hoc per-instance processing boosters (e.g., EAS [5]) could potentially further improve both the in-distribution and cross-distribution generalization performance of existing L2S solvers. We consider the above ideas as important directions for future research.

E.3 Results on TSP-200 and CVRP-200

We carry out experiments on larger scales, i.e., training our NeuOpt on TSP200 and CVRP200. For the size-specific hyper-parameters, we use ξ =0 . 125 for both problems; and use the CVRP problem settings: capacity ∆=70 and 30 dummy depots. All other hyper-parameters remain unchanged. The performance of our NeuOpt models for TSP-200 and CVRP-200 are depicted in Table 12, where we compare our models with traditional heuristics (as per Table 1). The results indicate that our models consistently find close-to-optimal solutions, even outperforming the LKH3 solver on CVRP-200. We note that due to prohibited longer training times, existing L2S solvers (like DACT [9]) may not be efficient for training on size 200, which underscores the better scalability of our approach.

E.4 Inference with multiple GPUs

Our implementation allows for accelerated inference using multiple GPUs. Recall that in Table 1, the recorded time is for solving a total of 10,000 instances on a single GPU. For practical applications, users can achieve significantly shorter runtimes using multiple GPUs, as illustrated in Table 13.

E.5 Visualization of exploration behaviour

Figure 12 presents the objective values (search trajectories) of the visited feasible or infeasible solutions within the first 50 search steps, when using a pre-trained NeuOpt-GIRE model to solve five randomly generated CVRP-20 instances, respectively. The blue line depicts the objective values of each visited solution where we use green dots to mark feasible solutions and use red crosses to mark infeasible ones. The red and green dotted lines represent the best-so-far infeasible and feasible objective values, respectively. As observed from the figure, our learned model exhibits a propensity to explore infeasible regions at the beginning of the search, so as to rapidly decrease the objective values. It then learns to correct the constraint violations on the found best-so-far infeasible solutions and begins exploring the feasible regions, leading to new best-so-far feasible solutions more

efficiently. Moreover, there is a trend of alternating searches between feasible and infeasible regions. This behaviour aligns with our motivation in Section 5, suggesting that exploring infeasible regions could foster shortcut discovery, promising boundary searches, and identification of possibly isolated feasible regions. This showcases the significance and effectiveness of our GIRE method.

0

10

20

30

40

50

6

8

10

12

0

10

20

30

40

50

6

8

10

12

0

10

20

30

40

50

8

10

0

10

20

30

40

50

5.0

7.5

10.0

12.5

15.0

0

10

20

30

40

50

6

8

10

12

In Figure 13, we further visualize the estimated distribution of the number of infeasible solutions within the last 5 visited solutions before finding a new best-so-far feasible solution. Results suggest that the probability of encountering at least one infeasible solution before finding a better best-so-far solution is around 80%, showcasing the importance of exploring both feasible and infeasible regions by GIRE (compared to purely visiting feasible solutions in the masking scheme).

T in [0.0k,0.2k)

T in [0.2k,0.4k)

T in [0.4k,0.6k)

T in [0.6k,0.8k)

T in [0.8k,1.0k)

0.11

0.07

0.05

0.05

0.06

0.15

0.12

0.11

0.16

0.10

0.19

0.19

0.20

0.16

0.17

0.22

0.21

0.24

0.18

0.25

0.20

0.22

0.21

0.23

0.21

0.14

0.19

0.20

0.21

0.19

P(#ifeasi. is 5/5)

P(#ifeasi. is 4/5)

P(#ifeasi. is 3/5)

P(#ifeasi. is 2/5)

P(#ifeasi. is 1/5)

P(#ifeasi. is 0/5)

E.6 Used assets and licenses

We list the used assets in Table 14. All of them are open-source for academic research use. Our code and pre-trained models have been made available on GitHub ( https://github.com/yining043/ NeuOpt ) using the MIT License.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a learning-to-search (L2S) solver called Neural k-Opt (NeuOpt) for routing problems. NeuOpt learns to perform flexible k-opt exchanges using a tailored action factorization method and a customized recurrent dual-stream decoder. The paper also proposes the Guided Infeasible Region Exploration (GIRE) scheme, which allows the autonomous exploration of both feasible and infeasible regions. The experiments on TSP and CVRP with up to 100 nodes show that NeuOpt could outperform other learning-based methods slightly. However, the solving speed could be very slow. It also cannot outperform HGS and LKH3 from OR fields. Overall, I think the topic and proposed method are interesting, however, the results are not significant.\n\nSTRENGTHS:\n* The paper is mostly well-written, except that Figure 1 is too complex to help the reviewer to understand the basic ideas of the proposed method. \n* The literature reviews are quite impressive, which include necessary classic and SoTA methods.\n* The proposed NeuOpt and Guided infeasible region exploration seem reasonable.\n* The proposed MDP for k-opt is interesting\n\nWEAKNESSES:\n* The experiments on TSP and CVRP with up to 100 nodes show that NeuOpt could outperform other learning-based methods slightly.  However, the solving speed could be very slow. It also cannot outperform HGS and LKH3 from OR fields. Overall, I think the results are not significant.\n* Given the complexity of the method and the code is not provided, reproducibility could be a problem.\n\nQUESTIONS:\nThe feasibility of solutions is not discussed in the experiments.  To what extent does the Guided Infeasible Region Exploration method impact the feasibility?  The reviewer encourages the authors to evaluate the method on VRPTW or TSPTW to make a more solid benchmark.\n\nLIMITATIONS:\nThe limitations have been discussed in the main body. In Section 7,  the first point \"it falls short against some L2P solvers (e.g., [5, 6]) for larger-scale TSPs\" is a good discussion about the limitation. However, the second and third points are actually the advantages of the paper, rather than the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to learn the k-opt operation, one of the famous local search methods, via neural networks. In particular, the authors model the k-opt operation as a sequential node selection process and use a recurrent dual-stream (RDS) decoder. Furthermore, a guided infeasible region exploration (GIRE) is suggested to encourage the policy to escape local optima.\nThe paper models the k-opt as a sequence of basic moves on open Hamiltonian paths (not cycles), which allows the neural network model easily learns the k-opt operation. Since the end of the sequence (i.e., E-move) is a kind of I-moves, k can be flexibly adjusted according to states. GIRE allows the model to explore infeasible regions, not only feasible regions; it encourages the model to escape local optima. In addition, the portion of infeasible solutions is dynamically adjusted.\n\nSTRENGTHS:\nThe paper is well-written in general, and the key ideas are clearly delivered. The extensive experiments with various baselines suggest that the proposed method can mitigate the inefficiency of L2S solvers.\n\nOne of the main benefits is that we can choose the proper K, the number of DA, and T according to the available computation budget. Also, the results show that NeuOpt gives promising performances with similar time with L2C algorithms.\n\nWEAKNESSES:\nThere is no analysis of where the performances come from. the k-opt operations are well-working with fixed k. Comparisons between the original k-opt and NeuOpt are required.\n\nSome detailed but important information is missing or hard to be found (e.g., encoding scheme, dynamic augmentation, initial tour)\n\nQUESTIONS:\nI have some questions about the details.\n\n1. Effectiveness of learning the S-move. I think that choosing the S-move also gives good performances.\n2. Why is LKH3 (known as better than LKH2) not employed as a baseline?\n3. Whether the encoding is necessary for every t?\n4. How is the problem augmented in detail? I’ve read through the appendix but couldn’t find it.\n\nMinor Comments:\n1. I think further explanation for ES features is required (maybe in the appendix)\n2. For better readability, additional cross-references for contents in the appendix are recommended.\n3. There is no explanation when the other papers are referenced (e.g., line 192, 257). I recommend adding further descriptions.\n\nLIMITATIONS:\nI think the dual-stream structure is restricted to consider additional context information.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose Neural k-Opt (NeuOpt) that factorizes a generic k-opt exchange operation as a series of base operations. They also introduce Guided Infeasible Region Exploration (GIRE) scheme for Capacitated Vehicle Routing Problem (CVRP) where one augments a reward function for RL with signals about exploring infeasible solution spaces. On the model architecture side, the authors propose a recurrent dual-stream decoder that consumes both moves and edges. The authors provided a comprehensive comparison with baseline methods for problems of sizes up to 100 cities.\n\nSTRENGTHS:\n1. I found the action factorization novel and general. A limitation of past k-Opt based learning-to-search methods is that they require a pre-determined value of k. The proposed factorization method can accommodate arbitrary k given an agent enough number of steps. \n\n2. GIRE seems a promising approach to incorporating information about near-feasible regions into explorations. As far as I know, this contribution is novel.\n\n3. The empirical comparisons with baseline methods are comprehensive. I appreciate the authors including an up-to-date list of baseline methods. There are some expected issues with reproductions but I definitely applaud the authors’ efforts here.\n\n4. The paper is relatively well-written. The authors did a good job of packing a dense set of information within the page limit.\n\nWEAKNESSES:\n1. The problem sizes considered are relatively small. Methods like DIFUSCO experimented with sizes up to 10000 cities. It is unclear how practically useful this method is with small problem sizes.\n\n2. Another common evaluation mode for these classes of problems is for the generalization ability of a trained model. That is, extrapolate the model performance on problems of larger sizes than those in the training set. This is not considered in this paper.\n\n3. In general, the empirical results are mixed. On TSP, NeuOpt is not better than DPDP and Att-GCN+MCTS. The comparison with DIFUSCO is obfuscated by a large time discrepancy from the original paper, possibly due to difficulty in reproducing their results. On CVRP, POMO+EAS+SGBS methods are comparable with NeuOpt.\n\n4. The main advantage of the factorized action representation is that one could potentially handle arbitrary k-Opt in a general way. However, in the experiments, the authors used a value of 4 for k which seemed limiting. In addition, in the ablation on values of k, larger k values worked better, so why did the authors not choose a larger value of k for the main evaluations?\n\n5. There are some places where the exposition could be improved. Please see my questions below.\n\nQUESTIONS:\n1. In section 5 about the reward shaping (near line 266), how to compute $P_t(\\mathcal{U}|\\mathcal{U})$ exactly? \n\n2. In the entropy computation, why $P_t(\\mathcal{U}|\\mathcal{F})$ and $P_t(\\mathcal{F}|\\mathcal{U})$ are not included? It is possible to transition between feasible and infeasible solution spaces. \n\n3. The actual entropy calculation involves hyperparameters $c_1$ and $c_2$. Can the authors provide more context on how they were chosen and their impact on the stability of training?\n\n4. In Table 1, for different variants of NeuOpt, what does the DA number refer to? Is that the $T_{DA}$ mentioned in section 4.3?\n\n5. In Figure 5(a), the results for K = 5 and 6 w/o E-move are missing.\n\nLIMITATIONS:\nThe authors could add discussions about the generalization ability of trained models as well as scaling to larger problem sizes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Neural k-Opt (NeuOpt), a deep learning-based vehicle routing solver, designed to handle k-opt exchanges for any k≥2. Unlike existing Learning-to-Search (L2S) solvers, NeuOpt employs a tailored action factorization method, which allows complex k-opt exchanges to be broken down into simpler basis moves. This approach grants the model the flexibility to determine an appropriate k for each search step, enabling a balance between coarse-grained (larger k) and fine-grained (smaller k) searches.\n\nMoreover, the paper introduces Guided Infeasible Region Exploration (GIRE) scheme that promotes exploration of both feasible and infeasible regions in the search space. GIRE enriches the policy network with additional features to indicate constraint violations and exploration behavior statistics, aiding in escaping local optima, discovering shortcuts to better solutions, and enhancing the model's understanding of the problem landscapes.\n\nNeuOpt is trained using reinforcement learning (RL) and incorporates a dynamic data augmentation method during inference for improved efficiency. Extensive experiments on classic vehicle routing problem variants (TSP and CVRP) demonstrate the superiority of NeuOpt and GIRE over some existing approaches, including traditional hand-crafted solvers.\n\nSTRENGTHS:\n- Novel Approach: The paper introduces a novel approach called Neural k-Opt (NeuOpt) to handle k-opt exchanges for vehicle routing problems. This approach offers flexibility by allowing k to be any value ≥2, which can potentially lead to better solutions and more efficient search processes.\n\n- Guided Infeasible Region Exploration (GIRE): The introduction of GIRE is a significant contribution that promotes exploration of both feasible and infeasible regions in the search space. This unique scheme bridges feasible regions, helps escape local optima, and forces explicit awareness of the VRP constraints, potentially leading to improved performance in constrained VRPs.\n\n- Comprehensive Evaluation: The paper claims to have extensive experiments on classic VRP variants (TSP and CVRP) to validate the proposed NeuOpt and GIRE. A thorough evaluation of the model's performance on various VRP instances can demonstrate its effectiveness and potential advantages over existing methods.\n\nWEAKNESSES:\n- Scalability: While the paper claims that NeuOpt outperforms other methods, including hand-crafted solvers, there is no mention of its scalability to larger and more complex instances of vehicle routing problems. The effectiveness of NeuOpt on larger and real-world datasets should be explored.\n\n- Complexity of Action Factorization: Although the tailored action factorization method is introduced to handle k-opt exchanges flexibly, it may introduce increased complexity in the model's architecture and training process. The paper should discuss potential trade-offs and computational costs associated with the proposed factorization.\n\nQUESTIONS:\n- Figure 1 is very hard to process...\n\n- lines 122-123: I believe the definition of TSP is slightly inaccurate. The main objective of the TSP is to find the optimal Hamiltonian cycle that **minimizes** the total distance or cost required to visit all the nodes exactly once and return to the starting node.\n\n- Can authors provide more discussions on why choosing k dynamically should escape local minimas?\n\nLIMITATIONS:\nsee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-23"}