Title: A Learning-based Capacitated Arc Routing Problem Solver Comparable to Metaheuristics While with Far Less Runtimes
Venue: NeurIPS 2024
Accepted: no
Keywords: Capacitated Arc Routing

Metaheuristics

Reinforcement learning
Abstract: Recently, neural networks (NN) have made great strides in combinatorial optimization problems (COPs). However, they face challenges in solving the capacitated arc routing problem (CARP) which is to find the minimum-cost tour that covers all required edges on a graph, while within capacity constraints. Actually, NN-based approaches tend to lag behind advanced metaheuristics due to complexities caused by non-Euclidean graph, traversal direction and capacity constraints. In this paper, we introduce an NN-based solver tailored for these complexities, which significantly narrows the gap with advanced metaheuristics while with far less runtimes. First, we propose the direction-aware attention model (DaAM) to in corporate directionality into the embedding process, facilitating more effective one-stage decision-making. Second, we design a supervised reinforcement learning scheme that involves supervised pre-training to establish a robust initial policy for subsequent reinforcement fine-tuning. It proves particularly valuable for solving CARP that has a higher complexity than the node routing problems (NRPs). Finally, a path optimization method is introduced to adjust the depot return positions within the path generated by DaAM. Experiments show that DaAM surpasses heuristics and achieves decision quality comparable to state-of-the-art metaheuristics for the first time while maintaining superior efficiency, even in large-scale CARP instances. The code and datasets are provided in the Appendix.

Abstract

Recently, neural networks (NN) have made great strides in combinatorial optimization problems (COPs). However, they face challenges in solving the capacitated arc routing problem (CARP) which is to find the minimum-cost tour that covers all required edges on a graph, while within capacity constraints. Actually, NN-based approaches tend to lag behind advanced metaheuristics due to complexities caused by non-Euclidean graph, traversal direction and capacity constraints. In this paper, we introduce an NN-based solver tailored for these complexities, which significantly narrows the gap with advanced metaheuristics while with far less runtimes. First, we propose the direction-aware attention model (DaAM) to incorporate directionality into the embedding process, facilitating more effective one-stage decision-making. Second, we design a supervised reinforcement learning scheme that involves supervised pre-training to establish a robust initial policy for subsequent reinforcement fine-tuning. It proves particularly valuable for solving CARPthat has a higher complexity than the node routing problems (NRPs). Finally, a path optimization method is introduced to adjust the depot return positions within the path generated by DaAM. Experiments show that DaAM surpasses heuristics and achieves decision quality comparable to state-of-the-art metaheuristics for the first time while maintaining superior efficiency, even in large-scale CARP instances. The code and datasets are provided in the Appendix.

1 Introduction

The capacitated arc routing problem (CARP) [7] is a combinatorial optimization problem, frequently arising in domains such as inspection and search-rescue operations. Theoretically, the CARP is established on an undirected connected graph G = ( V , E , E R ) that includes a set of nodes V connected by a set of edges E , and an edge subset E R ⊆ E that needs to be served, called required edges. Each required edge has a demand value which spends the capacity of the vehicle when it is served. In this context, all vehicles start their routes from the depot node depot ∈ V and conclude their journey by returning to the same depot . The main goal of a CARP solver is to serve all required edges with the lowest total path cost, while ensuring that the vehicle does not exceed its capacity Q .

According to [7], the CARP is recognized as an NP-Hard problem. Among all solver solutions, the 29

30

Memetic Algorithms (MA) [15, 25], first proposed in 2005, have still maintained unrivaled results in

solving the CARP challenge to this day. However, they have struggled with high time costs and the 31

exponential growth of the search space as the problem scale increases. Compared to the traditional 32

methods, NN-based solvers [16, 10, 20] are faster with the assistance of GPU, have therefore gained 33

increasing attention in recent years. However, unlike the decent performance in solving NRP, such as 34

vehicle routing problem (VRP) and travelling salesman problem (TSP), or ARP defined in Euclidean 35

graphs, such as rural postman problem (RPP) and Chinese postman problem (CPP), NN-based 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

methods usually lags far behind the traditional ones in solving CARP. This discrepancy is attributed to the inability of existing methods to effectively reduce the high complexity of solving CARP:

Lack of edge direction in embedding learning: ARP solvers need to determine the edges to be traversed along with the direction of traversal, easy for humans to achieve in one step but extremely challenging for computers. Existing methods didn't encode edge directionality in embedding, making them have to build edge sequences and determine edges' directions separately and leading to path generation without sufficient consideration.

Ineffective learning for solving CARP: CARP is more complex than NRPs and Euclidean ARPs owing to the non-Euclidean input , edge direction , and capacity constraints . Thus, learning methods for NRPs and Euclidean ARPs cannot be directly transferred to solve CARP or work well even if adapted, leaving a lack of effective learning strategies for CARP.

In this paper, we aim to address both above issues and propose an NN-based solver for CARP that competes with the state-of-the-art MA [25] while with far less runtimes. Firstly, we propose the direction-aware attention model (DaAM). It computes embeddings for directed arcs decomposed from undirected edges to align with the nature of ARP, thus avoiding missing direction information and enabling concise one-stage decision-making. Secondly, we design a supervised reinforcement learning method to learn effective heuristics for solving CARP. DaAM is pre-trained to learn an initial policy by minimizing the difference from the decisions made by experts, and fine-tuned on largerscale CARP instances by Proximal Policy Optimization with self-critical strategy. Finally, to further boost the path quality, we propose a path optimizer (PO) to re-decide the vehicles' optimal return positions by dynamic programming. In the experiments, our method approaches the state-of-the-art MAwith an average gap of 5% and is 4% better than the latest heuristics and gains high efficiency.

2 Related Work

2.1 Graph Embedding Learning

Graph embedding [3] aims to map nodes or edges in a graph to a low-dimensional vector space. This process is commonly achieved via graph neural networks (GNNs) [31]. Kipf et al. [13] introduced graph convolutional operations to aggregate information from neighboring nodes for updating node representations. Unlike GCN, GAT [27] allowed dynamic node attention during information propagation by attention mechanisms. Other GNN variants [9, 30] exhibited a similar information aggregation pattern but with different computational approaches. In this paper, since an arc is related to the outgoing arc of its endpoint but irrelevant to the incoming arc of that, we use attention mechanisms to capture the intricate relationships between arcs for arc embedding learning.

2.2 Learning for Routing Problems

The routing problem is one of the most classic COPs, and it is mainly categorized into two types according to the decision element: node routing problems and arc routing problems.

Node routing problems (NRPs), such as TSP and VRP, aim to determine the optimal paths that traverse all nodes in the Euclidean space or graphs. As the solutions to these problems are contextdependent sequences of variable size, they cannot be directly modeled by the Seq2Seq model [24]. To address this problem, Vinyals et al. [28] proposed the Pointer network (PN) to solving Euclidean TSP, which achieves variable-size output dictionaries by neural attention. Due to the scarcity of labels for supervised learning, Bello et al. [2] modeled the TSP as a single-step reinforcement learning problem and trained the PN using policy gradient [29] within Advantage Actor-Critic (A3C) [17] framework. Nazari et al. [19] replaced the LSTM encoder in PN with an element-wise projection layer and proposed the first NN-based method to solve the Euclidean VRP and its variants. To better extract correlations between inputs, Kool et al. [14] utilized multi-head attention for embedding learning and trained the model using REINFORCE [29] with a greedy baseline. To solve COPs defined on graphs, Khalil et al. [11] proposed S2V-DQN to learn heuristics, employing structure2vec [5] for graph embedding learning and n-step DQN [18] for training. While the mentioned NN-based approaches have achieved comparable performance to metaheuristics, they cannot be directly applied to solve ARP due to the modeling differences between ARP and NRP.

Arc routing problems (ARPs) involve determining optimal paths for traversing arcs or edges in graphs, with variants like RPP, CPP, and CARP. Truong et al. [26] proposed a DRL framework to

address the CPP with load-dependent costs on Euclidean graphs and achieved better solution quality 89 than metaheuristics. However, CARPs are defined on non-Euclidean graphs. Unlike Euclidean graphs 90 with given node coordinates, non-Euclidean graphs require manually extracting and aggregating the 91 node representations, a task that is typically learnable. Although several NN-based algorithms have 92 been proposed, they still lag significantly behind traditional methods. Li et al. [16] pioneered the use 93 of the NN-based approach in solving the CARP by transforming it into an NRP. They first determined 94 the sequence of edges and then decided the traversal direction for each edge. Hong et al. [10] trained a 95 PN in a supervised manner to select undirected edges in each time step, and also determined the edge 96 traversal direction as post-processing. Ramamoorthy et al. [20] proposed to generate an initial tour 97 based on edge embeddings and then split it into routes within capacity constraint. These approaches 98 lack edge directionality encoding, leading to edge selection without sufficient consideration and 99 necessitating a two-stage decision process or an additional splitting procedure. 100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

3 Background

The attention model (AM) [14] exhibits superior effectiveness in solving classic Euclidean COPs due to its attention mechanisms for extracting correlations between inputs. Therefore, we use the AM as the backbone and give a brief review in terms of the TSP. Given an Euclidean graph G =( V , E ) , the AM defines a stochastic policy, denoted as π ( x |S ) , where x = ( x 0 , ..., x | V |-1 ) represents a permutation of the node indexes in V , and S is the problem instance expressing G . The AM is parameterized by θ as:

where t denotes the time step. Specifically, the AM comprises an encoder and a decoder. The encoder first computes initial d h -dimensional embeddings for each node in V as h 0 i through a learned linear projection. It then captures the embeddings of h 0 i using multiple attention layers, with each comprising a multi-head attention (MHA) sublayer and a node-wise feed-forward (FF) sublayer. Both types of sublayers include a skip connection and batch normalization (BN). Assuming that l ∈ { 1 , ..., N } denotes the attention layer, the l th layer can be formulated as h l i :

The decoder aims to append a node to the sequence x at each time step. Specifically, a context embedding h ( c ) is computed to represent the state at the time step t . Then a single attention head is used to calculate the probabilities for each node based on h ( c ) :

̸

where W Q and W K are the learnable parameters of the last attention layer. u ( c ) j is an unnormalized log probability with ( c ) indicating the context node. C is a constant, and p i is the probability distribution computed by the softmax function based on u ( c ) j .

4 Method

4.1 Direction-aware Attention Model

In this section, we propose the direction-aware attention model (DaAM). Unlike previous methods that separately learn edge embeddings and determine edge directions, our model encodes direction information directly into the embedding, enabling one-stage decision-making. As shown in Fig. 1, the DaAM makes sequential decisions in two phases to select arcs. The first phase is a one-time transformation process, in which the arcs of the input graph are represented as nodes in the new directed complete graph. The second phase is executed at each time step, in which GAT is used to aggregate the inter-arc weights. Subsequently, AM is used to select the arc of the next action.

4.1.1 Arc Feature Formulation via Graph Transformation

Graph Transformation Motivated by the need to consider direction when traversing edges, we explicitly encode the edge direction by edge-to-arc decomposition. Let G =( V , E , E R ) denotes

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

Feature of Arc

Probability

Attention Queries

Prediction

Node in

𝐕

Edge in

𝐄

Arcs in

𝐴

Directed Edge in

𝐸

Arc nodes in

𝐺

Execute on Each Decision

Attention Model

(Encoder)

Attention Model

(Decoder)

ℎ

(")

$%&

ℎ

(")

$

ℎ

'

'

𝑝

'

ℎ

'

$

ℎ

&

'

𝑝

&

ℎ

&

$

ℎ

(

'

𝑝

(

ℎ

(

$

ℎ

)

'

𝑝

)

ℎ

)

$

Graph Transformation

(Execute Once)

𝑓

*

𝐆

𝐞

!"

𝑛

𝑚

𝐺

𝑎𝑟𝑐

!

𝑎𝑟𝑐

"

𝑒

#$

…

𝑎𝑟𝑐

!"

𝑎𝑟𝑐

"!

𝑛

𝑚

depot

MDS

|𝑒

#$

|

Graph

Attention

|𝑒

$$

|

|𝑒

%$

|

|𝑒

&$

|

𝑓

'

𝑓

&

𝑓

(

𝑓

)

the undirected connected graph as input, where V is the node set of G , E is the edge set of G , and E R ⊆ E is the required edge set. Firstly, given that an edge has two potential traversal directions, we decompose each edge e nm =( cost nm , demand nm , allow _ serve nm ) ∈ E R into two arcs { arc nm , arc mn } with opposite directions but the same cost, demand and serving state. Here n, m are the indexes of node in V . To simplify the representation below, we replace nm and mn with single-word symbols, such as i and j . In this way of edge decomposition, we obtain a set of arcs denoted as A R . Secondly, we build a new graph G = ( A R , E ) . Specifically, each arc in A R serves as a node in G , and directed edge set E is created, with e ij ∈ E representing the edge from node arc i to arc j . The weight | e ij | represents the total cost of the shortest path from the end node of arc i to the start node of arc j . In addition, we treat the depot as a self-loop zero-demand arc that allows for repeated serving, denoted as arc 0 . Consequently, we transform the input graph G into a directed complete graph G . By decomposing all edges in E R into arcs, it is natural to directly select the arcs from G during the decision-making. Given that the Floyd-Warshall algorithm is used to calculate the shortest path cost between any pair of nodes in G , the time complexity of our graph transformation is max( O ( | E R | 2 ) , O ( | V | 3 )) .

Arc Feature Formulation To establish a foundation for decision-making regarding arc selection, the features of the arcs are constructed as input for the subsequent model. Specifically, multidimensional scaling (MDS) is used to project the input graph G into a d -dimensional Euclidean space. The Euclidean coordinates of arc i 's start and end nodes, denoted as mds start ( i ) and mds end ( i ) , are then taken as the features of arc i to indicate its direction. As shown in Table 1, at time step t , arc i can be featured as:

where x t -1 is the index of the selected arc at the last time step, and t ∈ [1 , + ∞ ) . Our feature models arcs rather than edges and encodes the direction attribute of arcs through MDS. Therefore, it is more suitable than previous methods [10, 16] for ARPs that need to consider the direction of traversing.

4.1.2 Arc Relation Encoding via Graph Attention Network

Although AM is efficient in decision-making, according to Eq. (2), it cannot encode the edge weights between nodes in G , an important context feature, during learning. Therefore, we use graph attention network (GAT) [27] to encode such weights. At each time step t , for each arc arc i , we integrate the

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

weights between arc i and all arcs in A R along with their features into the initial embedding of arc i .

where W is a shared learnable parameter, [ ·||· ] is the horizontal concatenation operator, α ( · ) is a mapping from the input to a scalar, and σ ( · ) denotes the activation function. h 0 i denotes the initial feature embedding of arc i , which is taken as the input of subsequent AM. Since G is a complete graph, we use one graph attention layer to avoid over-smoothing [4].

4.1.3 Arc Selection via Attention Model

After aggregating the edge weights of G into the initial embeddings, we utilize AM to learn the final arc embeddings and make arc selection decisions. In the encoding phase described by Eq.2, for each arc { arc i } , we leverage N attention layers to process the initial embeddings { h 0 i } and obtain the output embeddings of the N th layer, i.e., { h N i } . In the decoding phase, we define the context node applicable to CARP:

where x t -1 indicates the chosen arc index at time step t -1 and x 0 is arc 0 . δ t is the remaining capacity at time step t , ∆ t = ∆( δ t > Q 2 ) is a variable to indicate whether the vehicle's remaining capacity exceeds half. Finally, according to Eq.(3), the decoder of AM takes the context node h N ( c ) and arc embeddings { h N i } as inputs and calculates the probabilities for all arcs, denoted as p i . The serviceable arc selected at time step t , i.e., arc x t , is determined by sampling or greedy decoding.

4.2 Supervised Reinforcement Learning for CARP

The decision-making of selecting arcs can be modeled as a Markov decision process with the following symbols regarding reinforcement learning:

State s t is the newest path of arcs selected from G : ( arc x 0 , ..., arc x t -1 ) , while the terminal state is s T with T indicating the final time step.

Action a t is the selected arc at time step t , i.e., arc x t . Selecting the action a t would add arc x t to the end of the current path s t and tag the corresponding arcs of arc x t with their features allow _ serve changed to 0. Notably, arc 0 can be selected repeatedly but not consecutively.

Reward r t is obtained after taking action a t at state s t , which equals the negative shortest path cost from the last arc arc x t -1 to the selected arc arc x t .

Stochastic policy π ( a t | s t ) specifies the probability distribution over all actions at state s t .

We parameterize the stochastic policy of DaAM with θ :

where S is a CARP instance. Starting from initial state s 0 , we get a trajectory τ = s 0 , a 0 , r 0 , ..., r T -1 , s T ) using π θ . The goal of learning is to maximize the cumulative reward: R ( τ ) = ∑ T -1 t =0 r t . However, due to the high complexity of CARP, vanilla deep reinforcement learning methods learn feasible strategies inefficiently. A natural solution is to minimize the difference between the model's decisions and expert decisions. To achieve this, we employ supervised learning to learn an initial policy based on labeled data and then fine-tune the model through reinforcement learning.

(

4.2.1 Supervised Pre-training via Multi-class Classification

In the pre-training stage, we consider arc-selection at each time step as a multi-class classification task, and employ the state-of-the-art CARP method MAENS to obtain high-quality paths as the label. Assuming that y t ∈ R | A R | denotes the one-hot label vector at time step t of any path, with y ( k ) t indicating each element. We utilize the cross-entropy loss to train the policy represented in Eq. (7):

We use the policy optimized by cross-entropy, denoted as π s , to initialize the policy network π θ and as the baseline policy π b in reinforcement learning.

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

4.2.2 Reinforcement Fine-tuning via PPO with self-critical strategy

During the fine-tuning phase, we use Proximal Policy Optimization (PPO) to optimize our model π θ ( a t | s t ) due to its outstanding stability in policy updates. Considering the low sample efficiency in reinforcement learning, we employ a training approach similar to self-critical training [21] to reduce gradient variance and expedite convergence. Specifically, We use another policy π b to generate a trajectory and calculate its cumulative reward, serving as a baseline function. Our optimization objective is based on PPO-Clip [23]:

where s is used to replace current state s t for symbol simplification, and a for a t . clip ( w,v min , v max ) denotes constraining w within the range [ v min , v max ] , and ϵ is a hyper-parameter. τ θ s denotes a trajectory sampled by π θ with s as the initial state, while τ b s for the trajectory greedily decoded by π b . In greedy decoding, the action with the maximum probability is selected at each step. R ( τ θ s ) -R ( τ b s ) serves as an advantage measure, quantifying the advantage of the current policy π θ compared to π b . We maximize Eq. (9) through gradient descent, which forces the model to select actions that yield higher advantages. The baseline policy's parameters are updated if π θ outperforms π b .

4.3 Path Optimization via Dynamic Programming

The complexity of the problem is heightened by the increasing capacity constraint, making it challenging for the neural network to make accurate decisions regarding the depot return positions. In this sec-, tion, we propose a dynamic programming (DP) based strategy to assist our model in optimizing these positions. Assuming that P is assigned with the terminal state s T = ( arc x 0 , arc x 1 , ..., arc x T -1 ) representing a generated path. Initially, we remove all the depot arcs in P to obtain a new path P ′ = ( arc x ′ 0 , arc x ′ 1 , ..., arc x ′ T ′ -1 ) , where { x ′ i | i ∈ [0 , T ′ -1] } denotes a subsequence of { x i | i ∈ [0 , T -1] } . Subsequently, we aim to insert several new depot arcs into the path P ′ to achieve a lower cost while adhering to capacity constraints. To be specific, we recursively find the return point that minimizes the overall increasing cost, which is implemented by the state transition

equation as follows:

where SC ( arc x ′ i , arc 0 ) = | e x ′ i 0 | denotes the shortest path cost from arc x ′ i to the depot. Q is the vehicle capacity. According to Eq. (10), we insert the depot arc arc 0 after an appropriate position arc x ′ i , which meets with the capacity constraint of the subpath P ′ i +1: T ′ -1 . f ( · ) denotes a state featuring dynamic programming. By enumerating the position i , we compute the minimum increasing cost f ( P ′ ) utilizing its sub-state f ( P ′ 0: i ) . The final minimum cost for path P is f ( P ′ ) + g ( P ′ ) , here g ( P ′ ) is the unoptimized cost of P ′ . Since P ′ includes only the required edges, i.e., T ′ = | E R | , the time complexity of DP is O ( | E R | 2 ) . During Path Optimization, we use beam search to generate two paths with the trained policy, one with capacity-constrained and one without. Both paths are optimized using DP and the one with the minimum cost is selected as the final result.

5 Experiments

5.1 Setup

Problem Instances. We extracted sub-graphs from the roadmap of Beijing, China, obtained from OpenStreetMap [8], to create CARP instances for both the training and testing phases. All instances are divided into seven datasets, each representing different problem scales, as presented in Table 2. Each dataset consists of 30,000 instances, further divided into two disjoint subsets: 20,000 instances for training and the remaining for testing. For each instance, the vehicle capacity is set to 100.

Implementation Details. Our neural network is implemented using the PyTorch framework and trained on a single NVIDIA RTX 3090 GPU. The heuristics and metaheuristics algorithms are

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

evaluated on an Intel Core i9-7920X with 24 cores and a CPU frequency of 4.4GHz. We optimize the model using Adam optimizer [12]. The dimension of MDS coordinates d is set to 8, and the learning rate is set to 1 e -4 . We set ϵ in the PPO training at 0.1. Notably, our PPO training does not incorporate discounted cumulative rewards, i.e., γ is set to 1.

Metrics and Settings. For each method and dataset, We compute the mean tour cost across all test instances, indicated by 'Cost'. Employing the state-of-the-art MAENS [25] as a baseline, we measure the 'Cost' gap between alternative algorithms and MAENS, indicated by 'Gap'. We compare our method against the heuristic Path-Scanning algorithms (PS) [6, 22, 1] and two NN-based algorithms. In the absence of publicly available code for prior NN-based CARP methods, we modify two NNbased NRP solvers to suit CARP, i.e, S2V-DQN [11] and VRP-DL [19]. Note that, for S2V-DQN, we replace structure2vec with GAT to achieve more effective graph embedding learning. For our method, we incrementally add supervised pre-training (SL), reinforcement learning fine-tuning (RL), and path optimization (PO) to assess the effectiveness of our training scheme and optimization, respectively. Due to the excessively long computation times of MAENS on larger-scale datasets, SL is only performed on Task20, Task 30, and Task40. The batch size for SL is set to 128. During the RL stage, greedy decoding is used to generate solutions, and except for the Task20 dataset, we utilize the training results obtained from the preceding smaller-scale dataset to initialize the model. The beam width in the PO stage is set to 2. For each dataset, we compare the mean cost of different methods on 10,000 problem instances.

5.2 Evaluation Results

Solution Quality Table 6 shows the result. Our algorithm outperforms all heuristic and NN-based methods across all scales, achieving costs comparable to MAENS, trailing by less than 8% . The advantage over PS demonstrates that neural networks can learn more effective policies than handcrafted ones, attributed to our well-designed modeling approach. Moreover, as the problem scale increases, it becomes time-consuming to obtain CARP annotation by MAENS. Therefore, we leverage the model pre-trained on small-scale instances as the initial policy for RL fine-tuning on Task50, Task60, Task80, and Task100, yielding commendable performance. This proves the generalization of our training scheme across varying problem scales. The performance gap with MAENS highlights our algorithm's superiority in CARP-solving approaches.

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

Task20

Task40

Task60

Task80

Task100

10

0

10

1

10

2

10

3

10

4

Time (seconds)

Run Time in Log Space vs Number of Edge

MAENS

PS

PS-Ellipse

PS-Efficiency

PS-Alt1

PS-Alt2

Ours (w/o PO)

Ours (w PO)

Task200

Task300

Task400

Task500

Task600

10

1

10

2

10

3

Time (seconds)

Run Time in Log Space vs Number of Edge

PS-Ellipse

PS-Efficiency

PS-Alt1

PS-Alt2

Ours (w/o PO)

Ours (w PO)

Datasets

Datasets

Generalization Ability. In Table 4, we assess DaAM's generalization on large-scale CARP instances using the policy trained on Task100. We remove MAENS and PS due to failing to run on large-scale graphs, and remove S2V-DQN and VRP-DL due to poor performance. Although DaAM is not trained on large-scale instances, it achieves or even exceeds the performance of PS, which shows its potential application on larger-scale CARP instances.

Run Time. We compare the total time required for solving 100 CARP instances across datasets Task20 to Task100 datasets using our method, MAENS, and PS algorithms, and show the run time in log space. For datasets Task200 to Task600, we compare the same metric using variants of PS and out method. For our method, we measured the solving time with and without PO. Fig. 2 demonstrates that our method exhibits a significant speed advantage over MAENS, even outperforming variants of PS [1] on most datasets. In comparison, the consumption time of MAENS increases exponentially as the problem scale increases. Our method efficiently generates paths for large-scale CARP instances by leveraging GPU data-level parallelism and CPU instruction-level parallelism.

Effectiveness of Combining MDS and GAT. To evaluate the combination of MDS and GAT for embedding exhibiting, we individually evaluate the performance of models using only MDS or GAT, as well as their combined performance. The experiment is conducted on Task30, Task40,

Task50, and Task60 by comparing the average performance of 1,000 instances on each dataset. In the RL stage, we use the policy pre-trained on Task30 for initialization. Table 5 indicates that using MDS or GAT individually yields worse quality in most cases, highlighting that combining MDS and GAT enhances the model's capacity to capture arc correlations. Fig. 3 depicts the convergence trends in these scenes, which shows that the synergy of MDS and GAT contributes to the stability of training.

Solution Visualization. For a more intuitive understanding of the paths generated by different methods, we visualize and compare the results of our method with PS [6] and MAENS across four road scenes in Beijing. Fig. 4 visualizes all results alongside scene information. We observe that our model obtains similar paths with MAENS since we leverage the annotation generated by MAENS for

0

20

40

60

80

740

760

780

800

820

840

Solution Cost

Task30

MDS + GAT

MDS

GAT

0

20

40

60

80

100

1010

1020

1030

1040

1050

1060

1070

Task40

MDS + GAT

MDS

GAT

0

20

40

60

80

Epochs

1250

1500

1750

2000

2250

2500

2750

Solution Cost

Task50

MDS + GAT

MDS

GAT

0

10

20

30

Epochs

1700

1750

1800

1850

1900

1950

Task60

MDS + GAT

MDS

GAT

MAENS

Ours (SL+RL+PO)

Path Scanning

Taiping Street, Beijing

𝐄

!

= 54;

𝐄

= 100;

𝐕 = 88

Sanlihe, Beijing

𝐄

!

= 125;

𝐄

= 125;

𝐕 = 103

Beijing Used Car Market, Beijing

𝐄

!

= 103;

𝐄

= 103;

𝐕 = 154

Area of

'

'

, Beijing

𝐄

!

= 79;

𝐄

= 138;

𝐕 = 116

Cost=1925

Cost=1584

Cost=1685

Cost=2948

Cost=2824

Cost=3208

Cost=2610

Cost=2122

Cost=2335

Cost=1933

supervised learning. MAENS paths exhibit superior spatial locality, clearly dividing the scene into 303 regions, whereas PS paths appear more random. 304

305

6 Conclusion and Limitations

In this paper, we propose a learning-based CARP solver that competes with state-of-the-art meta306 heuristics. Firstly, we encode the potential serving direction of edges into embeddings, ensuring 307 that edge directionality is taken into account in decision-making. Secondly, we present a supervised 308 reinforcement learning approach that effectively learns policies to solve CARP. With the aid of 309 these contributions, our method surpasses all heuristics and achieves performance comparable to 310 metaheuristics for the first time while maintaining excellent efficiency. 311

Limitations and future work. Decomposing undirected edges increases the decision elements, 312 which complicates the problem and may widens the gap between DaAM and traditional state-of-the313 art approaches as the problem instance scale increases. Our future work focuses on designing an 314 efficient graph transformation method that does not significantly increase problem complexity. 315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

References

Rafael Kendy Arakaki and Fabio Luiz Usberti. An efficiency-based path-scanning heuristic for the capacitated arc routing problem. Computers & Operations Research (COR) , 103:288-295, 2019.

Irwan Bello*, Hieu Pham*, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. In International Conference on Learning Representations (ICLR) , 2017.

Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE transactions on knowledge and data engineering (TKDE) , 30(9):1616-1637, 2018.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In AAAI conference on artificial intelligence (AAAI) , 2020.

Hanjun Dai, Bo Dai, and Le Song. Discriminative embeddings of latent variable models for structured data. In International conference on machine learning (ICML) , 2016.

Bruce L Golden, James S DeArmon, and Edward K Baker. Computational experiments with algorithms for a class of routing problems. Computers & Operations Research (COR) , 10(1):4759, 1983.

Bruce L Golden and Richard T Wong. Capacitated arc routing problems. Networks , 11(3):305315, 1981.

Mordechai Haklay and Patrick Weber. Openstreetmap: User-generated street maps. IEEE Pervasive computing , 7(4):12-18, 2008.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems (NeurIPS) , 2017.

Wenjing Hong and Tonglin Liu. Faster capacitated arc routing: A sequence-to-sequence approach. IEEE Access , 10:4777-4785, 2022.

Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. In Advances in neural information processing systems (NeurIPS) , 2017.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International 345 Conference on Learning Representations (ICLR) , 2015. 346

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional 347 networks. In International Conference on Learning Representations (ICLR) , 2017. 348

Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In 349 International Conference on Learning Representations (ICLR) , 2019. 350

Natalio Krasnogor and James Smith. A tutorial for competent memetic algorithms: model, taxon351 omy, and design issues. IEEE transactions on Evolutionary Computation (TEVC) , 9(5):474-488, 352 2005. 353

Han Li and Guiying Li. Learning to solve capacitated arc routing problems by policy gradient. 354 In IEEE Congress on Evolutionary Computation (CEC) , 2019. 355

356

357

358

359

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning (ICML) , pages 1928-1937, 2016.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G 360 Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 361 Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015. 362

Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takác. Reinforcement 363 learning for solving the vehicle routing problem. In Advances in neural information processing 364 systems (NeurIPS) , 2018. 365

Muhilan Ramamoorthy and Violet R. Syrotiuk. Learning heuristics for arc routing problems. 366 Intelligent Systems with Applications (ISWA) , 21:200300, 2024. 367

368

369

370

Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Selfcritical sequence training for image captioning. In IEEE conference on computer vision and pattern recognition (CVPR) , 2017.

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

Luís Santos, João Coutinho-Rodrigues, and John R Current. An improved heuristic for the capacitated arc routing problem. Computers & Operations Research (COR) , 36(9):2632-2637, 2009.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems (NeurIPS) , 2014.

Ke Tang, Yi Mei, and Xin Yao. Memetic algorithm with extended neighborhood search for capacitated arc routing problems. IEEE Transactions on Evolutionary Computation (TEVC) , 13(5):1151-1166, 2009.

Cong Dao Tran and Truong Son Hy. Graph attention-based deep reinforcement learning for solving the chinese postman problem with load-dependent costs. arXiv preprint arXiv:2310.15516 , 2023.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In International Conference on Learning Representations (ICLR) , 2018.

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in neural information processing systems (NeurIPS) , 2015.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning (ICML) , 2019.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A 394 comprehensive survey on graph neural networks. IEEE transactions on neural networks and 395 learning systems (TNNLS) , 32(1):4-24, 2020. 396

A Appendix 397

398

399

400

401

402

403

404

405

406

407

408

409

A.1 Source Code and Dataset

The source code of DaAM and the datasets used for testing are available at DaAM. Once the paper is accepted, we will promptly release the source code and datasets.

A.2 Pseudocode of PPO with self-critical strategy

Algorithm 1 presents the pseudocode for the PPO training algorithm we used. In the code implementation, the trajectory τ θ s can be replaced by ( s, a ) 's original trajectory τ o for efficiency. Once τ o is sampled, the cumulative rewards from any state s ∈ τ o can be quickly computed.

Algorithm 1 PPO algorithm with self-critical strategy

A.3 Experimental Results of Additional Datasets

For small-scale problem instances, we generated two additional datasets, Task30 and Task50. In Task30 the range of | V | is 25-30, while in Task50, it spans 55-60. Correspondingly, | E R | is set to 30 and 50, respectively The demand for each edge ranges from 5 to 10 in both tasks. Table 6 is the complete experimental data from the solution quality experiments.

A.4 Licences of Assets Used for Experiments 410

The code we used does not require special consent from the authors. We follow their licenses as 411 specified below: 412

413

https://github.com/wouterkool/attention-learn-to-route: MIT Licence.

414

https://github.com/Hanjun-Dai/graph_comb_opt: MIT Licence.

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Abstract and Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See section Conclusion and Limitations .

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This papar does not include theoretical results. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This papar discusses the detail to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

Answer: [Yes]

Justification: The source code and datasets are provided in the Appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See section Experiments .

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Since the experimental results are deterministic, we did not repeat the experiments multiple times. However, to reduce errors, we calculated the average over 10,000 problem instances for each dataset of any scale.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See section Experiments .

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The aim of our work is to provide a better solution for a class of combinatorial optimization problems, though it is difficult to predict its impact on society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See the Appendix .

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See the Appendix .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a learning-based method to address the Capacitated Arc Routing Problem (CARP). It involves breaking undirected edges into directed arcs and utilizing a graph attention network to build a Direction-aware Attention Model. In the training process, supervised learning is used to create the initial policy, followed by reinforcement learning based on policy gradients using Proximal Policy Optimization (PPO) to refine strategies. Lastly, dynamic programming is applied to optimize depot placements for path enhancement. Experimental outcomes show notable benefits of this algorithm in evaluation criteria.\n\nSTRENGTHS:\nIn general, the paper exhibits a well-organized structure with detailed experimental outcomes showcase through graphs and tables, facilitating readers in comprehending and visualizing the results effortlessly. The dataset employed comprises real-world scenarios, thereby boosting its practical relevance.\n\nWEAKNESSES:\nConverting the graph G from arcs to nodes represents a common approach in many heuristics for addressing CARP. This process adds complexity to the problem and increases its scale. The proposed method appears to lack enough novelty, with most components bearing resemblance to neural models designed for CVRP.\n\nQUESTIONS:\nIn the experimental section, it is highlighted that supervised learning was employed for generating the initial policy using MAENS to produce real labels. Nonetheless, supervised learning may face challenges with large-scale datasets. Is supervised learning indispensable for this aspect? Would the final model's performance be affected if the initial policy was randomly generated instead?\n\nIs path optimization applied during both the training and inference/testing phases, or is it specifically used for the inference phase only? Moreover, the time specified in Figure 2 likely pertains to the inference time for sample testing. I am also curious about the model's training time—approximately how long does it typically take?\n\nLIMITATIONS:\nIt appears that the paper focuses on an unlimited number of vehicles. How would the approach adapt to a specific set of vehicles?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors skillfully address challenges posed by non-Euclidean graphs, traversal direction, and capacity constraints with their novel NN-based solver in solving capacitated arc routing problem. The introduction of the direction-aware attention model and a supervised reinforcement learning scheme is particularly commendable. These innovations significantly narrow the gap with advanced metaheuristics, achieving superior efficiency and competitive decision quality.\n\nSTRENGTHS:\n1. The manuscript employs numerous innovative methods to solve the capacitated arc routing problem, achieving impressive results. \n2. It also shows promising performance in generalizing to larger problem instances.\n3. The combination of supervised and reinforcement learning is quite interesting. Using supervised learning for pre-training followed by fine-tuning with reinforcement learning is a noteworthy approach.\n4. The qualitative comparisons in real street scenes presented in Figure 4 are particularly interesting.\n\nWEAKNESSES:\n1. It's better to redraw the first part of Figure 1 to enhance its aesthetic quality.\n2. The baseline is not very recent. After S2V-DQN and S2V-DQN, there are still some excellent works that can be used to address the CARP problem.\n3. Some writing errors have been identified, such as in line 2 of Algorithm 1. Please review the entire manuscript to check.\n4. The completeness of the manuscript still requires supplementation and refinement.\n\nQUESTIONS:\nThe method proposed by the authors is innovative and interesting, but I still have some questions that I hope the authors can address:\n1. In the manuscript, what is the significance of transforming the initial undirected edges into fully connected directed edges at the beginning? If edges' directions are required, why not just convert them into bidirectional directed edges?\n2. In section 4.2, how is ϵ selected?\n3. In section 4.3, the authors remove all the depot arcs initially and then add them back. What is the purpose of these steps? Why does this not affect the calculation of vehicle capacity?\n4. In the Solution Visualization section, the authors have visualized the streets of Beijing. How does the model handle the non-Euclidean space in real-world scenarios?\n5. The detailed structure of the proposed model and algorithm should be further completed and upplemented in the appendix.\n\nLIMITATIONS:\nThe approach of decomposing undirected edges into directed ones introduces additional decision elements, which complicates the problem. It's better that the authors can find a more efficient graph processing method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new learning-based constructive heuristic for capacitated arc routing problems. In contrast to node routing problems such as the TSP and VRP, arc routing problems received comparably little attentition. To address the specific\nchallenges in the capacitated arc routing problems, the authors propose a Neural Network-based approach that uses a graph attention\nmodel considering arc directionality, a reinforcement learning approach with supervised pre-training and PPO-based fine-tuning. In\norder to improve solutions obtained by an RL-based construction approach, they propose a beam search approach for path optimization which, after turning the set of routes into a giant tour, splits the tour into routes by adding returns to the depot. A set of experiments\nshows that the proposed approach consistently yields better results than traditional hand-crafted constructive heuristics, and that their solutions almost match the quality of a time-consuming memetic algorithm that is only capable of solving small instance in a reaonable amount of time.\n\nSTRENGTHS:\nThe authors propose one of the first learning-based approaches for the capacitated arc routing problem (CARP). Their approach, in particular their graph embedding, explicitly addresses one of the challenges of learning-based construction algorithms for this problem by explicitly replacing the undirected edges by directed arcs. This idea is original and turns out to be helpful to create a well-performing heuristic. \n\nThe online performance of the approach surpasses hand-crafted constructive heuristics both in terms of runtime and efficiency. While for small instances, other metaheuristic approaches are better, it can be assumed that for large-scale instances, the proposed approach surpasses the state-of-the art of heuristic approaches. This is a significant result, since for node routing problems such as the CVRP, researcher have been struggling for years to design learning-based heuristics that achieve a performace that is comparable to hand-crafted heuristics. It should be mentioned, though, that in general, arc routing problems receive much less attention than node routing problems in the literature.\n\nThe paper comprises several insightful and, as far as I can tell, reasonably designed experiments, in particular showing the generalization capability to larger instances. \n\nThe paper provides both code and instances.\n\nWEAKNESSES:\nThe presentation of the CARP routing problem, the solutions approaches and related work lacks clarity in many places.\nAs an example, in the abstract, we find that the CARP consists in finding \"the minimum-cost tour\"hat covers all required edges on a graph, while within capacity constraints\". This is a a bit misleading description since we look for a set of routes instead of a tour.\n\nThe paper distinguishes \"heuristics\" and \"metaheuristics\", while clearly metaheuristics are a type of heuristics. Actually, what the authors appear to have in mind is \"constructive heuristics\" which sequentially construct a solution by adding edges to form routes. I suggest to formulate more precisely here.\n\nSimilarly, it would enhance the understanding of the paper to introduce the notion of \"route-first, cluster second\" and the related\nnotion of a \"giant tour\" which is commonplace in routing applications, to characterize respective existing work. It would even facilitate the\npresented path optimization which actually turns the presented approach into a route-first, cluster second approach. \n\n\nThe computational results are convincing, but the discussion should emphasize that a fair comparison can only be made between their\napproach wihout path optimization and the other constructive heuristics. It would indeed be interesting to see how the far the path\noptimization is able to improve the results of the other constructive heuristics.\n\nThe claims \"NN-based approaches tend to lag behind advanced metaheuristics\" (abstract) and \"NN-based methods usually lags far\nbehind the traditional ones in solving CARP\", \"they still lag significantly behind traditional methods\" are not valid. Actually, (Rahmamoorty et. al 2024) (reference 20) report that on average, they improve upon the memetic algorithm by 11% on average.\n\nWhen it comes to the evaluation of the path scanning approaches in the experiments, it is unclear how they are parameterized. From reading the paper (Aarakaki 2019) one sees that the parameter alpha and the number of iteration have a considerable impact both on solution time and solution quality, and (albeit on different instances), the average gaps for the path scanning approaches to the optimal (and to the memetic algrithm) reported in (Aarakaki 2019) are smaller than those found in the submission.\n\nThe description of the path improvement is not very clear; in particular the definition of the state used in the Dynamic Programming\nalgorithm. Is it a path? Is it the length of a path? Also, the statement \"f(*) denotes a state featuring dynamic programming\" is hard\nto decipher.\n\nTraining time is not discussed at all.\n\nQUESTIONS:\nPlease address the weaknesses mentioned above. Also:\n\n- How long does the training take?\n- Where does the implementation of the memetic algorithm come from?\n- How were the PS approaches parameterized?\n\nLIMITATIONS:\nLimitations are mostly addressed in a reasonable way. I suggest to add the following aspects:\n\nI think that for small instances, the approach by (Rahmamoorty et. al 2024) may surpass the results reported here, which should be mentioned. \n\nAlso, you should at least briefly mention the training time, since this makes it easier to assess the trade-off between offline effort and online performace.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper presents a novel neural network-based CARP solver that uses a direction-aware attention model to incorporate directionality into the embedding process. It then applies supervised reinforcement learning for subsequent fine-tuning.

STRENGTHS:
1. A learning-based CARP solver is proposed.  
2. The performance of the proposed solver on large-scale data is discussed.

WEAKNESSES:
1. The comparison algorithms were published five years ago, and there is no discussion of existing methods aimed at big data.  
2. The experiments only tested the self-constructed dataset and did not evaluate on public datasets.

QUESTIONS:
1.How is the runtime for different algorithms defined in this paper? Does the training time for the learning-based algorithms take into account? 
2.How can one assess the effectiveness of the arc features used in this paper in a more intuitive way?

LIMITATIONS:
The amount of data required for algorithm training needs to be discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14