Title: DeepACO: Neural-enhanced Ant Systems for Combinatorial Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural Combinatorial Optimization

Ant Colony Optimization

Evolutionary algorithm

Meta-heuristic

Deep reinforcement learning

Learned heuristic measure

Neural local search

Generalization
Abstract: Ant Colony Optimization (ACO) is a meta-heuristic algorithm that has been successfully applied to various Combinatorial Optimization Problems (COPs). Traditionally, customizing ACO for a specific problem requires the expert design of knowledge-driven heuristics. In this paper, we propose DeepACO, a generic framework that leverages deep reinforcement learning to automate heuristic designs. DeepACO serves to strengthen the heuristic measures of existing ACO algorithms and dispense with laborious manual design in future ACO applications. As a neural-enhanced meta-heuristic, DeepACO consistently outperforms its ACO counterparts on eight COPs using a single neural model and a single set of hyperparameters. As a Neural Combinatorial Optimization method, DeepACO performs better than or on par with problem-specific methods on canonical routing problems. Our code is publicly available at https://github.com/henry-yeh/DeepACO.

DeepACO: Neural-enhanced Ant Systems for Combinatorial Optimization

Abstract

Ant Colony Optimization (ACO) is a meta-heuristic algorithm that has been successfully applied to various Combinatorial Optimization Problems (COPs). Traditionally, customizing ACO for a specific problem requires the expert design of knowledge-driven heuristics. In this paper, we propose DeepACO, a generic framework that leverages deep reinforcement learning to automate heuristic designs. DeepACO serves to strengthen the heuristic measures of existing ACO algorithms and dispense with laborious manual design in future ACO applications. As a neuralenhanced meta-heuristic, DeepACO consistently outperforms its ACO counterparts on eight COPs using a single neural model and a single set of hyperparameters. As a Neural Combinatorial Optimization method, DeepACO performs better than or on par with problem-specific methods on canonical routing problems. Our code is publicly available at https://github.com/henry-yeh/DeepACO.

1 Introduction

Ant systems in nature are self-learners. They utilize chemical signals and environmental cues to locate and return food to the colony. The pheromone trails deposited by ants indicate the quality and distance of a food source. The intensity of pheromone trails increases as more ants visit and decreases due to evaporation, creating a self-learning foraging system.

Inspired by the ant systems in nature, researchers propose and develop Ant Colony Optimization (ACO) meta-heuristics for (but not limited to) Combinatorial Optimization Problems (COPs) [24, 26]. ACO deploys a population of artificial ants to explore the solution space through repeated solution constructions and pheromone updates. The exploration is biased toward more promising areas through instance-specific pheromone trails and problem-specific heuristic measures. Both the pheromone trail and the heuristic measure indicate how promising a solution component is. Typically, pheromone trails are initialized uniformly for all solution components and learned while solving an instance. On the contrary, heuristic measures are predefined based on prior knowledge of a problem, and devising proper heuristic measures for complicated COPs is quite challenging (an example is [49]).

Over the past decades, research and practice efforts have been dedicated to a careful design of heuristic measures in pursuit of knowledge-driven performance enhancement [57, 62, 33, 49, 26, 24]. However, this routine of algorithm customization exhibits certain deficiencies: 1) it requires extra effort and makes ACO less flexible; 2) the effectiveness of the heuristic measure heavily relies on expert knowledge and manual tuning; and 3) designing a heuristic measure for less-studied problems can be particularly challenging, given the paucity of available expert knowledge.

This paper proposes DeepACO, a generic neural-enhanced ACO meta-heuristic, and a solution to the above limitations. DeepACO serves to strengthen the heuristic measures of existing ACO algorithms and dispense with laborious manual design in future ACO applications. It mainly involves two learning stages. The first stage learns a problem-specific mapping from an instance to its heuristic measures by training neural models across COP instances. Guided by the learned measures, the second stage learns instance-specific pheromone trails while solving an instance with ACO. The heuristic measures learned in the first stage are incorporated into ACO (the second learning stage) by biasing the solution constructions and leading Local Search (LS) to escape local optima.

DeepACO is also along the line of recent progress in Neural Combinatorial Optimization (NCO) [12, 61, 7, 34]. Within the realm of NCO, DeepACO is more related to the methods that utilize heatmaps for algorithmic enhancement [32, 42, 90, 55, 42], but it is superior in its flexibility: DeepACO provides effective neural enhancement across eight COPs covering routing, assignment, scheduling, and subset problems, being the most broadly evaluated NCO technique to our knowledge. In addition, we propose three extended implementations for better balancing between exploitation and exploration: one featuring a multihead decoder, one trained with an additional topk entropy loss, and one trained with an additional imitation loss. They can be generally applied to heatmap-based NCO methods.

As a neural-enhanced version of ACO, DeepACO consistently outperforms its counterparts on eight COPs using a single neural model and a single set of hyperparameters after only minutes of training. As an NCO method, DeepACO performs better than or competitively against the stateof-the-art (SOTA) and problem-specific NCO methods on canonical routing problems while being more generalizable to other COPs. To the best of our knowledge, we are the first to exploit deep reinforcement learning (DRL) to guide the evolution of ACO meta-heuristics. Such a coupling allows NCO techniques to benefit from decades of ACO research (notably regarding theoretical guarantees [7, 26]), while simultaneously offering ACO researchers and practitioners a promising avenue for algorithmic enhancement and design automation.

In summary, we outline our contributions as follows:

We propose DeepACO, a neural-enhanced ACO meta-heuristic. It strengthens existing ACO algorithms and dispenses with laborious manual design in future ACO applications.

We propose three extended implementations of DeepACO to balance exploration and exploitation, which can generally be applied to heatmap-based NCO methods.

We verify that DeepACO consistently outperforms its ACO counterparts across eight COPs while performing better than or on par with problem-specific NCO methods.

2 Related work

2.1 Neural Combinatorial Optimization

Neural Combinatorial Optimization (NCO) is an interdisciplinary field that tackles COPs with deep learning techniques. In general, existing NCO methods can be categorized into end-to-end and hybrid methods, and DeepACO belongs to the latter methodological category.

End-to-end methods in the former category learn autoregressive solution constructions or heatmap generation for subsequent sampling-based decoding. Within this realm, recent developments include better-aligned neural architectures [82, 64, 54, 50, 89, 13, 46], improved training paradigms [6, 56, 52, 83, 10, 66, 40, 45], advanced solution pipelines [47, 51, 48, 20, 78, 18, 65, 19], and broader applications [88, 95, 16, 29, 75, 9]. End-to-end methods are admirably efficient, but their constructed solutions can be further improved with iterative refinement and algorithmic hybridization.

Therefore, the hybrid methods in the latter category incorporate neural learners to make decisions within/for heuristics or generate heatmaps to assist heuristics. These methods either let neural learners make decisions within algorithmic loops [60, 87, 21, 17, 94, 86, 59], or generate heatmaps in one shot to assist subsequent algorithms. In the latter case, Xin et al. [90] propose to learn edge scores and node penalties to guide the searching process of LKH-3 [39], a highly optimized solver for routing problems. Kool et al. [55] train neural models to predict promising edges in routing problems, providing a neural boost for Dynamic Programming. Fu et al. [32] train small-scale GNN to build heatmaps for large TSP instances and feed the heatmaps into Monte Carlo Tree Search for solution

improvements. Hudson et al. [42] utilize neural models to generate regret in Guided Local Search for TSP. DeepACO is along the line of these works, but superior in terms of its flexibility, i.e., it provides effective neural enhancement across eight COPs covering routing, assignment, scheduling, and subset problems while the existing hybridized methods mostly focus on a limited set of routing problems.

2.2 Ant Colony Optimization

Ant Colony Optimization (ACO) is a meta-heuristic and evolutionary algorithm (EA) [4, 73] inspired by the behavior of ants in finding the shortest path between their colony and food sources [26].

The representative ACO meta-heuristics such as Ant System (AS) [25], Elitist AS [25], and MAXMIN AS [76] provide general-purpose frameworks allowing for problem-specific customization. Such customization may involve designs of heuristic measures [49, 33], the incorporation of local search operators [3, 93], and the hybridization of different algorithms [44, 5]. DeepACO is not in competition with the most SOTA ACO algorithms for a specific COP. Instead, DeepACO can strengthen them with stronger heuristic measures and can in turn benefit from their designs.

ACO can utilize a group of techniques named hyper-heuristics [69] that are conceptually related to DeepACO. But hyper-heuristics mostly involve expert-designed heuristics to select from (heuristic selection hyper-heuristics) [27] or problem-specific and manually-defined components to evolve heuristics (heuristic generation hyper-heuristics) [14]. By comparison, DeepACO is more generic, requiring little prior knowledge of a COP. Its aim is not to compete with any hyper-heuristics; instead, for example, we can utilize hyper-heuristics to improve LS components in DeepACO.

Unlike the knowledge-driven ACO adaptations above, a recent method, namely ML-ACO [77], boosts the performance of ACO via Machine Learning (ML) for solution prediction. While ML-ACO provides an initial insight into coupling ML with ACO, it is preliminary and limited. ML-ACO tailors five features for Orienteering Problem and trains ML classifiers in a supervised manner. It entails high-demanding expert knowledge for feature designs and specialized solvers for optimal solutions, making itself inflexible and hard to scale. By comparison, DeepACO leverages DRL and demands little expert knowledge to apply across COPs.

3 Preliminary on Ant Colony Optimization

The overall ACO pipeline is depicted in Fig. 1. We begin with defining a COP model and a pheromone model. They are prerequisites for implementing ACO algorithms.

COP model Generally, a COP model consists of, 1) a search space S defined over a set of discrete decision variables X i , i = 1 , . . . , n , where each decision variable X i takes a value from a finite set D i = { v 1 i , . . . , v | D i | i } ; 2) a set of constraints Ω that the decision variables must satisfy; and 3) an objective function f : S → R + 0 to minimize. A feasible solution s to a COP is a complete assignment of all decision variables that satisfies all the constraints in Ω .

Pheromone model A COP model defines a pheromone model in the context of ACO. Without loss of generality, a pheromone model is a construction graph that includes decision variables as nodes and solution components as edges [26]. Each solution component c ij , that represents the assignment of value v j i to decision variable X i , is associated with its pheromone trial τ ij and heuristic measure η ij . Both τ ij and η ij indicate how promising it is to include c ij in a solution. Typically, ACO uniformly initializes and iteratively updates pheromone trails, but predefines and fixes heuristic measures.

As a motivating example, for TSP, a instance can be directly converted into a fully connected construction graph, where city i becomes node i and solution component c ij represents visiting city j immediately after city i . Then, η ij is typically set to the inverse of the distance between city i and j . After defining a COP model and a pheromone model, we introduce an ACO iteration, usually consisting of solution constructions, optional LS refinement, and pheromone updates.

Solution construction and local search (optional) Biased by τ ij and η ij , an artificial ant constructs a solution s = { s t } n t =1 by traversing the construction graph. If an ant is located in node i at the t -th construction step ( s t -1 = i ) and has constructed a partial solution s <t = { s t } t -1 t =1 , the probability of

ACO

Expert knowledge

Solution construction and

local search (optional)

Heuristic measures

Pheromone trials

Input instance

...

Problem instances

Heuristic measures

Heuristic learner

...

Training

Problem

definition

Pheromone

model

Heuristic measures

Pheromone trials

Input instance

Heuristic learner

Inference

DeepACO

Reward

Update

Update

Problem

definition

Pheromone

model

Solution construction and

local search (optional)

Solution construction and

neural

local search (optional)

selecting node j as its next destination ( s t = j ) is typically given by

Here, ρ is a COP instance, N ( s <t ) is the set of feasible solution components given the partial solution, and α and β are the control parameters, which are consistently set to 1 in this work unless otherwise stated. To simplify the notations, we omit the dependence on ρ for τ , η , c , and N . Based on Eq. (1), constructing a complete solution requires an n -step graph traversal. The probability of generating s can be factorized as

After solution constructions, local search (LS) is optionally applied to refine the solutions.

Pheromone update After solution constructions, the pheromone update process evaluates solutions and adjusts the pheromone trails accordingly, i.e., it increases the pheromone trails of components in the superior solutions while decreasing those in the inferior ones. The detailed update rules can differ depending on the ACO variation used.

ACO intelligently explores the solution space by iterating the above process, eventually converging on (sub)optimal solutions. We refer the readers to [26] for more details.

4 Methodology

DeepACO is schematically presented in Fig. 1 wherein a comparison is made with ACO. It dispenses with expert knowledge and learns a set of stronger heuristic measures to guide the ACO evolution. DeepACO involves parameterizing the heuristic space (Section 4.1), optionally interleaving LS with neural-guided perturbation (Section 4.2), and training a heuristic learner across instances (Section 4.3). Additionally, we introduce three extended designs (Section 4.4) to boost exploration.

4.1 Parameterizing heuristic space

We introduce a heuristic learner, defined by a graph neural network (GNN) with trainable parameters θ , to parameterize the heuristic space. The heuristic learner maps an input COP instance ρ to its

heuristic measures η θ ( ρ ) , where we rewrite it as η θ for notation simplicity. It contains non-negative real values η ij ; θ associated with each solution component c ij , ∀ i ∈ { 1 , . . . , n } , ∀ j ∈ { 1 , . . . , | D i |} . DeepACO constructs solutions following Eq. (2) but biased by η θ :

In particular, we exploited the neural models recommended by Joshi et al. [48] and Qiu et al. [66]. It consists of a GNN backbone relying on anisotropic message passing and an edge gating mechanism, and a Multi-Layer Perceptron (MLP) decoder mapping the extracted edge features into heuristic measures. We defer the full details of this neural architecture to Appendix A.

4.2 Local search interleaved with neural-guided perturbation

In ACO, local search (LS) is optionally applied to refine the constructed solutions. However, LS is a myopic procedure in that it greedily accepts any altered solution with a lower objective value and easily gets trapped in local optima. In DeepACO, we intend the well-learned heuristic measures to indicate the global optimality of solution components. Leveraging such indicators and including solution components with greater global optimality can lead to better solutions eventually, if not immediately. Nevertheless, it is unrealistic to solely rely on the learned heuristic measures due to the inherent complexity of COPs.

Based on these considerations, we propose LS interleaved with neural-guided perturbation (NLS for short) in Algorithm 1. NLS interleaves LS aiming for a lower objective value and neural-guided perturbation biasing the learned optima. In each iteration, the first stage utilizes LS to repeatedly refine a solution until (potentially) reaching local optima. The second stage utilizes LS to slightly perturb the locally optimal solution toward gaining higher cumulative heuristic measures.

Algorithm 1 NLS

1: Input: A solution s ; an objective function f ; well-learned heuristic measures η θ ; a local search operator LS that takes three inputs: a solution to refine, the targeted objective function, and the maximum iterations before encountering local optima; the number of perturbation moves T p ; the number of NLS iterations T NLS 2: Output: The best improved solution s ∗ 3: s = LS ( s , f, + ∞ ) // Improve s until reaching a local optimum 4: s ∗ = copy ( s ) 5: for iter = 1 → T NLS do 6: s = LS ( s , 1 η θ , T p ) // Perturb s toward higher cumulative heuristic measures with T p moves 7: s = LS ( s , f, + ∞ ) 8: s ∗ = arg min( f ( s ∗ ) , f ( s )) 9: end for

4.3 Training heuristic learner

We train the heuristic learner across COP instances. The heuristic learner θ maps each instance ρ to its heuristic measures η θ . Then, we minimize the expected objective value of both constructed solutions and NLS-refined constructed solutions:

where W is a coefficient for balancing two terms. Intuitively, the first loss term encourages directly constructing optimal solutions, which is, however, often held off by the complexity of COP. The second term encourages constructing solutions most fit for NLS, and it could be easier to learn high-quality solutions if coupling end-to-end construction with NLS. Even so, solely relying on the second term leads to inefficient training due to the small quality variance of the NLS-refined solutions. As a result, we find it useful to implement both terms for training. Note that the NLS process itself does not involve gradient flow.

In practice, we deploy Ant Systems to construct solutions stochastically following Eq. (3) for estimating Eq. (4). The pheromone trials are fixed to 1 to ensure an unbiased estimation. We apply a REINFORCE-based [85] gradient estimator:

where b ( ρ ) and b NLS ( ρ ) are the average objective value of the constructed solutions and that of the NLS-refined constructed solutions, respectively.

4.4 Toward better exploration

The vanilla DeepACO fully exploits the underlying pattern of a problem and learns a set of aggressive heuristic measures (visualized in Appendix C.6) according to Eq. (4). Still, preserving exploration is of critical importance since COPs often feature many local optima [89]. To that end, we further present three extended designs to enable a better balance of exploration and exploitation. Note that they can generally be applied to heatmap-based NCO methods.

4.4.1 Multihead decoder

Multihead DeepACO implements m MLP decoders on the top of the GNN backbone. It aims to generate diverse heuristic measures to encourage exploring different optima in the solution space. A similar multihead design has been utilized for autoregressive solution construction [89]. We extend this idea to the non-autoregressive heuristic generation here.

For training, Multihead DeepACO utilizes Eq. (4) calculated individually and independently for m decoders and an extra Kullback-Leibler (KL) divergence loss expressed as

where ˜ η k ij ; θ is the heuristic measure of the solution component c ij output by the k -th decoder after row-wise normalization.

In the inference phase, Multihead DeepACO deploys m ant groups guided by their respective MLP decoders while the whole ant population shares the same set of pheromone trails.

4.4.2 Topk entropy loss

Entropy loss (reward) incentivizes agents to maintain the diversity of their actions. It is often used as a regularizer to encourage exploration and prevent premature convergence to suboptimal policies [68, 37, 51]. In the context of COP, however, most solution components are far from a reasonable choice for the next-step solution construction. Given this, we implement a topk entropy loss while optimizing Eq. (4), promoting greater uniformity only among the top k largest heuristic measures:

Here, K i is a set containing top k solution components of decision variable X i , and ¯ η ij ; θ are the heuristic measures normalized within the set.

4.4.3 Imitation loss

If expert-designed heuristic measures are available, we can additionally incorporate an imitation loss, enabling the heuristic learner to mimic expert-designed heuristics while optimizing Eq. (4). It holds two primary merits. First, the expert-designed heuristic measures are less aggressive than the learned ones. So, it can function as a regularization to maintain exploration. Second, the heuristic learner can acquire expert knowledge. Perfecting the imitation first can guarantee that the learned heuristics are at least not inferior to the expert-designed counterparts.

Accordingly, the imitation loss is formulated as

where ˜ η ∗ ij and ˜ η ij ; θ are expert-designed and learned heuristic measures, respectively, both after row-wise normalization.

5 Experimentation

5.1 Experimental setup

Benchmarks We evaluate DeepACO on eight representative COPs, including the Traveling Salesman Problem (TSP), Capacitated Vehicle Routing Problem (CVRP), Orienteering Problem (OP), Prize Collecting Traveling Salesman Problem (PCTSP), Sequential Ordering Problem (SOP), Single Machine Total Weighted Tardiness Problem (SMTWTP), Resource-Constrained Project Scheduling Problem (RCPSP), and Multiple Knapsack Problem (MKP). They cover routing, assignment, scheduling, and subset COP types. Their definitions and setups are given in Appendix D.

Hardware Unless otherwise stated, we conduct experiments on 48-core Intel(R) Xeon(R) Platinum 8350C CPU and an NVIDIA GeForce RTX 3090 Graphics Card.

5.2 DeepACO as an enhanced ACO algorithm

In this part, we do not apply NLS and set W in Eq. (4) to 0 for training. It only entails minutes of training to provide substantial neural enhancement as shown in Fig. 2 (also see Appendix C.4). On the other hand, the extra inference time for DeepACO is negligible; for example, it takes less than 0.001 seconds for a TSP100.

DeepACO for fundamental ACO algorithms We implement DeepACO based on three fundamental ACO meta-heuristics: Ant System [25], Elitist Ant System [25], and MAX-MIN Ant System [76]. They are widely recognized as the basis for many SOTA ACO variations [44, 2, 5, 3, 93]. We compare the heuristic measures learned by DeepACO to the expert-designed ones (detailed separately for eight COPs in Appendix D) on 100 held-out test instances for each benchmark COP. The results in Fig. 2 show that DeepACO can consistently outperform its ACO counterparts, verifying the universal neural enhancement DeepACO can bring. We defer the results on more COP scales to Appendix C.1.

DeepACO for advanced ACO algorithms We also apply DeepACO to Adaptive Elitist Ant System (AEAS) [2], a recent ACO algorithm with problem-specific adaptations. In Fig. 3, we plot the evolution curves of AEAS based on the original and learned heuristic measures, respectively, and DeepACO shows clearly better performance. In light of this, we believe that it is promising to exploit DeepACO for designing new ACO SOTAs.

0

1000

2000

3000

4000

5000

# of evaluations

8

10

12

14

16

Avg. Obj.

TSP100

0

500

1000

1500

2000

# of evaluations

15

20

25

30

35

40

Avg. Obj.

CVRP100

0

500

1000

1500

2000

# of evaluations

17.5

20.0

22.5

25.0

27.5

30.0

Avg. Obj.

0

500

1000

1500

2000

# of evaluations

10

15

20

25

30

35

40

Avg. Obj.

PCTSP100

0

500

1000

1500

2000

# of evaluations

34

36

38

40

42

Avg. Obj.

SOP50

0

500

1000

1500

2000

# of evaluations

0

5

10

15

20

25

30

Avg. Obj.

SMTWTP100

0

500

1000

1500

2000

# of evaluations

175

180

185

190

195

Avg. Obj.

RCPSP120

0

500

1000

1500

2000

# of evaluations

40.0

42.5

45.0

47.5

50.0

52.5

55.0

Avg. Obj.

MKP300

ACO (AS)

ACO (MAX-MIN)

DeepACO (AS)

DeepACO (MAX-MIN)

0

500

1000

1500

2000

# of evaluations

4.5

5.0

5.5

6.0

Avg. Obj.

CVRP20

0

500

1000

1500

2000

# of evaluations

15.0

17.5

20.0

22.5

25.0

27.5

Avg. Obj.

CVRP100

0

500

1000

1500

2000

# of evaluations

60

80

100

120

Avg. Obj.

CVRP500

ACO (AEAS)

DeepACO (AEAS)

DeepACO for different pheromone models The pheromone model used in ACO can vary, even for the same problem [63]. Following the naming convention in [63], we use PH suc to denote the pheromone model that captures the successively chosen items and PH items the one that directly captures how good each item is. We extend DeepACO from PH suc to PH items using a Transformer [79] encoder equipped with an MLP decoder. We showcase such flexibility of DeepACO on MKP in Fig. 7, where DeepACO using PH items can still outperform its ACO counterparts. It validates that DeepACO can be readily extended to variations of pheromone models using properly aligned neural models. We defer further discussions to Appendix A and C.5.

DeepACO for better robustness to hyperparameter choice Increasing the robustness of ACO to hyperparameter choices has long been an important research topic [84, 92]. In Fig. 4, we adjust two important hyperparameters, i.e., Alpha that controls the transition probability of solution construction and Decay that controls the pheromone updates, evaluate DeepACO (AS) and ACO (AS) on the TSP100 test dataset, and plot their best objective values within the evolution of 4K evaluations. The results indicate that DeepACO is more robust to the hyperparameter choice given its much lower color variance. We thereby argue that data-driven training can also dispense with high-demanding expert knowledge for hyperparameter tuning.

DeepACO for real-world instances We draw all 49 real-world symmetric TSP instances featuring EUC_2D and containing less than 1K nodes from TSPLIB. We infer instances with n < 50 nodes using the model trained on TSP20, those with 50 ≤ n < 200 nodes using the model trained on TSP100, and the rest using the model trained on TSP500. The evolution curves of DeepACO and ACO are shown in Fig. 5, suggesting that DeepACO can consistently outperform its ACO counterparts even when generalizing across scales and distributions.

Alpha=0.5

Alpha=1

Alpha=2

Alpha=3

Alpha=5

Decay=0.1

Decay=0.5

Decay=0.9

Decay=0.95

Decay=0.99

DeepACO

Alpha=0.5

Alpha=1

Alpha=2

Alpha=3

Alpha=5

Decay=0.1

Decay=0.5

Decay=0.9

Decay=0.95

Decay=0.99

ACO

8.5

9.0

9.5

10.0

10.5

11.0

11.5

12.0

0

500

1000

1500

2000

# of evaluations

10

12

14

16

18

20

22

Avg. Obj.

TSPLIB

ACO (AS)

ACO (MAX-MIN)

DeepACO (AS)

DeepACO (MAX-MIN)

5.3 DeepACO as an NCO method

Comparison on routing problems As shown in Table 1, DeepACO demonstrates competitive performance against the SOTA NCO methods for TSP. Here, we apply the generic 2-opt for NLS ( T NLS = 10 , T p = 20 ) and set W in Eq. (4) to 9. Note that most NCO baselines are specialized for TSP or routing problems, while DeepACO is a general-purpose meta-heuristic and validated across eight different COPs. In addition, the results of ablation studies in Table 2 further validate our designs; that is, both neural-guided perturbation and training with LS effectively strengthen DeepACO. More results on TSP100 and CVRP are given in Appendix C.2.

For heatmaps with better exploration We implement the three extended designs introduced in Section 4.4 without LS and compare their evolution curves with vanilla DeepACO in Fig. 6. We apply 4 decoder heads for Multihead DeepACO and set K to 5 for the top-K entropy loss. The coefficients for the additional loss terms (i.e., L KL , L H , and L I ) are set to 0.05, 0.05, and 0.02, respectively. The results indicate that these designs lead to better performance on TSP20, 50, and 100 since small-scale COPs typically desire more exploration.

For enhancing heatmap decoding with pheromone update Though we can conduct pure solution samplings similar to [47, 66, 78] given the learned heuristic measures, we demonstrate the superiority of ACO evolution (without NLS) with self-learning pheromone update in Fig. 8. Furthermore, coupling heatmaps with ACO can benefit from extensive ACO research, such as advanced ACO variations, algorithmic hybridizations, and seamless incorporation of local search operators.

0

1000

2000

3000

4000

5000

# of evaluations

3.82

3.84

3.86

3.88

3.90

3.92

3.94

Average obj.

TSP20

0

1000

2000

3000

4000

5000

# of evaluations

5.8

5.9

6.0

6.1

6.2

6.3

6.4

Average obj.

TSP50

0

1000

2000

3000

4000

5000

# of evaluations

8.2

8.4

8.6

8.8

9.0

Average obj.

TSP100

Vanilla DeepACO

With mutihead decoder

With entropy loss

With imitation loss

0

200

400

600

800

1000

# of evaluations

47.5

50.0

52.5

55.0

57.5

60.0

62.5

65.0

Avg. Obj.

MKP300

0

200

400

600

800

1000

# of evaluations

75

80

85

90

95

100

Avg. Obj.

MKP500

ACO (AS)

ACO (MAX-MIN)

DeepACO (AS)

DeepACO (MAX-MIN)

0

500

1000

1500

2000

# of evaluations

3.82

3.84

3.86

3.88

3.90

3.92

3.94

Avg. Obj.

TSP20

0

1000

2000

3000

4000

5000

# of evaluations

8.2

8.4

8.6

8.8

Avg. Obj.

TSP100

DeepACO (AS)

DeepACO (MAX-MIN)

Sampling

6 Conclusion and limitation

In this work, we propose DeepACO to provide universal neural enhancement for ACO algorithms and automate heuristic designs for future ACO applications. We show that DeepACO performs

consistently better than its ACO counterparts and is on par with the specialized NCO methods. For the Operations Research (OR) community, DeepACO demonstrates a promising avenue toward leveraging deep reinforcement learning for algorithmic enhancement and design automation. For the Machine Learning (ML) community, DeepACO presents a versatile and adaptable NCO framework that can seamlessly integrate with SOTA ACO and EA techniques, including improved rules for solution construction and pheromone update [76, 11], algorithmic hybridizations [67, 91], and incorporation of sophisticated local search operators [3, 93].

However, DeepACO is potentially limited by compressing all learned heuristic information into an n × n matrix of heuristic measures. Due to the complexity of COPs and this restricted way of expressing problem patterns, DeepACO may fail to produce close-to-optimal solutions when not incorporating LS components. To address such limitations, we plan to investigate 3D or dynamic heuristic measures in our future work.

Acknowledgments and Disclosure of Funding

The authors appreciate the helpful discussions with Yu Hong, Juntao Li, and anonymous reviewers. Yu Hu, Jiusi Yin, and Tao Yu also contributed to this work. This work was supported by the National Natural Science Foundation of China (NSFC) [grant numbers 61902269, 52074185]; the National Key R&D Program of China [grant number 2018YFA0701700]; the Undergraduate Training Program for Innovation and Entrepreneurship, Soochow University [grant number 202210285001Z, 202310285041Z]; the Priority Academic Program Development of Jiangsu Higher Education Institutions, China; and Provincial Key Laboratory for Computer Information Processing Technology, Soochow University [grant number KJS1938].

References

D. Abitz, T. Hartmann, and M. Middendorf. A weighted population update rule for paco applied to the single machine total weighted tardiness problem. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference , pages 4-12, 2020.

A. Abuhamdah. Adaptive elitist-ant system for solving combinatorial optimization problems. Applied Soft Computing , 105:107293, 2021.

Z. A. Ali, H. Zhangang, and W. B. Hang. Cooperative path planning of multiple uavs by using max-min ant colony optimization along with cauchy mutant operator. Fluctuation and Noise Letters , 20(01):2150002, 2021.

Z. N. Ansari and S. D. Daxini. A state-of-the-art review on meta-heuristics application in remanufacturing. Archives of Computational Methods in Engineering , 29(1):427-470, 2022.

E. Babaee Tirkolaee, I. Mahdavi, M. M. Seyyed Esfahani, and G.-W. Weber. A hybrid augmented ant colony optimization for the multi-trip capacitated arc routing problem under fuzzy demands for urban solid waste management. Waste management & research , 38(2):156-172, 2020.

I. Bello, H. Pham, Q. V. Le, M. Norouzi, and S. Bengio. Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940 , 2016.

Y. Bengio, A. Lodi, and A. Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021.

I. BenMansour. An effective hybrid ant colony optimization for the knapsack problem using multi-directional search. SN Computer Science , 4(2):164, 2023.

F. Berto, C. Hua, J. Park, M. Kim, H. Kim, J. Son, H. Kim, J. Kim, and J. Park. Rl4co: an extensive reinforcement learning for combinatorial optimization benchmark. arXiv preprint arXiv:2306.17100 , 2023.

J. Bi, Y . Ma, J. Wang, Z. Cao, J. Chen, Y . Sun, and Y . M. Chee. Learning generalizable models for vehicle routing problems via knowledge distillation. arXiv preprint arXiv:2210.07686 , 2022.

C. Blum. Beam-aco-hybridizing ant colony optimization with beam search: An application to open shop scheduling. Computers & Operations Research , 32(6):1565-1591, 2005.

A. Bogyrbayeva, M. Meraliyev, T. Mustakhov, and B. Dauletbayev. Learning to solve vehicle routing problems: A survey. arXiv preprint arXiv:2205.02453 , 2022.

X. Bresson and T. Laurent. The transformer network for the traveling salesman problem. arXiv preprint arXiv:2103.03012 , 2021.

E. K. Burke, M. Gendreau, M. Hyde, G. Kendall, G. Ochoa, E. Özcan, and R. Qu. Hyperheuristics: A survey of the state of the art. Journal of the Operational Research Society , 64: 1695-1724, 2013.

J. Cai, P. Wang, S. Sun, and H. Dong. A dynamic space reduction ant colony optimization for capacitated vehicle routing problem. Soft Computing , 26(17):8745-8756, 2022.

J. Chen, Z. Zong, Y. Zhuang, H. Yan, D. Jin, and Y. Li. Reinforcement learning for practical express systems with mixed deliveries and pickups. ACMTransactions on Knowledge Discovery from Data (TKDD) , 2022.

X. Chen and Y. Tian. Learning to perform local rewriting for combinatorial optimization. Advances in Neural Information Processing Systems , 32, 2019.

H. Cheng, H. Zheng, Y. Cong, W. Jiang, and S. Pu. Select and optimize: Learning to aolve large-scale tsp instances. In International Conference on Artificial Intelligence and Statistics , pages 1219-1231. PMLR, 2023.

A. Chmiela, E. Khalil, A. Gleixner, A. Lodi, and S. Pokutta. Learning to schedule heuristics in branch and bound. Advances in Neural Information Processing Systems , 34:24235-24246, 2021.

J. Choo, Y.-D. Kwon, J. Kim, J. Jae, A. Hottung, K. Tierney, and Y. Gwon. Simulation-guided beam search for neural combinatorial optimization. arXiv preprint arXiv:2207.06190 , 2022.

P. R. d O Costa, J. Rhuggenaath, Y. Zhang, and A. Akcay. Learning 2-opt heuristics for the traveling salesman problem via deep reinforcement learning. In Asian Conference on Machine Learning , pages 465-480. PMLR, 2020.

P. R. d O Costa, J. Rhuggenaath, Y. Zhang, and A. Akcay. Learning 2-opt heuristics for the traveling salesman problem via deep reinforcement learning. In Asian conference on machine learning , pages 465-480. PMLR, 2020.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

M. Dorigo and T. Stützle. Ant colony optimization: overview and recent advances . Springer, 2019.

M. Dorigo, V. Maniezzo, and A. Colorni. Ant system: optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 26(1): 29-41, 1996.

M. Dorigo, M. Birattari, and T. Stutzle. Ant colony optimization. IEEE computational intelligence magazine , 1(4):28-39, 2006.

J. H. Drake, A. Kheiri, E. Özcan, and E. K. Burke. Recent advances in selection hyper-heuristics. European Journal of Operational Research , 285(2):405-428, 2020.

S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. (arXiv:1702.03118), Nov 2017. doi: 10.48550/arXiv. 1702.03118. URL http://arxiv.org/abs/1702.03118 . arXiv:1702.03118 [cs].

M. Fan, Y. Wu, T. Liao, Z. Cao, H. Guo, G. Sartoretti, and G. Wu. Deep reinforcement learning for uav routing in the presence of multiple charging stations. IEEE Transactions on Vehicular Technology , 2022.

S. Fidanova. Hybrid ant colony optimization algorithm for multiple knapsack problem. In 2020 5th IEEE International Conference on Recent Advances and Innovations in Engineering (ICRAIE) , pages 1-5. IEEE, 2020.

M. Fischetti, J. J. S. Gonzalez, and P. Toth. Solving the orienteering problem through branchand-cut. INFORMS Journal on Computing , 10(2):133-148, 1998.

Z.-H. Fu, K.-B. Qiu, and H. Zha. Generalize a small pre-trained model to arbitrarily large tsp instances. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7474-7482, 2021.

W. Gao, Q. Tang, B. Ye, Y. Yang, and J. Yao. An enhanced heuristic ant colony optimization for mobile robot path planning. Soft Computing , 24:6139-6150, 2020.

A. I. Garmendia, J. Ceberio, and A. Mendiburu. Neural combinatorial optimization: a new player in the field. arXiv preprint arXiv:2205.01356 , 2022.

Y. L. Goh, W. S. Lee, X. Bresson, T. Laurent, and N. Lim. Combining reinforcement learning and optimal transport for the traveling salesman problem. arXiv preprint arXiv:2203.00903 , 2022.

A. Gonzalez-Pardo, J. Del Ser, and D. Camacho. Comparative study of pheromone control heuristics in aco algorithms for solving rcpsp problems. Applied Soft Computing , 60:241-255, Nov 2017. ISSN 1568-4946. doi: 10.1016/j.asoc.2017.06.042.

T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

K. Helsgaun. An extension of the lin-kernighan-helsgaun tsp solver for constrained traveling salesman and vehicle routing problems. Roskilde: Roskilde University , pages 24-50, 2017.

A. Hottung, Y.-D. Kwon, and K. Tierney. Efficient active search for combinatorial optimization problems. arXiv preprint arXiv:2106.05126 , 2021.

Q. Hou, J. Yang, Y. Su, X. Wang, and Y. Deng. Generalize learned heuristics to solve large-scale vehicle routing problems in real-time. In The Eleventh International Conference on Learning Representations , 2022.

B. Hudson, Q. Li, M. Malencia, and A. Prorok. Graph neural network guided local search for the traveling salesperson problem. arXiv preprint arXiv:2110.05291 , 2021.

S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. (arXiv:1502.03167), Mar 2015. doi: 10.48550/arXiv.1502.03167. URL http://arxiv.org/abs/1502.03167 . arXiv:1502.03167 [cs].

G. M. Jaradat, A. Al-Badareen, M. Ayob, M. Al-Smadi, I. Al-Marashdeh, M. Ash-Shuqran, and E. Al-Odat. Hybrid elitist-ant system for nurse-rostering problem. Journal of King Saud University-Computer and Information Sciences , 31(3):378-384, 2019.

Y. Jiang, Y. Wu, Z. Cao, and J. Zhang. Learning to solve routing problems via distributionally robust optimization. arXiv preprint arXiv:2202.07241 , 2022.

Y. Jin, Y. Ding, X. Pan, K. He, L. Zhao, T. Qin, L. Song, and J. Bian. Pointerformer: Deep reinforced multi-pointer transformer for the traveling salesman problem. arXiv preprint arXiv:2304.09407 , 2023.

C. K. Joshi, T. Laurent, and X. Bresson. An efficient graph convolutional network technique for the travelling salesman problem. arXiv preprint arXiv:1906.01227 , 2019.

C. K. Joshi, Q. Cappart, L.-M. Rousseau, and T. Laurent. Learning the travelling salesperson problem requires rethinking generalization. Constraints , pages 1-29, 2022.

S. Kashef and H. Nezamabadi-pour. An advanced aco algorithm for feature subset selection. Neurocomputing , 147:271-279, 2015.

E. Khalil, H. Dai, Y. Zhang, B. Dilkina, and L. Song. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems , 30, 2017.

M. Kim, J. Park, et al. Learning collaborative policies to solve np-hard routing problems. Advances in Neural Information Processing Systems , 34:10418-10430, 2021.

M. Kim, J. Park, and J. Park. Sym-nco: Leveraging symmetricity for neural combinatorial optimization. arXiv preprint arXiv:2205.13209 , 2022.

R. Kolisch and A. Sprecher. Psplib - a project scheduling problem library: Or software - orsep operations research software exchange program. European Journal of Operational Research , 96(1):205-216, Jan 1997. ISSN 0377-2217. doi: 10.1016/S0377-2217(96)00170-1.

W. Kool, H. Van Hoof, and M. Welling. Attention, learn to solve routing problems! arXiv preprint arXiv:1803.08475 , 2018.

W. Kool, H. van Hoof, J. Gromicho, and M. Welling. Deep policy dynamic programming for vehicle routing problems. In International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research , pages 190-213. Springer, 2022.

Y.-D. Kwon, J. Choo, B. Kim, I. Yoon, Y. Gwon, and S. Min. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances in Neural Information Processing Systems , 33:21188-21198, 2020.

G. Leguizamon and Z. Michalewicz. A new version of ant system for subset problems. In Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406) , volume 2, pages 1459-1464. IEEE, 1999.

J. Levine and F. Ducatelle. Ant colony optimization and local search for bin packing and cutting stock problems. Journal of the Operational Research society , 55(7):705-716, 2004.

S. Li, Z. Yan, and C. Wu. Learning to delegate for large-scale vehicle routing. Advances in Neural Information Processing Systems , 34:26198-26211, 2021.

Y. Ma, J. Li, Z. Cao, W. Song, L. Zhang, Z. Chen, and J. Tang. Learning to iteratively solve routing problems with dual-aspect collaborative transformer. Advances in Neural Information Processing Systems , 34:11096-11107, 2021.

N. Mazyavkina, S. Sviridov, S. Ivanov, and E. Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research , 134:105400, 2021.

D. Merkle, M. Middendorf, and H. Schmeck. Ant colony optimization for resource-constrained project scheduling. IEEE Transactions on Evolutionary Computation , 6(4):333-346, Aug 2002. ISSN 1941-0026. doi: 10.1109/TEVC.2002.802450.

J. Montgomery, M. Randall, and T. Hendtlass. Solution bias in ant colony optimisation: Lessons for selecting pheromone models. Computers & Operations Research , 35(9):2728-2749, 2008.

M. Nazari, A. Oroojlooy, L. Snyder, and M. Takác. Reinforcement learning for solving the vehicle routing problem. Advances in neural information processing systems , 31, 2018.

X. Pan, Y. Jin, Y. Ding, M. Feng, L. Zhao, L. Song, and J. Bian. H-tsp: Hierarchically solving the large-scale travelling salesman problem. arXiv preprint arXiv:2304.09395 , 2023.

R. Qiu, Z. Sun, and Y. Yang. Dimes: A differentiable meta solver for combinatorial optimization problems. arXiv preprint arXiv:2210.04123 , 2022.

N. Rokbani, R. Kumar, A. Abraham, A. M. Alimi, H. V. Long, I. Priyadarshini, and L. H. Son. Bi-heuristic ant colony optimization-based approaches for traveling salesman problem. Soft Computing , 25:3775-3794, 2021.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

E. Singh and N. Pillay. A study of ant-based pheromone spaces for generation constructive hyper-heuristics. Swarm and Evolutionary Computation , 72:101095, 2022.

R. Skinderowicz. Population-based ant colony optimization for sequential ordering problem. In Computational Collective Intelligence: 7th International Conference, ICCCI 2015, Madrid, Spain, September 21-23, 2015, Proceedings, Part II , pages 99-109. Springer, 2015.

R. Skinderowicz. An improved ant colony system for the sequential ordering problem. Computers & Operations Research , 86:1-17, 2017.

R. Skinderowicz. Improving ant colony optimization efficiency for solving large tsp instances. Applied Soft Computing , 120:108653, 2022.

A. Slowik and H. Kwasnicka. Evolutionary algorithms and their applications to engineering problems. Neural Computing and Applications , 32:12363-12379, 2020.

S. Sohrabi, K. Ziarati, and M. Keshtkaran. Acs-ophs: Ant colony system for the orienteering problem with hotel selection. EURO Journal on Transportation and Logistics , 10:100036, 2021.

W. Song, N. Mi, Q. Li, J. Zhuang, and Z. Cao. Stochastic economic lot scheduling via selfattention based deep reinforcement learning. IEEE Transactions on Automation Science and Engineering , 2023.

T. Stützle and H. H. Hoos. Max-min ant system. Future generation computer systems , 16(8): 889-914, 2000.

Y. Sun, S. Wang, Y. Shen, X. Li, A. T. Ernst, and M. Kirley. Boosting ant colony optimization via solution prediction and machine learning. Computers & Operations Research , 143:105769, 2022.

Z. Sun and Y. Yang. Difusco: Graph-based diffusion solvers for combinatorial optimization. arXiv preprint arXiv:2302.08224 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

T. Vidal. Hybrid genetic search for the cvrp: Open-source implementation and swap* neighborhood. Computers & Operations Research , 140:105643, 2022.

T. Vidal, T. G. Crainic, M. Gendreau, N. Lahrichi, and W. Rei. A hybrid genetic algorithm for multidepot and periodic vehicle routing problems. Operations Research , 60(3):611-624, 2012. doi: 10.1287/opre.1120.1048.

O. Vinyals, M. Fortunato, and N. Jaitly. Pointer networks. Advances in neural information processing systems , 28, 2015.

C. Wang, Z. Yu, S. McAleer, T. Yu, and Y. Yang. Asp: Learn a universal neural solver! arXiv preprint arXiv:2303.00466 , 2023.

Y. Wang, L. Wang, G. Chen, Z. Cai, Y. Zhou, and L. Xing. An improved ant colony optimization algorithm to the periodic vehicle routing problem with time window and service choice. Swarm and Evolutionary Computation , 55:100675, Jun 2020. ISSN 2210-6502. doi: 10.1016/j.swevo. 2020.100675.

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning , pages 5-32, 1992.

Y. Wu, W. Song, Z. Cao, and J. Zhang. Learning large neighborhood search policy for integer programming. Advances in Neural Information Processing Systems , 34:30075-30087, 2021.

Y. Wu, W. Song, Z. Cao, J. Zhang, and A. Lim. Learning improvement heuristics for solving routing problems.. IEEE transactions on neural networks and learning systems , 2021.

Y. Wu, J. Zhou, Y. Xia, X. Zhang, Z. Cao, and J. Zhang. Neural airport ground handling. arXiv preprint arXiv:2303.02442 , 2023.

L. Xin, W. Song, Z. Cao, and J. Zhang. Multi-decoder attention model with embedding glimpse for solving vehicle routing problems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12042-12049, 2021.

L. Xin, W. Song, Z. Cao, and J. Zhang. Neurolkh: Combining deep learning model with linkernighan-helsgaun heuristic for solving the traveling salesman problem. Advances in Neural Information Processing Systems , 34:7472-7483, 2021.

H. Ye, H. Liang, T. Yu, J. Wang, and H. Guo. A bi-population clan-based genetic algorithm for heat pipe-constrained component layout optimization. Expert Systems with Applications , 213: 118881, 2023.

X. Zhou, H. Ma, J. Gu, H. Chen, and W. Deng. Parameter adaptation-based ant colony optimization with dynamic hybrid mechanism. Engineering Applications of Artificial Intelligence , 114:105139, Sep 2022. ISSN 0952-1976. doi: 10.1016/j.engappai.2022.105139.

Y. Zhou, X. Liu, S. Hu, Y. Wang, and M. Yin. Combining max-min ant system with effective local search for solving the maximum set k-covering problem. Knowledge-Based Systems , 239: 108000, 2022.

Z. Zong, H. Wang, J. Wang, M. Zheng, and Y. Li. Rbg: Hierarchically solving large-scale routing problems in logistic systems via reinforcement learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4648-4658, 2022.

Z. Zong, M. Zheng, Y. Li, and D. Jin. Mapdp: Cooperative multi-agent reinforcement learning to solve pickup and delivery problems. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9980-9988, 2022.

DeepACO: Neural-enhanced Ant Systems for Combinatorial Optimization (Appendix)

A Neural architecture

A.1 GNN

For pheromone model PH suc , we employ the neural architecture recommended by Joshi et al. [48] and Qiu et al. [66] as the GNN backbone, which relies on an anisotropic message passing scheme and an edge gating mechanism. The GNN we use has 12 layers. For the l -th layer, we denote the node feature of v i as h l i and the edge feature of edge < i, j > as e l ij . The propagated features are defined as

where U l , V l , P l , Q l , R l ∈ R d × d are learnable parameters, α denotes the activation function, BN denotes batch normalization [43], A j ∈N i denotes the aggregation operation over the neighbourhood of v i , σ is the sigmoid function, and ⊙ is the Hadamard product. In this paper, we define α as SiLU [28] and A as mean pooling.

We map the extracted edge features into real-valued heuristic measures using a 3-layer MLP with skip connections [38] to the embedded node feature h 12 i . We apply SiLU [28] as the activation function except for the output layer, where the sigmoid function is utilized to obtain normalized outputs.

A.2 Transformer

For pheromone model PH items , the architecture mostly follows the transformer encoder ( num _ hidden _ layers = 3 , hidden _ size = 32 , num _ attention _ heads = 2 ) but deprecates its positional encoding. On top of it, we add position-wise feedforward layers ( num _ hidden _ layers = 3 , hidden _ size = 32 ) that map the hidden representations of each solution component into its real-valued heuristic measure. Overall, it is similar to the neural networks used in [23, 35].

B Details of NLS

In NLS, neural-guided perturbation does not involve a new neural network other than that introduced in Section 4.1. It directly leverages the learned heuristic measures to guide the perturbation process. Specifically, each time a solution reaches local optima, it gets perturbed with local search (LS) which iteratively maximizes the heuristic measures of its solution components. Note that NLS can be implemented using an arbitrary LS operator.

Take TSP as an example. With 2-opt as the LS operator, NLS alternates between (a) optimizing a solution to minimize its tour length and (b) perturbing it to maximize the total heuristic measures of its edges. The two processes are similar, except that we use the inverse of heuristic measures in (b) while the distance matrix in (a) as indicators of good edges (line 6 in Algorithm 1). Our training is customized for NLS. The heuristic learner is trained to minimize the expected TSP length of the NLS-refined sampled solutions.

C Extended results

C.1 DeepACO for more COP scales

We evaluate DeepACO on more COP scales and present the results in Fig. 9. The evolution curves demonstrate that DeepACO can provide consistent neural enhancement across COP scales.

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

3.8

3.9

4.0

4.5

5.0

5.5

Avg. Obj.

TSP20

0

1000

2000

3000

4000

5000

# of evaluations

8

10

12

14

16

Avg. Obj.

TSP100

0

1000

2000

3000

4000

5000

# of evaluations

20

25

30

35

40

45

50

55

Avg. Obj.

TSP500

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Avg. Obj.

CVRP20

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

15

20

25

30

35

Avg. Obj.

CVRP100

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

80

100

120

140

160

180

Avg. Obj.

CVRP500

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

17.5

20.0

22.5

25.0

27.5

30.0

Avg. Obj.

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

20

25

30

35

40

45

50

55

Avg. Obj.

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

30

40

50

60

70

80

Avg. Obj.

OP300

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

4

5

6

7

8

9

10

11

Avg. Obj.

PCTSP20

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

10

15

20

25

30

35

40

Avg. Obj.

PCTSP100

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

25

50

75

100

125

150

175

Avg. Obj.

PCTSP500

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

13.0

13.5

14.0

14.5

15.0

15.5

Avg. Obj.

SOP20

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

34

36

38

40

42

Avg. Obj.

SOP50

0

1000

2000

3000

4000

5000

# of evaluations

70

75

80

85

Avg. Obj.

SOP100

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

0

1

2

3

4

5

6

Avg. Obj.

SMTWTP50

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

0

5

10

15

20

25

30

Avg. Obj.

SMTWTP100

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

0

200

400

600

800

1000

Avg. Obj.

SMTWTP500

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

58

59

60

61

62

63

64

Avg. Obj.

RCPSP30

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

88

90

92

94

96

98

100

Avg. Obj.

RCPSP60

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

175

180

185

190

195

Avg. Obj.

RCPSP120

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

45.0

47.5

50.0

52.5

55.0

57.5

60.0

Avg. Obj.

MKP300

0

250

500

750

1000

1250

1500

1750

2000

# of evaluations

60

65

70

75

80

85

Avg. Obj.

MKP500

ACO (AS)

ACO (MAX-MIN)

DeepACO (AS)

DeepACO (MAX-MIN)

C.2 TSP and CVRP

Table 3 and 4 gather the comparative experimental results on more diverse routing tasks, i.e., TSP100, CVRP100, 400, 1000, and 2000. They demonstrate DeepACO's consistent and competitive performance. 'Time' in the tables refers to the duration needed to solve a single instance, and the results of the baselines are drawn from their respective papers or [41]. NLS for CVRP is based on the local search strategy in the HGS-CVRP algorithm [80, 81].

NLS for CVRP The local search strategy proposed by Vidal [80] (denoted as LS HGS ) consists of searching among 5 collections of neighborhoods, including SWAP, RELOCATE, 2-opt, 2-opt*, and SWAP* neighborhood. NLS for CVRP follows Algorithm 1 and apply LS HGS as the local search strategy. In practice, we implement the code provided by Vidal [80] and set T NLS to 1 . To be noted that LS HGS occasionally produces infeasible solutions, and such solutions are discarded in NLS. We refer the reader to [80] for more details.

C.3 Generalizability of DeepACO

Across COPs The generalizability of DeepACO across COPs is rooted in the generalizability of the ACO meta-heuristic and the feasibility to represent many COPs' solutions using binary variables. In our paper, we evaluated DeepACO across 8 diverse COPs, including routing, assignment, scheduling, and subset COP types. Furthermore, we extend DeepACO to the Bin Packing Problem (BPP), a grouping problem, aiming at optimally splitting items into groups. We follow the experimental setup in [58] and demonstrate the consistent neural enhancement provided by DeepACO in Table 5.

Across scales Table 6 compares the performance of ACO, DeepACO trained on TSP100, and DeepACO trained on the respective test scale, all implementing vanilla LS (instead of NLS to ensure the same execution time for DeepACO and ACO). The results show that DeepACO still outperforms its ACO counterpart even with a significant distributional shift (i.e., from TSP100 to TSP1000).

C.4 Training duration

As presented in Table 7, DeepACO entails only minutes of training to provide substantial neural enhancement in Fig. 2. It is mainly because our training hybridizes 1) heatmap-based on-policy sampling without costly step-by-step neural decoding [66], and 2) parallel multi-start sampling leveraging the solution symmetries [56, 52].

C.5 Flexibility for a different pheromone model

The pheromone model used in ACO can vary, even for the same problem [63]. The pheromone model used in this work models the successive relationships of the nodes in a COP graph, which applies to most COPs but is by no means exhaustive. This section aims to show the flexibility of DeepACO for different pheromone models.

Here, we use the Multiple Knapsack Problem (MKP) (formally defined in Appendix D.8) as an example. In Section 5.2, we mostly use the pheromone model PH suc that models successively chosen items [63], where the pheromone trails are deposited on the edges connecting two items. One alternative, i.e., PH items , is to directly model how good each item is, where the pheromone trails are deposited on the items, i.e., the graph nodes. On the one hand, PH suc is reasonable because the quality of an item must be considered in conjunction with how it is combined with other items. On the other hand, PH items makes sense because a solution to MKP is order-invariant. Both pheromone models have their own advantages [63] and are used in the recent literature [30, 8].

Isomorphic GNN [48] is applied in this work to PH suc due to its ability to extract edge features. To apply DeepACO to variations of pheromone models, it is necessary to use a neural model properly aligned with the pheromone model. As a showcase, we use a lightweight Transformer [79] encoder (without positional encoding) with an MLP decoder on top of it to model PH items for MKP. Under the same experimental setup as in Section 5.1, we compare DeepACO with its ACO counterparts using PH items for MKP. The results presented in Fig. 7 demonstrate that DeepACO using PH items can also outperform its ACO counterparts. It validates that DeepACO can be readily extended to variations of pheromone models using properly aligned neural models.

C.6 Visualization of the learned heuristic

We visualize the learned heuristic in Fig. 10. After the graph sparsification (see Appendix D), we scatter the remaining solution components (i.e., the TSP edges in this case) in a unit square by normalizing (in a row-wise fashion) their manually designed and learned heuristic measures. We can observe that the solution components with a large learned heuristic measure do not necessarily

0.0

0.2

0.4

0.6

0.8

1.0

1

d

0.0

0.2

0.4

0.6

0.8

1.0

Learned heuristic measure

TSP20

0.0

0.2

0.4

0.6

0.8

1.0

1

d

0.0

0.2

0.4

0.6

0.8

1.0

Learned heuristic measure

TSP100

0.0

0.2

0.4

0.6

0.8

1.0

1

d

0.0

0.2

0.4

0.6

0.8

1.0

Learned heuristic measure

TSP500

obtain an equivalently large 1 d , and vice versa. In particular, DeepACO learns a set of more aggressive heuristic measures, where most solution components are associated with normalized measures close to zero.

D Benchmark problems

D.1 Traveling Salesman Problem

Definition Traveling Salesman Problem (TSP) seeks the shortest route that visits a given set of nodes and returns to the starting node, visiting each node exactly once.

Instance generation Nodes are sampled uniformly from [0 , 1] 2 unit.

Model inputs ATSP instance corresponds to a complete graph. Node features are their coordinates, and edge attributes are relative distances between nodes.

Graph sparsification All nodes are only adjacent to their k nearest neighbors to exclude highly unpromising solution components and speed up computation. The same graph sparsification is conducted for ACO baselines to ensure a fair comparison. k is set to 10 for TSP20, 20 for TSP50, 20 for TSP100, and 50 for TSP500.

Baseline heuristic The baseline heuristic measure is the inverse of edge length [72].

D.2 Capacitated Vehicle Routing Problem

Definition Capacitated Vehicle Routing Problem (CVRP) is a natural extension of TSP that involves multiple vehicles. In CVRP, each vehicle has a limited capacity, and each customer has its own demand that must be satisfied. The objective is to find the optimal route that satisfies all customer demands and minimizes the total distance traveled by the fleet.

Instance generation We follow the same instance generation settings as in previous work [94]. Customer locations are sampled uniformly in the unit square; customer demands are sampled from the discrete set { 1 , 2 , . . . , 9 } ; the capacity of each vehicle is set to 50; the depot is located at the center of the unit square.

Model inputs We transform a CVRP instance into a complete graph, where the node features are the demand values, and the edge attributes are relative distances between nodes.

Baseline heuristic The baseline heuristic measure is the inverse of edge length [15].

D.3 Orienteering Problem

Definition Following the problem setup in [54], each node i has a prize p i , and the objective of the Orienteering Problem (OP) is to form a tour that maximizes the total collected prize. The tour starts from a special depot node 0 and should be subject to a maximum length constraint.

Instance generation We uniformly sample the nodes, including the depot node, from the unit [0 , 1] 2 . We use a challenging prize distribution [31]: p i = (1 + ⌊ 99 · d 0 i max n j =1 d 0 j ⌋ ) / 100 , where d 0 i is the distance between the depot and node i . The maximum length constraint is also designed to be challenging. As suggested by Kool et al. [54], we set it to 4, 5, and 6 for OP100, OP200, and OP300, respectively.

Model inputs We transform an OP instance into a complete graph. The node features are twodimensional, consisting of nodes' prizes and distances to the depot. The edge attributes are relative distances between nodes. We do not include the maximum length constraint in input features because, in our case, it is a constant for each problem scale. It is easy to model the constraint explicitly as an additional node feature or context feature, or implicitly by using it to normalize nodes' distances to the depot.

Graph sparsification k is set to 20 for OP100, 50 for OP200, and 50 for OP300.

Baseline heuristic The baseline heuristic measure η ij = p j d ij [74].

D.4 Prize Collecting Traveling Salesman Problem

Definition Prize Collecting Traveling Salesman Problem (PCTSP) is a variant of TSP. In PCTSP, each node has an associated prize and penalty. The objective is to minimize the total length of the tour (which visits a subset of all nodes) plus the sum of penalties for unvisited nodes while also ensuring a specified minimum total prize collected.

Instance generation We follow the design of Kool et al. [54] for instance generation. All nodes are sampled uniformly in the unit square. The prize for each node is sampled uniformly from (0 , 1) . As a result, the minimum total prize constraint is set to n 4 for PCTSP n , such that n 2 nodes are expected to be visited. The penalty of each node is sampled uniformly from (0 , 3 L n 2 n ) , where L n is the expected tour length for TSP n and is set to 4, 8, 18 for n = 20 , 100 , 500 , respectively. For an additional depot node, its prize and penalty are set to 0.

Model inputs We transform a PCTSP instance into a complete graph. The node features are two-dimensional, consisting of nodes' prizes and penalties. The edge attributes are relative distances between nodes.

Baseline heuristic PCTSP is related to OP but inverts the goal [54]. The baseline heuristic measure of edge < i, j > is p j d ij [74].

D.5 Sequential Ordering Problem

Definition The Sequential Ordering Problem (SOP) involves finding a consistent and linear ordering of a set of elements. Consider that we have n jobs to handle sequentially for SOP n . Let x i be the processing time of job i and t ij the waiting time between job i and j . Specially, job 0 is a dummy node, and x 0 = 0 . We also have a set of constraints that indicate the precedence between jobs. The objective is to minimize the processing time for all jobs under precedence constraints.

Instance generation Wesample both x i and t ij uniformly from (0 , 1) . Westochastically sample the precedence constraints for all job pairs while ensuring at least one feasible solution and maintaining the constraints' transitive property; if job i precedes job j and job j precedes job k , then job i precedes job k . The probability of imposing the constraint on a pair of jobs is set to 0.2.

Model inputs We construct a directed graph where each job is a node and each job pair < i, j > , as long as job j does not precedes job i , is connected with a directed edge. The node feature is x i for job i and the edge attribute for job pair < i, j > is t ij .

Baseline heuristic The baseline heuristic measure η ij = 1 x j + d ij [70, 71].

D.6 Single Machine Total Weighted Tardiness Problem

Definition The Single Machine Total Weighted Tardiness Problem (SMTWTP) is a scheduling problem in which a set of n jobs must be processed on a single machine. Each job i has a processing time p i , a weight w i , and a due date t i . The objective is to minimize the sum of the weighted tardiness of all jobs, where the weighted tardiness is defined as the product of a job's weight and the duration by which its completion time exceeds the due date.

Instance generation Due dates, weights, and processing time are sampled as follows: t i ∼ n × Uni (0 , 1) , w i ∼ Uni (0 , 1) , p i ∼ Uni (0 , 1) .

Model inputs To ensure that all jobs can be the first to get processed, we include a dummy job with all its features set to zero. Then, we transform an SMTWTP instance into a directed complete graph with n +1 nodes. The features of a node are its normalized due date and weight. The attribute of edge < i, j > is p j .

Baseline heuristic The baseline heuristic prefers jobs with shorter due time: η ij = 1 t j [1].

D.7 Resource-Constrained Project Scheduling Problem

Definition Resource-Constrained Project Scheduling Problem (RCPSP) is to schedule a set of activities and minimize the schedule's makespan while satisfying precedence and resource constraints.

Instances Due to the complexity of generating random RCPCP instances, we draw the instances from PSPLIB [53] for training and held-out test.

̸

Model inputs As stated by Gonzalez-Pardo et al. [36], parallelism is not allowed if we sample directly from the constraint graph. So the input graph is an extension of it by adding edges between unrelated nodes, i.e., the edge between node i and j ( i = j ) exists only if j / ∈ S ∗ i ∧ i / ∈ S ∗ j , where S ∗ i denotes all the successors (direct and indirect) of node i on the constraint graph.

ACO algorithm The ACO algorithm follows the architecture proposed by Merkle et al. [62] but without the additional features. We sample the ant solutions in a topological order to accelerate the convergence.

Baseline heuristic The baseline heuristic is the combination of the greatest rank positional weight all (GRPWA) heuristic, and the weighted resource utilization and precedence (WRUP) heuristic [62].

D.8 Multiple Knapsack Problem

Definition Knapsack Problem (KP) is to select a subset of items from a given set such that the total weight of the chosen items does not exceed a specified limit and the sum of their values is maximized. Multiple Knapsack Problem (MKP), also known as m -dimensional Knapsack Problem, impose m different constraints on m types of weight. Formally, MKP can be formulated as follows:

Here, v j is the value of the item j , w ij is the i -th type of weight of the item j , and x j is the decision variable indicating whether the item j is included in the knapsack.

Instance generation The values and weights are uniformly sampled from [0 , 1] . To make all instances well-stated [57], we uniformly sample c i from (max j w ij , ∑ j w ij ) .

Model inputs For PH suc , we transform an MKP instance into a complete directed graph. The node features are their weights. The attribute of edge < i, j > is v i . For PH items , the model input is m +1 -dimensional feature vectors of n items. The feature vector of item j is a concatenation of v j , and w ij , i = 1 , . . . , m normalized by dividing c i , i = 1 , . . . , m respectively.

Baseline heuristic For PH suc , the baseline heuristic η ij = v j ∑ k w kj [30]. For PH items , the baseline heuristic η j = v j ∑ k w kj [30].

D.9 Bin Packing Problem

Definition The objective of the Bin Packing Problem (BPP) is to optimally arrange items within containers of a specific capacity, in order to reduce the overall number of containers used. We follow the specific setup in [58].

Instance generation Following [58], we set the number of items to 120 and the bin capacity to 150. Item sizes are uniformly sampled between 20 and 100.

Model inputs We transform a BPP instance into a complete graph where each node corresponds to an item. Node features are their respective item sizes, while edge features are set to 1. We additionally include a dummy node with 0 item size as an unbiased starting node for solution construction.

Baseline heuristic Heuristic measure of an item equals its size [58]; that is, the heuristic measure of a solution component in PH suc is the item size of its ending node.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to improve the performance of existing meta-heuristic algorithms using deep neural networks. At the same time, the method described in this paper makes it possible to design a high-performing meta-heuristic without requiring expert domain knowledge. In particular, the study in this paper is generally applicable to many types of COPs. This paper presents a deep neural network model to output the heuristic measure of ACO, as well as a training method for it. In addition, a neural-guided pertubating method is presented in the local search process. Experiments were conducted on 8 different types of CO problems, which showed that DeepACO significantly improves the performance of conventional ACO. It also showed excellent results when compared with the recent NCO studies. Finally this paper is excellently written and well organized.\n\nSTRENGTHS:\n**S1.** The method of combining meta-heuristic and deep neural networks is novel and applicable to various types of CO problems.  \n\n**S2.** The performance of the ACO meta-heuristic has been greatly improved by deepACO. In addition, the experiments were appropriately designed and performed.  \n\n**S3.** Compared to recent studies on NCO, DeepACO's performance was shown to be superior.\n\nWEAKNESSES:\n**W1.** In section 5.3, it is difficult to conclusively state that DeepACO demonstrates state-of-the-art performance when compared to the latest NCOs, because only the TSP500 and TSP1000 tasks are being used for comparison with existing NCOs. It would be nice to have comparative experiment results with existing NCOs for more diverse tasks. For example, adding CVRP 500/1000 experiments or adding TSP 100/CVRP 100, etc.  \n\n**W2.** In section 4.2, details about neural-guided perturbation are lacking. Specifically, there is a lack of explanation regarding the structure of the neural network used for perturbation, as well as details on the input and output of the network, and the processes of learning and inference.  \n\n​\n\nQUESTIONS:\n**Q1.** Please explain in more detail about neural-guided perturbation in 4.2. (refer to W2)  \n\n**Q2.** Since $\\tau_{i,j}$ is fixed to $1$ when training a heuristic learner(line 183), $\\tau$ is not considered in the training process. However, in Figure 4, DeepACO is robust against changes in alpha. What makes DeepACO robust to changes in alpha?  \n\n**Q3.** In line 252, you wrote \"We extend DeepACO from $PH_{suc}$ to $PH_{items}$ using a Transformer encoder equipped with an MLP decoder\". This is a new neural network different from the heuristic learner described in 4.3. Please explain in detail about this neural network.\n\n**Q4.** How many TSP test instances were used in the experiment in Table 1 and Table 2?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposes DeepACO, a generic framework leveraging deep reinforcement learning to automate heuristic designs. Specifically, DeepACO serves to strengthen the heuristic measures of existing ACO algorithms and dispense with laborious manual design in future ACO applications.  Experiments demonstrate that DeepACO consistently outperforms its ACO counterparts on eight combinatorial optimization problems (COPs).

STRENGTHS:
1. To the best of my knowledge, this work is the first to exploit deep reinforcement learning to guide the evolution of ACO meta-heuristics.   
2. Experiments demonstrate that DeepACO consistently outperforms its ACO counterparts on eight combinatorial optimization problems (COPs).

WEAKNESSES:
1. Previous work [1] has proposed a data-driven Heuristics Schedule framework in modern mixed-integer linear programming (MILP) solvers. The authors may want to discuss the major novelty of this work over previous work.  
2. It would be more convincing if the authors could show some generalization results, as the ability of generalizing to larger instances is also important in solving MILPs. 

[1] Chmiela, Antonia, et al. "Learning to schedule heuristics in branch and bound." Advances in Neural Information Processing Systems 34 (2021): 24235-24246.

QUESTIONS:
Please refer to Weaknesses for my questions.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DeepACO, a neural-enhanced solution to the limitations of Ant Colony Optimization (ACO) meta-heuristics, namely, the laborious manual design of heuristic measures and their heavy reliance on expert knowledge. ACO, which is a foraging system inspired by ant colonies, deploys artificial ants to explore solution spaces, and these solutions are biased toward promising areas through pheromone trails and heuristic measures. DeepACO innovates by learning a problem-specific mapping from an instance to its heuristic measures, leveraging neural models across instances, and incorporating these measures into ACO to influence solution constructions and help escape local optima. The paper also proposes three extended implementations of DeepACO to balance exploration and exploitation. DeepACO outperforms traditional ACO methods and is competitive against state-of-the-art and problem-specific Neural Combinatorial Optimization (NCO) methods across eight combinatorial optimization problems, even with only minutes of training.\n\nSTRENGTHS:\nNovelty: The paper proposes DeepACO, a new approach to enhancing Ant Colony Optimization (ACO) algorithms through a neural-enhanced meta-heuristic. This application of deep reinforcement learning to ACO is stated as being the first of its kind, thereby representing an innovative contribution to the field.\n\nBroad Applicability: DeepACO has been tested across eight different Combinatorial Optimization Problems (COPs), indicating its versatility and adaptability. It also shows competitive performance with specialized Neural Combinatorial Optimization (NCO) methods on the Travelling Salesman Problem (TSP), suggesting it is a robust method.\n\nAutomation: The paper presents a method to automate the design of heuristics for future ACO applications, potentially saving significant manual effort and expert knowledge.\n\nWEAKNESSES:\nLimitations in Current Implementation: The authors themselves note that DeepACO may underperform when not incorporating local search (LS) components due to the current restriction of compressing all learned heuristic information into an n × n matrix. This could limit the solution's effectiveness in complex COPs.\n\nAbsence of Comparison to Other Methods: While the paper mentions that DeepACO outperforms other ACO methods and is competitive with specialized NCO methods, there's no mention of a comparison with other non-ACO or non-NCO optimization methods. This could limit the ability to gauge how novel or effective DeepACO truly is in the broader context of optimization techniques.\n\nDependence on Machine Learning: The performance of DeepACO appears to be heavily dependent on machine learning. While this isn't necessarily a weakness, it could limit its application in situations where computational resources are restricted, or where machine learning models are difficult to train due to lacking training data.\n\nUnclear Generalizability: Although DeepACO is tested on eight different COPs, it's unclear how it would perform on a broader range of problems, particularly those that don't closely resemble the tested ones. More comprehensive testing across a diverse set of problems and datasets would provide stronger evidence of its generalizability.\n\nQUESTIONS:\nCould you provide further insight into the \"3D heuristic measures\" or \"dynamic heuristics\" you mention in the conclusion? How might they help to overcome the limitations of the current implementation?\n\nHow do you see DeepACO being applied in real-world situations? What kind of problems do you envision it solving most effectively?\n\nCould you discuss more about the process and challenges of training the deep reinforcement learning (DRL) model for guiding the evolution of ACO meta-heuristics?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This article proposes DeepACO, which is a generic framework leveraging deep reinforcement learning to automate heuristic designs.
DeepACO serves to strengthen the heuristic measures of existing ACO algorithms.
According to the experiments, DeepACO consistently outperforms its ACO counterparts on eight COPs using a single neural model and a single set of hyperparameters. It also performs better than or competitively against the problem-specific methods on the canonical Travelling Salesman Problem.

The article reviews the related works, explains ACO and the proposed methodology, and presents the experiments: their settings results and analysis. It is followed by supplementary materials.

STRENGTHS:
It is a well-written paper and I enjoyed reading it. All important concepts seem to be clearly explained.
The method was tested in 8 benchmark problems which is a huge strength. On all of them, DeepACO outperformed standard ACO methods.
In general, the results of the experiments are very good. The limitations of the method are discussed as well and the authors declared that the code used in experiments will be made publicly available and I found it in supplementary materials, indeed.

WEAKNESSES:
For some reason, hyperlinks to the bibliography do not work in the PDF that I received. I didn't find information on how the parameters alpha and beta (control parameters) were set in experiments. The writing can be slightly improved:
- p.7: "On the other hand" appears at the beginning of 2 consecutive sentences, so maybe in 1 of those sentences, it can be substituted.

QUESTIONS:
Question to Fig. 6: does it make sense to combine all the extensions and use them at once?

LIMITATIONS:
The authors adequately discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06