SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haoran Ye $^{1}$ , Jiarui Wang $^{1}$ , Zhiguang Cao $^{2,*}$ , Helan Liang $^{1,*}$ , Yong Li $^{3}$
<sup>1</sup> School of Computer Science & Technology, Soochow University
$^{2}$  School of Computing and Information Systems, Singapore Management University
$^{3}$  Department of Electronic Engineering, Tsinghua University
{hrye, jrwangfurffico}@stu.suda.edu.cn, zgcao@smu.edu.sg,
hlliang@suda.edu.cn, liyong07@tsinghua.edu.cn
Ant Colony Optimization (ACO) is a meta-heuristic algorithm that has been successfully applied to various Combinatorial Optimization Problems (COPs). Traditionally, customizing ACO for a specific problem requires the expert design of knowledge-driven heuristics. In this paper, we propose DeepACO, a generic framework that leverages deep reinforcement learning to automate heuristic designs. DeepACO serves to strengthen the heuristic measures of existing ACO algorithms and dispense with laborious manual design in future ACO applications. As a neural-enhanced meta-heuristic, DeepACO consistently outperforms its ACO counterparts on eight COPs using a single neural model and a single set of hyperparameters. As a Neural Combinatorial Optimization method, DeepACO performs better than or on par with problem-specific methods on canonical routing problems. Our code is publicly available at https://github.com/henry-yesh/DeepACO.
Ant systems in nature are self-learners. They utilize chemical signals and environmental cues to locate and return food to the colony. The pheromone trails deposited by ants indicate the quality and distance of a food source. The intensity of pheromone trails increases as more ants visit and decreases due to evaporation, creating a self-learning foraging system.
Inspired by the ant systems in nature, researchers propose and develop Ant Colony Optimization (ACO) meta-heuristics for (but not limited to) Combinatorial Optimization Problems (COPs) [24, 26]. ACO deploys a population of artificial ants to explore the solution space through repeated solution constructions and pheromone updates. The exploration is biased toward more promising areas through instance-specific pheromone trails and problem-specific heuristic measures. Both the pheromone trail and the heuristic measure indicate how promising a solution component is. Typically, pheromone trails are initialized uniformly for all solution components and learned while solving an instance. On the contrary, heuristic measures are predefined based on prior knowledge of a problem, and devising proper heuristic measures for complicated COPs is quite challenging (an example is [49]).
Over the past decades, research and practice efforts have been dedicated to a careful design of heuristic measures in pursuit of knowledge-driven performance enhancement [57, 62, 33, 49, 26, 24]. However, this routine of algorithm customization exhibits certain deficiencies: 1) it requires extra effort and makes ACO less flexible; 2) the effectiveness of the heuristic measure heavily relies on expert knowledge and manual tuning; and 3) designing a heuristic measure for less-studied problems can be particularly challenging, given the paucity of available expert knowledge.
* Corresponding authors: Zhiguang Cao and Helan Liang.
This paper proposes DeepACO, a generic neural-enhanced ACO meta-heuristic, and a solution to the above limitations. DeepACO serves to strengthen the heuristic measures of existing ACO algorithms and dispense with laborious manual design in future ACO applications. It mainly involves two learning stages. The first stage learns a problem-specific mapping from an instance to its heuristic measures by training neural models across COP instances. Guided by the learned measures, the second stage learns instance-specific pheromone trails while solving an instance with ACO. The heuristic measures learned in the first stage are incorporated into ACO (the second learning stage) by biasing the solution constructions and leading Local Search (LS) to escape local optima.
DeepACO is also along the line of recent progress in Neural Combinatorial Optimization (NCO) [12, 61, 7, 34]. Within the realm of NCO, DeepACO is more related to the methods that utilize heatmaps for algorithmic enhancement [32, 42, 90, 55, 42], but it is superior in its flexibility: DeepACO provides effective neural enhancement across eight COPs covering routing, assignment, scheduling, and subset problems, being the most broadly evaluated NCO technique to our knowledge. In addition, we propose three extended implementations for better balancing between exploitation and exploration: one featuring a multihead decoder, one trained with an additional top- $k$  entropy loss, and one trained with an additional imitation loss. They can be generally applied to heatmap-based NCO methods.
As a neural-enhanced version of ACO, DeepACO consistently outperforms its counterparts on eight COPs using a single neural model and a single set of hyperparameters after only minutes of training. As an NCO method, DeepACO performs better than or competitively against the state-of-the-art (SOTA) and problem-specific NCO methods on canonical routing problems while being more generalizable to other COPs. To the best of our knowledge, we are the first to exploit deep reinforcement learning (DRL) to guide the evolution of ACO meta-heuristics. Such a coupling allows NCO techniques to benefit from decades of ACO research (notably regarding theoretical guarantees [7, 26]), while simultaneously offering ACO researchers and practitioners a promising avenue for algorithmic enhancement and design automation.
In summary, we outline our contributions as follows:
- We propose DeepACO, a neural-enhanced ACO meta-heuristic. It strengthens existing ACO algorithms and dispenses with laborious manual design in future ACO applications.
- We propose three extended implementations of DeepACO to balance exploration and exploitation, which can generally be applied to heatmap-based NCO methods.
- We verify that DeepACO consistently outperforms its ACO counterparts across eight COPs while performing better than or on par with problem-specific NCO methods.
2.1 Neural Combinatorial Optimization
Neural Combinatorial Optimization (NCO) is an interdisciplinary field that tackles COPs with deep learning techniques. In general, existing NCO methods can be categorized into end-to-end and hybrid methods, and DeepACO belongs to the latter methodological category.
End-to-end methods in the former category learn autoregressive solution constructions or heatmap generation for subsequent sampling-based decoding. Within this realm, recent developments include better-aligned neural architectures [82, 64, 54, 50, 89, 13, 46], improved training paradigms [6, 56, 52, 83, 10, 66, 40, 45], advanced solution pipelines [47, 51, 48, 20, 78, 18, 65, 19], and broader applications [88, 95, 16, 29, 75, 9]. End-to-end methods are admirably efficient, but their constructed solutions can be further improved with iterative refinement and algorithmic hybridization.
Therefore, the hybrid methods in the latter category incorporate neural learners to make decisions within/for heuristics or generate heatmaps to assist heuristics. These methods either let neural learners make decisions within algorithmic loops [60, 87, 21, 17, 94, 86, 59], or generate heatmaps in one shot to assist subsequent algorithms. In the latter case, Xin et al. [90] propose to learn edge scores and node penalties to guide the searching process of LKH-3 [39], a highly optimized solver for routing problems. Kool et al. [55] train neural models to predict promising edges in routing problems, providing a neural boost for Dynamic Programming. Fu et al. [32] train small-scale GNN to build heatmaps for large TSP instances and feed the heatmaps into Monte Carlo Tree Search for solution
improvements. Hudson et al. [42] utilize neural models to generate regret in Guided Local Search for TSP. DeepACO is along the line of these works, but superior in terms of its flexibility, i.e., it provides effective neural enhancement across eight COPs covering routing, assignment, scheduling, and subset problems while the existing hybridized methods mostly focus on a limited set of routing problems.
2.2 Ant Colony Optimization
Ant Colony Optimization (ACO) is a meta-heuristic and evolutionary algorithm (EA) [4, 73] inspired by the behavior of ants in finding the shortest path between their colony and food sources [26].
The representative ACO meta-heuristics such as Ant System (AS) [25], Elitist AS [25], and MAX-MIN AS [76] provide general-purpose frameworks allowing for problem-specific customization. Such customization may involve designs of heuristic measures [49, 33], the incorporation of local search operators [3, 93], and the hybridization of different algorithms [44, 5]. DeepACO is not in competition with the most SOTA ACO algorithms for a specific COP. Instead, DeepACO can strengthen them with stronger heuristic measures and can in turn benefit from their designs.
ACO can utilize a group of techniques named hyper-heuristics [69] that are conceptually related to DeepACO. But hyper-heuristics mostly involve expert-designed heuristics to select from (heuristic selection hyper-heuristics) [27] or problem-specific and manually-defined components to evolve heuristics (heuristic generation hyper-heuristics) [14]. By comparison, DeepACO is more generic, requiring little prior knowledge of a COP. Its aim is not to compete with any hyper-heuristics; instead, for example, we can utilize hyper-heuristics to improve LS components in DeepACO.
Unlike the knowledge-driven ACO adaptations above, a recent method, namely ML-ACO [77], boosts the performance of ACO via Machine Learning (ML) for solution prediction. While ML-ACO provides an initial insight into coupling ML with ACO, it is preliminary and limited. ML-ACO tailors five features for Orienteering Problem and trains ML classifiers in a supervised manner. It entails high-demanding expert knowledge for feature designs and specialized solvers for optimal solutions, making itself inflexible and hard to scale. By comparison, DeepACO leverages DRL and demands little expert knowledge to apply across COPs.
3 Preliminary on Ant Colony Optimization
The overall ACO pipeline is depicted in Fig. 1. We begin with defining a COP model and a pheromone model. They are prerequisites for implementing ACO algorithms.
COP model Generally, a COP model consists of, 1) a search space  $\mathbf{S}$  defined over a set of discrete decision variables  $X_{i}, i = 1, \ldots, n$ , where each decision variable  $X_{i}$  takes a value from a finite set  $D_{i} = \{v_{i}^{1}, \ldots, v_{i}^{|D_{i}|}\}$ ; 2) a set of constraints  $\Omega$  that the decision variables must satisfy; and 3) an objective function  $f: \mathbf{S} \to \mathbb{R}_0^+$  to minimize. A feasible solution  $s$  to a COP is a complete assignment of all decision variables that satisfies all the constraints in  $\Omega$ .
Pheromone model A COP model defines a pheromone model in the context of ACO. Without loss of generality, a pheromone model is a construction graph that includes decision variables as nodes and solution components as edges [26]. Each solution component  $c_{ij}$ , that represents the assignment of value  $v_{i}^{j}$  to decision variable  $X_{i}$ , is associated with its pheromone trial  $\tau_{ij}$  and heuristic measure  $\eta_{ij}$ . Both  $\tau_{ij}$  and  $\eta_{ij}$  indicate how promising it is to include  $c_{ij}$  in a solution. Typically, ACO uniformly initializes and iteratively updates pheromone trails, but predefines and fixes heuristic measures.
As a motivating example, for TSP, a instance can be directly converted into a fully connected construction graph, where city  $i$  becomes node  $i$  and solution component  $c_{ij}$  represents visiting city  $j$  immediately after city  $i$ . Then,  $\eta_{ij}$  is typically set to the inverse of the distance between city  $i$  and  $j$ . After defining a COP model and a pheromone model, we introduce an ACO iteration, usually consisting of solution constructions, optional LS refinement, and pheromone updates.
Solution construction and local search (optional) Biased by  $\tau_{ij}$  and  $\eta_{ij}$ , an artificial ant constructs a solution  $s = \{s_t\}_{t=1}^n$  by traversing the construction graph. If an ant is located in node  $i$  at the  $t$ -th construction step ( $s_{t-1} = i$ ) and has constructed a partial solution  $s_{<t} = \{s_t\}_{t=1}^{t-1}$ , the probability of
ACO
Figure 1: The schematic diagrams of ACO and DeepACO. DeepACO 1) additionally trains a heuristic learner across instances, 2) applies the well-trained heuristic learner to generate heuristic measures during inference, and 3) optionally leverages the learned heuristic measures to conduct local search interleaved with neural-guided perturbation.
DeepACO
selecting node  $j$  as its next destination  $(s_{t} = j)$  is typically given by
$$
P \left(s _ {t} \mid \boldsymbol {s} _ {<   t}, \boldsymbol {\rho}\right) = \left\{ \begin{array}{l l} \frac {\tau_ {i j} ^ {\alpha} \cdot \eta_ {i j} ^ {\beta}}{\sum_ {c _ {i l} \in \boldsymbol {N} \left(\boldsymbol {s} _ {<   t}\right)} \tau_ {i l} ^ {\alpha} \cdot \eta_ {i l} ^ {\beta}} & \text {i f} c _ {i j} \in \boldsymbol {N} \left(\boldsymbol {s} _ {<   t}\right), \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {1}
$$
Here,  $\rho$  is a COP instance,  $N(s_{<t})$  is the set of feasible solution components given the partial solution, and  $\alpha$  and  $\beta$  are the control parameters, which are consistently set to 1 in this work unless otherwise stated. To simplify the notations, we omit the dependence on  $\rho$  for  $\tau$ ,  $\eta$ ,  $c$ , and  $N$ . Based on Eq. (1), constructing a complete solution requires an  $n$ -step graph traversal. The probability of generating  $s$  can be factorized as
$$
P (\boldsymbol {s} | \boldsymbol {\rho}) = \prod_ {t = 1} ^ {n} P \left(s _ {t} \mid \boldsymbol {s} _ {<   t}, \boldsymbol {\rho}\right). \tag {2}
$$
After solution constructions, local search (LS) is optionally applied to refine the solutions.
Pheromone update After solution constructions, the pheromone update process evaluates solutions and adjusts the pheromone trails accordingly, i.e., it increases the pheromone trails of components in the superior solutions while decreasing those in the inferior ones. The detailed update rules can differ depending on the ACO variation used.
ACO intelligently explores the solution space by iterating the above process, eventually converging on (sub)optimal solutions. We refer the readers to [26] for more details.
DeepACO is schematically presented in Fig. 1 wherein a comparison is made with ACO. It dispenses with expert knowledge and learns a set of stronger heuristic measures to guide the ACO evolution. DeepACO involves parameterizing the heuristic space (Section 4.1), optionally interleaving LS with neural-guided perturbation (Section 4.2), and training a heuristic learner across instances (Section 4.3). Additionally, we introduce three extended designs (Section 4.4) to boost exploration.
4.1 Parameterizing heuristic space
We introduce a heuristic learner, defined by a graph neural network (GNN) with trainable parameters  $\theta$ , to parameterize the heuristic space. The heuristic learner maps an input COP instance  $\rho$  to its
heuristic measures  $\pmb{\eta}_{\pmb{\theta}}(\pmb{\rho})$ , where we rewrite it as  $\pmb{\eta}_{\pmb{\theta}}$  for notation simplicity. It contains non-negative real values  $\eta_{ij;\pmb{\theta}}$  associated with each solution component  $c_{ij}, \forall i \in \{1, \dots, n\}, \forall j \in \{1, \dots, |\pmb{D}_i|\}$ . DeepACO constructs solutions following Eq. (2) but biased by  $\pmb{\eta}_{\pmb{\theta}}$ :
$$
P _ {\boldsymbol {\eta} _ {\boldsymbol {\theta}}} (\boldsymbol {s} | \boldsymbol {\rho}) = \prod_ {t = 1} ^ {n} P _ {\boldsymbol {\eta} _ {\boldsymbol {\theta}}} \left(s _ {t} \mid \boldsymbol {s} _ {<   t}, \boldsymbol {\rho}\right). \tag {3}
$$
In particular, we exploited the neural models recommended by Joshi et al. [48] and Qiu et al. [66]. It consists of a GNN backbone relying on anisotropic message passing and an edge gating mechanism, and a Multi-Layer Perceptron (MLP) decoder mapping the extracted edge features into heuristic measures. We defer the full details of this neural architecture to Appendix A.
4.2 Local search interleaved with neural-guided perturbation
In ACO, local search (LS) is optionally applied to refine the constructed solutions. However, LS is a myopic procedure in that it greedily accepts any altered solution with a lower objective value and easily gets trapped in local optima. In DeepACO, we intend the well-learned heuristic measures to indicate the global optimality of solution components. Leveraging such indicators and including solution components with greater global optimality can lead to better solutions eventually, if not immediately. Nevertheless, it is unrealistic to solely rely on the learned heuristic measures due to the inherent complexity of COPs.
Based on these considerations, we propose LS interleaved with neural-guided perturbation (NLS for short) in Algorithm 1. NLS interleaves LS aiming for a lower objective value and neural-guided perturbation biasing the learned optima. In each iteration, the first stage utilizes LS to repeatedly refine a solution until (potentially) reaching local optima. The second stage utilizes LS to slightly perturb the locally optimal solution toward gaining higher cumulative heuristic measures.
Algorithm 1 NLS
1: Input: A solution  $s$ ; an objective function  $f$ ; well-learned heuristic measures  $\pmb{\eta}_{\theta}$ ; a local search operator  $LS$  that takes three inputs: a solution to refine, the targeted objective function, and the maximum iterations before encountering local optima; the number of perturbation moves  $T_{p}$ ; the number of NLS iterations  $T_{NLS}$
2: Output: The best improved solution  $s^*$
3:  $s = LS(s, f, +\infty)$  // Improve  $s$  until reaching a local optimum
4:  $s^* = \text{copy}(s)$
5: for iter  $= 1\rightarrow T_{NLS}$  do
6:  $s = LS(s, \frac{1}{\eta_{\theta}}, T_p) // \text{Perturb } s \text{ toward higher cumulative heuristic measures with } T_p \text{ moves}$
7:  $\pmb{s} = LS(\pmb{s}, f, +\infty)$
8:  $\pmb{s}^{*} = \arg \min (f(\pmb{s}^{*}), f(\pmb{s}))$
9: end for
4.3 Training heuristic learner
We train the heuristic learner across COP instances. The heuristic learner  $\theta$  maps each instance  $\rho$  to its heuristic measures  $\eta_{\theta}$ . Then, we minimize the expected objective value of both constructed solutions and NLS-refined constructed solutions:
$$
\text {m i n i m i z e} \quad \mathcal {L} (\boldsymbol {\theta} | \boldsymbol {\rho}) = \mathbb {E} _ {\boldsymbol {s} \sim P _ {\eta_ {\boldsymbol {\theta}}} (\cdot | \boldsymbol {\rho})} [ f (\boldsymbol {s}) + W f (N L S (\boldsymbol {s}, f, + \infty)) ], \tag {4}
$$
where  $W$  is a coefficient for balancing two terms. Intuitively, the first loss term encourages directly constructing optimal solutions, which is, however, often held off by the complexity of COP. The second term encourages constructing solutions most fit for NLS, and it could be easier to learn high-quality solutions if coupling end-to-end construction with NLS. Even so, solely relying on the second term leads to inefficient training due to the small quality variance of the NLS-refined solutions. As a result, we find it useful to implement both terms for training. Note that the NLS process itself does not involve gradient flow.
In practice, we deploy Ant Systems to construct solutions stochastically following Eq. (3) for estimating Eq. (4). The pheromone trials are fixed to 1 to ensure an unbiased estimation. We apply a REINFORCE-based [85] gradient estimator:
$$
\nabla \mathcal {L} (\boldsymbol {\theta} | \boldsymbol {\rho}) = \mathbb {E} _ {\boldsymbol {s} \sim P _ {\eta_ {\boldsymbol {\theta}}} (\cdot | \boldsymbol {\rho})} \left[ \right.\left( \right.\left(f (\boldsymbol {s}) - b (\boldsymbol {\rho})\right) + W \left(f (N L S (\boldsymbol {s}, f, + \infty)) - b _ {N L S} (\boldsymbol {\rho}))\right) \nabla_ {\boldsymbol {\theta}} \log P _ {\eta_ {\boldsymbol {\theta}}} (\boldsymbol {s} | \boldsymbol {\rho}) \left. \right], \tag {5}
$$
where  $b(\pmb{\rho})$  and  $b_{NLS}(\pmb{\rho})$  are the average objective value of the constructed solutions and that of the NLS-refined constructed solutions, respectively.
4.4 Toward better exploration
The vanilla DeepACO fully exploits the underlying pattern of a problem and learns a set of aggressive heuristic measures (visualized in Appendix C.6) according to Eq. (4). Still, preserving exploration is of critical importance since COPs often feature many local optima [89]. To that end, we further present three extended designs to enable a better balance of exploration and exploitation. Note that they can generally be applied to heatmap-based NCO methods.
4.4.1 Multihead decoder
Multihead DeepACO implements  $m$  MLP decoders on the top of the GNN backbone. It aims to generate diverse heuristic measures to encourage exploring different optima in the solution space. A similar multihead design has been utilized for autoregressive solution construction [89]. We extend this idea to the non-autoregressive heuristic generation here.
For training, Multihead DeepACO utilizes Eq. (4) calculated individually and independently for  $m$  decoders and an extra Kullback-Leibler (KL) divergence loss expressed as
$$
\mathcal {L} _ {K L} (\boldsymbol {\theta} | \boldsymbol {\rho}) = - \frac {1}{m ^ {2} n} \sum_ {k = 1} ^ {m} \sum_ {l = 1} ^ {m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {| D _ {i} |} \tilde {\eta} _ {i j; \boldsymbol {\theta}} ^ {k} \log \frac {\tilde {\eta} _ {i j ; \boldsymbol {\theta}} ^ {k}}{\tilde {\eta} _ {i j ; \boldsymbol {\theta}} ^ {l}}, \tag {6}
$$
where  $\tilde{\eta}_{ij;\theta}^k$  is the heuristic measure of the solution component  $c_{ij}$  output by the  $k$ -th decoder after row-wise normalization.
In the inference phase, Multihead DeepACO deploys  $m$  ant groups guided by their respective MLP decoders while the whole ant population shares the same set of pheromone trails.
4.4.2 Top-  $k$  entropy loss
Entropy loss (reward) incentivizes agents to maintain the diversity of their actions. It is often used as a regularizer to encourage exploration and prevent premature convergence to suboptimal policies [68, 37, 51]. In the context of COP, however, most solution components are far from a reasonable choice for the next-step solution construction. Given this, we implement a top- $k$  entropy loss while optimizing Eq. (4), promoting greater uniformity only among the top  $k$  largest heuristic measures:
$$
\mathcal {L} _ {H} (\boldsymbol {\theta} | \boldsymbol {\rho}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j \in \mathcal {K} _ {i}} \bar {\eta} _ {i j; \boldsymbol {\theta}} \log \left(\bar {\eta} _ {i j; \boldsymbol {\theta}}\right). \tag {7}
$$
Here,  $\mathcal{K}_i$  is a set containing top  $k$  solution components of decision variable  $X_{i}$ , and  $\bar{\eta}_{ij;\theta}$  are the heuristic measures normalized within the set.
4.4.3 Imitation loss
If expert-designed heuristic measures are available, we can additionally incorporate an imitation loss, enabling the heuristic learner to mimic expert-designed heuristics while optimizing Eq. (4). It holds two primary merits. First, the expert-designed heuristic measures are less aggressive than the learned ones. So, it can function as a regularization to maintain exploration. Second, the heuristic learner can acquire expert knowledge. Perfecting the imitation first can guarantee that the learned heuristics are at least not inferior to the expert-designed counterparts.
Accordingly, the imitation loss is formulated as
$$
\mathcal {L} _ {I} (\boldsymbol {\theta} | \boldsymbol {\rho}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {| D _ {i} |} \tilde {\eta} _ {i j} ^ {*} \log \frac {\tilde {\eta} _ {i j} ^ {*}}{\tilde {\eta} _ {i j ; \boldsymbol {\theta}}}, \tag {8}
$$
where  $\tilde{\eta}_{ij}^{*}$  and  $\tilde{\eta}_{ij;\theta}$  are expert-designed and learned heuristic measures, respectively, both after row-wise normalization.
5 Experimentation