SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fu Luo $^{1*}$  Xi Lin $^{2*}$  Fei Liu $^{2}$  Qingfu Zhang $^{2}$  Zhenkun Wang $^{1\dagger}$
$^{1}$  Southern University of Science and Technology
luof2023@mail.sustech.edu.cn, {xi.lin, fliu36-c}@my.cityu.edu.hk,
qingfu.zhang@cityu.edu.hk, wangzhenkun90@gmail.com
Neural combinatorial optimization (NCO) is a promising learning-based approach for solving challenging combinatorial optimization problems without specialized algorithm design by experts. However, most constructive NCO methods cannot solve problems with large-scale instance sizes, which significantly diminishes their usefulness for real-world applications. In this work, we propose a novel Light Encoder and Heavy Decoder (LEHD) model with a strong generalization ability to address this critical issue. The LEHD model can learn to dynamically capture the relationships between all available nodes of varying sizes, which is beneficial for model generalization to problems of various scales. Moreover, we develop a data-efficient training scheme and a flexible solution construction mechanism for the proposed LEHD model. By training on small-scale problem instances, the LEHD model can generate nearly optimal solutions for the Travelling Salesman Problem (TSP) and the Capacitated Vehicle Routing Problem (CVRP) with up to 1000 nodes, and also generalizes well to solve real-world TSPLib and CVRPLib problems. These results confirm our proposed LEHD model can significantly improve the state-of-the-art performance for constructive NCO. The code is available at https://github.com/CIAM-Group/NCO_code/tree/main/single_objective/LEHD.
Combinatorial optimization (CO) holds significant practical value across various domains, such as vehicle routing [48], production planning [11], and drug discovery [34]. Due to the NP-hardness and the presence of numerous complex variants, solving CO problems remains an extremely challenging task [31]. In the past few decades, many powerful algorithms have been proposed to tackle different CO problems, but they mainly suffer from two major limitations. Firstly, solving each new problem requires extensive domain knowledge to design a proper algorithm by experts, which could lead to a significant development cost. In addition, these algorithms usually require an excessively long execution time due to the NP-hardness of CO problems [49]. These limitations make the classical algorithms undesirable and impractical for many real-world applications.
Recently, many learning-based neural combinatorial optimization (NCO) methods have been proposed to solve CO problems without handcrafted algorithm design. These methods build neural network models to generate the solution for given CO problem instances, which can be trained by supervised learning (SL) [51, 22, 13, 23, 28, 18] or reinforcement learning (RL) [4, 27, 17, 19, 6, 9, 53, 37, 29, 36, 24, 39, 40, 54, 52, 1]. Although these methods can achieve promising performance on
*Equal contributors
small-scale problems, they perform poorly on solving large-scale problems. Due to the NP-hardness of large-scale problems, the SL-based methods struggle to obtain enough high-quality solutions as labeled training data. Meanwhile, the RL-based methods could suffer from the critical issues of sparse rewards [47, 15] and device memory limitations for solving large-scale problems. Therefore, it is impractical to directly train the NCO model on large-scale problem instances. One feasible approach is to train the NCO model on small-scale problems and then generalize it to solve large-scale problems. However, the current purely learning-based constructive NCO methods have a very poor generalization ability, which diminishes their usefulness in solving large-scale real-world problems.
The current constructive NCO models typically have a Heavy Encoder and Light Decoder (HELD) structure, which could be an underlying reason for their poor generalization ability. The HELD-based model aims to learn the embeddings of all nodes via a heavy encoder in one shot and then sequentially construct the solution with the static node embeddings via a light decoder. This one-shot embedding learning may incline the model to learn scale-related features to perform well on small-scale instances. When the model generalizes to large-scale problem instances, the learned scale-related features may hinder the model from capturing the necessary relations among a significantly larger number of nodes. This work proposes a constructive NCO model with a novel Light Encoder and Heavy Decoder (LEHD) structure to tackle this issue. Instead of one-shot embedding learning, our proposed LEHD-based model learns to dynamically capture the relationships among the current partial solution and all the available nodes at each construction step via the heavy decoder. As the size of the available nodes varies with construction steps, the model tends to learn scale-independent features. Moreover, the model can iteratively adjust and refine the learned node relationships during the training. Therefore, it could be less sensitive to the instance size and has much better generalization performance on large-scale problem instances.
It becomes impractical to train the proposed LEHD model with RL due to the huge memory and computational cost of the heavy decoder structure. In this paper, we proposed a data-efficient training scheme, called learn to construct partial solution, to train the LEHD model in a supervised manner. With this scheme, the model learns to reconstruct partial solutions during the optimization process, which can be treated as a kind of data augmentation for robust model training. Finally, similar to the training process, we propose a flexible solution construction mechanism called Random Re-Construct (RRC) for efficient active improvement at the inference stage, which can significantly improve the solution quality with iterative local reconstructions.
- We propose a novel Light Encoder and Heavy Decoder (LEHD) model for generalizable neural combinatorial optimization. By only training on small-scale problem instances, the proposed LEHD model can achieve robust and promising performance on problems with much larger sizes.
- We develop a data-efficient training scheme and a flexible solution construction mechanism for the proposed LEHD model. The data-efficient training scheme enables LEHD to be trained efficiently via supervised learning, while the solution construction mechanism can continuously improve the solution quality through a customized inference budget.
- Our purely learning-based constructive method can achieve state-of-the-art performance in solving TSP and CVRP at various scales up to size 1000, and also generalizes well to solve real-world TSPLib/CVRPLib problems.
Constructive NCO with Balanced Encoder-Decoder In their seminal work, Vinyals et al. [51] propose the Pointer Network as a neural solver to directly construct solutions for combinatorial optimization problems in an autoregressive manner. This model has a balanced encoder-decoder structure where both the encoder and decoder are recurrent neural networks (RNNs) with the same number of layers. The original Pointer Network is trained by supervised learning, while Bello et al. [4] proposes an efficient reinforcement learning method for NCO model training. Some Pointer Network variants have been proposed to tackle different combinatorial optimization problems [37]. However, these models can only solve small-scale problem instances with sizes up to 100 and have poor generalization performance.
Constructive NCO with Heavy Encoder and Light Decoder Kool et al. [27] and Deudon et al. [10] leverage the Transformer architecture [46] to design more powerful NCO models. The Attention Model (AM) proposed by Kool et al. [27] has a Heavy Encoder and Light Decoder (HELD) structure with three attention layers in the encoder and one layer in the decoder. This model can obtain promising performance on various small-scale problems with no more than 100 nodes. Since then, AM has become a typical model choice, and many AM variants have been proposed for constructive NCO [53, 29, 54, 30, 24]. Among them, POMO proposed by Kwon et al. [29] is an AM model with a more powerful learning and inference strategy based on multiple optimal policies. The POMO model has a heavy encoder with six attention layers and a light one-layer decoder. It can achieve remarkable performance on small-scale problems with up to 100 nodes.
However, these constructive models with heavy encoder and light decoder structures still have a very poor generalization performance on problem instances with larger sizes, even for those with up to 200 nodes [23]. Directly training these models on the large-scale problem is also very difficult if not infeasible [23]. Different methods have been proposed to enhance the generalization ability for constructive NCO models in the inference stage, such as Efficient Active Search (EAS) [19] and Simulation-guided Beam Search (SGBS) [8]. While these methods can improve the constructive NCO model's generalization performance on problems with up to size 200, they struggle to solve problems with larger scales.
Very recently, a few approaches have been proposed to use the AM model to tackle large-scale TSP instances. Pan et al. [38] propose a Lower-Level Model that learns to construct the solution of sub-problems obtained by an Upper-Level Model. Another method proposed by Cheng et al. [7] uses an AM model to learn to reconstruct segments of a given TSP solution and selects the most improving segment to optimize it. However, these methods heavily rely on specialized designs for TSP by human experts, and cannot be used to solve other problems such as CVRP. In a concurrent work, Drakulic et al. [12] propose a novel Bisimulation Quotienting (BQ) method for generalizable neural combinatorial optimization by reformulating the Markov Decision Process (MDP) of the solution construction. Our proposed LEHD model has a different motivation and model structure from BQ, and a comprehensive comparison is provided in the experiment section.
Non-Constructive NCO Methods In addition to constructive NCO, there are also other learning-based approaches that work closely with search and improvement methods [33, 6, 18, 21]. Joshi et al. [22] propose a graph convolutional network to predict the heat map of each edge's probability in the optimal solution for a TSP instance. Afterward, approximate solutions can be obtained via the heat map guided beam search [22], Monte Carlo tree search (MCTS) [13], dynamic programming [28], and guided local search [20]. The heatmap-based approach was originally proposed to solve small-scale problems, and some recent works generalize it for solving large-scale TSP instances [13, 42]. However, they heavily rely on the carefully expert-designed MCTS strategy for TSP [13] and cannot be used to solve other problems.
Learning-augmented methods can also be applied to improve and accelerate heuristic solvers via operation selection [6, 35, 9], guided improvement [17, 5, 55] and problem decomposition [44, 32]. Many improvement-based methods have also been proposed to iteratively improve a feasible initial solution [3, 52, 36]. However, these methods typically rely on human-designed heuristic operators or more advanced solvers [14, 16] for solving different problems. In this work, we mainly focus on the purely learning-based NCO solver without expert knowledge.
3 Model Architecture: Light Encoder and Heavy Decoder
In this section, we propose a novel NCO model with the Light Encoder and Heavy Decoder (LEHD) structure to tackle the critical issue regarding large-scale generalization.
For a problem instance  $\mathbf{S}$  with  $n$  node features  $(\mathbf{s}_1,\dots ,\mathbf{s}_n)$  (e.g., the coordinates of  $n$  cities), a constructive model parameterized by  $\theta$  generates the solution in an autoregressive manner, i.e., it constructs the solution by selecting nodes one by one. Our LEHD-based model consists of a light encoder and a heavy decoder, as shown in Figure 1. The light encoder has one attention layer, and the heavy decoder has  $L$  attention layers. Given the problem instance with node features
Figure 1: The structure of our proposed LEHD model, which has a single-layer light encoder and a heavy decoder with  $L$  attention layers.
$\mathbf{S} = (\mathbf{s}_1, \ldots, \mathbf{s}_n)$ , the light encoder transforms each node features  $\mathbf{s}_i$  to its initial embedding  $\mathbf{h}_i^{(0)}$  through a linear projection. The initial embeddings  $\{\mathbf{h}_1^{(0)}, \ldots, \mathbf{h}_n^{(0)}\}$  are then fed into one attention layer to get the node embedding matrix  $H^{(1)} = (\mathbf{h}_1^{(1)}, \ldots, \mathbf{h}_n^{(1)})$ .
Attention layer The attention layer comprises two sub-layers: the multi-head attention (MHA) sub-layer and the feed-forward (FF) sub-layer [46]. Different from generic NCO models such as AM [27], the normalization is removed from our model to enhance generalization performance (see Appendix A). Let  $H^{(l-1)} = (\mathbf{h}_1^{(l-1)}, \ldots, \mathbf{h}_n^{(l-1)})$  be the input of the  $l$ -th attention layer for  $l = 1, \ldots, L$ , the output of the attention layer in terms of the  $i$ -th node is calculated as:
$$
\hat {\mathbf {h}} _ {i} ^ {(l)} = \mathbf {h} _ {i} ^ {(l - 1)} + \operatorname {M H A} \left(\mathbf {h} _ {i} ^ {(l - 1)}, H ^ {(l - 1)}\right), \tag {1}
$$
$$
\mathbf {h} _ {i} ^ {(l)} = \hat {\mathbf {h}} _ {i} ^ {(l)} + \mathrm {F F} (\hat {\mathbf {h}} _ {i} ^ {(l)}).
$$
We denote this embedding process as  $H^{(l)} = \mathrm{AttentionLayer}(H^{(l - 1)})$ .
The decoder sequentially constructs the solution in  $n$  steps by selecting node by node. In the  $t$ -th step for  $t \in \{1, \dots, n\}$ , the current partial solution can be written as  $(x_1, \dots, x_{t-1})$ , the first selected node's embedding is denoted as  $\mathbf{h}_{x_1}^{(1)}$ , the embedding of the node selected in the previous step is denoted as  $\mathbf{h}_{x_{t-1}}^{(1)}$ , and the embeddings of all available nodes is denoted by  $H_a = \{\mathbf{h}_i^{(1)} \mid i \in \{1, \dots, n\} \setminus \{x_1, \dots, x_{t-1}\} \}$ . Since the decoder has  $L$  attention layers, the  $t$ -th construction step of LEHD can be described as:
$$
\widetilde {H} ^ {(0)} = \mathrm {C o n c a t} (W _ {1} \mathbf {h} _ {x _ {1}} ^ {(1)}, W _ {2} \mathbf {h} _ {x _ {t - 1}} ^ {(1)}, H _ {a}),
$$
$$
\begin{array}{c} \widetilde {H} ^ {(1)} = \mathrm {A t t e n t i o n L a y e r} (\widetilde {H} ^ {(0)}), \\ \dots \end{array}
$$
$$
\widetilde {H} ^ {(L)} = \text {A t t e n t i o n L a y e r} (\widetilde {H} ^ {(L - 1)}), \tag {2}
$$
$$
u _ {i} = \left\{ \begin{array}{l l} W _ {O} \widetilde {\mathbf {h}} _ {i} ^ {(L)}, & i \neq 1 \text {o r} 2 \\ - \infty , & \text {o t h e r w i s e} \end{array} \right.,
$$
$$
\mathbf {p} ^ {t} = \operatorname {s o f t m a x} (\mathbf {u}),
$$
where  $W_{1}, W_{2}, W_{O}$  are learnable matrices. The matrices  $W_{1}$  and  $W_{2}$  are used to re-calculate the embeddings of the starting node (i.e., the node selected in the previous step) and the destination node (i.e., the node selected in the first step), respectively. The matrix  $W_{O}$  is used to transform the relation-denoting embedding matrix  $\widetilde{H}^{(L)}$  into a vector  $\mathbf{u}$  for the purpose of calculating the selection probability  $\mathbf{p}^t$ . The most suitable node  $x_{t}$  is selected based on  $\mathbf{p}^t$  at each decoding step  $t$ . Finally, a complete solution  $\mathbf{x} = (x_{1},\ldots ,x_{n})^{\intercal}$  is constructed by calling the decoder  $n$  times.
For the HELD model such as AM [27], the heavy encoder with  $L$  layers outputs the node embedding matrix as  $H^{(L)} = (\mathbf{h}_1^{(L)},\dots ,\mathbf{h}_n^{(L)})$ . The average of all nodes' embeddings (i.e.,  $\frac{1}{n}\sum_{i = 1}^{n}\mathbf{h}_i^{(L)})$ , the
Figure 2: Examples of generating partial solution for TSP and CVRP. For the TSP instance, its optimal solution is [1,2,3,4,5,6,7,8,9], and a partial solution is randomly sampled as [6,5,4,3,2]. For the CVRP instance, its optimal solution is [0,1,2,3,0,4,5,6,0,7,8,9,0], and a partial solution [2,3,0,6,5,4,0] is randomly sampled. We impose a restriction for CVRP that the partial solution must end at the depot.
starting node's embedding (i.e.,  $\mathbf{h}_{x_{t - 1}}^{(L)}$ ), the destination node's embedding (i.e.,  $\mathbf{h}_{x_1}^{(L)}$ ) are contacted to form the context node embedding (i.e.,  $\mathbf{h}_c^{(0)}$ ). Thereafter, the light decoder takes the context node embedding and the embeddings of all nodes (i.e.,  $H^{(L)}$ ) as input, and computes the selection probability through an MHA operation, the compatibility calculation, and the softmax function, i.e.,
$$
\mathbf {h} _ {c} ^ {(1)} = \mathrm {M H A} (\mathbf {h} _ {c} ^ {(0)}, H ^ {(L)}),
$$
$$
u _ {i} = \left\{ \begin{array}{l l} 1 0 \cdot \tanh  \left(\frac {\left(W _ {C} \mathbf {h} _ {c} ^ {(1)}\right) ^ {\intercal} W _ {E} \mathbf {h} _ {i} ^ {(L)}}{d}\right), & i \neq x _ {1: t - 1}, \\ - \infty , & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
where  $W_{C}$  and  $W_{E}$  are learnable matrices,  $d$  is a parameter determined by the embedding dimension. Since the HELD model's decoder only has a static node embedding matrix  $H^{(L)}$ , the relationships among the nodes it captures are not updated throughout the decoding process. In other words, the HELD model's performance relies heavily on the quality of the static node embedding matrix. When the problem size becomes larger, the ability of the static node embedding matrix to represent the relationships among all nodes becomes worse, resulting in the poor generalization performance of the model.
In our LEHD model, the heavy decoder dynamically re-embeds the embeddings of the starting node, destination node, and available nodes via  $L$  attention layers, thus updating the relationships among the nodes at each decoding step. Such a dynamic learning strategy enables the model to adjust and refine its captured relationships between the starting/destination and available nodes. In addition, as the size of the nodes varies during the construction steps, the model tends to learn the scale-independent features. These good properties enable the model to dynamically capture the proper relationships between the nodes, thereby making more informed decisions in the node selection on problem instances of various sizes.
4 Learn to Construct Partial Solution
The constructive NCO aims to construct the solution node-by-node using the decoder. The heavy decoder of our LEHD model can result in a higher computational cost than that of the HELD model at each construction step. Whereas the RL training method needs to generate the complete solution before computing the reward, which implies it requires significantly huge memory and computational costs. Furthermore, RL suffers from the challenging issue of sparse reward, especially for problems with large sizes.
According to Yao et al. [56], data augmentation (DA) can significantly reduce the number of required high-quality solutions (i.e., labels) for SL training, and DA-based SL can even outperform RL regarding solution accuracy and generalization performance. Operations based on the property of multiple optimality [29] and optimality invariance [25] are the most commonly used DA methods in NCO. In this work, we develop an efficient DA-based SL training scheme also based on the property of optimality invariance.
For each training data (i.e., a set of nodes), its label is a sequence of all the nodes (i.e., a valid tour). According to the property of optimality invariance, the partial solution of the optimal solution must
also be optimal. Therefore, we can randomly sample labeled partial solutions with various sizes and different directions from each labeled data, thus greatly enriching our training set. Based on these characteristics, we propose a training scheme called learning to construct partial solutions. The model learns to construct optimal partial solutions of various sizes and random directions, thereby resulting in a more efficient and robust training process.
4.1 Generate Partial Solutions during the Training Phase
For a routing problem instance  $\mathbf{S}$  with  $n$  nodes  $(\mathbf{s}_1,\dots ,\mathbf{s}_n)$ , its optimal solution can be represented by a specific node sequence  $\mathbf{x}$  (also called the tour). The partial solution  $\mathbf{x}_{sub}$  is a consecutive subsequence sampled from  $\mathbf{x}$  with a random size and direction. In this work, we let each partial solution have a length with  $w$  sampled from the discrete uniform distribution  $\mathbf{Unif}([4,|V|])$  where  $|V|$  is the number of nodes for each TSP/CVRP instance, since a problem with size less than 4 is trivial to solve. Each partial solution  $\mathbf{x}_{sub}$  is one augmented data point in our method. Two examples of generating partial solutions for TSP and CVRP are provided in Figure 2.
4.2 Learn to Construct Partial Solutions via Supervised Learning
The model learns to construct the partial solution node by node. The first node of  $\mathbf{x}_{sub}$  is the starting node, the last node of  $\mathbf{x}_{sub}$  is the destination node, and the remaining nodes constitute the available nodes to be selected by the model. For the example  $\mathbf{x}_{sub} = [6,3,5,4,1]$ , node 6 is the starting node, node 1 is the destination node, and nodes 3, 5, 4 are available nodes.
In each training step, the model provides each available node  $\mathbf{s}_i$  with a probability  $p_i$  of being selected. Meanwhile, the label  $\mathbf{x}_{sub}$  tells whether the node  $\mathbf{s}_i$  should be chosen in the current step (i.e.,  $y_i = 1$  or 0). The cross-entropy loss  $loss = -\sum_{i=1}^{u} y_i \log(p_i)$  is employed to measure the error made by the current model, where  $u$  is the number of available nodes. Then we can update the model parameters by a gradient-based algorithm (e.g., Adam) with respect to the loss. Throughout the training process, the number of available nodes gradually decreases, and the starting node dynamically shifts to the node selected in the previous step while the destination node remains constant. The training epoch ends when there are no more nodes available. The next epoch starts with new augmented data.
4.3 Generate the Complete Solution during the Inference Phase
During the inference phase, our proposed LEHD model employs a greedy step-by-step approach to construct the whole solution for the given instance S. In the beginning, a single node is randomly selected to serve as the destination node and also the initial starting node. The remaining nodes constitute the set of available nodes that the model can select step-by-step. Once a node is selected, it will be treated as the new starting node and removed from the set of available nodes, i.e., the starting node is dynamically shifting while the destination node remains constant. When no more available nodes exist, the whole solution is generated. In this way, LEHD can easily construct solutions for problems of various sizes as well as unseen large-scale problems.
5 Random Re-Construct for Further Improvement
The constructive model has an inductive bias, which means it could construct a better solution by following its preferred direction. Similarly, different starting and destination nodes will lead to different solution quality since the model may perform better when starting from some well-learned local patterns. The tour generated via greedy search could be not optimal and contain multiple suboptimal local segments (e.g., partial solutions). Therefore, rectifying these suboptimal segments during the inference phase can effectively enhance the overall solution quality.