Generative Adversarial Training for Neural Combinatorial Optimization Models

1 INTRODUCTION . Combinatorial Optimization Problems ( COPs ) are a family of problems with the goal of finding the best one ( s ) from a finite set of solutions . Due to the considerably large solution space , many important COPs are hard to solve , such as the vehicle routing problems ( Toth & Vigo , 2002 ) . Exact algorithms based on branch-and-bound ( Lawler & Wood , 1966 ) or its variants can provide elegant theoretical guarantees but the worst-case computational complexity is exponential , hence impractical for problems of medium and large sizes . In contrast , heuristic methods can usually attain good solutions in reasonable running time , which are often preferred in practical applications . Traditional heuristics are designed based on expert knowledge for specific problems , which usually requires a large amount of time and efforts to develop . These manually designed heuristics could suffer from relatively poor performance . Moreover , for less studied problems , sufficient expert knowledge may even be unavailable . Recent studies suggest that deep learning could greatly facilitate in automating the design of heuristics , and alleviating the heavy reliance on expert knowledge . With the prior that the instances may follow certain distribution ( e.g. , locations in a routing problem may uniformly scatter in an area ) , deep models can be trained to learn heuristics in an end-to-end way ( Dai et al. , 2017 ; Kool et al. , 2019 ; Chen & Tian , 2019 ) . It has been shown that these models perform well with relatively short running time on COP instances following the training distribution . However , after the trained model is deployed , it could encounter many instances following unknown distributions different from the training one , especially for real-life applications . As shown in many existing works and our experiments , when applied to infer the instances following a different distribution , the generalization performance of deep models gets much inferior , which severely hinders the practical use of the learned heuristics . Such mismatch between the training and testing distributions is always an important issue for most learning based methods . Especially , for neural combinatorial optimization ( NCO ) models , deep learning models are mostly trained on instances sampled from specific distributions and the solution quality intricately depends on the instance distributions . The generalization to instances with different distributions has been widely acknowledged for the importance ( e.g. , in Mazyavkina et al . ( 2021 ) ) and remains a challenge . To tackle this issue , we propose the Generative Adversarial Neural Combinatorial Optimization ( GANCO ) framework which is model agnostic and generally applicable to various neural combinatorial optimization models for solving different COPs . Instead of training an optimization model only on instances following the predefined distribution , another deep neural network is deployed as a generation model to produce training instances following the distributions on which the optimization model performs poorly . The generation model and optimization model are trained alternatively in an adversarial way . Specifically , the generation model is trained by reinforcement learning to maximize the performance gap of the current optimization model on the generated instances with respect to a traditional non-learning baseline algorithm . The optimization model is trained in the original way but using the training dataset augmented with the newly generated instances to improve the generalization performance , i.e. , to reduce the performance gap . As we will show in the experiments , the non-learning baseline algorithms do not need to be very strong or fast . To demonstrate the effectiveness of the proposed GANCO framework , we apply it to a representative neural combinatorial optimization model , the Attention Model ( AM , Kool et al . ( 2019 ) ) on various COPs including Traveling Salesman Problem ( TSP ) , Capacitated Vehicle Routing Problem ( CVRP ) , Orienteering Problem ( OP ) , Prize Collecting TSP ( PCTSP ) and 0-1 Knapsack Problem ( KP ) . In the extensive experiments , we show that the proposed GANCO framework improves the generalization performance of the original optimization model on various testing distributions with little sacrifice of performance on the original training distribution . Furthermore , we show that the proposed GANCO framework can be readily and effectively applied to other optimization models such as the Policy Optimization with Multiple Optima ( POMO , Kwon et al . ( 2020 ) ) . 2 RELATED WORKS . Most deep learning models are trained to learn the construction heuristics where the model picks the actions sequentially to construct a solution ( e.g. , Bello et al . ( 2017 ) ; Nazari et al . ( 2018 ) ; Kool et al . ( 2019 ) ; Hottung et al . ( 2021 ) ; Kwon et al . ( 2020 ) ) , which perform well with fairly short running time . While some other models learn the improvement heuristics to locally refine existing solutions , which usually search over a large number of solutions and are relatively slow . Moreover , they often need to fit in frameworks specifically designed for different problems , such as 2-opt ( d O Costa et al. , 2020 ) , large neighborhood search ( Hottung & Tierney , 2020 ) and combinations of local operators ( Lu et al. , 2020 ) . Most models are trained with reinforcement learning ( RL ) except for several ones with supervised learning , e.g. , Vinyals et al . ( 2015 ) and Joshi et al . ( 2019 ) . Though we use representative construction heuristic models with RL in the experiments , our proposed GANCO framework can also be applied to models of other types . The most well-known models trained with adversary are the Generative Adversarial Networks ( Goodfellow et al. , 2014 ; Yang et al. , 2018 ; Zhang et al. , 2019 ) , which serve to generate new data samples to imitate the training set . The generative network generates the sample candidates whereas the discriminative network classifies whether the sample is generated or genuine . In contrast , the goal of our proposed GANCO framework is to train the optimization model for better generalization ability where the generation model is used only to generate more informative training samples . The idea of using adversarial training to improve RL robustness has also been studied in other fields such as Atari games and robotics ( Pinto et al. , 2017 ; Khirodkar et al. , 2018 ; Dennis et al. , 2020 ) . However , existing works in this direction are not directly applicable to our task due to its unique properties . The key is how to define the adversary , which is specific to the application domain . In the field of COPs , we define the adversary as a hard instance generator , which is evaluated by performance gaps of the optimization model with respect to a traditional non-learning baseline algorithm . To the best of our knowledge , the most related work considering generating data samples for COPs is Liu et al . ( 2020 ) where the samples are generated by performing crossover and mutation to existing instances . The goal is to search better parameters configuration for given traditional solvers . Whereas our proposed framework is designed to generate instances with the guidance of deep reinforcement learning and improve the generalization performance for neural combinatorial optimization models . In our experiments , we also verify the necessity of our data generation method . 3 MODEL . For most learning based methods , the patterns are learned from the training data and usually applicable to similar testing samples . However , if the distributions of training and testing samples are considerably different , these learned patterns may not be general enough to guarantee desirable inference performance . As we will show in the experiments , this issue is particularly severe for COPs due to the intricate relationship between the solution and the instance distribution . Instead of designing a network architecture to achieve better generalization performance for specific problems , we aim to propose a model agnostic framework which is generally applicable to different optimization models and training methods . The only requirement for the optimization models is that they are able to learn effective patterns from the training data and perform well on similar testing samples , which could be fulfilled by most of the recent learning based models . To this end , we propose the Generative Adversarial Neural Combinatorial Optimization ( GANCO ) framework , as shown in Figure 1 . Rather than training the optimization model solely on instances sampled from the predefined training distribution , we also deploy another deep learning model to generate instances on which the optimization model performs poorly . The generation model is trained by reinforcement learning to maximize the performance gap between the optimization model and a non-learning baseline algorithm on the generated instances . And the optimization model is trained with the generated samples to improve the performance on similar instances . In other words , the generation model and the optimization model are trained to maximize and minimize the performance gaps on the generated instances , respectively . In such an adversarial training way , the lower bound of the optimization model performance on the instance distributions generated by the generation model will be pushed up . We detail the generation model and the GANCO framework in the following subsections . 3.1 GENERATION MODEL . As suggested in existing works ( Kool et al. , 2019 ; Kwon et al. , 2020 ) , many important COPs can be viewed as a graph of n nodes with different attributes , like the city coordinates for TSP and the item information for KP . Therefore , we formulate the data generation task as determining the attribute values for each node of an instance . However , the proposed framework could be easily adapted to other types of COPs , such as by determining the presence of each edge for problems depending on the graph connectivity ( e.g. , the Boolean Satisfiability Problem ) . The generation model Ω takes the noise sample h for each node as the input to output the distribution parameters for the node attributes . We format the noise input h ∈ Rn×2 as 2-dimensional variables for each node j = 1 , .. , n following the unit uniform distribution independently , i.e. , h ∼ U ( 0 , 1 ) . Pertaining to the network architecture , we use the self-attention block proposed in the Attention Model ( AM , Kool et al . ( 2019 ) ) , which consists of a multi-head self-attention layer ( Vaswani et al. , 2017 ) , two node-wise feed-forward layers with Batch Normalization ( Ioffe & Szegedy , 2015 ) and skip connection ( He et al. , 2016 ) . The noise input is node-wise linearly projected to H-dimension and encoded by k self-attention blocks . And another node-wise linear layer with a sigmoid activation projects the hidden vectors to the output distribution parameters Ω ( h ) ∈ Rn×d where d is the number of attributes for each node . Then we draw the sample x̃ from the Gaussian distribution N ( µ , σ2 ) with mean µ = Ω ( h ) and fixed standard deviation σ . The network outputs are kept in the same range [ 0 , 1 ] to use the same variance for different attributes . The sample x̃ is scaled to the corresponding valid ranges ( and discretized for discrete attributes ) to attain the node attributes x = A ( x̃ ) . The goal of the generation model is to generate instances on which the optimization model currently performs poorly . Since different instances have different optimal objective values , the attained objective values alone are not meaningful enough to evaluate the model performance . Although the optimality gap appears to be a good alternative metric in this case , it often requires expensive computation to attain the optimal solutions . On the other hand , while achieving approximate solutions , traditional non-learning algorithms tend to perform relatively stably on instances with different distributions , compared to the learning based methods . Therefore , the performance gaps of the optimization model with respect to a traditional non-learning baseline algorithm could be regarded as a favorable metric . Theoretically , the baseline algorithm performance can be arbitrarily inferior as long as it performs consistently ( e.g. , similar optimality gaps ) on different instances . As we will show in our experiments , the baseline algorithms do not need to be very strong or fast as the training for generation models converges in a relatively small number of iterations . The generation model is trained by the reinforcement learning ( RL ) . Specifically , the state of RL environment is the noise input h ∈ Rn×2 . The action is to sample the attribute values for a node . An episode is to sample the attribute values x = A ( x̃ ) with x̃ ∼ N ( Ω ( h ) , σ2 ) for all the nodes to form an instance . However , as the sampled attributes for one node will not affect the other nodes , the episode can be viewed as only containing one action with large action space to sample all the node attributes at the same time . The reward for an episode is the performance gap between the optimization model Φ and baseline algorithm B on the sampled instance x . The loss function is formally defined as follow , L ( Ω ) = −Eh∼U ( 0,1 ) , x̃∼N ( Ω ( h ) , σ2 ) G ( x , Φ , B ) , ( 1 ) where G ( x , Φ , B ) = ( OΦ ( x ) − OB ( x ) ) /OB ( x ) is the performance ( objective ) gap1 ; OΦ ( x ) and OB ( x ) are the objective values of the solutions found by the optimization model Φ and the baseline algorithm B for the generated instance x , respectively . Different from the generator in GAN ( Goodfellow et al. , 2014 ) which deterministically infers the instance given the noise input , we sample the noise input to infer the distribution and sample the instance from the output distribution . The generation model is trained by the REINFORCE algorithm ( Sutton et al. , 2000 ) with reward baseline as the mean of the gaps in a batch . The gradient to train the generation model is expressed as follow , ∇ΩL ( Ω ) ≈ − 1 N N∑ i=1 G ( xi , Φ , B ) − 1 N N∑ j=1 G ( xj , Φ , B ) ∇ΩlogN ( x̃i ; Ω ( hi ) , σ2 ) , ( 2 ) where i and j are the instance indices ; N is the batch size ; x̃i ∈ Rn×d ∼ N ( Ω ( hi ) , σ2 ) ; and N ( x̃i ; Ω ( hi ) , σ2 ) is the probability of x̃i sampled from the Gaussian distribution N ( Ω ( hi ) , σ2 ) .

Summary: This work presents GANCO, a framework that extends the training of RL heuristics for combinatorial optimisation problems (COPs) to include an adversarial agent that aims to generate hard-to-solve problem instances.  The RL heuristic and adversarial instance generation are alternately trained, with the intuition that the adaptive problem distribution encourages the solving agent to learn policies with better generalisation performance to unseen instances compared to those trained on a single static distribution.  Experiments train the Attention Model of Kool et al (and a subsequent extension, POMO) within the GANCO framework and show that across multiple well-studied COPs (including the travelling salesman problem), GANCO augmented training does result in policies that generalise better to unseen distributions at test time.