Learning Collaborative Policies to Solve NP-hard Routing Problems

1 Introduction . Routing is a combinatorial optimization problem , one of the prominent fields in discrete mathematics and computational theory . Among routing problems , the traveling salesman problem ( TSP ) is a canonical example . TSP can be applied to real-world problems in various engineering fields , such as robot routing , biology , and electrical design automation ( EDA ) [ 1 , 2 , 3 , 4 , 5 ] by expanding constraints and objectives to real-world settings : coined TSP variants are expanded version of TSP . However , TSP and its variants are NP-hard , making it challenging to design an exact solver [ 6 ] . Due to NP-hardness , solvers of TSP-like problems rely on mixed-integer linear programming ( MILP ) solvers [ 7 ] and handcrafted heuristics [ 8 , 9 ] . Although they often provide a remarkable performance on target problems , the conventional approaches have several limitations . Firstly , in the case of MILP solvers , the objective functions and constraints must be formulated into linear forms , but many real-world routing applications , including biology and EDA , have a non-linear objective . Secondly , handcrafted heuristics rely on expert knowledge on target problems , thus hard to solve other problems . That is , whenever the target problem changes , the algorithm must also be re-designed . Deep reinforcement learning ( DRL ) -routing frameworks [ 10 , 11 , 12 ] is proposed to tackle the limitation of conventional approaches . One of the benefits of DRL is that reward of DRL can be any 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . value , even from a black-box simulator ; therefore , DRL can overcome the limitations of MILP on real-world applications . Moreover , DRL frameworks can automatically design solvers relying less on a handcrafted manner . We note that the main objective of our research is not outperforming problem-specific solvers like the Concorde [ 9 ] , a TSP solver . Our problem-solving strategy based on DRL , however , ultimately focuses on practical applications1 including intelligent transportation [ 13 ] , biological sequence design [ 14 ] , routing on electrical device [ 15 ] and device placement [ 16 , 17 ] . Therefore , this paper evaluates the performance of DRL frameworks on TSP-like problems as a benchmark for potential applicability to practical applications , including speed , optimality , scalability , and expand-ability to other problems . TSP-like problems are excellent benchmarks as they have various baselines to compare with and can easily be modeled and evaluated . Contribution . This paper presents a novel DRL scheme , coined learning collaborative policies ( LCP ) , a hierarchical solving protocol with two policies : seeder and reviser . The seeder generates various candidate solutions ( seeds ) , each of which will be iteratively revised by the reviser to generate fine-tuned solutions . Having diversified candidate solutions is important , as it gives a better chance to find the best solution among them . Thus , the seeder is dedicated to exploring the full combinatorial action space ( i.e. , sequence of assignment action ) so that it can provide as diversified candidate solutions as possible . It is important to explore over the full combinatorial action space because the solution quality highly fluctuates depending on its composition ; however , exploring over the combinatorial action space is inherently difficult due to its inevitably many possible solutions . Therefore , this study provides an effective exploration strategy applying an entropy maximization scheme . The reviser modifies each candidate solution generated by the seeder . The reviser is dedicated to exploiting the policy ( i.e. , derived knowledge about the problem ) to improve the quality of the candidate solution . The reviser partitions the full trajectory into sub-tours and revises each subtour to minimize its traveling distance in a parallel manner . This scheme provides two advantages : ( a ) searching over the restricted solution space can be more effective because the reward signal corresponding to the sub-tour is less variable than that of the full trajectory when using reinforcement learning to derive a policy , and ( b ) searching over sub-tours of seeds can be parallelized to expedite the revising process . The most significant advantage of our method is that the reviser can re-evaluate diversified but underrated candidates from the seeder without dropping it out early . Since the seeder explores the full trajectory , there may be a mistake in the local sub-trajectory . Thus , it is essential to correct such mistakes locally to improve the solution quality . The proposed revising scheme parallelizes revising process by decomposing the full solution and locally updating the decomposed solution . Thus it allows the revisers to search over larger solution space in a single inference than conventional local search ( i.e. , number of iteration of the reviser is smaller than that of conventional local search 2-opt [ 18 ] , or DRL-based 2-opt [ 19 ] ) , consequently reducing computing costs . Therefore , we can keep the candidates without eliminating them early because of computing costs . The proposed method is an architecture-agnostic method , which can be applied to various neural architectures . The seeder and reviser can be parameterized with any neural architecture ; this research utilizes AM [ 12 ] , the representative DRL model on combinatorial optimization , to parameterize the seeder and the reviser . According to the experimental results , the LCP improves the target neural architecture AM [ 12 ] , and outperforms competitive DRL frameworks on TSP , PCTSP , and CVRP ( N = 20 , 50 , 100 , 500 , N : number of nodes ) and real-world problems in TSPLIB [ 20 ] . Moreover , by conducting extensive ablation studies , we show proposed techniques , including entropy regularization scheme and revision scheme , clearly contribute to the performance improvement . 2 Related Works . There have been continuous advances in DRL frameworks for solving various routing problems . DRL framework can generate solvers that do not rely on the ground-truth label of target problems : it can be applied to un-explored problems . DRL-based approaches can be categorized into two parts ; 1These works [ 13 , 14 , 15 , 16 , 17 ] are inspired by DRL frameworks [ 10 , 12 ] on combinatorial optimization constructive heuristics and improvement heuristics . We survey these two categories and current emerging hybrid approaches of machine learning ( ML ) with conventional solvers . 2.1 DRL-based Constructive Heuristics . Bello et al . [ 10 ] introduced an actor-critic algorithm with a policy parameterized by the pointer network [ 21 ] . They proposed a constructive Markov decision process ( MDP ) , where the action is defined as choosing one of the un-served nodes to visit , given a partial solution ; the policy is trained to add a node to provide a complete solution sequentially . Later , DRL-based constructive heuristics were developed to design the architecture of neural networks while preserving the constructive MDP [ 10 ] . Khalil et al . [ 11 ] proposed a DRL framework with a graph embedding structure . Nazari et al . [ 22 ] , Duedon et al . [ 23 ] and Kool et al . [ 12 ] redesigned the pointer network [ 21 ] using the transformer [ 24 ] and trained it with a policy gradient method [ 25 ] . The AM by Kool et al . [ 12 ] reports substantial results on various NP-hard routing problems , including TSP , PCTSP , CVRP , and orienteering problem ( OP ) in high-speed computation . AM-variants . After the meaningful success of the AM , many studies are expanded from the AM . Many engineering fields and industries apply AM into their domain . For example , Liao et al . [ 4 ] proposed a routing algorithm for the circuit using AM . Some researches focus on increasing the performances of AM on classic routing problems like TSP by simple techniques . Kwon et al . [ 26 ] proposed the POMO , effective reinforcement learning method for AM . They proposed a new RL baseline that can reduce the training variance of AM using the problem-specific property of TSP and CVRP . In addition , they presented an effective post-processing algorithm for TSP and CVRP . However , their proposed method has a limitation in that it is problem-specific because it uses the domain properties of TSP and CVRP ( e.g. , their method is limited to be applied to PCTSP. ) . Xin et al . [ 27 ] proposed AM-style DRL-model , MDAM , for NP-hard routing problems . Their method learns multiple AM decoders and derives various solutions through the multiple decoders . The goal of increasing the solution diversity is similar to our research . However , our study is different where it increases the entropy of a single decoder and improves the mistakes of various solutions through a reviser . 2.2 DRL-based Improvement Heuristics . Unlike the constructive MDP , DRL-based improvement heuristics are designed to improve the completed solution iteratively . Most researches on DRL-based improvement heuristics are inspired by classical local search algorithms such as 2-opt [ 18 ] and the large neighborhood search ( LNS ) [ 28 ] . Chen et al . [ 29 ] proposed a DRL-based local search framework , termed NeuRewriter , that shows a promising performance on CVRP and job scheduling problems . Wu et al . [ 30 ] , and Costa et al . [ 31 ] proposed a DRL-based TSP solver by learning the 2-opt . Their method improves the randomly generated solutions , unlike the method of Chen et al . [ 29 ] rewrites a solution given by a conventional heuristic solver . Hottung & Tierney [ 32 ] proposed a novel search method of VRP that destroys and repairs a solution repeatably inspired LNS . Their method gives promising performances on CVRP . Improvement heuristic approaches generally show better performance than constructive heuristics but are usually slower than constructive heuristics . In the case of TSP , the number of neural network ’ s inferences of constructive heuristics is the same as the number of cities to visit . However , the number of inferences of the improvement heuristics is generally much larger . 2.3 Hybrid Approaches with Conventional Solvers . There are several studies on hybrid approaches with conventional solvers having promising performance recently . Lu et al . [ 33 ] proposed a hybrid method , where the policy is learned to control improvement operators ( handcrafted heuristic ) . Significantly , they outperforms the LKH3 , which is widely considered as mountain to climb in machine learning ( ML ) communities . Joshi et al . [ 34 ] combined graph neural network ( GNN ) model with the beam search algorithm . They trained the GNN with supervised learning for generating a hit map of candidate nodes . Then trained GNN reduces a searching space for improvement heuristics . Similarly , Fu et al . [ 35 ] combined supervised GNN model with Monte Carlo tree search ( MCTS ) and Kool et al . [ 36 ] combined supervised GNN model with dynamic programming . Their method achieves significant performances , showing ML method can effectively collaborate with conventional operational research ( OR ) methods . The research scope of hybrid approaches and DRL-based methods is different . Hybrid approaches can overcome classical solvers in target tasks by collaborating with the classical solvers . However , hybrid approaches have inherited limitations from classical solvers that are poor expandability to other tasks . The DRL-based method can be applied to various real-world tasks without a classic solver ; we can also utilize DRL-based to unexplored tasks . This paper investigates the DRL-based NP-hard routing method without the help of classical solvers .

Summary: Deep reinforcement learning (DRL) has recently been applied to variety of optimization problems, outperforming traditional approaches. This paper investigates the application of DRL to the context of routing problems such as the classical travelling salesman problem. The methodology proposed involves two DRL agents: the seeder, which is tasked with generating candidates for the solution that provide a good coverage of the solution space, and the reviser, which is tasked with attaining better solutions based on the reduced solution space provided by the seeder. To generate diverse candidates, the seeder utilizes entropy regularization reward. The authors present experiments indicating that their framework, called learning collaborative policies (LCP) improves over previous DRL frameworks for various routing challenges (e.g., TSP and capacitated vehicle routing).