Efficient Active Search for Combinatorial Optimization Problems

1 INTRODUCTION . In recent years , a wide variety of machine learning ( ML ) based methods for combinatorial optimization problems have been proposed . While early approaches failed to outperform traditional operations research methods , the gap between handcrafted and learned heuristics has been steadily closing . However , the main potential of ML-based methods lies not only in their ability to outperform existing methods , but in automating the design of customized heuristics in situations where no handcrafted heuristics have yet been developed . We hence focus on developing approaches that require as little additional problem-specific knowledge as possible . Existing ML based methods for combinatorial optimization problems can be classified into construction methods and improvement methods . Improvement methods search the space of complete solutions by iteratively refining a given start solution . They allow for a guided exploration of the search space and are able to find high-quality solutions . However , they usually rely on problemspecific components . In contrast , construction methods create a solution sequentially starting from an empty solution ( i.e. , they consider a search space consisting of incomplete solutions ) . At test time , they can be used to either greedily construct a single solution or to sample multiple solutions from the probability distribution encoded in the trained neural network . Furthermore , the sequential solution generation process can be easily integrated into a beam search without requiring any problem-specific components . However , search methods like sampling and beam search offer no ( or very limited ) search guidance . Additionally , these methods do not react towards the solutions seen so far , i.e. , the underlying distribution from which solutions are sampled is never changed throughout the search . Bello et al . ( 2016 ) propose a generic search strategy called active search that allows an extensive , guided search for construction methods without requiring any problem specific components . Active search is an iterative search method that at each iteration samples solutions for a single test instance using a given model and then adjusts the parameters of that model with the objective to increase the likelihood of generating high-quality solutions in future iterations . They report improved performance over random sampling when starting the search from an already trained model . Despite promising results , active search has not seen adaption in the literature . The reason for this is its resource requirements , as adjusting all model parameters separately for each test instance is very time intensive , especially compared to methods that can sample solutions to multiple different instances in one batch . We extend the idea of active search as follows . ( 1 ) We propose to only adjust a subset of ( model ) parameters to a single instance during the search , while keeping all other parameters fixed . We show that this efficient active search ( EAS ) drastically reduces the runtime of active search without impairing the solution quality . ( 2 ) We implement and evaluate three different implementations of EAS and show that all offer significantly improved performance over pure sampling approaches . In our EAS implementations , the majority of ( model ) parameters are not updated during the search , which drastically reduces the runtime , because gradients only need to be computed for a subset of model weights , and most operations can be applied identically across a batch of different instances . Furthermore , we show that for some problems , EAS finds even better solutions than the original active search . All EAS implementations can be easily applied to existing ML construction methods . We evaluate the proposed EAS approaches on the traveling salesperson problem ( TSP ) , the capacitated vehicle routing problem ( CVRP ) and the job shop scheduling problem ( JSSP ) . For all problems , we build upon already existing construction approaches that only offer limited search capabilities . In all experiments , EAS leads to significantly improved performance over sampling approaches . For the CVRP and the JSSP , the EAS approaches outperform all state-of-the-art ML based approaches , and even the well-known heuristic solver LKH3 for the CVRP . Furthermore , EAS approaches assists in model generalization , resulting in drastically improved performance when searching for solutions to instances that are much larger than the instances seen during model training . 2 LITERATURE REVIEW . Construction methods Hopfield ( 1982 ) first used a neural network ( a Hopfield network ) to solve small TSP instances with up to 30 cities . The development of recent neural network architectures has paved the way for ML approaches that are able to solve large instances . The pointer network architecture proposed by Vinyals et al . ( 2015 ) efficiently learns the conditional probability of a permutation of a given input sequence , e.g. , a permutation of cities for a TSP solution . The authors solve TSP instances with up to 50 cities via supervised learning . Bello et al . ( 2016 ) report that training a pointer network via actor-critic RL instead results in a better performance on TSP instances with 50 and 100 cities . Furthermore , graph neural networks are used to solve the TSP , e.g. , a graph embedding network in Khalil et al . ( 2017 ) and a graph attention network in Deudon et al . ( 2018 ) . The first applications of neural network based methods to the CVRP are reported by Nazari et al . ( 2018 ) and Kool et al . ( 2019 ) . Nazari et al . ( 2018 ) propose a model with an attention mechanism and a recurrent neural network ( RNN ) decoder that can be trained via actor-critic RL . Kool et al . ( 2019 ) propose an attention model that uses an encoder that is similar to the encoder used in the transformer architecture Vaswani et al . ( 2017 ) . Peng et al . ( 2019 ) and Xin et al . ( 2021 ) extend the attention model to update the node embeddings throughout the search , resulting in improved performance at the cost of longer runtimes for the CVRP . Falkner & Schmidt-Thieme ( 2020 ) propose an attention-based model that constructs tours in parallel for the CVRP with time windows . While ML-based construction methods have mainly focused on routing problems , there are some notable exceptions . For example , Khalil et al . ( 2017 ) use a graph embedding network approach to solve the minimum vertex cover and the maximum cut problems ( in addition to the TSP ) . Zhang et al . ( 2020 ) propose a graph neural network based approach for the job shop scheduling problem ( JSSP ) . Li et al . ( 2018 ) use a guided tree search enhanced ML approach to solve the maximal independent set , minimum vertex cover , and the maximal clique problems . For a more detailed review of ML methods on different combinatorial optimization problems , we refer to Vesselinova et al . ( 2020 ) . While most approaches construct routing problem solutions autoregressively , some approaches predict a heat-map that describes which edges will likely be part of a good solution . The heat-map is then used in a post-hoc search to construct solutions . Joshi et al . ( 2019 ) use a graph convolutional network to create a heat-map and a beam search to search for solutions . Similarly , Fu et al . ( 2020 ) use a graph convolutional residual network with Monte Carlo tree search to solve large TSP instances . Kool et al . ( 2021 ) use the model from Joshi et al . ( 2019 ) to generate the heat-map and use it to search for solution to TSP and CVRP instances with a dynamic programming based approach . Improvement methods Improvement methods integrate ML based methods into high-level search heuristics or try to learn improvement operators directly . In general , they often invest more time into solving an instance than construction based methods ( and usually find better solutions ) . Chen & Tian ( 2019 ) propose an approach that iteratively changes a local part of the solution . At each iteration , the trainable region picking policy selects a part of the solution that should be changed and a trainable rule picking policy selects an action from a given set of possible modification operations . Hottung & Tierney ( 2020 ) propose a method for the CVRP that iteratively destroys parts of a solution using predefined , handcrafted operators and then reconstructs them with a learned repair operator . Wu et al . ( 2021 ) and de O. da Costa et al . ( 2020 ) propose to use RL to pick an improving solution from a specified local neighborhood ( e.g. , the 2-Opt neighborhood ) to solve routing problems . Hottung et al . ( 2021 ) learn a continuous representation of discrete routing problem solutions using conditional variational autoencoders and search for solutions using a generic , continuous optimizer . 3 SOLVING COMBINATORIAL OPTIMIZATION PROBLEMS WITH EAS . We propose three EAS implementations that adjust a small subset of ( model ) parameters in an iterative search process . Given an already trained model , we investigate adjusting ( 1 ) the normally static embeddings of the problem instance that are generated by the encoder model , ( 2 ) the weights of additional instance-specific residual layers added to the decoder , and ( 3 ) the parameters of a lookup table that directly affect the probability distribution returned by model . In each iteration , multiple solutions are sampled for one instance and the dynamic ( model ) parameters are adjusted with the goal of increasing the probability of generating high quality solutions ( as during model training ) . This allows the search to sample solutions of higher quality in subsequent iterations , i.e. , the search can focus on the more promising areas of the search space . Once a high-quality solution for an instance is found , the adjusted parameters are discarded , so that the search process can be repeated on other instances . All strategies efficiently generate solutions to a batch of instances in parallel , because the network layers not updated during the search are applied identically to all instances of the batch . Background RL based approaches for combinatorial problems aim to learn a neural network based model pθ ( π|l ) with weights θ that can be used to generate a solution π given an instance l. State-ofthe-art approaches usually use a model that consists of an encoder and a decoder unit . The encoder usually creates static embeddings ω that describe the instance l using a computationally expensive encoding process ( e.g. , Kool et al . ( 2019 ) ; Kwon et al . ( 2020 ) ) . The static embeddings are then used to autoregessively construct solutions using the decoder over T time steps . At each step t , the decoder qφ ( a|st , ω ) , with weights φ ⊂ θ , outputs a probability value for each possible action a in the state st ( e.g. , for the TSP , each action corresponds to visiting a different city next ) . The starting state s1 describes the problem instance l ( e.g. , the positions of the cities for the TSP and the starting city ) and the state st+1 is obtained by applying the action at selected at time step t to the state st . The ( partial ) solution πt is defined by the sequence of selected actions a1 , a2 , . . . , at . Once a complete solution , πT , fulfilling all constraints of the problem is constructed , the objective function value C ( π , l ) of the solution can be computed ( e.g. , the tour length for the TSP ) . Figure 1 shows the solution generation for a TSP instance with a model that uses static embeddings . The static embeddings ω are used at each decoding step to generate a probability distribution over all possible next actions and the selected action is provided to the decoder in the next decoding step . During testing , solutions can be constructed by either selecting actions greedily or by sampling each action according to qφ ( a|st , ω ) . Since the static embeddings are not updated during solution generation they only need to be computed once per instance , which allows to quickly sample multiple solutions per instance . We note that not all models use static embeddings . Some approaches update all instance embeddings after each action ( e.g. , Zhang et al . ( 2020 ) ) , which allows the embeddings to contain information on the current solution state st .

Summary: The paper proposes a new method of updating deep neural networks for combinatorial optimization problems during search using reinforcement learning. In particular, the authors show that by updating only part of the network, better results can be achieved at lower cost. They describe and evaluate their method on different combinatorial optimization problems, comparing to other machine-learning-based approaches as well as "traditional" solvers.