Efficient Active Search for Combinatorial Optimization Problems

1 INTRODUCTION . In recent years , a wide variety of machine learning ( ML ) based methods for combinatorial optimization problems have been proposed . While early approaches failed to outperform traditional operations research methods , the gap between handcrafted and learned heuristics has been steadily closing . However , the main potential of ML-based methods lies not only in their ability to outperform existing methods , but in automating the design of customized heuristics in situations where no handcrafted heuristics have yet been developed . We hence focus on developing approaches that require as little additional problem-specific knowledge as possible . Existing ML based methods for combinatorial optimization problems can be classified into construction methods and improvement methods . Improvement methods search the space of complete solutions by iteratively refining a given start solution . They allow for a guided exploration of the search space and are able to find high-quality solutions . However , they usually rely on problemspecific components . In contrast , construction methods create a solution sequentially starting from an empty solution ( i.e. , they consider a search space consisting of incomplete solutions ) . At test time , they can be used to either greedily construct a single solution or to sample multiple solutions from the probability distribution encoded in the trained neural network . Furthermore , the sequential solution generation process can be easily integrated into a beam search without requiring any problem-specific components . However , search methods like sampling and beam search offer no ( or very limited ) search guidance . Additionally , these methods do not react towards the solutions seen so far , i.e. , the underlying distribution from which solutions are sampled is never changed throughout the search . Bello et al . ( 2016 ) propose a generic search strategy called active search that allows an extensive , guided search for construction methods without requiring any problem specific components . Active search is an iterative search method that at each iteration samples solutions for a single test instance using a given model and then adjusts the parameters of that model with the objective to increase the likelihood of generating high-quality solutions in future iterations . They report improved performance over random sampling when starting the search from an already trained model . Despite promising results , active search has not seen adaption in the literature . The reason for this is its resource requirements , as adjusting all model parameters separately for each test instance is very time intensive , especially compared to methods that can sample solutions to multiple different instances in one batch . We extend the idea of active search as follows . ( 1 ) We propose to only adjust a subset of ( model ) parameters to a single instance during the search , while keeping all other parameters fixed . We show that this efficient active search ( EAS ) drastically reduces the runtime of active search without impairing the solution quality . ( 2 ) We implement and evaluate three different implementations of EAS and show that all offer significantly improved performance over pure sampling approaches . In our EAS implementations , the majority of ( model ) parameters are not updated during the search , which drastically reduces the runtime , because gradients only need to be computed for a subset of model weights , and most operations can be applied identically across a batch of different instances . Furthermore , we show that for some problems , EAS finds even better solutions than the original active search . All EAS implementations can be easily applied to existing ML construction methods . We evaluate the proposed EAS approaches on the traveling salesperson problem ( TSP ) , the capacitated vehicle routing problem ( CVRP ) and the job shop scheduling problem ( JSSP ) . For all problems , we build upon already existing construction approaches that only offer limited search capabilities . In all experiments , EAS leads to significantly improved performance over sampling approaches . For the CVRP and the JSSP , the EAS approaches outperform all state-of-the-art ML based approaches , and even the well-known heuristic solver LKH3 for the CVRP . Furthermore , EAS approaches assists in model generalization , resulting in drastically improved performance when searching for solutions to instances that are much larger than the instances seen during model training . 2 LITERATURE REVIEW . Construction methods Hopfield ( 1982 ) first used a neural network ( a Hopfield network ) to solve small TSP instances with up to 30 cities . The development of recent neural network architectures has paved the way for ML approaches that are able to solve large instances . The pointer network architecture proposed by Vinyals et al . ( 2015 ) efficiently learns the conditional probability of a permutation of a given input sequence , e.g. , a permutation of cities for a TSP solution . The authors solve TSP instances with up to 50 cities via supervised learning . Bello et al . ( 2016 ) report that training a pointer network via actor-critic RL instead results in a better performance on TSP instances with 50 and 100 cities . Furthermore , graph neural networks are used to solve the TSP , e.g. , a graph embedding network in Khalil et al . ( 2017 ) and a graph attention network in Deudon et al . ( 2018 ) . The first applications of neural network based methods to the CVRP are reported by Nazari et al . ( 2018 ) and Kool et al . ( 2019 ) . Nazari et al . ( 2018 ) propose a model with an attention mechanism and a recurrent neural network ( RNN ) decoder that can be trained via actor-critic RL . Kool et al . ( 2019 ) propose an attention model that uses an encoder that is similar to the encoder used in the transformer architecture Vaswani et al . ( 2017 ) . Peng et al . ( 2019 ) and Xin et al . ( 2021 ) extend the attention model to update the node embeddings throughout the search , resulting in improved performance at the cost of longer runtimes for the CVRP . Falkner & Schmidt-Thieme ( 2020 ) propose an attention-based model that constructs tours in parallel for the CVRP with time windows . While ML-based construction methods have mainly focused on routing problems , there are some notable exceptions . For example , Khalil et al . ( 2017 ) use a graph embedding network approach to solve the minimum vertex cover and the maximum cut problems ( in addition to the TSP ) . Zhang et al . ( 2020 ) propose a graph neural network based approach for the job shop scheduling problem ( JSSP ) . Li et al . ( 2018 ) use a guided tree search enhanced ML approach to solve the maximal independent set , minimum vertex cover , and the maximal clique problems . For a more detailed review of ML methods on different combinatorial optimization problems , we refer to Vesselinova et al . ( 2020 ) . While most approaches construct routing problem solutions autoregressively , some approaches predict a heat-map that describes which edges will likely be part of a good solution . The heat-map is then used in a post-hoc search to construct solutions . Joshi et al . ( 2019 ) use a graph convolutional network to create a heat-map and a beam search to search for solutions . Similarly , Fu et al . ( 2020 ) use a graph convolutional residual network with Monte Carlo tree search to solve large TSP instances . Kool et al . ( 2021 ) use the model from Joshi et al . ( 2019 ) to generate the heat-map and use it to search for solution to TSP and CVRP instances with a dynamic programming based approach . Improvement methods Improvement methods integrate ML based methods into high-level search heuristics or try to learn improvement operators directly . In general , they often invest more time into solving an instance than construction based methods ( and usually find better solutions ) . Chen & Tian ( 2019 ) propose an approach that iteratively changes a local part of the solution . At each iteration , the trainable region picking policy selects a part of the solution that should be changed and a trainable rule picking policy selects an action from a given set of possible modification operations . Hottung & Tierney ( 2020 ) propose a method for the CVRP that iteratively destroys parts of a solution using predefined , handcrafted operators and then reconstructs them with a learned repair operator . Wu et al . ( 2021 ) and de O. da Costa et al . ( 2020 ) propose to use RL to pick an improving solution from a specified local neighborhood ( e.g. , the 2-Opt neighborhood ) to solve routing problems . Hottung et al . ( 2021 ) learn a continuous representation of discrete routing problem solutions using conditional variational autoencoders and search for solutions using a generic , continuous optimizer . 3 SOLVING COMBINATORIAL OPTIMIZATION PROBLEMS WITH EAS . We propose three EAS implementations that adjust a small subset of ( model ) parameters in an iterative search process . Given an already trained model , we investigate adjusting ( 1 ) the normally static embeddings of the problem instance that are generated by the encoder model , ( 2 ) the weights of additional instance-specific residual layers added to the decoder , and ( 3 ) the parameters of a lookup table that directly affect the probability distribution returned by model . In each iteration , multiple solutions are sampled for one instance and the dynamic ( model ) parameters are adjusted with the goal of increasing the probability of generating high quality solutions ( as during model training ) . This allows the search to sample solutions of higher quality in subsequent iterations , i.e. , the search can focus on the more promising areas of the search space . Once a high-quality solution for an instance is found , the adjusted parameters are discarded , so that the search process can be repeated on other instances . All strategies efficiently generate solutions to a batch of instances in parallel , because the network layers not updated during the search are applied identically to all instances of the batch . Background RL based approaches for combinatorial problems aim to learn a neural network based model pθ ( π|l ) with weights θ that can be used to generate a solution π given an instance l. State-ofthe-art approaches usually use a model that consists of an encoder and a decoder unit . The encoder usually creates static embeddings ω that describe the instance l using a computationally expensive encoding process ( e.g. , Kool et al . ( 2019 ) ; Kwon et al . ( 2020 ) ) . The static embeddings are then used to autoregessively construct solutions using the decoder over T time steps . At each step t , the decoder qφ ( a|st , ω ) , with weights φ ⊂ θ , outputs a probability value for each possible action a in the state st ( e.g. , for the TSP , each action corresponds to visiting a different city next ) . The starting state s1 describes the problem instance l ( e.g. , the positions of the cities for the TSP and the starting city ) and the state st+1 is obtained by applying the action at selected at time step t to the state st . The ( partial ) solution πt is defined by the sequence of selected actions a1 , a2 , . . . , at . Once a complete solution , πT , fulfilling all constraints of the problem is constructed , the objective function value C ( π , l ) of the solution can be computed ( e.g. , the tour length for the TSP ) . Figure 1 shows the solution generation for a TSP instance with a model that uses static embeddings . The static embeddings ω are used at each decoding step to generate a probability distribution over all possible next actions and the selected action is provided to the decoder in the next decoding step . During testing , solutions can be constructed by either selecting actions greedily or by sampling each action according to qφ ( a|st , ω ) . Since the static embeddings are not updated during solution generation they only need to be computed once per instance , which allows to quickly sample multiple solutions per instance . We note that not all models use static embeddings . Some approaches update all instance embeddings after each action ( e.g. , Zhang et al . ( 2020 ) ) , which allows the embeddings to contain information on the current solution state st .

Summary: The paper deals with end-to-end learning of heuristics for combinatorial optimization problems. The authors propose an extension of the active search method of [Bello et al 2016], where only part of the model parameters are updated at test time for each instance. They propose three ways of applying this idea that consist in fine-tuning part of the instance embeddings, the parameters of an additional layer or directly the prediction scores of the model. Applied to the POMO method [Kwon et al 2020] for the TSP and CVPR and the L2D method of [Zhang et al 2020] for the JSSP, the proposed efficient active search leads to significant improvements on instances of the same size and larger than the training ones.