A new dog learns old tricks:  RL finds classic optimization algorithms

1 INTRODUCTION . Machine learning has led to dramatic improvements in our capabilities to solve problems previously considered intractable . Besides the obvious empirical evidence of success , there has also been a strong parallel effort in the theory of ML which aims to explain why , when , and how ML techniques work . Our goal in this paper is to explore whether machine learning can be used to learn algorithms for classic combinatorial optimization problems . We will define this question more specifically by connecting to three concepts from traditional algorithms and complexity theory . 1.1 INPUT LENGTH INDEPENDENCE AND THE CONNECTION TO RL . Firstly , by “ algorithm , ” we mean a uniform algorithm , one that works for inputs of all lengths , not just for specific input lengths from which the training data is drawn . Typically , models learned using ML techniques tend to be non-uniform , i.e. , depend on input length . Previous approaches to finding uniform models — the Neural Turing machines of Graves et al . ( 2014 ) and generally the use of recurrent models ( including LSTMs ) — all suffer from some drawback , most notably the difficulty of training by back-propagation and gradient descent over long sequences . A particularly clever approach , due to Kaiser and Sutskever ( 2015 ) , adopts the idea of learning “ convolution masks ” of finite size that , when repeatedly applied solve a problem of interest on inputs of arbitrary length ; however , the resulting learning problems appear intractable ( since the volume of computation grows at least cubically in their setup for most interesting problems , and stable ways to learn convolutions over such large grids are not well-understood ) . We expand on this point in Appendix F. Our first key insight is that for numerous combinatorial optimization problems , the primal-dual framework offers efficient solutions , and also lends itself to efficient online algorithms ( see , e.g. , Buchbinder et al . ( 2009 ) ) where the input arrives in small units , one at a time , and the algorithm makes a choice about the input ( e.g. , which advertiser to give a query impression to , which node to match in a graph , whether to include an item in the knapsack , etc. ) . In addition , there is usually a clear notion of reward associated with a set of decisions , and the goal is often to optimize the overall rewards collected . This naturally connects our goal to the field of reinforcement learning , and indeed we formulate our learning problems in the Markov Decision Process ( MDP ) framework and use tools from deep reinforcement learning using policy gradient and DQN methods . Specifically , for any optimization problem , the MDP state will consist of three parts : global parameters of the problem instance ( e.g. , Knapsack size ) , a data structure that we expect ( and train ) the algorithm to learn to maintain ( and whose size can depend on the global parameters ) , and the current input unit . We will train two agents — U that computes an update to the data structure , and D that makes the decision for each input using the data structure and the current input . The RL environment will then carry out the task of applying the update to the data structure , and present it back to the agents U and D as part of the state for the next input . We establish theoretically ( Appendix G ) that this simple framework is flexible to capture a wide class of problems , and empirically that the resulting algorithms are quite powerful . 1.2 ADVERSARIALLY CHOSEN INPUT DISTRIBUTIONS . An important question in both ML and Algorithms is what input instances is the algorithm expected to work on . The ML approach is to use a rich enough training set to capture future inputs distributions . Theoretical computer science ( TCS ) , by contrast , traditionally considers worst-case analysis : an algorithm is judged by its performance on the worst possible input ( specially crafted by an Adversary to beat the algorithm ) . This approach leads to theoretically robust guarantees on the algorithm . In stochastic models of input , the Adversary is somewhat restricted in order to better capture “ real ” inputs — including the Random Order and the IID models of input . Our second key insight is to bring this approach of adversarial input sets ( not to be confused with the notion of adversarial examples which fool ML models , for example , see Goodfellow et al . ( 2014 ) ) to the ML domain via two techniques to craft training sets : 1 . Universal Training Set A common way to prove lower bounds in the TCS literature is to come up with a distribution over inputs and show that no algorithm can perform better than some factor α ≤ 1 compared to the optimal solution , in expectation . This is a key ingredient in the technique of using Yao ’ s Lemma Yao ( 1977 ) to prove a lower bound on the performance of all randomized algorithms . For example , in the Adwords problem , there is a specific input distribution which is hard for all online algorithms ( Karp et al . ( 1990 ) ; Mehta et al . ( 2007 ) ) . Intuitively , one might expect that if an algorithm does perform well on the specified input distribution then it must have learned some characteristics of the optimal algorithm . We bring this idea to the ML literature by proposing to incorporate such instances into the training . 2 . High-Entropy Training Set In some cases , it may difficult to find a universal training set or the universal training set may admit algorithms which perform well on the training set while performing poorly on all other instances . To alleviate this problem we also propose to incorporate training sets that have high entropy . For example , in the Adwords problem , a randomized greedy algorithm is able to perform quite well on the adversarial instance so we incorporate a distribution which is explicitly bad for greedy . In the secretary problem , we provide inputs which come from many different distributions so that it is difficult for it to learn utilize any characteristics of the distributions . 1.3 DECODING THE NETWORK TO FIND THE ALGORITHM . Our third contribution is the following intriguing question , connected to the broad area of how to interpret ML models . Specifically , suppose that for a given problem , we do manage to learn a network of constant ( fixed ) size , which does well over inputs of varying lengths coming from varying distributions . Does this allow us to confidently say that the network has learned the correct algorithm ? One observation is that since the network is concise , it has to represent a succinct logic . How does that compare to the optimal pen-and-paper algorithm that computer scientists have developed for the problem ? We will answer such questions by plotting the input-output characteristics of the network learned for the different problems we consider , and compare them to the expected behavior of the traditional algorithms . It may even be possible to convert the network to an algorithmic form , but we leave such an attempt for future work . 1.4 SUMMARY OF RESULTS . We study three optimization problems in this work – the Adwords Problem ( aka Online Budgeted Allocation ) , the Online ( 0-1 ) Knapsack Problem , and the so called Secretary Problem1 . All three problems share the feature that they are all very well-studied online combinatorial problems with some probabilistic features , and importantly , the optimal algorithms for each of them have a concise algorithmic form ( e.g. , not represented implicitly as the solution of a dynamic program ) . For all three problems we use RL to find a “ uniform algorithm ” , i.e. , an input-length independent logic . We train the models using universal or high-entropy input distributions and find that the models discover the classic algorithms . To mention the highlights of each section : • Adwords problem : The model learns to find the Balance strategy ( Kalyanasundaram and Pruhs , 2000 ) for unweighted graphs , and the MSVV strategy ( Mehta et al. , 2007 ) for weighted graphs which optimally trades-off between load-balancing and greedy strategies . • Online Knapsack problem : The model learns to find an optimal threshold on value per unit size to use to either accept or reject incoming items . • Secretary Problem : The model learns the optimal “ Wait-then-Pick ” algorithm which samples the first 1/e fraction of the input stream and then picks the next item which is higher than any seen before . It also finds the optimal time-dependent value-threshold algorithm for i.i.d . input . Our results suggest that it might be possible to draw a formal connection between the online primaldual framework and RL , e.g. , to prove that the online optimization problems solvable in the primaldual framework admit efficient algorithms learnable via RL . We leave this as a fascinating open question for future work . Remark . In this paper , we use the standard REINFORCE algorithm for policy gradient , with the Adam optimizer . Our contribution is not in extending RL techniques , but in making the connection to algorithms , and showing how standard RL techniques can in fact find the classic “ pen-and-paper ” algorithms . Further , we do not optimize for the training set-up or hyperparameters ; in particular all our training is done over a single machine and training often takes less than a day or two . 1.5 RELATED WORK . We are taking a specific angle at the question of how machine learning solves optimization problems . There is a lot of previous work on the larger question of ML and optimization . A related previous work is that of Bello et al . ( 2016 ) which also studies combinatorial problems , particularly the Traveling Salesman Problem and Knapsack , and also uses policy gradient method for an RL framework to optimize the parameters of a pointer network . ( This paper also summarizes previous literature on combinatorial optimization using neural networks . ) Our work differs in a few ways , but specifically the goal is not only to solve the problem , but also to interpret the learned RL policy network and compare to the known optimal algorithms , both in performance and in structure . Moreover , the work of Bello et al . ( 2016 ) learns a recurrent network , which could become prohibitively expensive to train on data sets that are large enough to capture the complexity of TSP or Knapsack . Another closely related paper is ( Dai et al. , 2017 ) , which uses embeddings and RL to find heuristics to solve classic graph problems on specific distributions . The problems they consider are offline in nature , and the heuristics conform to an incremental ( greedy ) policy guided by scores generated by the RL agent . Specifically , their goal is to find new heuristics for specific distributions , which is different from the work here , where we ask if RL can discover the classic “ worst-case ” algorithms . Our work is also different in the same way from other work in the space of combinatorial problems , such as that on TSP and Vehicle routing ( Kool and Welling , 2018 ) , as well as to the growing literature on using RL to solve optimization for control problems ( see for e.g . Lillicrap et al . ( 2015 ) , Levine et al . ( 2016 ) ) . We also mention as a loosely related paper by Boutilier and Lu ( 2016 ) , which uses RL ( as a budgeted MDP ) the solve Budget Allocation problem , although that problem is different from the Adwords problem we consider , in that the question there is to optimally allocate a single advertiser ’ s budget . 1This is the somewhat unfortunate prevailing name for this problem of Optimal Stopping — we continue using this name in this paper to avoid technical confusion . One of the goals in this work is to find a uniform algorithm , i.e. , one which is independent of the input length , for which we use the RL approach and focus on online optimization problems . As mentioned earlier , there have been several other nice approaches for this problem , each with different difficulty level in the goals , and obstacles in learning . This includes the Neural Turing machines of Graves et al . ( 2014 ) and generally the use of recurrent models ( including LSTMs ) , and the “ convolution masks ” approach of Kaiser and Sutskever ( 2015 ) . Finally , from the algorithms literature , our three problems are very well-studied , especially with Knapsack ( e.g. , Dantzig ( 1957 ) ) and Secretary ( Dynkin , 1963 ) being decades old problems . The relatively recent Adwords problem ( Mehta et al. , 2007 ) is strongly motivated by online advertising ( see , e.g. , Mehta et al . ( 2013 ) ) , and for which solutions that merge both the theoretical approach and the ML approach could potentially be of high impact in practice of budgeted ad allocations . Algorithmic work on these problems is cited throughout the next sections .

Summary: The overall goal of this paper is to solve online combinatorial optimization (OCO) problems using reinforcement learning. Importantly, the authors are not seeking to establish new results for unsolved problems, but instead they are motivated by analyzing and comparing the quality of solutions predicted by reinforcement learners with respect to the well-known near-optimal strategies for some OCO tasks. In doing so, the authors focused on an MDP framework using policy gradient and DQN methods. This framework was trained on three OCO tasks; online budget allocation, online knapsack, and the secretary problem. For each, the trained model is consistent with the near-optimal “handcrafted’’ algorithms.