A new dog learns old tricks:  RL finds classic optimization algorithms

1 INTRODUCTION . Machine learning has led to dramatic improvements in our capabilities to solve problems previously considered intractable . Besides the obvious empirical evidence of success , there has also been a strong parallel effort in the theory of ML which aims to explain why , when , and how ML techniques work . Our goal in this paper is to explore whether machine learning can be used to learn algorithms for classic combinatorial optimization problems . We will define this question more specifically by connecting to three concepts from traditional algorithms and complexity theory . 1.1 INPUT LENGTH INDEPENDENCE AND THE CONNECTION TO RL . Firstly , by “ algorithm , ” we mean a uniform algorithm , one that works for inputs of all lengths , not just for specific input lengths from which the training data is drawn . Typically , models learned using ML techniques tend to be non-uniform , i.e. , depend on input length . Previous approaches to finding uniform models — the Neural Turing machines of Graves et al . ( 2014 ) and generally the use of recurrent models ( including LSTMs ) — all suffer from some drawback , most notably the difficulty of training by back-propagation and gradient descent over long sequences . A particularly clever approach , due to Kaiser and Sutskever ( 2015 ) , adopts the idea of learning “ convolution masks ” of finite size that , when repeatedly applied solve a problem of interest on inputs of arbitrary length ; however , the resulting learning problems appear intractable ( since the volume of computation grows at least cubically in their setup for most interesting problems , and stable ways to learn convolutions over such large grids are not well-understood ) . We expand on this point in Appendix F. Our first key insight is that for numerous combinatorial optimization problems , the primal-dual framework offers efficient solutions , and also lends itself to efficient online algorithms ( see , e.g. , Buchbinder et al . ( 2009 ) ) where the input arrives in small units , one at a time , and the algorithm makes a choice about the input ( e.g. , which advertiser to give a query impression to , which node to match in a graph , whether to include an item in the knapsack , etc. ) . In addition , there is usually a clear notion of reward associated with a set of decisions , and the goal is often to optimize the overall rewards collected . This naturally connects our goal to the field of reinforcement learning , and indeed we formulate our learning problems in the Markov Decision Process ( MDP ) framework and use tools from deep reinforcement learning using policy gradient and DQN methods . Specifically , for any optimization problem , the MDP state will consist of three parts : global parameters of the problem instance ( e.g. , Knapsack size ) , a data structure that we expect ( and train ) the algorithm to learn to maintain ( and whose size can depend on the global parameters ) , and the current input unit . We will train two agents — U that computes an update to the data structure , and D that makes the decision for each input using the data structure and the current input . The RL environment will then carry out the task of applying the update to the data structure , and present it back to the agents U and D as part of the state for the next input . We establish theoretically ( Appendix G ) that this simple framework is flexible to capture a wide class of problems , and empirically that the resulting algorithms are quite powerful . 1.2 ADVERSARIALLY CHOSEN INPUT DISTRIBUTIONS . An important question in both ML and Algorithms is what input instances is the algorithm expected to work on . The ML approach is to use a rich enough training set to capture future inputs distributions . Theoretical computer science ( TCS ) , by contrast , traditionally considers worst-case analysis : an algorithm is judged by its performance on the worst possible input ( specially crafted by an Adversary to beat the algorithm ) . This approach leads to theoretically robust guarantees on the algorithm . In stochastic models of input , the Adversary is somewhat restricted in order to better capture “ real ” inputs — including the Random Order and the IID models of input . Our second key insight is to bring this approach of adversarial input sets ( not to be confused with the notion of adversarial examples which fool ML models , for example , see Goodfellow et al . ( 2014 ) ) to the ML domain via two techniques to craft training sets : 1 . Universal Training Set A common way to prove lower bounds in the TCS literature is to come up with a distribution over inputs and show that no algorithm can perform better than some factor α ≤ 1 compared to the optimal solution , in expectation . This is a key ingredient in the technique of using Yao ’ s Lemma Yao ( 1977 ) to prove a lower bound on the performance of all randomized algorithms . For example , in the Adwords problem , there is a specific input distribution which is hard for all online algorithms ( Karp et al . ( 1990 ) ; Mehta et al . ( 2007 ) ) . Intuitively , one might expect that if an algorithm does perform well on the specified input distribution then it must have learned some characteristics of the optimal algorithm . We bring this idea to the ML literature by proposing to incorporate such instances into the training . 2 . High-Entropy Training Set In some cases , it may difficult to find a universal training set or the universal training set may admit algorithms which perform well on the training set while performing poorly on all other instances . To alleviate this problem we also propose to incorporate training sets that have high entropy . For example , in the Adwords problem , a randomized greedy algorithm is able to perform quite well on the adversarial instance so we incorporate a distribution which is explicitly bad for greedy . In the secretary problem , we provide inputs which come from many different distributions so that it is difficult for it to learn utilize any characteristics of the distributions . 1.3 DECODING THE NETWORK TO FIND THE ALGORITHM . Our third contribution is the following intriguing question , connected to the broad area of how to interpret ML models . Specifically , suppose that for a given problem , we do manage to learn a network of constant ( fixed ) size , which does well over inputs of varying lengths coming from varying distributions . Does this allow us to confidently say that the network has learned the correct algorithm ? One observation is that since the network is concise , it has to represent a succinct logic . How does that compare to the optimal pen-and-paper algorithm that computer scientists have developed for the problem ? We will answer such questions by plotting the input-output characteristics of the network learned for the different problems we consider , and compare them to the expected behavior of the traditional algorithms . It may even be possible to convert the network to an algorithmic form , but we leave such an attempt for future work . 1.4 SUMMARY OF RESULTS . We study three optimization problems in this work – the Adwords Problem ( aka Online Budgeted Allocation ) , the Online ( 0-1 ) Knapsack Problem , and the so called Secretary Problem1 . All three problems share the feature that they are all very well-studied online combinatorial problems with some probabilistic features , and importantly , the optimal algorithms for each of them have a concise algorithmic form ( e.g. , not represented implicitly as the solution of a dynamic program ) . For all three problems we use RL to find a “ uniform algorithm ” , i.e. , an input-length independent logic . We train the models using universal or high-entropy input distributions and find that the models discover the classic algorithms . To mention the highlights of each section : • Adwords problem : The model learns to find the Balance strategy ( Kalyanasundaram and Pruhs , 2000 ) for unweighted graphs , and the MSVV strategy ( Mehta et al. , 2007 ) for weighted graphs which optimally trades-off between load-balancing and greedy strategies . • Online Knapsack problem : The model learns to find an optimal threshold on value per unit size to use to either accept or reject incoming items . • Secretary Problem : The model learns the optimal “ Wait-then-Pick ” algorithm which samples the first 1/e fraction of the input stream and then picks the next item which is higher than any seen before . It also finds the optimal time-dependent value-threshold algorithm for i.i.d . input . Our results suggest that it might be possible to draw a formal connection between the online primaldual framework and RL , e.g. , to prove that the online optimization problems solvable in the primaldual framework admit efficient algorithms learnable via RL . We leave this as a fascinating open question for future work . Remark . In this paper , we use the standard REINFORCE algorithm for policy gradient , with the Adam optimizer . Our contribution is not in extending RL techniques , but in making the connection to algorithms , and showing how standard RL techniques can in fact find the classic “ pen-and-paper ” algorithms . Further , we do not optimize for the training set-up or hyperparameters ; in particular all our training is done over a single machine and training often takes less than a day or two . 1.5 RELATED WORK . We are taking a specific angle at the question of how machine learning solves optimization problems . There is a lot of previous work on the larger question of ML and optimization . A related previous work is that of Bello et al . ( 2016 ) which also studies combinatorial problems , particularly the Traveling Salesman Problem and Knapsack , and also uses policy gradient method for an RL framework to optimize the parameters of a pointer network . ( This paper also summarizes previous literature on combinatorial optimization using neural networks . ) Our work differs in a few ways , but specifically the goal is not only to solve the problem , but also to interpret the learned RL policy network and compare to the known optimal algorithms , both in performance and in structure . Moreover , the work of Bello et al . ( 2016 ) learns a recurrent network , which could become prohibitively expensive to train on data sets that are large enough to capture the complexity of TSP or Knapsack . Another closely related paper is ( Dai et al. , 2017 ) , which uses embeddings and RL to find heuristics to solve classic graph problems on specific distributions . The problems they consider are offline in nature , and the heuristics conform to an incremental ( greedy ) policy guided by scores generated by the RL agent . Specifically , their goal is to find new heuristics for specific distributions , which is different from the work here , where we ask if RL can discover the classic “ worst-case ” algorithms . Our work is also different in the same way from other work in the space of combinatorial problems , such as that on TSP and Vehicle routing ( Kool and Welling , 2018 ) , as well as to the growing literature on using RL to solve optimization for control problems ( see for e.g . Lillicrap et al . ( 2015 ) , Levine et al . ( 2016 ) ) . We also mention as a loosely related paper by Boutilier and Lu ( 2016 ) , which uses RL ( as a budgeted MDP ) the solve Budget Allocation problem , although that problem is different from the Adwords problem we consider , in that the question there is to optimally allocate a single advertiser ’ s budget . 1This is the somewhat unfortunate prevailing name for this problem of Optimal Stopping — we continue using this name in this paper to avoid technical confusion . One of the goals in this work is to find a uniform algorithm , i.e. , one which is independent of the input length , for which we use the RL approach and focus on online optimization problems . As mentioned earlier , there have been several other nice approaches for this problem , each with different difficulty level in the goals , and obstacles in learning . This includes the Neural Turing machines of Graves et al . ( 2014 ) and generally the use of recurrent models ( including LSTMs ) , and the “ convolution masks ” approach of Kaiser and Sutskever ( 2015 ) . Finally , from the algorithms literature , our three problems are very well-studied , especially with Knapsack ( e.g. , Dantzig ( 1957 ) ) and Secretary ( Dynkin , 1963 ) being decades old problems . The relatively recent Adwords problem ( Mehta et al. , 2007 ) is strongly motivated by online advertising ( see , e.g. , Mehta et al . ( 2013 ) ) , and for which solutions that merge both the theoretical approach and the ML approach could potentially be of high impact in practice of budgeted ad allocations . Algorithmic work on these problems is cited throughout the next sections .

Summary: This paper introduces a new framework to solve online combinatorial problems using reinforcement learning. The idea is to encode the current input, the global parameters, and a succinct data structure (to represent current states of the online problem) as MDP states. Such a problem can then be solved by deep RL methods. To train such models, the authors use a mixture of input distributions. Some come from hard distributions which are used to prove lower bounds in the TCS community, and the others are carefully constructed distributions to fool a specific set of algorithms. The authors made an important point that their algorithms are uniform in the TCS sense, i.e., the algorithm does not depend on the input length.