YaoGAN: Learning Worst-case Competitive Algorithms from Self-generated Inputs

1 INTRODUCTION . It has been a long-standing ambition of machine learning to learn from first principles , that is , a machine learning program that learns without any human-provided data beyond an appropriate definition of the task at hand ( Silver et al. , 2018 ) . The community has made impressive strides towards this goal in recent years , most notably in the domain of tabletop games such as chess , Go , shogi ( Silver et al. , 2018 ) and poker ( Brown & Sandholm , 2018 ) , where programs have been trained to play at superhuman levels without consuming any game-specific human expertise except , naturally , the rules of the game . In this work , we look at the task of using ML to design algorithms for optimization problems . Similar to tabletop games , the task of algorithm design has succinct and well-defined objectives , so learning from first principles is potentially feasible . Moreover , the task is very intriguing as it typically requires significant amount of time and expertise for humans to accomplish . In our context , the problem of learning algorithms from first principles is also related to an important issue in both ML and algorithms : what input instances is the learned algorithm expected to handle ? The typical ML approach is to train the program against a pre-defined training set , with the assumption that the training set is carefully chosen to capture future input distributions well , so the learned algorithm generalizes to future input of interest . On the other hand , the algorithm design community in theoretical computer science ( TCS ) traditionally considers algorithms with worst-case guarantees due to their robustness against arbitrary input . We focus on the latter domain in our work . Along the line of learning worst-case algorithms , a particularly relevant result is by Kong et al . ( 2018 ) that uses reinforcement learning to learn theoretically optimal algorithms for online optimization problems , and one of the key ideas they introduce is the notion of universal and highentropy training sets . Roughly speaking , such a training set serves as a very effective benchmark in the sense that the task of finding an algorithm with strong worst-case guarantee reduces to finding an algorithm that performs well against this training set . Not surprisingly , for a given optimization problem , coming up with such a training set can require significant domain expertise a priori . In the current work the goal is to go beyond the framework of universal training sets , and enable learning of algorithms without even this prior domain knowledge . To overcome this issue , we draw insights from the fields of game theory , analysis of algorithms , and online learning , as well as generative adversarial networks ( GANs , Goodfellow et al . ( 2014 ) ) . In particular , we introduce a training framework that aims to learn a strong worst-case algorithm and an effective benchmark training set simultaneously , all without relying on any human domain expertise beyond specifying how to compute the objective of the optimization problem . If successful , such an approach would have impactful ramifications : researchers could accelerate the speed of discovery by offloading to machines the time-consuming process of finding problem-specific insights , which can be crucial in improving the state of the art for many problems both in theory and in practice . At the high level , our framework has two components : the algorithm agent and the adversary . Their goals are to learn the optimal algorithm and a set of input instances that captures the essential difficulty of the given optimization problem , respectively . For simple optimization problems , rudimentary strategies such as random search or grid search can provide a fairly effective adversary . For problems with more interesting underlying space , we exploit deep learning to guide our search for the distribution of difficult inputs . Similar to a generative adversarial network ( GAN ) , we use two neural networks , one for the algorithm agent and another for the adversary . We start from scratch , train these two networks against each other , and let them evolve simultaneously . We call our program YaoGAN due to this structural similarity to GAN and the connection to the classic Yao ’ s Lemma ( see Section 2.1 for details ) ; this is our main technical contribution . In this initial effort , we follow Kong et al . ( 2018 ) in focusing on online optimization problems ( Buchbinder et al. , 2009 ) where each input is a sequence arriving over time , and the algorithm needs to make an immediate and irrevocable decision at each time step without knowing the future input . We further restrict our attention to problems that can be solved or approximated by algorithms requiring a small amount of memory . There are abundant well-studied and interesting problems bearing such properties in online optimization , typically when the structure of the problem exhibits certain Markov property so it is sufficient for the algorithm to maintain a succinct state capturing past information . We consider two online optimization problems , the ski rental problem and the fractional AdWords problem . Ski rental ( Karlin et al. , 1986 ) has been a staple introductory example in online analysis . The second problem , fractional AdWords ( Mehta et al. , 2007 ) , is ( a fractional relaxation of ) a wellknown problem motivated by online advertising . Moreover , the Adwords problem is closely related to numerous variants of the online bipartite matching problem , which is still an actively studied area in algorithm design , for example in the setting of ride-sharing platforms ( Dickerson et al. , 2017 ; Ashlagi et al. , 2019 ) . For these problems , we have a good understanding of the structures of the optimal algorithm and difficult inputs , so we can verify that our paradigm leads to something meaningful . Our experiments , although fairly limited and preliminary , demonstrate that our framework can be effective at finding algorithms as well as difficult input instances that are consistent with optimal results . We believe further study along this direction can lead to more exciting developments . We use generic neural network architecture and ML methods ( e.g. , Adam optimizer ( Kingma & Ba , 2014 ) ) in our experiments , and do not optimize the training set-up or hyperparameters , since our focus in this paper is not in extending ML techniques , but in applying our high-level paradigm to learn algorithms from first principles . We believe the co-evolution of new ML techniques tailored to this task will be essential to the ultimate success of this approach . We leave that aspect , as well as applying the framework to more open algorithm design problems , to future work . Related work . The most relevant paper to ours is an approach by Kong et al . ( 2018 ) that uses reinforcement learning ( RL ) to find competitive algorithms for online optimization problems including the AdWords problem , online knapsack and the secretary problem . They introduce the notions of universal and high-entropy training set that are tailor-made to facilitate the convergence to worstcase competitive algorithm . Deudon et al . ( 2018 ) and Kool et al . ( 2018 ) use RL to solve the traveling salesman problem ( TSP ) in the two-dimensional plane . Both uniformly sample the input points from the unit square and teach the network to optimize the expected performance on this input distribution without having access to exact solutions . Nevertheless , the learned algorithms from both approaches seem competitive with state-of-the-art methods , at least for modest input sizes . Vinyals et al . ( 2015 ) introduce the Pointer Network , a sequence-to-sequence model that successfully manages to approximately find convex hulls , Delaunay triangulations and near-optimal planar TSPs . They assume access to an exact solution and use supervised learning to train the model , while their input is uniformly sampled from the unit square . Graves et al . ( 2014 ) introduce the Neural Turing machines , a differentiable and computationally universal Turing machine model , and use supervised learning to teach them simple algorithms such as copying or sorting . Kaiser & Sutskever ( 2015 ) introduce the parallel and computationally universal Neural GPU and use supervised learning to teach it addition and multiplication that seemingly generalizes to arbitrary lengths . We point out some other related results without going into details : the work of Bello et al . ( 2016 ) studies applying RL on combinatorial problems including TSP and Knapsack ; the work of Dai et al . ( 2017 ) exploits embedding and RL to design heuristic algorithms for classic graph problems on specific distributions ; Boutilier & Lu ( 2016 ) uses RL to solve Budget Allocation problem ; Duetting et al . ( 2019 ) applies deep learning techniques to designing incentive compatible auction mechanisms . Readers insterested in ski rental and AdWords can consult Buchbinder & Naor ( 2009 ) and Mehta et al . ( 2013 ) for a more thorough coverage . To the best of our knowledge , previous results mostly focus on the algorithm side , and either learn algorithms designed to work on specific input distributions of interest , or utilize supervised learning to mimic a known method , or ( in the case of Kong et al . ( 2018 ) ) rely on domain expertise to provide hard input instances to train algorithms that are competitive in worst-case . We do not wish to detract from the value of optimizing an algorithm on a specific distribution if it is of special interest . Indeed , a practitioner optimizing for a distribution D could likely design algorithms that outperform classic solutions onD since classic solutions typically protect against worst-case inputs and might be overly pessimistic onD . Our goal is different : facilitate the discovery of new algorithms with strong worstcase performance on open problems where we do not have a known difficult input distribution at hand . Such learned strategies could then potentially be later fine-tuned to adapt to specific input distributions , while retaining some worst case guarantees . 1.1 SUMMARY OF RESULTS . As a brief introduction , we can summarize YaoGANs as essentially starting with a “ random ” algorithm network , then alternating between ( 1 ) finding the best response ( that will force the algorithm to exhibit its worst behavior ) , and ( 2 ) training the algorithm network on this best response . Details are given in Section 3 . We apply the YaoGAN method to two classic problems in the area of online optimization : the ski rental problem and a fractional version of the AdWords problem ( these are defined formally in Section 2 ) . The two problems exhibit a progression in terms of how hard it is to find the “ best responses. ” Finding best response inputs for ski rental is relatively easy—an approximately exhaustive search ( over a suitably parameterized space ) is sufficient—while for AdWords it seems to be a hard non-convex global optimization problem . Ski rental . A few minutes of training leads to an algorithm whose competitive ratio—the primary figure of quality for an online optimization algorithm—converges approximately to 1.59 , which is within 10−2 of the theoretically best value achievable . Furthermore , the behavior of trained algorithm exactly matches the one of Karlin et al . ( 1986 ) ( Appendix C ) . Fractional AdWords . For the fractional AdWords problem—a “ budgeted bipartite matching problem ” —with 3 advertisers with a budget of 3 each and 9 ad slots1 , about 10 hours of training is sufficient to achieve a competitive ratio of 0.73 , which is within 10−2 of the theoretically best value achievable for this problem . The trained algorithm qualitatively recovers both the greedy strategy and the balance strategy ( Kalyanasundaram & Pruhs , 2000 ) , two defining aspects of the optimal solution , as well as seems to find the correct form of tradeoff between the strategies in tested scenarios ( details in Section 4 ) . 1Note that while the problem solved here is defined on “ fixed size ” instances ( 27 real numbers ) , this is already well beyond the regime of executing a simple exhaustive search to discover the optimal algorithm .

Summary: This paper introduces a new approach to solve optimization problems without relying on any human-provided data beyond the specification of the optimization problem itself. The approach is inspired by the two-player zero-sum game paradigm and follow a generative adversarial network (GAN) setting. One network is trained to output the optimal behavior for a given problem, while the other is trained to output difficult instances of the given problem. These two networks are trained simultaneously and compete against the other until some equilibrium is achieved.