ADAPTIVE GENERATION OF PROGRAMMING PUZZLES

1 INTRODUCTION . Computers still perform worse than humans at elementary programming and mathematical reasoning problems , let alone the AI grand challenge of designing , analyzing , and implementing sophisticated algorithms . For example , many symbolic mathematical software packages fail to solve nn n = 1010 10 for n. While programs can be taught to immediately answer such puzzles , it points to a fundamental lack of understanding that limits their ability to perform complex programming and mathematics . The first question faced in the daunting task of teaching computers to program is , how should one represent programming problems to teach computers ? The ideal representation may be different from what suits humans . Consider a question “ What number follows the sequence 1-2-1-4-1-6-1 ? ” ( Barrett et al. , 2018 ; Saxton et al. , 2019 ) . It requires : ( a ) understanding some English and ( b ) guessing which sequence the test-writer considers most natural among the many consistent ones . We refer to these biases as human priors . Program synthesis problems involve human priors as well . They are usually specified in English and/or by examples ( Gulwani et al. , 2012 ) such as “ Extract the area code from a phone number ” and/or “ ( 212 ) 123-4567 ” → “ 212 ” . These involve prior knowledge about phone numbers , which may impede computers from learning the fundamentals of programming and math.1 In summary , to advance artificial reasoning , the ideal programming problem representation should be : Objective : a candidate solution ought to be unambiguously validated as to whether or not it satisfies the specification . In particular , a problem-solving algorithm should be able to determine correctness without knowledge of English or consulting an answer key . Challenging : Capture problems that human programmers can solve easily but which foil computers . Diverse : Capture a rich range of useful programming problems from easy to hard . Unbiased : Avoid dependence on human priors , such as language or spatio-temporal biases . Programming Puzzles As a better standard for evaluating and advancing artificial reasoning , we propose Programming Puzzles . In a puzzle , given the source code of a function f in a fixed programming language ( e.g . Python ) , the goal is to find an input x such that f ( x ) returns True . 1After creating NAPS ( Zavershynskyi et al. , 2018 ) , a “ Natural Programming Synthesis ” dataset of programming contest problems with English descriptions and input-output examples , one of the authors later conceded that “ ML , specifically natural language understanding , is not there yet. ” ( Polosukhin , 2018 ) As Figure 1 illustrates , short puzzles can capture extremely difficult problems such as factoring or subset-sum as well as easy questions such as list reversal or solving ( x+ 1 ) x+1 == 100100 for x . Importantly , such puzzles are objective – a candidate solution can easily be evaluated for correctness . Not all programming problems can be nicely defined as puzzles . In some problems , writing the puzzle is as hard as solving the problem , e.g . long addition of two numbers . Other problems involve human priors , e.g . alphabetizing a list of names by last names , where language rules determine whether “ Mary De Leon ” is sorted under “ D ” or “ L ” . However , puzzles with polynomial-time definitions constitute the complexity class NP , which contains P problems such as shortest-path as well as hard problems such as factoring and finding a neural network with a sufficiently low training loss . Troublemaker Our second main contribution is a “ Troublemaker ” algorithm that automatically generates puzzles in an adaptive manner to challenge any given puzzle-solving system . A primitive system may be given easy puzzles , while an advanced system may be given hard puzzles ( or puzzles targeting a specific discovered weakness ) . Automatic problem generation has been applied to train program synthesis systems ( Balog et al. , 2016 ; Christakopoulou & Kalai , 2018 ) and in numerous related domains such as mathematical problems ( Saxton et al. , 2019 ) and IQ tests ( Barrett et al. , 2018 ) . However , these problems were generated non-adaptively , often via random sampling . In this work , we use a host of state-of-the-art neural and symbolic machine intelligence techniques as target solvers , and show that Troublemaker learns to generate challenging puzzle datasets for all of them . Troublemaker operates by searching for puzzles in a given grammar that would be hard for a given solver . The search process is made adaptive by guiding it with a trained neural network . The network is optimized to balance ( a ) difficulty of the puzzle w.r.t . the solver runtime , ( b ) diversity of generated puzzles , and ( c ) puzzle brevity . When the solver is trainable , this generation process results in an adversarial setup , where Troublemaker finds increasingly hard problems as the training progresses . Contributions This paper introduces ( 1 ) programming puzzles , a class of objective problems that can be used to evaluate and improve the reasoning ability of existing AI systems ; ( 2 ) an adaptive puzzle generator that can be targeted at any given puzzle solver ( s ) . Note that we do not set out to create novel approaches for puzzle solving – we evaluate our puzzle generation on state-of-the-art neural and symbolic solvers . We focus on adaptive problem generation and establishing the puzzle domain as suitable for teaching machines how to program , leaving the improvements of puzzle solving techniques to future work . 2 RELATED WORK . Program Synthesis The prior work on program synthesis is too large to survey here ( see the survey by Gulwani et al. , 2017 ) . As mentioned , prior work on program synthesis largely works from input-output examples xi , yi where the goal is to find a “ natural ” function f such that f ( xi ) = yi . This representation is subjective – the issue is not that there are multiple correct programs but rather that the correctness of a given solution f is debatable and can not be determined from examples alone . Similarly , the research on semantic parsing ( Liang , 2016 ) describes problems in natural language such as English , for which program correctness is even more debatable . From the point of view of expressing the intent of an end user , these representations may be quite natural and the ambiguity may be inherent to the problem . However , for the purpose of teaching computers programming and reasoning basics , such ambiguity impedes learning . An exception is a closely-related work of Christakopoulou & Kalai ( 2018 ) , which inspired this paper . They consider a specification that is itself a program that takes a whole problem-solving program as input . The specification program then generates a number of random inputs to the candidate program and ensures that the candidate satisfies them all . Similarly to our work , this specification in the form of source code is given to the problem solver . A dual view of puzzles in that their solutions are programs , and the puzzle itself serves as a simple formal specification . Every puzzle solution can be then viewed as a constant program that satisfies this specification . In the programming languages community , the intent is often defined by a formal specification with the goal of finding any program that satisfies it ( Manna & Waldinger , 1980 ; Bornholt & Torlak , 2017 ; Alur et al. , 2018 ) . The greater expressive power in some languages inherently means that it is not always trivial to determine whether a solution satisfies the specification . Evaluating Intelligence and Reasoning Program synthesis systems that employ machine learning models are often trained on automatically generated programming problems ( Balog et al. , 2016 ; Christakopoulou & Kalai , 2018 ; Devlin et al. , 2017 ; Parisotto et al. , 2016 ) . However , the way they generated problems was non-adaptive , i.e . independently of the solver . Recent work ( Shin et al. , 2019 ) has noted that such randomly generated datasets often fail to capture important properties of the desired program distribution , and thus result in a biased program synthesizer . Our Troublemaker framework allows adapting the dataset to the evaluation metric of a particular synthesizer . As mentioned in Section 1 , other tasks for evaluating and motivating progress in machine intelligence have been proposed , including mathematical word problems ( Saxton et al. , 2019 ) and IQ tests ( Barrett et al. , 2018 ) . None of them achieve our desiderata of objectivity and independence of human priors . One notable example is the ARC task ( Chollet , 2019 ) , developed concurrently with this work . While it is also designed to establish a new standard in measuring intelligence and intentionally forgoes all human priors , its problem specification is still given by input-output pairs and thus is subjective . Adversarial and Curriculum Learning The Troublemaker framework aims to generate short and diverse puzzles that are difficult for a given puzzle solver . As such , its main goal is to evaluate the solver and set an adaptive milestone to facilitate AI progress . However , a similar technique can be used to adaptively generate a training set , aiming to improve a given solver after discovering its weaknesses . In the machine learning community this is known as curriculum learning ( Bengio et al. , 2009 ; Graves et al. , 2017 ; Sukhbaatar et al. , 2018 ) . The Alice-Bob framework of Sukhbaatar et al . ( 2018 ) is particularly relevant . In it , a teacher agent and a student agent engage in self-play where the teacher generates tasks for the student right at the edge of the student ’ s “ comfort zone ” , thus encouraging the student to continually improve . This is in contrast to Troublemaker , which over time generates problems just beyond the solver ’ s comfort zone to expose its limitations . When a solver is trainable , Troublemaker proceeds by iterating between improving the solver and the puzzle generator . Troublemaker , Alice-Bob , and similar iterative frameworks drive inspiration from generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . In addition to wide usage in improving generative models via learned adversarial objectives , the GAN ideas have also been recently used to improve robustness of trained agents ( Pinto et al. , 2017 ) or to automatically adapt their reward functions ( Durugkar & Stone , 2018 ) . These works are typically applied to reinforcement learning based environments , while Troublemaker focuses on the problem of generating hard and diverse puzzles without any assumptions on the target solver . 3 PROBLEM GENERATION AS A ZERO-SUM GAME . Our goal is to find a distribution over a diverse set of hard problems . The framework can be defined abstractly in terms of set P of problems and set S of solvers , where each problem p ∈ P computes a Boolean function p : X → { > , ⊥ } for on some set X , and each solver S ∈ S computes a function S : P → X . Problem p is said to be easy for S if p ( S ( p ) ) = > and hard for S otherwise . The set S captures the resource constraints on solvers ( e.g. , our experiments have a time budget B by wrapping each solver in an appropriate timeout ) , and the set P captures restrictions on problems such as length limit and any other restrictions of interest such as solvability . The Problem-generation game is a two-player zero-sum game between a Generator that chooses an arbitrary distribution D ∈ ∆ ( P ) over P , where ∆ ( T ) denotes the set of probability distributions over set T , and a Learning Solver who chooses S ∈ S. For example , we implement S as a set of neural guided solvers whose parameters are trained by the Learning Solver . The payoff to the Generator is v ( D , S ) = Ep∼D [ r ( p , S , D ) ] , where , r is a reward function and D ( p ) is the probability of generating problem p : r ( p , S , D ) = { λ ( − log2D ( p ) ) + ( 1− λ ) if p ( S ( p ) ) 6= > 0 if p ( S ( p ) ) = > . ( 1 ) The reward is parameterized by λ ∈ [ 0 , 1 ] which offers a trade-off : at λ = 0 , any hard problem earns the Generator a reward of 1 , while at λ = 1 the reward is the expected negative log-likelihood over hard problems . The reward function may seem peculiar at first since it depends not only on problem ’ s hardness but also on its probability of being generated . At λ = 0 the reward does not account for the diversity of the distribution of problems—against any solver S a generator could maximize v with the distribution that always outputs a single hard problem p ∈ S , yet we want D to generate diverse hard problems . To account for diversity , the reward therefore depends on D as well . As one varies λ from 0 to 1 , one expects the fraction of hard problems to decrease but the entropy of the hard problems to increase , a measure of greater diversity .

Summary: This paper proposes a method for generating hard puzzles with a trainable puzzle solver. This is an interesting and important problem which sits at the intersection of symbolic and deep learning based AI. The approach is largely GAN inspired, where the neural solver takes the role of a discriminator, and the generator is trained with REINFORCE instead of plain gradient descend.