ADAPTIVE GENERATION OF PROGRAMMING PUZZLES

1 INTRODUCTION . Computers still perform worse than humans at elementary programming and mathematical reasoning problems , let alone the AI grand challenge of designing , analyzing , and implementing sophisticated algorithms . For example , many symbolic mathematical software packages fail to solve nn n = 1010 10 for n. While programs can be taught to immediately answer such puzzles , it points to a fundamental lack of understanding that limits their ability to perform complex programming and mathematics . The first question faced in the daunting task of teaching computers to program is , how should one represent programming problems to teach computers ? The ideal representation may be different from what suits humans . Consider a question “ What number follows the sequence 1-2-1-4-1-6-1 ? ” ( Barrett et al. , 2018 ; Saxton et al. , 2019 ) . It requires : ( a ) understanding some English and ( b ) guessing which sequence the test-writer considers most natural among the many consistent ones . We refer to these biases as human priors . Program synthesis problems involve human priors as well . They are usually specified in English and/or by examples ( Gulwani et al. , 2012 ) such as “ Extract the area code from a phone number ” and/or “ ( 212 ) 123-4567 ” → “ 212 ” . These involve prior knowledge about phone numbers , which may impede computers from learning the fundamentals of programming and math.1 In summary , to advance artificial reasoning , the ideal programming problem representation should be : Objective : a candidate solution ought to be unambiguously validated as to whether or not it satisfies the specification . In particular , a problem-solving algorithm should be able to determine correctness without knowledge of English or consulting an answer key . Challenging : Capture problems that human programmers can solve easily but which foil computers . Diverse : Capture a rich range of useful programming problems from easy to hard . Unbiased : Avoid dependence on human priors , such as language or spatio-temporal biases . Programming Puzzles As a better standard for evaluating and advancing artificial reasoning , we propose Programming Puzzles . In a puzzle , given the source code of a function f in a fixed programming language ( e.g . Python ) , the goal is to find an input x such that f ( x ) returns True . 1After creating NAPS ( Zavershynskyi et al. , 2018 ) , a “ Natural Programming Synthesis ” dataset of programming contest problems with English descriptions and input-output examples , one of the authors later conceded that “ ML , specifically natural language understanding , is not there yet. ” ( Polosukhin , 2018 ) As Figure 1 illustrates , short puzzles can capture extremely difficult problems such as factoring or subset-sum as well as easy questions such as list reversal or solving ( x+ 1 ) x+1 == 100100 for x . Importantly , such puzzles are objective – a candidate solution can easily be evaluated for correctness . Not all programming problems can be nicely defined as puzzles . In some problems , writing the puzzle is as hard as solving the problem , e.g . long addition of two numbers . Other problems involve human priors , e.g . alphabetizing a list of names by last names , where language rules determine whether “ Mary De Leon ” is sorted under “ D ” or “ L ” . However , puzzles with polynomial-time definitions constitute the complexity class NP , which contains P problems such as shortest-path as well as hard problems such as factoring and finding a neural network with a sufficiently low training loss . Troublemaker Our second main contribution is a “ Troublemaker ” algorithm that automatically generates puzzles in an adaptive manner to challenge any given puzzle-solving system . A primitive system may be given easy puzzles , while an advanced system may be given hard puzzles ( or puzzles targeting a specific discovered weakness ) . Automatic problem generation has been applied to train program synthesis systems ( Balog et al. , 2016 ; Christakopoulou & Kalai , 2018 ) and in numerous related domains such as mathematical problems ( Saxton et al. , 2019 ) and IQ tests ( Barrett et al. , 2018 ) . However , these problems were generated non-adaptively , often via random sampling . In this work , we use a host of state-of-the-art neural and symbolic machine intelligence techniques as target solvers , and show that Troublemaker learns to generate challenging puzzle datasets for all of them . Troublemaker operates by searching for puzzles in a given grammar that would be hard for a given solver . The search process is made adaptive by guiding it with a trained neural network . The network is optimized to balance ( a ) difficulty of the puzzle w.r.t . the solver runtime , ( b ) diversity of generated puzzles , and ( c ) puzzle brevity . When the solver is trainable , this generation process results in an adversarial setup , where Troublemaker finds increasingly hard problems as the training progresses . Contributions This paper introduces ( 1 ) programming puzzles , a class of objective problems that can be used to evaluate and improve the reasoning ability of existing AI systems ; ( 2 ) an adaptive puzzle generator that can be targeted at any given puzzle solver ( s ) . Note that we do not set out to create novel approaches for puzzle solving – we evaluate our puzzle generation on state-of-the-art neural and symbolic solvers . We focus on adaptive problem generation and establishing the puzzle domain as suitable for teaching machines how to program , leaving the improvements of puzzle solving techniques to future work . 2 RELATED WORK . Program Synthesis The prior work on program synthesis is too large to survey here ( see the survey by Gulwani et al. , 2017 ) . As mentioned , prior work on program synthesis largely works from input-output examples xi , yi where the goal is to find a “ natural ” function f such that f ( xi ) = yi . This representation is subjective – the issue is not that there are multiple correct programs but rather that the correctness of a given solution f is debatable and can not be determined from examples alone . Similarly , the research on semantic parsing ( Liang , 2016 ) describes problems in natural language such as English , for which program correctness is even more debatable . From the point of view of expressing the intent of an end user , these representations may be quite natural and the ambiguity may be inherent to the problem . However , for the purpose of teaching computers programming and reasoning basics , such ambiguity impedes learning . An exception is a closely-related work of Christakopoulou & Kalai ( 2018 ) , which inspired this paper . They consider a specification that is itself a program that takes a whole problem-solving program as input . The specification program then generates a number of random inputs to the candidate program and ensures that the candidate satisfies them all . Similarly to our work , this specification in the form of source code is given to the problem solver . A dual view of puzzles in that their solutions are programs , and the puzzle itself serves as a simple formal specification . Every puzzle solution can be then viewed as a constant program that satisfies this specification . In the programming languages community , the intent is often defined by a formal specification with the goal of finding any program that satisfies it ( Manna & Waldinger , 1980 ; Bornholt & Torlak , 2017 ; Alur et al. , 2018 ) . The greater expressive power in some languages inherently means that it is not always trivial to determine whether a solution satisfies the specification . Evaluating Intelligence and Reasoning Program synthesis systems that employ machine learning models are often trained on automatically generated programming problems ( Balog et al. , 2016 ; Christakopoulou & Kalai , 2018 ; Devlin et al. , 2017 ; Parisotto et al. , 2016 ) . However , the way they generated problems was non-adaptive , i.e . independently of the solver . Recent work ( Shin et al. , 2019 ) has noted that such randomly generated datasets often fail to capture important properties of the desired program distribution , and thus result in a biased program synthesizer . Our Troublemaker framework allows adapting the dataset to the evaluation metric of a particular synthesizer . As mentioned in Section 1 , other tasks for evaluating and motivating progress in machine intelligence have been proposed , including mathematical word problems ( Saxton et al. , 2019 ) and IQ tests ( Barrett et al. , 2018 ) . None of them achieve our desiderata of objectivity and independence of human priors . One notable example is the ARC task ( Chollet , 2019 ) , developed concurrently with this work . While it is also designed to establish a new standard in measuring intelligence and intentionally forgoes all human priors , its problem specification is still given by input-output pairs and thus is subjective . Adversarial and Curriculum Learning The Troublemaker framework aims to generate short and diverse puzzles that are difficult for a given puzzle solver . As such , its main goal is to evaluate the solver and set an adaptive milestone to facilitate AI progress . However , a similar technique can be used to adaptively generate a training set , aiming to improve a given solver after discovering its weaknesses . In the machine learning community this is known as curriculum learning ( Bengio et al. , 2009 ; Graves et al. , 2017 ; Sukhbaatar et al. , 2018 ) . The Alice-Bob framework of Sukhbaatar et al . ( 2018 ) is particularly relevant . In it , a teacher agent and a student agent engage in self-play where the teacher generates tasks for the student right at the edge of the student ’ s “ comfort zone ” , thus encouraging the student to continually improve . This is in contrast to Troublemaker , which over time generates problems just beyond the solver ’ s comfort zone to expose its limitations . When a solver is trainable , Troublemaker proceeds by iterating between improving the solver and the puzzle generator . Troublemaker , Alice-Bob , and similar iterative frameworks drive inspiration from generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . In addition to wide usage in improving generative models via learned adversarial objectives , the GAN ideas have also been recently used to improve robustness of trained agents ( Pinto et al. , 2017 ) or to automatically adapt their reward functions ( Durugkar & Stone , 2018 ) . These works are typically applied to reinforcement learning based environments , while Troublemaker focuses on the problem of generating hard and diverse puzzles without any assumptions on the target solver . 3 PROBLEM GENERATION AS A ZERO-SUM GAME . Our goal is to find a distribution over a diverse set of hard problems . The framework can be defined abstractly in terms of set P of problems and set S of solvers , where each problem p ∈ P computes a Boolean function p : X → { > , ⊥ } for on some set X , and each solver S ∈ S computes a function S : P → X . Problem p is said to be easy for S if p ( S ( p ) ) = > and hard for S otherwise . The set S captures the resource constraints on solvers ( e.g. , our experiments have a time budget B by wrapping each solver in an appropriate timeout ) , and the set P captures restrictions on problems such as length limit and any other restrictions of interest such as solvability . The Problem-generation game is a two-player zero-sum game between a Generator that chooses an arbitrary distribution D ∈ ∆ ( P ) over P , where ∆ ( T ) denotes the set of probability distributions over set T , and a Learning Solver who chooses S ∈ S. For example , we implement S as a set of neural guided solvers whose parameters are trained by the Learning Solver . The payoff to the Generator is v ( D , S ) = Ep∼D [ r ( p , S , D ) ] , where , r is a reward function and D ( p ) is the probability of generating problem p : r ( p , S , D ) = { λ ( − log2D ( p ) ) + ( 1− λ ) if p ( S ( p ) ) 6= > 0 if p ( S ( p ) ) = > . ( 1 ) The reward is parameterized by λ ∈ [ 0 , 1 ] which offers a trade-off : at λ = 0 , any hard problem earns the Generator a reward of 1 , while at λ = 1 the reward is the expected negative log-likelihood over hard problems . The reward function may seem peculiar at first since it depends not only on problem ’ s hardness but also on its probability of being generated . At λ = 0 the reward does not account for the diversity of the distribution of problems—against any solver S a generator could maximize v with the distribution that always outputs a single hard problem p ∈ S , yet we want D to generate diverse hard problems . To account for diversity , the reward therefore depends on D as well . As one varies λ from 0 to 1 , one expects the fraction of hard problems to decrease but the entropy of the hard problems to increase , a measure of greater diversity .

Summary: This paper proposes a trainable 'puzzle' program synthesizer that outputs a program f with a specific syntax. These 'puzzles' are structured as boolean programs, and a program solver solves the puzzle by finding an input x such that f(x) = True.  The authors motivate this task by making a case that puzzles of this sort are a good domain for teaching computers how to program.