Neurally Guided Genetic Programming for Turing Complete Programming by Example

1 INTRODUCTION . The ability to synthesise source code from examples , in which a source code implementation of a function is created based on one or more demonstrations of input-output mapping , is a fundamental question in machine learning . We specifically study this question in the form of scenarios where large corpora of existing code are not available ( e.g. , human-written programs in open-source repositories ) . The immediate applications of this would allow non-programmers to generate programs , or experts to abstract away trivial coding tasks . In addition , from a machine learning perspective , it allows complex functions to be generated in a symbolic and human-readable form – which can be subjected to a wide range of static analysis tools to model generality or correctness . To date this challenge has been studied using neural-network-driven synthesis , genetic programming , and SMT solvers . However , at present these approaches are significantly constrained in the complexity of the target language in which code is synthesised . Neural synthesis for example , such as the DeepCoder architecture ( Balog et al. , 2017 ; Zohar & Wolf , 2018 ) , shows success on simple domain-specific languages , but the search space of more realistic Turing-complete languages is vast by comparison and is unlikely to be representable in a neural network on current or near-future hardware . Genetic programming , meanwhile , is limited by our ability to specify a fitness function which can successfully navigate to a solution for a particular problem – in the highly irregular and often flat fitness landscape of program space ( Kinnear , 1994 ; Renzullo et al. , 2018 ) . SMT solvers by comparison lack the analytical power to handle loops without human guidance , constraining their applicability ( Srivastava et al. , 2010a ; So & Oh , 2018 ; Srivastava et al. , 2010b ) . In this paper we examine code synthesis from examples for a Turing-complete language which can be cross-compiled into C/Java . We use just 10 input/output examples to describe each problem for which we need to synthesise a matching function ( i.e. , providing unsorted and sorted arrays of integers to describe sorting ) ; because our target language yields a total search space size of 5 ∗ 10119 possible program permutations , scalability of the synthesis technique is crucial . In this context , we use a novel combination of genetic programming ( GP ) and neural networks ( NNs ) ; GP is used to navigate within program space from a given starting point using a general-purpose fitness function , while NN methods are used to provide prediction of high-level features which help guide the GP to higher-probability success areas in which to search . In essence this technique allows the NN to model only highly abstract features of program space , allowing it to scale to vast program search spaces , while the GP can then incrementally traverse program space from an NN-derived starting point to a correct solution . We bootstrap this process by using an unstructured form of curriculum learning , in which successfully-found functions are used as seed programs to generate synthetic corpora on which to train new neural networks , leading to further high-utility source code feature predictions for new problems . Initially this curriculum learning is based on programs that can be successfully found using GP alone with our generic fitness function , which then allows us to synthesise more complex programs using NN inference . Our key results demonstrate that GP is able to solve simple synthesis problems unaided , and that synthetic corpora generated from these problems allow popular neural network architectures to identify high-utility search hints on more complex problems to guide the genetic programming search . In one of our problem sets , this has the effect of allowing the framework to successfully synthesise 7 of the 10 programs that had never been found by GP alone , and among other programs moves success rates from 38 % to 55 % . All of our source code is provided online ( pending de-anonymisation ) . 2 RELATED WORK . Code synthesis from I/O examples has been studied using three major approaches : deductive solvers ; neural networks ( NNs ) with search ; and genetic programming ( GP ) . For code synthesis in a Turingcomplete language , deductive solvers have yet been shown to operate well with loop-based flow control operators ( although frameworks which manually define any non-linear program flow can yield good performance ( So & Oh , 2018 ) ) . Neural synthesis , by comparison , is limited by how much of program space for a general-purpose target language can be captured in a NN model , while GP is limited by the difficulty of deriving a fitness function to navigate to a solution ( Renzullo et al. , 2018 ) . In the remainder of this section we focus on NN and GP approaches in more detail . Neural synthesis Neural synthesis works by training a neural network on a sub-sample of the entirety of program space for the target language ( e.g. , sampling at a uniform interval or at random ) . When presented with a new problem as an I/O example , the neural network will then be asked to predict which lines of code ( or particular operators ) are likely to be present in the solution based on similar I/O transforms observed in the training set from the above sub-sample . The system will then perform an exhaustive search of program space to fill in the remaining ( non-predicted ) features . Notable examples here include DeepCoder and RobustFill , among others ( e.g. , Balog et al . ( 2017 ) ; Zohar & Wolf ( 2018 ) ; Devlin et al . ( 2017 ) ; Chen et al . ( 2019 ) ; Singh & Kohli ( 2017 ) ) The key limitation to this approach is that it must be able to train on a detailed enough sub-sample of program space , and store this sample inside a neural network , to make meaningful predictions on program features for unseen problems . While this works for highly simplified languages ( DeepCoder , for example , has no loop operators ( Balog et al. , 2017 ) ) , the search space size of a Turingcomplete language is astronomical by comparison . If we consider that the capacity of a feed-forward ReLu neural network to differentiate between classes ( in our case programs ) , termed its VapnikChervonenkis ( VC ) dimension , at best grows as a linear function of w ∗ L ∗ log ( w ) where w is the total number of weights and L the number of layers ( Bartlett et al. , 2019 ) , it is unlikely that a neural network on current hardware would be able to represent a useful sub-sample of possible program permutations yielded by the search space of a general-purpose language . Genetic programming GP relies on iterative travel through program space from a starting point ( often an empty program ) to the solution , guided by a fitness function ( Vanneschi & Poli , 2012 ; Taleby Ahvanooey et al. , 2019 ) . The field has a long history ( Forsyth , 1981 ) but still shows results ( Miranda et al. , 2019 ) that are competitive with neural networks ( Ain et al. , 2020 ) , and an ability to tackle complex problems mixing diverse datatypes ( Pantridge & Spector , 2020 ) . Unlike neural synthesis , a GP approach does not need to encode the entirety of program space in a model , and so can in theory work in a scalable fashion on high dimensional search spaces as long as a fitness function is provided which can guide the search incrementally towards a solution . The key problem with GP for code synthesis is that large areas of program space are difficult to navigate , exhibiting large plateaus of neutrality ( no behavioural change despite significant code change ) and highly irregular responses to code change ( jagged fitness landscape ) ( Renzullo et al. , 2018 ; Kinnear , 1994 ) . A fitness function may also fail to capture higher level properties , for example “ all outputs are even ” or “ all values from the input are repeated in the output ” , which may otherwise be identified by a neural network . Combining neural network prediction with genetic programming Given the limitations of both NNs and GP in themselves , we hypothesise that the combination of the two techniques may provide the best features of both while mitigating their respective limitations in the context of code synthesis in a Turing-complete language with a very large program search space . While a NN can not feasibly model all of program space , and so can not be expected to predict each line to be synthesised , it does have the potential to predicting a small number of higher-level features which only require a very limited internal model of program space . This can be combined with a GP search process which fixes these lines in place to constrain the search area , and can use this partially-constructed program in combination with one or more generic fitness functions to guide the search to a successful result . We further find that we can use successfully-found programs to generate synthetic training sets for a NN focused on that area of program space , which then leads to further high-value predictions of likely features for more complex programs . 3 METHODOLOGY . Our system accepts up to ten I/O examples from a user and synthesises the source code of a function which converts those inputs to their corresponding outputs . We use 10 examples as a number which a user may be willing to input while representing lower effort than writing the function manually . If our system fails to synthesise source code for a given problem , the user can specify a simpler but related I/O problem ; the successful synthesis of this simpler problem can then lead to subsequent success on the more complex problem . As a target programming language for synthesis , we use a Turing-complete language ( previously used by the authors of ( Wild & Porter , 2019 ) ) which can be cross-compiled directly into C/Java/Python . The language features primitive loop operators , variable declarations , and conditional branch operators ; a full listing of its operators is given in appendix A.1 . We use a combination of GP guided by NN prediction to approach this problem , and show successful synthesis of a wide range of programs including sorting . In this paper we study three specific parts of the above overall framework : ( i ) the success rate of GP by itself of finding programs in our problem sets ; ( ii ) the effect of different hints ( suggested lines of code ) on the success rate of GP ; ( iii ) the ability of a NN to predict high-utility hints , after being trained on a synthetic corpus derived from high-success program finds from the GP alone . In the remainder of this section we describe our problem sets , genetic programming and NN implementations in more detail , then present the set of experiments that we conduct . 3.1 SPECIFICATION OF PROBLEM . We use two different problem sets which represent human-useful programs of the kind that may be input into our system . Our first problem set is array-to-array programs such as extract even numbers , append arrays , or sort . Our second problem set is provided with an image to draw on a canvas and must synthesise the program which draws that image ( this problem set is taken from So & Oh ( 2018 ) ) . Each problem within each problem set is presented to the system as a set of 10 IO examples . These are generated for each problem by feeding in 10 inputs and corresponding outputs of the form the problem requires ( either a randomly generated input array and integer , or a canvas size ) . These 10 inputs are randomly generated from a fixed seed , to reduce internal variability between tests , allowing more accurate evaluation of the changes made by alterations to parameters or by guidance to the GP . This is designed to represent an unbiased input set .

Summary: This work considers training neural networks to provide "hints" (suggested lines of code) as a way of augmenting genetic programming systems. At test time, a neural network conditions on input-output examples and learns to predict 1-2 lines of code which are present in a program satisfying those input-outputs. At train time, the system learns in a self supervised manner by bootstrapping off previously discovered programs, augmenting them to produce a large corpus of similar problems, and then training the network to predict lines of code present in this augmented data set, similar to DeepCoder.