Discriminator Based Corpus Generation for General Code Synthesis

1 INTRODUCTION . Automated code synthesis is increasingly being studied as a way to lower the entry bar for nonexperts to create computer software , and to aid in generally taming the complexity of large-scale systems by allowing engineers to specify their intentions at a higher level of abstraction . The approach of neural code synthesis in particular has recently gained a lot of attention , applying advances in neural networks to the problem of automated synthesis . We specifically study the approach of programming by example , in which a small set of input-output examples are presented to the system to serve as a guide to the desired functionality of a program . Based on an analysis of these examples the synthesis system returns a source-code program able to replicate that functionality . Recent research in this field demonstrates promising results , including DeepCoder Balog et al . ( 2017 ) and Zohar & Wolf ( 2018 ) . However , research to date is limited to using domain-specific languages and often linear sequential programs without conditions or loops . We also take a neural-network-based approach to this problem in an attempt to gain inter-program inference across the training examples given to our system , potentially allowing the system to learn general aspects of programming to help synthesize new programs from unseen input/output examples . Unlike existing recent work , however , we target a general-purpose low-level programming language for code synthesis with a much larger search space of possible programs . This presents a major challenge in generating a training corpus for the neural network . Where related research has used uniform sampling methods through program search space ( Sun et al . ( 2018 ) Chen et al . ( 2017 ) ) , or even enumerative approaches ( Balog et al . ( 2017 ) ) , such approaches are wholly inadequate over larger search volumes – with sparse sampling producing very poor inference results . To solve this training corpus generation problem we propose a novel discriminator-based system , in which new sub-corpora are iteratively created , continually measuring their functional properties against those of the problems it is attempting to solve . This process works by learning how similar the I/O mappings of generated programs are to I/O problems requested by users ; by selecting programs which result in increasingly similar I/O mappings we simultaneously choose programs with similar underlying source code features , until we are able to solve I/O problems requested by users . We demonstrate that the resultant training corpus is greatly superior to a conventionally generated corpus via uniform sampling , when using a more generalised programming language for synthesis . We measure the performance of our approach by comparing against similar research on neural code synthesis which uses uniform or enumerative sampling for training , demonstrating that our 1 Under review as a conference paper at ICLR 2020 discriminator-informed corpus generation approach far exceeds uniform sampling , by a factor of 2 , in terms of find-rates . We also compare against a general baseline using genetic programming ( GP ) ; this baseline produces a surprising result that GP has a broader range of programs found , although its probability of resolving any given user-provided problem is worse . Our approach offers an effective way to generate a training corpus for a high-dimensional program search space , capable of finding a wide range of unseen useful programs based only on input/output examples , without any labelled training data . At a high level our research also demonstrates that the structure of the training corpus provided to a neural network greatly affects its performance on general purpose code generation tasks , and we argue that it should therefore represent a core focus of the code synthesis community ’ s efforts alongside work on neural network and language structures . In the remainder of this paper we firstly assess the literature in the field , focusing on neural code synthesis and specifically its corpus generation techniques . In Sec . 3 we then present the methodology we use to build our system , based on both a synthesis network and a discriminator network for corpus generation . We then evaluate our approach in Sec . 4 by comparing it against traditional training corpus generation approaches for neural code synthesis . [ code to reproduce our results will be made open-source should this paper be accepted , and this line will be changed to the link to the repository ] 2 BACKGROUND . Program synthesis has been studied for nearly as long as machine learning has been considered , having been proposed in 1950 by Alan Turing Turing ( 1950 ) . Three main subfields exist : logic-based solvers Feng et al . ( 2017 ; 2018 ) ; Feser et al . ( 2015a ; b ) ; Solar-Lezama et al . ( 2006 ) ; stochastic-search-based Genetic Programming ( GP ) Koza ( 2010 ) ; Haraldsson & Woodward ( 2014 ) ; and neural network based approaches Kant ( 2018 ) . While solvers are excellent for restricted domains , they have limited applicability in programs containing loops due to their reliance on predicate logic . Genetic programming has been used to good effect in specific domains , but lacks the cross-program inference power that neural networks offer . We hold neural code synthesis to be an interesting area of study following the results of DeepCoder ( Balog et al . ( 2017 ) ) , which suggests that neural network can identify higher level features of program outputs ( sortedness , evenness , offset from zero mean ... ) ; combined with this is the opportunity of inter-program inference within the representational model of the neural network . Within the subfield of neural synthesis , we focus on programming by example Gulwani & Jain ( 2017 ) , which takes a set of inputs and outputs which demonstrate a desired functionality . One of the core works in this field is DeepCoder Balog et al . ( 2017 ) , a system able to recognise subfunctions within a larger program consisting of up to 5 of these sub-functions called sequentially . This was significantly improved in subsequent work Zohar & Wolf ( 2018 ) , which used a novel partial program execution model with results fed back into the synthesis neural network , to enhance search speed and increase the length of programs that can be found up to 14 . Both approaches operated in the same Domain Specific Language , which focused on arrays of integers . In terms of corpus generation , DeepCoder uses a simple enumerative strategy , as its DSL creates a small enough program space to list all possible programs , while Zohar & Wolf ( 2018 ) use random sampling . Two other major domains exist in the field of neural code synthesis ; In both of these , we see that synthetic training sets are predominantly generated by uniform sampling ( Sun et al . ( 2018 ) ; Chen et al . ( 2017 ) ; Devlin et al . ( 2017 ) ) . The first of these two domains is string-manipulation functions Gulwani ( 2011 ) ; Raza & Gulwani ( 2017 ) , as these have immediate and obvious usefulness to non-expert users . While these tasks can be handled by solvers , there are good applications for neural synthesis systems , such as RobustFill Devlin et al . ( 2017 ) , or neurally-augmented solver systems Vijayakumar et al . ( 2018 ) . The second domain is deduction of the behavioural algorithm of an agent in an environment , with the examples being sequences of actions taken by the agent in a grid environment . This has been studied both in terms of trained neural networks recognising behavioural patterns Sun et al . ( 2018 ) Bunel et al . ( 2018 ) or in reinforcement learning based approaches to program generation Chen et al . ( 2017 ) . 2 Under review as a conference paper at ICLR 2020 Our work differs from these domains ’ use of random uniform sampling for training sets , as well as from the unscalable enumerative strategy of DeepCoder , in that we introduce a strategy for generating a targeted training corpus without requirement for human-provided labelled data , and we investigate how this generation strategy significantly boosts inference performance . Finally , the most similar work to our own is Shin et al . ( 2019 ) , which investigate the effects of inputs to programs in terms of the produced I/O examples ’ effect on training success . This differs from our work in that they explore which inputs to feed into sampled programs , while we are presenting a new program sampling technique . This approach may well be complementary to our own if using as part of our training corpus generation pipeline . 3 METHODOLOGY . Our overall approach to code synthesis is to feed in a set of ten input/output ( I/O ) examples to a neural network , and have the output layers of that neural network select the most likely operation for each line of code for a function of a given upper length limit ( where the number of output layers is equivalent to the maximum number of lines of code in the function , each neuron in an output layer is equivalent to selecting one particular operator for that line , and each line of code can be set to ‘ no-op ’ ) . The probabilistic nature of a neural network ’ s output allows us to gain a set of possible programs to search through up to a given search depth ; this basic approach is similar in spirit to DeepCoder Balog et al . ( 2017 ) in that it is attempting to guess the probability of each ‘ feature ’ of a program – except in our case these features are relatively low-level instructions . Neural networks for code synthesis are usually trained on a uniformly sampled set of programs from the total space of all possible programs ; while this is viable for simple domain-specific languages , it becomes intractable for more general purpose languages . With the aim of achieving code synthesis using a much more general programming language , our methodology focuses on how training data can be effectively drawn from a much larger search space without human input . Our approach to this uses a novel hybrid solution inspired by genetic programming but using a neural network as a fitness function . We term this neural network a discriminator , as it attempts to discriminate between the algorithms the genetic programming element is currently generating , and the likely features needed by requested I/O examples for human-useful functions . 3.1 LANGUAGE USED . We reuse the programming language used in Wild & Porter ( 2019 ) , a simplified C-like language , which can be compiled out to C or Java source code . The language features variable assignments , conditional operators , and loop operators . These allow complex flow-control operations , allowing us to study algorithms which go beyond linear concatenation of functions often used in the literature . We enhance the problem complexity considered by Wild & Porter ( 2019 ) by removing the restrictions on array length , instead implementing an operation which instantiates empty arrays of given length . We also implement operations to allow array length to be determined , and the ability for a function to call itself , to allow recursive functions . Our methodology also avoids using any humanprovided hints about the likely features of source code for problem solutions , instead relying only on a set of I/O examples for unsolved problems . The full set of language operators available and restrictions used is included as an appendix 6.1 .

Summary: This paper presents a technique based on genetic programming to generate a suitable training corpus of programs and I/O examples using a trained discriminator network. Given a set of human useful programs and the corresponding I/O examples, the main idea of the approach is to use genetic programming to iteratively construct new corpus leveraged by a discriminator that aims to make the I/O examples of the corpus closer to the I/O examples of the human-useful set. This approach is evaluated on 40 human useful corpus of array programs and it is able to synthesize more programs than random corpus or genetic programming based baselines.