Title: Programming With a Differentiable Forth Interpreter
Authors: Matko Bošnjak
Tim Rocktäschel
Jason Naradowsky
Sebastian Riedel
Venue: ICLR 2017

Abstract: There are families of neural networks that can learn to compute any function, provided sufficient training data. However, given that in practice training data is scarce for all but a small set of problems, a core question is how to incorporate prior knowledge into a model. Here we consider the case of prior procedural knowledge, such as knowing the overall recursive structure of a sequence transduction program or the fact that a program will likely use arithmetic operations on real numbers to solve a task. To this end we present a differentiable interpreter for the programming language Forth. Through a neural implementation of the dual stack machine that underlies Forth, programmers can write program sketches with slots that can be filled with behaviour trained from program input-output data. As the program interpreter is end-to-end differentiable, we can optimize this behaviour directly through gradient descent techniques on user specified objectives, and also integrate the program into any larger neural computation graph. We show empirically that our interpreter is able to effectively leverage different levels of prior program structure and learn complex transduction tasks such as sequence sorting or addition with substantially less data and better generalisation over problem sizes. In addition, we introduce neural program optimisations based on symbolic computation and parallel branching that lead to significant speed improvements. 

TL;DR: This paper presents the first neural implementation of an abstract machine for an actual language, allowing programmers to inject prior procedural knowledge into neural architectures in a straightforward manner.

Review discussion:
Interesting ideas. A few questions:
- Could you explain in more detail what the Manipulate and Permute decoders do in Sec 3.2? Manipulate just uses an MLP to turn h into a set of m vectors? What is the structure of Permute?
- Is it possible to cast your learning problems as discrete search problems? If so, how large would the search spaces be for the problems considered in the experiments?
- What are the reasons for choosing a squared error objective in Sec 3.5 over cross entropy?
- Is it possible to discretize a learned model and interpret it as a discrete program like in Neural Random Access Machines (Kurach et al)? Would this improve the results on generalizing to longer sequences than seen in training?
Thank you for your questions. Here are our answers:
Answer 1
Manipulate decoder:
In a case of a static encoder, manipulate produces the output of size m * w (m is the number of manipulated stack elements, w is the stack width) which is essentially equal to encoder biases reshaped into m * w.
In a case of a observe decoder, the output of the encoder is h = W * input (where W is a learnt matrix, h is of size m * w). Manipulate produces the output of size m * w again, where m * w is essentially the reshaped output of the encoder h.
In addition to that, we enable the definition of multiple intermediate hidden layers by chaining linear layers of a set size (linear N) and activation functions (sigmoid, tanh). In this case, manipulate IS an MLP (at least one hidden layer).
Permute decoder:
The output of the permute decoder is a linear combination of m! state vectors, one for each permutation of the m chosen stack elements. The weights of this combination are calculated as softmax(W*h) where h is the encoder vector.
We will make sure to add these details to the next iteration of the paper.
Answer 2
When considering the particular problems presented in the paper, there are 600 states for the bubble sort and 4000 for the adder.
However, an aim of our architecture is to allow working with dense vectors as well (e.g. as the output of an RNN). In that sense, we don't look at the problem as a discrete learning problem.
Answer 3
Yes, squared error objective allows us to work with vectors that represent continuous values, not just discrete values. We did experiment with cross entropy loss for problems where the machine state contains only discrete symbols, but in our experiments this did not make a big difference.
Answer 4
Indeed we do that, as mentioned in paragraph 4: "During test time we employ memory element discretisation, replacing differentiable stacks and pointers with their discrete counterparts." And yes, that improves generalisation to longer sequences.
This paper develops a differentiable interpreter for the Forth programming
language. This enables writing a program "sketch" (a program with parts left
out), with a hole to be filled in based upon learning from input-output
examples. The main technical development is to start with an abstract machine
for the Forth language, and then to make all of the operations differentiable.
The technique for making operations differentiable is analogous to what is done
in models like Neural Turing Machine and Stack RNN. Special syntax is developed
for specifying holes, which gives the pattern about what data should be read
when filling in the hole, which data should be written, and what the rough
structure of the model that fills the hole should be. Motivation for why one
should want to do this is that it enables composing program sketches with other
differentiable models like standard neural networks, but the experiments focus
on sorting and addition tasks with relatively small degrees of freedom for how
to fill in the holes.
Experimentally, result show that sorting and addition can be learned given
strong sketches.
The aim of this paper is very ambitious: convert a full programming language to
be differentiable, and I admire this ambition. The idea is provocative and I
think will inspire people in the ICLR community.
The main weakness is that the experiments are somewhat trivial and there are no
baselines. I believe that simply enumerating possible values to fill in the
holes would work better, and if that is possible, then it's not clear to me what
is practically gained from this formulation. (The authors argue that the point
is to compose differentiable Forth sketches with neural networks sitting below,
but if the holes can be filled by brute force, then could the underlying neural
network not be separately trained to maximize the probability assigned to any
filling of the hole that produces the correct input-output behavior?)
Related, one thing that is missing, in my opinion, is a more nuanced outlook of
where the authors believe this work is going. Based on the small scale of the
experiments and from reading other related papers in the area, I sense that it
is hard to scale up differentiable forth to large real-world problems. It
would be nice to have more discussion about this, and perhaps even an experiment
that demonstrates a failure case. Is there a problem that is somewhat more
complex than the ones that appear in the paper where the approach does not work?
What has been tried to make it work? What are the failure modes? What are the
challenges that the authors believe need to be overcome to make this work.
Overall, I think this paper deserves consideration for being provocative.
However, I'm hesitant to strongly recommend acceptance because the experiments
are weak.
Thank you for your review and feedback. It’s great to hear that you found the paper provocative and inspiring. You are right that for the tasks in our experiments we are neuralising problems that could also have been solved by an exhaustive search over discrete programs.
However, such brute-force enumeration only works if the inputs to the program are discrete, but not dense vector representations (such as image representations or word vectors). Our architecture is designed to be able to backpropagate through the entire execution of a program, thus allowing us to calculate a gradient with respect to input representations.
Furthermore, output representations can be used in a downstream neural network that is trained jointly with d4.
Admittedly, we did not test this explicitly in our experiments, but instead first focused on providing a testbed for understanding how much prior structural bias a neural architecture needs to solve a certain problem.
In addition, we added the Seq2Seq baseline to both tasks and expanded the experimental section. We also expanded the adding task description and experimental setting with an additional sketch (based on choose) and showed that it performs better than manipulate, and explained why.
Furthermore, we added a “Discussion” section where we discuss some of the failure cases, and underlying reasons, as well as proposing ways to circumvent them. The conclusion section has been expanded with future work offering both the nuanced outlook and the long-term goals of the framework.
This paper presents an approach to do (structured) program induction based on program sketches in Forth (a simple stack based language). They turn the overall too open problem of program induction into a slot filling problem, with a differentiable Forth interpreter, for which one can backprop through  the slots (as they are random variables). The point of having sketches/partial programs is that one can learn more complex programs than starting from scratch (with no prior information). The loss that they optimize (end to end through the program flow) is a L2 (RMSE) of the program memory (at targeted/non-masked adresses) and the desired output. They show that they can learn addition, and bubble sort, both with a Permute (3-way) sketch and with a Compare (2-way) sketch.
The idea of making a language fully differentiable to write partial programs (sketches) and have them completed was previously explored in the  probabilistic programming community and more recently with TerpreT. I think that using Forth (a very simple stack-based language) as the sketch definition language is interesting in itself, as it is between machine code (Neural Turing Machine, Stack RNN, Neural RAM approaches...) and higher level languages (Church, TerpreT, ProbLog...).
Section 3.3.1 (and Figure 2) could be made clearer (explain the color code, explain the parallel between D and the input list).
The experimental section is quite sparse, even for learning to sort, there is only one experimental setting (train on length 3 and test on length 8), and .e.g no study of the length at which the generalization breaks (it seems that it possibly does not), no study of the "relative runtime improvement" w.r.t. the training set (in size and length of input sequences). There are no baselines (not even at least exhaustive search, one of the neural approaches would be a plus) to compare to.
Similarly, the "addition" experiment (section 4.2) is very shortly described, and there are no baselines either (whereas this is a staple of "neural approaches" to program induction). Does "The presented sketch, when trained on single-digit addition examples, successfully learns the addition, and generalises to longer sequences." mean that it generalizes to three digits or more?
Overall, the paper is very interesting, but it seems to me like the experiments do not support the claims, nor the usefulness, enough.
Thank you for pointing out the error in Figure 2. It indeed contained a color coding error. We updated the paper, both explaining this figure in more detail and appropriately fixing section 3.3.1.
Regarding additional baselines, we want to note that a Seq2Seq model has already been tested in the NPI paper (Reed et al. 2015) on the sorting and addition tasks and it was found that there is no generalisation for sequences longer than the ones trained on.
Still, we added a Seq2Seq baseline to the paper and can confirm the results by Reed et al. (2015).
Furthermore, we updated the paper with a detailed analysis regarding at which sequence lengths the models are not able to generalise. We argue that due to test-time discretisation, once the model learns the correct behaviour, it is able to generalise to a sequence of any length. We ran our experiments on sequences up to length 64 (we ran it on sequences of lengths 1 2 4 8 16 32 and 64, but displayed only 8 and 64 for brevity) and observed perfect generalisation for all these test lengths. We also expanded both the description of the addition task and its experimental section by adding another more specific sketch to it (based on choose).
The experiments show that we are able to generalise to long sequences (tested up to length 64). Note that due to the usage of one-hot encoding for the sequence length counter we are not able to run our models on much longer sequences, a limitation that we know discuss in the “Discussion” section in the paper. We believe a solution to this issue could be floating point representation, which we plan to investigate for future work.
This paper presents an approach to make a programming language (Forth) interpreter differentiable such that it can learn the implementation of high-level instruction from provided examples. The paper is well-written and the research is well-motivated.
Overall, I find this paper is interesting and pleasure to read.
However, the experiments only serve as proof of concept. A more detailed empirical studies can strength the paper.
Comments:
- To my knowledge, the proposed approach is novel and nicely bridge programming by example and sketches by programmers. The proposed approach borrow some ideas from probabilistic programming and Neural Turing Machine, but it is significantly different from these methods. It also presents optimisations of the interpreter to speed-up the training.
- It would be interesting to present results on different types of programming problems and see how complex of low-level code can be generated.
Thank you for your review and the suggestion regarding improving our empirical studies.  We have included additional experiments, baselines, and discussions that we hope address at least some of these concerns.
Dear reviewers, thank you for your thoughtful reviews. Based on your comments we uploaded an updated version of the paper in which we:
- added results of a Seq2Seq baseline to both sorting and adding tasks
- added a much more detailed quantitative analysis, regarding at which sequence length the models are not able to generalise, together with a qualitative evaluation
- expanded the addition task section with an additional sketch (choose), more text clarifying the sketches and a detailed quantitative analysis
- added a “Discussion” section clearly stating the limitations of the presented framework, and discussing failures
expanded the conclusion with additional future work, showing both nuanced outlook, and the long-term goals of the framework
- fixed Figure 2 and added details to section 3.3.1

Meta-review:
role: PC
data: comment: This work is stood out for many reviewers in terms of it's clarity ("pleasure to read") and originality, with reviewers calling it "very ambitious" and "provocative". Reviewers find the approach novel, and to fill an interesting niche in the area. All the reviewers were interested in the results, even if they did not buy completely the motivation (what "practically gained from this formulation", how does this fit in with prob programming).
 
 The main quality and impact issue is the lack of experimental results and baselines. Several reviewers find that the experiments "do not fit the claims", and ask for any type of baselines, even just enumeration. Lacking empirical evidence, there is a desire for a future plan showing what this type of approach could be useful for, even if it cannot really scale. I recommend this paper to be submitted to the workshop track.

Decision: Invite to Workshop Track