Title: Lifelong Perceptual Programming By Example
Authors: Alexander L. Gaunt
Marc Brockschmidt
Nate Kushman
Daniel Tarlow
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning

Abstract: We introduce and develop solutions for the problem of Lifelong Perceptual Programming By Example (LPPBE). The problem is to induce a series of programs that require understanding perceptual data like images or text. LPPBE systems learn from weak supervision (input-output examples) and incrementally construct a shared library of components that grows and improves as more tasks are solved. Methodologically, we extend differentiable interpreters to operate on perceptual data and to share components across tasks. Empirically we show that this leads to a lifelong learning system that transfers knowledge to new tasks more effectively than baselines, and the performance on earlier tasks continues to improve even as the system learns on new, different tasks.

TL;DR: Combination of differentiable interpreters and neural networks for lifelong learning of a model composed of neural and source code functions

Review discussion:
This is a wonderful paper on program induction, with a very nice framework for combining code with neural networks.
I'd like to ask the authors a simple question, which in no way attempts to diminish the great value of this paper, but rather prompts for clarification.
Can you learn more complex programs such as sorting with weak supervision? (and without curriculum, which amounts to strong supervision of hierarchy). Reed et al learn sorting with only 8 demonstrations, but imitation is of course a stronger form of supervision.
Can the family of tasks that can be learned with weak supervision be easily characterised? Does classical complexity theory offer an answer?
Thank you for the question and the nice feedback.
To answer the last question first, we are unaware of any formal answer to the question. What is learnable depends not only on the supervision but also on the assumptions and priors that are encoded into the model (either explicitly or implicitly). What we infer based upon seeing examples [2, 1, 4]->[1, 2, 4] and [4, 3]->[3, 4] depends on the priors and inductive bias of the model.
Thus in some sense, this question boils down to "what inductive bias can/should be encoded over program space, and how powerful is it to disambiguate a 'correct' program (one that generalizes strongly) from an 'incorrect' program (one that does not)?"
We don't have a clear answer to this right now and it is beyond the scope of this paper, but our long-term view is the following: (1) the explicit source code representation provides strong inductive bias towards learning an algorithm that generalizes in a strong way; (2) this inductive bias could be strengthened further by studying how people use programming languages in practice and incorporating this as a prior over source code representations; and (3) learning in a lifelong setting can achieve many of the benefits of a curriculum and thus help towards building up to more complicated programs from simpler ones.
Finally, we believe that an ultimate lifelong learning system should be able to learn from many types of supervision, ranging from very strong to very weak, and I/O examples are one important point on that spectrum that we should be studying.
The authors explore the idea of life-long learning in the context of program generation.
The main weakness of this paper is that it mixes a few issues without showing strong results on any of them. The test tasks are about program generation, but these are toy tasks even by the low standards of deep-learning for program generation (except for the MATH task, they are limited to 2x2 grid). Even on MATH, the authors train and discuss generalization from 2-digit expressions -- these are very short, so the conclusiveness of the experiment is unclear.
The main point of the paper is supposed to be transfer learning though. Unluckily, the authors do not compare to other transfer learning models (e.g., "Progressive Neural Networks") nor do they test on tasks that were previously used by others. We find that only testing on a newly-created task with a weak baseline is not sufficient for ICLR acceptance.
After clarifying comments from the authors and more experiments (see the discussion above), I'm now convinced that the authors mostly measure overfitting, which in their model is prevented because the model is hand-fitted to the task.
While the idea might still be valid and interesting, many harder and much more diverse experiments are needed to verify it. I consider this paper a clear rejection at present.
In addition to responding in the general response comment above, we have now addressed many of the main issues raised by this review:
- Added a Neural GPU baseline for the MATH task.
- Modified baselines for MATH to train on up to 5-digit expressions, showing the conclusions are unchanged.
- Add a comparison to Progressive Neural Networks (PNNs), showing that our method substantially outperforms PNNs.
- Added a response (see the thread above under "General Response") to the new claim that we are "mostly measuring overfitting." Our comment argues why we do not believe this to be the case.
This paper proposes an extension of TerpreT by adding a set of functions that can deal with inputs in the form of tensor with continuous values. This potentially allows TerpreT to learn programs over images or other “natural” sources.
TerpreT generates a source code from a set of input/output examples. The code is generate in the form of a TensorFlow computation graph based on a set of simple and elegant program representations. One of the limitation of TerpreT is the type of inputs it can work with, this work aim at enriching it by adding “learnable functions” that can deal with more complex input variables.
While I really like this direction of research and the development of TerpreT, I find the contribution of this work to be a bit limited. This would have been fine if it was supported by a strong and convincing experimental section, but unfortunately, the experimental section is a bit weak: the tasks studied are relatively simple and the baselines are not very strong.
For example let us consider the SUM2x2 problem:  all the images of digits are from MNIST, which can be classify with an error of 8% with a linear model (and even better with neural networks), There is also a linear model that given 4 numbers will compute the 2x2sum of them that is: y=Ax where x is the vector containing the 4 numbers and A = [1 0 1 0;1 1 0 0;1 0 0 1; 0 1 0 1]. This means a succession of two linear models can solve the sum2x2 problems with little trouble.
While I'm aware that this work aims at automatically finding the combination of simple models to achieve this task end-to-end, the fact that the solution is a set of 2 consecutive linear models makes it a bit too simple in my humble opinion.
Overall, I think that this paper proposes a promising extension of TerpreT that is unfortunately not backed by experiments that are convincing enough.
Thanks for the thoughtful review.
The preliminary tasks are less about individual task complexity, and more about the ability to learn modular knowledge that can be transferred to more complex tasks. The MATH task is on a level of complexity comparable to a failure case of an existing system in the literature that aims to learn algorithms [1].
We have now added Progressive Neural Network and Neural GPU baselines which are strong baselines for transfer learning and learning algorithms respectively. Our approach outperforms these baselines and the qualitative conclusions of the paper are unchanged.
[1] Eric Price, Wojciech Zaremba & Ilya Sutskever; Extensions and Limitations of the Neural GPU (submitted to ICLR 2017)
I think the paper is a bit more solid now and I still stand by my positive review. I do however agree with other reviewers that the tasks are very simple.
While NPI is trained with stronger supervision, it is able to learn quicksort perfectly as shown by Dawn Song and colleagues in this conference. Reed et al had already demonstrated it for bubblesort. If the programs are much shorter, it becomes easy to marginalise over latent variables (pointers) and solve the task end to end. The failure to attack much longer combinatorial problems is my main complaint about this paper, because it makes one feel that it is over-claiming.
In relation to the comments concerning NPI,  Reed et al freeze the weights of the core LSTM to then show that an LSTM with fixed weights can continue learning new programs that re-use the existing programs (ie the trained model can create new programs).
However, despite this criticism, I still think this is an excellent paper, illustrating the power of combining traditional programming with neural networks. It is very promising and I would love to see it appear at ICLR.
This paper makes a valuable contribution to the emerging research area of learning programs from data.
The authors mix their TerpreT framework, which enables them to compile programs with finite integer variables to a (differentiable) TensorFlow graph,  and neural networks for perceiving simple images. This is made possible through the use of simple tapes and arithmetic tasks.  In these arithmetic tasks, two networks are re-used, one for digits and one for arithmetic operations.  This clean setup enables the authors to demonstrate not only the avoidance of catastrophic interference, but in fact some reverse transfer.
Overall, this is a very elegant and potentially very useful way to combine symbolic programming with neural networks. As a full-fledged tool, it could become very useful.
Thus far it has only been demonstrated on very simple examples. It would be nice for instance to see it demonstrated in all the tasks introduced in other approaches to neural programming and induction: sorting, image manipulation, semantic parsing, question answering. Hopefully, the authors will release neural TerpreT to further advance research in this domain.
We thank all the reviewers for their time and comments on our work.
To summarize, two reviewers state that this is an interesting line of research, and that this paper could be a valuable contribution to the emerging field of learning programs from data.
However the reviewers raise a concern that the tasks presented in the experimental section are too simple and the baselines may be weak.
In particular, AnonReviewer1 notes that if strong supervision were available the ADD2x2 task could be solved by a baseline consisting of 2 consecutive linear models.
However, in this paper we are concerned entirely with the LPPBE paradigm, where weak supervision and lifelong learning are key. We argue that the ADD2x2 and APPLY2x2 tasks in the experimental section were not chosen for complexity, but instead chosen to cleanly illustrate different benefits of our method relative to a purely neural baseline subject to these LPPBE restrictions. AnonReviewer5 agrees that our choice of tasks allows for a clean illustration of the key qualitative conclusions: success with weak supervision (Figure 5) and avoidance of catastrophic forgetting / demonstration of reverse transfer (Figure 6).
Having progressed through the illustrative early tasks, the lifetime of learning culminates in the MATH task, which is a non-trivial task of some practical use (evaluation of handwritten arithmetic expressions of arbitrary length). We argue that AnonReviewer1’s analysis concentrates on a linear model for our simplest ADD2x2 task and ignores this more complex MATH task (which even with strong supervision would require at least a recurrent non-linear model).
We thank the AnonReviewer3 for the suggestion to include a Progressive Neural Network (PNN) as a more modern baseline model. We will include results using PNNs in a final version of the paper. Note that this baseline does not affect Figure 5 (which shows the success of our model over all weakly supervised baselines on a single task), and while PNNs may prevent catastrophic forgetting in Figure 6, they will not exhibit reverse transfer. We therefore feel that the main qualitative messages of the paper are not significantly changed with the addition of this baseline.
Ultimately, as we mention in the paper, the complexity of the tasks that we consider is limited by the known difficulty in training differentiable interpreters. The tasks we present in this paper require search over program spaces of size up to 10^12 syntactically distinct programs. Solution to problems of this scale is comparable to the state of the art for methods using weakly supervised differentiable interpreters. It is an open area of research to improve the performance of these models, and this work aims to provide motivation for this line of research by producing the first demonstration of the value of differentiable interpreters as a glue for combining symbolic programming with neural networks.
I thank the authors for their reply. I find including PNNs and also more complex fully supervised models for MATH (e.g., the Neural Turing Machine) is essential. I would consider increasing my score if these were included, but of course only once this is done.
Thank you for your comment. We have now included the PNN baseline in the current revision of the paper, and we have improved the LSTM baseline in the MATH task.
We believe that the new LSTM results now make for a convincing and natural baseline, but to ensure that we have satisfied your concerns, we would like to ask for clarification on the request for more complex fully supervised models for MATH (e.g. the Neural Turing Machine).
specifically:
(i) We are not sure what is meant by 'fully supervised': All of our models are weakly supervised with input-output examples.
(ii) The Neural Turing Machine has been shown to be effective at learning copy/recall/sorting algorithms which require shifting patterns around in memory. It is not clear that NTMs are more effective than LSTMs at learning to perform arithmetic operations, nor is it clear how to include perceptual inputs to a NTM. We believe that solving these issues with NTMs might be considered a separate line of work rather than a baseline for the present work.
Thanks for the new LSTM baseline. I think it makes a good addition. I see it in Figure 8 now, but cannot find the numbers. From what I see:
* the NTPT and LSTM perform the same on 5-digit test when trained on 5-digit training (is this true?)
* the NTPT generalizes a bit better; it also has less parameters if I understood correctly?
Was dropout used in the LSTM? NTM should be more effective at generalization, but indeed, it hasn't been applied to arithmetics too often. You could try the Neural GPU which generalizes much better than an LSTM. Could you also clarify how many parameters the LSTM/NTPT have, if the LSTM has dropout (to generalize well with a lot of parameters, it certainly should), and what the exact numbers are (it's hard to read them from Figure 8).
Thank you for your comment. In light of your questions we have made three further improvements to the generalization section (5.2) and Fig 8 (updates shown in red in the pdf).
(1) Dropout
_Summary_: We improved the performance of both the LSTM baseline and NTPT by training for longer.
_Details_: We find that dropout significantly slows down learning, but ultimately, by using dropout we were able to improve the baseline performance by increasing the baseline model size to a 3 layer LSTM with 1024 units in each layer (details of this architecture are now included at the end of section 4). This configuration required 8M training instances to reach convergence.
To be fair, we also extended the training of the NTPT model to 8M instances. The source code learned by NTPT converges rapidly (after 30k instances), but we find that extended training on the MATH task continues to improve digit classifier (net_0) in the library of neural functions from 95% to 97.5% (converged) accuracy on a single digit classification task. The performance of the NTPT model is very sensitive to the performance of the classifiers for long arithmetic expressions (e.g. consider the ratio 0.975^16/0.95^16 = 1.5 for the 16 digit case).
Therefore, this small boost in classifier performance has a significant effect on the generalization accuracy.
_Numerical values_: Below we provide the requested numerical values for the accuracies plotted in Fig. 8b:
# Digits | 2 | 3 | 4| 5| 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16
LSTM (2-digit) | 96.2 | 19.6 | 18.2 | 19.0 | 18.0 | 17.6 | 18.3 | 17.7 | 18.4 | 18.9 | 18.9 | 18.5 | 18.7 | 17.8 | 19.1
LSTM (5-digit) | 94.2 | 92.1 | 90.8 | 89.3 | 88.4 | 88.0 | 87.0 | 85.7 | 85.2 | 85.0 | 84.4 | 84.4 | 83.9 | 82.9 | 82.8
NTPT (2-digit) | 96.2 | 94.3 | 93.3 | 92.1 | 91.2 | 91.3 | 90.1 | 89.6 | 88.8 | 88.8 | 88.7 | 88.1 | 88.3 | 87.3 | 87.1
The numbers in parentheses indicate the maximum expression length observed during training. The conclusion remains unchanged: NTPT trained on only 2-digit expressions outperforms the best LSTM baseline even when the LSTM is provided with up to 5-digit expressions during training.
_Changes to paper_: We have updated Fig 8b with the results from this extended training (and we include the numerical values for the 16 digit test accuracy). We also include a comment on the sensitivity of NTPT to the classifier accuracies in section 5.2.
(2) Neural GPU baseline
_Summary_: We performed an investigation into a Neural GPU baseline on the MATH task as requested, but find that it shows worse generalization than the LSTM. Independent findings from [1] support our conclusion.
_Details_: We obtained an implementation of the Neural GPU from the original authors [2] (which includes all training tricks highlighted in [3]). To assess this model, we first evaluated the relative performance of the Neural GPU, the LSTM and TerpreT models on a simpler, perception-free version of the MATH task where digits and operators are supplied as one-hot vectors and there is not the additional complication of decoding handwritten symbols. Following [1] we tried Neural GPUs with 128, 256, or 512 filters, and found best performance with the largest model. We find that the Neural GPU can perfectly fit the training data (expressions of length <=5 digits), but generalizes very poorly to longer expressions. The LSTM on the other hand is seen to be a very strong baseline (see the numerical values below and Fig. 8a).
Our findings are supported by [1] where the authors consider a task similar to our MATH problem. There are two main differences between our experiment and [1] which affect the difficulty of the task in different ways – these are:
(a) we use a decimal representation of numbers while [1] uses binary (both [3] and [1] suggest that decimal representation tends to increase the difficulty of the task for the Neural GPU)
(b) we consider shorter test cases: up to 32 tokens (digits + operators) on the input vs. 201 tokens in [1] (this reduces the difficulty of our task).
Using a 512-filter Neural GPU we achieve a 25% accuracy on our 32 token decimal MATH task, and argue that this seems compatible with the results from [1] where a 512-filter Neural GPU achieves a similarly low ~30% accuracy on the 201 token binary MATH task.
Our experimental findings (& [1]) indicate that the LSTM is a much stronger baseline than the Neural GPU for our task. Since the gap in performance is so large on this simplified, perception-free version of the MATH task, we drop the Neural GPU when we re-introduce the perceptual component into the MATH experiments (Fig 8b).
_Numerical values_: Below are the numerical values for the accuracies of all models on the perception-free MATH task (plotted in Fig 8a).
# Digits | 2 | 3 | 4| 5| 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16
Neural GPU (5-digit) | 100 | 100 | 100 | 100 | 99.56 | 92.82 | 68.55 | 54.35 | 47.36 | 44.82 | 42.72 | 38.18 | 34.33 | 29.0 | 25.0
LSTM (5-digit) | 100 | 100 | 100 | 99.9 | 99.2 | 98.7 | 98.2 | 96.4 | 96.2 | 95.6 | 94.7 | 93.2 | 93.0 | 91.1 | 92.8
NTPT (2-digit) | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100
_Changes to the paper_: To justify our belief that the LSTM is a strong baseline, we include the results for the performance of the Neural GPU, LSTM and TerpreT on the simplified, perception-free version of the MATH task in the paper (Fig 8a). A short description of the results in Fig 8a is also included in section 5.2.
(3) Parameter counts
_Summary_: The algorithmic part of NTPT has six orders of magnitude fewer parameters than the LSTM baseline.
_Numerical values_: Below we report the number of parameters in each model excluding parameters in the perceptual networks (net_0/1):
NTPT: 32
LSTM: 21.1M
Neural GPU (512 filters): 43.8M
The perceptual nets themselves contribute an additional 0.5M parameters to each model.
_Changes to the paper_: These parameter counts are included in Fig 8a.
[1] Eric Price, Wojciech Zaremba & Ilya Sutskever; Extensions and Limitations of the Neural GPU (submitted to ICLR 2017)
[2] https://github.com/tensorflow/models/tree/master/neural_gpu
[3] Łukasz Kaiser & Ilya Sutskever; Neural GPUs Learn Algorithms (ICLR 2016)
I'm still trying to understand the above results. One thing that stands out to me is the number of parameters:
NTPT: 32
Is this correct? Only 32 floats, that's the whole model? It's hard to believe it can train on harder tasks. Clearly I missed something, I'll re-think it. Could you confirm that the model has 32 parameters? Thanks!
Yes, the NTPT model for MATH contains 32 independent floats that define distributions over 19 discrete values. These discrete values define a combinatorially large space of around 10^6 different programs in the MATH model (and in other cases the space is larger, such as fig 3b having a program space of over 10^12 syntactically distinct programs).
In addition to these parameters, there are 0.5M parameters in the "perceptual" neural networks that define the mappings from images to distributions over discrete values in the programs.
While it may seem that 32 values is small, there are a few things to consider:
- Each joint configuration of the discrete variables gives rise to a different program; a small number of variables can parameterize a very large space of functions
- NTPT separates the parameterization of the "algorithmic" and "perceptual" components of the model. There are many tasks where the algorithmic component is relatively simple given a good mapping from perceptual to discrete representations (e.g., navigation). If we can compactly parameterize the algorithmic component and do a good job of jointly learning the perceptual and algorithmic components (as NTPT allows), then we expect stronger generalization, which is what we see in experiments.
- Representationally it is easy to specify a Turing machine or assembly code-like model for the algorithmic component of a NTPT model (see [1]), so there is no ceiling on the representational capacity of the algorithmic component. The challenge currently holding us back from scaling up further is not the representational capacity of the model; it's in the optimization.
[1] Alexander L. Gaunt et al. Terpret: A probabilistic programming language for program induction. (http://arxiv.org/abs/1608.04428)
Thank you for your explanation. It took me some time to ponder how your experiments can be true.
For example, it was clearly established that Neural GPU generalizes much better than a LSTM baseline on arithmetic tasks, while you show the opposite. The number of parameters solved the riddle: a powerful model with 40000000 parameters will overfit 5-digit training much easier than a less-powerful LSTM with half that many, and, clearly, a 32-parameter model will not overfit.
Thus, it seems like your experiments mostly measure overfitting, and it is a strong concern that NTPT only performs well because it is small, and the problems are hand-picked to make it work.
While this might be false, much more extensive experiments are needed, including NTPTs with much more parameters and many more diverse tasks. If the results hold in the extensive experiments, it might become a great technique.
But in the present version, the results are simply not sufficent.
There are some errors in this comment:
1. "a powerful model with 40M parameters will overfit 5-digit training much easier than a less-powerful LSTM with half that many, and, clearly, a 32-parameter model will not overfit."
It is not generally valid to compare model capacity just by counting parameters, especially when dealing with very different model classes. It is also not valid in the specific case being discussed here.  In [1], the Neural GPU model has 200k parameters while the baseline LSTM has 30k parameters, but the LSTM still "overfits" (in the strong generalization sense) on the Long Binary Addition task while the Neural GPU does not.
Further, [2] states "It is not self-evident that larger [Neural GPU] model would generalize better, but the empirical results are clear."
To confirm this in our specific setting, we ran additional experiments reducing the number of parameters for the Neural GPU, and the trend is that generalization performance gets worse as the number of parameters decreases (see the generalization performance on 8-digit expressions below).
Thus, the issue with the Neural GPU is not overfitting, and explaining generalization performance in terms of number of parameters is overly simplistic and does not match experimental results.
# filters | 32 | 64 | 128 | 256 | 512
# parameters | 169k | 671k | 2.7M | 10.7M | 43.8M
8-digit accuracy | 22.5% | 40.1% | 62.7% | 63.3% | 68.6%
2. "it was clearly established that Neural GPU generalizes much better than a LSTM baseline on arithmetic tasks, while you show the opposite"
We are aware of only two experiments where Neural GPU was compared to LSTM. These are the ones in [1] showing that Neural GPU outperforms LSTM+attention on long addition and long multiplication when:
(a) data are represented in binary,
(b) expressions are always of length 2, and
(c) a separate model is trained for each operator (addition vs multiplication).
Further, it does not appear that the experimental protocol used in [1] is applicable to the case where there are only short data sequences available for training.
Specifically, the results rely upon access to the longer test data in order to select which setting from the grid should be used, and the text reports that generalization performance is sensitive to the choice of seed.
In contrast, in our experiment:
(a) data are represented in decimal
(b) expressions are of variable length,
(c) a single model is trained on expressions that have a mix of operators, and
(d) we do not assume access to longer validation data.
Our setting is more challenging and defines a stricter definition of strong generalization than [1] (namely, generalizing beyond the length of data used for training *or validation*). In our setting, there is no evidence that Neural GPU generalizes better than an LSTM baseline.
Regarding (a), the literature [1, 2] reports that Neural GPU performance degrades when numbers are represented in decimal, but there are no experiments showing how the LSTM baseline is affected.
Regarding (b) & (c), [2] experiments with evaluating variable-length expressions with mixed mathematical operators (albeit on binary numbers). They report that a Neural GPU is not able to generalize well to longer sequences, which is consistent with our experiments. There is no LSTM baseline in [2].
In summary, there is no contradiction between our experiments and existing work, and our experiments are not "mostly measuring overfitting".
Our paper is about supplying a very different type of inductive bias than models like an LSTM or Neural GPU. There are different assumptions being made in NTPT (the key one being that there is a component of the model that is well described using source code). We don't claim that this is the right assumption for all tasks (e.g., image classification), but there is a large space of tasks (e.g., evaluating a mathematical expression, navigation, complex policies in RL) where this assumption is valid, and in these cases we believe NTPT-like models will generalize more strongly from less data than models like LSTM or Neural GPU.
[1] Łukasz Kaiser & Ilya Sutskever; Neural GPUs Learn Algorithms (ICLR 2016)
[2] Eric Price, Wojciech Zaremba & Ilya Sutskever; Extensions and Limitations of the Neural GPU (submitted to ICLR 2017)
It might be that you misunderstood my comments. I find your points very convincing in the discussion about "how much the number of parameters contributes to overfitting" and "LSTM vs Neural GPU", I agree with a lot of what you say.
But this is about comparing your 32-parameter model with dozens-of-millions parameter baselines. It's simply not a fair comparisons when you train those baselines at length 5. I read the other papers more carefully now, for example the Neural GPU was trained on length 40 and evaluated on lengths 400 or so in the original paper -- did you try that?
But my point is bigger than a specific training procedure for a specific baseline. Deep learning works great because it is at the intersection of learnable and powerful models. Your model with 32 parameters does not look powerful. You only have 1 non-trivial task (MATH) and your baseline experiments are not convincing. If you want to convince the readers that a model with 32 parameters can really be powerful, you'll have to include a number of tasks, not just MATH, which your model can be hand-tuned for. How about including all the algorithmic tasks from other papers that considered them? And training baselines on a lot of long data for a fair comparison.
At present, I simply don't believe that a model with 32 parameters can be a powerful deep learning model, and your experiments are not sufficient to warrant that belief.
Moreover, I find some of your claims (such as claims that your model outperforms multi-million parameter baselines) misleading.
While true on a single task that you designed, I don't believe they will hold up otherwise. The misleading presentation is a strong argument for me that the paper should be rejected.
We thank the reviewers again for their suggestions. We have now rewritten the experiment section, and believe that it is considerably clearer and more complete thanks to their suggestions. The main changes that address the reviewers' concerns are:
(1) We have improved the description of the baseline models and included a Progressive Neural Network baseline in the 2x2 tasks as requested. These updates can be seen in section 4, figure 6c and figure 7.
(2) We additionally study a Neural GPU baseline for the MATH task as an example of a more modern neural architecture for learning algorithms.
In addition, we have pointed out that the MATH task is similar in complexity to tasks in contemporary literature [1]. These updates can be seen in section 5.2 and figure 8a.
(3) We have extended the training of the LSTM baseline in the MATH task beyond two digit expressions and ensured that we have trained all models to convergence. We also now test on longer expressions up to 16 digits. These updates can be seen in section 5.2 and figure 8b.
The qualitative conclusions of the paper are unchanged:
(1) We demonstrate simultaneous learning of source code and neural networks to solve algorithmic tasks with perceptual inputs.
(2) In a lifetime of learning, NTPT outperforms all baselines, and is the only model to demonstrate reverse transfer.
(3) The source code representation of the algorithm for the MATH task leads to better generalization performance than baseline models on long expressions.
[1] Eric Price, Wojciech Zaremba & Ilya Sutskever; Extensions and Limitations of the Neural GPU (submitted to ICLR 2017)

Meta-review:
role: PC
data: comment: The reviewers were quite divided on this submission, which proposes a method for lifelong learning in the context of program generation. While a novel idea, the experiments and baselines are simply not clear enough or convincing enough, and the method itself is not clearly conveyed. The paper makes strong claims that are not substantiated with more compelling or challenging experiments. Since this is an interesting, novel idea in a relevant area, however, I think it should be invited as a workshop paper.

Decision: Invite to Workshop Track