Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an extension of TerpreT by adding a set of functions that can deal with inputs in the form of tensor with continuous values. This potentially allows TerpreT to learn programs over images or other “natural” sources.

TerpreT generates a source code from a set of input/output examples. The code is generate in the form of a TensorFlow computation graph based on a set of simple and elegant program representations. One of the limitation of TerpreT is the type of inputs it can work with, this work aim at enriching it by adding “learnable functions” that can deal with more complex input variables.

While I really like this direction of research and the development of TerpreT, I find the contribution of this work to be a bit limited. This would have been fine if it was supported by a strong and convincing experimental section, but unfortunately, the experimental section is a bit weak: the tasks studied are relatively simple and the baselines are not very strong.

For example let us consider the SUM2x2 problem:  all the images of digits are from MNIST, which can be classify with an error of 8% with a linear model (and even better with neural networks), There is also a linear model that given 4 numbers will compute the 2x2sum of them that is: y=Ax where x is the vector containing the 4 numbers and A = [1 0 1 0;1 1 0 0;1 0 0 1; 0 1 0 1]. This means a succession of two linear models can solve the sum2x2 problems with little trouble. While I'm aware that this work aims at automatically finding the combination of simple models to achieve this task end-to-end, the fact that the solution is a set of 2 consecutive linear models makes it a bit too simple in my humble opinion. 

Overall, I think that this paper proposes a promising extension of TerpreT that is unfortunately not backed by experiments that are convincing enough.

review_title: null

final_score: rating: 2: Strong rejection
review_content: The authors explore the idea of life-long learning in the context of program generation.

The main weakness of this paper is that it mixes a few issues without showing strong results on any of them. The test tasks are about program generation, but these are toy tasks even by the low standards of deep-learning for program generation (except for the MATH task, they are limited to 2x2 grid). Even on MATH, the authors train and discuss generalization from 2-digit expressions -- these are very short, so the conclusiveness of the experiment is unclear. The main point of the paper is supposed to be transfer learning though. Unluckily, the authors do not compare to other transfer learning models (e.g., "Progressive Neural Networks") nor do they test on tasks that were previously used by others. We find that only testing on a newly-created task with a weak baseline is not sufficient for ICLR acceptance.

After clarifying comments from the authors and more experiments (see the discussion above), I'm now convinced that the authors mostly measure overfitting, which in their model is prevented because the model is hand-fitted to the task. While the idea might still be valid and interesting, many harder and much more diverse experiments are needed to verify it. I consider this paper a clear rejection at present.
review_title: Final Review: Fine idea but very basic tasks, weak baselines, and misleading presentation

final_score: rating: null
review_content: 
This is a wonderful paper on program induction, with a very nice framework for combining code with neural networks.

I'd like to ask the authors a simple question, which in no way attempts to diminish the great value of this paper, but rather prompts for clarification.

Can you learn more complex programs such as sorting with weak supervision? (and without curriculum, which amounts to strong supervision of hierarchy). Reed et al learn sorting with only 8 demonstrations, but imitation is of course a stronger form of supervision.

Can the family of tasks that can be learned with weak supervision be easily characterised? Does classical complexity theory offer an answer?
review_title: How far can one push the notion of weak supervision in program induction

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
I think the paper is a bit more solid now and I still stand by my positive review. I do however agree with other reviewers that the tasks are very simple. While NPI is trained with stronger supervision, it is able to learn quicksort perfectly as shown by Dawn Song and colleagues in this conference. Reed et al had already demonstrated it for bubblesort. If the programs are much shorter, it becomes easy to marginalise over latent variables (pointers) and solve the task end to end. The failure to attack much longer combinatorial problems is my main complaint about this paper, because it makes one feel that it is over-claiming.

In relation to the comments concerning NPI,  Reed et al freeze the weights of the core LSTM to then show that an LSTM with fixed weights can continue learning new programs that re-use the existing programs (ie the trained model can create new programs). 

However, despite this criticism, I still think this is an excellent paper, illustrating the power of combining traditional programming with neural networks. It is very promising and I would love to see it appear at ICLR.

===========
This paper makes a valuable contribution to the emerging research area of learning programs from data.

The authors mix their TerpreT framework, which enables them to compile programs with finite integer variables to a (differentiable) TensorFlow graph,  and neural networks for perceiving simple images. This is made possible through the use of simple tapes and arithmetic tasks.  In these arithmetic tasks, two networks are re-used, one for digits and one for arithmetic operations.  This clean setup enables the authors to demonstrate not only the avoidance of catastrophic interference, but in fact some reverse transfer. 

Overall, this is a very elegant and potentially very useful way to combine symbolic programming with neural networks. As a full-fledged tool, it could become very useful. Thus far it has only been demonstrated on very simple examples. It would be nice for instance to see it demonstrated in all the tasks introduced in other approaches to neural programming and induction: sorting, image manipulation, semantic parsing, question answering. Hopefully, the authors will release neural TerpreT to further advance research in this domain.

 
review_title: Review and review update

Meta-review: The reviewers were quite divided on this submission, which proposes a method for lifelong learning in the context of program generation. While a novel idea, the experiments and baselines are simply not clear enough or convincing enough, and the method itself is not clearly conveyed. The paper makes strong claims that are not substantiated with more compelling or challenging experiments. Since this is an interesting, novel idea in a relevant area, however, I think it should be invited as a workshop paper.
Decision: Invite to Workshop Track