Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Having just been introduced to TerpreT by this paper, it appears to me as if this paper might be pandering to the ICLR community just a bit by failing to mention your earlier claims (in "TerpreT: A Probabilistic Programming Language for Program Induction") that "the overwhelming trend in the experiments is that the techniques from the programming languages community outperform the machine learning approaches by a significant margin," where, specifically, the gradient-based optimization in this paper is precisely this disparaged machine learning approach.  So, what's actually new here ("we developed our models in TerpreT which hides [] technicalities") and which is the reason you're pushing gradient descent learning here: a) because it's ICLR or b) because by taking into account the nuggets of advice given in the paper you actually find that the gradient methods now outperform the programming languages techniques previously favored?
review_title: Just an application of TerpreT?

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors talk about design choice recommendations for performing program induction via gradient descent, basically advocating reasonable programming language practice (immutable data, higher-order language constructs, etc.).  

As mentioned in the comments I feel fairly strongly that this is a marginal at best contribution beyond TerpreT, an already published system with extensive experimentation and theoretical grounding.  To be clear I think the TerpreT paper deserves a large amount of attention.  It is truly inspiring.

This paper contradicts one of the key findings in the original paper but doesn't provide convincing evidence that gradient-based evaluators for TerpreT are superior or even, frankly, appropriate for program induction.   This is uncomfortable for me and makes me wonder why gradient-based methods weren't more carefully vetted in the first place or why more extensive comparisons to already implemented alternatives weren't included in this paper.  

My opinion: if we want to give the original TerpreT paper more attention, which I think it deserves, then this paper is above threshold.  On the other hand it's basically unreadable, actually contradicts its mother-paper in not well-defended ways, and is irreproducible without the same so I think, unfortunately, it's below threshold.  
review_title: very interesting but probably too derivative from earlier already published work

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents small but important modifications which can be made to differentiable programs to improve learning on them. Overall these modifications seem to substantially improve convergence of the optimization problems involved in learning programs by gradient descent. That said, the set of programs which can be learned is still small, and unlikely to be directly useful. 
review_title: Weak accept

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a set of recommendations for the design of differentiable programming languages, based on what made gradient descent more successful in experiments.

I must say i’m no expert in program induction. While i understand there is value in exploring what the paper set out to explore -- making program learning easier -- i did not find the paper too engaging. First everything is built on top of Terpret, which isn’t yet publicly available. Also most of the discussion is very detailed on the programming language side and less so on the learning side. It is conceivable that it would be best received on a programming language conference. A comparison with alternatives not generating code would be valuable in my opinion, to motivate for the overall setup.

Pros: 
Useful, well executed, novel study.
Cons:
Low on learning-specific contributions, more into domain-related constraints. Not sure a great fit to ICLR.

review_title: Not sure if very interesting to the ICLR community

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper discusses a range of modelling choices for designing differentiable programming languages. Authors propose 4 recommendations that are then tested on a set of 13 algorithmic tasks for lists, such as "length of the list", "return k-th element from the list", etc. The solutions are learnt from input/output example pairs (5 for training, 25 for test).

The main difference between this work and differentiable architectures, like NTM, Neural GPU, NRAM, etc. is the fact that here the authors aim at automatically producing code that solves the given task.

My main concern are experiments - it would be nice to see a comparison to some of the neural networks mentioned in related work. Also, it would be useful to see how this model is doing on typical problems used by mentioned neural architectures (problems such as "sorting", "merging", "adding"). I'm wondering how this is going to generalize to other types of programs that can't be solved with prefix-loop-suffix structure.

It is also concerning that although  1) the tasks are simple, 2) the structure of the solution is very restricted and 3) model is using extensions doing most of the work, the proposed model still fails to find solutions (example: A+L model that has “loop” fails to solve “list length” task in 84% of the runs).


Pro:
- generates code rather than black-box neural architecture
- nice that it can learn from very few examples

Cons:
- weak results, works only for very simple tasks, missing comparison to neural architectures
review_title: review

final_score: rating: null
review_content: It would be interesting to know how well one of the neural network models does on the evaluated tasks, as additional baseline besides lambda squared. While they do not produce source code they can still probably solve the tasks, but to which extent ?
review_title: Comparison with neural network models

final_score: rating: null
review_content: It'd be interesting to see a comparison between the memory model proposed in this paper and the one used in "Hybrid computing using a neural network with dynamic external memory". More specifically, the way allocation and releasing works in Graves et al 16 seems more robust than what is done here.
review_title: Comparison with the memory model of Graves et al 16

final_score: rating: 7: Good paper, accept
review_content: This paper presents design decisions of TerpreT [1] and experiments about learning simple loop programs and list manipulation tasks. The TerpreT line of work (is one of those which) bridges the gap between the programming languages (PL) and machine learning (ML) communities. Contrasted to the recent interest of the ML community for program induction, the focus here is on using the design of the programming language to reduce the search space. Namely, here, they used the structure of the control flow (if-then-else, foreach, zipWithi, and foldli "templates"), immutable data (no reuse of a "neural" memory), and types (they tried penalizing ill-typedness, and restricting the search only to well-typed programs, which works better). My bird eye view would be that this stands in between "make everything continuous and perform gradient descent" (ML) and "discretize all the things and perform structured and heuristics-guided combinatorial search" (PL).

I liked that they have a relevant baseline (\lambda^2), but I wished that they also included a fully neural network program synthesis baseline. Admittedly, it would not succeed except on the simplest tasks, but I think some of their experimental tasks are simple enough for "non-generating code " NNs to succeed on.

I wished that TerpreT was available, and the code to reproduce these experiments too.

I wonder if/how the (otherwise very interesting!) recommendations for the design of programming languages to perform gradient descent based-inductive programming would hold/perform on harder task than these loops. Even though these tasks are already interesting and challenging, I wonder how much of these tasks biased the search for good subset of constraints (e.g. those for structuring the control flow).

Overall, I think that the paper is good enough to appear at ICLR, but I am no expert in program induction / synthesis.

Writing:
 - The paper is at times hard to follow. For instance, the naming scheme of the model variants could be summarized in a table (with boolean information about the features it embeds).
 - Introduction: "basis modern computing" -> of
 - Page 3, training objective: "minimize the cross-entropy between the distribution in the output register r_R^{(T)} and a point distribution with all probability mass on the correct output value" -> if you want to cater to the ML community at large, I think that it is better to say that you treat the output of r_R^{(T)} as a classification problem with the correct output value (you can give details and say exactly which type of criterion/loss, cross-entropy, you use).


[1] "TerpreT: A Probabilistic Programming Language for Program Induction", Gaunt et al. 2016
review_title: Review

Meta-review: Quality, Clarity: There is no consensus on this, with the readers having varying backgrounds, and one reviewer commenting that they found it to be unreadable. 
 
 Originality, Significance:
  The reviews are mixed on this, with the high score (7) acknowledging a lack of expertise on program induction.
 The paper is based on the published TerpreT system, and some think that it marginal and contradictory with respect to the TerpreT paper. In the rebuttal, point (3) from the authors points to the need to better understand gradient-based program search, even if it is not always better. This leaves me torn about a decision on this paper, although currently it does not have strong support from the most knowledgeable reviewers.
 That said, due to the originality of this work, the PCs are inclined to invite this work to be presented as a workshop contribution.
Decision: Invite to Workshop Track