Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: From the title (Extensions ... of the Neural GPU) and chapter 4 (Improvements to the Neural GPU), it sounds like you extend or improve the Neural GPU model. From the text, I understand that you don't modify the Neural GPU model but you use improved training techniques and an improved implementation for memory efficiency. I think you should make that more clear in the title. So, there is no modification to the Neural GPU model?

From Figure 2, it seems that gradient noise could slightly help. Have you used that in your remaining experiments? What exactly have you used in your remaining experiments?

For some cases, you describe the length of train data sequences and test data sequences but I haven't found that information for example for the experiments in Figure 3 and maybe others. How long was it? How much longer was test compared to train?

You state that the NTM number of steps is O(n). But I think that is actually variable, right?

You don't have any comparisons to other related models, like Grid LSTM, NTM, Adaptive Computation Time, etc?

review_title: extensions / improvements to NeuralGPU model, test data length, NTM steps

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overall the paper has the feel of a status update by some of the best researchers in the field. The paper is very clear, the observations are interesting, but the remarks are scattered and don't add up to a quantum of progress in the study of what can be done with the Neural GPU model.

Minor remark on the use of the term RNN in Table 1: I found Table 1 confusing because several of the columns are for models that are technically RNNs, and use of RNNs for e.g. translation and word2vec highlight that RNNs can be characterized in terms of the length of their input sequence, the length of their input, and the sizes (per step) of their input, output, and working memories.

Basic model question: How are inputs presented (each character 1-hot?) and outputs retrieved when there are e.g. 512 “filters” in the model ?  If inputs and outputs are 1-hot encoded, and treated with the same filters as intermediate layers, then the intermediate activation functions should be interpretable as digits, and we should be able to interpret the filters as implementing a reliable e.g. multiplication-with-carry algorithm. Looking at the intermediate values may shed some light on why the usually-working models fail on e.g. the pathological cases identified in Table 3.

The preliminary experiment on input alignment is interesting in two ways: the seeds for effective use of an attentional mechanism are there, but also, it suggests that the model is not presently dealing with general expression evaluation the way a correct algorithm should.

The remarks in the abstract about improving the memory efficiency of Neural GPU seem overblown -- the paragraph at the top of page 6 describes the improvements as using tf.while_loop instead of unrolling the graph, and using swap_memory to use host memory when GPU memory runs short. These both seem like good practice, but not a remarkable improvement to the efficiency of the model, in fact it would likely slow down training and inference when memory does in fact fit in the GPU.

The point about trying many of random seeds to get convergence makes me wonder if the Neural GPU is worth its computational cost at all, when evaluated as means of learning algorithms that are already well understood (e.g. parsing and evaluating S-exprs). Consider spending all of the computational cycles that go into training one of these models (with the multiple seeds) on a traditional search through program space (e.g. sampling lisp programs or something).

The notes on the curriculum strategies employed to get the presented results were interesting to read, as an indication of the lengths to which someone might have to go to train this sort of model, but it does leave this reviewer with the impression that despite the stated extensions of the Neural GPU model it remains unclear how useful it might be to practical problems.
review_title: scattered but interesting remarks don't quite add up to a publishable unit

final_score: rating: null
review_content: -
review_title: -

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper investigates on better training strategies for the Neural GPU models as well as studies the limitations of the model.

Pros:
* Well written.
* Many investigations.
* Available source code.

Cons:
* Misleading title, there is no extension to the Neural GPU model, just to its training strategies.
* No comparisons to similar architectures (e.g. Grid LSTM, NTM, Adaptive Computation Time).
* More experiments on other tasks would be nice, it is only tested on some toy tasks.
* No positive results, only negative results. To really understand the negative results, it would be good to know what is missing to make it work. This has not been studied further.
* Some details remain unclear or missing, e.g. if gradient noise was used in all experiments, or the length of sequences e.g. in Figure 3.
* Misleading number of NTM computation steps. You write O(n) but it is actually variable.

After the results from the paper, the limitations still remain unclear because it is not clear exactly why the model fails. Despite showing some examples which make it fail, it was not studied in more detail why it failed for those examples, and how you could fix the problem.

review_title: interesting paper but requires some more work

final_score: rating: null
review_content: I guess you should write ICRL 2017 there.
review_title: header typo: ICLR 2016

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors investigate the neural GPU model introduced by Kaiser and Sutskever. In section 3 they claim its performance is due to the O(n^2) number of steps it can perform for each example. In the subsequent section they highlight the importance of curriculum training and empirically show that larger models generalize better. In section 5 they construct examples that reveal failure modes. In the last section they compare the performance given different input formats.

The paper is well written. It contains an exhaustive set of experiments which provide insight into the details of training the neural GPU model. It pushes the boundary of algorithms that can be learned further. On the other hand, the paper seems to lack a coherent message. It also fails to provide any insight into the how and why of the observations made (i.e. why curriculum training is essential and why certain failure modes exist).

The introduction contains several statements which should be qualified or explained further. As far as I am aware statistical learning theory does not guarantee that empirical risk minimization is consistent when the number of parameters is larger than the number of examples; the generalization performance depends on the VC dimension of the function space instead. Furthermore, the suggested link between adversarial examples and learning algorithms is tenuous, and references or a further explanation should be provided for the contentious statement that deep neural networks are able to match the performance of any parallel machine learning algorithm.

The authors argue that the neural GPU performs O(n^2) “steps” on each example, which allows it to learn algorithms with super-linear complexity such as multiplication. This analysis seems to overlook the parallel nature of the neural GPU architecture: Both addition and multiplication have O(log n) time complexity when parallelism is used (cf. a carry-lookahead adder and a Wallace tree respectively).

In section 4 the authors show that their larger models generalize better, which they argue is not self-evident. However, since both training and test error decrease it is likely that the smaller models are underfitting, in which case it is not counter-intuitive at all that a larger model would have better generalization error.

It is interesting to see that progressively decreasing the number of terms and increasing the radix of the number system works well as a learning curriculum, although it would be nice to have a stronger intuitive or theoretical justification for the latter.

The final section claims that neural GPUs are cellular automata. Further justification for this statement would be useful, since cellular automata are discrete models, and the equivalence between both models is in no way obvious. The relationship between global operations and changing the input format is circuitous.

In conclusion, the paper provides some useful insights into the neural GPU model, but does not introduce original extensions to the model and does not explain any fundamental limitations. Several statements require stronger substantiation.

Pro:

* Well written
* Exhaustive set of experiments
* Learning algorithms with decimal representation
* Available source code

Cons:

* No coherent hypothesis/premise advanced
* Two or three bold statements without explanation or references
* Some unclarity in experimental details
* Limited novelty and originality factor

Typos: add minus in “chance of carrying k digits is 10^k” (section 5); remove “are” from “the larger models with 512 filters are achieve” (section 4); add “a” in “such model doesn’t generalize” (section 4).
review_title: null

Meta-review: This paper is clearly written, and contains original observations on the properties of the neural GPU model. These observations are an important part of research, and sharing them (and code) will help the field move forward. However, these observations do not quite add up to a coherent story, nor is a particular set of hypotheses explored in depth. So the main problem with this paper is that it doesn't fit the 'one main idea' standard format of papers, making it hard to build on this work.
 
 The other big problem with this paper is the lack of comparison to similar architectures. There is lots of intuition given about why the NGPU should work better than other architectures in some situations, but not much empirical validation of this idea.
Decision: Reject