Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents an approach called a neural computer, which has a Differential Neural Computer (DNC) at its core that is optimised with an evolutionary strategy. In addition to the typical DNC architecture, the system proposed in this paper has different modules that transfer different domain representations into the same representation, which allows the system to generalise to different and unseen tasks.

The paper is interesting and well written but I found that the contributions of this paper could be made more clear. 

First, the idea of evolving a Neural Turing machine was first proposed in Greve et al. 2016, which the authors cite, but only in passing in the conclusion. Greve et al. paper introduced the idea of hard attention mechanisms in an NTM through evolution and the benefits of having a memory structure that does not have to be differentiable. However, if the reader of this paper is not careful, they would miss this fact. I, therefore, suggest featuring Greve’s paper more prominently and highlighting the differences/similarities to the current paper earlier in the introduction.

Second, the idea of learned modules to allow the approach to work across different domains is interesting, but I’m wondering how novel it really is? Isn’t this basically just like feature engineering and changing the underlying representation, something that we have been doing for a long time? Also, the domains that this approach can be applied to seem potentially limited in that the two problems have to already be very similar; In fact, I probably wouldn’t call them different tasks but the same task with a different visual representation. 

I also had a question about the DNC training. Is the DNC version also trained with NES? It would be good to know how much of the difference between the proposed approach and the DNC is because of the training method (NES vs SGD) or other factors.  

Once the points raised above and the specific contributions of this paper are made more clear, I would suggest accepting it. 

####After rebuttal####
The authors' response and the revised paper address most of my concerns now. Since I do believe the approach could be more novel, I'm keeping my weak accept, but do think it is very interesting work. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper introduces a neural controller architecture for learning abstract algorithmic solutions to search and planning problems. By combining abstract and domain-specific components, the model is able to mimic two classical algorithms quite closely across several domains. The precision of the learning is very high; verified by generalization to substantially larger problem sizes and different domains. One notable conclusion is that Evolutionary Strategies is able to learn algorithmic solutions whose precision is on par with deterministic algorithms. The method of triggering learning based on curriculum level performance is a notable feature that nicely couples generalization progress with learning, and yields insightful learning curves.

Although the paper shows that learning abstract algorithmic solutions is possible, it is not clear that the framework is able to learn such solutions when the training signals have not already been broken down into their smallest constituent parts. In other words, it is not clear what this framework could be used for, since it appears the experimenter must already possess a complete specification of the target algorithm and breakdown of the domain. Is there a setting where this framework could be used to learn something the experimenter is not already aware of? Or is the main point that it is technically possible to get an NN to learn this behavior?

Although it is clear that models are achieved that satisfy R1-R3, it is not clear exactly what problem formulation is being considered. It would be very helpful if the paper included a formal problem definition so that the purpose of each framework component and differences w.r.t. prior work are clear. 

Similarly, the motivation for each of the data dependent modules is not clear. Is there something fundamental about this particular decomposition into modules? Or are these just the modules that were necessary given the specifics of the algorithms that were learned in experiments? How does this framework generalize to other kinds of algorithms?

Are the comparison methods (DNC and stackRNN) unable to generalize to larger problem sizes? Including the full comparisons on generalization would give a more complete picture. Similarly, the figures are missing the comparisons for Learning to Plan for DNC and stackRNN.

Is the comparison w.r.t. training time in Figure 3c fair, since the proposed framework pretrains the submodules?

Is there a fundamental problem of DNC being addressed here? E.g., are there some critical types of submodules where making them differentiable is not an option?

Is the algorithm limited to cases where the number of actions at each state is equal? I.e., could it be applied to algorithms like shortest path in the DNC paper?

Finally, the last line talks about intriguing applications to the real world, but the running example in the paper is sorting. Is there some hypothetical but concrete example of how this framework could help in the real world, and do something better than a hard-coded classical algorithm? Or discover a new algorithm?

----------

After the rebuttal and discussion, I am convinced the paper is a useful contribution, and have increased my rating. However, I still think the presentation can be much improved to improve the clarity of the contribution. I would hope to see the following addressed in the final version:

1. More detailed and precise discussion of how the approach relates to prior work. As is, this discussion is informal and scattered throughout the paper. Enumerating the distinctions in one place would make the contribution much more clear.

2. The above would also help make the explicit contribution more clear. E.g., the "Contribution" paragraph currently does not contain the fact that the machine has only been applied to "planning" problems. This should be included to avoid making the contribution seem overly general.

3. More formal description of what each module does. Right now, they are described informally. Actually seeing the equations of what each computes would make it easier to understand how and why they all fit together.

I still see the main contribution as "It is technically possible to train the abstract controller of a neural computer for planning using NES, so that R1-R3 are satisfied." Ideally, the above clarifications would make it more clear that this is the main contribution, or that there are some other key contributions beyond this.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes modifications and modular extensions to the differential neural computer (DNC). The approach is nicely modular, decoupling the data modules from algorithmic modules. This enables the authors to pretrain the data modules with supervised learning and to train the small algorithmic modules with neural evolution strategies (NES). NES is a global optimization method (which may be understood as policy gradients where the parameters of the neural policy are the actions) and consequently this enables the authors to use discrete selection mechanisms instead of the soft attention mechanisms of DNC.

The authors correctly argue for generalization to different and unseen tasks (R1), independence of data representation (R2) and independence of task environment (R3). These are important points, and the authors are able to achieve these by using a modular approach where the interface modules are pretrained.

While what can be achieved with modularity is important, the modules themselves are rather simple and leave open the question of scalability. NES clearly does not scale to higher dimensional spaces, and the complexity or real say vision modules is the problem for many people --- one does not just learn the vision module and freeze it. 

The idea that modularity can be used to attain greater generality and domain independence has already been explored at ICLR to some extent. In fact, some authors have shown theoretically provable generalisation via verification approaches: https://openreview.net/forum?id=BkbY4psgg

In summary, this paper demonstrated the advantages of modular neural systems, but fails to address the important issue of making sure the modules scale to real problems.



review_title: Official Blind Review #1

Meta-review: The authors present a method that optimizes a differentiable neural computer with evolutionary search, and which can transfer abstract strategies to novel problems.  The reviewers all agreed that the approach is interesting, though were concerned about the magnitude of the contribution / novelty compared to existing work, clarity of contributions, impact of pretraining, and simplicity of examples.  While the reviewers felt that the authors resolved the many of their concerns in the rebuttal, there was remaining concern about the significance of the contribution.  Thus, I recommend this paper for rejection at this time.
Decision: Reject