Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new memory augmented neural network (MANN) model called recurrent entity network (EntNet). EntNet can be considered as a bank of RNNs with gating mechanism to update the hidden states of the RNNs and the hidden states act like the memory slots. The model is very much relevant to NTM style architectures. It is known that training the controller in NTM to read/write from memory slots is challenging. EntNet cleverly pushes the complexity of the controller to individual memory slots. It is as if each slot has a controller and they all act in a distributed manner.

Authors report strong results in bAbI tasks where the model achieves state of the art performance. Synthetic world experiments justify that the model learns to capture the world dynamics. However it is not clear if this will be scalable to complex real tasks. EntNet also achieves reasonable performance with one-shot reading of the passage in CBT task.

I see EntNet as a generalization of RNNs and has some advantage over NTMs when it comes to training complexity. This is definitely a good contribution to the conference. I see that EntNet can have several other applications in future.

Authors have provided convincing answers to my pre-review questions.

Few more questions:
1. Do you fix the size of the f vector set in equation (1)? If so, to what size in all the experiments?
2. There are so many training details in the paper which makes it difficult to reproduce the results. Can the authors release the source code to reproduce all the results in the paper? I am willing to increase my rating if authors can release the code.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: The work proposes a variant of a recurrent neural network that can selectively update a fixed number of multiple memory slots to update entity states.

The architecture is well motivated, especially with the motivating example, and the operation is shown to validate the intuition as shown in visualizations.

Experimental results, datasets and the baselines used are sufficient to quantitatively show the strength of the proposed architecture.

A limitation is failing to (explicitly) generalize to unseen entities, however this is not a trivial problem on its own and the authors have addressed to this issue and proposed several ideas as workarounds.

I consider the work as a good conference contribution.
review_title: null

final_score: rating: null
review_content: 1. Error values in tables 1 and 2 range from 0 to 1 or 0 to 100? I can guess that it is from 0 to 1, but good to make it explicit in the caption.
2. You say that you report NTM results in Table 3, but it is missing.
3. Can you please modify table 4 and make the story part and key/NN part two separate figures or tables? My initial impression from the table was that for each line in the story you have a key/NN. It is quite confusing.
4. For CBT, authors say that setting U=V=0 and W=I helps. What is the performance of the general EntNet without these changes? Is it too worse? I think it is good to report those numbers as well.
5. I think EntNet+BoW results should go to Table 3 (and not to appendix). In table 3, all the models are not exactly comparable since different models use different encoder representations. So authors should report the performance of EntNet with proposed encoder representation and EntNet with BoW encoder representation in the main table itself.
6. How much time does it take to train EntNet on babi tasks? 10 runs with different initialization for all tasks seems to be too much for me. It is worth investigating and solving the variance problem than simply increasing the computation.
review_title: Few questions

final_score: rating: 7: Good paper, accept
review_content: The paper proposed a multi-memory mechanism that memorizes different information into different components/entities. It could be considered as a mixture model in RNN. This is a very interesting model and result is convincing.

A limitation is that we do not know how to generalize to some unseen entities and how to visualize what entities the model learned.
review_title: null

final_score: rating: null
review_content: I really enjoyed the visualization in Table 4. Have you checked key vectors in the case where you do not explicitly tie them to see if there is anything interpretable still, e.g. by looking at which s_t activates which w_j? Or is it mostly distributed and does not focus to a single concept/topic/entity?
review_title: Interpretability

final_score: rating: null
review_content: 1. The paper mentioned it could use POS tagging to define entities for finding entities. Have you tried it for some experiments/cases? 
2. If the entities are extracted from data, would there be different number of entities for a testing data? Then the length of your hidden state h_j would be varied based on the data. 
review_title: Question about entity

Meta-review: Reviewers found this work to be a "well-motivated", "good contribution", and "clever". The idea was clearly conveyed and reviewers were convinced that the approach was simpler than others like NTMs. Experiments are sufficient, and the work will likely be used in the future. 
 
 Pros:
 - Well-explained and expected to be widely implemented
 - Experimental results convincing on the tasks.
 
 Cons:
 - Several questions about "generalization to some unseen entities". 
 - Reliance on synthetic tasks unclear if "scalable to complex real tasks"
 - Training process seems quite complicated (although again simpler that NTMs)
Decision: Accept (Poster)