Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: I struggle to understand figure 2, despite the length of the caption. Perhaps labelling the images themselves a bit more clearly.
review_title: Figure 2 difficult to make sense of

final_score: rating: null
review_content: You introduce InvNorm as an alternative to the softmax. Maybe I missed it but I didn't really understand the motivation behind this.
review_title: Motivation for this InvNorm

final_score: rating: 7: Good paper, accept
review_content: The Neural Turing Machine and related “external memory models” have demonstrated an ability to learn algorithmic solutions by utilizing differentiable analogues of conventional memory structures. In particular, the NTM, DNC and other approaches provide mechanisms for shifting a memory access head to linked memories from the current read position.

The NTM, which is the most relevant to this work, uses a differentiable version of a Turing machine tape. The controller outputs a kernel which “softly” shifts the head, allowing the machine to read and write sequences. Since this soft shift typically “smears” the focus of the head, the controller also outputs a sharpening parameter which compensates by refocusing the distribution.

The premise of this work is to notice that while the NTM emulates a differentiable version of a Turing tape, there is no particular reason that one is constrained to follow the topology of a Turing tape. Instead they propose memory stored at a set of points on a manifold and shift actions which form a Lie group. In this way, memory points can have have different relationships to one another, rather than being constrained to Z.

This is mathematically elegant and here they empirically test models with the shift group R^2 acting on R^2 and the rotation group acting on a sphere.

Overall, the paper is well communicated and a novel idea.

The primary limitation of this paper is its limited impact. While this approach is certainly mathematically elegant, even likely beneficial for some specific problems where the problem structure matches the group structure, it is not clear that this significantly contributes to building models capable of more general program learning. Instead, it is likely to make an already complex and slow model such as the NTM even slower. In general, it would seem memory topology is problem specific and should therefore be learned rather than specified.

The baseline used for comparison is a very simple model, which does not even having the sharpening (the NTM approach to solving the problem of head distributions becoming ‘smeared’). There is also no comparison with the successor to the NTM, the DNC, which provides a more general approach to linking memories based on prior memory accesses.

Minor issues:
Footnote on page 3 is misleading regarding the DNC. While the linkage matrix explicitly excludes the identity, the controller can keep the head in the same position by gating the following of the link matrix.
Figures on page 8 are difficult to follow.

review_title: mathematically elegant, limited impact

final_score: rating: null
review_content: In the introduction in the first paragraph in the last sentence, you mean "these works" instead of "this work"?

Can you make the difference to NTM more clear in the introduction?
Esp, NTM are already differentiable for end-to-end training and have a robust relative indexing scheme. Why is a Lie-Access NTM different in that regard?

In Tape-Based Memory, what is the probability simplex \Delta^2?

How does it compare to Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes?

In chapter 3, the general space \mathcal{K} is continuous?

What metric $d$ do you use and which have you tried?

When you introduce InvNorm, what do you mean with "limit value"?

In section 4.4, what do you mean by "L_2-normalization"?

In Models and Baselines, you have "described the background", which is probably missing a word.
The first model (a), the LSTM encoder-decoder, does it have attention?

review_title: difference to NTM or DNTM, details

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: *** Paper Summary ***

This paper formalizes the properties required for addressing (indexing) memory augmented neural networks as well as how to pair the addressing with read/write operation. It then proposes a framework in which any Lie group as the addressing space. Experiments on algorithmic tasks are reported.

*** Review Summary ***

This paper brings unity and formalism in the requirement for memory addressing while maintaining differentiable memories. Its proposal provide a generic scheme to build addressing mechanisms. When comparing the proposed approach with key-value networks, the unbounded number of memory cells and the lack of incentive to reuse indexes might reveal impractical. 

*** Detailed Review ***

The paper reads well, has appropriate relevance to related work. The unified presentation of memory augmented networks is clear and brings unity to the field. The proposed approach is introduced clearly, is powerful and gives a tool that can be reused after reading the article. I do not appreciate that the growing memory is not mentioned as a drawback. It should be stressed and a discussion on the impact it has on efficiency/scalability is needed. 
review_title: This paper brings unity and formalism in the requirement for memory addressing while maintaining differentiable memories. Its proposal provide a generic scheme to build addressing mechanisms. When comparing the proposed approach with key-value networks, the unbounded number of memory cells and the lack of incentive to reuse indexes might reveal impractical. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces a novel memory mechanism for NTMs based on differentiable Lie groups. 
This allows to place memory elements as points on a manifold, while still allowing training with backpropagation.
It's a more general version of the NTM memory, and possibly allows for training a more efficient addressing schemes.

Pros:
- novel and interesting idea for memory access
- nicely written
 
Cons:
- need to manually specify the Lie group to use (it would be better if network could learn the best way of accessing memory)                                 
- not clear if this really works better than standard NTM (compared only to simplified version)
- not clear if this is useful in practice (no comparison on real tasks)

review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new memory access scheme based on Lie group actions for NTMs.

Pros:
* Well written
* Novel addressing scheme as an extension to NTM.
* Seems to work slightly better than normal NTMs.
* Some interesting theory about the novel addressing scheme based on Lie groups.

Cons:
* In the results, the LANTM only seems to be slightly better than the normal NTM.
* The result tables are a bit confusing.
* No source code available.
* The difference to the properties of normal NTM doesn't become too clear. Esp it is said that LANTM are better than NTM because they are differentiable end-to-end and provide a robust relative indexing scheme but NTM are also differentiable end-to-end and also provide a robust indexing scheme.
* It is said that the head is discrete in NTM but actually it is in space R^n, i.e. it is already continuous. It doesn't become clear what is meant here.
* No tests on real-world tasks, only some toy tasks.
* No comparisons to some of the other NTM extensions such as D-NTM or Sparse Access Memory (SAM) (https://arxiv.org/abs/1610.09027). Although the motivations of other NTM extensions might be different, such comparisons still would have been interesting.

review_title: interesting new 

Meta-review: The paper presents a Lie-(group) access neural turing machine (LANTM) architecture, and demonstrates it's utility on several problems. 
 
 Pros:
 Reviewers agree that this is an interesting and clearly-presented idea.
 Overall, the paper is clearly written and presents original ideas.
 It is likely to inspire further work into more effective generalizations of NTMs.
 
 Cons:
 The true impact and capabilities of these architectures are not yet clear, although it is argued that the same can be said for NTMs. 
 
 The paper has been revised to address some NTM features (sharpening) that were not included in the original version.
 The purpose and precise definition of the invNorm have also been fixed.
Decision: Accept (Poster)