Title: Lie-Access Neural Turing Machines
Authors: Greg Yang
Alexander Rush
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Supervised Learning

Abstract: 
  External neural memory structures have recently become a popular tool for
  algorithmic deep learning
  (Graves et al. 2014; Weston et al. 2014).  These models
  generally utilize differentiable versions of traditional discrete
  memory-access structures (random access, stacks, tapes) to provide
  the storage necessary for computational tasks.  In
  this work, we argue that these neural memory systems lack specific
  structure important for relative indexing, and propose an
  alternative model, Lie-access memory, that is explicitly designed
  for the neural setting.  In this paradigm, memory is accessed using
  a continuous head in a key-space manifold. The head is moved via Lie
  group actions, such as shifts or rotations, generated by a
  controller, and memory access is performed by linear smoothing in
  key space. We argue that Lie groups provide a natural generalization
  of discrete memory structures, such as Turing machines, as they
  provide inverse and identity operators while maintaining
  differentiability. To experiment with this approach, we implement
  a simplified Lie-access neural Turing machine (LANTM) with
  different Lie groups.  We find that this approach is able to perform
  well on a range of algorithmic tasks.

TL;DR: We generalize Turing machines to the continuous setting using Lie group actions on manifolds.

Review discussion:
You introduce InvNorm as an alternative to the softmax. Maybe I missed it but I didn't really understand the motivation behind this.
You are right we did not really explain this. We’ll expand on this in the next version of the paper.
These two methods can exactly be thought of different kernels for computing the expectation over memories via kernel regression. Softmax is a gaussian kernel and invnorm is an inverse distance kernel.
Behaviorally, in invnorm, the weighting vector becomes concentrated at a single memory i only when the read key k is very close to k_i, the address of memory i.
On the other hand, in softmax, the weighting vector becomes concentrated at memory i as long as k is relatively closer to k_i than to other addresses k_j, j != i.
Thus invnorm encourages the controller to output exact addresses, while softmax is OK with controller emitting loosely approximate addresses. The former has the advantage that, for example in R^2 with the shift group, when trying to learn relative head movement to go from k_i to k_{i+1}, the controller will learn k_{i+1} - k_i. With softmax, the controller may learn something wildly different;
for example, it perhaps learns to retrieve memory i with \tilde k_i := k_i + (k_{i+1} - k_i)/3 and to retrieve memory i+1 with \tilde k_{i+1} = k_{i+1} - (k_{i+1} - k_i)/3, and therefore it learns the head shift \tilde k_{i+1} - \tilde k_i = (k_{i+1} - k_i)/3. If the memories are laid out on a straight line, with the same shift vector between each consecutive memories, then invnorm seems likely to learn the correct head movement, but softmax might not.
I struggle to understand figure 2, despite the length of the caption. Perhaps labelling the images themselves a bit more clearly.
In the introduction in the first paragraph in the last sentence, you mean "these works" instead of "this work"?
Can you make the difference to NTM more clear in the introduction?
Esp, NTM are already differentiable for end-to-end training and have a robust relative indexing scheme. Why is a Lie-Access NTM different in that regard?
In Tape-Based Memory, what is the probability simplex \Delta^2?
How does it compare to Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes?
In chapter 3, the general space \mathcal{K} is continuous?
What metric $d$ do you use and which have you tried?
When you introduce InvNorm, what do you mean with "limit value"?
In section 4.4, what do you mean by "L_2-normalization"?
In Models and Baselines, you have "described the background", which is probably missing a word.
The first model (a), the LSTM encoder-decoder, does it have attention?
>Question: In the introduction in the first paragraph in the last sentence, you mean "these works" instead of "this work"?
We were referring to the collection of works mentioned in the previous sentence, and not our paper. Sorry for the confusion; this will be fixed in the next version of the paper.
>Can you make the difference to NTM more clear in the introduction?
Esp, NTM are already differentiable for end-to-end training and have a robust relative indexing scheme. Why is a Lie-Access NTM different in that regard?
As mentioned in the last sentence of that paragraph in the paper, the set of NTM’s head movement does not compose a group.
In particular it is not true that every head movement has an inverse movement. This is briefly explained in footnote 1.
In short, NTM’s head movement is a Markov kernel, which mixes the probabilities encoded in the NTM’s head. Intuitively, a mixture cannot be “unmixed” by mixing further; formally, it can be shown that a Markov kernel does not have an inverse that is a Markov kernel if it is not the delta kernel (all values concentrated at one point). This problem manifests as the loss of concentration in NTM’s head after shifting repeatedly. This was fixed in an ad hoc way in the NTM paper via “sharpening coefficients” that boosts concentration, but they have to be learned.
>In Tape-Based Memory, what is the probability simplex \Delta^2?
It is the set {(x, y, z) in R^3: x, y, z >= 0, x + y + z = 1}.
>How does it compare to Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes?
The D-NTM paper’s results are in some sense orthogonal to ours. Its contributions were mainly Dynamic Least Recently Used (LRU) addressing, regularization methods, and hard addressing using REINFORCE. In our paper under review here, we argue for a principled way to purpose different manifolds as memory storage and different Lie groups as relative head movement.
In contrast, D-NTM’s memory is still modelled after a traditional memory array and it does not have relative head movement. A priori, there is no reason that ideas from both papers cannot be combined in a coherent way.
>In chapter 3, the general space \mathcal{K} is continuous?
When we were talking about the general criteria for head movement in the first and second paragraphs, \mathcal{K} may be any space satisfying those criteria, which may or may not be "continuous."
For example, the classic Turing machine has \mathcal{K} = Z.
In the last paragraph, when we talked about the differentiability criteria, we argued for \mathcal{K} to be a smooth manifold (to be more technically accurate, a connected smooth manifold with dimension > 0), and thus "continuous".
>What metric $d$ do you use and which have you tried?
In the experiments, $d$ is Euclidean distance (L_2 distance) for both the memory based in R^2 and memory based on the sphere S^2: d(x, y) = \sqrt{\sum_i (x_i - y_i)^2}.
We didn't try other distance functions for R^2, because Euclidean distance seems like the most appropriate choice.
For the sphere, we tried angle distance: d(x, y) = arccos(<x, y>).
This performs very similar to Euclidean distance because the two functions are close to proportional when x and y are close together.
Another choice would be cosine similarity (dot product): d(x, y) = 1 - <x, y>.
But as explained in footnote 5, this is the same as the Euclidean distance.
>When you introduce InvNorm, what do you mean with "limit value"?
It is really a minor technical point.
It means that we define the InvNorm function when q = k_i, which a priori causes both the enumerator and denominator to be infinite, to be the limit value of the function when q approaches k_i.
In particular, if all k_i are distinct, then when q = k_i, InvNorm takes the value (0, ..., 0, 1, 0, ..., 0), where the 1 appears in position i, corresponding to k_i.
>In section 4.4, what do you mean by "L_2-normalization"?
L_2-normalization is the map x |--> x/\|x\|, i.e. divides the coordinates of a point x by the 2-norm \|x\|.
>In Models and Baselines, you have "described the background", which is probably missing a word.
>The first model (a), the LSTM encoder-decoder, does it have attention?
The LSTM encoder-decoder baseline does not have attention.
The RAM baseline would be roughly the same as LSTM with attention.
*** Paper Summary ***
This paper formalizes the properties required for addressing (indexing) memory augmented neural networks as well as how to pair the addressing with read/write operation. It then proposes a framework in which any Lie group as the addressing space. Experiments on algorithmic tasks are reported.
*** Review Summary ***
This paper brings unity and formalism in the requirement for memory addressing while maintaining differentiable memories. Its proposal provide a generic scheme to build addressing mechanisms.
When comparing the proposed approach with key-value networks, the unbounded number of memory cells and the lack of incentive to reuse indexes might reveal impractical.
*** Detailed Review ***
The paper reads well, has appropriate relevance to related work. The unified presentation of memory augmented networks is clear and brings unity to the field. The proposed approach is introduced clearly, is powerful and gives a tool that can be reused after reading the article. I do not appreciate that the growing memory is not mentioned as a drawback. It should be stressed and a discussion on the impact it has on efficiency/scalability is needed.
Thank you for reading and reviewing our paper.
> I do not appreciate that the growing memory is not mentioned as a drawback. It should be stressed and a discussion on the impact it has on efficiency/scalability is needed.
Thanks for the comment. We agree this is an issue with key/value type approaches. We will discuss this more explicitly and mention possible future corrections.
The paper proposes a new memory access scheme based on Lie group actions for NTMs.
Pros:
* Well written
* Novel addressing scheme as an extension to NTM.
* Seems to work slightly better than normal NTMs.
* Some interesting theory about the novel addressing scheme based on Lie groups.
Cons:
* In the results, the LANTM only seems to be slightly better than the normal NTM.
* The result tables are a bit confusing.
* No source code available.
* The difference to the properties of normal NTM doesn't become too clear. Esp it is said that LANTM are better than NTM because they are differentiable end-to-end and provide a robust relative indexing scheme but NTM are also differentiable end-to-end and also provide a robust indexing scheme.
* It is said that the head is discrete in NTM but actually it is in space R^n, i.e. it is already continuous. It doesn't become clear what is meant here.
* No tests on real-world tasks, only some toy tasks.
* No comparisons to some of the other NTM extensions such as D-NTM or Sparse Access Memory (SAM) (https://arxiv.org/abs/1610.09027).
Although the motivations of other NTM extensions might be different, such comparisons still would have been interesting.
Thanks for reviewing our paper.
> In the results, the LANTM only seems to be slightly better than the normal NTM.
We disagree with this interpretation of the results. In most tasks the best LANTM gets almost 100% coarse on generalization, but the corresponding simplified NTM never gets more than 50% coarse on generalization.
> The result tables are a bit confusing.
Can you clarify this comment? We closely followed past work in this area when presenting the results.
> No source code available.
We plan on open sourcing the code and will include a link in the next version.
> The difference to the properties of normal NTM doesn't become too clear. Esp it is said that LANTM are better than NTM because they are differentiable end-to-end and provide a robust relative indexing scheme but NTM are also differentiable end-to-end and also provide a robust indexing scheme.
Let us reiterate the key motivation of the paper: the set of NTM’s head movement does *not* compose a group.
In particular it is not true that every head movement has an inverse movement. This is briefly explained in footnote 1.
In short, NTM’s head movement is a Markov kernel, which mixes the probabilities encoded in the NTM’s head. Intuitively, a mixture cannot be “unmixed” by mixing further; formally, it can be shown that a Markov kernel does not have an inverse that is a Markov kernel if it is not the delta kernel (all values concentrated at one point). This problem manifests as the loss of concentration in NTM’s head after shifting repeatedly. This was fixed in an ad hoc way in the NTM paper via “sharpening coefficients” that boosts concentration, but they have to be learned.
> It is said that the head is discrete in NTM but actually it is in space R^n, i.e. it is already continuous. It doesn't become clear what is meant here.
We agree that NTM is continuous. Our point is that it does not form a group.
> No tests on real-world tasks, only some toy tasks.
We agree that the tasks used are mainly synthetic, but this is true on most work studying NTMs. Our tasks capture essential aspects of algorithmic learning, like for loops (repeat copy), sorting (priority sort), and arithmetic. They are nontrivial, as demonstrated by the gap between baseline performances and LANTM performances.
> No comparisons to some of the other NTM extensions such as D-NTM or Sparse Access Memory (SAM) (https://arxiv.org/abs/1610.09027).
Although the motivations of other NTM extensions might be different, such comparisons still would have been interesting.
These are very interesting papers but they target aspects of the NTM that are orthogonal to the topology of addressing schemes.
For example, the contributions of the D-NTM paper were mainly Dynamic Least Recently Used (LRU) addressing, regularization methods, and hard addressing using REINFORCE. In our paper under review here, we argue for a principled way to purpose different manifolds as memory storage and different Lie groups as relative head movement.
In contrast, D-NTM’s memory is still modeled after a traditional memory array and it does not have relative head movement. A priori, there is no reason that ideas from both papers cannot be combined in a coherent way.
The Neural Turing Machine and related “external memory models” have demonstrated an ability to learn algorithmic solutions by utilizing differentiable analogues of conventional memory structures.
In particular, the NTM, DNC and other approaches provide mechanisms for shifting a memory access head to linked memories from the current read position.
The NTM, which is the most relevant to this work, uses a differentiable version of a Turing machine tape. The controller outputs a kernel which “softly” shifts the head, allowing the machine to read and write sequences. Since this soft shift typically “smears” the focus of the head, the controller also outputs a sharpening parameter which compensates by refocusing the distribution.
The premise of this work is to notice that while the NTM emulates a differentiable version of a Turing tape, there is no particular reason that one is constrained to follow the topology of a Turing tape.
Instead they propose memory stored at a set of points on a manifold and shift actions which form a Lie group. In this way, memory points can have have different relationships to one another, rather than being constrained to Z.
This is mathematically elegant and here they empirically test models with the shift group R^2 acting on R^2 and the rotation group acting on a sphere.
Overall, the paper is well communicated and a novel idea.
The primary limitation of this paper is its limited impact.
While this approach is certainly mathematically elegant, even likely beneficial for some specific problems where the problem structure matches the group structure, it is not clear that this significantly contributes to building models capable of more general program learning.
Instead, it is likely to make an already complex and slow model such as the NTM even slower. In general, it would seem memory topology is problem specific and should therefore be learned rather than specified.
The baseline used for comparison is a very simple model, which does not even having the sharpening (the NTM approach to solving the problem of head distributions becoming ‘smeared’). There is also no comparison with the successor to the NTM, the DNC, which provides a more general approach to linking memories based on prior memory accesses.
Minor issues:
Footnote on page 3 is misleading regarding the DNC.
While the linkage matrix explicitly excludes the identity, the controller can keep the head in the same position by gating the following of the link matrix.
Figures on page 8 are difficult to follow.
Thank you for reading and reviewing our paper.
> The primary limitation of this paper is its limited impact. [...I]t is not clear that this significantly contributes to building models capable of more general program learning.
Instead, it is likely to make an already complex and slow model such as the NTM even slower.
What leads you to think that Lie-Access is complex or slower? We feel that this model is simpler than NTM/DNC and contributes to program learning by providing a more natural interpretation of memory than soft versions of discrete memory. Our model requires only simple lookups in R^2 and could certainly be optimized to run fast.
For example, all the advances made in the recent paper "Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes" by Rae et al. could be applied to Lie-Access Memory.
> In general, it would seem memory topology is problem specific and should therefore be learned rather than specified.
We feel this criticism should apply as well to NTM and DNC. They specify a rigid memory system modelled after discrete memory arrays. By trying different memories topologies we were simply experimenting with types of access and not trying to advocate problem specific topology. Likely an R^n key space would be sufficient for most problems.
> The baseline used for comparison is a very simple model, which does not even having the sharpening (the NTM approach to solving the problem of head distributions becoming ‘smeared’). There is also no comparison with the successor to the NTM, the DNC, which provides a more general approach to linking memories based on prior memory accesses.
We will add sharpening to the simplified NTM in our experiments. We had hoped to compare to DNC, but were not able to reimplement it. There are now reimplementations though that we will compare to.
Thanks for your response. I think that if you are able to add sharpening to the baseline and compare with the DNC this would definitely be more compelling, in particular, it would be helpful to know how much empirically driven approaches like "sharpening" help.
If you had space, a short analysis/ comment on computational impact of your approaches would also be helpful.
We have finished results adding sharpening to the RAM/tape model. Weight sharpening only confers small advantage over vanilla on the copy, bigram flip, and double tasks, but deteriorates performance on all other tasks. These results will appear in the final version of the paper.
Since there is neither an official nor unofficial implementation replicating LSTM-controlled DNC, and implementing DNC from scratch requires more time than is available, we won’t be able to compare DNC directly with LANTM. We hope to add this in a future version.
The paper introduces a novel memory mechanism for NTMs based on differentiable Lie groups.
This allows to place memory elements as points on a manifold, while still allowing training with backpropagation.
It's a more general version of the NTM memory, and possibly allows for training a more efficient addressing schemes.
Pros:
- novel and interesting idea for memory access
- nicely written
Cons:
- need to manually specify the Lie group to use (it would be better if network could learn the best way of accessing memory)
- not clear if this really works better than standard NTM (compared only to simplified version)
- not clear if this is useful in practice (no comparison on real tasks)
Thank you for taking the time to read and review our paper.
> need to manually specify the Lie group to use (it would be better if network could learn the best way of accessing memory)
We see the choice of Lie Group as analogous to the choice of memory layout in discrete models. In a perfect world that could be learned, but we think something like an R^n key space is a fine general-layout.
> not clear if this really works better than standard NTM (compared only to simplified version)
We will implement some of the tricks in a standard NTM such as sharpening.
However, we see it as a benefit that these are not necessary in Lie-access.
> not clear if this is useful in practice (no comparison on real tasks)
We realize that the tasks used are mainly synthetic, but this is true for most work studying NTMs. Our tasks capture essential aspects of algorithmic learning, like for loops (repeat copy), sorting (priority sort), and arithmetic. They are nontrivial, as demonstrated by the gap between baseline performances and LANTM performances.
We have finished results adding sharpening to the RAM/tape model. Weight sharpening only confers small advantage over vanilla on the copy, bigram flip, and double tasks, but deteriorates performance on all other tasks. These results will appear in the final version of the paper.

Meta-review:
role: PC
data: comment: The paper presents a Lie-(group) access neural turing machine (LANTM) architecture, and demonstrates it's utility on several problems. 
 
 Pros:
 Reviewers agree that this is an interesting and clearly-presented idea.
 Overall, the paper is clearly written and presents original ideas.
 It is likely to inspire further work into more effective generalizations of NTMs.
 
 Cons:
 The true impact and capabilities of these architectures are not yet clear, although it is argued that the same can be said for NTMs. 
 
 The paper has been revised to address some NTM features (sharpening) that were not included in the original version.
 The purpose and precise definition of the invNorm have also been fixed.

Decision: Accept (Poster)