Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a method called neural attention memory. It constructs the attention output leveraging a memory matrix. The computational cost is linear to the sequence length (vs. quadratic in the conventional attention mechanism). The method is evaluated for long-range sequence tasks. It is also used to derived variants of LSTM and neural tuning machine. 



strength_and_weaknesses: Strength
Efficient attention is a topic of general interests to the community. The presentation of the proposed method is clear. It is shown that the method can be applied to different settings such as LSTMs and NTMs.

Weakness
1. The novelty in compression with the linear attention work (Katharopoulos et al 2020) is very limited. In fact the mathematical forms are almost the same. The difference is that this work uses "unit vector normalization" so that they do not need to compute the causal masking factor Z. It is unclear to me what benefit, if any, the unit vector normalization provides.
2. In the long-range task evaluation section, the proposed method performs worse than Katharopoulos et al 2020. Also, other linear transformer works (discussed in Section 2.1) should be compared with too. 
3. One more comment: although the proposed approach is linear to the sequence length, the writing process is a sequential operation. In practice, this can be slow on GPUs/TPUs in comparison with conventional attention which can be fully parallelized. It would be nice to discuss the implementation as well as the break even sequence length with efficient GPU or TPU implementations. 
clarity,_quality,_novelty_and_reproducibility: Clarify: The presentation of the problem and proposal is clear. 
Quality: The quality of this work is subpar because it is unclear why the proposed method is better than Katharopoulos et al 2020 and other alternatives.
Novelty: Very limited considering Katharopoulos et al 2020.
Reproducibility: The algorithm seems to be straightforward to implement. 
summary_of_the_review: The proposed method is very limited in its novelty in comparison with Katharopoulos et al 2020. The empirical results do not justify the merits of the method among other alternatives.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a memory-based attention mechanism. It leverages the outer product of the key and query to write into a memory matrix. To retrieve information from the memory, the query does a dot product with the memory matrix. The benefit of this attention mechanism is that the computation complexity is reduced from quadradic to linear. Experiment results show the efficiency of the proposed method. The author further proposes NAM-based LSTM and NTM, both models achieve stronger performance compared to their vanilla version while maintaining a similar computation complexity.
strength_and_weaknesses: Strength:
1. The proposed NAM transformer is fairly efficient compared to the standard transformer model and slightly faster than the linear transformer model.
2. The newly proposed LSAM and NAM TM models expand the memory capacity of the original LSTM and NTM model and achieve strong compositional generalization.

Weakness
1. The NAM is not substantially different from the linear transformer and achieves lower performance compare to the linear transformer while the speedup is similar.
2. The performance of LSAM and NAM TM still falls behind the SOTA models in the SCAN task.
clarity,_quality,_novelty_and_reproducibility: The experiments in section 5.4 lack detail. For example, the SCAN task includes four different splits: simple, length, jump, and turn left. It's unclear on which split the author tested their model. The novelty of this paper is also limited, a similar technique has been proposed by the linear transformer.
summary_of_the_review: Overall the paper proposes a new outer product-based memory mechanism, that has an incremental novelty. The author also leverages the proposed method to extend recurrent neural networks such as LSTM and NTM. The compositional generalization results are interesting, but the detail of the experiments requires further clarification.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes neural attention memory (NAM) as an efficient memory-augmented attention mechanism. The authors propose a well-motivated read and write mechanism that is efficient and convenient for hardware implementations. Then, they show a self-attention mechanism inspired by NAM. The resulting NAM Transformer is comparable to Linear Transformer (direct competitor). The authors also design an LSTM-like and an NTM-like architectures that use NAM as attention mechanisms. The authors demonstrate that the NAM-augmented architectures are superior to direct competitors, such as LSTM, NTM, DCN, UT and TF on simple algorithmic tasks, such as Fibonnaci, Palindrom, Reduce and SCAN. Sometimes the proposed methods generalize very well on OOD tasks.
strength_and_weaknesses: Strengths:

1. Simple and working idea. 

2. Meaningful implementation that can be applied in hardware efficiently, and it could help few-shot learning as well.

Weaknesses:

1. The gains are only marginal compared to the Linear Transformer (see Table 1).

2. It seems to me that the Self-attention NAM model is a bit disconnected from Attention NAM. Namely, currently posed, there are no read/ write operations in SelfAttn_NAM. Could you elaborate a bit further, or potentially test some SelfAttn mechanisms with read / write?

3. It would be nice to see experiments on the domains you suggested trying, e.g. few-shot learning.

4. Minor:

* I think you mean "bigger speadups at image classification" in place of  "bigger speadups at text classification" when discussing Table 1. 
clarity,_quality,_novelty_and_reproducibility: The paper's clarity is great: I managed to follow the details seamlessly. 

The quality is also good: the experiments are well-executed.

Originality gets close to existing ideas (Linear Transformer) but it is nevertheless unique in its implementation and applications on relevant tasks.


summary_of_the_review: The paper is well executed and the constructions are meaningful, so I recommend weakly the paper to be accepted. To improve the paper: it would be nice to experiment with read and write into the self-attention mechanism. Further improvements might come from trying some simple few-shot learning tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes neural attention memory (NAM), an alternative to the standard attention mechanisms, which suffer from well known limitations: their quadratic dependency with respect to the sequence length, and thus their inefficiency when dealing with long sequences, as well as their difficulty to solve tasks that require memory states.

NAM follows the same query-key-value structure of scaled dot-product attention; it first writes a memory matrix by adding outer products of key-value pairs, and then reads it by multiplying the memory matrix with a unit query vector.  NAM attention reduces the computational complexity to linear with respect to the sequence length. 

They perform experiments on long-range arena tasks, comparing a transformer with NAM self-attention with a vanilla transformer and with the Linear Transformer of Katharopoulos et al. (2020). They further experiment with their memory augmented neural network designs (LSAM and NAM-TM) on compositional generalization tasks like number sequence prediction, sequence reduction, and SCAN. 

strength_and_weaknesses: Strengths:

* The paper is well written and is easy to follow. 
* The motivation is very clear from the beginning: trying to address two limitations of current attention mechanisms (inefficiency when dealing with long sequences and stateless design).

Weaknesses:
* The discussion about memory-augmented neural networks (MANN) in $\S2.2$. seems limited. It would be beneficial to discuss the similarities/differences of your proposal and recent works that try to augment the transformer architecture with a memory network (e.g., [1], [2]). The same applies for the experiments in $\S5$. 
* The writing of $\S4.1$ could be improved. In particular, I find the explanation of NAM-based attention a bit confusing and unstructured. Also, I don’t think the connection between $\S3$ and $\S4$ is very clear. 
* Several transformer architectures that try to efficiently model long sequences have been proposed in the last couple of years. I believe that your experimental results would be stronger if you compared NAM to some of these – the linear transformer is just one. 

Minor Comments/Questions:

* I would slightly change the sentence “Attention mechanisms of deep neural networks (Bahdanau et al., 2014; Luong et al., 2015) provide differentiable methods of choosing (attending) one item from a variable-length sequence.”, because attention may be spread/not concentrated on one item only. 
* Typo in “algorithmic patterns that require more that pushdown automata in practice”
* Is something missing in “Note that $V^\top(p_w \odot \mu(K))$ is one of the special cases of constructing the memory matrix by setting the erase probabilities to zeros W R operations.”?
* Typo (*hyperparameters*) in “The implementation details and hypereparameters can be found at the source code in the supplementary materials.” Also, it would be better to have the hyperparameters listed in, e.g., an appendix, and not only as part of the code.
* Can you further comment on “However, NAM-TM is not the best-performing model for the SCAN task.”?
* Typo in “For example, a document tensor $D$ can constructed by sum”
* Typo in “NAM’s cost does not depend the sequence



[1] Compressive Transformers for Long-Range Sequence Modelling, Rae et al., ICLR 2020

[2] $\infty$-former: Infinite Memory Transformer, Martins et al., ACL 2022


clarity,_quality,_novelty_and_reproducibility: The motivation is clear and the two sets of experiments are adequate to show how NAM addresses the limitations of current systems. However, I still think that you could rewrite some parts in order to make the connection between $\S3$ and $\S4$ easier.

The paper is well structured and the idea is interesting. I think, however, that you could still compare your methods to other works (e.g., other long-range arena transformers and more recent MANN models).

To the best of my knowledge, NAM’s formulation is novel. In practice, a transformer with NAM is similar to a Linear Transformer with some differences, highlighted in $\S4.2$ (unit vector normalization instead of ELU kernel function; no need to compute the causal masking factor Z).

Code is provided as supplementary material. Are you planning to release the code to everyone?


summary_of_the_review: The paper is well written and addresses an important problem. The formulation is interesting and might be used for other applications that were not explored in this paper. See the Weakness section for further details on what I think the paper would benefit from.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a memory-based attention mechanism called neural attention memory (NAM), to avoid the quadratic bottleneck of transformers. The proposed strategy is similar to prior work with linear transformers (Katharopoulos et al. 2020): it first writes a memory matrix by adding outer products of key-value pairs, and then reads it by multiplying the memory matrix with a unit query vector. NAM attention reduces the computational complexity to linear with respect to the sequence length. Experiments are shown in long-range arena and on compositional generalization tasks like number sequence prediction, sequence reduction, and SCAN. While the paper addresses a relevant problem and the empirical results look fairly convincing, reviewers pointed out as the main weakness the insufficient novelty with respect to previous methods, in particular the linear transformer, and lack of comparison with other methods. The revised version addresses some of these points, but it is insufficient to alleviate the main concerns. I therefore recommend rejection. I recommend the authors take into account the comments of the reviewers to improve their paper in a future iteration of their work.
justification_for_why_not_higher_score: Written in the meta-review.
Decision: Reject