Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper shows that Transformer Attention approximates Sparse Distributed Memory (SDM) model under given conditions.  It further confirms that those conditions are satisfied on pre-trained GPT2 Transformer models.  In addition to theoretical analysis, some results are provided using various datasets including MNIST.  Furthermore, there is a section on biological interpretation of Attention and SDM.  
main_review: This paper shows that the relationship between SDM and Attention exists because SDM’s read operation, when approximated as an exp sum of exponentials, it actually approximates softmax rule used in Attention.   They use Query-Key Normalized transformer to show the relationship to SDM, and original GPT2 models to generalize the results.

The connection between SDM and Attention was insightful, and nicely presented.  Evaluation was very limited, which his concerning as one of the difficulties of SDM-based models is precisely lack of accuracy when it comes to more complex image-based datasets beyond MNIST.   The idea that SDM/Attention model could be better interpreted through biology is very attractive, but that section in the paper seemed to be quite a stretch, and did not present convincing evidence that this is indeed the case.  

limitations_and_societal_impact: None
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors focus on attention mechanisms in neural networks, and show that under certain conditions the attention mechanism in Transformer networks can approximate a type of associative memory.

main_review: Main merits:
- The work presented is novel and it seems solid.

Main limitations:
- The paper may be organized better, as it is a bit hard to read in its present form.
- A more thorough experimental validation would be useful..
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper shows that Transformer's Attention closely approximates Sparse Distributed Memory (SDM), a biologically plausible associative memory model, under certain conditions on the data normalization. The authors first provide a review of Kanerva’s SDM. Then, they show that Attention approximates SDM theoretically and empirically via the pretrained GPT2 architecture. Here, the intersection between 2 Hamming circles in SDM is approximated by an exponential function of key and query, which forms the softmax function in Attention.  This close relationship between Attention and SDM leads to a hypothesis that Attention is performing the same associative memory operations as the cerebellum. Moreover, there are discussions on related works including Hopfield Network, Memory-augmented Neural Networks and Kanerva Machine. 
main_review: While Attention has become an important mechanism in deep learning, it remains unclear why it works so well. The finding of this paper may provide one explanation for Attention’s success. Prior works have pointed out the relationship between Attention and Hopfield Network, Fast-weight or Associative Memory. SDM has been used as motivations for memory-augmented neural networks.  However, this is the first time that someone have shown the close relationship between Attention and SDM both theoretically and empirically. Key mathematical techniques are the approximation of binomial coefficients using exponentials (equation (4)) and transforming the non-continuous Hamming distance to a continuous version including the cosine similarity (equation (6)). Those techniques, yet simple, are crucial to building a bridge between SDM and Attention.
The paper is well-written and easy to follow. People who are not familiar with the topic can still get the key ideas. Figures are clear and informative.
Questions to the authors:
1.	SDM is well-known for the ability of iterative reading, i.e. to feed the previous output of the read operation into the read operation again to get a better-match pattern until convergence. Does Attention have this ability? If yes, can it be shown theoretically or via experiments?
2.	Section 4 shows that components of a Transformer block are closely related to SDM. However, Transformer encoder and decoder consist of many blocks stacked together. This makes the model more powerful. How can this be explained in terms of SDM?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors showed that the attention mechanism used in Transformers is related to the Sparse Distributed Memory, a classical computational model of associative memory in the brain. 
main_review: Originality
This result bears some similarity with the Ramsauer 2020 paper that connects “modern Hopfield Networks” with Attention. But other than that, it appears original.

Quality
The authors clearly spent a lot of time on this submission. The Supplemental Materials is 55 page long.

Clarity
The description of SDM seems unnecessarily complex and somewhat confusing. I thought I understood SDM when I read Kanerva’s 1992 book, but I was kind of confused after reading the authors’ description. If SDM is almost equivalent to Attention, and Attention can be described in one line (line 122-123), why does it take two pages to describe SDM?

Some description of SDM’s relationship to the brain seems unnecessarily long and not particularly insightful. For example, section 5 SDM’s Mapping to Cerebellum. It’s not clear what this section adds.

Significance
It is not clear what significant insights we can obtain by relating SDM with Transformer Attention. Many models are similar to one another if we make some approximations and assumptions. Attention without softmax is equivalent to a neural network with a single hidden layer and no nonlinearity.

limitations_and_societal_impact: I don’t see clear limitations and potential negative societal impact beyond what’s mentioned above.
review_title: null

Meta-review: This paper shows connections between the attention function and sparse distributed memory. The insights are interesting and could be useful for those who want to understand why it works so well as well as those who want to extend it further. All reviewers generally agree that this is a good paper. Reviewer PZYv has concerns about the clarity of the paper and the significance of the insights. While I agree that the writing could be improved to make it easier to read for general readers (which the authors promised they will address in the version), I believe that the insights from this paper is useful. The authors also promised to add more discussions based on feedback from Reviewer PZYv, as summarized in their author response. I recommend accepting the paper.
Decision: Accept (Poster)