Attention Approximates Sparse Distributed Memory
Authors: Trenton Bricken
Cengiz Pehlevan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformer
Attention
Sparse Distributed Memory
Associative Memory
Cerebellum

Abstract: While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well. Here, we show that Transformer Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.

Attention Approximates Sparse Distributed Memory
Trenton Bricken
Cengiz Pehlevan
Systems, Synthetic and Quantitative Biology Harvard University
Applied Mathematics Harvard University
trentonbrickendg..nrvvee edu
cpehlevan@seas harvard. haw.rraaeeeaaaa.aarreeeeeeeeed
Abstract
While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why works so well. Here, show that Transformer Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.
Introduction
Used most notably in the Transformer, Attention has helped deep learning to arguably approach human level performance in various tasks with larger models continuing to boost performance
22 5, 7 9 However, the heuristic motivations that produced Attention leave open the question of why performs so well [ITO] Insights into why Attention is effective would not only make it more interpretable but also guide future improvements.
Much has been done try and explain Attention's success, including work showing that Transformer representations map more closely to human brain recordings and inductive biases than other models 12 Our work takes another step in this direction by showing the potential relationship between Attention and biologically plausible neural processing at the level of neuronal wiring, providing novel mechanistic perspective behind the Attention operation. This potential relationship is created by showing mathematically that Attention closely approximates Sparse Distributed Memory (SDM).
SDM is associative memory model developed in 1988 solve the "Best Match Problem" where we have a set of memories and want to quickly find the "best" match to any given query [ [4] In the development of its solution, SDM respected fundamental biological constraints, such as Dale's law, that synapses are fixed to be either excitatory or inhibitory and cannot dynamically switch (see Section 1 for SDM overview and [[3] (15) for deeper review). Despite being developed independently of neuroanatomy, SDM's biologically plausible solution maps strikingly well onto the
cerebellum I [6
Abstractly, the relationship between SDM and Attention exists because SDM's read operation uses intersections between high dimensional hyperspheres that approximate the exponential over sum of exponentials that Attention's 's softmax function (Section 2). Establishing that Attention approximates SDM mathematically, we then test it in pre-trained GPT2 Transformer models simulations (Appendix [B.7). We use the Query-Key Normalized Transformer variant show that the relationship to SDM holds well. We then original GPT2 models help confirm this result and make it more general.
3
(Section 3) and
2 to directly
1 'This cerebellar relationship additionally compelling by the fact that cerebellum like neuroanatomy exists in many other organisms including numerous insects (eg. the Drosophila Mushroom Body) and potentially cephalopods 17
9 20 21
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Using the SDM framework, we are able to go beyond Attention and interpret the Transformer architecture as a whole, providing deeper intuition (Section 4). Motivated by this mapping between Attention and SDM, we discuss how Attention can be implemented in the brain by summarizing SDM's relationship to the cerehellum (Section 5). In related work (Section 6), we link SDM to other memory models [2 24 including how SDM is generalization of Hopfield Networks and, in turn, how our results extend work relating Hopfield Networks to Attention 25 2d. Finally, we discuss limitations, and future research directions that could leverage our work (Section 7).
1 Review of Kanerva's SDM
Here, we present a short overview of SDM. A deeper review on the motivations behind SDM and the features that make it biologically plausible can be found in [I3 [T SDM provides an algorithm for how memories (patterns) are stored in, and retrieved from, neurons in the brain. There are three primitives that all exist in the space of n dimensional binary vectors:
Patterss (p) - have two components: the pattern address, p E {0, 13", ", is the vector representation of memory; the pattern "pointer", D E {0, 1)"", is bound to the address and points to itself when autoassociative or to a different pattern address when heteroassociative. A heteroassociative example is memorizing the alphabet where the pattern address for the letter points to pattern address b, b points to c etc. For tractability in analyzing SDM, we assume our pattern addresses and pointers are random. There are m patterns and they are indexed by the superscript μ E (m....
Neurons (x)- in showing SDM's relationship to Attention it is sufficient to know there are r neurons with fixed addresses x", E {0, 1)" that store set of all patterns written them. Each neuron will sum over its set of patterns to create a superposition. This creates minimal noise interference between patterns because of the high dimensional nature of the vector space and enables all patterns to be stored in an n dimensional storage vector denoted x E 2 constrained to the positive integers. Their biologically plausible features are outlined in L3 15 When we assume our patterns are random, we also assume our neuron addresses are randomly distributed. Of the 2n possible vectors in our binary vector space. SDM is "sparse" because it assumes that < 2n neurons exist in the space.
Query (e) - is the input to SDM, denoted & E {0, 1)ah. The goal in the Best Match Problem is to return the pattern pointer stored at the closest pattern address the query. We will often care about the maximum noise corruption that can be applied to our query, while still having it read out the correct pattern. An autoassociative example is wanting recognize familiar faces in poor lighting. Images of faces we have seen before are patterns stored in memory and our query is a noisy representation of one of the faces. We want SDM return the noise-free version of the queried face, assuming is stored in memory.
SDM uses the Hamming distance metric between any vectors defined: d(a. b) = 1, b/- The all ones vector 1 is of n dimensions and a b/ takes the absolute value of the element-wise difference between the binary vectors. When it is clear what two vectors the Hamming distance is between, we will sometimes use the shorthand d, := d(a. b).
The Hamming distance is crucial for determining how many neurons read and write operations are distributed across. The optimal Hamming distance for the read and write circles denoted d², depends upon the number and distribution of patterns in the vector space and what the memories are being used for (e.g. maximizing the number of memories that can be stored versus the memory system's 's robustness to query noise). We provide three useful reference d' values, using equations outlined in Appendix B.5 The Signal-to-Noise Ratio (SNR) optimal dŠNR maximizes the probability a noise-free query will return its target pattern 15). The memory capacity optimal de maximizes the number of memories that can be stored with a certain retrieval probability and also assumes a noise-free query. The critical distance d maximizes, for given number of patterns, the amount of noise that can be applied to a query such that it will converge to its correct pattern [5).
These d' are only approximate reference points for later comparisons to Transformer Attention, first and foremost because they assume random patterns to make their derivations tractable. In addition, Transformer Attention will not be optimizing for just one of these objectives, and likely interpolates between these optimal d' s as wants to have both a good critical distance to handle noisy queries and reasonable memory capacity. These optimal d' are function of n, and m. For the Transformer

---

Attention setting D. where = 64, r = 2" and m S 1024, dğNR 11, dMem = 5, dCD = 15, as
derived in Appendix B.5
Write patterns into nearby neurons
Write radius
Pattem stored
Writing in a second pattern
Neurons superposition patterns
Original
D
Neuron
Read patterns from neurons near query
Neuron view:
Query stored pettern
Circle Intersections with original pattern locarions.
S
Query
Figure 1: Summarizing the SDM read and write operations. Top Row three patterns being written into nearby neurons. The first write operation; 2. Patterns are stored inside nearby neurons and the original pattern location is shown; 3. Writing a second pattern: 4. Writing a third pattern and neurons storing superposition of multiple patterns. Bottom Row shows the read operation. Neuron view (left) shows the query reading from nearby neurons with the inset showing the number times each pattern read. The four blue patterns are a majority which would result in one step convergence. Pattern view (right) is crucial to relating SDM to Attention and defined Eq. below. We abstract away the neurons by assuming they are uniformly distributed through the space. This allows us to consider the circle intersection between the query and the original locations of each pattern where blue has the largest circle intersection.
isomorphic perspectives of
1.1 SDM Read Operation
For the connection to Attention we focus on the SDM read operation and briefly summarize the write operation: all patterns write their pointers P, in distributed fashion all neuron addresses located within Hamming distance d. This means that each neuron will store a superposition of pattern pointers from those pattern addresses within d: x patterns distributed fashion across nearby neurons, SDM's 's read operation retrieves stored pattern pointers from all neurons within distance of the query and averages them. This average is effectively weighted because the same patterns have distributed storage across multiple neurons being read from. The pattern weighting will be higher for those patterns with addresses nearer the query because they have written their pointers to more neurons the query reads from. Geometrically, this weighting of each pattern can be interpreted as the intersection of radius d circles") that are centered on the query and each pattern address P for all μ. A high level overview of the SDM read and write operations is shown in Fig. m
(p=d(p
Pp. Having stored
The 2n possible neurons that have both stored this pattern's pointer Pp and been read by E is: JO, (Pa: d) (n, d)), where |- is the cardinality operator and O,(E, d) = {x, {0, 1)" d(E, Xx) S dd is the set of all possible neuronal addresses x, within radius d of g. Mathematically,
IIn this binary space, the Hamming distance around vector fact hypereube dimensional unit cube lie on the surface of an n dimensional sphere with radius vn/2 and we refer to this as a circle because of our two dimensional diagrams. We adopt this useful analogy, taken from Kanerva's book on SDM [3]. throughout the paper.
vertices of
3

---

SDM's 's read operation sums over each pattern's pointer, weighted by its query circle intersection:
Epep (n gnew = "pper (n. O., d)l
{ 0,
1. ife 2
g(e)
(I)
else
and g acts elementwise on vectors. The denominator normalizes all of the weights so they sum to I in the numerator and enables computing if the element-wise majority value is 0 or 1, using the function g()). Intuitively, the query will converge to the nearest "best" pattern because it will have the largest circle intersection weighting. The output of the SDM read operation is written updating the query & + 5 new so that it can (but is not required to) apply the read operation iteratively if full convergence its "best match" pattern is desired and not achieved in update.
The circle intersection (derived Appendix B.T for the read and write operations d, the dimensionality and the vector distance between the query and pattern: d, = d( d(pa. s E), so we use the shorthand I(dy, d, nT):
calculated as function of the Hamming radius
T2- de
(n- d
(m-d))
(a.)
dg
I(d,, n, = On. d) n (O.(E.
Σ
Σ
(2)
c=
Eq sums over the number of possible binary vectors that can exist at every location inside the circle intersection. Taking inspiration from [27] this is new and more interpretable derivation of the circle intersection than that originally developed [3.. Eq. 2 is approximately exponential for the closest, most important patterns where d(pa, <) s 2d, which crucial to how SDM approximates Attention.
This is shown for representative instance SDM in Fig. 2 The details of this approximation provided in Appendix B.2 but at high level the binomial coefficients can be represented as binomial distributions and then approximated by normal distributions that contain exponentials. With the correctly chosen constants, ç₁ and cz. that are independent of the vector distance d(pa. E), can make the following approximation:
I(d(Pa: E), n) ai exp ( - d(pa.
(3)
Expected Log Circle Intersection d
A-61,
1077
d-11
μ p
1011
Log Eiete interastion
a
1040
d
10a
10
15
20
Vector Distance, d.g
Hamming Distance d, between two vectors
Figure 2: (Left) SDM's read operation using the Hamming radius d (for reading and writing) and the vector distance d, d( Recall that during the write operation, pattern addresses P write their pattern pointer P to neurons located at the addresses x" (denoted here black dots) within radius d. During the read operation, the query & reads from each neuron within radius d, thus creating an intersection. (Right) As d, between the query and pattern increases (x-axis), the size of their circle intersection falls approximately exponentially (y-axis). We use Eq, 2 with n 64 and dš 11, while varying d,, up to the distance of d, 2d beyond which point there is no circle intersection. We plot the y-axis on log scale to show how, because the curve is approximately linear, the circle intersection is approximately exponential. See Appendix B.2 for formal analysis of the exponential approximation the circle intersection creates that is robust across parameters n, d, and r.

---

2
Attention Approximates SDM
To be able to handle a large number of patterns, we let the pattern address matrix with each pattern as a column be: Pa [Pa Pa Pu.... ... P with pointers P. Ipp P Pp
The Attention update rule [ using its original notation is:
fnew = =Vsottax(33330) = (W,Y )oft)aa
where K, V, and Q symbolize the "key", "value", and "query" matrices, respectively. is single query vector and Y represents the raw patterns to be stored in memory. The softmax exp(3x)) Σ exp(3x;), where the exponential acts element-wise and Attention sets 3 = 1/ vn. Softmax normalizes a vector of values to sum to and gives the largest values the most weight due to the exponential function, to what extent depending on B. We can re-write this using our notation, including distinguishing continuous vectors IR" from binary ones by putting a tilde above them:
new PPssofmax(3PP T E)
(4)
We write W&Y- Pa as the raw input patterns Y are projected by the learnt weight matrix We into the SDM vector space to become the addresses Pa. Similarly, W.Y Po and
Q = E.
Showing the approximation between SDM Eq. and Attention Eq. requires two steps: (i) Attention must L normalize its vectors. This a small step because the Transformer already uses LayerNorm
before and after its Attention operation that later relate L normalization; (ii) A coefficient for the softmax exponential must be chosen such that it closely approximates the almost exponential decay of SDM's circle intersection calculation.
23
To proceed, define map from binary vectors b to L normalized continuous vectors å, b, h(a) = å such that for any pair of pattern addresses the following holds:
d(a. b) - 1 (1-aab)
(5)
where [-] is the floor operator. We assume that this map exists, at least approximately. This map allows us to relate the binary SDM circle intersection (Eq, 2 to the exponential used in Attention (Eq. by plugging into the exponential approximation Eq.
I(d(Pa: n) = I (1," nh pT p..))..dn) = ci exp ( (- C2 2 (1-p%_))
2
2 c3 exp ()
where с3 encompasses the constants outside the exponential. We replaced the remaining constants in the exponential with 8, that is a function of n and d and is an approximation due to the floor operation.
Finally, these results allow us to show the relationship between Attention and SDM:
Eper I [3 PT d, Pr
EpEp cxp (3p Posoffmaxx(811) (3PT Eper exp
(7)
EpEp I ((13(1 PT d,
Alternatively, instead of converting cosine similarity to Hamming distance to use the circle intersection Eq. 2 in binary vector space, we can extend SDM to operate with L normalized pattern and neuron addresses (Appendix B.3 This continuous SDM circle intersection closely matches its binary counterpart in being approximately exponential:
2d 2d,n) n) cxexp (B.D.E).
I.
(8)
n
*Pattern pointers can point to a different vector space and thus do not need to be L normalized, However, in canonical SDM they point to pattern addresses in the same space so write them as also being L2 normalized our equations.
5

---

We use Ie to denote this continuous intersection. use Eq. 5 to map our Hamming d to cosine similarity. and use coefficients 24 and B. to acknowledge their slightly different values. Then, we can also relate
Attention as we did in Eq. 7 to continuous SDM as follows:
PPsooftax(3PP55 ΣpEp exp( d)
EpEp To 2d
Pb
n
Epep (
ΣPEP Io 1
2d
We have written Attention with L normalized vectors and expanded out the softmax operation to show that it is approximated when we replace the exponential weights by either the binary or continuous SDM circle intersections (Eqs. 7 and respectively). The right hand side of Eq. 0 is identical to Eq. 2 aside from using continuous, L normed vectors and dropping the elementwise majority function g-) that ensured our output was a binary vector. In the Transformer, while the
Attention equation does not contain any post-pres post-processing function to its query update & it is then post-processed by going through a linear projection and LayerNorm [ and can be related to g()).
To fit / to binary SDM, we convert the Hamming distances into cosine similarity using Eq. 5 and use
univariate log linear regression:
log E((((p.:))
log(cs) + ((PT
(10)
We expect the exponential behavior to break at some point, if only for the reason that if dp. <) > 2d the circle intersection becomes zero. However, closer patterns are those that receive the largest weights and "attention" such that they dominate in the update rule and are the most important.
binary and continuous SDM for our smallest optimal deem 5 and largest dep = 15 to show not only the quality of the approximations but also how many orders of magnitude smaller the normalized weights are when d(p.. c) > For these plots, we plug into our binary circle intersection equation each possible Hamming distance from 0 to 64 when n 64 and converting Hamming distance to cosine similarity, doing the same for our continuous circle intersection. Here use our binary intersection values to fit (3, creating the exponential approximation. To focus our exponential approximation on the most important, closest patterns, we fit our regression to those patterns d(p... { < d and allow it to extrapolate to the remaining values. We then normalize the values and plot them along with an smaller inset plot in log space to better show the exponential relationship. In both plots, looking at the log inset plot first, the point which the circle intersection in blue ceases to exist or be exponential corresponds to a point in the main normalized plot where the weights are == O.
In Fig. we plot the softmax approximation
The number of neurons r and how well they cover the pattern manifold are important considerations that will determine SDM's performance and degree of approximation to Attention. Increasing the number of neurons the circle intersection can be accomplished by increasing the number of neurons in existence, ensuring they cover the pattern manifold, and reducing the dimensionality of the manifold to increase neuron density In SDM's original formulation, it was assumed that neuronal addresses were randomly distributed and fixed in location, however, extensions to SDM 2y) have proposed biologically plausible competitive learning algorithms to learn the manifold B0 To ensure the approximations SDM tight, we test random and correlated patterns in an autoassociative retrieval task across different numbers of neurons and SDM variants (Appendix [B.]). These variants
fl
include SDM implemented using simulated neurons and the Attention approximation with a fitted
3 To summarize, Attention closely approximates the SDM update rule when it uses I,² normed continuous vectors and correctly chosen B.
3 Trained Attention Converges with SDM
For many instantiations of SDM, there exists a 3 that can be found via the log linear regression Eq. 10 that makes Attention approximate well. However, depending on the task hand, there are instantiations of SDM that are better than others as highlighted by the different d' optimal values. If
'This can be done by learning weight projection matrices like in Attention make the manifold lower dimensional and also increase separability between patterns.
'The code for running these experiments, other analyses, and reproducing all figures is available at https //github com http:///weaabbbeaaatttttttttttpa

---

Normalized Weights for Circle Intersections
Normalized Weights
Intersections Fitted Bin. B 10:11
nw64 | dd=5 (P-40-13 Fitted Bin. f - 35 484
n=64
d=15 p 1e-05
0.35
Bin. Circle Intersect
Circle Bin. Exp w/
füln Pap fitted
Nomabbin Weight
0.6
Plot
0.25
Log Plot
Nomalzed
0.20
0.15
0.10
0.05
177
20
20
Hamming
Hamminng
Figure 3: Representative plots showing the relationships in Eqs. 7 and between the softmax approximation (green) and the normalized binary (blue) and continuous (orange) circle intersections using Transformer Attention parameters. Here the softmax approximation uses the Eq. I0 regression to fit its / binary SDM. We use Eq. 5 to relate Hamming distance and cosine similarity for our vector distances on the x-axis. The insets use a log y-axis to show the circle intersections approximately exponential when d((p.. <) < 2d. (Left) Uses dem 5, corresponding the Hamming distance reading and writing to = 4e - 13 of the vector space. (Right) Uses dčo = 15.
These results hold well for other values of and but focus on the Attention setting.
Attention in the Transformer model is implementing SDM, we should expect for trained Attention to use 3s that correspond to reasonable instances of SDM. We use as reference points these optimal d's. Attention learns useful pattern representations that are far from random so this SDM/ 33 that fits the optimal d's are only weak reference for what 3 values might be reasonable. However, because these d' definitions of optimality span from maximizing convergence with query noise, to maximizing memory capacity with noise free queries, we should expect the Transformer deal- ing with noisy queries and wanting reliable re- trieval to interpolate between these d' values. Here, we provide empirical evidence that this is indeed the case. We analyze the 3 coeffi- cients learnt by the "Query- Key Normalization" Transformer Attention variant [22]. Query-Key Normalization makes it straightforward to find 33 because it is learnt via backpropagation and easy to interpret because it uses cosine similarity between the query and key vectors. To further evaluate the convergence between Attention and SDM 3 coefficients and make it more general, we also investigate the GPT2 architecture 3 However, in this case we need to infer "effective" values from the size of query key dot products in the softmax. This makes these results, out- lined in Appendix A.2 more approximate but they remain largely in agreement with the learnt Bs of Query-Key Norm.
Learnes Coefficients for Attention Heaas different translation tasks.
Rcp 10.1
Bax - 15.9
Rous 35.5
COUrt
15
25 Learned Coefficien:
Figure Histogram showing learned | coeffi- cients for all Attention heads across layers for the translation tasks used (22.. We plot the 3 val- ues for Attention that approximate the different d' definitions showing how the 3s are interpolating between them. 3CD is optimal for critical distance (maximum noise for each query); 3S R is optimal for Signal-to-Noiss ratio. This assumes there is no query noise and SDM wants to minimize noise from other queries. BMem maximizes memory ca- pacity and also assumes no query noise.
The Kuery-Key Norm Attention heads learn 3 E [10, 25] as shown Fig. Note that the whole range of 3 E [10, 25) interpolates between the d' values well, in particular with the critical distance optimal that realistically assumes noisy queries and the SNR optimal, where having high signal to noise ratio is desirable for both memory capacity and critical distance (see Appendix B.5).
'These values were graciously provided by the authors of [22 in private correspondence for their trained Transformer models.
7

---

In requiring that Attention vectors be L normalized and 3 fitted, SDM predicted Query-Key Norm. This is interesting because Query-Key Norm has evidence of improving Transformer performance and training speed [22 31 We say more about its advantages and caveats in Appendix A.1
4 Transformer Components Interpreted with SDM
We can leverage how Attention approximates SDM to interpret many components of the Transformer architecture This exercise demonstrates the explanatory power of SDM and, in relating it to additional unique Transformer components, widens the bridge upon which ideas related to SDM and neuroscience can cross into deep learning and vice versa.
crucial component of the Transformer is its Feed Forward (FF) layer that interleaved with the Attention layers and uses approximately 2/3rds of the Transformer's parameter budget 1 Attention's ' Transformer implementation and SDM deviate importantly in Attention's 's use of ephemeral patterns from the current receptive field. In order to model temporal dependencies beyond the receptive field, we want Attention to be able to store persistent memories. Work has compellingly shown that the FF layers store these persistent memories R2
SDM can be interpreted as this FF layer because in [32 the FF layer was substituted with additional, persistent key and value vectors that Attention learnt independently rather than projecting from its current inputs. This substitution performed on par with the FF layer which, combined with deeper analysis in [33, shows the FF layers are performing Attention with long term memories and thus can be directly interpreted as SDM.
3 34
Another crucial component of the Transformer its use of LayerNorm 22 35 M. LayerNorm has a natural interpretation by SDM as implementing a similar constraint to L normalization. However, while it ensures all of the vectors are on the same scale such that their dot products are comparable to each other, it does not constrain these dot products to the interval [-1, like cosine similarity. In addition to providing an interpretation for the importance of LayerNorm, this discrepancy has led to two insights: First, as previously mentioned, it predicted that Query-Key Normalization could be a useful inductive bias (more details in Appendix A.I Second, it has provided caution to interpretations of Attention weights. Fig. X of Appendix A.3 shows that there are many value vectors that receive very small amounts of attention but have large L norms that dominate the weighted summation of value vectors. This makes it misleading to directly interpret Attention weights without L normalization of the value vectors and this has not been done in work including []] 37 38). Beyond helping future interpretations of Attention, we tested L normalized value vectors potentially useful inductive bias by training a GPT2 model with it. Our results showed that this did not change performance but 1 normalization should still be performed in cases where the Attention weights will be interpreted. See Appendix A.3 for a full discussion.
Finally, multi-headed Attention is a Transformer component where multiple instantiations of Attention operate at the same hicrarchical level and have their outputs combined. Multi healii-heding allows SDM to model probabilistic outputs, providing an interpretation for why it benefits Attention. For example, if we are learning a sequence that can go "A- + B" and "A- + Z" with equal probability, we can have one SDM module learn each transition. By combining their predictions we correctly assign equal probability to each. This probabilistic interpretation could explain evidence showing that Attention heads pay attention to different inputs and why some are redundant post training
39
20 10).
An important difference between SDM and the Transformer that remains to be reconciled is in the Transformer's 's hierarchical stacking of Attention. This is because, unlike in the traditional SDM setting where the pattern addresses (keys) and pattern pointers (values) are known in advance and written into memory, this cannot be done for layers of SDM beyond the first that will need to learn latent representations for its pattern addresses and pointers (keys and values). Transformer Attention solves this problem by learning its higher level keys and values, treating each input token as its own query to generate a new latent representation that is then projected into keys and values [.]. This does not mean SDM would fail to benefit from hierarchy. As concrete example, the operations of SDM are related to the hierarchical operations of 41 More broadly, we believe thinking about how to learn the latent keys and values for the higher levels of SDM could present new Transformer imprroeeentss. A key breakthrough of the recent Performer architecture that highlights the arbitrariness of the original Transformer solution is its use of reduced set of latent keys and values 42
For a summary of the components that make up the full Transformer architecture see Fig. 9 of Appendix
A.4
8

---

5 A Biologically Plausible Implementation of Attention
Here, we provide an overview of SDM's biological plausibility to provide biologically plausible implementation of Attention. SDM's read and write operations have non trivial connectivity require- ments described in [k] 15] Every neuron must: (i) know to fire if it within Hamming distance of an input; (ii) uniquely update each element of its storage vector when writing in a new pattern; (iii) output its storage vector during reading using shared output lines so that all neuron outputs can be summed together.
Unique architectural features of the cerebellar cortex can implement all of these requirements, specifically via the three way convergence between granule cells, climbing fibers and Purkinje cells: (i) all granule cells receive inputs from the same mossy fihers to check if they are within d of the incoming query or pattern; (ii) each granule cell has a very long parallel fiber that stores memories in synapses with thousands of Purkinje cells [ , updated by LTP/LTD (Long Term Potentiation/Depeee Deppasssepppre from joint firing with climbing fibers; (iii) all granule cells output their stored memories via their synapses to the Purkinje cells that perform the summation operation and use their firing threshold to determine if the majority bit was 1 or O, outputting the new query Moreover, the Drosophila mushroom body highly similar to the cerebellum and the previous cell labels for each function can be replaced with Kenyon cells, dopaminergic neurons, and mushroom body output neurons, respectively
[ [5]
While SDM fits the unique features of the cerebellum well, this connection has limitations. Expla- nations for some of the original model's limitations have been put forward to account for sparse dendritic connections of Granule cells 34] and the functions of at least two of the three inhibitory interneurons: Golgi, Stellate and Basket cells 2 45. However, there are futher challenges that remain, including better explanations of the inputs to the mossy and climbing fibers and outputs from the Purkinje cells; in particular, how the mossy and climbing fiber inputs synchronize for the correct spike time dependent plasticity [6).. Another phenomenon that SDM does not account for is the ability of Purkinje cells to store the time intervals associated with memories +7. Further research is
necessary to update the state of SDM's biological plausibility with modern neuroscientific findings.
6 Related Work
Previous work showed that the modern Hopfield Network, when made continuous and optimized differently, becomes Attention 25 26 This result was motivation for this work because Hopfield Networks are another associative memory model. In fact, it has been shown that SDM is a generalization of the original Hopfield Network (Appendix B.6 24 While SDM is a generalization of Hopfield Networks, their specific discrepancies provide different perspectives on Attention. Most notably, Hopfield Networks assume symmetric weights that create an energy landscape, which can be powerfully used in convergence proofs, including showing that one step convergence is possible for the modern Hopfield Network, and by proxy, Attention and SDM when it is close approximation However, these symmetric weights come at the cost of biological plausibility that SDM provides in addition to its geometric framework and relation to Vector Symbolic Architectures
25
26
20
Other works have tried to reinterpret remove the softmax operation from Attention because the normalizing constant can be expensive to compute 19 [0] However, while reducing computational cost, these papers show that removing the softmax operation harms performance. Meanwhile, SDM not only shows how Attention can be written as a Feedforward model [5] but also reveals that through simple binary read and write operations, (the neuron is either within Hamming/cosine distance it's not) the softmax function emerges with no additional computational cost.
Since the publication of SDM, there have been number of advancements not only to SDM specif- ically, but also through the creation of related associative memory algorithms under the name of "Vector Symbolic Architectures" [51]. Advancements to SDM include using integer rather than binary vectors (52], handling correlated patterns (29.. and hierarchical data storage [53]] Vector Symbolic
Architectures, most notably Holographic Reduced Representations, have ideas that be related
back to SDM and the Transformer in ways that may be fruitful [54] 55
58 59
The use of external memory modules in neural networks has been explored most notably with the Neural Turing Machine (NTM) and its followup, the Differentiable Neural Computer (DNC) 23 (D) In order to have differentiable read and write operations to the external memory, they use the softmax

---

function. This, combined with their use of cosine similarity between the query and memory locations, makes both models closely related to SDM. A more recent improvement to the NTM and DNC directly inspired by SDM is the Kanerva Machine [24 61
However, the Kanerva Machine remains distinct from SDM and Attention because it does not apply the a Hamming distance threshold on the cosine similarity between its query and neurons. Independent of these discrepancies, we believe relating these alternative external memory modules to SDM presents a number of interesting ideas that will be explored in future work.
62 G
7 Discussion
The result that Attention approximates SDM should enable more cross pollination of ideas between neuroscience, theoretical models of associative learning, and deep learning. Considering avenues for future deep learning research, SDM's relationship Vector Symbolic Architectures is particularly compelling because they can apply logical and symbolic operations on memories that make SDM more powerful [55 64 65 66 [7]. SDM and its relation to the brain can inspire new research in not only deep learning but also neuroscience, because of the empirical success of the Transformer and its relation to the cerebellum, via SDM.
Our results serve as a new example for how complex deep learning operations can be approximated by, and mapped onto, the functional attributes and connectivity patterns of neuronal populations. At a time when many new neuroscientific tools are mapping out uncharted neural territories, we hope that more discoveries along the lines of this work connecting deep learning to the brain will be made
Limitations While our work shows a number of convergences between SDM, Attention, and full Transformer models, these relationships remain approximate. The primary approximation is the link between SDM and Attention that exists not only in SDM's 's circle intersection being approximately exponential but also its use of binary rather than continuous space. Another approximation is between optimal SDM Hamming radii d' and Attention 3 coefficients. This is because we assume patterns are random to derive the d' values. Additionally, in the GPT2 Transformer models we must infer their effective 3 values. Finally, there is only an approximate relationship between SDM and the full Transformer architecture, specifically with Feed Forward and LayerNorm components.
Conclusion
We have shown that the Attention update rule closely approximates SDM when I norms its vectors and has an appropriate 3 coefficient. This result has been shown to hold true in both theory and empirical evaluation trained Transformer models. SDM predicts that Transformers should normalize their key, query and value vectors, preempting the development of Query-Key Normalization and adding nuance to the interpretation of Attention weights. We map SDM onto whole, relate it to other external memory implementations, and
the Transformer architecture highlight extensions to SDM. By discussing how SDM can be mapped to specific brain architectures, provide potential biological implementation of Transformer Attention. Thus, our work highlights another link between deep learning and neuroscience.
Acknowledgements
Thanks to Dr. Gabriel Kreiman, Alex Cuozzo, Miles Turpin, Dr. Pentti Kanerva, Joe Choo-Choy, Dr. Beren Millidge, Jacob Zavatone- Veth, Blake Bordelon, Nathan Rollins, Alan Amin, Max Farrens, David Rein, Sam Eure, Grace Bricken, and Davis Brown for providing invaluable inspiration, discussions and feedback. Special thanks to Miles Turpin for help working with the Transformer model experiments. We would also like to thank the open source software contributors that helped make this research possible, including but not limited to: Numpy, Pandas, Seipy, Matplotlib, Py Torch, HuggingFace, and Anaconda. Work funded by the Harvard Systems, Synthetic, and Quantitative Biology PhD Program.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
10

---

[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016.
[3] Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, and Ilya Sutskever. Better language models and their implications. OpenAl Blog https://openai. 2019.
connbloogbetter-lgggug--mmmm
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vose Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[5] Mark Chen, A. Radford, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. ICML 2020, 2020.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
[7] Anonymous. Videogen: Generative modeling of videos using (vq)-(vae) and transformers. In Submitted to International Conference on Learning Reprexentations, 2021. under review.
[8] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, and Scott Gray. Dall-e: Creating images from text, Jan 2021.
[9] Gwern. May 2020 news and on gpt-3, Dec 2019.
[10] Anna Rogers, O. Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
[11] Martin Schrimpf, I. Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, N. Kanwisher,
J. Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative reverse- engineering converges on model for predictive processing. bioRviv, 2020.
[12] Shikhar Tuli, Ishita Dasgupta, Erin Grant, and T. Griffiths, Are convolutional neural networks or transformers more like human vision? ArXiv, abs/2105.071997 2021.
[13] Pentti Kanerva. Sparse distributed memory. MIT Pr., 1988.
[14] Marvin Minsky and Seymour Papert. Time vs. memory for best matching an open problem, page 222 - 225. 1969.
[15] Kanerva. Sparse distributed memory and related models. 1993.
[16] M. Kawato, Shogo Ohmae, and Terry Sanger. 50 years since the marr, ito, and albus models of the cerebellum. Neuroscience, 462: 2021.
[17] M. Modi, Yichun Shuai, and G. Turner. The drosophila mushroom body: From architecture to algorithm in a learning circuit. Annual review of neuroscience, 2020.
[18] G. Wolff and N. Strausfeld. Genealogical correspondence of a forebrain centre implies an executive brain in the protostome-dduttersstom bilaterian ancestor. Philosophical Transactions of the Royal Society: Biological Sciences, 371, 2016.
[19] Ashok Litwin-Kumar, K. D. Harris, R. Axel, H. Sompolinsky, and L. Abbott. Optimal degrees of synaptic connectivity. Neuron, 93:1 13:1153-1164.40 2017.
[20] T. Shomrat, Nicolas Graindorge, C. Bellanger, G. Fiorito, Y. Loewenstein. and B. Hochner. Alternttive sites of synaptic plasticity in two homologous "fan-out fan-in" learning and memory networks. Current Biology. 21:1773-1782, 2011.
[21] S. Shigeno and C. W. Ragsdale. The gyri of the octopus vertical lobe have distinct neurochemical identities. Journal of Comparative Neurology. 523, 2015.
[22] A. Henry, Prudhvi Raj Dachapally, S. Pawar, and Yuxuan Chen. Query-key normalization for transformers. In EMNLP, 2020.
[23] A. Graves, G. Wayne, and Ivo Danihelka. Neural turing machines. ArXiv, abs/1440,5 2014.

---

[24] Yan Wu, Greg Wayne, Alex Graves, and Timothy Lillicrap. The kanerva machine: A generative distributed memory, 2018.
[25] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner. Milena Pavlović, Geir Kjetil Sandve. Victor Greiff, David Kreil, Michael Kopp. Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need, 2020.
[26] D. Krotov and J. Hopfield. Dense associative memory for pattern recognition. In NIPS, 2016.
[27] Louis A. Jaeckel. An alternative design for a sparse distributed memory. 1989.
[28] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
[29] James D. Keeler. Comparison between kanerva's sdm and hopfield-type neural networks. Cognitive Science, 12(3):299 - 329, 1988.
[30] David E. Rumelhart and David Zipser. Feature discovery by competitive learning. 1985.
[31] Guang Yang, Xiang Chen, K. Liu, and C. Yu. Deeppseudo: Deep pseudo-code generation via transformer and code feature extraction. ArXiv, abs/2102.06360, 2021.
[32] Sainbayar Sukhbaatar, E. Grave, Guillaume Lample, H. Jégou, and Armand Joulin. Augmenting self-attention with persistent memory. ArXiv, abs/1907.011770 2019.
[33] Mor Geva, R. Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. ArXiv, abs/2012. 149113...1441,.. 2020.
[34] N. Carlini, Florian Tramèr, Eric Wallace, M. Jagielski, Ariel Herbert-Voss, K. Lee, A. Roberts, Tom Brown, D. Song, Ülfar Frlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. ArXiv, abs/2012.00803. 2020.
[35] Kevin Lu, Aditya Grover, P. Abbeel, and Igor Mordatch. Pretrained transformers universal computation engines. ArXiv, abs/2103.05247, 2021.
[36] Jesse Vig. Visualizing attention transformer-bant language representation models. ArXiv, abs/1904.02679, 2019.
[37] Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna. Carey Radebaugh, Emily Reif, and Ann Yuan. The language interpretability tool: Extensible, interactive visualizations and analysis for nip models. In EMNLP, 2020.
[38] Joris Baan, Maartje ter Hoeve, M. V. D. Wees, Anne Schuth, and M. Rijke. Do transformer attention heads provide transparency in abstractive summarization? ArXiv, abs/ hts/1907.0g 0050000 2019.
[39] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? ArXiv, abs/1905. 10650, 2019.
[40] J Alammar. Interfaces for explaining transformer language models, 2020.
[41] Yubei Chen, Dylan M. Paiton, and Bruno Olshausen. The sparse manifold transform. In NeurlPS, 2018.
[42] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Be- langer, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. ArXiv, abs/2009. 14794, 2021.
[43] Eriola Hoxha, F. Tempia, Lippiello, and M. C. Miniaci. Modulation, plasticity and patho- physiology of the parallel fiber-purkinje cell synapse. Frontiers in Synaptic Neuroscience, 8, 2016.
[44] Louis A. Jaeckel. A class of designs for sparse distributed memory. 1989.
[45] Eren Sezener, Agnieszka Teabska-Barwinskee Dimitar Kostadinov, Maxime Beau, Sanjukta Krishnagopal, David Budden, Marcus Hutter, Joel Veness, Matthew M. Botvinick, Claudia Clopath, Michael Häusser, and Peter E. Latham. A rapid and efficient learning rule for biological neural circuits. bioRxiv, 2021.
[46] H. Markram, Lubke, M. Frotscher, and Sakmann. Regulation of synaptic efficacy by coincidence of postsynaptic aps and epsps. Science, 275:213 215, 1997.
12

---

[47] Charles R. Gallistel. The coding question. Trends in Cognitive Sciences, 21:498-5555 00888,008-000 2017.
[48] D. Krotov and J. Hopfield. Large associative memory problem in neurobiology and machine learning. ArXiv, (es/2208806699)) 2020.
[49] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franccois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. ArXiv, abs/2006.162.3 2020.
[50] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In ICML, 2021.
[51] P. Kanerva. Hyperdimensional computing: An introduction to computing in distributed repre- sentation with high-dimensional random vectors. Cognitive Computation, 1:139-159, 2009.
[52] R. Prager and F. Fallside. The modified kanerva model for automatic speech recognition. Computer Speech and Language, 3:61-81, 1989.
[53] L. Manevitz and Yigal Zemach. Assigning meaning to data: Using sparse distributed memory for multilevel cogmitive tasks. Neurocomputing. 14:15-39, 1997.
[54] Ivo Danihelka, Greg Wayne, B. Uria, Nal Kalchbrenner, and A. Graves. Associative long short-term memory. In ICMI., 2016.
[55] T. Plate. Holographic reduced representations: Convolution algebra for compositional dis- tributed representations. In I/CAI, 1991.
[56] P. Kanerva. Binary spatter-coding of ordered k-tuples. In ICANN, 1996.
[57] P. Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artif. Intell., 46: 159-216, 1990.
[58] R. Gayler. Multiplicative binding. representation operators and analogy. 1998.
[59] Jeff Hawkins and Subutai Ahmad. Why neurons have thousands of synapses, theory of sequence memory in neocortex. Frontiers in Neural Circuits, 10:23, 2016.
[60] A. Graves, Greg Wayne, M. Reynolds, T. Harley, Ivo Danihelka, Agnieszka Grabska- Barkkk--Bawiisk,,, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, J. Agapiou, Adrià Puig- domènech Badia, K. Hermann, Yori Zwols, Georg Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and Demis Hassabis. Hybrid computing using neural network with dynamic external memory. Nature, 538:471-4766 2016.
[61] Y. Wu, Gregory Wayne, K. Gregor, and T. Lillicrap. Learning attractor dynamics for generative memory. In NeurlPS, 2018
[62] K. Gregor, Danilo Jimenez Rezende, Frédéric Besse, Y. Wu, Hamza Merzic, and Aaron van den Oord. Shaping belief states with generative environment models for rl. ArXiv, abs/1906.092377 2019.
[63] Adam Marblestone, Yan Wu, and Greg Wayne. Product kanerva machines: Factorized bayesian memory, 2020.
[64] C. Eliasmith. How to build a brain: A neural architecture for biological cognition. 2013.
[65] B. Lake, Tomer D. Ullman, J. Tenenbaum, and S. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2016.
[66] S. Piantadosi. The computational origin of representation. Minds and Machines, 31:1-58, 2021.
[67]
Yoshua Bengio and Gary Marcus. Ai debate Yoshua bengio gary marcus. 2019.
[68]
Alon, Daniel R. Goodwin, Anubhav Sinha, A. Wassie, Fei Chen, Evan R. Daugharthy, Yosuke Bando, Atsushi Kajita, Andrew G. Xue, Karl Marrett, Robert Prior, Yi Cui, A. Payne, Chun- Chen Yao, Ho-Jun Suk, Ru Wang, Chih chieh Yu, Paul W. Tillberg, P. Reginato, N. Pak, S. Liu, Sukanya Punthambaker, Eswar P. R. Iyer, Richie E. Kohman, J. Miller, E. Lein, Ana Lako, N. Cullen, S. Rodig, K. Helvie, Daniel L. Abravanel, N. Wagle, B. Johnson, J. Klughammer, Michal Slyper, Julia Waldman, J. Valbuenn, O. Rozenblatt-Rosen A. Regev, G. Church, Adam H. Marblestone, and E. Boyden. Expansion sequencing: Spatially precise in situ transcriptomics in intact biological systems. bioRxiv, 2020.
[69] C. Xu, Michal Januszewski, Z. Lu, S. Takemura, K. Hayworth, G. Huang, K. Shinomiya, Jeremy B. Maitin Shepaad,, David Ackerman, Stuart E. Berg, T. Blakely, J. Bogovic, Jody Clements, Tom Dolafi, Philip M. Hubbard, Dagmar Kainmueller, W. Katz, Takashi Kawase,
13

---

K. Khairy, Laramie Leavitt, Peter H. Li, Larry Lindsey. Nicole L. Neubarth, D. J. Olbris, H. Otsuna, Eric T. Troutman, L. Umayam, Ting Zhao, M. Ito, Jens Goldammer, T. Wolff, R. Svirskas, P. Schlegel, E. Neace, Christopher J Knecht, Chelsea X. Alvarado, Dennis A Bailey, Samantha Ballinger, J. Boryez, Brandon S Canino, Natasha Cheatham, Michael Cook, M. Dreher, Octave Duclos, Bryon Eubanks, K. Fairbanks, S. Finley, N. Forknall, Audrey Francis, Gary Hopkins, Emily M Joyce, Sungjin Kim, Nicole A. Kirk, Julie Kovalyak, Shirley Lauchie, Alanna Lohff, Charli Maldonado, Emily A. Manley, Sari McLin, C. Mooney, Miatta Ndama, Omotara Ogundeyi, Nneoma Okeoma, Christopher Ordish, Nicholas L. Padilla, Christopher Patrick, Tyler Paterson, Elliott E Phillips, E. M. Phillips, Neha Rampally, Caitlin Ribeiro, Madelaine K Robertson, J. Rymer, S. Ryan, Megan Sammons, Anne K. Scott, Ashley L. Scott, A. Shinomiya, C. Smith, Kelsey Smith, N. L. Smith, Margaret A. Sobeski, Alia Suleiman, J. Swift, Satoko Takemura, Iris Talebi, Dorota Tarnogorska, Emily Tenshaw, Temour Tokhi, J. Walsh, Tansy Yang. J. Horne, Feng Li, Ruchi Parekh, P. Rivlin, V. Jayaraman, K. Ito, S. Saalfeld, R. George, I. Meinertzhagen, G. Rubin, H. Hess, Louis K. Scheffer, Viren Jain, and Stephen M. Plaza. A connectome of the adult drosophila central brain. bioRxiv, 2020.
[70] Kristin M Scaplen, M. Talay, John D) Fisher, Raphael Cohn, Altar Sorkaç, Y. Aso, G. Barnea, and Kaun. Transsynaptic mapping of drosophila mushroom body output neurons. bioRxiv, 2020.
[71] Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. ArXiv, abs/1910.05899 2019.
[72] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, S. Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, L. Wang, and T. Liu. On layer normalization in the transformer architecture. ArXiv, abs/2002.047455 2020.
[73] W. Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. ArXiv, abs/2100..0961, 2021.
[74] J. Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[75] Aäron van den Oord, Oriol Vinyals, and K. Kavukcuoglu. Neural discrete representation
learning. In NIPS, 2017.
[76] M. Nour. Surfing uncertainty: Prediction, action, and the embodied mind. British Journal of Psychiatry, 210:301 - 302, 2017.
[77] Yongjae Lee and Woo Chang Kim. Concise formulas for the surface area and intersection of two hyperspherical caps. KAIST Technical Report, 2017.
[78] P. Chou. The capacity of the kanerva associative memory. IEEE Trans. Inf. Theory, 35:281-298. 1989.
[79] Gordon M. Shepherd. The Synaptic Organization of the Brain. Oxford University Press, 2004.
[80] D. V. Van Essen, C. Donahue, and M. Glasser. Development and evolution of cerebral and cerebellar cortex. Brain, Behavior and Evolution, 91:158 169, 2018
[81] E. D' Angelo and Stefano Casali. Seeking unified framework for cerebellar function and dysfunction: from circuit operations to cognition. Frontiers in Neural Circuits, 6, 2013.
[82] D. Timmann and I. Daum. Cerebellar contributions to cognitive functions: A progress report after two decades of research. The Cerebellum, 6:159-162, 2008
[83] Nitish Srivastava, Geoffrey E. Hinton, A. Krizhevsky, Ilya Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn Res., 15:1929-1958, 2014.
[84] J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, (955)225542 1982.
[85] M. Demircigil, Judith Heusel, M. Löwe, Sven Upgang, and F. Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288-299, 2017.
[86] J. Hopfield and D. Tank. "neural" computation of decisions in optimizatioon problems. Biological Cybernetics, 52: 141-122, 2004.
[87] Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005.
[88] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
14

---

[89] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ (bbs/141..680,, 2015.
15


Reviews:
[{"review_id": "yxnbeDnsHhP", "reviewer": "Reviewer_PZYv", "summary": "The authors showed that the attention mechanism used in Transformers is related to the Sparse Distributed Memory, a classical computational model of associative memory in the brain. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\nThis result bears some similarity with the Ramsauer 2020 paper that connects “modern Hopfield Networks” with Attention. But other than that, it appears original.\n\nQuality\nThe authors clearly spent a lot of time on this submission. The Supplemental Materials is 55 page long.\n\nClarity\nThe description of SDM seems unnecessarily complex and somewhat confusing. I thought I understood SDM when I read Kanerva’s 1992 book, but I was kind of confused after reading the authors’ description. If SDM is almost equivalent to Attention, and Attention can be described in one line (line 122-123), why does it take two pages to describe SDM?\n\nSome description of SDM’s relationship to the brain seems unnecessarily long and not particularly insightful. For example, section 5 SDM’s Mapping to Cerebellum. It’s not clear what this section adds.\n\nSignificance\nIt is not clear what significant insights we can obtain by relating SDM with Transformer Attention. Many models are similar to one another if we make some approximations and assumptions. Attention without softmax is equivalent to a neural network with a single hidden layer and no nonlinearity.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don’t see clear limitations and potential negative societal impact beyond what’s mentioned above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "fK_oizWjXLO", "reviewer": "Reviewer_sTq7", "summary": "This paper shows that Transformer's Attention closely approximates Sparse Distributed Memory (SDM), a biologically plausible associative memory model, under certain conditions on the data normalization. The authors first provide a review of Kanerva’s SDM. Then, they show that Attention approximates SDM theoretically and empirically via the pretrained GPT2 architecture. Here, the intersection between 2 Hamming circles in SDM is approximated by an exponential function of key and query, which forms the softmax function in Attention.  This close relationship between Attention and SDM leads to a hypothesis that Attention is performing the same associative memory operations as the cerebellum. Moreover, there are discussions on related works including Hopfield Network, Memory-augmented Neural Networks and Kanerva Machine. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While Attention has become an important mechanism in deep learning, it remains unclear why it works so well. The finding of this paper may provide one explanation for Attention’s success. Prior works have pointed out the relationship between Attention and Hopfield Network, Fast-weight or Associative Memory. SDM has been used as motivations for memory-augmented neural networks.  However, this is the first time that someone have shown the close relationship between Attention and SDM both theoretically and empirically. Key mathematical techniques are the approximation of binomial coefficients using exponentials (equation (4)) and transforming the non-continuous Hamming distance to a continuous version including the cosine similarity (equation (6)). Those techniques, yet simple, are crucial to building a bridge between SDM and Attention.\nThe paper is well-written and easy to follow. People who are not familiar with the topic can still get the key ideas. Figures are clear and informative.\nQuestions to the authors:\n1.\tSDM is well-known for the ability of iterative reading, i.e. to feed the previous output of the read operation into the read operation again to get a better-match pattern until convergence. Does Attention have this ability? If yes, can it be shown theoretically or via experiments?\n2.\tSection 4 shows that components of a Transformer block are closely related to SDM. However, Transformer encoder and decoder consist of many blocks stacked together. This makes the model more powerful. How can this be explained in terms of SDM?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "UKxK23X9oyG", "reviewer": "Reviewer_1c9u", "summary": "The authors focus on attention mechanisms in neural networks, and show that under certain conditions the attention mechanism in Transformer networks can approximate a type of associative memory.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main merits:\n- The work presented is novel and it seems solid.\n\nMain limitations:\n- The paper may be organized better, as it is a bit hard to read in its present form.\n- A more thorough experimental validation would be useful..", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.75"}, "extra_text": {}}, {"review_id": "CtAc4re8tLY", "reviewer": "Reviewer_XvDB", "summary": "This paper shows that Transformer Attention approximates Sparse Distributed Memory (SDM) model under given conditions.  It further confirms that those conditions are satisfied on pre-trained GPT2 Transformer models.  In addition to theoretical analysis, some results are provided using various datasets including MNIST.  Furthermore, there is a section on biological interpretation of Attention and SDM.  ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper shows that the relationship between SDM and Attention exists because SDM’s read operation, when approximated as an exp sum of exponentials, it actually approximates softmax rule used in Attention.   They use Query-Key Normalized transformer to show the relationship to SDM, and original GPT2 models to generalize the results.\n\nThe connection between SDM and Attention was insightful, and nicely presented.  Evaluation was very limited, which his concerning as one of the difficulties of SDM-based models is precisely lack of accuracy when it comes to more complex image-based datasets beyond MNIST.   The idea that SDM/Attention model could be better interpreted through biology is very attractive, but that section in the paper seemed to be quite a stretch, and did not present convincing evidence that this is indeed the case.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows connections between the attention function and sparse distributed memory. The insights are interesting and could be useful for those who want to understand why it works so well as well as those who want to extend it further. All reviewers generally agree that this is a good paper. Reviewer PZYv has concerns about the clarity of the paper and the significance of the insights. While I agree that the writing could be improved to make it easier to read for general readers (which the authors promised they will address in the version), I believe that the insights from this paper is useful. The authors also promised to add more discussions based on feedback from Reviewer PZYv, as summarized in their author response. I recommend accepting the paper.
Decision: Accept (Poster)