Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture
Authors: Thomas F Burns
Tomoki Fukai
Christopher Earls
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) demonstrate an impressive ability to utilise information within the context of their input sequences to appropriately respond to data unseen by the LLM during its training procedure. This ability is known as in-context learning (ICL). Humans and non-human animals demonstrate similar abilities, however their neural architectures differ substantially from LLMs. Despite this, a critical component within LLMs, the attention mechanism, resembles modern associative memory models, widely used in and influenced by the computational neuroscience community to model biological memory systems. Using this connection, we introduce an associative memory model capable of performing ICL. We use this as inspiration for a novel residual stream architecture which allows information to directly flow between attention heads. We test this architecture during training within a two-layer Transformer and show its ICL abilities manifest more quickly than without this modification. We then apply our architecture in small language models with 8 million and 1 billion parameters, focusing on attention head values, with results also indicating improved performance at these larger and more naturalistic scales.

Published in Transactions on Machine Learning Research (07 (225))
Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture
Thomas F Burns
SciAT Center, Cornell University, USA
Tomoki Fukai
Neural Coding and Brain Computing Unit, OIST, Japan
Christopher J Earls
SciAT Center, Center for Applied Mathematics, School of Civil and Environmental Engineering, Cornell University, USA
Reviewed OpenReview: https: https://openrevieu openreview. net/ formm? id= LcIFm/LIRR
Abstract
Large language models (LLMs) demonstrate an impressive ability to utilise information within the context of their input sequences to appropriately respond to data unseen by the LLM during its training procedure. This ability known as in-context learning (TCT.). Humans and non-human animals demonstrate similar abilities, however their neural archi- tectures differ substantially from LLMs. Despite this, critical component within LLMs, the attention mechanism, resembles modern associative memory models, widely used in and influenced by the computational neuroscience community to model biological mem- ory systems. Using this connection, we introduce an associative memory model capable of performing ICL. We use this as inspiration for a novel residual stream architecture which allows information directly flow between attention heads. We test this architecture during training within a two-layer Transformer and show its ICL abilities manifest more quickly Ihan without this modification. We then apply our architecture in small language models with 8 million and 1 billion parameters, focusing attention head values, with results also indicating improved performance at these larger and more naturalistic scales.
1 Introduction
Transformers (Vaswani al. 2017) popular and performant class of artificial neural networks. Large Language Models (1.LMs), decorated exemplars of Transformers, have demonstrated impressive capabilities in a wide array of natural language tasks (Brown et al. 2020 One particularly notable capability, known as in-context learning (ICL), has gained significant attention. ICL, as witnessed in LLMs, occurs when a model appropriately adapts to tasks or patterns in the inference-tine input data which was not provided during the model's ss training procedure (Lynch & Sermanet 2021 Mirchandani et al., 2023 Duan et al. 2023 This ability immediately learn and generalise from new information, especially with only single or few exposures to information, is hallmark of sophisticated cognitive abilities seen in biological systems; human and non-human animals are capable of rapidly adapting their behaviour in changing contexts to achieve novel goals (Miller & Cohen 2001 Ranganath & Knight 2002 Boorman et al.) 2021 Rosenberg et al. 2021), and can infer and apply previously-unsed
and even arbitrary rules without significant learning periods (Goel & Dolan 2000 Rougier et al. 2005 Mansouri et al. 2020 Levi et al. 2024). Developing models rooted in computational neuroscience, such as associative memory, to connect our understanding of related phenomena in both artificial and natural intelligence may provide valuable insights for developing more adaptive and versatile language models, in addition to helping more deeply understand more general

---

Published in Transactions on Machine Learning Research (07/2025)
neural systems (see Appendix A for discussion the connection between associative memory models and modern neuroscience).
While various explanations have been proposed for how LLMs learn to perform ICL OOlsson et al. 2022
Von Oswald et al. 2023 Li et al. 2023 Reddy 2024), there has been little work to develop explanations which also offer heurobiologically-P segiicall-plaussble models of similar abilities seen humans and non-human animals. One exception this is the recent work Ji-Ai et al. (2024), which illustrates connection between ICL in LLMs and the contextual maintenance and retrieval model of human episodic memory from psychology. This model proposes that memory items are stored contextual composition of stimulus- and source-related content available during, and near-in-time, to memory item's presentation. The model comports with several reported psychological phenomena in humans (Lohnas et al., 2015 Cohen & Kahana 2022 Zhou et al. 2023), and can be constructed using a Hebbian learning rule (Howard et al., 2005). Hebbian learning (Hebb, 1949), that neurons which 'fire together, wire together', is foundational to the theoretical underpinnings of associative memory models [Nakano] 1972. Amari 1972. Little 1974 Hopfield 1982), which propose that
memory items are stored by strengthening the connections belween neurons which become activated during and near-in-time to the stimulus corresponding to memory items" presentations. How associative memory is
nourophysiologccall implemented is well-studied (Amit 1990 Buzsáki
2010 Khona & Ficte 2022 Burns
et al. 2022 and this is complemented by well-developed theoretical literature, inchuding work noting the
close resemblance to a core ingredient of LLMss the attention mechanisms of Transformers (Ramsauer et al.
2021
Bricken & Pehlevan 2021 Kozachkov et al. 2022 Burns & Fukai 2023
Burns 2024 Gershman et al.
2025
1.1
Contributions
Given these existing links, further developing connections between the framework of associative memory and ICT may offer deeper insights or improvements. In the following sections, we:
introduce one-layer associative memory model which can perform ICL on classification task, and which analogously allows attention values to directly represent input data;
using the same task, and inspired by our explicit associative memory model, show how creating a residual stream of attention values between attention heads in a two-layer Transformer speeds- up ICL during training (compared to the vanilla. Transformer and applying the same technique to queries or keys); and
demonstrate that naively applying the same idea in small language models (LMs) indicates perfor- mance improvements scale to larger models and more naturalistic data.
A central theme in our innovation is a focus on the role of values in the attention mechanism, and archi- lecting a simple 'look-back' method in the form a residual connection of values. Residual connections, also known as "skip' or 'shorteut" connections, can be described as those which connect neurons which are other- wise indirectly connected through a more prominent pathway. First identified in experimental neuroscience (Lorente de N6. 1938) and considered since the dawn of theoretical neuroscience and artificial neural net- works (McCulloch & Pitts 1943 Rosenblatt 1961), researchers continue to find residual connections useful in modern applications (Dalmaz et al. 2022 Huang et al.) 2023 Zhang et al. 2024b). A noticeable feature of Transformers is its use of the so-called residual stream, wherein data, once processed by the attention and feedforward layers, is added back to itself. What can therefore be considered a 'cognitive workspace' (Juliani et al. 2022) has been shown to contain rich structure, amendable to popular (Elhage et al., 2021 and emerging (Shai et al.) (0224) interpretability methods. Our work illustrates that specific additional resid- ual connections can lead to enhanced performance in ICL tasks, and we speculate may also aid in future interpretability efforts.
2

---

Published in Transactions on Machine Learning Research (07/2025)
2 ICL classification with one-layer associative memory network
2.1 ICL classification task and mathematical set-up
Let X E Rx be the input sequence data, where is the dimension of each token embedded within a suitable latent space, and a is the number of tokens in the sequence] Each token considered either an object, o, or label, I. Input X consists of a sequence of multiple pairs of objects and labels. We say pair of tokens is contiguous sub-sequence of two column vectors from within X, consisting of one object token followed by one label token. For example, the j-th pair X' E Rxx2 consists of one object token of E I followed by one label token P E RR. Our input sequence X will therefore consist of multiple pairs, and each pair may appear more than once. The final token of a sequence, denoted as 2a, corresponds to the label token of the last pair, which itself has appcared at least once prior to this final instance in X. However, the true label token data of this final label token is replaced with the zero vector, and the network's 's task is to correctly predict this true label token given the previous in-context instance of the tokens' data (as illustrated in Figure Dp).
The token embeddings, representing objects and labels, follows Reddy (2224), where all token embeddings are drawn from similar statistical distributions. For every instance of a pair, label loken embeddings come from fixed vectors, whereas object token embeddings are constructed from combinations of fixed and random vectors. Each label token I' is an dimensional vector pp, whose components are i.i.d. sampled from a normal distribution having mean zero and variance 1/e. Each object token embedding, o', is given by
o = Mot + En
vI+
where Moi, which is fixed across all instances in X of the pair, and 7, which is drawn randomly for cach instance in X of the pair, are e dimensional vectors whose components, like pis are i.i.d. sampled from a normal distribution having mean zero and variance 1/e. The variable e controls the inter-instane variability of objects and, in the following, is set to 0.1 unless otherwise stated. This mcans that the final, testod instance of a pair has, as its object token, a slightly different appearance than the previous instance(s) seen in X and is not a perfect match, where fo* provides the commonality between these variations. Adding these variations makes the task less trivial and slightly more naturalistic.
In 2.2 we show it is possible to perform ICL with such pairs in single forward step of a one-layer associative memory network, written in the language of a single Transformer attention head (Vaswani et al. 2017). To show this, and for the benefit of subsequent sections, we briefly summarize the classical Transformer set-up, In Transformers, each attention head consists of learnt parameters weight matrices W W e R and W" e xo with which, when taken together with the input data sequence X, we calculate the queries Q, keys K, and values V matrices using
@= K = W and V W"X.
The values and o are the reduced embedding dimensions for the attention operation (i.e., to facilitate multi-headed attention, etc.). Here we use ft As in the input data, is the dimension of the unreduced embedding space of the tokens.
It is then useful to define the SOFTMAX function for matrix arguments. For a matrix M E Rext we write :::MMi; : E R for the i-th row and 1 == M [ E R for the j-th column. Then, we define the SOFTMAX exp(Mit,.t:)) where i and j are the vector component function for a matrix M as SOFTMAX(M)[4+ ty] Σ, exp(MM(t.t)))
indices. We use this to compute allention-basee embeddings of the data X, denoted by X, as
X = -SOFTMXX
(1)
in which we refer to the term KJ T Q as the scores S R'X,, In Transformers and Transformer- based models such as LLMs, this new data X is then recombined with data from other attention heads, before passing through a multi-layer perceptron. Layers of attention heads and multi-layer perceptrons are stacked atop one-another to perform increasingly sophisticated computations.
We provide notation tables in Appendix
3

---

Published in Transactions on Machine Learning Research (07/ (07/)025)
2.2 Associative Memory for ICL (AMICL) model
In our associative memory model, which we call Associative Memory for ICL (AMICL), we take inspiration from the transformations applied to the input data X to create the basis of what can be considered (Ramsauer et al. 2021) as an associative memory update step in Equation I Instead of using parameterised values, we use a simple set of assignments for each token's key, query, and value vectors. For all embedded tokens i << s, we set h; = %: = a-1 as the keys and queries, where lowercase Latin letters denote indexed column vectors, taken from the matrices that are denoted with upperease Latin letters. For simplicity, we wrap the indices along the token sequence such that the first loken, #1, and the last, loken, Ia, are used to generate the sub-sequence (I,, d.e. we assign token index 0 to s. For token J, with 2, as the token column vector, we set 9. ax.-1tz the query and k2 as the zero vector, which acts as its key. For all tokens, the values column vector is equal to the token column vector, i.e., Uj value, but, after testing within the range [0, 2). is set as a = 2. (The justifications for testing a within this range and setting a = 2 is given later in this subsection.)
-
a+1
23.- The value of can be any arbitrary real
We then perform next token prediction using the universal associative memory framework Millidge et al, 2022), which can be interpreted as a generalisation of Equation | In particular, Equation in the universal associative memory framework has the form SABBBCCOO((SPPAAATTNIIIIILLAAAS SIMILARITY function is chosen as scaled dot product, the SEPARATION function is chosen SOFTMAX, and the PROJECTION function is chosen as product.
Q),, V), where the
By constructing the queries and keys in the AMICL model using what is essentially an 'average' between the current. token and the previous token, with a scalar weighting of a on the previous token, we allow the similarity function to identify the relevant pair in the context (in a similar sense to that of a 1D convolution operation); which is then potentially amplified by the scparation function (e.g., in the SOFTMAX case), for projection to the appropriate matching final token. In this way, and as illustrated in Figure [ the AMICL model essentially performs anto-association
the final pair of tokens, treating the penultimate token context clue for the final token (in the same way that auto-associative memory dynamics traditionally use partial memory identity information to complete the full, remaining pattern information (Nakano) 1972:
Amari 1972
Little 1974 Hopfield 1982)).
Normatively, this design reflects the goal the ICL task: to recall or reconstruct the most appropriate output token given its prior contextual association. Using the identity function ensures that what is retrieved is the token itself, not transformation of it, mirroring the associative memory setting where retrieval should produce the original pattern.
The reason for choosing the values to have the identity of the original data, then, is because we cast this ICL task as an auto-association problem, and by the universal associative memory framework (Millidge et al., 2022), the projection function in the case of auto-association is the identity function (also see $3.1 of Millidge et al. (0222).
From biological perspective, this is also consistent with Hebbian learning in attractor networks, where activation patterns corresponding to past stimuli are reactivated as-is during recall (Nakano) 1972 Amari
1972
Little
1974:
Hoplield 1982.. rather than being replaced by arbitrary transforms.
This then justifies the choice of exploring the range of a in the positive reals only, since although negative values are mathematically possible, they would create an anti-Hebbian or repelling force, counter-acting the designed (Hebbian) auto-association task created by the query and key constructions.
As an intuitive sketch, AMICL can be thought of as implementing the algorithm illustrated in Figure [ first, we identify the final pair, where the final token, 2,, which should be a label token, has been set to the zero vector (Figure D,, where '?' represents the unknown token data and other tokens' data are shown); second, we consider all possible contiguous pairs in the context, e.g., (Iz, xa), (2z, zz)...., without knowledge of their data, which we as designers know will correspond to pairs like, e.g., (o", (P,P), (12, o²), (o², (2)....... (Figure D. where each contiguous pair is enclosed by a yellow rectangle); third, we compare all of the contiguous pairs in the sccond step with the final pair identified in the first step (which we are attempting to 'pattorn completo"). and, upon seeing which context pair is most similar in final pair (in this example, (o' (o), complete the pattern appropriately by setting x., 1 (Figure
4

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
1
o²
!
[3
o²
o²
b)
o²
2
0
12
O
[3
[
o²
01
[1 0 2
12
1
0
1
Figure 1: Depiction of the AMICL algorithm on label object pairs, consisting of three steps: (a) consider the penultimate and final tokens as a 'local pattern' which has been 'partially corrupted' by the missing data in the final token, illustrated by the black rectangle enclosing this final pair; (b) search for matching, 'complete local patterns' by grouping all previous contiguous token pairs in the context, with each pair illustrated by a yellow rectangle enclosing the tokens; and (c) complete the final, corrupted local pattern based on matching to the nearest complete in-context pair, which in this case is (o', I so the final token data is assigned l!.
Varying the parameter a between 0 and 2 for combinations of similarity and seperation functions showed that using the Dor PRODUCT similarity with ARGMAX separation function resulted in practically perfect ICL ability at a 2 1.5 whereas using PEARSON's CORRELATION similarity with ARGMAX separation function saturated at a performance level of 85% accuracy for the ICL pairs task (see Figure 12 in Appendix and the saturation of TCT. performance PEARSON's CORRELATION both saturating at values at 2 1.5 we set 2.
Cp. Due to the practically perfect TCT. performance in the DOT PRODUCT
While setting 2 and PROJECTION IDENTITY, we tested all combinations of:
SIMILARITY E (DOT PRODUCT, PEARSON's CORRELATION, MANHATTAN DISTANCE, EUCLIDEAN DISTANCE); with
SEPARATION e (IDENTITY, SOFTMAX, ARGMAX).
Manual inspection of the resulting allention matrices indicated that the DOT PRODUCT and PEARSON's CORRELATION similarity functions combined with the ARCMAX separation function provided the cleanest ICL (sce Figure 11 in Appendix C for an example). Varying the token sequence length is be- tween 10 and 1, 1,000 shows no variation in performance; varying the embedding dimension of the tokens between 10 and 1, 000 showed that for 2 50, the ICL task performance was practically perfect (see Figure
13 in Appendix
3 Residual attention streams in two-layer Transformer
3.1 Residual attention streams
In the AMICL model, following the path of information from the input to the queries, keys, and values (as illustrated in Figure 2),, we can see that the queries Q and keys K flow from a shared function f of the input X. Whereas, the values are given directly by the input X. In the language of the universal associative memory framework (Millidge al. 2022). we can say that the similarity and separation factors (determined by the queries Q and keys K) are coupled by the shared function f whereas the projection factor (given by the values V) is simply the input itself, i.e., auto-association.
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Seen through the lens of the traditional self-attention mechanism, this implies that similar construction of the queries and keys is possible, such that using simple identity function from the input for the values could replicate this behaviour in more sophisticated data, task, and model scenarios. However, in our initial experimentation with this idea, whereby we eliminated the W" matrices altogether, we witnessed significant model performance degradation and training instability (see Appendix
B also principally limits the model's expressiveness in the self-attention mechanism to auto-association, whereas many interesting tasks make of some amount of hedero-association see Burnss (0024). This led us to consider an alternative form of projection, V which more explicitly retain the prior input X. In particular, we introduce a residual connection between the values data of successive layers in the Transformer, which we call a residual values stream and, more generally, a residual allention stream applied to values (Figure p)
which could in principle mix auto- and hetero-association, via the creation of values
b)
Xn-1
X
K
V
Q
K
X,
W
Qn
K₁
Vn
Figure 2: Partial diagrams of (a) the AMICL model and (b) our residual attention stream architecture with two Transformer layers, n and n 1, shown here implemented for values (the added residual is shown green). Boxes represent variables. Full arrows represent functions, with overlaid circles indicating relevant variables or functions (no circle is used for the identity function). Dotted arrows represent functions and variables omitted in the diagram for space.
As an alternative source of intuition, one can consider the informal interpretation of self-attention as con- sisting of: queries 'asking each token a question about other tokens'; keys 'responding to each token with the answer to the queries' questions', i.e., they align with or 'attend to' the queries; and values giving the (weighted) answers those questions, which are operationalisod as small additions to the current token vectors. Within this informal conceptual model, we can consider our residual atlention streams as retaining additional bock-back' information in the answers.
Our residual attention stream also generates additional gradient signals during training, which could them- selves be beneficial for the nefwork to develop ICL capabilities. We therefore also test the same residual stream architecture, separately, for queries and keys. In principle, is also possible to apply the residual stream architecture combinations of queries, keys, and values. But, to maintain stronger connection the AMICL model, we focus on testing the residual value stream separately, as well as the keys and queries, again separately, for comparison.

---

Published in Transactions on Machine Learning Research (07 (225))
3.2 Two-layer Transformer architecture
We Irain classic and modified versions of two-layer Transformer the same task described in Section 2 Following Reddy (2024), the common architectural features between the two versions consist of two single- head attention layers followed by a throe-layer multi-layer perceptron with 128 ReLU ncurons followed by a softmax layer to give probabilities over the l labels. The network trained with the same task as the AMICL model, using the cross-entropy loss between the predicted and actual final label token I3. Within the modified architecture, we add the first attention head's queries, keys, or values to the second attention head's queries, keys, values, respectively. More formally, in our modified version of the Transformer architecture, we calculate the first attention layer as
Q1 = Wfx, = = and V1 = W
and then, in the second attention layer, for residual queries stream calculate
Q2 = x + Q2,
K2 = W and V22 = x
or for residual keys stream calculate
Q2 = W K2 = W + K1, and
1/2 - W
or for residual values stream (shown in Figure 2b) we calculate
Q2 = W
K2 = w and V/2 = W + Vi.
3.3 Supplemental ICL tasks
As in Reddy 2024h, while we train the originally-dcseribed ICL task, create supplemental tasks which are not used for training but rather act proxy measurements of progress for different computational strategies for completing the task: training data memorisation and ICL capability generalisation. Namely, these supplemental tasks are:
In-weights (IW): a series of object-labed pairs is presented where the final pair is not found within the prior context but is present in the training data;
In-context (IC): a series of novel object label pairs, not seen in the training data (i.e. an entirely re-drawn set of Po and pr values) but following exactly the same statistical structure, is presented; and
In-context 2 (IC2): a series of object label pairs are presented, where the objects are found in the training data but have been assigned new labels (i.e., the objects retain their Hox values but the labels have their ue values re-drawn).
The reduced embedding dimensions of the attention operation are both set to 128, i.e., 4 = 128. Fach network architecture was trained on four random seeds, with batch size of 128, vanilla stochastic gradient descent, and learning rate of 0.01.
3.4 Performance of the attention residual streams on the ICL classification task
Figure shows the accuracies for the task being trained (Test) and the three supplemental tasks (IW, IC, and IC2) for each architecture. It also shows the Test loss for all architectures and all loses for the values residual stream architecture. For our residual attention stream modifications, we observe general leftward shift in all but the IW task, with the value residual stream architecture showing the largest shift. The IW task also shows a slower learning rate, which can be attributed to the relative difficulty of the network memorising the training data.
To quantify these shifts, we report statistics in Table I of when the first training snapshots we recorded reached accuracy threshold of 0.95. We find the values stream networks perform best, reaching the
7

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Test
IW
architectures
1.0
1.0
0.8
0.8
Accuracy
0.6
Accuracy
0.6
Test loss
Classic
Queries
Keys
0.4
Classic
0.4
Classic
Values
Querles
Queries
0.2
Keys Values
0.2
Keys
Values
0.0
0.0
100
150
50
100
150
50
100
150
IC
IC2
Values residual stream architecture
1.0
1.0
0.8
0.8
Accuracy
0.6
Accuracy
0.6
Loss
2-
Test
IC
IC2
0.4
Classic
0.4
Classic
IW
Queries
Querles
0.2
Keys
0.2
Keys
Values
Values
0.0
0.0
100
150
100
150
Training snapshots (taken every 500 steps)
Figure 3: Accuracies and losses for the Test, IW, TC, and IC2 tasks training time for the classic (unmodified) network and residual queries, keys, and values stream networks. Dark lines represent means, light lines represent individual trials.
same level of accuracy the classic (unmodified) networks with 24% fewer training steps. The same result is also seen at lower thresholds (see Tables 5 and 6 in Appendix D for thresholds of 0.5 and 0.9, respectively). We also performed t-tests at the 0.95 accuracy threshold, which showed significant differences between the performance of the classic (unmodified) network and the residual queries and values stream networks on the IC and IC2 tasks (p < 0.01, / < -3.9), but not between the classic and residual keys stream networks (p 0.39, t > -10).. The residual values stream networks also perform significantly better (p 0.03, t 3.00) than all other networks both the IC and IC2 task, except the residual queries stream network for the IC task (p 0.06, t 2.33)..
Table 1: Mean + standard deviation of training snapshot number where accuracy first exceeded 0.95 for the IC and TC2 tasks in the classic (unmodified), and residual queries, keys, and values stream networks. Fastest training times are bolded.
Classic
Queries (ours)
Keys (ours)
Values (ours)
IC
64.5
5.12
52.5 +
61.75 0.83
49.0 + 2.12
IC2
63.25
4.15
51.75 + 1.92
61.25 0.83
47.75 1.3
4 Residual value streams in a toy language model
4.1 Toy language model architecture and training details
To provide an initial indication as to whether the benefit for ICL performance seen in Section D scales to larger models and naturalistic data, we also test the residual value stream architecture by training toy LMs. The Transformer-basee model contains approximately 8 million parameters spread over eight layers, each with 16 attention heads. The context window is 256 tokens long and dimension of the model is 256, Le.

---

Published in Transactions on Machine Learning Research (07/2025)
each token is mapped to a 256-dimensional vector. Next-token prediction is trained for three epochs on the Tiny Stories dataset, such that they can generate simple and short children's 's stories (Eldan & Li, 2023). Each network architecture was trained in three separate instances, using different random seeds for each instance but controlling for randomness between architectures by re-using the same set of seeds for the two architectures.
We implement the residual value stream in a naive way: in each but the first layer, the values of each attention head receives additional input of the values from the attention head with the same index in the previous layer Although there are alternative implementations of such residual streams, chose naive and straight approach to provide an initial indication of the potential scalability of this architectural change and to not alter the number of learnt parameters between the networks.
The residual values stream networks achieved 1.65 training loss and L validation loss, slightly lower than the training and validation losses for the classic networks, which were 1.67 0.05 and 1.56 0.01, respectively. Measured by wall clock computation time the residual values stream networks also took slightly longer to train, 30.18 = 0.49 hours compared to 29.12 + 0.40 hours for the classic networks. This additional computation time is attributable to the additional gradient computations required by the 16 additional residual streams connecting the attention head values at cach but the first layer.
4.2 Evaluation of ICL ability using simplistic natural language task
As a proxy evaluation of the ICL ability of each model in natural language, we utilise a simple indirect object identification (TOT) task. In natural language, direct object is the noun that receives the action of the verb and an indirect object is noun which receives the direct object, e.g., in the sentence "A passed B the ball": "passed" is the verb; "A" is the subject; "the ball" is the direct object; and "B" is the indirect object. In the TOT task, we test. for correct completion of sentences like "When A and B were playing with a ball, A passed the ball to", where " B" is the indirect object and considered the correct complction.
GPT-2 Small can perform instances of the IOI task, and the circuit responsible for this ability has been identified (Wang et al.) 2023). As shown in this section, our toy LMs have some measurable ability to complete instances of this task, and so use this to compare the performance of the classic (unmodified) and residual value stream archilectures.
For the 101 task, we tested the following sentences:
1. "When John and Mary went the shops, gave the bag to", where was either "John" or "Mary", with correct completions " MMay"" and " "John", respectively.
2. "When Tom and James went to the park, * gave the ball to", where was either "Tom" or "James". with correct completions " James" and " Tom", respectively.
3. "When Dan and Emily went to the shops, * gave an apple to", where * was either "Dan" or "Emily", with correct completions " "mily" and " Dan", respectively.
4. "After Sam and Amy went to the park, gave drink to", where was cither "Sam" or "Amy", with correct completions " Amy" and Sam", respectively.
For each sentence and variation thereof (swapping the identities of the subject and indirect object, as indi- cated by the symbol above), we recorded the next token probabilities of the correct and incorrect names. As summarised in Table we find the classic networks are much less capable than the residual values stream networks across the four sentences, the classic networks correctly identify the indirect object with prob- ability of ~ 7% while the residual values stream networks do so with a probability of ~ 41%, ~ 590%
Principaally the choice of which attention heads form the residual values stream is arbitrary given the independent nature of the same head index for computational
each head. However, when implemented with multi-head attention, it preferable convenience.
Using a PC equipped with an AMD Ryzen Threadripper PRO 3975WX 32-Cor Cores CPU, 130GB of RAM, and 4x NVIDIA RTX 44000 GPUs.
9

---

Published in Transactions on Machine Learning Research (07 (07/2025)
improvement. Similarly, the classic networks more regularly mistake the subject for the indirect object with probability of ~ 5% while the residual values stream networks do so with probability of ~ 3%, a ~ 60% reduction.
Table 2: Mean + standard deviation probabilities (%) of correct and incorrect responses to each sentence
for the IOI task for the classic (unmodified) and residual values stream networks. Best scores bolded.
Classic
Sentence
Sentence 2
Sentence 3
Sentence 4
Correct (higher better)
11.73 15.62
11.88 + 9.09
3.83 6.09
1.54 + 2.31
Incorrect (lower is better)
6.93 5.70
7.99 + 9.52
0.51 + 0.66
4.86 + 8.35
Residual values stream (ours)
Sentence
Sentence 2
Sentence 3
Sentence 4
Correct (higher better)
42.89 + 10.58
43.44 + 14.45
49.24 + 9.89
28.56 5.92
Incorrect (lower is better)
5.68 = 3.76
6.8 = 10.40
0.03 + 0.03
0.18 + 0.15
We propose the following explanations for why the residual values stream variant outperforms the classic model so handedly despite the two models having very similar final loss values and the former having only slightly smaller loss:
especially in small models, even small differences in the training loss can translate large differences in the test accuracy on downstream tasks (Tau et al.] 2023
algorithmically, models with only very small even no loss differences can perform the task in principally different ways (Power et al. 2022 Bushnaq et al. 2024 and
provision of the residual values stream improves sample complexity for the 101 task, which while implicitly present in the rugo-segressvve training set-up, is not what is primarily mcasured by the training loss (which is more generally measuring performance on next-token prediction, implicitly consisting of much larger variety of natural language tasks).
5 Residual value streams in small language model
5.1 Small language model architecture and training details
To test the performance of residual value streams on a larger scale, we trained two milliilion param- eter (1B) language models in the style of a Llama 3 model (Grattafiori et al. 2024). These models were trained on English-language documents, amounting approximately 84 billion tokens, from random subset of the Nemotron-CC-Ii 2024a). As in the toy language model described in $ the only way in which these models differed was in the presence or absence of the residual values stream. As in the MM-parameter models, the 1B models showed no significant. difference training loss (sce Figure
Residual values stream Classic
datasct (Su et al.)
Loss
4-
2-
10000
20000
25000
30000
35000
40000
Step
Each model comprised of 16 layers, 2,048 hidden di- mensions, 32 attention heads, and eight key-value heads. These models support context length of 4,096 lokens and incorporate several modern ar- chitectural features: rotary positional embeddings (RoPE) (Su al. (024b) with base of 500,000, SwiGLU activation functions (Shazeer 2020 with a multi-
Figure 4: Training loss over steps for the 1B models.
layer perceptron (MLP) expansion factor of four, and RMSNorm for layer normalisation [Zhang & Sennrich 2019).
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
The model trained for 40,000 iterations with global batch size of 512. We used the AdamW optimizer (Loshchilov & Hutter 2019) with 31 = 0.9, 822 0.95, 1e - 8, and gradient clipping at 1.0. The learning rate followed a cosine decay schedule with an initial rate of 3e-4, 500 warmup steps, 5,000 cooldown steps, and minimum learning rate of 0.0. The model was trained using the Llama-3.1-8B tokenizer (Grattafiori et al. 2024), which has a vocabulary size of 131,072 tokens. We used BF precision throughout.
Tables
and summarise the training parameters.
Nvcabb
Layers
Wheads
dmodel
dMLP
Batch Size
Sequence Length
131,072 16
32
2,048
8,192
512
4,096
Table 3:
used in the 1B models.
Optim. B1
32
Nwarmpp
NCooldown
learning rate
AdamW
0.9
0.95
1e-8
1%
10%
3e-4
Table 4: Optimizer parameters for the 1B models.
5.2 Small language model evaluation and results
After training these models, we evaluated them on single- and five-shot language understanding tasks,
0.550
namcly: AI2 Reasoning Challenge (ARC) (Clark
Classic
iet al.] 2018), Physical Interaction: Question An-
0.525
Residual Value Stream
swering (PIQA) (Bisk et al.) 2019)), OpenBookQA
(Mihaylov ct al. 2018). and HellaSwag (Zellers
Average Accuracy
0.500
et al. 2019).
0.475
Figure 5 shows the average accuracy of the resid- ual value stream architecture compared to the clas- sic baseline, showing modest but sustained improve- ments over our single- and five-shot language under- standing tasks. We observed very high correlation between the two models' performance (0.993), which is expected for paired samples evolving over training steps with the same training data. We conducted a paired two-sample for means I-test on the aver- age accuracy and found the p-values (both one-tail and two-tail) are much smaller than = 0.05 (the t-critical one- and two-tail values were 1.694 and 2.037, respectively). Therefore, we may reject the null hypothesis, i.e., the mean difference between the paired samples is not zero. Given this, we con- firm the improvement in performance at the 1B scale is statistically significant across these tasks.
0.450
0.400
0.375
0.350
0
5000 10000 1000015000200000 20000 2500030000 35000
Training steps
Figure 5: Average accuracy across training steps for our evaluations (single- and five-shot). Individual eval- uation results can be found in Figure 14
However, these gains appear localisod to particular tasks. In Figure 14 we can see that the most noticcable gains are in the ARC and OpenBookQA tasks whereas there almost no change in the PIQA and HellaSwag results. We therefore also evaluated our model on extended version of the IOT task, which dub TOL- Hard, In the HOI-Hard task, there are 25 sentences like:
'(A} and (B) were commuting to the (PLACE).'
'After lunch, {A} and (B) went to the (PLACE)..
11

---

Published in Transactions on Machine Learning Research (07/2025)
'The (PLACE) {A} and {B} went to had (OBJECT)."
FFiiends {A} and (B) found {OBJECT) at the (PLACE)."
'(A} and (B) were planning to visit the (PLACE)."
where {A} and (B) are a unique pair of person names, {PLACE} a unique physical place (e.g., park, library, coffee shop, elc.), and {OBJECT) a unique physical object (e.g., book, pen, flower, key, etc.). Following 25 unique sentences of this type, there is a final, unfinished sentence designed to test if the model can correctly identify the indirect ob- ject. The final sentences randomly selected pair of person names occurring in one the previous 25 unique sentences, and matched the place and object mentioned) in one of the following final sentences:
1.0
0.8
Accuracy
0.6
0.4
'At the (PLACE), (GIVER) wanted to give a (OBJECT) to'
0.2
At the [PLACE), [GIVER] decided to give the {OBJECT) to"
Residual Value Stream
0.0
5000 10000 15000 20000 25000 30000 35000 40000 Training steps
"Then at the (PLACE), (GIVER) gave [OBJECT) to'
Figure 6: Accuracy across training steps for 1,000 com- plctions of the IOI-Hard task evaluation.
'While at the (PLACE), (GIVER) handed (OBJECT) to"
where the (PLACE) and (OBJECT) was before, the (GIVER) was randomly chosen (A} (B), and the correct completion was (A) if (B) was cho-
sen as the (GIVER] and vice-versa.
Figure shows the accuracy of the 101-Hard task over training steps for both models, where 1,000 randomly- generated prompts of the above style were created, using 32 unique names, 30 unique objects, and 30 unique places. In this task, which much more directly lesting ICL ability, we see a very clear performance gain in our model compared to the classic model.
6 Discussion
Our study builds upon the understanding of ICL in Transformer models by exploring connections asso- ciative memory models from computational neuroscience. We introduced AMICL, associative memory model that performs ICL using approach akin to single-layer Transformer allention head but with an implied residual stream from the inputs to the values. This itself is notable given ICL, is not typically seen in single-layer Transformers OOssoon et al.) 2022) outside of simple linear regression tasks (Zhang et al. 2024a Lu et al. 2024). Inspired by this, we proposed a novel residual stream architecture in Transformers, where information flows directly between attention heads. We demonstrate this a two-layer Transformer, showing showwin increased efficiency in learning on an ICL classification task. Moreover, by extrapolating this architecture small Transformer-based language models, we illustrated enhanced ICL capabilities are possible a larger and more naturalistic scale.
6.1 Connections computational neuroscience
Our results offer potential insights for neural network design and our understanding of biological cognition:
12

---

Published in Transactions on Machine Learning Research (07/2025)
Neural network architecture. The simplicity of our neuroscience-inspin architectural mod- ification, attention residual streams, provides a promising extension for designing more adaptive Transformer models. Given one can interpret this change as providing additional shared workspaces across model layers (in the form of the additional residual streams), our results can also be viewed as demonstrating another example where functional modularisation can provide a performance benefit (Kaiser& & Hilgetag 2006; Chen et al. 2013), which also seen cases of highly distributed repre- sentations (Viiov & Mrsic-FF Flogel| 2022), of which natural language appears no exception (Mikolov
ct al. 2013 Hernandez et al. 2024
Biological cognition. Drawing parallels between artificial networks and cognitive neuroscience theories contributes a deeper understanding of memory systems in biological entities. As we have shown by our AMICL model, the associative memory framework is capable of ICL in single layer, with an implicd 'skip connection' present between the imput and the values. This suggests that similar lypes of connections may analogously exist in biological networks to adapt and generalize from limited exposure. For instance, associative memory is often related to the hippocampal formation [Aitt] 1989), an area in the brain important for memory and learning. An open question in this area of neuroscience is what the compulational role of observed skip connections are. In particular, while some information in the hippocampus goes directly from area CA3 to CA1, another area- CA2 is often skipped, but does receive input from CA3, which it then passes on to CA1 (Cui ct al. 2013). This is interesting not just anatomically, but also functionally, since CA2 is considered responsible for specialised tasks and context-switching (Dudek et al., 2016 Robert et al. 2018) Whether our AMICL model or residual attention stream modification can be considered analogous to skip connections witnessed in the hippocampus remains to be sludied. However, it offers a promising window, e.g., one could now study the effects of changing the mumber and quality of residual attention streams, to test whether connection types or relative proportions similar to that seen in the hippocampus also show improved performance (for some types of data or tasks). We also suggest experimental neuroscientists perform classical ablation study of CA3 to CA1 skip connections at different intensities and measure the performance of an object- pair recognition task with the same structure as described in 22.1 and $3.3 We hypothesise that the strength and number of remaining CA3 to CA1 skip connections will correlate with performance on the IC and IC2 tasks but not the IW task. Further, we also hypothesise the effect size of this performance difference will correlate with the effect size of the difference in performance causod by ablation of CA2 itself, since we theorise these CA3 to CA1 skip connections are mostly valuable only in the presence of CA2 to CA1 connections.
In establishing an additional potential mechanistic link between associative memory frameworks and ICL tasks, this work builds upon the broader nouroscience-insprred interpretation of Transformer attention mech- anisms (Ramsauer et al. 2021). Indeed, /hhoo 2023) conjectures that LLMs performing ICL do so using an associative memory model which performs kind of pampenn-oompltton using provided context tokens, conditioned by the learnt LLM parameters. Using this perspective, Zhao (2023) constructs different token sequences to be used as contexts to prepend onto the same final token sequence representing the tested task. By actively choosing context tokens which more 'closely' resemble the final tokens the LLM is being tested with, the LLM shows improved task performance, presumably by utilising the increased relevancy of the context tokens for ICL. This perspective is further developed in Jiang et al. (2024h, who show how LLMs can be 'hijacked' by purposeful use of contexts with particular semantics. When L.LSs such as Gemma-2R-IT and LLaMA-7B are given the context of "The Eiffel Tower is the city of", they successfully predict the next token as Paris". However, JJiang et al. (2024) demonstrate that prepending the context with the sentence "The Eiffel Tower is not in Chicago." sufficient number of times, these LLMs incorrectly predict the next token as Chicago". We may interpret these prepended data as acting (crudely) as 'distractors', and in the associative memory sense as causing an over-activation of competing memory items which interferes with accurate task performance.
Beyond associative memory, our architecture also shows structural similaritics to other models from com- putational neuroscience. Notably, AMICL shares characteristics with successor representation (SR) models, particularly models where discount factor governs the contribution of future states (Janner et al. 2020
13

---

Published in Transactions on Machine Learning Research (07/2025)
In AMICL, the parameter plays similar role, and in all of our implementations of the value residual stream is implicitly set to 1 (since we do not weight the added residual stream). Altering or weighting the value residual stream would enable control over the influence of past representations within the residual stream. This connection suggests that our model may be modified to encode a temporal structure over repre- sentations, akin to how SRs integrate over future expected states. Furthermore, recent work has highlighted parallels between SRs and temporal context models as applicd to LLMs (Ji-An et al.] 2024), particularly through the lens of reinforcement learning (RL) and temporal difference learning (Sutton 1988; Gershman et al., 2 These insights open path for future work: instead of passing values from single attention head, the model could be extended by learning weighted combinations of values across heads and layers with the weights themselves learned via RL-based methods further enriching the capacity to encode temporal or relational structure.
6.2 Potential representational benefit of residual value streams
The representational benefit of the residual attention stream modification, specifically the residual values stream, may be that it enhances the model's ability to retain and propagate contextual information across layors in a more persistent and structured way. In standard Transformers, attention values are rocomputed independently at each layer, potentially discarding loo much in the form of intermediate representations, which themselves retain relevant information regarding imput associations. By directly passing the attention values from one layer to the next, the model effectively gains memory-like mechanism that may help preserve previously computed associations, which could be particularly beneficial for tasks requiring ICL.
This residual pathway may enable a form of representational continuity: information that was relevant in earlier layers remains accessible in later layers, even if the current query-key similarity focuses elsewhere. This could allow the model to better align and integrate information across time or token positions, supporting pattern completion and associative recall, which can be interpreted as core functions of TCL. This may allow for more robust binding between stimulus features and outcomes, especially when only partial cues are available in context. Thus, the modification potentially enriches the model's internal representations with Longer-range more stable associations that are critical for adaptive, few-shot generalisation.
6.3 Limitations and future work
A key limitation of this paper is that, while it introduces a novel and neuroscience-inspired architectural modification and demonstrates improved ICL performance across synthetic tasks and small-scale language model, its generalisability remains uncertain. The performance improvements, though statistically signifi- cant, are modest and task-specific when evaluated on broader benchmarks, such as PIQA and HellaSwag, where the proposed architecture shows minimal gains. Additionally, the residual stream mechanism is im- plemented in a relatively naïve fashion, leaving open questions about its optimal design and scalability to larger, more complex models. The paper also does not explore potential trade-offs, such as computational cost versus benefit or impacts on tasks unrelated to ICL, limiting conclusions about its broader utility in real-world applications.
Future efforts may seek to more deeply understand our results in the context of varying the structure and order of context tokens in associative memory networks, and during inference and training of LLMs. Along these lines, Russin et al. (2024) recently showed that LLMs and Transformers exhibit similar improvements as seen in humans on tasks with and without rule-like structures, depending on the order and organisation of context and training tokens. In particular, ICL performance improved when context tokens appcared in rennantcallyy-ellvannt 'chunks' or 'blocks', as seen in humans when completing tasks with rule-like structures. Whereas, when training samples were interleaved, Transformers saw performance improvements (as measured by the extent to which the network rote-learnt training examples), similar to humans completing tasks lacking rule-like structures.
Other recent works have demonstrated how Transformer based language models store and retrieve knowledge, using synthetic tasks to isolate specific mechanisms (Bictti et al. 2023 Nichani et al.] (024)). Bictti et al. (2023) analyse how Transformers balance global knowledge learned during training with context-specific knowledge acquired during inference. They show that weight matrices function as associative memories,
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
with different learning dynamics for global bigrams and in-context bigrams, and provide theoretical insights into how gradients support this process. Nichani et al. (2024) focus on factual recall, proving that shallow Transformers can achieve near-optimal storage capacity using either self-attention or MLP components as associative memorics. They also show that such models can trade off between these components and exhibit sequential learning dynamics during training. We believe could by highly valuable for future work analyse how such learning dynamics vary the case of the residual values stream modification, and explore whether an analogous modification could be made to the MLP components.
The performance improvements from introducing residual values stream suggests that ICL can be thought of as an associative process where continuous information reinforcement enhances model memory, efficiency, and prediction accuracy on novel data. Nonetheless, this raises several questions. Firstly, more comprehen- sive testing on varied datasets and different scales of language models can address whether the improvements we observed generalise beyond our specific tasks and data setups. Further, remains to be clarified whether similar benefits manifest when dealing with other natural language processing tasks, such as sentiment anal- ysis translation, or whether there exist, any trade-offss between ICL and other abilities. Additionally, while these initial results present compelling further exploration of optimisation parameter settings for such architectures would strengthen understand- ing. Quantitative studies on computation cost versus accuracy improvements will also better-inform model architecture design and selection for deployment in real-world scenarios, as well as potential competition between learning flexibility and task accuracy.
for testing attention residual streams in larger models,
6.4 Conclusion
This research demonstrates potential conceptual and practical advancements in enhancing Transformers' adaptive capabilities through a neuroscience-inspee inspiired mechanism. By bridging neural computational princi- ples with associative memory insights, we offer new directions for research into more intelligent and dynamic models, with potential improvements for LLMs. Our associative memory model and Transformer architec- ture not only bolsters existing computational frameworks, but also offers fertile ground for computational neuroscientists to analyse the computational role of skip connections in memory systems. As progress, these interdisciplinary approaches may ultimately yield richer cognitive models that parallel, or even emulate. biological intelligence.
Acknowledgments
This effort was supported by the Cornell University SciAl Center. T.B. and C.E. were funded by the Office of Naval Research (ONR), under Grant Number N00014-23-1-221 T.F. supported by JSPS KAKENHI no. JP23H05476. T.B. thanks Gautam Reddy for code sharing, Björn Deiseroth for computing support, and Fabien C. Y. Benureau for helpful discussions.
Code availability
Code
used
in
this project
be
found
https://github com/tfburns/
AMICL-and www.ieaa--atttttttiioiooooo
15

---

Published in Transactions on Machine Learning Research (07 (225))
References
Shun'ich Amari. Learning patterns and pattern sequences by self-organizing nets of threshold elements.
IEEE Transactions on Computers, C-21(11::117-1206, 1972. doi: 10.1109/-CC1992.233477
Danicl J Amit. Modeling brain function: The world of attractor neural networks. Cambridge university press, 1989.
Daniel J Amit. Attractor neural networks and biological reality: associative memory and learning. Future Generation Computer Systems, 6(2):111-119, 1990.
Daniel J. Amit, Hanoch Gutfreund, and H. Sompolinsky. Storing infinite numbers of patterns in spin-glass model of neural networks. Phys. Rev. Lett., 55:1530-1533, Sep 1985. doi: 10.1103/PhyseewI.co Ptys
URL https://link aps. org/ doi//0.//1///////// 1103//hyyRevLett..56 1530
Dmitriy Aronov, Rhino Nevers, and David W. Tank. Mapping of non-spatial dimension by the
hippocampal-entorhin circuit. Nature, 543(7647)::199-72 Mar 2017. ISSN 1476-4687. doi: 10.1038/ nalure21692.
Xiaojun Bao, Eva Gjorgieva, Laura K. Shanahan, James D. Howard, Thorsten Kahnt, and Jay Gottfried. Grid-like neural representations support olfactory navigation of two-dimensional odor space. Neuron, 102(6):0066-175.c,, Jun 2019. ISSN 0896-6273. doi: http://.ccrro..0119... org/ 10166/ neuron. 2019. 034
URL /tttss//p://doi
Alison L. Barth and James F A Poulet. Experimental evidence for sparse firing in the neocortex. Trends in Newrosciences, :5(5) 355, 2012. ISSN 01662236. doi: 00.106//..ins.2012..3.
Jacob L. S. Bellmund, Peter Gärdenfors, Edvard I. Moser, and Christian F. Doeller. Navigating cognition: Spatial codes for human thinking. Science, 362 (6415) 2018. doi: 10.1126/science.ae science.aat6766: URL https://wwx
science.ec... peclence.org/donn wttp/oo//as//......oooo http:////bbbb/0111///eeeeeeee/// www.:/111///cceeeeeeeee aat6766
Alex Bellos He ate all the pi Japanese man memorisos to 111,700 digits. The Guardian, 2015. URL httpp///ww...ggggacacccc/cccc/ca
and/2015//2/ -kirr-aaaaaaaaahhiii haraguchi,
day- menooryyeeeorrrttiioooooodddeee- japaness-dappeaeeeeeeeee akira-aiiiaaaa
Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36:1560-155 15800,158800000 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense natural language, 2019. URL httpp:: wttp://.wrrgaaaaa/11..... 11641
T. V P Bliss and A. R. Cardner-Medwin Long-lasting potentiation of synaptic transmission in the dentate area of the unanassthetized rabbit following stimulation of the perforant path. The Journal of Physiology, 232(2):357 374, 1973. ISSN 14697793. doi: 00.113/jjhhyste 19t.lysiiol.19....oo 1973.sp0102744
Erie D Boorman, Sarah C Sweigart, and Seongmin A Park. Cognitive maps and novel inferences: a flexibility hierarchy. Current Opinion in Behavioral Sciences, www.weeeeeeeeee.cccc....... wtwp:e/wweeeeeeeteoeeeeoooeeeeee http://cieenn/arricleeee doi.org/ 1oi.orr/10.100 523621546210003355
2021. ISSN 2352- 1546. doi: https://
URL https: httt:::/www.ssiieeeddrrr...c.. Computational cognitive neuroscience.
Melina Paula Bordone, Mootaz M. Salman, Haley E. Titus, Elham Amini, Jens V. Andersen, Barnali Chakraborti, Artem V. Diuba, Tatsiana G. Dubouskaya, Eric Ehrke, Andiara Espindola de Freitas, Guilherme Braga de Freitas, Rafaella A. Gonçalves, Deepali Gupta, Richa Gupta, Sharon R. Ha, Is- abel A. Hemming, Minal Jaggar, Emil Jakobsen, Punita Kumari, Navya Lakkappa, Ashley P. L. Marsh, Jessica Mitlöhner, Yuki Ogawa, Ramesh Kumar Paidi, Felipe C. Ribeiro, Ahmad Salamian, Suraiya Saleem, Sorabh Sharma, Joana M. Silva, Shripriya Singh, Kunjbihari Sulakhiya, Tesfaye Wolde Tefera, Behnam Vafadari, Anuradha Yadav, Reiji Yamazaki, and Constanze I. Seidenbecher. The energetic brain a review from students students. Journal of Neurochemistry, 151(2)) 2019. doi: https: 14829
/ (tbbi.org/10.11111.. 14829. URL. htap:::///olillll.rrrrrrl..lll
16

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Timothy F Brady, Talia Konkle, George A Alvarez, and Aude Oliva. Visual long-term memory has massive storage capacity for object details. Proceedings of the National Academy of Sciences, 105(38))14335-1144 2008.
Trenton Bricken and Congiz Pehlevan. Attention approximates sparse distributed memory. In A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Jehoshua Bruck and Vwani P. Roychowdhury. On the Number of Spurious Memories in the Hopfield Model. IEEE Transaclions on Information Theory, 36(2):393 397, 1990. ISSN 15579654. doi: 10. 11110/181552488 18.24886.
Mare Brysbaert, Michaël Stevens, Pawel Mandera, and Emmanuel Keuleers. How many words do we know? Practical estimates of vocabulary size dependent on word definition, the degree of language input and the participant's s age. Frontiers in Psychology, 7, 2016. ISSN 1664-1078. doi: 10.3389 (a/8899/f/g.22// 2016.011. URL
https: //wp.
1tw./////........././//../// 2016.01116
Thomas F Burns. Semantically-corrlaated memories in dense associative model. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (ods.). Proceedings of the 11st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 4936 4970. PMLR, 21 27 Jul 2024. httpsstps:// h/tbocee//prooeiiings mlr.
meaass/v33/burneg
html
Thomas F Burns and Tomoki Fukai. Simplicial hopfield networks. In The Eleventh International Conference on Learning Representations, 2023. URL
Thomas F Burns, Tatsuya Haga, and Tomoki Fukai. Multiscale and extended retrieval of associative
memory structures in a cortical model of local-global inhibition balance. eNeuro, 9(3), 2022. doi:
1..12///EEURO..0M 1023-22.2022 URL httpps//wwit http://wwwww....o../////// www.eeeeweeeooooooeee /ttp://r/g/.cttttet////////aaaa http://////EEEE00000022222020210 002.022.220220000002222222222220
Lucius Bushnaq, Jake Mendel, Stefan Heimersheim, Dan Braun, Nicholas Goldowsky-Dill Kaarel Hänni, Cindy Wu, and Marius Hobbhahn. Using degeneracy in the loss landscape for mechanistic interpretability. arXiv preprint Xix:222055 10927, 2024.
György Buzsáki. Neural syntax: Cell assemblies, synapsembles, and readers. Neuron, 68(3):362 385, 2010. URL https://ww..cceneciiieees
ISSN 0896-6273. doi: hoap/sciencee/rtiill/ppppppp http:////..r///////990///0000310
Rishidev Chaudhuri and lla Fiete. Computational principles of memory. Nature Newroscience, 19(3):394-44) -439, March 2016.
Rishidev Chaudhuri and aaa Ficte. Bipartite expander Hopfield networks self-decoding high- capacity error correcting codes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL. https://proceeedings neurips httprissscc/paper/211/////
97006277052020005777666888 Paper.ppppppp
Spyridon Chavlis and Panayiota Poirazi. Drawing inspiration from biological dendrites to empower ar- tificial neural networks Current Opinion in Neurobiology, 70:1-10, 2021. ISSN 0959-4388. doi: http://o.........0..oooooo......
URL http://wwww..cccoocccoccc.ccaaca pi1/509559488821000444 Computational Neuroscience.
Yuhan Chen, Shengjun Wang, Claus C Hilgetag, and Changsong Zhou. Trado-off between multiple con- straints enables simultancous formation of modules and hubs in neural systems. PLoS Comput. Biol., 9 1002933, March 2013.
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Peter H. Chipman, Chi Chung Alan Fung, Alejandra Pazo Fernandez, Abhilash Sawant, Angelo Tedoldi, Atsushi Kawai, Sunita Ghimire Gautam, Mizuki Kurosawa, Manabu Abe, Kenji Sakimura, Tomoki Fukai, and Yukiko Goda. Astrocyte GluN2C NMDA receptors control basal synaptic strengths of hippocampal CA1 pyramidal neurons in the stratum radiatum. Elife, 10, October 2021.
Kimberly M. Christian and Richard F. Thompson. Neural Substrates of Eyeblink Conditioning: Acquisition and Retention. Learning and Memory, 10(6):427-455, 2003. ISSN 10720502. doi: 10.1101/lm.9600,
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafford. Think you have solved question answering? try arc, the ai2 reasoning challenge. orXiv preprint arXiv:1803.05.4 2018.
Claudia Clopath, Tobias Bonhoeffer, Mark Hübener, and Tobias Rose. Variance and invariance of neuronal long-term representations. Philos Trans R Soc Lond B Biol Sci, 372(1715), March 2017.
Rivka T Cohen and Michael Jacob Kahana, A memory-based theory of emotional disorders. Psychological Review, (29(4):74,, 2022.
Alexandra O Constantinescu, Jill X. O' O'Reilly and Timothy E. J. Behrens. Organizing conceptual knowledge in humans with a gridlike code. Science, (62( 2016. doi: 10.1126 sciencc. URL htttps://wee wwwence.innc...oo wttp:///..rrr//o///////////////. 1126/ 1wt.:0..111///cccenne sciesccn.
Zhenzhong Cui, Charles R Gerfen, and W Scott Young 3rd. Hypothalamic and other connections with dorsal ca2 area of the mouse hippocampus. Journal of comparalive neurology, 521 (8) (8):11844441800) 1866, 2013.
Onat Dalmaz, Mahmut Yurt, and Tolga Çukur. Resvit: residual vision transformers for multimodal medical image synthesis. IEEE Transactions on Medical Imaging, 41 (10):2598-261) 2022.
Christopher J. Darwin, Michael T. Turvey, and Robert G. Crowder. An auditory analogue of the Sperling partial report procedure: Evidence for brief auditory storage. Cognitive Psychology, 3(2):255-20) 1972. ISSN 00100285. doi: 10.1016/0010-0285(( 0010-0285(72)90000-22
Nathaniel D. Daw and Kenji Doya. The computational neurobiology of learning and reward. Current Opinion in Neurobiology, 16(2):199 204, 2006. ISSN 09594388. doi: 10.1016/j.comb.20066 wttpmm..00.....00000
Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Stalistioul Physics, 168(2):288 299, Jul 2017. ISSN 1572- 9613. doi: 00.1007/110955007--111 1806-.. URL
Adi Doron, Alon Rubin, Aviya Cenmelech-Chovav, Netai Benaim, Tom Carmi, Ron Refaeli, Nechama Novick, Tirzah Kreisel, Yaniv Ziv, and Inbal Goshen. Hippocampal astrocytes encode reward location. Na- ture, Aug 2022. ISSN 1476-4687. doi: (0.1008//411556-0222051114 941586-022-05146-6
URL
Hanyu Duan, Yixuan Tang, Yi Yang, Ahmed Abbasi, and Kar Yan Tam. Exploring the relationship between in-context learning and instruction tuning, 2023. URL https://arxiv..rg/ wwwgg//ga///...../ 10367
Serena M Dudek, Georgia M Alexander, and Shannon Farris. Rediscovering area ca2: unique properties and functions. Nature Reviews Neuroscience, 17(2):89-100) 1000899--022-100 2016.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL httppe:/aaarivvoor//b//////////s
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Thans- former Circuits Thread, 1(1):12, 2021.
Sarah Etherington, Susan E Atkinson, Greg J Stuart, and Stephen R Williams. Synaptic Integrationn chap- ter 1, pp. 1-15. Wiley, 2010. ISBN 9780470015902. doi: http:://ddiooooo//oooooo/////f/2 pub2. URL
18

---

Published in Transactions on Machine Learning Research (07/2025)
Samuel J Gershman, Christopher D Moore, Michael T Todd, Kenneth A Norman, and Per B Sederberg. The successor representation and temporal context. Neural Compulation, 24(6):1553-15 15688, 2012.
Samuel J. Gershman, Ila Fiete, and Kazuki Irie. Key-value memory in the brain. Neuron, 113(11)
1707.e1, 2025. ISSN 0896-6273. doi:
www.on..2..........omom URL https://www
ISSN 10747427. doi: http://dcc...cooooo/oooo///////a
Paul E. Gilbert and Raymond P. Kesner, Memory for objects and their locations: The role of the hippocam- pus in retention of object-place associations. Newrobiology of Learning and Memory, 81(1):39 45, 2004.
Vinod Goel and Raymond J. Dolan. Anatomical Segregation of Component Processes in an Inductive Inference Task. Journal of Cognitive Neuroscience, 12 01 2000. ISSN 0898-929X. doi: 10.
1162/0898929005113776 URL https://doi..org/0... /t.///////////// 110811111666/0999991111199999999
A. M. Gordon, G. Westling, K. J. Cole, and R. S. Johansson. Memory reprosentations underlying motor commands used during manipulation of common and novel objects. Journal of Neurophysiology, 69(6); 1789-1797, 1993. ISSN 00223077. doi: 10.1152/jm.1 9ttph.1999......7783393
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Tourct, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Fsiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve. Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graemne Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hamnah Korevaar, Hu Xu, Hugo Touvron, Tlyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evti- mov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landznat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Peng- wei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Sil- veira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos- seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ra- manathan, Viklor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaclle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharic Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain,
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annic Dong, Annie Franco, Amuj Goyal, Aparajita Saraf, Arka- bandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assal Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Ed- ward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Flo- rez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Tbrahim Damlaj, Igor Molybog, Igor Tufanov, Tliss Leontiadis, Irina-Flena Veliche, Itai Gat, Jake Weiss- man, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Baptitt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthow Lennie, Matthias Reso, Maxim Groshov, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Nor- man Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyag- ina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sncha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanic Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Guplæ, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjum Zhang, Tim Matthews, Timothy Chou, Tzook Shakod, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Tonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xisojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https: //tp://aavvvv//////////ea www.xiv.wgg//a... 2407 21783
Eva-Maria Griesbauer, Ed Manley, Jan M. Wiener, and Hugo J. Spiers. London taxi drivers: A review of neurocognitive studies and an exploration of how they build their cognitive map of London. Hippocampus, 32(1):3-20 2022. doi: https:/ http:://ddo.org/10000///////// 1002 hhip:..hpp..........
URL https://omlliellibrarry watery.wwleyy wiley. com/
23395 23395
20

---

Published in Transactions on Machine Learning Research (07/2025)
Vincent Gripon and Claude Berrou. Sparse neural networks with large learning diversity. JEEE Transactions on Neural Networks, 22(7):1087-1096 2011. ISSN 10459227. doi: 10.109/TNN.2011214666
Lukas N. Groschner, Jonatan G. Malis, Birte Zuidinga, and Alexander Borst. biophysical account of multiplication by a single neuron. Nature, 603(7999):119 123, Mar 2022. ISSN 1476-4687. doi: 10.1038/ $41586-022-04422 URL
022 http:/411588-0222244------ 04428-3
Bengt Gustafsson and Holger Wigström. Physiological mechanisms underlying long-term potentiation. Trends in Neurosciences, 11 (4):156-162, 1988. ISSN 01662236. doi: (10066/0166-223((88))1011333
Eric Hart and Alexander C. Huk. Recurrent circuit dynamics underlie persistent activity in the macaque frontoparietal network. eLife, 9re52460, May 2020. ISSN 2050-084X. doi: N..7554/eLife.5240 URL
http::://dooorr////// 10./10..///// 7554/eLife 52460 32379044ppmid;
Donald Olding Hebb. The organization of behavior: A neuropsychological theory. Wiley, 1949.
Nathan G. Hedrick, Zhongmin Lu, Eric Bushong, Surbhi Singhi, Peter Nguyen, Yessenia Magaña, Sayyed
Jilani, Byung Kook Lim, Mark Ellisman, and Takaki Komiyama. Learning binds new inputs into function ional synaptic clusters via spinogenesis. Nature Neuroscience, 25(6):726 737, Jun 2022. ISSN 1546-1726. doi:
URL https: 10tg://11.11111/111111181883 0ttp://44116800222--0---------
Suzana Hereulano-Houzzl. The human brain in numbers: lincarly scaled-up primate brain. Frontiers in Human Nearoscience, 3, 2009. ISSN 1662-5161. doi: 10.3389//nerr.000.03..1.
URL https://www.
frontiersin Trontiersin.org/attiicoo www.oorrrgggre/////ooo...
09.0090003. 0.1. 29.9
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg. Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. The Twelfth International Conference on Learning Representations, 2024. URL https://opperrrriio wttprriiiwneneteeeeeeeeeeeeeeeee foumiid
w7LU2s1422
Heiko Hoffmann. Sparse associative memory. Neural Computation, 31(5):908- 1014, 2019. ISSN 1530888X.
doi: 10.1162/neco a 01181.
Benjamin Hoover, Duen Horng Chau, Hendrik Strobelt, and Dmitry Krotov. A universal abstraction for hierarchical Hopfield networks. In The Symbiosis of Deep Learning and Differential Equations II, 2022.
URL htttps:///ppnrreviee wnt.:/oommm...d/sSSssllll
J J Hopficid. Searching for memories, Sudoku, implicit check bits, and the iterative use of not-always- correct rapid neural computation. Neural Computation, 20(5):1119 1164, 2008. ISSN 08997667. doi: 10.1162/neco.2008 09-08009-06-455.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554 2558, 1982. doi: pttp://.nns..........
URL attps://wwy www.:paa............oo... ppass.or//ao///bb//000/.... 10tp///110010////ma pnas. 79. 79.8..4 2554
Mare W Howard, Mrigankka S Fotedar, Aditya V Datey, and Michael E Hassclmo. The temporal context model in spatial navigation and relational learning: Loward a common explanation of medial temporal lobe function across domains. Psychological review, 112(1):75, 2005.
Zhongzhan Huang, Pan Zhou, Shuicheng YAN, and Liang Lin. Scalelong: Towards more stable training of diffusion model via scaling network long skip connection. In Thirty-seventhh Conference Neural Information Processing Systems, 2023. URL https://openrevier htt.eeeeeettooorom//dooo0031111
Michael Janner, Igor Mordatch, and Sergey Levine. Gamma-models: Generative temporal difference learning for infinite-horizon prediction. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (cds.), Advances in Neural Information Processing Systems, volume 33, pp. 1724-1735. Cur- ran Associates, Inc., 2020. URL T2Z10088226661110006677777771222 http://eettna.ararrrse//ppppepep
21

---

Published in Transactions on Machine Learning Research (07/2025)
R. Jenkins, AJ Dowsett, and AM Burton. How many faces do people know? Proceedings of the Royal Society
B, 285(1888:2001813)) 2018.
Li Ji-An, Corey Y. Zhou, Marcus K. Benna, and Marcelo G. Mattar. Linking in-context learning in trans- formers to human episodic memory, 2024. URL http:s//arraav.org/aa//222. wrg..b//2//....... 14992
Yibo Jiang, Goutham Rajendran, Pradcep Ravikumar, and Bryon Aragam. Do 1lms dream of clephants (when lold not to)? latent concept association and associative memory in transformers, 2024. URL
attps://arxivv www.woggg......eo.../.........at wetpaaaa../2.222// 2406 18400
Arthur Juliani, Ryota. Kanai, and Shuntaro Sasai Sasai. The perceiver architecture is a functional global workspace. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022.
Marcus Kaiser and Claus C Hilgetag. Nonoptimal component placement, but short processing paths, due to long-distance projections in neural systems. PLoS Comput. Biol., 2(7):e95, July 2006.
Ayaka Kato, Kazumi Ohta, Kazuo Okanoya, and Hokto Kazama. Dopaminergic neurons dynamically update sensory values during navigation. bioRriv, 2022. doi: 21..101/2222.0.17.50000000 URL https://ww/ 08 17. 504092
biorxiv.org/ httgg///nnnne/eeea//////////////
Mikail Khona and lla R Fiete. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23(12):744-766 2022.
Do Hyun Kim, Jinha Park, and Byungnam Kahng. Enhanced storage capacity with errors in scale-free Hopfield neural networks: An analytical study. PLoS ONE, 12(10), 2017. ISSN 19326203. doi: 10.1371/ 001.088.....011468..... 0184683.
journal.pone.00
Leo Kozachkov, Ksenia V. Kastanenka, and Dmitry Krotov. Building Transformers from neurons and astrocyttes bioRaiv, 2022. doi: 10.1101 ear1y/2022/1/11/2222 10. 10.0.....1111 511910
10.1101/2020 100120022.1011255111000 URL https://ww.biorxiz wiorxi.wwwb...i..........o...w.. www.bbiiwwoiooooooooooooooooei org/contie content/
Dmitry Krotov and John Hopfield. Dense associative memory for pattern recognition. In Proceedings of the 30th International Conference on Newral Information Processing Systems, NIPS'16, pp. 1180 1188, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Anders Lansner. Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations. Trends in Neurosciences, 32(3):178 186, 2009. ISSN 01662236. doi: 10.1016/j (.001/j.tins tins.2008. 12. 002.
Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: A singularity-aware complexity measure. ar preprinl arXiv:2908.1 1188..111.... 2023.
S. J. Lederman and R. L. Klatzky. Haptic perception: A tutorial. Attention, Perception, and Psychophysics, 71(7):1439-1455, 2009. ISSN 19433921. doi: 10.3758/APP.71.77
Amir Levi, Noam Aviv, and Eran Stark. Learning to learn: Single session acquisition of new rules by frooly moving mice. PNAS Nezus, ppaacg2,, 05 2024. ISSN 2752-6542. doi: (0000///aaanxxs/////
URL httpt::///aee org/ 10. 1093/ 1ew.110093ppaeeee tatasnexus/pgaa2200
Yingcong Li, Muhammed Emrullah Tldiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers algorithms: Generalization and stability in in-context learning. In Andreas Krause, Emma Brunskill, uy Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19565-19594. PMLR, 23-29 Jul 2023. URL https://proceeddngs. wwwprrcceedings..m..p.e.emm wwwpss/e22/1122211111 html
William A Little. The existence of persistent states in the brain. Mathematical Biosciences, 19(1):101- 120, 1974. ISSN 0025-5564. doi: http:///ooooo///00...00000555550
URL https://www.
22

---

Published in Transactions on Machine Learning Research (07/2025)
Lynn J Lohnas, Sean M Polyn, and Michael J Kahana. Expanding the scope of memory search: Modeling intralist and interlist effects in free recall. Psychological review, 122(2):337, 2015.
Rafael Lorente de Nó. Analysis of the activity of the chains of internuncial neurons. Journal of Neurophysi- ology, 1(3):207 244, 1938.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL http:::/ppaareev.eees..tttrrrrrp
Matthias Löwe and Franck Vermet. The Hopfield model a sparse Erdös- graph. Journal of Statistical Physics, 143(1):205 214, Apr 2011. ISSN 1572-9613. doi: 10.1007/110055-0100006111 URL https:
7/doi org/10. 1007/111555001111117
Yue M Lu, Mary I Letey, Jacob A Zavalone- Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptolic theory of in-context learning by linear attention. arXiv preprint vrxxt=2405.1175.1 2024.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data, 2021. URL btopp:://rriivy org/ httggabb/220055 07648
Matthias Lôwe. On the storage capacity of Hopfield models with correlated patterns. The Annals of Applied Probability, 8(4):1216 1250, 1998. doi:
URL fttps://doi.org/1.//
aoap/1028903378
N. J. Mackintosh. Condilioning and associalive learning. Oxford University Press, Oxford, 1983.
Farshad Alizadeh Mansouri, David Freedman, and Mark J Buckley. Emergence of abstract rules in the primate brain. Nature Reviews Neuroscience, 21((1):595-65 610, 2020.
Stephen Maren. Neurobiology of Pavlovian Fear Conditioning. Annual Review of Neuroscience, 24(1): 897-931, 2001. ISSN 0147-006X. doi: 1114///mmrre.enurr.2444444
D Marr. Simple memory: a theory for archicortex. Phálos Trans Soc Lond Biol Sei, 262(841):23 81, July 1971.
William Mau, Michael E Hasselmo, and Denise J Cai. The brain in motion: How ensemble fluidity drives memory-updatinn and flexibility. eLife, 9ce63550, Dec 2020. ISSN 2050-084X. doi: /
URL https://doi. wtt.://rrrr......///...f....o.. 00t.0055///L....6555550
Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5:115-133, 1943.
R. McFliece, F. Posner, F. Rodemich, and S. Venkatesh. The capacity of the Hopfield associative memory. IEEE Transactions on Information Theory, 33(4):461 482, 1987. doi: 10.1109/TTT..187.. 10877733............
Frances K. McSweeney and Eric S. Murphy (eds.). The Wiley Blackwell Handbook of Operant and Classical Conditioning. John Wiley & Sons, Ltd, Oxford, UK, may 2014. ISBN 9781118468135 doi: 10.1002/ 9781118468135, URL http:///oi. www.y. cwm/ 10. 1002/9781118468185
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? dataset for open book question answering, 2018. URL https://arxiv, http:////a/wiw..//aa////////////
warg/aba/18000 0tt///88900000 00789
Tomas Mikolov, Πya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality, 2013. URL https://arxiv.co httpe///aaxxi..o/aabbb////////// 4546
Earl K Miller and Jonathan D Cohen. An integrative theory of prefrontal cortex function. Annual review of neuroscience, 24(1):167-202, 2001.
Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Universal hop- field networks: A general framework for single-shot associative memory models. In International Confer- ence on Machine Learning, pp. (6561-15583 PMLR, 2022.
23

---

Published in Transactions on Machine Learning Research (07/2025)
B. Milner. Amnesia following operation on the temporal lobes. In C. Whitty and O. Zangwill (eds.), Amnesia, pp. 109-133. Butterworth, London, 1966.
G.A. Milner. The Magical Number Seven, Plus or Minus Two: Some Limils on Our Capacity for Processing Information. Psychological Review, 101(2):343 352, 1955. URL http://spider wwwp:ppiieerraa..rgrg... apw. ottp////.........e/gtttto///////
1994/ppr11/rev101333 html
Ali A. Minai and William B. Levy. The dynamics of sparse random networks. Biological Cybernetics, 70(2): 177-187, 1993. ISSN 03401200. doi: 10.1007/BF002000
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kan- ishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines, 2023. URL
A. A. Mofrad and M.G Parker. Nested-clique network model of neural associative memory. Neural Compu- tation, 29:1681 1695, 2017.
Gianluigi Mongillo, Omri Barak, and Misha Tsodyks. Synaptic Theory of Working Memory. Science, 319 (5869):1543- 2008. ISSN 10959203. doi: 10.1126/science.11 1150769.
Kaoru Nakano. Associatron-a model of associative memory. IEEE Transactions Systems, Man, and Cybernelics, :35-2 388, 1972. doi: (0.100/TSMCC.19124303133
Eshaan Nichani, Jason D Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories. arXiv preprint or 2rXi 2.06588, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatffeld-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https://ttaane https://tannsfrmerrccccutttstts /2022/ /in-context- learning-and-inng wunddinnducioon-ehaass//aee hhweeeadddd/d.hmm.m....
Christos H. Papadimitriou, Santosh S. Vempala, Daniel Mitropolsky, Michael Collins, and Wolfgang Maass. Brain computation by assemblies neurons. Proceedings of the National Academy of Sciences, 117(25): 14464 14472, 2020, doi: 10.1073/pmas.2001.0 2001893117.
URL bstps://wwat .nas.
p 2001893117
Scongmin A. Park, Douglas S. Miller, and Eric D. Boorman. Inferences on a multidimensionall social hicrarchy use a grid-like code. Nature Neuroscience, 24(9):1292 1301, Sep 2021. ISSN 1546-1726. doi: 10.1038/
s415588-021-00916-3 URL httpp::/ddi...rgg////
Gertrudis Perea, Marta Navarrete, and Alfonso Araque. Tripartite synapses: astrocyles process and control
synaptic information. Trends in Newrosciences, 32(8):421 14111141,111 2009.
Brad E. Pfeiffer and David Foster. Autoassociative dynamics in the generation of sequences of hippocampal place cells. Science, (49(444)):180-188, 2015. ISSN 10959203. doi: 10.1126/ 101116//cinncc..o wcciencc.aaa9663.
Alethca Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint 1xXi:2201.02177, 2022.
Mclissa A. Preziosi and Jennifer H. Coanc. Remembering that big things sound big: Sound symbolism and associative memory. Cognitive Research: Principles and Implications, 2(1), 2017. ISSN 2365-7464. doi:
10.1186//11235--66-0007.7
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In International Conference on Learning Representa- lions, 2021. URL httppp///pppaeeeeee..eeeee/ooada
24

---

Published in Transactions on Machine Learning Research (07/2025)
Charan Ranganath and Robert T Knight. Prefrontal cortex and episodic memory: Integrating findings from neuropsychology and functional brain imaging. The cognilive neuroscience of memory: Encoding and retrieval, 1:83, 2002.
Nelson Rebola, Mario Carta, and Christophe Mulle. Operation and plasticity of hippocampal CA3 circuits: Implications for memory encoding. Nature Reviews Newroscience, 18(4):209-28) 2017. ISSN 14710048. doi: 10. 10.1038/nmm... 2017.10..
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in in-context clas- sification task. In The Twelfth International Conference on Learning Representations, 2024. URL
R.A. Rescorla and A.R. Wagner. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A.H. Black and W.F. Prokasy (eds.), Classical Condilioning II: Current Research and Theory, pp. 64 99, Appleton Century Crofts, New York, 1972.
Mark Rigby, Federico Grillo, Benjamin Compans, Guilherme Neves, Julia Gallinaro, Sophie Nashashibi, Gema Vizcay-Barrent Florian Levet, Jean-Baptiste Sibarita, Angus Kirkland, Roland A. Fleck, Claudia Clopath, and Juan Burrone. Multi-synaptic boutons are feature of cal hippocampal connections that
may underlic network synchrony. bioRriv, 2022. doi: 493836
493836. URL httpps://ww.
biorxiv. wtgp:/oottem/aarl///2///////////
Daniel S. Rizzuto and Michael J. Kahana. An autoassociative neural network model of paired-associate lcarning. Neural Computation, 13(9):2075-2092 2001. ISSN 08997667. doi:
Vincent Robert, Sadiyah Cassim, Vivien Chevaleyre, and Rebecca A Piskorowski. Hippocampal area ca2: properties and contribution to hippocampal function. Cell and tissue research, 373:525- 540, 2018.
Matthew Rosenberg, Tony Zhang, Pietro Perona, and Markus Meister. Mice in labyrinth show rapid learning, sudden insight, and elficient exploration. eLife, 10:e66175, jul 2021. ISSN 2050-084X. doi: 77755///eiif.666 URL https://doi.co
66175
Frank Rosenblatt. Principles of neurodynamics: Perceptrons and the theory of brain mechanisms, 1961.
Nicolas P Rougier, David C Noelle, Todd S Braver, Jonathan Cohen, and Randall C O'Reilly Prefrontal cortex and flexible cognitive control: Rules without symbols. Proceedings of the National Academyy of Sciences, 102(20):7338 7343, 2005.
Jacob Russin, Ellie Pavlick, and Michael J Frank. Human curriculum effects emerge with in-context learning in neural networks. ArXiv, 2024.
Saratha Sathasivam and Wan Ahmad Tajuddin Wan Abdullah. Logic Learning in Hopfield Networks. Modern Applied Science, 2(3), 2008. ISSN 1913-1844. doi: 10.5539 mmas.v2n3551.
W. B. Scoville and B. Milner. Loss recent memory after bilateral hippocampal lesions. Journal of Neurolgy, Newrosurgery, and Psychiatry, 20(1):11 21, 1957.
Philipp Seidl, Philipp Renz, Natalia Dyubankova, Paulo Neves, Jonas Verhoeven, Jörg K. Wegner, Mar- win Segler, Sepp Hochreiter, and Günter Klambauer. Improving few- and zero-shot reaction template prediction using modern Hopfield networks. Journal of Chemical Information and Modeling, 62(9):2111 2120, 2022. doi: (al.02/acc.jcim.m... 35034452.
URL BalLhttss: //doi, org/10. 102100022sss... http://acsscci....oo.. 1c01065 PMID:
Adam Shai, Paul M. Ricchers, Lucas Teixcira, Alexander Gictelink Oldenziel, and Sarah Marzen. Transform- ers represent belief state geometry in their residual stream. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL http::///ppaaaeeeeee..ee/////t/p
25

---

Published in Transactions on Machine Learning Research (07/)025)
Sugandha Sharma, Sarthak Chandra, and Ila Ficte. Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold. In Kamalika Chaudhuri, Stefanie Jegelka, Te Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19658-19682. PMLR, 17-23 Jul 2022. URL htttpe://proceedinggs. mlr prwssppessseeeee 21116//shama22b.... htmml
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05200 2020.
George Sperling. A Model for Visual Memory Tasks, Human Factors: The Journal of Human Factors and Ergonomics Society, 5(1):19-31, 1963. ISSN 15478181. doi: 10.1177/001372203300000000
Reisa Sperling, Flizabeth Chua, Andrew Cocchiarella, Frin Rand-Giovannetti, Russell Poldrack, Daniel L. Schacter, and Marilyn Albert. Putting names to faces: Successful encoding of associative memories activates the anterior hippocampal formation. Newrolmage, 20(2):1400-1410, 2003. ISSN 10538119. doi: 10.1016/S1053-88 (0030003311-
Larry R. Squire. Memory and the Hippocampuss: A Synthesis From Findings With Rats, Monkeys, and Humans. Psychological Review, 99(2):195-28) 231, 1992. ISSN 0033295X. doi: 0007700033295...00
Larry R. Squire, Arthur P. Shimamura, and David G. Amaral. Memory and the Hippocampus. Neural Models of Plasticily, pp. 208 239, 1989. doi: 10.1016/b978-0-111 1e---0------888-- 14896644.0016-4.
Lionel Standing. Learning 10000 pictures. The Quarterly journal of experimental psychology, 25(2):207 222, 1973.
Amos Storkey. Increasing the capacity of a Hopfield network without sacrificing functionality. Lecture Notes in Computer Science (including subseries Lecture Notes Artificial Intelligence and Lecture Notes in Bioinformatics), 1327:451-456, 1997. ISSN 16113349. doi:
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Moham- mad Shoeybi, and Bryan Calanzaro. Nemotron-ce: Transforming common crawl into a refined long-horizon pretraining dataset, 2024a. URL https://arxiv..ot wtwgg////////// abs 2412. 02595
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024b.
R. J. Sutherland and J. W. Rudy. Configural association theory: The role of the hippocampal formation in learning, memory, and amnesia. Psychobiology, 17(2):129-11) 144, 1989. ISSN 08896313. doi: 10.3758/ BFF0337828..
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9 44, 1988.
A Treves and D J Amit. Metastable states in asymmetrically diluted Hopfield networks. Journal of Physics A: Mathematical and General, 21(14):3155 3169, jul 1988. doi: 10.1088 10.1088/0305- (1805-4470/21/14/0111 URL https://doi.oow org/ 0000118//030--444////////00
Danil Tyulmankov, Ching Fang, Annapurna Vadaparty, and Guangyu Robert Yang. Biological learn- ing in key-value memory networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and
J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 22247- 22258. Curran Associates, Inc., 2021. URL https://proceedding: hwt.roceeding..eurrpp.....eepppp pdf.
Michael T. Ullman. Contributions of memory circuits to language: The declarative/proecduaal model. Cogmition, 22(112))2311211 2004. ISSN 00100277. doi: mo.101//.cognition..o 10.0..0..00.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Aduances Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL Sateee22sss/deeeffooocccceeeaaaa pddf
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Edward K. Vogel, Geoffrey F. Woodman, and Steven J. Luck. Storage of features, conjunctions, and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 27 (1):92 114, 2001. ISSN 00961523. doi:
Tvan Voitov and Thomas D. Mrsic-Flogee Cortical feedback loops bind distributed representations of working memory, Nature, 608(7922) : 389, Aug 2022. ISSN 1476-4687. doi: 10.1008//41588-022--0555 URL
attps://dbi.org// 10381:..11//44411 020-58060222-00-1------
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, An- drey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Reseurch, pp. 35151 35174. PMLR, 23 29 Jul 2023. URL https: //proceedings.co mlr www.mmlr.ppr.... www....pees/vv22222 wowa////ooowwwwwld2/ html
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretabil- ily in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference Learning Representations, 2023. URL
Richard A. Watson, C. L. Buckley, and Rob Mills. Optimization in "sell-modeling" complex adaptive systems. Complexity, 16(5):17-20 2011a. ISSN 10990526. doi: 10. (0.1002/cbx:22334.
Richard A. Watson, Rob Mills, and C. L. Buckley. Global adaptation in networks of selfish components: Emergent associative memory at the system scale. Artificial Life, 17(3):147-166, 2011b. ISSN 10645462. doi: 10. //rrlll a 00029.
Melanie Weber, Pedro D. Maia, and J. Nathan Kutz. Estimating memory deterioration rates following neurodegeneration and traumatic brain injuries in Hopfield network model. Frontiers in Newroscience, 11(NOV), 2017. ISSN 1662453X. doi: 10.3389 wwnins.iiiinn///ee wwtfffiiinn.......000 2017.00623.
Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitner, Johannes Brandstetler, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al. Modern Hopfield networks and attention for immune repertoire classification. Advances in Neural Information Processing Systems, 33:18832-18845, 2020.
Tom J. Wills, Colin Lever, Francesca Cacucci, Neil Burgess, and John Oeeeee. Attractor dynamics in the hippocampal reprosentation of the local environment. Science, 308(5723):873- 2005. ISSN 00368075.
doi: 10.1126/scieiie sccence.eeeee... 110te90.... 108905.
István Winkler and Nelson Cowan. From sensory to long-term memory: Evidence from auditory memory reactivation studies. Experimental Psychology, 52(():3 20, 2005. ISSN 16183169. doi: 10.1027/1618-310 52.1.3.
Alexander Woodward, Tom Froese, and Takashi Ikegami. Neural coordination can be enhanced by occasional interruption of normal firing patterns: A self-optimzzing spiking neural network model. Neural Networks, 62:39 46, 2015. ISSN 18792782. doi:
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL httpp:://axxxx...r. wwtp://w.wwrr..bb...b...o
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49)11-55, 2024a.
Zixuan Zhang, Kaiqi Zhang, Minshuo Chen, Yuma Takeda, Mengdi Wang, Tuo Zhao, and Yu-Xiang Wang. Nonparametric classification on low dimensional manifolds using overparameterized convolutional residual networks. In The eighth Annual Conference on Neural Information Processing Systems, 2024b.
URL http:pppaaeeireeeeeeo/oooaaanaaa
27

---

Published in Transactions on Machine Learning Research (07/2025)
Jiachen Zhao. In-context exemplars clues to retrieving from large associative memory. In Associative Memory & Hopfield Networks in 2023, 2023. URL http:///wggggpeoeeeeee/////////a
Corey Y Zhou, Deborah Talmi, Nathaniel Daw, and Marcelo G Mattar. Episodic retrieval for model- evaluation in sequential docision tasks, 2023.
Appendix
A Connection between associative memory models and modern neuroscience
Associative memory falls under the broader category of long-term memory the type of memory capable of lasting a lifetime. Yet, memory also operates on at least two additional, functionally distinct timescales: short-lerm memory and sensory memory. Short-term (or working) memory retains information over tens of seconds and serves as a limited, passive buffer that enables manipulation and use of that information (Milner 1955: Mongillo et al. 2008). In contrast, sensory memory operates on even briefer timescales - typically just lew seconds. 1t is the initial stage where sensory inputs such as visual (Sperling 1963 Vogel et al., 2001)
auditory (Darwin et al., 1972 Winkler & Cowan 2005), and other sensory modalities (Gordon et al 1993
Lederman & Klatzky| 2009) are actively "remembered" before either being discarded encoded further. A general model for forming long-term memories thus involves: (1) recciving sensory imput; (2) briefly storing features of that input in sensory memory; (3) selectively retaining and manipulating aspects of this input within short-term memory, possibly integrating multiple sources; and (4) consolidating the result into long- term memory. Theoretical and computational insights into these processes are valuable for two reasons: (i) they may inform the development of intelligent systems inspired by biological memory mechanisms; and (ii) they contribute understanding the neural basis of memory, with potential applications in both clinical and cognitive contexts.
In both humans and other animals, associative memory typically refers to any form of long-term memory involving the linking or "association" of distinet stimuli, allowing the recall of one item upon encountering the other. Classical examples include name face associations (Sperling et al.) 2003), object sound pairings (Preziosi & Coane 2017), and object-location associations (Gilbert & Kesner 2004). These forms are considered part of explicit or declarative memory (Ullman 2004), which encompasses memories that can be consciously retrieved or verbalized. In contrast, implicit non-declarative memory supports associative processes that occur unconsciously automatically, such as those formed through classical conditioning [Maren 2001 Christian & Thompson 2003) and operant conditioning (Mackintosh 1983 McSweeney &
Murphy 2014
Computational models of implicit associative memory have a long history, beginning with foundational models such as the Rescorla- modern field of reinforcement learning (Daw & Doya 2006).. Explicit associative memory, on the other hand, appears to require greater computational sophistication, suggested by the complexity of its biological
Rescorla-Wagner Wagner model (Rescorla & Wagner 1972), which laid groundwork for the
underpinnings (Chaudhuri & Fiete 2016 Clopath et al. 2017 Mau et al. 2020
A seminnal model of explicit associative memory is the associative memory model Marr 1971 Nakano 1972
Amari (192);; Little (1974). popularised as the Hopfield network (Hopfield. 1982). In its simplest form, when storing single memory, the associative memory model allows for a configuration of neuron thresholds such that a partial activation reliably lcads to full memory recall. This behaviour corresponds to attractor dynamics converging to stable memory state, analogous to neuronal assemblies in the hippocampus (Wills
et al. 2005 Pfeiffer & Foster 2015 Rebola et al. 2017
The hippocampus is widely regarded as a central structure in memory processing. Along with the entorhinal, perirhinal, and parahippocampal cortices, it plays a critical role in the formation of explicit memories (Scoville 1957 Milner 1966 Squire 1992p. It acts as a temporary store for new information, which is later
& MMinner consolidated in the cortex (Squire et al. 1989 Sutherland & Rudy 1989). Traditionally, these processos have been attributed to Hebbian learning and long-term term potentiation (Biss& & Hardner-Medwis 1973 Gustafsson
& Wigström 1988), though recent. research has highlighted additional mechanisms that may contribute to
28

---

Published in Transactions on Machine Learning Research (07/2025)
memory formation and maintenance (Rigby et al., 2022
Groschner al. 2022
Hedrick et al.
2022
Kato
et al.
2022
Papadimitriou et al. 2020: Hart & Huk 2020 Chipman et al. 2021 Perea et al. 2009 Doron
et al.
2022
Etherington et al. 2010 Chavlis & Poirazi 2021).
Despite its strengths, the classical associative memory model has limitations as a full account of long- term memory. For example, its memory capacity scales linearly with the number of neurons n, supporting ap- proximately 0.14n stable patterns before performance degrades due to spurious attractors (Amit et al. 1985
McEliece et al.) 1987 Bruck & Roychowdhury 1990). This capacity is further reduced under correlated
input conditions (Löwe 1998), sparse connectivity (Treves & Amit 1988 Löwe & Vermet 2011), or both
[Burns ct al., 2022). Yet, biological systems often operate in such sparse regimes (Minai & Levy 1993;
Lansner 2009; Barth & Poulet, 2012), and human memory regularly handles highly structured and over-
lapping information (Constantinescu et al. 2016
Aronov et al.) 2017 Bellmund et al.) 2018 Bao et al.
2019 Park et al.) [021]] Griesbauer et al.; 2022]
Nevertheless, humans can remember thousands of highly
similar images (Standing 1973; Brady et al. 2008
recognize countless faces (Jenkins et al. 2018
learn
tens of thousands of words (Brysbaert el. al. 2016).
and even recite over 100,000 digits of (Bellos 2015
all without significant interference.
Although modern associative memory networks have achieved much higher theoretical capacities Krotov & Hopfield 2016 Demircigil et al. 2017). the empirical data and biological constraints - succh as the finite number of neurons (Herculano- [Hereulaaoo-Houeel] 2009) and the metabolic costs of sustaining their connections (Bordone et al. 2019) suggest there are deeper computational and physiological principles at work. Even within associative memory-type frameworks, memory capacity can be interpreted not just as storage volume, but trade-off involving recall fidelity, interference, and cognitive efficiency.
Still, these limitations do not diminish the relevance of associative memory networks or their successors for understanding memory phenomena in biological and machines. They remain powerful tools in neuroscience
(Sathasivam & Wan Abdullah 2008 Rizzuto & Kahana 2001 Weber et al. 2017). machine learning (Widrich
let al. 2020
Seidl et al. 2022). and in bridging the gap between artilicial and biological systems (Sharma
jtt al.
2022
Hoover et al. 2022 Chaudhuri & Fiete 2019 Tyulmankov et al. 2021 Kozachkov el al. 2022).
Substantial progress has already been made in expanding their capabilities, including increased capacity
and efficiency (Storkey 1997 Hopficid 2008 Krotov & Hopfield 2016 Gripon & Berrou 2011 Mofrad &
Parker 2017:
Burns & Fukai 2023 sparse representalions (Kim el al. 2017 2017: 2019) and the
incorporation of biologically inspired mechanisms (Watson et al. 2011a Woodward et al. 2015 Burns
et al. 2022 Burns 2024). The current work aims to contribute further to these areas by: (1) understanding
the capacity of the associative memory framework to perform ICL (demonstrated with the AMICL model);
and (2) testing whether incorporating associative menory-inspired modifications to Transformers improves their capabilities (demonstrated with the residual values stream).
B Control experiments
In the following experiments, we use variants of a one-layer Transformer architecture and task as in $3. Tn all cases, the matrix WT is removed, to align with a naive version of the AMICL model. The only difference in the following four cases is the nunber of ReLU layers and whether these are placed before (Figure 7 after (Figure or both before and after (Figures 9 and 10 the single (value-absent) attention layer.
In all cases: the in weights (IW) task performs best, and close to perfect accuracy given sufficient training steps; the in-context (IC) task with novel data achieves approximately chance performance (20%); the in- context 2 (IC2) task with misaligned data initially performs near chance level but as the network begins memorising the training data (as illustrated through the delayed but superior improvement seen in the IW task), decays to 0% accuracy; and the test converges to mid-point between the IW and IC task performance levels, indicating a near-equal performance trade-off occurs in the learnt algorithm.
Two other general trends are observable: (i) having ReLU layers follow the attention layer proves a smoother performance trajectory (compare Figures 7 and 8]; and (ii) having more ReLU layers increases the speed of the learning trajectories.
29

---

Published in Transactions on Machine Learning Research (07/202) 2025)
1.0
KC2
IV
Accuracy
0.2
0.0
50
100
150
200
250
300
Figure 7: Performance of a single (value-absent) attention layer followed by three ReLU layers over training steps.
1.0
Test
0.8
Accuracy
0.6
0.2
0.0
50
100
150
200
250
300
Figure 8: Performance of model with three RcLU layers followed by a single (valuc-abscnt) attention layer over training steps.
Test
Accuracy
0.4
0.2
100
150
200
250
300
Figure 9: Performance of a model with two ReLU layers, followed by a single (value-absent) attention layer, followed by two ReLU layers over training steps.
30

---

Published in Transactions on Machine Learning Research (07/2025)
Test
IC2
0.8
Accuracy
0.0
50
100
150
200
250
300
Figure 10: Performance of a model with three ReLU layers, followed by single (valuo-absent) attention layer, followed by three ReLU layers over training steps.
C Extended figures
Dot product
Pearson's correlation
Manhattan distance
Euclidean distance
Identity
Softmax
Argmax
Figure 11: Attention matrices for label-object pairs in AMICL, using different similarity and separation functions with = 2.
31

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
Dot product with argmax
Pearson's correlation with argmax
1.0
0.8
0.4
Proportion
0.2
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
1.5
150
175
2.00
0.00
0.25
0.50
0.75
1.00
125
1.50
1.75
2.00
Figure 12: Proportion of correct trials, tested over 1, 000 trials for values between a and a = 2 in AMICL for label object pairs using the DOT PRODUCT (left) and PEARSON's CORRELATION (left) similarity with the ARGMAX separation function.
Dot product with argmax (a 2)
Dot product with argmax (a = 2)
1.0
1.0
0.8
Proportion
100
500
1000
10
100
500
1000
Sequence length
Token dimensions
Figure 13:
Proportion of correct trials, tested over 1, 000 trials for varying the values of the sequence length (o, left) and token dimensions (e, right) between 10 and 1,000 in AMICL for label-ot object pairs using a = 2 with the DOT PRODUCT similarity function and ARGMAX separation function.
Extended tables
Table 5: Mean + standard deviation of training snapshot number where accuracy first exceeded 0.5 for the IC and IC2 tasks in the classic (unmodified), and residual queries, keys, and values stream networks. Queries (ours)
Classic
Keys (ours)
Values (ours)
IC
51.0 + 4.24
41.5 + 2.29
49.75 + 0.43
32.75 = 0.83
IC2
51.5 + 4.39
41.5 + 1.66
49.25 + 0.83
33.25 + 1.09
32

---

Published in Transactions on Machine Learning Research (07 /025)
ARC
ARC (five-shot)
Classic
Residual Value
0.45
0.40
Accurocy
Accurocy
0.35
0.30
0.30
0.25
0.25
0.20
Training steps
Training steps
(five-shot)
Accurscy
Accuracy
0.70
0.70
0.65
0.65
0.60
5000
10000
15000
20000
25000
30000
35000
40000
5000
10000
15000
20000
25000
30000
35000
40080
steps
Training steps
OpenBookOA
OpenBookQA (five-shot)
0.425
0.40
0.375
Accutayy
0.350
Accuracy
0.35
0.325
0.300
0.250
0.225
5000
10000
15000
20000
25000
30000
35000
40000
10000
15000
25000
30000
35000
40000
Training stops
Training stops
HellaSwag
HellaSwag (five-shot)
Accuracy
Accuracy
0.50
0.45
0.45
0.40
0.40
0.35
0.35
0.30
5000
10000
15000
20000
25000
30000
25000
40000
5000
10000
15000
20000
25000
30000
35000
40080
Training
Training
Figure 14: Accuracy across training steps for our evaluations the 1B model, single- and five-shot.
33

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 6: Mean + standard deviation of training snapshot number where accuracy first exceeded 0.9 for the IC and IC2 tasks in the classic (unmodified), and residual queries, keys, and values stream networks.
Classic
Queries (ours)
Keys (ours)
Values (ours)
IC
59.25
4.49
48.25 + 1.48
57.5 + 0.5
42.25 + 1.3
IC2
59.0 + 4.53
48.0 + 1.87
57.25 + 0.83
42.0 + 0.71
E Notation and abbreviations tables
A comprehensive list of all notations and abbreviations used in this paper is provided in the tables below.
Abbreviations
LLMs
large language models
ICL
in-context learning
LMs
langauge models
iid.
independent and identically distributed
IW
in-weight
TC
in-context
IC2
in-context
101
indirect object identification
CA1, CA2, CA3
Ammonis 2,
34

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Variables
X,
The input sequence data X, E 12 with column vectors, corresponding to the token embeddings, each of dimension e. Where the subscript i is present, this denotes the data is taken from the i th Transformer layer, i.e., the residual stream data.
24
A token embedding 2; R, where the subscript i denotes the column position in the input sequence X.
20
The token embedding x, E Ie of the final column, i.e., the final token, in the input sequence X. An object token embedding IR', where the superscript identifies the object-label identity. A label token embedding 1 RR, where the superscript: i identifies the object label identity. A vector μu Rx whose components are ii.d. sampled from normal distribution having mean zero and variance 1/e. Used for constructing the token embeddings of object label where for each object, o', and label, 14, the vector 14 fixed.
o¹
1
A fixed real number E IR which controls the inter-instance variability of objects, and is set to 0.1 unless stated otherwise.
A vector v E R' whose components are i..d. sampled from normal distribution having mean zero and variance 1/e. Redrawn and used for adding inter-instance variability the construction of each object token embedding.
W7. w
Query and key weight matrices W W R respectively, of the th Transformer layer. Value weight matrix w E g of the i-th Transformer layer.
W
Q
Queries matrix Q E R of the i-th Transformer layer, calculated using W? and X Keys matrix K E xa of the th Transformer layer, calculated using W and X, Values matrix gx of the i-th Transformer layer, calculated using W and X..
K;
V.
S.
The scores matrix, S. E R which is equal to KTQ,-
A queries column vector, qi RAA where the subscript i denotes the column position in the queries matrix Q
q:
A keys column vector, & E R8d where the subscript i denotes the column position in the keys matrix K.
A values column vector, E R', where the subscript i denotes the column position in the values matrix
V.
Dimensions
Dimensionality e E N+ of each token embedding.
Number of tokens N+ in the input sequence data X.
Reduced token embedding dimension E N+ for keys and queries in the attention operation. Reduced token embedding dimension E N for values in the attention operation.
Number of unique labels N+ the imput sequence data
35


Reviews:
[{"review_id": "ZRpOTCurpt", "reviewer": "Reviewer_4pDz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper bridges neuroscience and machine learning by drawing on associative memory models from biological systems to improve Transformer architectures. This cross-disciplinary approach is novel.\n\nWeaknesses:\n- The writing needs to be completely polished. There are no subsections in method part and experiment part at all, making it difficult for readers to follow. There should be reasonable subsections where you introduce background, settings, etc., instead of mixing everything together.\n- The experiments are performed on language models that are too small. Considering the size of current LLMs, the authors need to conduct experiments on few hundred million parameter models to be more persuasive.\n- Comparing the steps to reach 100% accuracy as shown in Figure 3 is not a good metric. The authors should pick a hard enough dataset and compare accuracy under the same steps. This is more consistent with the actual situation.\n- For the sixth sub-figure in the figure 3, why is the loss increasing on IC2 task?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces two main contributions:\n- Associative Memory for In-Context Learning (AMICL):\nA one-layer associative memory model inspired by biological memory systems that can perform in-context learning (ICL) using a simplified attention-like mechanism. It demonstrates that even a single-layer architecture, when designed with biologically-inspired mechanisms, can support effective ICL.\n- Residual Attention Stream Architecture:\nA architectural modification to Transformers that introduces residual connections between the value vectors of attention heads across layers. This residual stream improves the speed and accuracy of ICL during training, especially in small models, and scales well to more realistic language tasks (e.g., TinyStories, IOI tasks).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Reconstruct the layout, add more subsections to better demonstrate the paper.\n- Train on larger language models, at least show some scaling curves.\n- Select a harder dataset where the models cannot reach 100% accuracy to evaluate.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9IIREFsUnI", "reviewer": "Reviewer_KET7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, I think the connection between computational neuroscience models of associative memory and in-context learning in transformers is interesting and deserving of further study. Additionally, this paper has a number of interesting findings, and the proposed architectural modification seems to produce some importance benefits. \n\nThe primary weakness is that, in its current form, many aspects of the paper seem somewhat disjointed or not sufficiently motivated. For example, the proposed architectural modification -- a residual stream connecting value embeddings between successive layers -- seems reasonable, and produces some benefits, but the connection to the initial associative memory experiments is tenuous. The modification is motivated by noting that in the initial AMICL model, values are simply a copy of the input token embeddings. However, this does not directly translate into the proposed modification. The proposed modification provides a direct path between value embeddings in different layers, not between the input and the value embeddings. A closer match to the AMICL model would be simply eliminate the value embedding matrices ($W^v$), or to have a residual stream from the inputs to value embeddings at downstream layers. \n\nAnother concern is that is that there's no justification provided for this initial decision (making values equal to inputs) in the AMICL model. This appears to be an arbitrary design decision in the AMICL model, rather than being based either on a specific source of biological inspiration, or factors related to performance of the model. So, although there is some connection between this design choice in the AMICL model and the proposed modification to transformers, it does not seem to have anything to do with associative memory per se. This somewhat undermines the framing of the paper, according to which the modification to the transformer architecture is supposed to be related to or inspired by associative memory models in computational neuroscience.\n\nAdditionally, although the proposed modification does seem to yield performance benefits on the IOI task, there is not much explanation or discussion of *why* this modification yields improved performance on this particular task, especially given that the modified architecture doesn't really yield improved overall performance on the language modeling task used for training. \n\nFinally, it might be informative to include additional control models in the experiments. In particular, I am curious whether it matters that the additional residual streams are passed between corresponding components (e.g. from values in layer l-1 to values in layer l), or do the performance benefits come merely from having additional pathways to aid in backpropagating gradients? For instance, it would be interesting to evaluate a version of the model with residual streams between queries/keys in layer l-1 and values in layer l, or simply randomly adding residual streams between elements in subsequent layers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work explores the relationship between computational neuroscience approaches to associative memory (e.g., the literature on hopfield networks) and in-context learning in transformers. A variant on the self-attention mechanism is shown to be capable of in-context learning, and the relationship to associative memory is analyzed. Taking inspiration from this, the authors then propose an architectural modification to transformers, and present evidence that this modification improves performance in both a toy in-context learning task and naturalistic pretraining.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper would benefit from addressing the following issues:\n- A tighter connection should be established between the AMICL model and the proposed modification to transformers. Additionally, other models could be explored that have a stronger connection to the AMICL model, e.g., eliminating the $W^v$ matrices, or incorporating residual stream connections between the input in layer l-1 and the value in layer l.\n- The decision to make values equal to inputs in the AMICL model should be better motivated, especially given that this is the main inspiration for the modified transformer architecture. What is the basis for this design choice? Is it based on a particular source of biological inspiration, or is there a specific reason to think that it should yield improved performance?\n- More explanation should be provided for why the modified transformer architecture yields performance benefits on the IOI task, despite not meaningfully improving overall language modeling performance.\n- More control models should be explored, especially including other possible configurations for additional residual streams.\n- In figure 3, the bottom right plot shows loss increasing for the IC2 task. This seems inconsistent with the increasing accuracy for this task. Is there an explanation for this discrepancy?", "broader_impact_concerns": "There is no need for a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S2lLw4yyub", "reviewer": "Reviewer_g8Zb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The definition of in-context learning is quite clear. The further analysis and case studies can represent the real ICL behaviors of language models.\n2. The experiment setting is computation-friendly. Using a two-layer Transformer for toy-dataset experiments are cheap and easy to reproduce.\n\nWeaknesses:\n1. Most of the experiments con conducted under a very small scale. Usually, the toy dataset shows the intrinsic capability of an architecture, where the accuracy is near to 100% or random. However, The experiments in Figure 3 only shows the convergence speed, which can not show the capability of difference architectures. For example, [1] shows that data-driven priors make big influence on the toy-dataset performance.\n2. Evaluation on real in-context training tasks is infeasible for only 8-million model size. Therefore, the experiment is not solid for the claim that value stream improves ICL.\n3. The value is only the linear projection of input layers. Therefore, most of the properties in the value residual can be obtained from hidden residual:\n$$\nV_2=W_2^V \\mathrm{norm}(X_1+X_2)+V_1=W_2^V\\mathrm{norm}(X_1+X_2)+W_1^V\\mathrm{norm}(X_1)\n$$\nSince$X_1+X_2$ already contains the residual information of $X_1$, another residual can not probably bring new capabilities.\n\n\n[1] Never Train from Scratch: FAIR COMPARISON OF LONG\u0002SEQUENCE MODELS REQUIRES DATA-DRIVEN PRIORS", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed value stream as a residual branch to enhance the in-context learning capabilities of large language models. The evaluation is in the ICL-specific dataset. In a two-layer Transformer, value stream shows better convergence speed than baseline models. In 8-layer small language models, value stream shows a better performance on ICL tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Personally, I completely understand that the small-scale of the experiment is due to the limited computation resources available. However, I do not believe that the experiments on the very small model can make any effective evaluations and conclusions.  I think the authors are better to try other research directions.\n\nIf focused on the the paper itself, I think the authors should either make the theoretical or empirical studies more solid. For theory perspective, the paper should prove that value residual broadens the expressiveness of the baseline Transformers. For empirical perspective, the real ICL and general evaluation tasks should be included. Adding the scaling analysis is better since the advantages of a simple residual enhancement can easily disappear with the scaling model depth.", "broader_impact_concerns": "No other concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JmXKjO8XlJ", "reviewer": "Reviewer_rkAz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main weakness of this manuscript is that it makes a number of imprecise and poorly-justified claims. I think the main idea is somewhat interesting, but the clarity and precision of the paper must be improved. \n\nFirst, I think the \"AMICL\" model could be introduced more clearly. This is extremely important, as this is the key conceptual contribution of the paper. I suggest that the authors unpack the last few paragraphs of Section 2; for instance they might convert the equations for the keys and queries into display equations. There are also a number of statements regarding the construction of the model that require justification, for instance: \n\n- \"The value of a can be any arbitrary positive real value, but, after testing within the range [0, 2], is set as a = 2.\" Where is justification for this provided? Why is an upper limit of 2 selected? \n\n- \"Seen through the lens of the traditional self-attention mechanism, construction of the AMICL model in Section 2 can be interpreted as implying that creating the values V in a way which more explicitly retains the prior input X can facilitate ICL.\" How does the construction of the model imply this, until it is demonstrated that this model displays advantages in ICL performance? \n\nI also have strong concerns regarding the authors' claims about their model's relationship to biology. Most simply, the authors claim that their residual value stream architecture is \"biologically-inspired\". The study of associative memory models has its roots in neuroscience, but it is not clear to me that there is anything inherently biological about this architectural modification. I list several more concerns regarding biological claims below: \n\n- In the Introduction, the authors state that \"Developing models which connect our understanding of common mechanisms underlying these related phenomena in both artificial and natural intelligence may provide valuable insights for developing more adaptive and versatile language models, in addition to helping us more deeply understand the brain.\" The assumption that there are necessarily common mechanisms is, in my opinion, quite non-trivial. To wit, showing that common mechanisms underlie rapid learning in LLMs and in animals would be a highly significant discovery. \n\n- The Discussion includes two paragraphs on potential biological insights arising from the proposed models, which make sweeping claims without adequate justification. First, how can the results \"be viewed as supporting cognitive functional specialisation and modularisation as being both natural and commonplace in neural systems\"? Perhaps they demonstrate one example where functional modularization can provide a performance benefit, but how does this show these features to be \"natural and commonplace\"? I do not follow. Second, the authors raise the question of whether their \"AMICL\" model or residual attention stream modification can be considered analogous to skip connections witnessed in the hippocampus\". However, they provide no specific predictions that could be tested, merely an analogy at the level of the existence of skip connections. I would suggest that the authors reify or remove these claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This short paper brings an associative memory perspective to bear on the phenomenon of in-context learning in Transformers. Its main contribution is a proposed augmentation of the standard Transformer architecture by additional residual streams carrying value information between attention blocks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have already discussed at some length my concerns regarding the manuscript's clarity and its biological claims under Strengths and Weaknesses. In short, I think substantial re-writing would be required in order for me to recommend acceptance. Here I list other concerns. \n\n- In the discussion, the authors state that \"This itself is notable given ICL is not typically seen in single-layer Transformers (Olsson et al., 2022).\" This claim must be qualified, as it depends on what ICL task one has in mind. As shown by Zhang et al., \"Trained Transformers Learn Linear Models In-Context\" and Lu et al., \"Asymptotic theory of in-context learning by linear attention\", a single attention block is sufficient for ICL of simple linear regression tasks. \n\n- References to a line of work by Bietti and colleagues analyzing Transformers from an associative memory perspective (e.g., \"Birth of a Transformer: A Memory Viewpoint\" or \"Understanding Factual Recall in Transformers via Associative Memories\") are notably absent. \n\n- One other limitation of this work in its current form is that it uses only two datasets: the synthetic ICL classification task from Reddy (2024), and the TinyStories dataset. It would be interesting to probe whether the proposed model displays strong performance on other synthetic ICL tasks, e.g., in-context regression.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)