Fast weight programming and linear transformers: from machine learning to neurobiology
Authors: Kazuki Irie
Samuel J. Gershman
Venue: Accepted by TMLR ()

Abstract: Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNs, known as Fast Weight Programmers (FWPs), can be interpreted as a neural network whose synaptic weights (called fast weights) dynamically change over time as a function of input observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPs and models of synaptic plasticity in the brain, suggesting a convergence of natural and artificial intelligence.

Published in Transactions on Machine Learning Research (12/2025)
Fast weight programming and linear transformers: from machine learning to neurobiology
Kazuki Irie"
kirie@@fos.havvard.de edu
Deportment of Psychology and Center for Brain Science Horvard University, Cambridge, MA, USA
Samuel J. Gershman"
vershmann@fas.havvevn edu
Department Psychology and Center for Brain Science
Kempner Institute for the Study of Natural ond Artificial Intelligence Horvard University, Combridge, MA, USA
Reviewed on OpenReview: https: openreview. net/ form?? id= TDG9EkNMQR
Abstract
Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNSs, known as Fast Weight Programmers (FWPs), can be interpreted as neural network whose synaptic weights (called fast weights) dynamically change over time as function of inpul observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPss and models of synaptic plasticity in the brain, suggesting convergence of natural and artificial intelligence.
1 Introduction
While early development of artificial neural networks (ANNs) was loosely inspired by neuroscience (McCulloch and Pitts, 1943; Rosenblatt, 1958; Fukushima, 1980), ANNs have rapidly evolved into an independent subfield of machine learning (MT.) and artificial intelligence (AT) on their own (Rumelhart et al., 1986b; McClelland et al., 1986; Ivakhnenko, 1971; LeCun et al., 2015; Schmidhuber, 2015). While some argue for the continued influence of neuroscience on the development of ANNs (Zador et al., 2023; Hassabis al., 2017; Macpherson et al., 2021), in reality, progress in ANNs has been largely driven by the pursuits of computer scientists to develop ever more powerful, general, and efficient ML models, rather than through their original motivation to model brain computation under neurobiological constraints (Gershman, 2024). In a twist of fate, such a detachment the ANN research from the traditional goals of neuroscience has led to more open-ended and flourishing developments in ANN models, which, in turn, have attracted interest from cognitive neuroscientists, as they represent the best existing computational systems for processing vision, audio, and language (Achiam et al., 2023; Pratap et al., (724)-modalitaee that. are central human cognition in real-life scenarios. Such successes of ANNs have stimulated many cognitive neuroscientists to seriously examine ML-driven models as hypotheses to explain neural computation in the brain (Kriegeskorte, 2015; Yamins and DiCarlo, 2016; Schrimpf et al., 2021; Gershman et al., 2025).
However, given the rapid progress in ML, there is still a significant gap between the computational modeling toolkits in the two fields. In particular, boosted by the recent successes of ChatGPT (Achiam et al., 2023)
Kazuki Trie and Sam Gershman took the lead in writing the machine learning and nenroscience sections, respectively.

---

Published in Transactions on Machine Learning Research (12/2025)
and other language models, myriad of sequence processing neural network architectures have been proposed to improve upon the standard transformer architecture (reviewed later) (Vaswani et al., 2017). While keeping track of every such models has become particularly challenging today, as the lack of formal naming conventions makes their (often obvions) mathematical relations opaque (c.g., how does "Mamba2" (Dao and Gu, 2024) relate to "Gated Linear Atlention" (Yang el al., 2024a) ?? answered in Sec. 3.4), it seems useful to summarize key elements of such advances in sequence and memory modeling in ML that are arguably relevant neuroscience.
In this Primer, we present special family of recurrent neural networks (RNNs; Elman (1989); Jordan (1986)) called Fast Weight Programmers (FWPs; Schmidhuber (1992b); Schlag et al. (2021a); Iric et al. (2021)). that has been well established in machine learning, but has yet. to see broad dissemination and application within the neuroscience community. Unlike the conventional RNN with one-dimensional vector-forn hidden states, states FWPs are two-dimensional (2D) matrices (sce Figure for illustrations). As we will argue, such 2D-state RNNs are particularly relevant for neuroscience, as the matrix-form states can be interpreted lime-varying synaplic weights that maintain short-term memory unlike conventional RNNs, in which all the synaptic weights are fixed after training.
Additionally, FWPs naturally introduce a novel perspective on achieving biologically compatible local learning in ANNs, with an intuitive connection to the now-popular MT. concept of in-conted learning. Overall, FWPs may address certain longstanding limitations of ANNs as models of their biological counterpart, by providing a novel timescale for learning and memory.
The FWP concept is entry point for understanding a multitude of sequence models recently proposed in machine learning. In fact, many such models be directly expressed as an instantiation of FWPs, with specific choice of the update rule used to modify the fast synaplic weights (see Table for preview); and we also review a formal connection between FWP and the transformer architecture (Vaswani et al., 2017).
We hope this Primor will stimulate neuroscientists to rethink computational modcling of cortain neurobiological Teatures ANNs, through unique properties of FWPs; and help them familialize with the slate-of-the-arl. sequence and memory models from machine learning.
Glossary (Machine Learning)
Efficient sequence model: a parameterized sequence model whose training can be parallelized over the sequence length, and whose inference time-complexity linear in sequence length.
In-context learning: an ability of a sequence model to learn new task when sequence of task demonstrations is fed to its input.
Metalearning: process leveraging learning experiences to acquire or improve the ability learn.
Model expressivity: the range of computations the model can perform.
Sequence processing neural networks: type of artificial neural network designed to handle sequence of inputs.
2 Preliminaries
Before delving into fast weight programmers (FWPs) in the next section, we briefly review some of the conventional, gencral-purposs sequence processing neural networks: the conventional RNN (Elman, 1989; 1990) (Sec. 2.1) and related state space models (Sec. 2.2), and the transformer neural network (Vaswani et al., 2017) (Sec. 2.3). This will be useful later for contrasting, relating and characterizing properties of FWPs compared to these conventional sequence models.
Note that our main focus here is on sequence processing RNNs, rather than other "non-sequential RNNs" such as Maaui-LittleHHopffed networks (Amari, 1972; Little, 1974; Hopfield, 1982). We also assume that readers are familiar with the general idea of machine learning that "trainable parameters" of model can be trained by using some learning algorithm (e.g., backpropagation through time; BPTT (Rumelhart al., 1986a; Werbos, 1990)) given some dataset environment and objective function to be optimized.
2

---

Published in Transactions on Machine Learning Research (12/2025)
b
W
W
Y1
St
t-1
Update
Rule
R
S
K
Vt
9
W
W
w/
*
000
000
*
000
W
00
0
xt
X
V
V,
V/2
Attention
K
qt
ky
K2
W
W
000
000
o0
000
W
X
Figure An Illustaaionn of sequence procossing in a: conventional recurrent neural networks (RNNs), b: fast weight programmers (FWPs), and c: transformers. One time step of recurrent computation is shown. all figures, colored circles indicate time-step specific variables (green) that are not retained for the next time step, temporally changing model satt/sshott-temm memory (yellow), and model parameters that are fixed/frozee after training (blue). In particular, the hidden state W, of an FWP is context-dependent time time-varying varying matrix, whereas the hidden state of conventional RNN 8, is vector, and that of transformer is the key-value memory matrices K, and V. whose size grows lincarly with the sequence length. In b, black arrows indicate computation in the fast/main net of the FWP, while the remaining gray arrows correspond to the computation performed by the slow how//pogrammes net. Activation functions and variables that are specific certain variants of FWPs (such dynamic learning rate or state decay factor) are omitted for clarify; see Table for a specific choice of the update rule used in various models.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Throughout our Primer, T, T, H, d, drey, dn and dout denote positive integers; and @ denote element-wise multiplication and outer product, respectively. Our vectors are column vectors, which implies that outer product also writes as a @ = ab E _= for arbitrary vectors a E IR" and b E I (this remark is useful to recognize outer products in certain equations).
2.1 Conventional recurrent neural networks
At every time step f, the conventional sequence processing RNN with a hidden state 8t-1 E R reccives an input 2 E Rdd and produces an output 8, E Reduut as follows:
= OW 81-1 W'
(1)
where a is an activation function (e.g., tanh), and W R E Wddns Xdow and W E Rddo are recurrent and input weight matrices, respectively, which are the trainable parameters of the model. The initial state sn is typically set to the "zero vector" whose entries are all zero. We omit the additive bias term inside which is irrelevant for our discussion. See Figure la for an illustration.
Note that, in machine learning, this RNN architecture is never used as-is in practice today, as it is known to critically suffer from the classic "vanishing gradient problem" (Hochreiter, 1991; Bengio et al., 1994; Hochreiter et al., 2001a) when trained using a gradient-descent based learning algorithm, and to yield sub-optimal performance in practice (note that another well- dell-known problem, the "exploding gradient problem" can be rather casily remediated by heuristically clipping/trume /truncatunnn large gradients to a certain value (Graves, 2013; Mikolov, 2012)). Instead, more sophisticated "gated architectures" such as long short-term memory (LSTM (Hochreiter and Schmidhuber, 1997; Gers et al., 2000; Greff et al., 2016)) are typically used. The core temporal dynamics of LSTMs maintains two recurrent states C2 E Iddat and 8, E Rdout
= r2 C C--++ ds ①
(2)
$c = o2 @ tanh(c;)
(3)
where the "gate functions" T2, ée, O E Rou and "cell input" 2, E Rdutt (c, is often referred to "cell state") are all parameterized functions of the recurrent state 8:-1 and input D; in Eq. 1.
Nevertheless, the vanilla RNN with the above Eq. is sufficient as a contrastive example to later highlight the unique properties of FWPs. In this RNN model, during training, the "synaptic weights" W and W are modulated by the learning algorithm (typically by using gradients computed through backpropagation through time); however, once the training ends, these weights become frozen and immutable. At test time, the state vector 8, is the only time-varying variables which carry the model's short-term memory to process sequence elements over time. Later, we will show how FWPs critically differ from the conventional RNN this aspect.
2.2 State Space Models
Given the current ML research trend (Tiezzi et al., 2025) of developing efficient sequence models delineed models for which training can be parallelized over the time dimension, whereas inference time-complexity is linear in sequence length (Yau et al., 2025), the so-called "state space models (SSMs)" or "lincar RNNs" have become popular (Gu et al., 2022; 2021). An SSM can be directly obtained from the conventional RNN above by simply removing the (non-linear) activation function a in Eq. 1:
8: = R 8 W'
(4)
While deriving such a model from the original RNN is straightforward, this model is also known as a "lincar time-invarianl dynamical system" (which is originally defined through a contimuous-tine differential equation, whose discretization yields Eq. 4 exactly, which is typically followed by an extra equation ye = W st to produce the output vector y: E Rd using an extra weight matrix Wo R xdbut). We do not further delve into such an alternative view here, as it provides no additional insight into the resulting computational model for our purpose.
4

---

Published in Transactions on Machine Learning Research (12/2025)
The definition of an SSM can be generalized to include models with time-varying weight matrices W.a e Rdout Xdoout and W! E RRo Xdn indexed by t:
8t W 8.-1 + w/
(5)
By rocognizing that such dcfinition can also express models in which 8 is matrix instead of a vector, we will see a connection to fast. weight programmers (Sec. 3).
An interesting class of SSMs can be obtained by setting the weight matrices W." and W in Eq. 5 to be diagonal matrices with diagonals rt E Rdon and &s E Rdout (by sctting dout = dm), respectively; we obtain the following element--wse recurrence:
8: = r2 ④ 8t-1 + d ④ Zt
(6)
It should be noted that this equation is identical to the cell update in an LSTM (Eq. 2), except that, in this SSM, Te and é are functions of xy or a few carlier observations (e.g., the last four; Tt, æt-1, Tz-2, 01-3) only, i.., the "gate functions" are not recurrent (no dependency on 8,-1), and the cell input 2, in Eq. is reduced to at.
An efficient SSM can be obtained by deliberately using this linear element-wise recurrence of Eq. as the core temporal processing operation of the model (no other form of recurrence is used). This choice has two crucial consequences: first, it enables training parallelism (i.e., there exists an efficient algorithm to compute 8t for all t in parallel (Martin and Cundy, 2018; Blelloch, 1990); which is not the case for the conventional, fully recurrent network); second, it sacrifices model expressivily, i.e., abilities to perform certain computations (Merrill et al., 2020; Grazzi et al., 2025; Merrill et al., 2024) (we discuss this further in Sec. 3.6) as a side note, there is also a third consequence, which is that element-wise recurrence enables efficient online lcarning through a tractable real-time recurrent learning algorithm (Mozer, 1989; Gori et al., 1989; Zuechet al., 2023; Trie et al., 2024), but further discussion is out of scope here."
This "cell-only LSTM" is currently often called "Mamba", after the name of an efficient "hardwarc-awaro implementation" (Gu and Dao, 2024), i.e., an implementation that takes into account the low-level memory hierarchy of modern graphic processing units (GPUs). However, we note that many have previously proposed similar models based on Eq. 6 under various names including Quasi RNNs (Bradbury et al., 2017) and the Simple Recurrent Unit (Lei et al., 2018), among others (Balduzzi and Ghifary, 2016; Li et al., 2018; Gonnet and Deselaers, 2020) to achieve improved efficiency over LSTM.
2.3 Transformer neural networks
Here we review the Transformer neural network (Vaswani et al., 2017) which is today's de facto standard sequence processing model architecture in ML. It also has a direct mathematical connection to FWPs (Schmidhuber, 1992b; Schlag al., 2021a), as we will review in the next section.
Transformer models come in three distinct variations: encoder-decoder ( (Vaswani et al., 2017), encoder-only (De- vlin et al., 2019), or decoder-only (Liu et al., 2018) architectures. We focus on the decoder-only (also known as the "causal model" archilecturr), which is a general-purpose sequence model used for example in language mod- els (Al-Rfou et al., 2019; Dai et al., 2019; Baevski and Auli, 2019; Irie et al., 2019) including OpenAl's GPT se- rics (Radford ct al., 2019; Brown et al., 2020); for brevity, we refer to this as "the" transformer architecture here.
A typical transformer architecture consists of multiple layers, interleaving "self-attention" layer (Parikh et al., 2016; Cheng et al., 2016; Lin et al., 2017; Bahdanan et al., 2015) and two-layer feedforward block, each com- bined with a residual connection (He et al., 2016b;a; Srivastava et al., 2015) and a layer normalization operation (Ba et al., 2016b). Among these layers, the core temporal/memory memory processing operation in the transformer is carried out by the self-attention layer; therefore, we focus on describing its sequential dynamics here.
Like the conventional RNN in Sec. 2.1, at every time step 1, a causal self-attention layer receives an input 3, E dh and produces an output y: E Rdoa, while maintaining the so-called "key-value memory", represented
As further aside, in theory the expressivity of such clement-wise RNNs could be enhanced by using complex-valued diagonals in the recurrent weight matrix (Orvieto et al., 2023; Ran-Milo al., 2024) (which yields expressivity of real-valued full matrix); however, stable implementation thereof is often reported to be challenging in practice (Elelimy et al., 2024).
5

---

Published in Transactions on Machine Learning Research (12/2025)
by two matrices K, RRd and Vi xt as follows:
9: = W9x; xs Kj W k It ; = W at
(7)
K, = 1 kb V = [V_ Uy]
(8)
y1- Altention( K2, Vi, = Visoftmax(K) g))
(9)
Σ 0,
(10)
where qe, ki Rddey Ur E Rdome in Eq. 7 are different projections of the input called query, key, and value, respectively through matrices W Bh xdin, W E Rh Xdi, and WV E R xdi which are the trainable parameters of the model. The operation [] as in [K,-1, k2] denotes concatenation of vector & E Rue to matrix K2-1 E R ((t-1) along the time dimension, yielding K, E R x Ko and Vo are initially empty. We omit, the dley scaling inside softmax, as well as the output projection, which are typically used but are irrelevant to our discussion here. In Eq. 10, Oxt, E IR denotes the T-th element of the vector softmax K, q:) R'. See Figure 1c for illustration.
The computation in Eq. 9 is called "attention", which is rather an intuitive name when we read Eq. as (1) comparing the query vector from the current step & to the keys from all the time steps through dot product to produce a score for each of / keys (K, q. E RR', (2) sharpening and normalizing these similarity scores through the softmax function to obtain "attention scores" (at,r 2 0 for all r from 1 to t with Σ Qt,, = 1), and (3) using the resulting scores as coefficients to compute the weighted average of all the value vectors (as is explicitly shown in Eq. 10) to produce the output; this effectively implements a form of attention as the model focuses on certain key-value memory elements at each step t.?
Additionally, practical transformers typically make use of "multi-head sclf-attention", where multiple indepen- dent heads within the same layer perform self-attention in parallel. By introducing an extra typer-paramett H as the number of heads, after projection of Eq. 7, each query/kor taar//ky/aallu /value vector is split into H sub-vectors of the same size (dkey and dout are set to be a multiple of H); the self-attention operation above is computed independently for the H sets of query wrry/key /key/vallee valuue vectors. The results from each head (each of size dout/ H) are concatenated to produce the output Y/ E Rdou..
One important distinction between the transformer and RNNs (Sec. 2.1) is their computational complexities. As can see in Eq. 8, the size of key and value memory matrices lincarly grows with the time step (i.e., sequence length) unlike in the conventional RNN whose state has constant size. This results in quadratic time complexity w.r.t. sequence length in the attention computation of Eq. 9-wheraas complexity is lincar in RNNs (i.e., compute is constant per time step). Consequently, practical self-attention requires predetermining a maximum sequence length, also called context window size, and any old events that fall outside the window are discarded. On the other hand, training of a transformer can be highly efficient. All the computations above are parallelizable over the sequence element by introducing the so-called "attention mask" inside the softmax as follows. By denoting an input sequence with T elements as X = (ar1 .., [T] Rdu XT (X, x, for all t) and the outputs Y [y,.... yr] E Rdon XT (and by analogously denoting queries, keys, and values for T steps as Q, K E R. xT and V E RdouxT XT, respectively), parallel computation performs:
Q = - K = w x; V = W'
(11)
Y = V /oftmaxx( M O (K (T)))
(12)
where M E RTXT is the so-called attention mask. These equations are equivalent to the sequential Eqs. 7-9 by setting M to be the upper triangular matrix, i.e,, M.j 1 if i < and Mij -oo otherwise; which explicitly sets certain attention weights to zero, as the causal model cannot access data from the future.
Practical implementations of transformers have been highly optimized. In particular, an efficient hardware- aware implementation is available (Dao, 2023), leveraging the "online softmax algorithm" (Rabe and Staats, 2021; Milakov and Gimelshein, 2018), which significantly reduces memory requirement (crucial for GPU
2It has been recognized that Eq. can also be interpreted single-step iteration that minimizes special energy function defining Amari-Litte Hopfeel network; we refer Ramsaner al. (2021) for further details.
6

---

Published in Transactions on Machine Learning Research (12/2025)
efficiency) compared to the naive algorithm that explicitly stores K Q (2023) for further details.
KTQ E in Eq. 12; we refer to Dao
3 Fast Weight Programmer Neural Networks
Here we present the concept of fast weight programming, and the resulting sequence models as well as their key properties.
3.1 Basic Instantiation: FWP with a purely additive update rule
Before introducing the general definition of fast weight programmers in the next Sec. 3.2. Here we first provide the most. basic instantiation of FWPs: an "FWP with a purely additive outer product update rule" (Schmidluber, 1992b) as an illustrative example; we refer to it as "vanilla FWP"
Like the conventional RNN and transformer, a vanilla FWP is a general-purpose sequence model. At every time step I, the model receives an input a, Re and produces an output y: E Rd while maintaining the so-called "fast weight" matrix We Rdout Xdxay a short-term memory storage, as follows:
q: = WPxx; k = W k x4 oz = WV.. x
(7)
W. = W + 1s 2 (k() y: = Wid(qi)
(13)
(14)
where Eq. 7 is the same as in the transformer (Sec. 2.3) with trainable parameters WQ Ru Xdin, WK E RRdwy Xdi,, and WV E agooutduu and as we'll discuss in Sec. 3.3, the connection between this model and the transformer does not end here. ø is an activation function discuss later (while the activation v is optional, in practice it is often also applied to v). The "fast weight matrix W. E Rdout x daay in Eq. 13 is initially set to 0, i.e., Wo O. See Figure lb for illustration.
This model can be viewed as a system of two networks (Schmidhuber, 1992b) where one net the slow net, corresponding to Eq. 7 (note that the three equations Eq. 7 can be grouped into a single matrix multiplication by defining a "slow weight" matrix W [Wq, Wk WV] (dund) Xxdayy by row concatenation)-lennn to program or train another net, the fast net (Eq. 14) by generating its weight changes (Eq. 13). The fast weight change is defined through update rule; here Eq. 13 takes the functional form of a Hebbian-like lcarning rule (Konorski, 1948; Hebb, 1949) whose update term is a simple outer product term. 3 The connection to SSMs (Sec. 2.2) is also noticeable: Eq. 13 is essentially a linear RNN with 2-dimensional state W, (which could potentially be flattened and operationalized to be a one-dimensional vector state).
From a neuroscientific viewpoint, we may interpret this mechanism as rapid synaptic modulation (Panichello et al., 2024; Spaak and Wolff, 2025) property which is absent in the conventional RNNs (Sec. 2.1). Under this view, it might be natural not to consider the fast and slow net as representing the same type of fneural network" but rather, it may be more appropriate to consider the slow net as representing some "molecular network""which is reminiscent of Denis Bray's view on ANNs (Bray, 1995; 2003; 2009), implementing molecular mechanisms that support. learning and memory in the (fast) neural network. We provide further neurobiological discussion in Sec. 4.
From the memory system perspective, Eqs. 13-14 also correspond an associative memory storing key/value pairs pairs-alsy also called correlation matrix memory (Kohonen, 1972) (see also Steinbuch and Piske (1963); Willshaw et al. (1969)), whose writing operation is outer product of the key and value vectors (Eq. 13); and its peading/retriene operation is matrix-vector multiplication between the memory matrix and a query vector (Eq. 14). From a cognitive science view point, this model can also be seen as a learnable 2D version of the "tensor product representations" (outer product is the 2D tensor product) (Smolensky, 1990; Schlag and Schmidhuber, 2018; Schlag et al., 2019; 2021b) an ANN model to bind two pieces of information (Greff et al., 2020).
Indeed, a very similar model was more recently proposed (Limbacher and Legenstcin, 2020), with neuroscientific motivation leverage Hehbian plasticity for sequence processing, and later extended spiking neural networks (Lambacher al., 2023); see also Najarro and Risi (2020).
7

---

Published in Transactions on Machine Learning Research (12/2025)
3.2 Core Concept
As is examplified by the model in Sec. 3.1, a fast weight programmer (FWP) (Schmidluber, 1992b; Irie et al., 2021) is defined as a neural network system in which one (sub)network, called slow net, generates modifications to weights of another (sub)network, called fast net, as function of a sequence of input observations. By conceptualizing weights of neural network as its mrogram/software (Schmidhuber, 1990a), such a slow net which modifies the weights of a fast net is programmer. The weights of the fast net are fast, because they can rapidly change as a response to observations received at every time step, while those of the slow net are slow because they are typically trained by some learning algorithu that updates slow weights on the sequence level (and they typically become fixed after training).
In most of the existing and practical FWP models, both fast and slow nets are one-layer networks, and the weight update rule is some outer-product based one. However, the general concept of fast weight programming is more general and has such a restriction (Trie et al., 2021); in principle, more complex (e-g., deeper) slow/fas ffatt nets or other update rules could be used. In fact, the core concept of FWPs is the idea of training& network to train (e.g., generate weights of) another network-an an idea which has been rebranded as "hypernetworks" (Ha et al., 2017) in the modern deep learning literature. One common challenge though is to deal with the high-dimensionallty of ANN weights, which are typically too large to be directly parameterized as outputs of an ANN (except for tiny networks (Gomez and Schmidhuber, 2005)). The use of outer product elegantly overcomes this challenge by generating two small vectors instead of one large matrix, and it's arguably more practical compared to other alternative methods that rely on weight compression (Trie and Schmidhuber, 2021) or sparsity (Munkhdalai, 2020).
Another issue is that naively applying the BPTT learning algorithm to FWP would require storing all the intermediate weight matrices for each time step for backpropagation, which would yield memory requirements that can casily exceed the amount of memory available on GPUs. Therefore, practical FWP model designs also require compute-effiiint recomputability of fast weights (e.g., through reversibility of the update rule; we refer to the corresponding discussions in prior work (Schlag et al., 2021a; Irie and Schmidluber, 2021)) or chunk-wise processing (sce Box 1).
As a historical note, McCulloch and Pitts (1943) also discussed recurrence and dynamic synapses (which they called 'circle' and 'alterable synapses', respectively) in their seminal paper on ANNs. Their proposal was to replace dynamic synapses by recurrence which can potentially simulate the effect of dynamic synapses (see their informal "theorem 7"). However, fixed weights of ANNs were later criticized as a limitation from both the thcoretical neuroscience and machine learning standpoints (von der Malsburg, 1981; Feldman, 1982; McClelland, 1985), and the possibility to introduce fast synaptic modulations was investigated. In particular, von der Malsburg (1981) and Hinton and Plaut (1987) proposed networks whose effective weights are defined as multiplicative (von der Malsburg, 1981) or additive (Hinton and Plaut, 1987) superposition of fast and slow changing weights. However, none of these carly works has proposed mechanism to learn the dynamics of synapses (e.g,, Hinton and Plaut (1987) merely used two different learning rates for the fast and slow weights, while jointly training both sets of weights through the same gradient descent learning algorithm). It was only in the carly 1990s that end-to-end differentiable and Icarnable synaptic modulation dynamics above (originally called "fast weight controllers"). featuring the "programming" part-the the 'P' in PWP-was proposed (Schmidhuber, 1991; 1992b), as an alternative to the conventional recurrence (Sec. 2.1) and was also computationally motivated by the idea of reducing the ratio of trainable parameters to temporally changing variables in sequence processing networks (Schmidhuber, 1993b). The FWP concept has seen a recent, revival (Schmidhuber, AT Blog, 2021; Irie and Schmidhuber, 2022) mainly due to its formal connection to the transformer and its potential to overcome certain limitations of transformers (as we'll see in the next. sections).
"Note that this fast vs. slow distinction may not hold in some edge cases where some fully ouline learning algorithm, such as real-time recurrent learning with update frequency of one time step, is used to learn the slow weights (in which case the slow weights also updated at every time step)- even though such learning algorithm is common in practice; see, c.g., Irie et al. (2024). Nevertheless, this terminology conceptually appropriate and illustrative of the characteristic timescale difference underlying FWPs.
8

---

Published in Transactions on Machine Learning Research (12/2025)
3.3 Formal Connection to Transformers
Here we present the formal connection between the vanilla. FWP (Sec. 3.1) and the transformer (Sec. 2.3). We discuss this relation in two didactic steps by looking into (1) transformer without softmax, and (2) transformer with an alternative (but still normalized) attention-score function.
Transformer without softmax. First, we examine the consequence of simply removing the softmax in the self-attention of Eq. 9, which yields:
yI = V. q:) = (V,K,)) )q
(15)
The removal of softmax opens up the possibility to reorganize the computation by first multiplying V,K,' before multiplying with the query qt- By denoting this key-valuc product as W, V,K," Rdoo x xdxar, can further express it in terms of the column vectors in each of the key and value matrices (recall their definition of Eq. 8) as in the following Eq. 16:
1
W, = V.K, = Σ ur & k
(16)
W + Us & k.
(17)
Further isolating the last term in the sum of Eq. 16 yields the above Eq. 17 which expresses a recurrent formula for We Overall, the sequential dynamic of the transformer without softmax can be rewritten as:
q: = WW xz k; = W xz 02 = W x
(7)
W. W,-1 24 @ kr
(17)
y: W.g.
(18)
which we can recognize as being identical to the vanilla FWP in Sec. 3.1, up to the missing activation function applied to and kg- This means that, the exact input/output mapping of a transformer without softmax can be equivalently expressed by an FWP.
This equivalence result may be intriguing at first, because even without the softmax, the transformer model stores key-value memory storage that grows with the sequence length (potentially to infinity), while the FWP has a fixed-size memory storage in the fast weight matrix (compare Figure 1b with Figure 1c). This highlights the role of softmax as a powerful retrieval function enabling sharp discrimination between large set of memory elements; without such a discriminative retrieval function, a key-value memory system with even an infinitely growing memory size is merely as powerful as an FWP system with a fixed memory size.
Transformer with linearized attention (Linear transformer). While the derivation above based on the simple removal of softmax is straightforward and captures the core matrix-algebra manipulation underlying the equivalence between the transformer and the vanilla FWP, we can also revisit the removal of the softmax, instead replacing the ooftmax-normalln attention score computation softmax(K] q:) in Eq. 9 (the softmax kernel) by another kernel function, which computes the normalized attention scores (for r from to f) as:
d,,, ((k,) (((() E.
(19)
(kk,)) T())))
for an arbitrary function ó with positive co-domain. Compared to the case above where we simply removed the softmax, we have extra & applied to the keys and the query, and the denominator that normalizes the attention score. Despite these differences, we can similarly reorganize the computations in the corresponding
HHinton (2022) asks: "For sequential data, is possible use fast weights mimic simplified transformer?" The answer is yes, as shown in Schlag al. (2021a), which review here.

---

Published in Transactions on Machine Learning Research (12/2025)
self-attention computation follows:
(Er=1 us @ (((k)) Φ(q:)
Σ+= o (r((k, T((a))
yt d
(20)
r=1 E (kk,,) ((()))
E ((kk.)) δ(q:)
1 W()(q()
(21)
2, ((q.)
where, in the numerator, W, is defined as W. = Ei= Dr @ (k,, E Rauad whose recurrent update function can be derived similarly to Eqs. 16-17, and 2₁ E R.dwy in the denominator of Eq. 21 has the following recurrent update equation with 2g = 0:
21
o(k,)) = zc-1 + o(ks)
(22)
Overall, the sequential dynamics of a transformer model based on an alternative normalized attention function defined by Eq. 19 can be rewritten as:
q: W xx ; k W xt ; 01 WV xz
(7)
W, WW, + 11 @ ((k:)
(13)
z1-1 + (kis)
(22)
Y: 2J T(q))) Wid(q:)
(21)
This is the "recurrent form" of the so-called "linear transformer" (Katharopoulos et al., 2020). This system is identical to the FWP of Sec. 3.1 up to the normalizing denominator z, o(g:) E R in Eq. 21 and tracking of the extra time-varying variable z (Eq. 22). From this view, the vanilla FWP is essentially an "unnormalized linear transformer" (ULTRA)). In fact, recent work extending linear transformer models (discussed in Sec. 3.4) has shown that such normalization is unnecessary in practice (Schlag et al., 2021a; Sun et al., 2023; Yang et al., 2024a;b).
While "linear" in the name "linear transformer" could highlight how the linearized attention function of Eq. 19 allows for its computation to be reorganized unlike the softmax attention, it primarily refers to its time complexily. Unlike the quadratic complexity of the sofumax atlention (Sec. 2.3), the computation per step is constant w.r.t. the time step/sequends length in this model; the resulting time complexity is linear w.r.t. sequence length like with RNNs. This is an example of efficient sequence models as its training is parallelizable using the "attention form" " parallel computation analogous to that of softmax atlention (Eq. 12) can be derived, while its inference is linear-time complexity by using the recurrent form above. Naturally, such a computational advantage comes with cost: performance of the linear transformer largely lags behind that of the quadratic transformer in practice (Katharopoulos et al,, 2020; Schlag et al., 2021a). However, as we'll see in the next Sec. 3.4, more effective but still elficient models can be derived by extending the vanilla FWP model.
As a side note, while the original linear transformer by Katharopoulos et al. (2020) simply used Φ(x) = (LU((:)) + (where E.U denotes "exponential ial linear unit" (Clevert et al., 2016)), both Choromanski et al. (2021) and Peng et al. (2021) proposed a lincar transformer that uses random feature kernels (i.c., & is not just a judiciously chosen activation function but also involves up-projection using randomly sampled features) which are formal approximations of the softmax in theory. However, such approximation methods (which hold with infinite many random features) do not perform well in practical scenarios.
As a historical note, while the derivation of the recurrent form of the linear transformer from its altention form was provided by Katharopoulos et al. (2020) (2020), the same mathematical derivation can also be found in Ba et al. (2016a) (2016) which connected special instantiation of recurrent FWPs (Schmidhuber, 1993b) (in which a recurrent hidden state used as both keys and values) to a form of attention. More broadly speaking, the mathematical derivation relating the vanilla FWP to unnormalized attention is the
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: A few variations of Fast Weight Programmers with the corresponding update rules and underlying local (minimizod) objective functions. W., W,- and W are matrices; V, and k are vectors, a, is a vector with elements in (0, 1); 1 denotes a vector whose elements are all one; À and X are scalars in (0, 1); The is a non-negative scalar serving
learning rate in the update rule (the update rules that do not involve n use & learning rate of 1). @ and @ denote outer product and element-wise/ Hisnnennbwiie/HHaammarr Hadamard product, respectively. Derivations can be found in Appendix A.
Model
State Update Rule
Local Loss (W)
Vanilla FWP
W; Wi + 60 (b(c))
-u, Wo(k))
Use Classic Rule
DeltaNet (Schlag al., 2021a)
Wi- Wi 1 ( W: 10(k)) @ d(k)
+lle: Wo(k))
OjaNet (Irie et al., 2022b)
W. W:-1 + 1.0. 8 (Ø(ks) W, 18))
- (o(kk)) + W
Introduce Docaying
RetNet Sun et al. (2023)
Wi = AWI-1 + @ ((k)
-u ('(k))) + - Wi]
Mamba2 (Dao and Gu, 2024)
W. = A.W,-1 + @ ((k))
-u Wo(kk) +
Gazed RFA (Pong et al., 2021)
W. A-W.-1 + (1- A @ ((+e)
-(1 As)or Wotkk) +
mLLTM in XLSTM (Beck et al., 2024)
W. A,W,1
((k,)
-ne, Wo(k)) + الب WI
GLA (Yang et al., 2024a)
W, = (a, & 1 Wi-1 vi @ (kk)
-u," Wo(k)) + 311((v] a.
@ WII
Combine Mathods
Gated DeltaNet (Yaug et al., 2025)
Wi = A.W. 1 m W Wi-1o(k.)) @ ((k))
+ll - Wo(k)) + WII
same as the classic derivation in ML that derives the duality between the perception and its dual form, kernel machines by Aizerman et al. (1964) (1964); based on this parallel, the vanilla FWP computation is the primal form, and (unnormalized) attention is its dual form (Irie et al., 2022a).
A practical implication of this equivalence that FWPs are typically used as a drop-in replacement to the self-at tention operation in the transformer architectures, while preserving other transformer components, including two-layer feedforward blocks, layer normalization, residual connections, as well as the use of multiple heads (Vaswani et al., 2017).
3.4 Going beyond the vanilla FWP: exploring fast weight update rules
A core characteristics of FWPs the use of an update rule that has form of learning rule in the forward dynamics of the system to train a subnetwork on the fly naturally motivates us to explore and improve the update rule used in Eq. 13.
For example, one idea is to replace the purely Hebbian-like learning rule of Eq. 13 by the crror-correcting delta-rule (Widrow and Hoff, 1960). This yields the following model, called "DeltaNet" (Schlag el al., 2021a):
q: = W we h; = w xt or = WV xt Be = uT x,
(23)
W. Wi- (B Wi-io(k() @ o(k:)
(24)
y: We)(p)
(14)
where in addition to the query/key /vlue generated by the slow net in Eq. 23, an extra trainable parameter
vector 2 E Rh is introduced to generate a scalar variable B, E R, which will serve as dynamic learning rate (in the multi-head case, different learning rates are generated for each head); in Eq. 24, ve is typically set to 2 times the sigmoid function (the factor 2 is crucial to introduce negative eigenvalues in the state transition matrix enabling improved expressivity (Grazzi et al., 2025); we discuss in Sec. 3.6).
Eq. 24 corresponds to a rank-one update of the fast weight matrix, from Wi- to W., through the delta learning rule (Widrow and Hoff, 1960), where the slow net-generated variables, V., o(k:), and (B,),, play the role of terget, input, and learning rate of the delta rule, respectively. See Box 3 for further comments on the delta rule.
From the memory system perspective (Schlag el al., 2021a), this update rule can also be interpreted as follows: instead of naively adding the new key-value association (ks, v2) to be stored in memory (as is the case for the purely additive rule of Eq. 13), we first chock the old value that is currently associated to the new key k, by querying the current memory matrix W,-1o(k;); which we remove from the memory, while adding the new value v,. The effective residual value vector to be added to the memory is their "delta", i.e., V,- wkk).
11

---

Published in Transactions on Machine Learning Research (12/2025)
In practice, DeltaNet has been shown to consistently outperform the vanilla FWP with the purely additive update rule (Sec. 3.1) across many tasks including language modeling (Schlag et al., 2021a; Trie et al., 2021; Yang et al., 2024b), reinforcement learning in game environments (Irie et al., 2021), time series classification (Iric et al., 2022b), and image generation (Iric and Schmidhuber, 2023). One natural question is whether DeltaNet still efficient, i.e., whether its training is parallelizable, and the answer is yes; Yang et al. (2024b) have derived a parallel training algorithm for DeltaNet.
More broadly, many recently proposed efficient sequence models as Gated Linear Attention (GLA) (Yang et al., 2024a), Mamba2 (Dao and Gu, 2024), RetNet (Sun et al., 2023), mLSTM in xLSTM (Beck et al., 2024), and Gated DeltaNet (Yang et al., 2025) can also be expressed as an FWP with specific state/weight update rule; the corresponding summary is shown in Table 1. This FWP view facilitates relating and comparing these can categorize that many of these models simply introduce a decay factor on the weight and differ from each other in the type of docay usod: RetNet uses a constant scalar docay, whereas Mamba2 uses a context / /dpenddennn scalar (produced as function of the inpul; similarly to the dynamic learning rate of DeltaNet in Eq. 23), while GLA dynamically produces different decay rates for each row of the fast weight matrix. Oja's rule (Oja, 1982) is also a natural extension to the naive Hebbian rule; however, OjaNet was reported to empirically underperform DeltaNet on certain sequence processing applications (Irie et al., 2022b); which may be an intuitive result as Oja's rule performs principal component analysis (Oja, 1982), while the delta rule is for error correction. Certain other rules can be naturally derived as extensions of the delta rule (Yang et al., 2025; Peng et al., 2025). Further discussions of local objectives underlying different update rules are provided in the next section and in Table 1.
which may be nebulous solely from their names. For example, we
As a side note, some of the early FWP-like models whose development predates 2020 (Schlag and Schmidhuber, 2017; Munkhdalai and Yu, 2017; Munkhdalai and Trischler, 2018; Miconi et al., 2018; 2019; Keller et al., 2018; Munkhdalai et al., 2019) (at the time when sequence model development was much less dominated by the training efficionc;; we remind that the GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020) language models were introduced in 2019 and 2020, respectively), are somewhat harder to fit in this table, as they tended to use fast weights within the LSTM architecture; but we can find the same core idea: replacing certain weight matrices in the LSTM by fast weights modified over time through update rule.
Practical considerations. To determining the to-go model for a specific task, our current recommendation is to try both DeltaNet and Gated DeltaNet variants (Table 1): while Gated DeltaNet has been reported to outperform DeltaNet on language modeling tasks, consistency of this advantage in other tasks has not been confirmed yet (e.g., for reinforcement learning in certain game environments, we found nightt/memory decay of the gated variant to hurt; unpublished work). Our general recommendation is to avoid relying solely on existing language modeling results when applying FWPs as general-purpose sequence models to other tasks.
As for practical considerations, the choice of & (applied to key and query vectors) has a direct impact on both good performance and stability (especially when the delta rule is used (Schlag et al., 2021a)). Our current recommendation is to set & to be the element-wise sigmoid lincar unit (SiLU = x C sigmoid(z)) followed by the L2 normalization as proposed by Yang et al. (2024b). Further discussion on the practical training algorithm can be found in Box
Efficient implementations for most models listed in Table are openly available on the actively maintained "[aah-llinea--atteniion repository (Yang and Zhang, 2024); using these models are typically as easy as using an LSTM in PyTorch, and could be a good starting point for any other FWP model development.
Box 1: Chunk-wise Parallel Training Algorithm for FWPs
In practice, the FWP models discussed here are trained using so-called "chunk-wise paral- lel training" algorithm, which is a hybrid approach leveraging both the recurrent and attention form of FWPs (Hua et al., 2022; Sun et al., 2023; Yang et al., 2024a). While the exact algorithın is derived for each FWP model, the main idea is to divide a training sequence into small chunks and causally
12

---

Published in Transactions on Machine Learning Research (12/2025)
a
b
Sequence Model Dynamics
Yt
Learning Algorithm
Sequence Model
Yt
Learning Algorithm/DDnamiis
Dynamics
(delayed) target y 1-1
target Vt X observation
X observation
Figure 2: An Illustration constrasting at a conventional view on sequence model with a learning algorithm, and b: a metalcarned (or in-context learning) system that embeds learning poprrthhm/ddnaannn within its sequential dynamics. In a, the sequence model only observes an input, and produces an output (black), while the learning algorithm receives the expected target and the model output, and takes care of adjusting the parameters of the sequence model to improve upon the given task (gray). In contrast, in b, the system itself observes the input and the (delayed) expected target, and self-improvement on the task, iee. learning, is part of its sequential dynamics.
process one chunk after another; the intra-chunk computation leverages the parallel computation, while the inter-chunk contributions are computed using the recurrent form.
Here we illustrate the main idea by focusing the algorithm for the vanilla FWP with é set to identity. Let S and denote positive integers. We denote all the model outputs in the n-th chunk of size as Yn E Rdomxx X9 which can be computed as:
Y, WnQn + Vn Vn OM) Wn-1 = W. + V,K,
(25)
where Qu Kn, E Rd X8 and Vn Rd denote matrices containing all the query, key, value vectors for chunk n, respectively, and W, Rduuddwy is the state of the fast weight matrix after obscrving all the inpuls from the beginning of the sequence up to chunk (exclusive), with Wo O, and M E S is the causal mask. In Eq. 25, the first and second terms of the left equation correspond to the inter- and intra-chunk computations, respectively, while the right equation is the chunk-level fast weight update.
Analogous algorithms can be rather straightforwardy derived for all the FWPs based on weight decays (see Table 1). While non trivial, an algorithm for DeltaNet can also be derived ( (Yang et al., 2024b) and scales well in practice. Actual implementations for different models can be found, e.g., in the open-source code available at the flash-linear-attenion repository (Yang and Zhang, 2024).
3.5 Local online learning, metalearning, and conception of in-context learning
The concept and structure of FWPs also offer unique insights into the idea of local online learning and metalearning, with implications for research on learning mechanisms compatible with biological constraints.
Local online lcarning. The structure of FWPs captures the fundamental idca of expressing the lcarning dynamics, i.e., the process of "training a network", within the model's sequential dynamics (Cotter and Conwell, 1990; 1991; Younger et al., 1999; 2001; Hochreiter et al., 2001b) slow net learns to "train" a fast net as a part of sequence processing. This view is further reinforced in the case of DeltaNet in which the classic delta rule- conventonnn used the "backward pass", i.c., in the process of learning the (slow) weights of an ML system- is used in the "forward pass" to perform online updates of the fast weights based
13

---

Published in Transactions on Machine Learning Research (12/2025)
on the variables (mput/target llrrreigeaannnggggg. rate) produced by the slow net on the fly; essentially performing local online training of the fast net.
This local optimization aspect becomes even more prominent by explicitly writing down the local objective function that is optimized by the corresponding update rule. For example, the classic delta rule (Eq. 24) corresponds to the derivative (w.r.t. the fast net weights Wt) of the squared loss ||m, - Wro(k))lB wetween the "target" Df and the output Wro(k;) that the fast net would produce if d(k,) were fed to its input (which is consistent with the idea of binding o(k;) to vs by storing the corresponding key/val vea/value pair in the memory matrix Wt); see Box 3 for further details. More generally, the update rules used in typical FWP models have a corresponding local objective function, as summarized in the last column of Table 1. For example, using a state/weight decay in the update rule corresponds to introducing the L2 regularization on the fast weight matrix [|? in the local objective function (disregarding constant factors), where the regularization strengths are the weight decay factors.
As a side note, such an idea of an optimized model (the slow net itself is objective function, e.g., by gradient descent) that internally optimizes certain objective function is often called "mnesa-ptimizztioon" (Hubinger et al., 2019) and the corresponding hidden objective is called "mesa- objective". A natural extension of such a view on FWPs has given rise to another class of FWP models, in which, instead of defining a single-step update rule, a local objective function is directly defined, and the model output is produced by finding the corresponding optimum by using an explicit optimizer. For further details, we refer to concrete examples of this model family, such as MesaNet (von Oswald et al., 2025; 2023b) and Titan (Behrouz et al., 2024; 2025a) (see also Behrouz et al. (2025b)), as well as the related concepts of "test-time waining/regressiin (Sun et al., 2025; Wang et al., 2025).
for an external
Metalearning. The FWP concept of training a model to train another model (or itself) is also the essence of metalearning in ML (Schmidhuber, 1987; Chalmers, 1990; Bengio et al., 1991; Hochreiter et al., 2001b). While any general-purpose sequence models (including any models discussed in this Primer) can be potentially trained to become an online learner through metalearning (as we explain below), the structure of FWPs provides an in- tuitive conception: the slow net implements a learning algorithm for the fast net. Remarkably, von Oswald et al. (2023a) have effectively derived a specific slow weight configuration for a vanilla FWP to implement the gradient descent learning algorithm for regression problems in its forward dynamics (see Box 2 for further details).
One crucial ingredient for metalearning we have not discussed yet is the "error feedback". For any sequence model to become an online learner (capable of effectively learning new tasks through obsorvations), error feedback needs to be provided to the model, in addition to the inpul observation. There are two common ways to do so. One way (which we call the flelayed-feedbbcck setting following Irie et al. (2022c)) is to food the ground truth target from the previous time step as an additional input to the model (i.c., with a one time-step delay); in this case, the model continually receives an input o, and a delayed feedback Út-1 and predicts y/ at every time step (Hochreiter et al., 2001b; Santoro et al., 2016). Alternatively, in the heapchrrooou--eedbaaee
setting (Mishra et al., 2018), we feed both an input o; and the corresponding target ye to the model at every time step as demonstrations of the task; and for an imput on which want the model to make prediction, no larget is provided; instead, we replace it with special values that indicate it is not a demonstration and that a prediction is being requested.
In both cases, such formulations turn the problem of learning itself into a sequence learning problem; by (meta-)rraining a sequence model on many such example sequences, each representing a different task (i.e., a different lcarning experience), we can obtain an online learner capable of learning a new task by observing some lask demonstrations (i.e., pairs of an observation and the expected target from the task).
While such an online learning capabilities is often called in-conteat learning (Brown et al., 2020; Garg et al., 2022; Raventós et al., 2023) today, and it is often (mislcadingly) described as a somewhat magical capability of transformer-based large language models (LLMs); by using some metalearning process like the one described above, we can meta-train any sequence model to become an online learner involving modalities beyond languages. For example, the seminal work by Hochreiter et al. (2001b) trained an LSTM to perform in-context prnessionn-dmonssttre
the "fighd-weight lcarning" concept advocated by Cotter, Conwel, and Younger (Cotter and Conwell, 1990; 1991; Younger et al., 1999) in the early 1990s, while Santoro et al. (2016) and
14

---

Published in Transactions on Machine Learning Research (12/2025)
Mishra et al. (2018) performed in-context image classification all predating the term in-context learning. There are numerous such examples across tasks, modalities, and model architectures (Bosc, 2015; Santoro et al., 2016; Duan et al., 2016; Wang et al., 2017; Munkhdalai and Yu, 2017; Munkhdalai and Trischler, 2018; Mishra et al., 2018; Micomi et al., 2018; 2019; Munkhdalai et al., 2019; Sandler et al., 2021; Kirsch and Schmidhuber, 2021; Huisman et al., 2023), and they are not specific to the transformer architecture or the language modality.
From the metalearning perspective, it is not surprising that LLMs capable of in-context learning, given that the task of rego-segressive next-token predicton--uddrrliingg underlying language modeling precisely follows the form roquired by metalcarning: prediction with error feedback (the delayod-foedback version above); and the internet-scale text could provide data necessary for such meta-training (Irie and Lake, 2025).
Box Slow Weight Configuration Implementing Gradient Descent
Here we review how Oswald et al. (2023a) constructed slow weight configuration that implements a gradient descent learning algorithm in the forward pass of the vanilla FWP (Sec. 3.1) for linear regression problems.
We consider a regression task with imput and output dimensions dx and dy, respectively, and the corresponding data set f(z)) for 2, E Rd and f(z)) E IR for from 1 to T with an unknown function f.
Let's first describe what the gradient descent algorithm would do for linear regression. If we had linear model with weight matrix Wo E Rd, xdx, and performed one step of gradient descent
the loss 2 D (z:) Wo the resulting weight matrix would be Wo AWT with AWT
(E)()(=))-- - Woz,) @ (here we use a learning rate of but the construction can be easily extended to the case with an arbitrary learning rate). Given new input x" E Rd, prediction of the linear model with the updated weight matrix would be (Wn+ 1hh-AP AWT)z* which can be further expressed as (Wo+ AWr)** x AWTx* for small initialization Wo.
The goal here is to construct a weight configuration of Wº, W WV the vanilla FWP that can reproduce the algorithm above, by simply following the FWP equations of Sec. 3.1. This construction uses layer single-head head vanilla FWP model with 6 function set to identity. The feedback scheme is synchronous, that is, we feed to the model sequence of demonstration vectors 2; = [zz, f(z)) E Rd each is a concalenation of an observation z₁ E Rd and the ground truth target / ((z)) E Iy
The proposed weight configuration is the following:
W -W
( Id.
(Oa xd. O4, xd,
Od, 0d,
0d,xd, WV = Wo Ia
0d,
(26)
where Id, xd, and Ia, E Rddxxd are identity matrices, O, are block matrices filled with zeros with the corresponding dimensions *, and Wo E Rdy In terms of the notation of Sec. 3.1, we have du = dout = dkey = dx + dy. For an input 2. = f(x)) E +d, this yields:
q. Kg =
Zt
Od 8: = Woze - f(ze)
(27)
02
With these key vvauue vectors, we can easily derive that, at time step t, the corresponding fast weight state is:
W. = Wi-1 1k W ((Wozz 0d,
Odexdy
(28)
f(z()) & me
Odyxdy
Od xd.
Od, xd.
Od, xd, AAW,
Odgxdy
(29)
Σ7=1 Wooz -((+)) 2 zr Od, xd,
Od,xdy
15

---

Published in Transactions on Machine Learning Research (12/2025)
where we can already recognize that AW, (in the left bottom block) is the gradients of the squared loss using imputs. The FWP output is:
x x1 y: = W.q. Wiz
(30)
The last dy elements of output Yt. which is - W is the same (up to a negalive sign) to the linear model predictor trained by gradient descent as described above; a simple linear readout layer (with weight vector wour [Oa, x1, -11,x11) can casily apply the negative sign and extract this part of ye- Overall, at every step t, this FWP implements gradient descent using the corresponding data points up to t.
After having processed T demonstration inputs, to make prediction on new x² without its target f(z"); ); we can set the corresponding "target" part of the model input as x² = , Wo=*| to obtain wout y 2W, corresponding to the gradient descent-trained linear predictor.
compabb-compatibbee learning. The core idea of parameterizing local learning part of the model's sequen- tial dynamics (see Figure 2)-and metalearning the corresponding in-context learning algorithm- prompts us to rethink general research on biologically plausible learning in ANNs (Schmidhuber, 1989; 1990b; Mazzoni et al., 1991; O'Reilly, 1996; Bengio et al., 2015; Lillicrap et al., 2016; Pozzi et al., 2018; Najarro and Risi, 2020; Boopathy and Fiete, 2022; Hinton, 2022), which has been striving to address the longstanding critique of back- propagation in deep ANNs as being incompatible with biology (Crick, 1989; Zipser and Rumelhart, 1990) (e.g., common critique is that backpropagation uses the transpose of the same weight matrix used in the forward pass in the backward pass, raising the "weight transport problem"). While meta-training still uses the biologi- cally implausible algorithm (e.g., BPTT), once the model is trained, it becomes capable of in-context learning which is a local learning algorithm for which there is no obvious incompatibility with biology at high level.
In particular, with FWPs in mind, one potentially fruitful view is, instead of drawing a parallel between the BPTT-based learning of slow weights with learning in the brain, we could introduce another time scale, and draw parallel between local learning of fast weights with learning in the brain, and learning of slow weights could be analogous evolution that has shaped our molecular mechanism and ability to learn (in fact, in lieu of BPTT, evolution strategy algorithms have also been used to train the slow weights of FWPs (Gomez and Schmidhuber, 2005); see also Chalmers (1990)).
As a side note, learning in ANNs has also received numerous critiques from cognitive scientists, who pointed out shortcomings such as the inability to learn from a few examples, to learn compositionally (Fodor and Pylyshyn, 1988), or to learn continually (McCloskey and Cohen, 1989). A recent series of work (Santoro et al., 2016; Lake and Baroni, 2023; Irie al., 2025a) has addressed these classic challenges altogether through a common framework of metalearning and in-context learning (Trie and Lake, 2025).
3.6 Expressivity of the FWP models
In addition to the computational complexity, expressivity is a critical property to compare and categorize various types of sequence models in ML. Expressivity is about determining what types of computation the model can perform, which fundamental question in computer science, and is arguably also important in modeling cognitive abilities in psychology. One common misconception is that, given all the universal approximation and Turing completeness results available for these ANNs/RNNs (Siegelmann and Sontag, 1991; Pérez et al., 2019; Pérez et al., 2021; Orvicto et al., 2024), they are all equally powerful. This is not the case for practical models with finite resources; different sequence models differ in their "practical computational ability" (Weiss et al., 2018) and in the type of tasks they can solve.
Tools to evaluate expressivity. Expressivity is often evaluated through formal language recognition tasks (Giles et al., 1989; Pollack, 1988; Gers and Schmidhuber, 2001; Schmidhuber et al., 2001; Weiss et al., 2018; Hahn, 2020; Merrill et al., 2020; Bhattamishra al., 2020: Delétang et al., 2023; Irie al., 2023; Merrill and Sabharwal, 2023; Strobl et al., 2024; Merrill et al., 2024; Beck et al., 2024). Formal languages are convenient.
16

---

Published in Transactions on Machine Learning Research (12/2025)
tools here because they provide us with diverse set of tasks that require different types of sequence and memory processing abilities derived from the Chomsky hierarchy (Chomsky, 1956; Hopcroft and Ullman, 1969). For example, parity (given a sequence of Os and 1s, the task of determining whether the number of Is is odd) or modular arithmetic (addition and multiplication of integers modulo some integer) tasks can be represented as "regular languages" which require state-tracking ability to solve the task (Grazzi et al., 2025; Merrill et al., 2024; Sarrof et al., 2024); while certain "context-froe" or " "context-sensitivee" grammars, such as the tasks of recognizing that a given sequence follows pattern of a"b" or b"b"c", can evaluate models' ability to count (Bhattamishra et al., 2020).
However, one important remark here is that the Chomsky hicrarchy, which classifies theorctical models of compulation, does not strietly capture the expressivity hierarchy of practical neural nelworks the inability to solve certain regular language tasks does not imply a systematic failure on context-free grammar tasks, even though they are "stronger" than the regular languages in the Chomsky hierarchy. For example, this is exactly the case for the transformer: cither completely fails or struggles with certain regular languages, but their performance is excellent for both context-free and context-sensitive counting tasks (Bhattamishra el. al., 2020).
Expressivity of FWPs. The FWW models in Table 1 differ in their expressive power. For this comparison, let's assume dout = dma = d, and regroup terms in the update rule equations that involve the current state Wr-1, to obtain a canonical SSM-like form:
W = B.
(31)
for arbitrary matrices A,, B., C. E Rdxd that have no dependency on variables from the previous time step --1 1, where A, and B, are the "state transition matrices". For example, by denoting the identity matrix I E Raxd, the update rule of DeltaNet (see Eq. 24) can be rewritlen as:
W = W + (B, - wk-1 @ (k:)
(24)
= Wi- (B))) @ (k:)) + (B)) @ (kk))
(32)
that is, A, = (I- @ ((k)) for DeltaNet, which, as pointed out by Yang et al. (2024b), is a generalized Householder matrix (in Box 3, we provides an overview of the useful facts about the delta rule discussed in this work); and B, I.
More generally, the canonical form of Eq. 31 can tell lot about the expressive power of the model by looking at the form that A, and B, take, because it is the state transition matrices which dictate the type of state transition the model can perform. The expressivity of models is limited when both A, and B, are reduced to an identity matrix (as is the case for the vanilla FWP; A, = B, = I) or a diagonal matrix whether all the diagonal values are the same (c.g., in RetNet, Mamba2, and mLSTM, A₁ = XI or A, = X, or different as in GLA (A, = a, & 1 = Da)g(a:)), akin to dlement-wise recurrence (Sec. 2.2); see Table 1. For example, these diagonal state-transition- modular arithmetic, while DeltaNet models can handle such state-tracking tasks (Grazzi et al., 2025).
models fail at recognizing certain regular languages such as parity and
While earlier work on expressivity analysis of FWPs has been mostly empirical (Irie et al., 2021; 2023), there has been increasingly more work that aims to analyze and improve FWPs through the theoretical angle (see, e.g., Merrill et al. (2024); Sarrof et al. (2024); Muca Cirone et al. (2024); Movahedi et al. (2025)), in particular, Siems et al. (2025) introduced "DeltaProduct" which extends DeltaNet by using more than one application of the delta rule per time step, yielding A, with a product of Householder matrices, achieving an improved expressivity in the resulting model.
In machine learning, the current challenge is to improve the expressivity and the general model performance, while maintaining the efficiency of sequence models (Yau et al., 2025). As a counterexample, introducing extra recurrence (by feeding back the output of fast net to the input of the slow net in the next time step (Irie el al., 2021)) or self-reference (by merging the slow and fast nets into a single network that modifies itself (Schmidhuber, 1992a; 1993a; Irie et al., 2022c)) can further improve the expressivity of FWPs, but it makes the models' training inefficient. However, more exploratory model developments by ignoring the requirement for efficient parallel training- which is desired solely from the machine learning standpoint, in principle- may also be fruitful for computational modeling in neuroscience.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Complementarity of memory systems in machine learning. Reproduced from Irie et al. (2025b)
Property
Transformer
Fast weight programmer
Complexity
quadratic
linear
Context length
bounded
unbounded
Retrieval precision
high
low
Expressivity
high (with certain update rules)
3.7 Complementarity of memory systems
While recent developments of FWPs have produced efficient sequence models that are both more efficient and more expressive than the standard transformer and competitive in practice on average across many language tasks (Yang et al., 2025; Siems et al., 2025; von Oswald et al., 2025) the transformer still outperforms FWPs on precise retricval tasks by a large margin (Iric al., 2025b), suggesting the raison d'être softmax attention. remains an open question whether FWP models
Their overall complementarity is summarized in Table standard transformers. For now, an engincering be further improved to match the retrieval quality of solution to achieve the best of both worlds is to combine the two in hybrid architecture (Beck al., 2024; Yang et al., 2025), which is reminiscent of the classic "complementary learning systems" (McClelland et al., 1995; O' O'Reilly and Norman, 2002) in which two complementary systems are allocated to collectively achieve incompatible goals that are unattainable by each individual system, through division of labor.
Box 3: Key facts and intuitions about the delta rule
Here we briefly summarize three useful facts about the delta rule.
1. Gradient descent on the squared error regression loss. The delta rule equation can be derived from the following regression problem. Consider function Rd - Rdbut parameterized by matrix W E RRdo that transforms an arbitrary input E Rd to output f(Wx) E Rdout where is an arbitrary differentiable function RRdout + Rdout, Given a data point with input æ E gdo and Larget ÿ E Idouu to which we want to fit this function, we can minimize the squared error E(W) 2 9 f(Wx)ll2 between the model output and the target, using gradient descent.
DE = - (['((x)) ④ (i ((Wx)) & æ, which yields the following
The corresponding gradient is: aW
update term to be added to W when one step of gradient descent is applied with a learning rate WF aE
AW - ow = mf @ (g f(Wx))) & æ. In the case where the model is a simple linear layer, i.e, when f is identity, this term becomes: AW = 77 Wx) & æ, which corresponds to the delta rule used in DeltaNet, where the key o(k;) and value v, which play the role of input and target, respectively, and the learning rate T/s, are dynamically generated.
2. Improved update rule for a "key-value associative memory" system. The delta rule can be seen as improved update rule for "key-value associative memory" system. Recall that linear layer with an outer-product weight updale rule can implement such memory system corresponding to Kohonen's correlation matrix memories (Kohonen, 1972) (in fact, Kohonen used the "key-data" terminology instead of "key-value"). Generally, the defining components of a memory architecture the case of a linear layer system, its
are its storage and the associated reading/wrrtting primitives. weight matrix serves as the storage. The reading operation is the multiplication between query input g, and the memory matrix W: y = Wq. A basic writing operation based on outer-product which adds a key-value association (k, to the memory is: W. W. consider 3-dimensional keys and 2-dimensional values. Given empty memory Wo 02x3 E g2x3 key-value association, with arbitrary value vector E R2, and the key & = [0. 1, E R33 (which is a one-hot vector), can be stored to the memory by adding the corresponding outer-product to Wn: W, = W, k 02x11 02x1]. We can retrieve the corresponding value by using the
k. As an illustration, we
18

---

Published in Transactions on Machine Learning Research (12/2025)
corresponding key as the query q [0, 1, 1, for memory reading: Wiq = v. However, when the same association (k, u) is presented again to the system, the updated memory state becomes W2 [0: 20; 0), which breaks the (k,, o)-association (the wrongly stored association is (k, 2v)), as this naive additive rule does not check the current memory content. In contrast, the delta rule preserves the memory state in that case, as it only writes the "delta", i.e., the difference between the target value to be stored and the current value associated to the key.
Improved transition matrix in linear RNNs. Finally, as discussed in Sec. 3.6, the effect of the delta rule can also be understood in terms of transition matrix in linear RNN. The state transition matrix of the FWP with the purcly additive rule is the identity matrix. In contrast, the delta rule introduces state update term that is dependent the current memory state; this yields more expressive, non-diagonal transition matrix (Eq. 32).
Glossary (Neuroscience)
Ion channel: protein that forms pore in the cell membrane, allowing specific ions (e.g., sodium Na* calcium Ca to pass through. lon channels are crucial for generating and transmitling electrical signals neurons.
acid receptor) membrane protein (ionotropic glutamate receptor) that forms glutamate-gated ion channel whose permeability depends on subunit composition: receptors that include the GluA2 subunit are impermeable to Ca²+, while those lacking GluA2 allow Ca entry. It opens rapidly and mediates fast excitatory transmission, contributing to synaptic plasticity.
AMPA receptor:
NMDA receptor: (N-mmethy----ssparate receptor) A membrane protein (ionotropic glutamate receptor) that forms a glutamate- and voltage-gated ion channel permeable to Nat, Kt, and Ca2+, requires depolarization to relieve Mg block before opening: while Na and potassium (K+) also pass, the Ca influx is the principal signal that initiates synaptic plasticity.
Glutamate: A small amino acid neurotransmütler that binds to receptors such as AMPA and NMDA, activating ion channels that mediate excitatory signaling and synaptic plasticity.
Phosphorylation: An addition of phosphate group protein other molecule, typically by enzyme called a kinase. Phosphorylation can alter a protein's activity, interactions, or localization.
Post-translational modification: A chemical change to a protein after synthesized, such as plosphorylation. These modifications regulate protein function, stability, and signaling.
4
Neurobiology
Here we discuss how FWPs might be implemented in the brain (Sec. 4.1). This is necessarily speculative, though we will support our assertions with available evidence. In Sec. 4.2, we more broadly highlight. properties of FWPs that are relevant as a synaptic plasticity model for neuroscience. Our hope is that these idcas will inspire new directions in the study of synaptic plasticity.
4.1 A neurobiological implementation of fast weight programming
To simplify the exposition, we will drop the time index and focus on the special case where the keys and queries the same, and the values have a direct. relation to the queries (as specify below); thus, we have k. We consider a "postsynaptic" population of neurons that receive "presynaptic" input and generate firing rates y. The presynaptic neurons receive input from sensory representation which we leave implicit here. As in many models of neural activity, we will assume that the postsynaptic firing rate can be approximated by a linear combination of presynaptic inputs, Dj = Σ, W5:03-
The synaptic strengths evolve according to a generalized Hebbian learning rule, where strength increases due the coincidence of presynaptic firing and a postsynaptic activity trace:
AW vyki,
(33)
19

---

Published in Transactions on Machine Learning Research (12/2025)
where A;- = 9, is the firing rate of presynaptic neuron i, and by is an "activity trace" encoded by postsynaptic neuron j, which we assign to the accumulated postsynaptic calcium level. We model the calcium trace as a linear combination of the presynaptic inputs, ej = E; Usiq: (though in reality the relationship is nonlinear due to the voltage-dependence of calcium conductance; c.g., we can possibly introduce some non-lincar activation function on v). This is equivalent to the FWP setup with W = UWQ and W = =KK
We hypothesize that the synaptic strength matrix W corresponds to the density/ (ollty/connt /conductance of AMPA receptors, while the matrix U governing the calcium response corresponds to the density haftty/connt /conductance of .MDA receptors. This distinction is motivated by several facts. First, firing rates are primarily governed by sodium channels linked to AMPA receptors, whereas intracellular calcium levels are primarily governed by calcium channels linked to NMDA receptors. Second, AMPA receptor plasticity can be induced quickly (on the order of seconds (Gustafsson et al., 1989)) by Hebbian stimulation protocols- fast enough to contribute to performance (at least in principle) on working memory tasks (Erickson et al., 2010; Lansner et al., 2023). In contrast, induction of NMDA receptor plasticity typically slower (Hunt and Castillo, 2012). These two forms of plasticity can also be induced independently. Third, AMPA receplor plasticity critically depends on calcium influx, which activates a cascade of protein synthesis and nost-translational modification. This is consistent with the dependence of fast weight modification on U/. the putative calcium trace. In particular, the fastest change induced by Hebbian stimulation is the phosphorylation of AMPA receptors by aalcium-activatee kinases like PKA and CaMKIT (Tee et al., 2000; Soderling and Derkach, 2000)-
The generalizations of the vanilla FWP architecture (discussed in Sec. 3.4) suggest further nuances to this picture. For example, DeltaNet uses vn -, Wyrq in place of v, in the Hebbian update. A possible biological interpretation is that recent neural activity sets a plasticity threshold on postsynaptic calcium, reversing the direction of plasticity when calcium levels are below the threshold. Indeed, this is a venerable idea in models of synaptic plasticity (Shouval et al., 2002; Graupner and Brunel, 2012).
Another generalization discussed in Sec. 3.4 (sce also Table 1) is the use of various forms of decay on the weights and/or states. For example, RetNet assumnes scalar decay of all fast weights. This is broadly consistent with the observation that changes to synaptic strength continuously decay due to a variety of processes (e.g., molecular turnover, diffusion of synaptic components, stochastic is also controlled by homeostatic processes that seek to keep neural activity near set point (Turrigiano, 2008). Mamba2 assumess that. decay is input-dependent, which is broadly consistent with the role of activity- dependent mechanisms in determining the decay rate of synaptic plasticity (Abrahan and Williams, 2003). In particular, protein synthesis triggered by calcium influx plays a central role in the conversion of short-term synaptic changes (e.g., AMPA receptor phosphorylation) to long-term changes (trafficking of new AMPA receptors to the postsynaptic membrane). GLA further extends this input-dependent decay by modeling separate decay rates for each postsynaptic neuron.
activity). Decay
The idea that synaptic plasticity plays out at multiple timescales through several different mechanisms has become widely accepted (Citri and Malenka, 2008), including some mechanisms (such as short-term depression and facilitation) that are even faster than the fast Hebbian plasticity described carlier (Zador and Dobrunz, 1997). Our goal in this section was to link a subset of these mechanisms to the computational ideas underlying FWP systems. This leaves fertile ground for future exploration of other potential links, including transformer-like computation in the brain (Ellwood, 2024; Whittington et al., 2022; 2025; Kozachkov et al., 2023; Gershman et al., 2025).
4.2 Other prospects and considerations for neuroscience
More broadly, the base FWP equation presented here (Eqs. 13-14) may be extended (or restricted) to accommodate certain neurobiological aspects that are not supported by the conventional computational models in neuroscience. In particular, the following properties are prominent.
First, FWPs could implement a quite broad class of synaptic modifications (Magee and Grienberger, 2020), including both Hebbian and non-Hebbian ones, by conceiving extensions in which the key, value, and query representations come from independent sources (i.c., they do not all have to be a function of the shared input 2x) or even from different time steps. In particular, FWPs can naturally support non-Hebbie learning, where the synaptic weight modification does not depend on the postsynaptic firings. For example, behavioral
20

---

Published in Transactions on Machine Learning Research (12/2025)
timescale synaptic plasticity (BTSP; Bittner al. (2017); Wu and Maass (2025)) in the hippocampus of mammals is well known to be non-Hebbian (i.e., it does not depend on the input-output correlation). BTSP involves different sub-regions of the hippocampus, CA1 and CA3, and a part of entorhinal cortex called EC3. Their functional relationship can be paramcterized as an FWP, in which the fast network maps imput CA3 activities (query 4t) to output CA1 activations (output yr), whose synaptic weights are modulated by CA3 (key ke) and gated by EC3 (value æt) not- that an outer product, like other products, can implement a gate. The FWP framework also supports the cases where either both the slow or fast networks in such a system employ recurrent connectivity (Irie et al., 2021). In contrast, implement Hebbian learning, FWPs may use variables from previous time steps as keys and values (as is done in certain recurrent FWPs (Schmidhuber, 1993b)). Overall, the FWP framework provides a unified formalism for modeling synaptic plasticity across types and timescales, which can facilitate the development of computational models in neuroscienlific studies (cf., e.g., Aitken and Mihalas (2023)).
Second, unlike traditional auto-associative memory models (Anderson, 1970; Amari, 1972; Nakano, 1972) in neuroscience which focus on retrieval of clean patterns based on partially corrupted patterns (Amari, 1972; Hopfield, 1982) (see also Kanerva (1988); Millidge et al. (2022)), FWPs implement flexible letero-associative memory, which is functionally more general (Kohonen, 1972; Steinbuch, 1961); as long as the query-key matching function is discriminative enough, it allows to store arbitrary key-value associations, regardless of whether the value is a denoised version of the key (for auto-association) or any other arbitrary patterns.
Finally, the current form of FWPs designed for machine learning purposes could be extended to include additional properties that are currently missing from a neurobiological perspective, such as stochasticity or the time window for plasticity- an important characteristic that distinguishes different types of synaptic plasticity. We hope this Primer provides a solid foundation that inspires future exploration of such potential extensions.
5 Conclusion
The main goal of this Primer was to introduce the concept of fast weight programmers (FWPs) a special class of recurrent neural networks (RNNs) with wwo-dimensional hidden states at the nexus of machine learning and computational neuroscience.
We have highlighted unique properties of FWPs that are relevant from various perspectives in these fields. We have argued that the use of dynamically changing synaptic weights as form of short-term memory offers a compelling abstract computational model for synaptic plasticity, capturing timescales that traditional R.NNs with static weights cannot. In machine learning, such sequential dynamics have been playing a central role in developing modern sequence models, as they allow for both sequence-level training, and therefore, scalability-and more expressive computations than those supported by the now popular transformer. Furthermore, the ability of FWPs to intuitively express local learning that is, learning that only involves locally available variables within their own sequential dynamics through weight/ reight/state /state update rules provides a novel perspective and a promising framework for learning mechanisms compatible with biological constraints.
maallllism-rruchi for efficient.
Finally, we have also explored a neurobiological implementation of FWP--ike computations in the brain, and broadly discussed the FWP concept as a general and promising framework that supports various types of synaptic plasticity rules that are known in neuroscience, including both Hebbian and non-Hebbian rules. While highly speculative and preliminary, we hope this work opens new avenues for modeling synaptic modulation and for discussing their roles in learning and memory in the brain.
Acknowledgments
The authors are grateful for support from the Kempner Institute for the Study of Natural and Artificial Intelligence, a Polymath Award from Schmidt Sciences, and the Department of Defense MURI program under ARO grant V991NNF-2311-0277 Kazuki Irie thanks Imanol Schlag and Jürgen Schmidhuber for introducing him to the world of fast weights, while at the Swiss AT lab IDSTA.
21

---

Published in Transactions on Machine Learning Research (12/2025)
References
Wickliffe C Abraham and Joanna M Williams. Properties and mechanisms of LTP maintenance. The Neuroscienlist, 9:463 474, 2003.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. Preprint XXiu:2303.08774 2023.
Kyle Aitken and Stefan Mihalas. Neural population dynamics of computing with synaptic modulations. Blife, 12, 2023.
Mark A. Aizerman, Emmanuil M. Braverman, and Lev I. Rozonoer. Thcoretical foundations of potential function method in pattern recognition. Automation and Remote Control, 25(6):917-99) -936, 1964.
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proc. Conference Artificial Intelligence (AAAT), pages 3159- 3166, Honolulu, HI, USA, January 2019.
Shun-Ichi Amari. Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions on computers, 100(11):1197-1266 1972.
James A Anderson. Two models for memory organization using interacting traces. Mathemalical Biosciences, 8:137-160, 1970.
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin lonescu. Using fast weights to attend to the recent past. In Proc. Advances in Newral Information Processing Systems (NIPS), pages Barcelona, Spain, December 2016a.
Jimmy Ba, Jamie Ryan Kiros, and GeofTrey E Hinton. Layer normalization. Preprint arXiv:1607.06450. 2016b.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In Int. Conf. on Learning Representations (ICLR), New Orleans, LA, USA, May 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA, May 2015.
David Balduzii and Muhammad Ghifary. Strongly-typed recurrent neural networks. In Proc. Int. Conf- Machine Learning (ICML), New York City, NY, USA, June 2016.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short-term memory. In Proc. Advonces in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. Preprint arXiv:2501.00663 2024.
Ali Behrouz, Zeman Li, Pranceth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Mcisam Razaviyayn, and Vahab Mirrokni. Atlas: Learning to optimally memorize the context at test time. Preprint Xiv:22505.22 23795, 2025a.
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. Nested learning: The illusion of deep learning architectures. In Proc. Advances in Neural Information Processing Systems (NeurIPS), San Diego, CA, USA, December 2025b.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule, In International Joint Conference Newral Networks (TJCNN), Seatle, WA, USA, 1991.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. JEEE Transactions on Neural Nelworks, 5(2):157-166, 1994.
Yoshua Bengio, Dong-Hyur Lee, Jorg Bornschein, Thomas Mesnard, and Zhouhan Lin. Towards biologically plausible deep learning. Preprint Xrx: 04156, 2015.
Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 7096 7116, Virtual only, November 2020.
Katie C Bittner, Aaron D Milstein, Christine Grienberger, Sandro Romani, and Jeffrey C Magee. Behavioral time scale synaptic plasticity underlies cal place fields. Science, 357(6355):1033- 1008,, 2017.
Guy E Blelloch. Prefix sums and their applications. Technical report, School of Computer Science, Carnegie Mellon University Pittsburgh, PA, USA, 1990.
Akhilan Boopathy and Ila Ficte. How to train your wide neural network without backprop: An input-weight alignment perspective. In Proc. Int. Conf. Machine Learning (ICML), Baltimore, MA, USA, July 2022.
Tom Bosc. Learning to learn neural networks. In NIPS Workshop on Reasoning, Attention, Memory, Montreal, Canada, December 2015.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-reeurrent neural networks. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
Dennis Bray. Protein molecules as computational elements in living cells. Nature, 376(6533) -722 1995.
Dennis Bray. Molecular networks: the top-down view. Science, 301(5641):1864 1865, 2003.
Dennis Bray. Wetware: a computer in every living cell. Yale University Press, 2009.
Tom B Brown et al. Language models are few-shot learners. In Proc. Advances in Neural Informalion Processing Systems (NeurIPS), Virtual only, December 2020.
David J Chalmers. The evolution of learning: An experiment in genetic connectionism. In Connectionist
Models Summer School, San Mateo, CA, USA, 1990.
Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 551 561, Austin, TX, USA, November 2016.
Noam Chomsky, Three models for the description of language. IRE Transactions on information theory, 2 (3):113 124, 1956.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Poter Hawkins, Jared Davis, Afroz Mohinddin, Lukasz Kaiscr, et al. Rethinking attention with performers. In Int. Conf. on Learning Representations (ICLR), Virtual only, 2021.
Ami Citri and Robert C Malenka. Synaptic plasticity: multiple forms, functions, and mechanisms. Neuropsy- chopharmacology, 33:18-41 -1, 2008.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUss). In Int. Conf. on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016.
Neil E Cotter and Peter R Conwell. Fixcd-weight networks can learn. In Proc. Int. Joint Conf. on Neural Nebworks (IJCNN), pages 553 559, San Diego, CA, USA, June 1990.
Neil E Cotter and Peter R Conwell. Learning algorithms and fixed dynamics. In Proc. Int. Joint Conf. on Neural Networks (IJCNN), pages 799 801, Seattle, WA, USA, July 1991.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Francis Crick. The recent excitement about neural networks. Nature, 337 1989.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhuldinov. -ransformer-XL: Attentive language models beyond a fixed-length context. In Proc. Association for Computational Linguistics (ACL), pages 2978 2988, Florence, Italy, July 2019.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, Preprint arXiv:2307.08691, 2023.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proc. Jnt. Conf. on Machine Learning (ICML), Vienna, Austria, July 2024.
Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Marcus Hutter, Shane Legg, and Pedro A Ortega. Neural networks and the Chomsky hierarchy. In Int. Conf. on Learning Representations (ICLR), Kigali, Rwanda, May 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. Tn Proc. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (NAACL-HLTT), pages 4171 4186, Minneapolis, MN, USA, June 2019.
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Pieter Abbeel. RL²: Fast reinforcement learning via slow reinforcement learning. Preprint arXiv:1611.0e71 12277979 2016.
Esraa Elclimy, Adam White, Michael Bowling, and Martha White. Real-time recurrent learning using trace units reinforcement learning. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
Tan Ellwood. Short-term Hebbian learning can implement transformer-like altention. PLOS Compulational Biology, 20, 2024.
Jeffrey L Elman. Structured representations and connectionist models. In Proc. Conference of Cognitive Science Society (CogSci), pages 17-25, Ann Arbor, MI, USA, August 1989.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179 211, 1990.
Martha A Erickson, Lauren A Maramara, and John Lisman. A single brief burst induces GluRl-dependent associative short-term potentiation: potential mechanism for short-term memory. Journal of Cognâtive Neuroscience, 22:2530 2540, 2010.
Jerome A Feldman. Dynamic connections in neural networks. Biological cybernetics, 46(1):27-33 39, 1982.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analysis. Cognition, 28(1-2):3 71, 1988.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetios, 36(4):193-20) 202, 1980.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context?
A case study of simple function classes. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, November 2022.
Felix A Gers and Jürgen Schmidhuber. Lstmn recurrent networks learn simple context-free and context-sensitive languages. IEEE Transactions on Neural Networks, 12(6):1333-1384 2001.
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with LSTM.
Neural computation, 12(10))2451-247 2000.
Samuel Gershman. What have we learned about artificial intelligence from studying the brain? Biological cyberneties, 118(1):1-5, 2024.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Samuel J Gershman, lla Fiete, and Kazuki Irie. Key-value memory in the brain. Neuron, 113, 2025.
C. Lee Giles, Guo-Zheng Sun, Hsing-Hen Chen, Yee- Yeo-Chun Lec, and Dong Chen. Higher order recurrent networks and grammatical inference. In Proc. Advances in Newral Information Processing Systems (NIPS), pages 680-387, Denver, CO, USA, November 1989.
Faustino Gomez and Jürgen Schmidhuber. Evolving modular fast-weight networks for control. In Proc. International Conference on Artificial Neurol Networks (ICANN), 2005.
Pedro Gonnet and Thomas Deselaers. IndyLSTMs: independently recurrent LSTMs. In Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pages 3352-3356, Barcelona, Spain, 2020.
Marco Gori, Yoshua Bengio, and Renato De Mori. BPS: learning algorithm for capturing the dynamic nature speech. In Proc. Int. Joint Conf. on Neural Networks (IJCNN), pages 117-423, Washington, DC, USA, 1989.
Michael Graupner and Nicolas Brunel. Calcium-based plasticity model explains sensitivity of synaptic changes to spike pattern, rate, and dendritic location. Proceedings of the National Academy of Sciences, 109: 2012.
Alex Graves. Generating sequences with recurrent neural networks. Preprinl Xiv: 0850, 2013.
Riccardo Grazzi, Julien Siems, Jörg KH Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear RNNs through negalive eigenvalues. In Inl. Conf. on Learning Representations (ICLR), Vancouver, Canada, April 2025.
Klaus Greff, Rupesh K Srivastava, Jan Koutnik, Bas R Steunebrink, and Jürgen Schmidluber. LSTM: A search space odyssey. IEEE Transactions on neural networks and learning systems, 28(10):2222-222) 22332, 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. Preprint arXiv:2012 2080000888 2020.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Conference on Language Modeling (COLM), 2024.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and contimuoustinng models with linear state space layers. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2021.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In Inl. Conf. on Learving Representations (ICLR), Virtual only, April 2022.
B Gustafsson, F Asztely, E Hanse, and H Wigström. Onset characteristics of long-term potentiation in the guinea-pig hippocampal cal region in vitro. European Journal of Neuroscience, 1:382-38 394, 1989.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In Inl. Conf. Learning Representations (ICLR), Toulon, France, April 2017.
Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156-171, 2020.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Maithew Botvinick. Neurosciencee-ispprred artificial intelligence. Neuron, 95(2):245 258, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. Tn Proc. European Conf. on Computer Vision (ECCV), pages 630 645, Amsterdam, Netherlands, October 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conf. on Computer Vision and Pottern Recognition (CVPR), pages 770-778, Las Vegas, NV, USA, June 2016b.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Donald O Hebb. The organization of behavior; neuropsycholocigal theory. A Wiley Book in Clinical Psychology, 62:78, 1949.
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. Preprint or Xrv: 13945, 2022.
Geoffrey Hinton and David C Plaut. Using fast weights deblur old memories. In Proc. Conf. of Cognitive Science Society, pages 177-186, Seatle, WA, USA, July 1987.
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma thesis, Technische Universität München, 9131) 1991.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-177 1780, 1997.
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001a.
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In Proc.
Inl. Conf. on Artificial Neural Networks (ICANN), volume 2130, pages 87-94, Vienna, Austria, August. 2001b.
John E Hoperoft and Jeffrey D Ullman. Formal languages and their relation to automata. Addison-Wesley, Wesley, 1969.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilitics. Proc. of the National Academy of Sciences (PNAS), 7955:22554-288 -258,, 1982.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoe V. Le. Transformer quality in linear time. In Proc. Int. Conf. Machine Learning (ICML), Baltimore, MA, USA, July 2022.
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalso, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. Preprint arXiv:1906. 01000, 2019.
Mike Huisman, Thomas M. Moerland, Aske Plaat, and Jan van Rijn. Are LSTMs good few-shot learners? Machine Learning, pages 1-28, 2023.
David L Hunt and Pablo E Castillo. Synaptic plasticity of NMDA receptors: mechanisms and functional implications. Current Opinion in Neurobiology, 22:496-508, 2012.
Kazuki Irie and Brenden M Lake. Overcoming classic challenges for artificial noural networks by providing incentives and practice. Nature Machine Intelligence, 2025.
Kazuki Trie and Jürgen Schmidhuber. Training and generating neural networks in compressed weight. space. In ICLR Neural Compression Workshop, Virtual only, May 2021.
Kazuki Irie and Jürgen Schmidhuber. Learning to control rapidly changing synaptic connections: An alternative type of memory in sequence processing artificial neural networks. NewrIPS Workshop on Memory in Artificial and Real Intelligence (MemARI), December 2022.
Kazuki Irie and Jürgen Schmidhuber. Images as weight matrices: Sequential image generation through synaptic learning rules. Tn Tnt. Conf. on Learning Representations (TCLR), Kigali, Rwanda, May 2023.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. Language modeling with deep Transformers. In Proc. Interspeech, pages 3905 3909, Graz, Austria, September 2019.
Kazuki Tric, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond lincar transformers with recurrent fast weight programmers. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2021.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In Proc. Irl. Conf. on Machine Learning (ICML), Baltimore, MD, USA, July 2022a.
Kazuki Trie, Francesco Faccio, and Jürgen Schmidhuber. Neural differential equations for learning to program neural nets through continuous learning rules. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, December 2022b.
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In Proc. Int. Conf. on Machine Learning (ICML), pages 9660-9677 Baltimore, MA, USA, July 2022c.
Kazuki Irie, Róbert Csordás, and Jürgen Schmidluber. Practical computational power of linear transformers and their recurrent and self-referential extensions. In Proc. Conf. on Empirical Methods in Notural Language Processing (EMNLP), Sentosa, Singapore, 2023.
Kazuki Irie, Anand Gopalakrishnan, and Jürgen Schmidhuber. Exploring the promise and limits real-time recurrent learning. In Int. Conf. on Learning Representations (ICLR), Vienna, Austria, May 2024.
Kazuki Irie, Róbert Csordés, and Jürgen Schmidluber. Metalearning continual learning algorithms. Transac- tion on Machine Learning Research (TMLR), 2025a.
Kazuki Irie, Morris Yau, and Samuel J Gershman. Blending complementary memory systems in hybrid quadratic-linear transformers. In Proc. Advances in Neural Information Processing Systems (NeurIPS), San Diego, CA, USA, December 2025b.
Alexey Grigorevich Ivakhnenko. Polynomial theory of complex systems. IEEE Transactions Systems, Man, and Cybernetics, pages 364-378, 1971.
Michael Jordan. Attractor dynamics and parallelism in connectionist sequential machine. In Proc. Conf. of the Cognâtive Science Society, pages 531-546, Amherst, MA, USA, August 1986.
Pentti Kanerva. Sparse distributed memory. MIT press, 1988.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs:
Fast autoregressive transformers with linear attention. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2020.
T Anderson Keller, Sharath Nittur Sridhar, and Xin Wang, Fast weight long short-term memory. Preprint arXiv.1804.06511, 2018.
Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 14122 14134, Virtual only, December 2021.
Teuvo Kohonen. Correlation matrix memories. IEEE Transactions on Compulers, (4))3333 359, 1972.
Jerzy Konorski. Conditioned reflezes and neuron organization. Cambridge University Press, 1948.
Leo Kozachkov, Ksenia V Kastanenka, and Dmitry Krotov. Building transformers from neurons and astrocytes. Proc. of the National Academy of Sciences (PNAS), 120, 2023.
Nikolaus Kriegeskorte. Deep neural networks: new framework for modeling biological vision and brain information processing. Annual review of vision science, 1(1):417-446, 2015.
Brenden M Lake and Marco Baroni. Human-like systematic generalization through a meta-learning neural network. Nature, 623(7985):115 121, 2023.
Anders Lansner, Florian Fiebig, and Pawel Herman. Fast Hebbian plasticily and working memory. Current Opinion in Newrobiology, 83:102809, 2023.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 444, 2015.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Hey-Kyoung Lee, Michacla Barbarosie, Kimihiko Kameyama, Mark F Bear, and Richard L Huganir. Regulation of distinct AMPA receptor phosphorylation sites during bidirectional synaptic plasticity. Nature, 405: 955 959, 2000.
Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. Simple recurrent units for highly parallelizable recurrence. In Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), pages 4470-4481, Brussels, Belgium, November 2018.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (IndRNN): Building longer and deeper RNN. In Proc. IEEE Conf. Computer Viston and Pattern Recognition (CVPR), pages 5457-5466, Salt Lake City, UT, USA, June 2018.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1):13276, 2016.
Thomas Limbacher and Robert Legenstein. H-Mem: Harnessing synaptic plasticity with Hebbian memory networks. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, Docember 2020.
Thomas Limbacher, Ozan Özdenizei, and Robort Legenstein. Memory-dependent computation and learning in spiking neural networks through Hebbian plasticity. IEEE Transactions on Neural Networks and Leurving Systems, 2023.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
William A Little. The existence of persistent states in the brain. Mathematical biosciences, 19(1-2):101 120, 1974.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In Inl. Conf. on Learning Representalions (ICLR), Vancouver, Canada, April 2018.
Tom Macpherson, Anne Churchland, Terry Sejnowski, James DiCarlo, Yukiyasu Kamitani, Hidehiko Takahashi, and Takatoshi Hikida. Natural and artificial intelligence: A brief introduction to the interplay between Al and neuroscience rescarch. Newral Networks, 144:603-613, 2021.
Jeffrey C Magee and Christine Grienberger. Synaptic plasticity forms and functions. Annual review of Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In Int. Conf. on Learning Representations (ICLR), Vancouver, Canada, April 2018
neuroscience, 43(1):95-117, 2020.
Pietro Mazzoni, Richard A Andersen, and Michael I Jordan. A more biologically plausible learning rule for neural networks. Proc. of the National Academy of Sciences (PNAS), 88(10):4433 4433-4437, 1991.
James L McClelland. Putting knowledge in its place: A scheme for programming parallel processing structures on the fly. Cognitive Science, 91)::113-144 146, 1985.
James L McClelland, David E Rumelhart, and PDP Research Group. Parallel Distributed Processing, Explorations in the Microstructure of Cognition, Volume 2: Psychological and Biological Models. MIT Press, 1986.
James McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and ncocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological revieu, 102(3):419, 1995.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and molivation, volume 24, pages 109-165, 1989.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathemalical biophysics, 5(4):115-133, 1943.
William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log- precision transformers. Transactions of the Association for Computational Linguistics (TACT), 11, 2023.
William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, and Fran Yahav. A formal hierarchy of RNN architectures. In Proc. Association for Computational Linguistics (ACL), pages 443 459, Virtual only, July 2020.
William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In Proc. Inl. Conf. on Machine Learning (ICML), Vienna, Austria, July 2024.
Thomas Miconi, Kenneth Stanley, and Jeff Chune. Differentiable plasticity: training plastic neural networks with backpropagation. In Proc. Inl. Conf. on Machine Learning (TCML), pages 3559-3568, Stockholm, Sweden, July 2018.
Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O. Stanley. Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. In Int. Conf. on Learning Representations (ICLR), New Orleans, LA, USA, May 2019.
Tomáš Mikolov. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012.
Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. Preprint 1805xv1805.020 2018.
Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Universal Hopfield networks: A general framework for single-shot associative memory models. In Proc. Inf. Conf. on Machine Learning (ICML), Baltimore, MA, USA, July 2022.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In Int. Conf. on Learning Representations (ICLR), Vancouver, Cananda, 2018.
Sajad Movahedi, Felix Sarnthein, Nicola Muca Cirone, and Antonio Orvicto. Fixed-point RNNs: Interpolating from diagonal to dense. In Proc. Advances in Neural Information Processing Systems (NeurIPS), San Diego, CA, USA, December 2025.
Michael C. Mozer. A focused backpropagation algorithm for temporal pattern recognition. Complex Systems, 3(4):349-335) 1989.
Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons. Theoretical foundations of deep selective state-space models. In Proc. Advances in Neural Information Processing Systems (NewrIPS), Vancouver, Canada, December 2024.
Tsendsuren Munkhdalai. Sparse meta networks for sequential adaptation and its application to adaptive language modelling. Preprint arXiv:2009001800 2020.
Tsendsuren Munkhdalai and Adam Trischler. Metalearning with Hebbian fast weights. UrXiu:1807.00017 2018.
Preprint
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proc. Int. Conf. on Machine Learning (ICML), pages 2554-2563, Sydney, Australia, August 2017.
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearned neural memory. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 13310 13321, Vancouver, Canada, December 2019.
Elias Najarro and Sebastian Risi. Meta-learning through Hebbian plasticity in random networks. In Proc. Aduances in Newral Information Processing Systems (NewrIPS), Virtual only, December 2020.
29

---

Published in Transactions on Machine Learning Research (12/2025)
Kaoru Nakano. Associatron-a model of associative memory. IEEE Transactions on Systems, Man, and Cyberneties, pages 380-388, 1972.
Erkki Oja. Simplified neuron model a principal component analyzer. Journal of mathematical biology, 15 (3):267-275 1982.
Randall C O'Reilly.
Biologically plausible error-driven learning using local activation differences: The generalized recirculation algorithm. Neural computation, 8(5):895 938, 1996.
Randall C O'Reilly and Kenneth A Norman. Hippocampal and neocortical contributions to memory: Advances in the complementary learning systems framework. Trends in cognitive sciences, 6(12):505-510, 2002.
Antonio Orvieto, Samuel Smith, Albert Gu, Amushan Fernando, Çaglar Gülçehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Proc. Int. Conf. on Machine Learning (ICML), Honolulu, HI, USA, July 2023.
Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L. Smith. Universality of linear recurrences followed by non-linear projections: Finite-width guarantees and benefits of complex eigenvalues. In Proc. Int. Conf. Machine Learning (ICML), Vienna, Austria, July 2024.
Matthew F Panichello, Donatas Jonikaitis, Yu Jin Oh, Shude Zhu, Ethan B Trepka, and Tirin Moore. Intermittent rate coding and cue-specific ensembles support working memory. Nature, pages 1 8, 2024.
Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A dccomposable attention model for natural language inference. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP)) pages 2249 2249-2255, Austin, TX, USA, November 2016.
Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, et al. RWKV-7 goose with expressive dynamic state evolution. Preprint 1rXiu:2508.4456. 2025.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong, Random feature attention. In Int. Conf. on Learning Representations (ICLR), Virtual only, 2021.
Jorge Pérez, Javier Marinkovic, and Pablo Barceló. On the Turing completeness of modern neural network archilectures. In Proc. Tnl. Conf. on Machine Learning (ICML., New Orleans, LA, USA, May 2019.
Jorge Pérez, Pablo Barceló, and Javier Marinkovic. Attention is turing complete. The Journal of Machine Learning Research (JMLR), 22(1):3463 3497, 2021.
Jordan Pollack. Recursive auto-associative memory" devising compositional distributed representations. In Proc. Meeting of the Cognitive Science Society, volume 10, 1988.
Isabella Pozzi, Sander Bohté, and Pieter Roelfsema. A biologically plausible learning rule for deep learning in the brain. Preprint arXiv:181 111111111111000 768, 2018.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazcl-Zarandi, et al. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research (JMLR), 25(7)), 2024.
Markus N Rabe and Charles Staats. Self-atlention does not need o(n²) memory. Preprint or 0122221122005688 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. [Online] https://blog www.oog.ppnniicooo. httpr//onn.i.o////eeee/gggggnoie http///teeeeamgggggooooss/////// 2019.
Hubert Ramsauer, Bernhard Schäff, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstotter, and Sepp Hochreiter. Hopfield networks is all you need. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021.
30

---

Published in Transactions on Machine Learning Research (12/2025)
Yuval Ran-Milo, Eden Lumbroso, Edo Cohen-Karlik, Raja Giryes, Amir Globerson, and Nadav Cohen. Provable benefits of complex parameterivations for structured state space models. In Proc. Aduances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
Allan Raventós, Manshoej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, December 2023.
Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 323(6088) 536, 1986a.
David E Rumelhart, James McClelland, and PDP Research Group. Parallel Distributed Processing, Explorations in the Microstructure of Cognition, Volume 1: Foundations. MIT Press, 1986b.
Mark Sandler, Max Vladymyrov, Andrey Zhmoginov, Nolan Miller, Tom Madams, Andrew Jackson, and Blaise Agūera y Arcas. Meta-learning bidirectional update rules. In Proc. Int. Conf. on Machine Learning (ICML), pages 9288 9288-9300, Virtual only, July 2021.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. Meta- learning with nemory-augmented neural networks. In Proc. Int. Conf. on Machine Learning (ICML), pages 1842-1850, New York City, NY, USA, June 2016.
Yash Raj Sarrof, Yana Veitsman, and Michael Halm. The expressive capacity of state space models: A formal language perspective. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024.
Imanol Schlag and Jürgen Schmidhuber. Gated fast weights for on-the-fly neural program generation. In NIPS Melalearning Workshop, Long Beach, CA, USA, December 2017.
Imanol Schlag and Jürgen Schmidhuber. Learning to reason with third order lensor products. In Proc. Advances in Newral Information Processing Systems (NIPS), pages 9981-9993, Montréal, Canada, December 2018.
Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. Enhancing the transformer with explicit relational encoding for math problem solving. Preprinl arXiv:1910.0611. 2019.
Imanol Schlag, Kazuki Tric, and Jürgen Schnidluber. Linear Transformers are secretly fast weight program- mers. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2021a.
Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative inference using fast weight. memory. In Int. Conf. on Learning Representations (ICL.), Virtual only, May 2021b.
Jürgen Schmidhuber. Evolutionary principles in relf-referential learning, or on learning how to learn: the meta-meto-... hook. PhD thesis, Technische Universität München, 1987.
Jürgen Schmidhuber. Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments. Institul für Informalik, Technische Universität München. Technical Report FKI-126, 90, 1990a.
Jürgen Schmidhuber. Networks adjusting networks. In Proc. Distributed Adaptive Neural Information Processing, 1990b.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative recurrent nets. Technical Report FKI-147-91, Institut für Informatik, Technische Universität München, March 1991.
31

---

Published in Transactions on Machine Learning Research (12/2025)
Jürgen Schmidhuber. Steps towards "self-referential" learning. Technical Report CU-CS-627-92, Dept. of Comp. Sci., University of Colorado at Boulder, November 1992a.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131 139, 1992b.
Jürgen Schmidhuber. self-referential weight matrix. In Proc. Int. Conf. on Artificial Neural Networks (ICANN), pages 446 451, Amsterdam, Netherlands, September 1993a.
Jürgen Schmidhuber. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In International Conference on Artificial Neural Networks (ICANN), pages 460-463, Amsterdam, Netherlands, September 1993b.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
Jürgen Schmidhuber. 26 March 1991: Neural nets learn to program neural nets with fast weights like today's Transformer variants. 2021: New stuff!, AI Blog, 2021. URL https://people. idsia. ch/- juergen// fast mast--veggtt-prorrames 19wer-9991--1 1191-tzee tran-ttrasformer html.
Jürgen Schmidhuber, Sepp Hochreiter, and Yoshua Bengio. Evaluating benchmark problems by random guessing. A Field Guide to Dynamical Recurrent Nehworks, 2001.
Jürgen Schmidhuber. local learning algorithm for dynamic feedforward and recurrent networks. Connection Science, 144):403-412, 1989.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive procossing. Proc. National Academy of Sciences (PNAS), 118(45), 2021.
Harel Z Shouval, Mark F Bear, and Leon N Cooper. A unified model of NMDA receptor-dependent ional synaptic plasticity. Proceedings of the National Academy of Sciences, 99:10831-10885 2002.
H. T. Siegelmann and E. D. Sontag. Turing computability with neural nets. Applied Mathematics Letters, 4 (0) 1991.
Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. Deltaproduct: Improving state-tracking in linear RNNs via householder products. In Proc. Advances Neural Information Processing Systems (NeurIPS), San Diego, CA, USA, December 2025.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159 216, 1990.
Thomas R Soderling and Victor A Derkach. Postsynaptic protein phosphorylation and LTP. Trends in Neurosciences, 23:75-80 100,8004 2000.
Felke Spaak and Michael J Wolff. Rapid connectivity modulations unify long-term and working memory. Trends in Cognitive Sciences, 2025.
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. In the Deep Learning workshop at Int. Conf. on Machine Learning (ICML), Lille, France, July 2015.
Karl Steinbuch. Die lernmatrix. Kybern., 1(1):36-45 45, 1961.
Karl Steinbuch and Uwe A. W. Piske. Learning matrices and their applications. IEEE Transactions on Electronic Computers, 1466)8846-662, 1963.
Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin. What formal languages can transformers express? A survey. Trans. Assoc. Comput. Linguistics (TACL), 2024.
32

---

Published in Transactions on Machine Learning Research (12/2025)
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjum Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Samni Koyejo, et al. Tearning to (learn at test time): RNNs with expressive hidden states. In Proc. Int. Conf. on Machine Learning (ICML), Vancouver, Canada, July 2025.
Yutao Sun, Li Dong, Shaohan Huang, Shumingg Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. Preprinl Stl:2307.08621 2023.
Mattco Ticzzi, Michele Casoni, Alcssandro Betti, Tommaso Guidi, Marco Gori, and Stefano Melacci. Back to recurrent processing at the crossroad of transformers and stale-space models. Nature Machine Intelligence, 2025.
Gina G Turrigiano. The self-tuning neuron: synaptic scaling of excitatory synapses. Cell, 135:422-4335 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 5998 6008, Long Beach, CA, USA, December 2017.
Christoph von der Malsburg. The correlation theory of brain function. Internal Report 81-2, Goettingen: Department of Neurobiology, Max Planck Intilute for Biophysical Chemistry, 1981.
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintser, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Proc. Int. Conf. on Machine Learning (ICML), Honolulu, HI, USA, July 2023a.
Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, Razvan Pascanu, et al. Uncovering mesa- optimization algorithms in Transformers. Preprint or Xiv:22300 058558 2023b.
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sicberling, Alexander Meulemans, et al. MesaNet: Sequence modeling by locally optimal test-time training. Preprint ar xrx::220060022 2025.
Jane Wang, Zeb Kurth-Nelson, Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M. Botvinick. Learning to reinforcement learn. In Proc. Annvull Meeting of the Cognitive Science Society (CogScí), London, UK July 2017.
Ke Alexander Wang, Jiaxin Shi, and Emily B Fox. Test-time regression: a unifying framework for designing sequence models with associative memory. Preprint XrXiv:2501. 12352, 2025.
Gail Weiss, Yoav Goldberg, and Eran Yahav. On the practical computational power of finite precision rmns for language recognition. In Proc. Association for Compulational Linguisties (ACL), pages 740 745, Melbourne, Australia, July 2018.
Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the TEFE, 78 (10):1550 1560, 1990.
James CR Whittington, Joseph Warren, and Tim EJ Behrens. Relating transformers to models and neural representations of the hippocampal formation. In Int. Conf. on Learning Representations (ICLR), Virtual only, April 2022.
James CR Whittington, William Dorrell, Timothy EJ Behrens, Surya Ganguli, and Mohamady El-Gaby. A tale of two algorithms: Structured slots explain prefrontal sequence memory and are unified with hippocampal cognitive maps. Neuron, 113(2):321 333, 2025.
Bernard Widrow and Marcian E Hoff. Adaptive switching circuits. In Proc. IRE WESCON Convention
Record, pages 96 104, Los Angeles, CA, USA, August. 1960.
David J Willshaw, O Peter Buneman, and Hugh Christopher Languet-Higgine hol-hologcaphic associative memory. Nature, 222((197):960 962, 1969.
33

---

Published in Transactions on Machine Learning Research (12/2025)
Yujie Wu and Wolfgang Maass. A simple model for behavioral time scale synaptic plasticity (BTSP) provides content addressable memory with binary synapses and one-shot learning. Nalure communicalions, 16(1): 342, 2025.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356-365, 2016.
Songlin Yang and Yu Zhang. Fla: triton-based library for hardware--fficient implementations of linear attention mechanism, January 2024. URL https://github. http://athuu.cmmfllloaaooee/
dinear-attentai
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficien training. In Proc. Inl. Conf. on Machine Leurwing (ICML), Vienna, Austria, July 2024a.
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada, December 2024b.
Songlin Yang, Jan Kautz, and Ali Hatamizadch. Gated delta networks: Improving Mamba2 with delta rule. In Inl. Conf. on Learning Represenlations (ICLR), Vancouver, Canada, April 2025.
Morris Yau, Sharut Gupta, Valerie Engelmayer, Kazuki Irie, Stefanie Jegelka, and Jacob Andreas. Sequential- parallel duality in prefix scannable models. Preprint arXiv:2506.10916 2025.
A Steven Younger, Peter R Conwell, and Neil E Cotter. Fixed-weight. on-line learning. IEEE Trunsactions on Nearal Networks, 10(2):272-28) 2-322222232222 1999.
A Steven Younger, Sepp Hochreiter, and Peter R. Conwell. Meta-learning with backpropagation. In Proc. International Joint Conference on Neural Networks (IJCNN)), 2001.
Anthouy Zador, Sean Escola, Blake Richards, Bence Ölveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, et al. Catalyzing next-generation artificial intelligence through NeuroAL. Nature communications, 14(1), 2023.
Authony Zador and Lynn E Dobrunz. Dynamic synapses in the cortex. Newron, 19:1 4, 1997.
David Zipser and David E Rumelhart. Computational neuroscience, 1990.
Nicolas Zucchet, Robert Meier, Simon Schug, Asier Mujika, and João Sacramento. Online learning of long-range dependencies. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, December 2023.
34

---

Published in Transactions on Machine Learning Research (12/2025)
A Derivations connecting the update rules and local losses
Here we provide derivations connecting the update rules and the loss functions provided in Table 1. In the following, we omit the activation function & on keys (i.e., we replace o(k,) by kx), which does not play any role in the derivations w.r.t. W. Here we also do not specify any dimensions and instead work with arbitrary ones including ranges for the running indices in the sums. i,J, m denote positive integers.
Vanilla FWP. The local loss is defined as similarity term:
((W)) = v Wk, =
Σ Well Σ WL,
Wl.mkijm
(34)
where kyjm denotes the m-th element of vector k;; we use the notation to clearly separate the time index t from the coordinate index m.
By taking the derivative w.r.t. element Wij i R of matrix W, we obtain:
ac, (W) = -Ut ;
(35)
Ow 1.
UL. (W) = U2 8 kg Using a learning rate of 1, one
which yields the matrix form with an outer product: aW step of gradient descent corresponds the weight update:
th. W. = W ow
(W,-1) Wi- + 1s & kg
(36)
DeltaNet.
The local loss is the least square between a target v. and the current "net output" Wk;
(W) 1110 w Σ (me Σ Wim
(37)
The derivative is:
oc, aw.
(W) -(Uiji -
Σ
W Wemkke))Ka
(38)
m
which yields the matrix form:
oL. (W) = - - Wk;) 8 ke
(39)
aW
Using a learning rate of ne one step of gradient descent yields:
aL,
W. W - n aw
(W--1) = Wi-1 + ( W,-1k)) @ kt
(40)
Note that this derivation essentially corresponds to how Widrow and Hoff (1960) derived the delta rule.
OjaNet. The local objective is defined as the sum of the similarity loss in the vanilla FWP case with an
additional constraint term:
L.WW) = n2 Wk; 2 W v. =
Σ Dejt
W Wi kelm + Σ
Σ
W Wi.mvyll
(41)
m
m
The derivative is:
OC.
(W) = keei + Uejo Σ Wis
(42)
oW
35

---

Published in Transactions on Machine Learning Research (12/2025)
which yields the matrix form:
Oc, (W) -2 @ (k, WT v.)
(43)
aw
Using a learning rate of nt, one step of gradient descent yields:
ah, W = Wi Th ow (W(-1) = Wi- + m @ (k: - w._
(44)
This corresponds to Oja's rule (Oja, 1982) by treating 8, as the net output W,-ik, (the same way we treat the first term as Hebbian-like term). The corresponding loss above directly parallels Oja's objective of stabilizing the Hebbian rule by preserving the norm of the weight voctor-i.c., enforcing unit-norm constraint in the single-neuron case, which in our matrix formulation generalizes to row-wise orthonormality constraint: WW = I. The additional quadratic term in our loss, ]| v, 12, serves the same purpose as it introduces the same correction torm that keeps WWT ~ I. This equivalence can be shown formally by solving the constrained optimization problem subject to WWT = I, introducing Lagrange multipliers, and recovering the same update rule.
State Decay Variants. The local loss function for all the state decaying variants correspond to the similarity loss as in the vanilla FWP case with additional L2 regularization term the fast weight matrix. Different variants differ from each other in how the L2 term is woighted/scaled
For example for RetNet, the local loss function is:
(,(W) - Wolk)) + 1 X
Σ 2iit Σ
W Wim
ke/m +
1-
ΣV
W 1.
(45)
2
m
The derivative is:
ac,
(W) = vejkkel + (1- W)
(46)
aw, 1.
which yields the matrix form:
ac. (W) = v Kr (1 W)W
(47)
aw
Using a learning rate of 1, one step of gradient descent yields:
Oh, W. W ow (W W. uu & k; (1 (--))) -W--- + @ kz
(48)
The derivation analogous for Mamba2, xLSTM, and Gated RFA.
The case of GLA is worth its own derivation its formula looks slightly more complex different scales (elements of az) are used for different rows of matrix W. Its loss function is:
dh, ow (W) u, W (olk)) + I((V[ a.) 1) W
(49)
= Σ Uz Σ W Wimnay kimm
ΣΣΣ Σ ((1 aqp Wi.))
(50)
where (1- az) denotes a vector of the same size as ag whose entries are 1 - azls for all i.
The derivative
ac, (W)
Ueiikkel + (1 Wi
(51)
ow
36

---

Published in Transactions on Machine Learning Research (12/2025)
which yields the matrix form:
aL. (W) = 0 2 ks + ow
w
(52)
Using a learning rate of 1, one slep of gradient descent yields:
dL.
W. Wi- Ow (W--1) Wi- + k. 101-(11- az) o Wi-1 (a, 1) W 1-1 + 0s ke
(53)
(54)
Gated Del The Gated DeltaNet case can be straightforwardly obtained by combining the derivations of the DeltaNet case and the state decay case above, and using learning rate ne-
37


Reviews:
[{"review_id": "fo8VhtZhAd", "reviewer": "Reviewer_JBY1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides a review of Fast Weight Programmers (FWPs) and establishes its formal connections with state space models, recurrent neural networks and transformers. The authors explore variations of FWPs found in existing literature and provide practical considerations for training these models. The paper emphasizes how FWPs capture learning dynamics through local online optimization, with update rules corresponding to specific local objective functions. Additionally, the authors examine the relationship between FWPs and meta-learning/in-context learning, and reflect on two formulations of meta-learning as sequence learning problems: the delayed-feedback setting and the synchronous-feedback setting. Furthermore, they propose viewing fast-weight learning in FWPs as analogous to learning in the brain, while slow-weight learning corresponds to evolution. They also review the use of formal languages for evaluating expressivity and discusses how expressivity differences among FWP variants are influenced by model architectures. Lastly, the paper proposes a neurobiological implementation of FWPs where the fast weights are influenced by the AMPA receptors, the slow weights are influenced by NMDA receptor and suggests that the FWP could implement both Hebbian and non-Hebbian synaptic modifications. Overall, the paper provides a comprehensive and accessible introduction to FWPs that bridges machine learning and neuroscience, making it a valuable resource for researchers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors mathematically demonstrate the formal connections between FWPs and various classes of models. They provide key facts and intuitions from cited papers in a clear and concise manner.  Furthermore, they comprehensively cite relevant literature throughout the paper, grounding claims in existing work, and they build understanding for readers progressively. While drawing connections to neurobiology, they clearly mention where their claims are speculative. \n\nAs a review/primer paper, the work appropriately synthesizes existing research rather than providing new empirical validation, while the mathematical connections remain clear and claims remain well-supported by cited literature.", "audience_yn": "Yes", "audience_explanation": "The authors provide a comprehensive review of FWPs, their properties, and their connections with other models in the existing literature. In addition, they suggest how FWPs might be implemented in the brain. This work would be of interest to multiple audiences: the machine learning community working on efficient sequence models and the emerging neuroAI community bridging machine learning and neuroscience. Given its pedagogical approach and progressive structure, it can serve as an accessible primer for researchers new to FWPs.", "requested_changes": "- Page 5, Section 2.2: When introducing 'One type of SSM that has recently received much attention in ML,' I would like to request the authors to provide a citation for this specific SSM variant.\n- Page 12, Section 3.4: The statement 'while Gated DeltaNet has been reported to outperform DeltaNet on language modeling tasks, consistency of this advantage in other tasks has not been confirmed yet (e.g., we found the opposite trend for reinforcement learning in certain game environments)' appears to lack a citation. I would request the authors to provide a reference for the reinforcement learning results mentioned, or clarify if this is unpublished work.\n- Section 4.1: Please add brief background explanations of AMPA and NMDA receptors for readers without neuroscience training. As many members of the machine learning community may be unfamiliar with these neurobiological mechanisms.\n- In Table 1, the local objective functions are listed but their derivations for most models are not presented. If possible, it would be helpful to add the derivations for these local objective functions.\n- There is a typo on page 17 - \"diagnoal\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FXtllNXe62", "reviewer": "Reviewer_HYzD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "First of all, thank you for writing this work, I really enjoyed reading it.\nThe paper presents the Fast Weight Programming (FWP) framework and discusses how it relates to existing recurrent neural networks (RNNs) and transformers. In particular, the authors show that both state-space models and linear transformers can be expressed using the FWP equations, hence providing a unifying notation that helps bridge these architectures.\nIn the second part, the paper explores potential extensions of FWP to model computational processes in the brain, though these discussions remain highly speculative.\n\nStrengths\n• The paper is well written and the connection with transformers is clear and easy to follow.\n• The idea of framing different existing formulations (RNNs, state-space models, transformers) under a unifying mathematical notation is excellent. It provides conceptual clarity, helps highlight commonalities, and facilitates comparison of models in terms of novelty and usefulness.\n\nWeaknesses\n1. Connection to RNNs less clear.\nThe link between FWP and standard RNNs is less straightforward than the transformer case. Conceptually, the idea of a 2D hidden state in FWP makes sense, but in a standard RNN, the hidden-to-hidden transformation W_h*h_t is fixed. In FWP, W_t appears to replace this product, but it would help to clarify this equivalence explicitly.\nAlso, what are the direct analogs of q_t, k_t, and v_t in the standard RNN formulation?\n2. Terminology: \"Fast\" may be misleading.\nThe term fast only makes sense when the \"slow\" network learns via a slower algorithm such as backpropagation through time (BPTT). However, if the slow network were trained using an online learning rule, the distinction in timescale might disappear.\nMoreover, during inference (when slow weights are fixed), referring to them as \"slow-learning\" weights is technically inaccurate. The terminology would make more sense in a continual learning context.\n3. Naming convention.\nThe phrase \"training a network to generate the weights of another network\" aligns more closely with the established hypernetwork definition than FWP. \"Fast Weight Programming\" may not fully capture this relationship (though I recognise this is naming convention).\n4. Speculative neuroscience connections.\nThe section \"Other prospects and considerations for neuroscience\" draws interesting but highly speculative parallels between FWP and the CA3–CA1 hippocampal circuitry. The analogy is not fully accurate, as CA3 is known to be recurrently connected while the slow weights in FWP are not.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most claims are clear and supported by accurate reasoning. However, some statements lack supporting evidence.\nFor example, in the Practical considerations paragraph:\n\"(e.g., we found the opposite trend for reinforcement learning in certain game environments); general weight/memory decay may hurt in certain scenarios.\"\nThere are no results or references provided to support this claim (though I understand this is a Primer rather than an empirical paper).\nAdditionally, the comment in the final paragraph of Section 2.1 might not hold under online or real-time recurrent learning scenarios, and this could be briefly acknowledged.", "audience_yn": "Yes", "audience_explanation": "The topic is highly relevant to both the machine learning and computational neuroscience communities, and it will likely attract considerable interest from readers working at the intersection of these areas.", "requested_changes": "Mainly address the weaknesses listed above and provide clarification or evidence where appropriate.\nAmong them, point (2) regarding the definition and interpretation of \"fast\" vs \"slow\" learning is the most critical for strengthening the conceptual clarity and supporting an acceptance recommendation.\nThe remaining points would further improve the paper’s precision and readability.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5dPc1LhMtK", "reviewer": "Reviewer_9Co4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper gives a primer on Fast Weight Programmers (FWPs), an advancement over RNNs, which maintains a memory matrix instead of a vector in RNNs. The paper goes on to draw connections between linear transformers and FWPs highlighting that FWPs have high expressivity while bein efficient, yet are harder to train. Further the authors also draw connections with neurobiology.\n\nThe paper is a very well written review on the existing methods in this space Linear RNNs and SSMs, yet does not introduce any new insight, but rather focuses on drawing connections between existing ideas.\n\nSmall correction, in the first figure, the key and query are interchanged. Attention is between key and query.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is a review of existing concepts, highlighting their potential connections.", "audience_yn": "Yes", "audience_explanation": "Readers who are not familiar with this line of work might benefit from this work.", "requested_changes": "Fix the typo in Fig 1.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)