Nested Learning: The Illusion of Deep Learning Architectures
Authors: Ali Behrouz
Meisam Razaviyayn
Peilin Zhong
Vahab Mirrokni
Venue: NeurIPS 2025 poster ()
Keywords: Neural Learning Module
Deep Learning
Memory
Recurrent Neural Networks
Primary area: deep_learning

Abstract: Over the last decades, developing more powerful neural architectures and simultaneously designing optimization algorithms to effectively train them have been the core of research efforts to enhance the capability of machine learning models. Despite the recent progresses, particularly in developing Language Models (LMs), there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improved, and find ''effective solutions,''. In this paper, we present a new learning paradigm, called Nested Learning (NL), that coherently represents a model with a set of nested, multi-level, and/or parallel optimization problems, each of which with its own ''context flow''. NL reveals that existing deep learning methods learns from data through \emph{compressing} their own context flow, and explain how in-context learning emerges in large models. NL suggests a path (a new dimension to deep learning) to design more expressive learning algorithms with more ''levels'', resulting in higher-order in-context learning abilities. In addition to its neuroscientifically plausible and mathematically white-box nature, we advocate for its importance by presenting three core contributions: (1) Deep Optimizers: Based on NL, we show that well-known gradient-based optimizers (e.g., Adam, SGD with Momentum, etc.) are in fact associative memory modules that aim to compress the gradients with gradient descent. Building on this insight, we present a set of more expressive optimizers with deep memory and/or more powerful learning rules; (2) Self-Modifying Titans: Taking advantage of NL's insights on learning algorithms, we present a novel sequence model that learns how to modify itself by learning its own update algorithm; and (3) Continuum Memory System: We present a new formulation for memory system that generalizes the traditional viewpoint of ``long-term/short-term memory''. Combining our self-modifying sequence model with the continuum memory system, we present a learning module, called Hope, showing promising results in language modeling, continual learning, and long-context reasoning tasks.

Nested Learning: The Illusion of Deep Learning
Architectures
Ali Behrouz
Meisam Razaviyayn Google Research USA razaviyayn@google.. com
Peiling Zhong Google Research USA peilinz@google.com
Google Research
USA
alibehrouz@google.coom
Vahab Mirrokni
Google Research USA mirrokni@google..omm
Abstract
Over the last decades, developing more powerful neural architectures and simul- taneously designing optimization algorithms effectively train them have been the core of research efforts to enhance the capability of machine learning models. Despite the recent progresses, particularly developing Language Models (LMs), there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improved, and find "effective solutions," "" this paper, present
learning paradigm, called Nested Learning (NL), that coherently represents a model with a set nested, multi-level, and/or parallel optimization problems, each of which with its own "context flow". NL reveals that existing deep learning methods learns from data through compressing their context flow, and explain how in-context learning emerges in large models. NI. suggests path (a new dimension to deep learning) to design more expressive learning algorithms with more "levels". resulting in higher-order in-context learn- ing abilities. In addition to its neuroscientifically plausible and mathematically white-box nature, we advocate for its importance by presenting three core contribu- tions: (1) Deep Optimizers: Based on NL, we show that well-known gradient-based optimizers (e.g., Adam, SGD with Momentum, etc.) are fact associative memory modules that aim to compress the gradients with gradient descent. Building on this insight, we present a set of more expressive optimizers with deep memory and/or more powerful learning rules; (2) Self- Modifying Titans: Taking advantage of NI's insights on learning algorithms, we present a novel sequence model that learns how to modify itself by learning its own update algorithm; and (3) Continuum Memory System: We present new formulation for memory system that general- izes the traditional viewpoint of "long-ternn "long-term/Shoot-teen memory". Combining our self-modifying sequence model with the continuum memory system, we present a learning module, called HOPE, showing promising results in language modeling, continual learning, and long-context reasoning tasks.
Introduction
This version of the paper has been extensively summarized to fit the page limit of NeurIPS camera ready, and some materials, experiments, better baselines, discussions, and methods are in the full version, which might make some parts this version hard to follow or cause inconsistencies. To avoid such cases, please read our arXiv version [1].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Uniform and Reuaalle Structure
Namppatticty the brun's sbility nampunoe taell sta mxchuniara like foening new synspses, stren beenghhninggwweie weakening existig ones, ressoting signals through altemate pathwoys etc. Suct shilty requires eniforn and rusalle structoe aeross heam..
Low Erequincy Nourons
Theta Ware (4: 8 (ll)
Mid Trequenty Nauuonss
Agala Woree (R- 12 li))
isto nesrons (s., ltver leaily deep MI each which with contect dow objectiv. This doige parrikes uniferm ruuable sinature for laming.
Nestel Leamning (NI) architettres
Frequencc Neuruns
Multi Time Seale Update
Itighest Frequency Neurons
sxcilitions
actirity
Gonmma Warre
Nusably the eatlies updute thei
centralard cleck synchnuntie cycles, whernas beer interrae information kongee,
saly
Bota Warm (12- 30 Hs)
(30 300 Ita)
frequency
Laseer
In Ni. Ail..grmmeeesss in euch "lewt" "am spdaned with their can speetic fregamcc and docn not noly un ennll contraliand rlcck. The Hore's design allona the curler layers update theis acivvlt quickly is high Biequnac eydeo, whereu Jater layers Integrate information over longer. slower cycles.
Figure 1: The uniform and reusable structure as well as multi time scale update in the brain are the key components unlock the continual learning in humans. Nested Learning (NI.) allows for multi time-scale update for each component of the brain, while showing that well-known architectures such as Transformers are in fact linear layers with different frequency updates.
For decades, Al research has focused on designing machine learning algorithms that learn from data [2-5] or experience [6-8]; often by optimizing an objective L(0) over parameters E O with gradient-basee methods. While traditional machine learning techniques required careful engineering and domain expertise to design feature extractors, limiting their ability to directly process and learn from natural data [9], deep representation learning offered a fully automated alternative discover the representations needed for the task. Thereafter, deep learning has been an inseparable part of the large-scale computational models with seminal success chemistry and biology [10], games [11, 12], computer vision [13, 14], and multimodal and natural language understanding [15-17].
Stacking of multiple layers, as it is done in deep learning models, provides the models with larger capacity, better expressive power in representing complex features, and more internal computations (e.g., #FLOPS) [18-20], all of which are critical and desirable characteristics for static tasks that require in-distribution predictions over previously fixed set. This deep design, however, is not universal solution to all the challenges and cannot help the expressive power of the models in multiple aspects, for example: (i) The computational depth of deep models might not change with more layers [21, 22], leaving their ability to implement complex algorithms untouched compared to traditional shallow approaches [23]; (ii) The capacity of some class of parameters might show marginal improvement with increasing the depth/width of the model [24]: (ii) The training process might converge to a suboptimal solution, mainly due to the suboptimal choice of the optimizer or its hyperparameters: and (iv) The model's ability to fast adapt to a new task, continually learn, and/or generalize to out-of-distriutiio data might not changed with stacking more layers and requires more careful designs.
The core part of the efforts to overcome the above challenges and to enhance the capability of deep learning models concentrate on: (I) developing more expressive class of parameters (i.c.. neural architectures) [13, 25-28]; (2) introducing objectives that can better model the tasks [29- 32]; (3) designing more efficient/effectivve optimization algorithms to find better solutions or with more resilience to forgetting [33-3]]; and (4) scaling the model size to enhance its expressivity, when the "right" choice of architecture, objective, and optimization algorithms are made [24, 37, 38). Collectively, these advancements and new findings on scaling patterns of deep models have established the foundations upon which Large Language Models (LLMs) have been built.
The development of LLMS marks a pivotal milestone in deep learning research: a paradigm shift from task-specific models to more general-pure pprepal-prrpose systems with various emergent capabilities as a result of scaling the "right" architectures [38, 39]. Despite all their success and remarkable capabilities in diverse sets of tasks 40, 41], LLMs are largely static after their initial deployment phase, meaning that they successfully perform tasks learned during pre- or post- post-training, but are unable to continually acquire new capabilities beyond their immediate context. The only adaptable component of LLMs is their in-context learning ability-a (known to be emergent) characteristic of LLMs that enables fast adaption to the context and so perform zero- or few-shot tasks [38]. Beyond in-context learning, recent efforts to overcome the static nature of LLMs either are computationally expensive, require external components, lack generalization, and/or might suffer from catastrophic forgetting [42-44], which has led researchers to question if there is a need to revisit how to design machine learning

---

models and if a new learning paradigm beyond stacking of layers is required to unleash the capabilities of LLMs in continual setups.
Current Models only Experience the Immediate Present. As an analogy and to better illustrate the static nature of LLMs, we use the example of anterograde amnesia-a neurological condition where a person cannot form new long-term memories after the onset of the disorder, while existing memories remain intact [45]. This condition limits the person's 's knowledge and experiences to short window of present and long past-before the onset of the disorder which results in continuously experiencing the immediate present as if it were always new. The memory processing system of current LLMs suffer from a similar pattern. Their knowledge is limited to either, the immediate context that fits into their context window, or the knowledge in MLP layers that stores long-past, before the onset of "end of pre-training " This analogy, has motivated us to take inspiration from neurophysiology literature and how brain consolidate its short-term memories:
1.1 Human Brain Perspective and Neurophysiological Motivation
Human brain is highly efficient and effective when it comes to continual learning (a.k.a, effective context management), which is often attributed to neuroplasticity tte brain's 's remarkable capacity to change itself in response new experiences, memories, learning, and even damage [46, 47]. Recent studies support that the formation of Long-term memory involves at least two distinct but complementary consolidation processes [48-50]: (1) A rapid "online" consolidation (also known as synaptic consolidation) phase occurs immediately or soon after learning, even during wakefulness. This is when new and initially fragile memory traces are stabilized and begin transferring from short-term to long-term storage; (2) An "offline" consolidation (also known systems consolidation) process repeats the replay of the recently encoded patterns-during sharp-wave ripples (SWRs) in the hippocampus, coordinated with cortical sleep spindles and slow oscillations-streeeee strengthens and reorganizes the memory and supports transfer to cortical sites [51-53].
Coming back to the analogy of anterograde amnesia, evidence indicates that the condition can impact both stages, but especially the online consolidation phase, mainly due to the fact that hippocampus is the gateway for encoding new declarative memories, and so its damage means new information never will be stored in long-term memory. As mentioned above, the design of LLMs, and more specifically Transformer- based backbones, suffers from similar condition after the pre-training phase. That is, the information provided in the context, never impacts the long-term memory parameters (e.g.. feedforward layers), and so the model is not capable of acquiring new knowledge or skill, unless the information is still stored the short-term memory (e.g., attention). To this end, although the second stage is equally, even more, crucial for the consolidation of memories. and its absence can damage the process and might cause loss of memory [54, 55], in this work, we focus on the first stage: memory consolidation as online process.
Notations. We let E Nxda be the input, MM. represent the state of memory/model M at time f, K be the keys, V be the values, and Q be the query matrices. We use bold lowercase letters with subscript to refer to the vector corresponds to the input (i.e., kr, Us and qt). We further refer to the distribution of any entities f as p(f). Through the paper, we use simple MLPs with Lm 2 1 layers and residual connection as the architecture of the memory module M(.)) When it is needed, we parameterized the memory module with H. 2 (W1, W WEM }. which at least includes the parameters of linear layers in the MLP. We use superscript with parenthesis refer parameters in different levels of nested learning (different update frequency): i.e., W(O.
2 Nested Learning
This section discusses the motivations, formal definitions, and general high-level implications of Nested Learning (NL). We start with a formulation of associative memory and then by using step- by-step examples, we build the intuition behind architecture decomposition and its connection to modeling a neural network as an integrated system of optimization problems. We aim to first show how existing methods and concepts in deep learning fall under the NL paradigm and then we present new formulations that go beyond traditional methods and/or provide insights on how to improve existing algorithms and designs.
3

---

Deep Learning
Neural Learning Modiule
mrgn CIMER
Grudiaal Fire
Levelo
Gradione
Mast Mal-M--FGMMMl F..M.K.
Gradienk Flox
Gradient
urgnns CIM.N vay
Plorr
Mi Ma--MM-FMMMMM (
Gradient Blow
The above figure illustrates the nasted learning reposentation of hybrid Attention),
deep learning representation (ax Slttien image NL) internal goudient its training process with the architecture, Ni. makes all the internal process transporent and mathematically white-box. box.
Lanval 3
acpurutes
Gradlnt
Mus M-TEIMLEE
Figure Nested Learning Paradigm that represent a machine learning model and its training procedure as a set of nested optimization problems. (Left) An example of Hybrid architecture. While deep learning perspective, as the flattened image of NL, does not provide insight about the depth of computation in the blocks, NL transparently represent all the inner gradient flows. (Right) A Neural Learning Module: A computational model that learns how to compress its own context flow. For example, the first level corresponds to the model's the most outer-loop training, often refer to as "pre-training" step.
2.1 Associative Memory
Associative memory- the ability to form and retrieve connections between events-is a fundamental mental process and is an inseparable component of human learning [56]. Often in the literature, the concept of memorization and learning used interchangeably; in neuropsychology literature, how- ever, these two are clearly distinguished. More specifically, following neuropsychology literature [57], build our terminology based the following definition of memory and learning:
Learning vs. Memorization:
Memory is a neural update caused by an impul, and learning is the process for acquiring effective and usefud memory.
In this work, our goal is to first show that all the elements of a computational sequence model, including optimizers and neural networks, are associative memory systems that compress their own context flow. Broadly speaking, associative memory is an operator that maps a set of keys to a set of values. We follow the general definition of associative memory by Behrouz al. [58]:
Definition (Associative Memory). Given of keys K. C Rd and values V C R. associative memory is an operator M K. that maps two sets of keys K. and values V. To learn such mapping from the data, an objective ((;-)) measures the quality of the mapping and M can be defined as:
M = arg min ((M(K):V). M
(1)
While the operator itself is memory and the mapping acts as a memorization process (i.e., memoriz- ing the connections of events in the context), acquiring such effective operator based on the data, is learning process. is notable that, here, keys and values can be any arbitrary events that memory aims to map them and are not limited to tokens. Later in this section, will discuss that given context flow, keys and values might be tokens, gradients, sub-sequences, etc. Furthermore, while the term of associative memory more common in neuroscience and neuropsychology literature, the above formulation also closely related to data compression and low- di-ennioniill representation. That is, interpret the optimization process in Equation I the training process of network M(.) that aims to compress the mappings into its parameters and so represent them lower dimensional space.
In sequence modeling, where keys and values are input tokens (e.g., tokenized text), the choice of objective and the optimization process for solving Equation 1 can result in distinct sequence

---

modeling architectures (see [59] and [58]) such as global/local softmax attention [27], or other modern recurrent models [28. 60, 61]. This simple formulation of sequence models provides us with better understanding of their internal process and also tool to simply compare their modeling power based on their objective and optimization process. In the following, using step-by-ster examples, we discuss how this formulation can be applied to all components of a neural architecture (including its optimization process in pre-training) and in fact, how a model an integrated system of multi-level, nested, and or parallel memories, each of which with its own context flow.
A Simple Example of MLP Training, We start with simple example, in which we aim to train a 1-layer MLP (parameterized with W) for task T and on dataset 2 Duain {ar...... 2 PPuull ] by optimizing the objective ((::-) with gradient descent. In this case, the training process is equivalent to the following optimization problem:
W = arg min L(W; Durii),
(2)
W
whose optimization by gradient descent results in weight update rule equivalent to:
W4+1 = W M++1 V w. ((W, 24+1) W + 1 V Mo+ ((W1; F++1, where F++1 Dirain
(3)
where Vt+11 W is the output of the model for input T++1- Given this formulation, one can let 1 V M 1 [(W and reformulate the backpropagation process as the solution to an optimization problem on finding optimal associative memory that maps input data points
Durain {2x [Puiill to their corresponding ue+1 V y <(W,; That is, we let M(-) W; parametrizes the memory, and use dot-produs similarity to measure the quality of W, mapping between Fz+1 and V 114,,(WW, I++1):
W4+1 = arg min [W2411, M441) W
I
W.l1
(5)
2m++1
arg min (Wz, V Vuneo L(W;: ((4)) 211+1
IW - -..12.
W
In the above formulation, Ue+1 V yoss L(W,; can be interpreted as local surprise signal in representation space that quantifies the mismatch between the current output and the structure the objective C(-: .) enforces. Therefore, this formulation translates the training phase of the model as process of acquiring effective memory that maps data samples to their Local Surprise Signal (LSS) in representation space-defined as the mismatch between the current output and the structure enforced by the objective -(::-). Accordingly, in this example, our model has a single gradient flow over the data samples, which is only active over dataset Drain {I1, I Dean and will be frozen for any other data samples afterwards (a.k.a a inference or test time).
Next, in the above examplc, we replace the gradient descent algorithm with its enhanced momentum- based variant, resulting an update rule of:
W4+1 W.- M1+1,
(7)
m = m, - Mh+1 Ww, L(W;; 1z41) = m, -74+1 V V L(W,; (2+1) 8 Te+1-
(8)
In Equation 8, given the previous state of Equation 7 (at time 1), the value of Vw, L(W6 or similarly V Vetz ((+; are independent of recurrence in Equation 8 and so can be pre-computed beforehand. To this end, we let u++1 = Ww, (W, and so Equation can be reformulated as:
W+11 Wt m1+1.
m = arg min m
- (m, Vw, (WW,51 + 7-+1 [m - m11
(10)
arg min (m 2++1, V Me+1 L (W,: (x+1)) + 7 ||m m.
(11)
where the optimization problem in Equation 10 is equivalent to on step of gradient descent with adaptive learning rate of 7/+1- Given these formulation, one can interpret the momentum term as either: (1) a key-less associative memory that compress the gradients into its parameters, or (2) an associative memory that learns how to map data points to their corresponding I.SS-value Interestingly, this formulation reveals that gradient descent with momentum is indeed two-level
5

---

optimization process, where the memory is optimized by simple gradient descent algorithm. This process is closely related to Fast Weight Programs (FWPs) [62], where the weight update process (i.e., Equation 9) is the slow network that its momentum weight is generated by fast network (i.e.. Equation 10).
Concluding the above examples, we observed that the training process of a l-layer MLP with: (I) Gradient descent is I-level associative memory that learns how to map data points to their corresponding LSS-value; and (2) Gradient descent with momentum is a 2-level associative memory (or optimization process) that the inner-level learns to store gradient values into parameters, and then the outer-level updates the slow weight (i.e.. W)) with the value of the inner-level memory. While these are the most simple examples with respect to both architecture and optimizer algorithms, one might ask if similar conclusion can be made in more complex setups.
An Example of A Archttectuaal Decomposition. In the next example, we replace the MLP module with a linear attention [60]. That is, we aim to train a 1-layer linear attention for task T and on a sequence of Darin = { , Z (DDaaul) } by optimizing the objective L with gradient descent. Recalling the unnormalized linear attention formulation:
ke x. Wk:
=== W.:
q: :==:Wo Wg
(12)
MM = M + v,ki
(13)
1/- M.g:
(14)
As discussed in earlier studies [58, 59], the recurrence in Equation 13 can be reformulated as the optimization process of matrix-valued associative memory M,(-), in which, it aims to compress the mappings of keys and values into its parameters. In more details, in Definition 1, if we let [(Me-1; k,, vz) := (MMeikke ve) and aim to optimize the memory with gradient descent, the memory update rule is: (Note that V VL(M--1: ki, v2)) = v, and we let learning rate 1): = 1)
M arg min (Mik-=1 1c+1) + IIM Mellé with gradient descent, (15)
M
M = M, V VÉ(Me; ke+1, v2+1) #2+1) = .M + 0et1ke-11
(16)
which is equivalent to the update rule of an unnormalized linear attention in Equation 13. Also, note that as we observed in the first example, training a linear layer with gradient descent is I-layer optimization problem of an associative memory (see Equation 3) and so the general training/updating process of projection layers (i.e., We. W., and Wg) itself an optimization process of associative memory. Accordingly, this setup, i.e., training linear attention with gradient descent, can be seen as a two-level optimization process, where the lote (also known as training process) optimizes the projection layers with gradient descent, while the inner-loop optimizes the inner memory of M, with gradient descent.
Note that, as discussed above, here, we have two associative memories, and so each of which has their own optimization process and gradient flow. That is, in the optimization of outer-level parameters of Wk. We, and Wa there is no gradient with respect to parameter MM()) and so there is no backpropagation through it. Similarly, the inner-level, there is no backpropagation through projection layers and they are considered frozen. Furthermore, it is notable that in this example, the above formulation is also closely connected FWPs perspective of linear attentions [63], where projections are considered slow weights, and memory update in Equation 13 is the fast weight update rule.
Architectural Decomposition with More Levels. In both above examples, discussed simple cases, where they can be translated into 2-1evel optimization processes, which also coincides with their FWPs interpretations. In practice, however, we need use more powerful optimization algorithms to train the model, and/or use more powerful recurrent update rule for memory. As a simple example, assume we use gradient descent with momentum to train linear attention model. In the above examples, we show that how the linear attention component can be decomposed into two nested optimization problem. Similarly, here the model can be represented as a 2-1evel optimization problem, where (1) the inner level optimizes the memory to compress the context using gradient descent (see Equation 15), and (2) the outer level optinizes the projection layers with gradient descent with momentum. Interestingly, from the first example, we know that "gradient descent with momentum" algorithm itself is indeed a 2-level optimization problem where the momentum term itself is an associative memory that compress the past gradients into its parameters.

---

2.2 Nested Optimization Problems
In the previous section, we provided examples to demonstrate how one can decompose a machine learning model into set of nested or multi-level optimization problems. Next, we first aim to present formal formulation for nested learning problems and then define Neural Learning Module-an an integrated computational system that learns from data.
As we observed in the previous section, while decomposed the model into set of optimization process, it is still unclear if we can define a hierarchy (or order) over these problems, and uniquely represent the model in this format. Inspired by the hierarchy of brain waves that indicates the information processing frequency rate of each part (discussed in Section 1), we use the update rate of each optimization problem to order the components in multiple levels. To this end, we let the one update step over one data point to be the unit of time, and define the update frequency rate of each component as:
Definition 2 (Update Frequency). For any component of A, which can be parametric component (e.g. learnable weights or momentum term in gradient descent in momentum) or a non parameetiiiic component (e.g., attention block), we define its frequency, denoted as SA, as its number of updates per unit of time.
Given the above update frequency, we can order the components of machine learning algorithm based on operator (- > -). We let A to be faster than B and denote A > B if: (1) fA > .B. or (2) fA fB but the computation of the B's state at time requires the computation of AAs state at time In this definition, when A Y B and B X A, we let A f B, which indicates that A and B has the same frequency update, but their computation is independent of each other (Later, we provide an example of this cases in AdamW optimizer). Based the above operator, we sort the components into an ordered set of "levels", where (1) components in the same level have the same frequency update, and (2) the higher the level is, the lower its frequency. Notably, based on the above definition, each component has its own optimization problem and so context. While we optimize the component's inner objective with gradient-basee based optimizers, the above statement is equivalent to having exclusive gradient flow for each component in the model. In general case, however, one can use non-pane pana-armettric solution (as we later discuss about attention).
Neural Learning Module. Given the above definition of nested learning problems, we define neural learning module as a new way of representation of machine learning models that shows the model as an interconnected system of components, each of which with its own gradient flow. Note that, orthogonal to deep learning, nested learning allows us to define neural learning models with more levels, resulting in more expressive architecture.
Nested learning allows computational models that are composed of multiple (multi-layer) levels to learn from and process data with different levels of abstraction and time-scales.
Next, we study optimizers and well-known deep learning architectures from the nested learning perspective, and provide examples that how NL can help to enhance those components.
2.3 Optimizers as Learning Modules
In this section, we start by understanding how well-known optimizers and their variants are special instances of nested learning. Recall the gradient descent method with momentum,
W W + m
m 0:4 m; VL (Wa.;)
(17)
where matrix (or vector) m; is the momentum at state i and o and 7/s are adaptive learning and momentum rates, respectively. Assuming 04+1 1, the momentum term can be viewed as the result of optimizing the following objective with gradient descent:
min m
(m vL(W..)))
(18)
This interpretation shows that momentum can indeed be viewed as meta memory module that learns how to memorize gradients of the objective into its parameters. Building this intuition,
7

---

can show that Adam is the optimal associative memory for the L2 regression objective on models' gradients. Next, we show that how this perspective can result in designing more expressive optimizers:
More Expressive Association. As discussed earlier, momentum is value-less ciative memory and so has limited expressive power. To address this issue, following the original definition of associative memory (i.e., mapping keys values), we let value parameter v P, and so the momentum aims to minimize:
Extension:
min (m VZ(W;=;)" P.)),
(19)
using gradient descent, resulting in the update rule:
W++1 = W; + m;+1
m 0441m, M.P.VL
(20)
This formulation is equivalent to using preconditioning the momentum GD. fact, preconditioning means that the momentum term is an associative memory that learns how to compress the mappings between P; and the gradient of preconditioning can improve the expressivity of the initial version of GD with momentum per se is value-less memory (i.e., mapping all gradients to a single value), the above perspective gives more intuition about what preconditioning are more useful. That is, the momentum acts as memory that aims map gradients their corresponding values, and function of gradients (e-g., information about Hessian) can provide the memory with a more meaningful mappings.
VC(W;; While any reasonable choice (e.g., random features)
More Expressive Objectives. As discussed by Behrouz et al. [58], optimizing an inner objective of dot-product similarity results in Hebbian- update rule, which can cause the memory to be less effective. A natural extension of this internal objective is to use (-)-) regression loss (for measuring the corresponding key-value mapping fitness) and minimize the loss func- tion VLV x;) P.Il2 resulting in the update rule of:
Extension:
W W; m,+1-
(21)
m (o. I- VL (Wax,) T.WWE,,TVVCC (WW m: M
(22)
This update is based on delta-rule [64] and it allows the memory (momentum) to better manage its limited capacity and better memorize the series of past gradients.
Extension: More Expressive Memory. As discussed earlier, momentum can be viewed as meta memory model that uses a linear layer (i.e., matrix-valued) to compress the past gradient values. Due to the linear nature of momentum, only linear functions of past gradients can be learned by its internal objective. To increase the learning capacity of this module, one alternative is use alternative powerful persistent learning modules: i.c., replacing a linear matrix- memory for momentum with an MLP. Therefore, momentum as the a memory for the past gradients, has more capacity to capture the underlying dynamics of the gradients. To this end, we extend the formulation in Equation 17 as:
W.+1 W; + m,+1 (u:),
and m m 7/ V ( (m; u, I),
(23)
where u, = VL (W; and (-) is the internal objective of momentum (e.g., dot product similarity (m(u ), 1)). We refer to this variant as Deep Momentum Gradient Descent (DMGD).
Extension: None Linear Outputs. Building upon the above perspective, which we see momentum neural architecture, one common technique to enhance the representation power of momentum memory module is to use non-lincarity on top of its output [28, 65]. That is, we re-formulate Equation 23
(24) where (--) is an arbitrary non-linearity. As example, we let (()) Mewton-Schulz(( where Newton-Se -SwrtonSSchu(() is the iterative Newton-Schulz method [66], and m(-) be a linear layer; the resulted optimizer is equivalent to Muon optimizer [34].
W4+1 = W, + a (m.+1 (u.),, and m:+1 a m.
(2) (m I),

---

Going Beyond Simple Backpropagation. As discussed earlier in Section 2.1, the pre-training process and backpropagation form of associative memory, where input data mapped to the surprised caused by its predicted output Vg. z.W;
W++1 Wt- - 7 W., (W m4) = W,- 7 V (( @ 2t, where 24 ~ Duain (25)
which from the associative memory perspective equivalent one step of gradient descent in optimization process of:
min (Wae, V. ((We W
(26)
As we discussed in Appendix D, the above formulation cause ignoring the dependencies of data samples like hy. To extend it more powerful formulation where also consider the dependencies of data points (which is extremely important when we use optimizer in the token space as they are not independent), we L2 regression objective with one step of gradient descent as follows:
min W
| V ((W:;
(27)
This formulation results in new variant of gradient descent, which can be simplified as follows:
W++1 We [I - 2 M V w, ((We;
(28)
W. (I - 2,22 7 V LWW; m4) & Te.
where I₁ Dirann
(29)
Later, this optimizer the internal optimizer of our HOPE architecture.
3 HOPE: A
Learning Module with Continuum Memory
Existing architectural backbones consist of (1) a working memory module (e.g., attention), which is responsible to actively fuse the information across sequence length, and (2) feed-forward layer (e.g.. MLP) that fuse information across features and acts as the persistent memory or knowledge storage of pre-training phase. From the NL perspective, pre-training the phase that the most outer level of the learning module is updated over its limited context flow. Accordingly. in the continual setup, such pre-training phase is also rarely updated over time, and its corresponding knowledge storage needs to rarely be updated over time. Given this intuition, we extend the traditional view-point of long-term/short-eee memory system and suggest knowledge storage feed-forward for each level (frequency domain).
Given the definition of frequency and arbitrary chosen objective L. Continuum Memory System (CMS) is formalized as chain of MLP blocks MLP(fi(.),,.. MLP(Sh)(.). each of which associated with a chunk size of ( = max, CCC such that given input x {x1,..., Ør} the output of the chain is calculated as (we disregard normalizations for the sake of clarity):
yt =MLP/*) (MLP(J> ()...) (M..MM)(()))))))
(30)
where the parameters of (-th MIP block, i.e., ((()), are updated every (t0) steps:
( ++1 = o(5e) Ei-ioco ne f(0! ()) i (mod ((O),
(31)
otherwise.
In Appendix C, we discuss different variants of this formulation, including fully nested MLP layers. Here 1/2" (f) are learning rates corresponds to @(ft) and f(.) is the error component of an arbitrary optimizer (e.g., V (f) in gradient descent). The MLP block in conventional Transformers architecture [27] is a special instance of this formulation, where k = 1 and frequency is zero (after pre-training). It is notable that Equation 31 provides important interpretation: parameters e(fx) are responsible for compressing their own context into the their parameters and so they are a representative of abstract knowledge of their context.
HOPF. We further present self-referential learning module based on Titans [28] and our variant of gradient descent in Appendix Combining this self-referential sequence model with continuum memory system results in HOPE architecture.

---

Hope Nosted Learning
Hope Deep Learning
Trazsformers Neated Learning
Transformers Deep Learning
Chunki sngih: ICX HPTE Longth 1 Proguenos)
lengtle urttiemple
Lows Frsciencc FFN
Mid Frequency FEN
Proguencyy 16
Irequeecy
Counk Longl: BPTT Lenght:
Atteetton
Proqucses; do
Chunk Lengh: artlleeg
Soldiidg Thans
Chusk Lougle Progurtes taking
Prejooone IN
Linner
Linoe
Figure comparison of Hope architectural backbone with Transformers (Normalization and potential data-dependent components are removed for the sake of clarity). It is notable that while self-modifying Titans replacement for the attention block, CMS is replacement for the static MLP block.
Table
Performance of HOPE and baselines on language modeling and common sense reasoning tasks. Hybrid models are marked with
Model Wiki. ppl
LMB.
LMB.
PIQA
Hella.
Wino. acc f
ARC-e acc
ARC-c acc_.n
SIQA acc
BoolQ acc ?
Avg.
ppl 1
acc
acc 1
acc n
760M
params 30B tokens
Transformer++
25.21
27.64
35.78
66.92
42.19
51.95
60.38
32.46
39.51
60.37
48.69
RetNet
26.08
24.45
34.51
67.19
41.63
$2.09
63.17
32.78
38.36
57.92
48.46
DeltaNet
24.37
24.60
37.06
66.93
41.98
50.65
64.87
31.39
39.88
59.02
48.97
TTT
23.51
67.25
43.92
$0.99
33.81
40.16
59.58
47.32
Samba'
20.63
22.71
39.72
69.19
47.35
52.01
66.92
33.20
38.98
61.24
$1.08
Titans (LMM)
20.04
21.96
37.40
69.28
48.46
52.27
66.31
35.84
40.13
62.76
51.56
HOPE (ours)
20.53
20.47
39.02
70.13
49.21
52.70
66.89
36.05
40.71
63.29
52.26
1.3B params 100B tokens
Transformer++
18.53
18.32
42.60
70.02
50.23
53,51
68.83
35.10
40.66
57.09
52.25
RetNer
19.08
17.27
40.52
70.07
49.16
54.14
67.34
33.78
40.78
60.39
$2.02
DeltaNet
17.71
16.88
42.46
70.72
50.93
$3.35
68.47
35.66
40.22
55.29
$2.14
Samba
16.13
13.29
44.94
70.94
53.42
55.56
68.81
36.17
39.96
62.11
54.00
Titans (LMM)
11.41
49.14
73.09
56.31
59.81
72.43
40.82
42.05
60.97
56.82
HOPE (ours)
15.11
11.63
50.01
73.29
56.84
60.19
72.30
41.24
42.52
61.46
57.23
4 Experiments
For the sake of space, in the main paper, we report the results of the HOPE'S evaluation on language modeling, and common-sense reasoning, tasks. However, we report extensive set of results, including on experiments on optimizers, emergence of in-context learning, continual learning abilities of HOPF, ablation studies, long-context tasks, etc. in the appendix. Details about the experimental setups and other used datasets are in Appendix
Language Modeling and Common-sense Reasoning. We follow recent sequence modeling stud- ies [28, 67, 68] and report the results of HOPE and baselines with size of 760M, and 1.3B on language modeling and also commonsense reasoning downstream tasks [69-75]. These results are reported in Table 1. HOPE demonstrate very good perfomance across all the scales and benchmark tasks, out- performing both Transformers and recent modern recurrent neural networks, including DeltaNet [63] and Titans [28]. Comparing HOPE to Titans and DeltaNet, we can see that dynamically changing the key, value, and query projections based on the context as well deep memory module can result in model with lower perplexity and higher accuracy in benchmark results.
10

---

References
[1] Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. Nested learning: The illusion of deep learning architectures. arXiv preprint arXiv:25/2.24695, 2025.
[2] Walter Pitts. The linear theory of neuron networks: The dynamic problem. The bulletin of mathematical biophysics, 5:23-31, 1943.
[3] Warren S McCulloch. The brain computing machine. Electrical Engineering, 68(6):492-49) 1949.
[4] Warren S McCulloch and Walter Pitts. The statistical organization of nervous activity. Biomet- rics, 4(2):91-99, 1948.
[5] Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210-25 229-22,, 1959.
[6] David Silver and Richard Sutton. Welcome to the era of experience. Google AI, 1, 2025.
[7] Richard S Sutton, Andrew Barto, al. Reinforcement learning: An introduction, volume 1.
1998.
[8] Jonathan H. Connell and Sridhar Mahadevan. Robot learning. Robotica, 17(2):229-233) 1999.
doi: 10.0077/0223337799977772
[9] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7533) 436-444, 2015.
[10] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):883-89,, 2021.
[1I] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):
484 489, 2016.
[12] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):11400 140-::::11111111 144, 2018.
[13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreviees
[15] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint 0ix.:25577002661 2025.
[16] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19433 2024.
[17] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint XXXv:2303.08774 2023.

---

[18] Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances neural information processing systems, 27, 2014.
[19] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
[20] Joel Hestmess, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 171..00019 2017.
[21] William Merrill, Ashish Sabharwal, and Noah A Smith. Saturated transformers are constant- depth threshold circuits. Trunsactions of the Association for Computational Linguistics, 10: 843-856, 2022.
[22] Clayton Sanford, Daniel Hsu, and Matus Telgarsky. Transformers, parallel computation, and logarithmic depth. In Forty-first International Conference on Machine Learning, 2024. URL https://openreviee net/ forum? wwtpummmmddicaa idCCZabbkhBB..
[23] William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In Forty-first International Conference on Machine Learning, 2024. URL https: wwtprr/eww..eettourid//////oooo
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[25] Juergen Schmidhuber and Sepp Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
[26] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-202, 1980.
[27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: //poceeee/c/sc///sss/ neurips cc/paper_ffie httppper_ffll/paapppppeeeeeeeeep Paper pdf.
[28] Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.0066: 2024.
[29] David Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back-propagating errors. nature, 323(6088):5333-53 1986.
[30] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Orair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144 2020.
[31] Shaden Alshammari, John Hershey, Axel Feldmann, William Freeman, and Mark Hamilton. I-con: A unifying framework for representation learning, arXiv preprint arXiv:2504. 16929, 2025.
[32] R Devon Hjelm, Alex Fedorov, Samuel Aavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019. URL
net/ feet/forum? 3o- 1dBBk1r3jockk
[33] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.698) 2014.
12

---

[34] K Jordan, Y Jin, V Boza, Y Jiacheng, F Cecista, L Newhouse, and J Berssten. Muon: An optimizer for hidden layers in neural networks, 2024b. URL https://kellerjordan github. io/posts/muon, 2024.
[35] Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842-1850. PMLR, 2018.
[36] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: Improving and stabilizing shampoo using adam for language modeling. In The Thirteenth International Conference on Learning Representations, 2025.
URL https:// http::://oppnrevier wwt//eetforrnn wwwoormmiiidddiaa Id=IxxXXXPN..
[37] Jordan Hoffmmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint 2...xx:2203.555333 2022.
[38] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 2020.
[39] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in neural information processing systems, 36:55565-55581, 2023.
[40] Erik Nijkamp. Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https: /tattpa:/oopeee http:////ppppeeeeeee/eemeemmeee wwwee//oorriiddaaieteoeaaaaa
[41] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36: 61501-61513, 2023.
[42] Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, et al. Cartridges: Lightweight and general-purpose long context representations via self-study. arXiv preprint 206.062506.06266 2025.
[43] hongzhou yu, Tianhao Cheng, Yingwen Wang, Wen He, Qing Wang. Ying Cheng, Yuejie Zhang, Rui Feng. and Xiaobo Zhang. M-oo Enhancing medical knowledge reason- ing ability of LLM from supervised fine-tuning to test-time training. In Second Conference on Language Modeling, 2025. URL httpp:://ppereeeeeetee
[44] Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning. In Forty-second International Conference on Machine Learning. 2024.
[45] William Beecher Scoville and Brenda Milner. Loss of recent memory after bilateral hippocam- pal lesions. Journal of neurology, neurosurgery, and psychiatry. 20(1):11, 1957.
[46] Alvaro Pascual-Leone, Amir Amedi, Felipe Fregni, and Lotfi B Merabet. The plastic human brain cortex. Annu. Rev. Neurosci., 28(1):377-401, 2005.
[47] Michael V Johnston. Plasticity in the developing brain: implications for rehabilitation. Devel- opmental disabilities research reviews, 15(2):94-101, 2009.
[48] Akihiro Goto, Ayaka Bota, Ken Miya, Jingbo Wang, Suzune Tsukamoto, Xinzhi Jiang, Daichi Hirai, Masanori Murayama, Tomoki Matsuda, Thomas McHugh, Takeharu Nagai, and Yasunori Hayashi. Stepwise synaptic plasticity events drive the early phase of memory consolidation. Science, 374(6569)8557-63,, 2021. doi: 10. 126/science.abb91995 URL. http://wwwww.cinccc//c org/ http:eeerrrr////oo///e/eo//o//// htt///////b///e// wttp///scenceeeeeeeeeeeeeeeeeeee abj9195.
13

---

[49] Uwe Frey and Richard GM Morris. Synaptic tagging and long-term potentiation. Nature, 385 (6616):533-536, 1997.
[50] Wannan Yang, Chen Sun, Roman Huszár, Thomas Hainmueller, Kirill Kiselev, and György Buzsáki. Selection of experience for memory by hippocampal sharp wave ripples. Science, 383(6690): 1483, 2024.
[51] Daoyun and Matthew Wilson. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nature neuroscience, 10(1):100-107, 2007.
[52] Adrien Peyrache, Mehdi Khamassi, Karim Benchenane, Sidney I Wiener, and Francesco P Battaglia. Replay of rule-learning related neural patterns the prefrontal cortex during sleep. Nature neuroscience, 12(7):919-92 2009.
[53] David J Foster and Matthew A Wilson, Reverse replay of behavioural sequences in hippocam- pal place cells during the awake state. Nature, 440(7084))680-6633 2006.
[54] Sean PA Drummond, Gregory G Brown, Christian Gillin, John L Stricker, Eric C Wong. and Richard B Buxton. Altered brain response to verbal learning following sleep deprivation. Nature, 40366770))55--557, 2000.
[55] Seung-Schik Yoo, Peter T Hu, Ninad Gujar, Ferene Jolesz, and Matthew P Walker. deficit in the ability to form new human memories without sleep. Nature neuroscience, 10(3): 385-392, 2007.
[56] Scott Terry. Learning and memory: Basic principles, processes, and procedures. Routledge, 2017.
[57] Hideyuki Okano, Tomoo Hirano, and Evan Balaban. Learning and memory. Proceedings of the National Academy of Sciences, 12433:12403-12244, 2000.
[58] Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It's all connected: journey through test-time memorization, attentional bias, retention, and online optimization. arXiv preprint arXiv:2504.13177 13173, 2025.
[59] Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners. arXiv preprint 24t.:2407.14277 2024.
[60] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mas:: Fast autoregressive transformers with linear attention. In International conference on muchine learning, pages 1156-5165. PMLR, 2020.
[61] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprini arXiv:2307.000000 08621, 2023.
[62] Juergen Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. accepted for publication in. Neural Computation, 1992.
[63] Imanol Schlag, Kazuki Irie, and Juergen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pages PMLR, 2021.
[64] DL Prados and SC Kak. Neural network capacity using delta rule. Electronics Letters, 25(3): 197-199, 1989.
[65] Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.046200 2024.
[66] Nicholas J Higham. Functions of matrices: theory and computation. SIAM, 2008
[67] Jiaxi Hu, Yongqi Pan, Jusen Du, Disen Lan, Xiaqiang Tang, Qingsong Wen, Yuxuan Liang, and Weigao Sun. Improving bilinear rnn with closed-loop control. The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
14

---

[68] Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, and Vahab Mirrokni. Atlas: Learning to optimally memorize the context at test time. arXiv preprint urXiv:2505.237755 2025.
[69] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL
https://openreviee
[70] Denis Paperno, German Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring broad discourse context. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525-15344 Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 18633/v1//16-11111 1144. URL https: http://caaaatttogo..o/ooo org/p16- 1144/.
[71] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[72] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pages 4791 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18.18653/ 10/165//v//P1-1111 1/722-147.. URL https: / haaaaacaaannoooooo org/P19-1472/.
[73] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
[74] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:/803.00 1803.05457 2018.
[75] Maarten Sap, Hannah Rashkin, Derek Chen. Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP), pages 4463-4473 Hong Kong, China, November URL https:
2019. Association for Computational Linguistics. doi: 10.11853/1//D1111. // /aaclanhholoyy.... org//19-
[76] Matteo Tiezzi, Michele Casoni. Alessandro Betti, Tommaso Guidi, Marco Gori, and Stefano Melacci. On the resurgence of recurrent models for long sequences: Survey and research opportunities in the transformer era. arXiv preprint arXivv2402.001132, 2024.
[77] Bo Peng, Eric Alcaide, Quentin Gregory Anthony. Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Nguyen Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan S. Wind, Stanistaw Wozniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL https:/ / http///ppeerrreeeee//e/ee/eeeeee wwwpareviee.nee/ffooooooe wit.forrnndd7Saaaaaaaaapp
[78] Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023, URL https:// /ttp://///ppereieeeeee wwwpenrvweeweeeeeeeeeeeeeeeeeeee fttt://formmiddAAaeee Aaw.dkK
[79] Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In The Eleventh International Con- ference on Learning Representations, 2023. URL https:/ http://ppppreeeeeeeeeeeeee net/ fortmfii g40TKRKfS7R.
15

---

[80] Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint arXiv:2403. 199888 2024.
[81] Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. Eagle and finch: Rwkv with matrix-valuee -vauud states and dynamic recurrence. arXiv preprint arXiv:2404.0555 000902 2024.
[82] Bo Peng. Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, et al. Rwkv-7" goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503. 14456, 2025.
[83] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in Neural Information Processing Systems, 37:115491-11552, 2024.
[84] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint urXiv:24/2.06444 2024.
[85] Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. Deltaproduct: Increasing the expressivity of deltanet through products of householders. arXiv preprint arXiv:2502. 10297, 2025.
[86] John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2555 1982.
[87] Juergen Schmidhuber. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In ICANN'93: Proceedings of the International Conference Artificial Neural Networks Amsterdam, The Netherlands 13-16 September 1993
3. pages 460-463. Springer, 1993.
[88] Donald Olding Hebb. The organization of behavior: A neuropsychological theory. Psychology press, 2005.
[89] Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 1, page 397. NIH Public Access, 2017.
[90] Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalcarned neural memory. Advances in Neural Information Processing Systems, 32, 2019.
[91] Kazuki Irie, Imanol Schlag, Robert Csordas, and Juergen Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers. Advances in neural information processing systems, 34:7703-7717, 2021.
[92] Ke Alexander Wang, Jiaxin Shi, and Emily B Fox. Test-time regression: a unifying framework for designing sequence models with associative memory. arXiv preprint arXiv:2501. 2025.
[93] Kazuki Irie, Robert Csordas, and Juergen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In International Conference on Machine Learning, pages 9639-9659. PMLR, 2022.
[94] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Juergen Schmidhuber. A modern self- referential weight matrix that learns to modify itself. In International Conference on Machine Learning, pages 9660-9677. PMLR, 2022.
[95] Dzmitry Bahdanau. Neural machine translation by jointly learning align and translate. arXiv preprint 140.v:1409.044 2014.
[96] Yufeng Zhang. Boyi Liu, Qi Cai, Lingxiao Wang, and Zhaoran Wang. An analysis of attention via the lens of exchangeability and latent variable models. arXiv preprint arXiv:2212. 14852, 2022.
[97] Jianqing Fan. Local polynomial modelling and its applications: monographs on statistics and applied probability 66. Routledge, 2018.
16

---

[98] Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, al. Minimax-O1: Scaling foundation models with lightning attention. arXiv preprint arXiv:250.083113 2025.
[99] Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. Poly SketehFormer: Fast transformers via sketching polynomial kernels. In Ruslan Salakhutdinov, 7ico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 22748-22770 PMLR, 21-27 Jul 2024. URL https: aetess/v3330 / //proceedings. mlr. prwpsppeeeeppp.e v235 22a55kkacha224. htmll. html.
[100] Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re. Simple linear attention language models balance the recall- throughput tradeoff. In Forty-first International Conference on Machine Learning, 2024. URL net/
https://openrevie www.eeew.ee/fruureeeeee http///oronnndddffd00
[101] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prud- nikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter, xlstm: Extended long short-term memory. arXiv preprint MxXxv:2400.04517 2024.
[102] Kazuki Irie, Francesco Faccio, and Jürgen Schmidhuber. Neural differential equations for learning to program neural nets through continuous learning rules. Advances in Neural Information Processing Systems, 35:38614-38628 2022.
[103] Johannes Von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vla- dymyrov, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.0558 2023.
[104] Yifan Zhang, Zhen Qin, and Quanquan Gu. Higher-order linear attention. arXiv preprint arXiv:2510.27258, 2025.
[105] Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of mathemati- cal biology, 15(3):267-273, 1982.
[106] Kazuki Irie and Samuel Gershman. Fast weight programming and linear transformers: from machine learning to neurobiology. arXiv preprint arXiv:2508.08433. 2025.
[107] Seppo Linnainmaa. The representation of the cumulative rounding error of an algorithm as Taylor expansion of the local rounding errors. PhD thesis, Master's Thesis (in Finnish), Univ. Helsinki, 1970.
[108] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of muchine learning research, 12(7), 2011.
[109] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview mini-batch gradient descent. Cited on, 14(8):2, 2012.
[110] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadeneshelli and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. International conference on muchine learning, pages 560-569. PMLR, 2018.
[111] Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
[112] Sashank Reddi, Ahmed Hefny, Suvrit Sra, Bamabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. International conference on machine learning, pages 314-323, PMLR, 2016.
[113] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020. URL https: https:://pperreiiieeeeeeee/ wttprrrviieeeeeeffrrrree net/forum? id=rkgzZaEKDr.
17

---

[114] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, Yifeng Lu, al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36:49205-49233, 2023.
[115] Jürgen Schmidhuber. Gödel machines: self-referential universal problem solvers making provably optimal self-imprrvvmmnns. arXiv preprint cs/0309048, 2003.
[116] Jürgen Schmidhuber. A 'self-referential weiitt matrix. In International conference on artificial neural networks, pages 456-450. Springer, 1993.
[117] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2924 2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 101186//11//11113000 URL httptss///cclaaatoooooggg wwwelogy..oo///// org/N19- 1300/.
[118] Bo Peng. Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, et al. Rwkv-7" goose" with expressive dynamic state evolution. arXiv preprint WxXiv22503.14555 2025.
[119] Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint 2tti:2200600522 2024.
[120] Guilherme Penedo, Hynek Kydliček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:3081 17:30811-30884 1-30849, 2024.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction both accurately state the main theoretical contri- butions of the paper and accurately discuss our experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in the main text, including its scope, assumptions, and potential areas for future improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: All assumptions are stated in the body of the theorems directly. As much of the proofs as possible is provided in the main paper text. We provided all detailed proofs of lemmas and theorems introducing this work.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification: In the paper, we provide all the details need to produce the results, including the details of the implementation. All the datasets used in this paper are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp cc/
Moollic/guides/CoeeSuuue wtt.ssammmiiiiooiiooo.o.o.....ce for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. http/uulccggguueeee httpc/gggiideeoeeeuuubbiiiiiiiii asiinnoliccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yee Yes]
Justification: Detailed descriptions of the experiment and the associated code are provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: To account for variability and support statistical validity, we report the results computed over multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have used TPUv5 to perform all the experiments in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttppp::///urrp.... wwteriis..ccppppiee wttpssccccpppcccccccceeeeccccctt htcc///bbbl//chhhtcciiieeeeeeeee
Answer: [Yes]
Justification: There are no violations of the NeurIPS Code of Ethics in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provided a discussion in the Appendix about the impact of the work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit all external assets and explicitly acknowledge their licenses and terms of use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo////oo/// datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any user studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No usage of LLMs in the methodology or experiments.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //teur/is....eee ceaurrips.cc/Cofeeeeeeeee /c/Cooferrences/2255111 for what should or should be described.
25

---

A Additional Related Work
In this section, we provide more comprehensive review of literature that are connected our work:
Modern Deep Learning Architectures. Recent research has focused on developing efficient recurrent alternatives to Transformers to mitigate their quadratic computational cost and limitations in modeling long contexts, primarily driven by the faster inference and training capabilities of recurrent architectures [76]. Initial models, including RetNet [61], RWKV [77]. and S5 [78], employed data-independent transition matrices with Hebbian-like update rules. Subsequent approaches began integrating input-dependent parameters into these linear architectures (e.g. SSMs [79, 80J. RWK [81]) or adopted more expressive memory updating mechanisms based on the delta rule [59, 63, 82- 84]. More recent advancements have extended these memory architectures to deeper models, utilizing delta-rule-like [65] or momentum-basee [28] update rules. To further improve delta-rule-based sequence models, Siems et al. [85] proposed employing multiple gradient descent updates per token, leading to enhanced expressiveness in state tracking tasks.
Fast Weight Programs. The conceptualization of recurrence as key-value associative memory systems originates from Hopfield networks [86] and fast weight programmers, where dynamic fast programs function as writable memory within recurrent neural networks [62, 63, 87]. Among the learning paradigms for these systems, the Hebbian [88] and delta [64] rules are prominent and have been extensively investigated in the literature [62. 63, 83, 84, 89-91]. Also, recently, the duality of meta-learning perspective and the architecture has been in the core of attention of several studies [58, 63, 92, 93]. Furthermore, our HOPE, can be seen as the generalization of SRWM [94], where instead of matrix valued memory, HOPE uses a neural networks and also its update rule is generalized using: (1) a data-dependent gating mechanism that helps to forget the past data; (2) We update the memory using gradient descent with momentum, where SRWM is equivalent to update the memory using a simple gradient descent. Also, while SRWM is not parallelizable, we present a parallelizable training algorithm for our HOPE.
B
Reformulating Modern Architectures as Learning modules
Modern sequence models such as Transformers [27] and recurrent models [28, 60, 63, 65] are the backbones of recent advances in language models. Recently, the equivalency of such models with associative memories that aim to learn a mapping from keys to values from data have been studied in different settings and objectives [58, 59, 65, 92]. Particularly,
focus the general framework of Miras [58], which defines associative memory as Definition and optimizes the internal objective (called "attentional bias") with choice of optimization algorithm
arbitrary class of functions (i.e., memory architecture). While this formulation alone indicates that the well-known architectures are instances of nested systems of associative memory, next, we review this equivalency for some learning rules and architectures.
From now on, we assume that keys { values (u. and queries (4 are given: they often are defined as the projections of the input, i.e.,
k; xz k.
V.- xz W..
q, W.
(32)
In this design, since projection parameters (i.e., WR. We, and W₂) are optimized in a lower frequency level, the sequence model component (e.g., self-attention) has higher frequency and so the learning process of the associative memory happens in a lower level. Accordingly, for the sake of clarity, we only discuss the higher frequency level (i.e., the internal learning process of the associative memory).
Softmax Attention. From the associative memory viewpoint: given keys (ke) values {or)-,, and queries (g.)/11 Sof tmax attention block [27, 95] can be reformulated as non-parametric solution to the (2()) regression objective with Nadaraya-Wart Watson estimators [96, 97]:
M arg min M
Σ
L s(k,, Al M
Σー
Σ 1=1
L
s s(k q)
Vi,
(33)
ビ skk, .q)
where I. is the sequence length [65]. This formulation optimizes the memory M(-) with respect to the entire context; however, one design choice can be to limit the optimization process to the past
26

---

tokens, resulting in:
s(ki, q)
M arg min
Σ s(k, -,) M1 i=t-ct1
i=t-ct1 2jjt s(kj, Vi
Σ
(34)
which is equivalent to the sliding window attention (SWA). Therefore, attention and its more ex- pressive variants [92] also are instances of Definition 1, when instead of gradient descent or other parametric methods, we find the optimal non-parametric solution to the mapping.
RNNs with Hebbian Rule. The first generation of modern recurrent architectures (c.g.. Linear attention [60]. RetNet [61], RWKV [77], lightening attention [98]) are based on Hebbian-like learning rules [88]. For this class of models, the inner objective measure the quality of mapping between keys and values is the dot- dot-product similarity. That is, given a matrix-valus
memory M RRdx, keys and values k, E Rd objective L(M; k2, vı) == 2(M,, vz), and a kernel o(-), we optimize the equivalent associative memory optimization problem (see Definition 1) with gradient descent and weight decay, resulting in:
aa==aMM---- V. (MI v.) a. + v. (k:])
(35)
which recovers the original linear attention recurrence [60]. Given different settings for a, (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also (6-) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule [60, 61, 81, 99-101]. Therefore, the variants of linear attention with Hebbian rule can be reformulated as the process of an optimization problem, which the memory aims learn the mapping between keys and values based on dot-product similarity objective, with gradient descent.
RNNs with Delta Rule. To improve the memory management and to enhance the memory capacity of the above group, several studies suggest replacing Hebbian rule with Delta rule the learning algorithm in recurrent neural networks [63], resulting in models such as DeltaNet [63], Longhorn [59], and RWKV7 [82]. When letting M E Rdxn, delta rule is equivalent to optimizing MSE objective f. = MMMkk vellž with Ret, (MM, ||Me Mr-1llf. as local retention, and stochastic gradient descent as the optimizer:
Mr M Tt V M E & (ke), = -II-=(II mikkk, M + v.kk
(36)
=IIM, M M optimization algorithms with weight decay (e.g., regularizing with || for given g > 0), multiple steps of gradient descent, and/or different formulations of learnable parameters such as and can result in diverse variants of delta rule [58, 59, 65, 67, 82, 85, 91, 92]. Therefore, Delta rule and its variants are all instances of an optimization problem, in which the model aims to learn mapping between keys and values based on the regressesoonn objective.
MM, Using other forms of retention gates (e.g., (M,, =
Omega, Oja's, and Non-Euclidean Learning Rules. More recently. there have been growing interests in designing architectures from the associative memory perspective (see Definition 1) and use more complex internal objectives, and/or optimization algorithms, resulting in learning algorithms beyond Delta and Hebbian rules [58, 68, 102-104). More specifically, to enhance the stability of Hebbian rule (discussed in Equation 35), Irie et al. [102] introduced OjaNet based on Oja's rule [105] with the following recurrence:
Beyond Conventional Learning Rules:
Mr = a, + The v. (o(k.)) MT_1ve).
(37)
In the associative memory formulation (as in Definition 1), this recurrence can simply be reformulated as one step of gradient descent as:
M, M The V Ma-1 E ((
(38)
M_
27

---

where [(M;k, vy) = -2(MK, vz) + IIMMT vell and o(-) is a kernel [102, 106]. Although this design enhances the Hebbian learning rule by enforcing a unit-norm constraint for the single-neuron. it has been reported to empirically underperform models based on Delta learning rule [102]. To further enhance the Delta rule through the design of more expressive objectives, recently, Behrouz et al. [58] suggested going beyond Euclidean spaces and use Lp = || norm for the internal regression objective, showing better empirical performance and robustness in long context tasks compared to Delta rule and its variants.
While the majority of learning rules are online update mechanisms Becchnnsms-mmeanng meaning that at each state, the models only need to keep the memory and the current (batch of) input-Omega rule [68] suggest an update rule based on set of past (batches of) inputs (or all inputs). More specifically, given memory M with an arbitrary structure, keys and values k2 Rd an arbitrary objective L(M; ky, vz). and kernel (()), Omega rule is defined as:
M₁ = ai a Mi-1-
Σ ne E(Me: ((ks), v.), i=1-c+1
(39)
where с 2 1 is the local window of cached inputs. Note that in the special case of Yt, and c equal the entire context length, the optimal solution of the above design collapses into an online case, where the update rule only depends on the current state and the current input [103]. For further discussion with more details about representing architectures as associative memories and so an optimization problem, we refer the reader to Behrouz et al. [58].
Note on Gating in Modern Sequence Models. One the recent architectural changes in modern language models is the gating of linear layer's output with the output of the sequence model. Despite significant improvement resulted by this method, it is still unclear that how it enhances the performance. The main difference between feedforward network and modern recurrent memory modules (e.g., linear attention [60] or deep memory modules [28]) when their initial state of the memory is meta-learned, is the second level in memory modules that perform in-context learning and adapt its state with the context. From this viewpoint, when the initial value of the memory is not meta-learned, it only relies on the in-context adaption of the memory and so there is no persistent memory system that stores the knowledge of pre-training in this block. Therefore, when the initial value of memory is not meta-learned, which common earlier variants of linear transformers, the gating of linear attention acts as a persistent memory and the initialization of the memory module.
C Details on initialization of CMS
As discussed earlier, different levels might have different process of knowledge transfer. Accordingly, while the above formulation suggests a spectrum of memory systems in different levels and so with different frequencies, their connections can vary based on the design. In the following, we diseuss some potential variants:
Nested Continuum Memory Systems. The first variant is a fully nested continuum memory system, in which the initial state of the MIP block in level s + 1 is meta-learned in level s. This design allows for higher-order in-context learning ability, where each of the levels has its own context flow and re-initialized after the end of the context. More specifically, given arbitrary 1 S S k.
@ff.11) arg min Eracts ((O,T:
(40)
where c(*) is the context length of the MLP block in s-th level. Following this design, the end of the optimization process of each block (i.e., after [C(*] /C({+1)( steps.) the value of the memory will be re-initialized to gff++1), Note that the update mechanism of each block in its own level remain unchanged (i.e.. Equation 31).
Sequential Continuum Memory Systems. In the second variant, the MLP blocks are located sequentially (i.e., the output of the MLP block in level is the input for the MI.P block in level + 1) and also the initial state of MLP blocks are all connected through backpropagation in the lowest
28

---

In this section, we start with viewing backpropagation process and optimizing a neural network from the associative memory and data compression perspective. Then, we discuss how variants such as frequency level. Given an arbitrary 1 s s k.
f-f = arg min B ((,,T:0)
mig
(41)
where C() is the context length of the MLP block in the lowest frequency level. Since the initial state of all memories are meta-learned in the lowest frequency, the most persistent knowledge of all components is the compression of the same context flow.
Independent (Head-wise) Continuum Memory Systems. In this variant, we keep the knowledge transfer process in Equation 41, but change the output computation in Equation 30. While the previous formulation designs the memory system as a sequence of blocks, and so making their input/out dependent to each other, this variant uses independent blocks with different context length and then combine them using an aggregation process:
y Agg (MLP(F) (a:), (x))
(LLP)(((a)))
(42)
The above Agg (.) is an arbitrary function that aggregates all the inputs to compute the output. For example, one straightforward and simple design choice is to use a learnable weighted sum of the input.
CMS Design Helps with Continual Learning. Based the design of CMS, fair question is to ask: Why and how CMS can help with longer context length and generally continual learning. Here, we provide simple answer to this question: Viewing MLP blocks in CMS the storage of model's knowledge catastrophic forgetting can happen when we update a block and as its result, the old knowledge stored in its parameters
forgotten. In CMS design, however, when updating an arbitrary block of (.P(f.).)) for some 1 S S k, the potentially forgotten knowledge from MLP(-)(-) is still stored in other components such MLP((.)) (.), where s' < Also, in this case (i.e.. the knowledge is already forgotten from MLP(F))(.) but it is still in MLP((.))() for s' < s) the knowledge transfer through backpropagation (for their initial state) can circle back the knowledge to MLP(f)) ()). resulting in a loop through time dimension, and so hardly forgetting important knowledge.
Is CMS Efficient Enough? common concern when updating the parameters of model in a continual manner is its efficiency. Therefore, fair question is to ask if CMS causes significant computational overhead for the model. To answer this question, let recall from Appendix that modern recurrent neural networks are also continually updating a subset of their parameters (i.e., their memory state). These parameter updates, however, take advantage of sequence parallelization as well as updating only small number of parameters. To this end, for CMS, we highlight two points:
In the CMS design, at each time, updates are restricted to blocks approaching their scheduled update time (based on their frequency). As a simple example, consider a Transformers but with replacing its MLP blocks with CMS. Let the model have Laaer layers, 4 levels of MLP blocks CMS with highest frequency of f. and hidden dimension of din On average, the di of parameters, which consists of only small number
update cost is for O (} Liner
of parameters at each time.
The update mechanism of Equation 31, not only helps with the enhancing the persistent memory of the model, but it also unlocks the sequence parallelization for higher frequency levels. More specifically, for input x, when * O (mod ((()) there is no sequential process inside the chunk and all the computations for tokens correspond different values of # O (mod ((O) can be done in parallel. The details of such training algorithm is the same as the training procedure in Behrouz et al. [28]. Sun et al. [65].
Therefore, in summary, CMS can be fast in practice, mainly due to the fact that it updates only small number of parameters each time, and also its design unlocks sequence parallelization.
D Reformulating Modern Optimizers Instances of NL
29

---

momentum -aarendtu----dddddd optimizers are instances of nested associative memory systems. Finally, we discuss alternative methods leading to deep optimizers with higher expressive power from the associative memory perspective.
D.1 Backpropagation as an Associative Memory
Updating the weights of neural network through backpropagation [29, 107] has been the critical component of training large-scale deep neural networks. Intuitively, in this optimization process, first, the error of the model's output with respect to target is calculated, and then each layer is updated based on its contribution to this eπror. This section aims to explain this process through the lens of associative memory and discuss how it fits within the nested learning paradigm. For the sake of clarity and simplicity, we assume deep MLP model, but all the derived formulations in the following can simply be adapted to other architectures as well. Given an MLP with I. layers parameterized with (We + be) the required gradients in backpropagation are computed as:
OC
T de Āg-1
and ò: Joe (zt) (We (W2+15e+1)
(43)
a
local output surprise for layer
where 2ς We 1 + by is pre-activation, and so ãt de (ze) is the output of f-th layer, Φe(.) is its non-linearity, and J de (-) is the Jacobian. Therefore, the update of the L-th layer with gradient descent is computed as:
We = We, - 74.. 8g a; #
(44)
Here, -t-1 1 is the input of the layer and 5e measures the local error signal for layer & or equivalently is a metric that measures the surprise of layer l's output given its input. Similar to our example in Section 2.1, write Equation 44 as:
1
We We arg min (W 52) + W
IIW
11,
(45)
276.+1
which is an associative memory module that aim to map the input of each layer tt-1 its local error signal, 8, (see Defimition 1). That is, the above formulation implies that the training process of neural network with gradient descent and backpropagation can be viewed as a compression process, in which each layer stores the mappings between its input and the corresponding local error signal.
D.2 Momentum-basse Optimizers as Associative Memories
Momentum-based optimizers are the major components of modern machine learning models' train- ing [33, 34, 108]. To explain momentun-based optimizers as associative memories, let us start from simple gradient descent algorithm:
W = W₁ - 7)c Vw, C(w;; x.+1),
(46)
which updates the current state of the weights based on the momentary gradient (surprise). This update rule does not incorporate the previous tokens and also the loss landscape that have been traversed so far, resulting slower (or less robust) convergence in many scenarios. To this, momentum- based gradient descent methods incorporate an Exponential Moving Averages (EMAs) of past gradients:
Wen = Wi, + m&
(47) where matrix (or vector) m. is the momentum at state t and a and 7)t are (adaptive) learning and momentum rates, respectively, and ò; and Φ;-1 are defined the same in Equation 43. Similar to Equation 45 and one of the examples in Section 2.1, assuming a++1 = 1, the momentum term can be viewed as the result of optimizing the following objective with gradient descent:
m = a1,4+1m - 7 Vw, C (W,, (141)) = am,1-1m - M1,+1 de 2z
min (m क; dr)).
(48)
The case of 0++1 # 1 is equivalent to GD on the above minimization plus an eeruglaario on the momentum term. Thus, momentum can indeed be viewed as associative memory module
30

---

that learns how to compress the past gradients of the objective into its parameters. Contrary to Equation 45, which was a simple 1-level associative memory and the update was directly applied to the memory, here the state of the momentum determines the update for the weights. In other words, it a 2-level optimization procedure, which the inner-loop learns the momentum and the outer-loop uses the state of the momentum to update the weights.
From this perspective, we can generalize the definition of momentum from EMAs to any arbitrary associative memory module that aims to compress the past gradients or maps the input of each token to its corresponding local error. This generalized momentum can be expressed as:
Wt,, Wt, + M
(49)
(50)
where my is the solution of the following associative memory, optimized by gradient descent:
min È (m; (6-))-))))
(51)
Here, the objective (()) is different from the original objective of the problem at hand, and Z() the objective that defines the momentum and measures the quality of its mappings. fact, the momentum term in this formulation aims to adapt in-context (recall that the context of the momentum is the gradients) to the local error rates based on the input of the layer. Most popular optimizers are formulated as element- wise update rule (for computational efficiency reasons) and so we first explore the element- associative memory formulation of momentum and connect to popular optimizers such Adam [33]. Showing that Adam can be viewed as the optimal associative memory to the Ly-regression objective that aims to predict the variance of gradients, we discuss other similar algorithms such R.SPProp [109], SignSGD and its momentum- -assdd variants [1 [1]0]. NAdam [11 1],
AMSGrad [1 12), RAdam [113], 3. and Lion [114] are also instances of an associative memory that aims to compress the gradients. We then go beyond element-wiss formulation and show that AdaGrad [108] is also an associative memory module. Due to the connection of AdaGraa with optimizers such as Shampoo [35] and Soap [36]-i.e., as the approximation of the preconditioning term-we then conclude that all these optimizers can be re-formulated as associative memory. Next, we discuss another class of optimizers based on preconditioning and reformulate them from NL's perspective in more details:
Preconditioning and Approximation of Hessian. Another class of algorithms is preconditioning algorithms where the idea is to approximate Hessian inverse to mimic the behavior of Newton's s algorithm. Formally, gradient descent with preconditioning defined as:
W We, - T-- P 9
(52)
where preconditioner Pi+1 is often a positive-definit matrix. A critical interpretation of precon- ditioner is their role in performing gradient descent in a transformed coordinate system, which can be viewed as a mapping from gradients to that system of interest. Accordingly, we reformulate and interpret the preconditioner in Equation 52 as associative memory that maps the set of gradients (or a function of gradients denoted as g) to the system of our choice, denoted as g:
We,1 = We, 71+1
P ++1 (
(53)
where internally (in a nested level), P++1 learns how to perform this mapping using an objective:
min P
(P (g):
(54)
Given this viewpoint, the main question is about finding the best coordinate system that can empower the compression process. The most simple variant an identity mapping, where we preserve the metric system and use P to map g (i.e., gradients in this case) to itself, resulting in preconditioning terms in Adam [33] and AdaGrad [108]. These results, along with the representation of Adam and its variants as associative memories, show that not only momentum-based optimizers are associative memories, but they also can be decomposed into a set of nested learning problems, each of which optimized with gradient descent. In more general form, however, can more nested levels and optimize the inner problems in Equation 54 with gradient descent, resulting in:
P P G41 V E (Pi: 9:41 1: 1
(55)
31

---

In the NL framework, to design an effective preconditioning, one needs to find the right choice of 9 and C. This viewpoint can also lead to other classes of algorithms with gradient/momentum orthogonalization: e.g., Muon and its variants [34]. Recalling Muon optimizer [34]:
W 1441 We, NewtonSchulzz (me,.1)
m = 02, me, 1Je,++
(56)
where NewtonSchullz performs steps of Newton- Schuulz orthogonalization process. From the above discussion about the general formulation of preconditioning, one NewtonSchulz, (-) operator as a mapping from gradients of momentum term to a proper metric system. The choice of proper coordinate system in Muon is orthogonalize the gradients and we aim find mapping P (-) by minimizing loss function minp C(P; O, m) where objective L(:s measures the quality of mapping from O to either m or by P(-). A critical challenge this process that the parameter O itself is not given and so the mapping requires learning both the mapping and the proper orthogonal space. A simple formulation measuring orthogonalization, can be achieved by defining the objective as:
((P(g):)) = [P(g) P(g) 1
(57)
where P(g) is the orthogonal space that we aim to directly learn from gradients. This objective ensures that the gradients (or momentum) and their mapping are relatively close while the mapping is to an orthogonal space. Optimizing the above objective find O P(g) with one step of gradient descent results in:
0++1 = 4 Vo. ((O:g.)) = 11+1 (0, 9: 20, (OTO.-1)
(58)
which recovers the 3-degree polynomial (initial value Og 9.). In summary, the higher-frequeney level learns the orthogonal mapping and then the lower-frequency process use the learned mapping to optimize the weights.
E Details on Hope Architecture
A general formulation for the associative mesory-based blocks is to project the data into keys, values, and queries and learns how to map keys to values and how to retrieve from the mapping based on queries. More formally, for a parametric associative memory, let æ Rd for forl- = 1..... L be the input, we have:
k WK
Ut = W
1) = 2 W', with an optimization algorithm
q:
xt Wqp g.
a Bt Wa.
(59)
min L(M:kg vz),
(60)
V. Mig.
(61)
For the sake of clarity, we use red (resp. blue) to highlight computations/weight in the upper level (resp. lower level). We can add new level for each of WR, W., Wq. W,, and Wa. and allow them to be updated in-context. For the sake of efficiency, a simple version is to share the values for all the components in the nested system of associative memories:
k. -MRA-1 (xt+,
V: Mu,t- (1:)),
q. == =M= (are),
T) MM (are).
ae = Ma,e-1 (Iz),
(62)
min ((M:; De, vz),
with an optimization algorithm,
O E (k, v. (,,,,,,,,,)
(63)
Mu
min [(Mmom: kr, vz),
with an optimization algorithm,
(64)
y. Mmmmm (q:)
(65)
where the initial states of all memories, i.e., MM for any O E {k, U. q,77 a. memory) meta- learned across all sequences/contexts. As discussed earlier, the meta-learning of the initial states of memories is essential for both fast-adaption, training stability, robustness noise in the data.
This design provides a fully adaptive memory, where all the components can adapt themselves in-context. It, however, (1) still lacks self-modification, where the model in response to new data
32

---

changes its own parameters or learning process [115]: (2) has suboptimal design as it shares of keys and values for all the memories. In continual learning, where the model requires consistent weigh//knoleegge update in response to new data, is critical for the model to not solely rely on data, and instead learns how to modify itself when is needed. Motivated by the above points, and inspired by the self-modifying mechanisms that generate their own values based on the context [94, 115, 1 116), we present self-modifying deep associative memory where the models generate their own values:
D1 = Mmemoor,t-11 (qt),
k; = Ma.t-1 (x)),
0t = Ma (2+))
T/ = M (x:)
0₁ = Mo,c-1 (2t))
(66)
+ Ma, (1t)
(Generating its own values for each memory)
(67)
min C (Mai Ke. 00.1)
with an optimization algorithm,
D {/, v, q, 73 a, memory),
Ma
(68)
where q, 2, W is the only non-adaptive projection, 7/: is the learning rate in optimization process, and ar is the retention gate (forget gate or weight decay) in the optimization process. Note that, again, the initial states of all memories, i.e., Mn,o for any O E (k, v. q. 17, at memory] } are meta-learned across all sequences/contexts, and so are optimized in the higher levels (or outer-loop).
Learning the mappings for associative memory modules (see Equation 68) requires a choice of optimization algorithm as well as an objective L that measures the quality of mappings. A simple and common choice for objective and optimization process are T-zregression loss, and gradient descent algorithm. As for the objective, we use I.g-regression loss, i.e., ((M; k, v) = [l.M(k) - =1ß.- As discussed earlier, the choice of optimizer highly depends on the context of optimization. For example, gradient descent from associative memory perspective is based on dot-produc similarity and so the update at each step, is solely based on the input and does not incorporate the previous data samples to the update. When performing optimization in the token space, however, we know tokens are highly correlated. Therefore, we use our DGD with weight decay, resulting in general update rule of:
y: = Mmomoor,.t 1-1 (qt), a, = Ma,e-1 (xt),
k; -M (xt),
Ur- = MM (2))
TT
M
(xt).
(69)
t -Ma,r-1 (vt),
(Generating its own values for each memory)
(70)
Mo. MM (arI - nkikk
TI V. (Mo,-i;k,,
(71)
0 E (k, v, q. a, memory).
Here, the architecture of the memories are arbitrary and even we are not forced to use the same architecture for all components. We use a 2-layer MLP block the architecture of all the memories:
Ma(+) = (-) (W)())
(72)
E.1 Fast and Parallelizable Training
In the above, we discussed how to design model that can learn to generate its own latent values and so modify itself. The main challenge from the practical point of view is the efficiency of the method and if its training is parallelizable. We follow the chunk-wise training algorithm of non-linear update rules [28, 65] and use update frequency of fa = CD no limitation to use different chunk-sizes, in our experiments, we use two different value of chunk sizes, one for the update of Mmemory() (-) and the other for all the other memories in the self seff-referentaal Titans.
L. where L. is the context length. While there is
In more details, given an input sequence (at) and chunk size 1 S C < T., we split the sequence into [&] chunks of (=(((-1))++) )C_ for i = 1,..., [6],
and then generate all elements in Equa- tion 69 the end of each chunk for the next chunk. This allows for generating all the elements for the entire chunk in parallel, before starting the computation for this chunk. Furthermore, to update the memory modules based on Equation 71, we take the gradient with respect to the last state of the previous chunk. Again, this allows for computing all the gradients for the next chunk in parallel. In more details, given this chunk-win updating procedure, the update rule for the self-referential Titans is computed as:
33

---

y, M Mmemony, CxI1 (gt); k = 1 (ax), vt Mo,ox( (ax))
T M, b1 (xt),
ar a= ===Mo,cx(d] (zt),
(73)
on. - -M..x (0t),
(Generating its own values for each memory)
Ma. Mo.-- a mikikit) -1 WLM (Mr.cxt8)s Ke ot.)) ,r,
(74)
DE {k, v, 0, a, memory).
Here, the architecture of the memories are arbitrary and even we are not forced to use the same architecture for all components. We use 2-layer MLP block as the architecture of all the memories:
Mo(+) (.) (o.,o(WD.2())).
(75)
Since all the gradients as well as new keys, values, learning-rates, and weight decays can be computed in parallel before starting the processing of the current chunk, the above updates accepts the fast parallclizable dual form that is discussed by Sun et al. [65] and Behrouz et al. [28]. To better illustrate the above update rule for self-referential Titans, let derive the recurrent formula for the simplest case of valued memory. We derive the recurrent form for two different objectives:
Dot not-prrducct similarity C(M; k. v) = - (Mi, 0): Given this objective and linear memory, the gradient is calculated as ok which results in update rule of:
Mr Mn (arI m)kkk? 77 m
(76)
DE {k, u. 1) memory]
In-regression loss: Given this objective and linear memory, the gradient calculated as (Mk - vv)k which results in update rule of:
Mn. MM,-- (atI nokek ne (Ma,cxisike - - kT,
(77)
0 E (k, v, 9, a, memory).
E.2 Hope Neural Learning Module
In the previous sections, we first discussed Continuum Memory System (CMS) that allows for more persistent storage of memories and defines memory as a spectrum of blocks with different frequencies of update. Due the larger capacity and constraints for scaling the parameters, often CMS requires simple learning rule but higher capacity to store more persistent knowledge. On the other hand, in the previous section, we discussed the design of self-modifying Titans, where it can generate its own keys and so learning update to better adapt to the context. Contrary to CMS, the self- self-modifying Titans has a small capacity but is using complex and expressive learning rule. Accordingly, these two systems seem to be complementary and their combination can enhance the model expressiveness from different aspects.
To this end, we present HOPE architecture: A neural learning module that incorporates self-modifying Titans followed by Continuum Memory System. The HOPE design is illustrated in Figure Formally, let x E Fd for t = 1,....L L be the input, the HOPE forward pass is defined as (we remove the normalization and convolution layers for the sake of clarity):
o, = Mmemory, (q.),
k = -MMM-1111 (zz).
v. = (MMe--1(E)).
(78)
7) Mr,t- (xx),
as Ma.- (xx),
(79)
0n, = Mn, (v.),
(80)
Mn.- = -M aa - mkkkk? - C.M.n (Mo,-11; (n,))
(81)
O [k, v. q. 17, memory).
= MLP( (pp(ff((MM((--))-(-.. -MM
(82)
where the block's output for token t is y,- In our experiments, also normalize q and ls with L2 normalization and also use local convolutions with window size of 4.
Hope-Attention. We also use another variant of HOPE, which we simply replace the self- moddfyinii Titans with softmax global attention [27].
34

---

Table 2: Architectural Details.
Model
Block
Dim
Head
Peak LR
Token
170M
12
768
16
3e-3
15B
340M
24
1024
16
1.5e-3
15B
760M
24
1536
16
1.25e-3
30B
1.3B
18
2048
7c-4
100B
Experimental Setups
To study HOPE backbone of language model and evaluate common language modeling and common-sense reasoning tasks with the setup of:
Datasets: We evaluate HOPE and baselines on Wikitext [69], LMB [70], PIQA [71], Hel-
laSwag [72], WinoGrande [73], ARC-easy (ARC-e) and ARC-challenge (ARC-c) [74].
SIQA [75], and BoolQ [117] benchmarks.
Baselines: As for the baselines, of the models that are purely based on Hebbian- or Delta-rule, and two modern matrix- valued recurrent models with the best performance compared others: i.e., RWKV-7 [1 [118]
RetNet [61] and DeltaNet [63] the representatives
and Comba [67]. As another group of baselines, we compare with attention-fiee free deep memory modules with diverse internal attentional bias of dot-product, L2, and Lp regression: i.c., TTT [65]. Miras [58], DLA [68] and Titans [28]. Finally, we also compare with Transformers [27] well as the hybrid of attention and linear RNN, Samba [119].
Training: We train models with about 760M and 1.3B parameters, trained with 30B and 100B tokens, respectively. from a mixture of Feb-WedEEdu [120] and long- cong-context documents with vocabulary size of 32K train all the models from scratch. All models are trained with standard next-token prediction for language modeling, optimized using Adam with tuned learning rate for each model, and with the default optimizer configuration in Behrouz et al. [28].
35


Reviews:
[{"review_id": "Tl8DUqRrEb", "reviewer": "Reviewer_TV1V", "summary": "The paper proposes a new paradigm called Nested Learning (NL), which aims to unify the long-separated fields of neural architecture design and optimization algorithms in deep learning. The authors argue that this separation is unnatural and demonstrate how a complete learning system can be viewed as a set of nested optimization problems. This framework considers meta-learning as a two-level special case and can be extended to more levels and more complex structures.\n\nThe HOPE dynamizes the traditional key and value projection matrices, allowing them to adapt at each step of sequence processing, thereby achieving in-context learning at all levels. Experimental results show strong performance on language modeling, commonsense reasoning, and long-context benchmarks.", "questions": "Have the authors considered a \"relaxed\" nested version that allows a small amount of gradient information to flow between levels?", "limitations": "Yes.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe HOPE model performs exceptionally well on multiple benchmarks.\n\nWeaknesses:\nTo achieve efficient training, the HOPE model appears to use a block-wise hybrid training method, which is inconsistent with its theoretical description.", "was_revised": true, "final_justification": "The authors have sufficiently clarified my concerns regarding the gap between theory and practice (block-wise training) as a necessary trade-off. My score remains unchanged.", "rebuttal": "We thank the reviewer for their time and valuable comments. We are glad that you found the performance of our model exceptionally well on different benchmarks. We respond to your comments below:\n\n\n> To achieve efficient training, the HOPE model appears to use a block-wise hybrid training method, which is inconsistent with its theoretical description.\n\n**Response:** To answer your question, let us first clarify two points: (1) Similar to other types of architectures, there is a trade-off between efficiency and performance of the model. This trade-off is controlled by the size of the chunks. If we set the chunk size equal to 1, then the model becomes fully recurrent and so matches all the motivations, but this comes with the cost of hard parallelizability.  (2) Choosing larger chunk size can be viewed an approximation of the exact recurrence. In this viewpoint, as you mentioned, our theoretical description is not fully aligned with our design. However, the model still performs gradient descent to optimize the internal objective, but the updating the memory weights are happening every C tokens (which can be interpreted as splitting the input sequence, not necessarily changing the recurrence). So from this perspective, there is no gap between our theory and our design.\n\n  \n> Have the authors considered a \"relaxed\" nested version that allows a small amount of gradient information to flow between levels?\n\n**Response**: Thank you for bringing up this setting and we agree this is indeed interesting to consider. Please note that the general formulation for Nested Learning allows such cases, however, since each box corresponds to a specific gradient flow, if there is a full gradient flow between blocks, then those two are sharing a gradient flow and so we would have considered them as a single block from the beginning. However, as you mentioned, we might want to transfer some gradient steps to other levels in a certain special design. This is an interesting direction for future work, which is out of the scope of our current work. \n\n$ ~ $  \n\n\n--- \n## General Note on Contributions: \n\n--- \nFinally, we would like to kindly bring some points about the contributions of our work to your attention: (1) Please note that in our submission, we not only present a new learning paradigm that can explain and unify several well-known concepts, but we also present a new architecture that provides a noticeable performance benefit over the baselines in a wide range of benchmarks. n addition,, we study and compare the in-context learning ability of different architectures in the Appendix, which alone has often been the only focus of some similar studies.  (2) While the current focus of the community in improving architectures is on scaling the parameters and data, Nested Learning presents a new dimension to enhance the model capability, which we believe is a solid contribution. This paradigm can help design more efficient architectures with fewer parameters, and inspire more research studies to build on .  (3) Due to the recent community interest in designing alternative sequence models and understanding the in-context learning ability of  different architectures, we believe that Nested Learning is a timely contribution and its publication can help the community to better design architectures capable of in-context learning and better design architecture-specific optimizers that can lead to better performance, even for current models. \n  \n\nWe hope that our above responses have addressed your concerns, and we would be more than happy to answer any remaining concern."}, {"review_id": "SExpbaDtrH", "reviewer": "Reviewer_aBwg", "summary": "The paper proposes a conceptual framework that treats a model and the optimiser that trains it as a stack of nested optimisation problems. The authors formalise \"prime\" modules (that decompose to themselves) and a general nesting relation, as well as interpret popular optimisers as associate memory learners. They then prove a generic theorem that a neural network represented as an instance of NL can at least learn one-step gradient descent. Finally, they propose HOPE, a neural learning module that is dynamically updating its key/value/query projections using a single internal objective.", "questions": "- How is NL different from bi-level/meta-learning formalisms? \n- What does it mean for a model to be \"mathematically white-box\" (see abstract)? How does this yield claimed interpretability gains?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strenghts:** \n- The conceptual framework of considering architectures and optimisers jointly is intellectually appealing  \n- HOPE achieves small but consistent gains over strong recurrent baselines \n\n**Weaknesses:** \n- The induction proof for theorem 1 is largely intuitive. It assumes that nesting objectives automatically provides a valid inner-gradient update without articulating convergence conditions under which this is guaranteed. \n- The paper was overall hard to follow", "was_revised": true, "final_justification": "The framework of jointly considering architectures and optimizers represents an intellectually appealing direction that could inspire future research. The work would benefit from a major revision addressing presentation concerns, but the core insights and empirical validation justify acceptance.", "rebuttal": "We thank the reviewer for their time and valuable comments. We are glad that you found our framework intellectually appealing. We respond to your comments below:\n\n\n> The induction proof for theorem 1 is largely intuitive. It assumes that nesting objectives automatically provides a valid inner-gradient update without articulating convergence conditions under which this is guaranteed.\n\n**Response:** Please note that our proof is based on induction and is by construction. In fact, we do not rely on convergence conditions and it is completely possible that the nested optimization problem does not converge to a global optima. This theorem, however, indicates that **one step of gradient descent** in the inner problem can be learnt by the outer problem.   \n\n\n\n> The paper was overall hard to follow \n\n**Response**: We thank the reviewer for bringing this point to our attention. We adhere to extensively proofreading the paper and clear points that are unclear. Also, if there is a specific part that you want us to clarify or improve, we would be happy to provide additional details and improve it in the final version of the paper. \n\nHaving said that, we would like to explain that our paper advocates for a completely new learning paradigm.  Therefore, we need to introduce several new terms and concepts,which might not be fully aligned with the readers’ prior viewpoint. This naturally makes the paper more difficult to follow compared to the majority of papers in the community.. We kindly ask you to consider this point in your evaluation. We will, however, follow your suggestion and will do our best to improve the writing in the final version of the paper as much as we can.\n\n\n\n> How is NL different from bi-level/meta-learning formalisms?\n**Response:** This is a great question. The main differences are as follows:\n\n- As illustrated in Figure 1, levels are not necessarily fully nested and we might have parallel building blocks at the same level. An example of this is to optimize linear recurrent models with gradient descent with momentum (we have provided this example in Sec. 2.2).  On the other hand, meta-learning always consists of an inner-loop and an outer-loop.    \n- Even considering fully nested layers, NL is not necessarily a bi-level optimization problem and allows to have several nested levels. Please note that even extending meta-learning to the case of meta-meta-...-meta learning, these nested optimizations are different as here there is a flexibility to have 1 outer-loop or each level be the outer-loop of its 1 level below. This flexibility is not in meta-meta-...-meta learning problem. From this perspective, NL is closer to hierarchical games.  \n- In NL, we can have modules at different levels but each of which is the most outer level of its branch, meaning that there is no other module that meta-learn them.  \n- Levels such as momentum are not fully meta-learning levels while they shape a two-level nested problem, as they can be collapsed into one single level, but with caching previous states.\n\n\n> What does it mean for a model to be \"mathematically white-box\" (see abstract)? How does this yield claimed interpretability gains?\n\n**Response:** This is an important question. Thank you for asking. What we wanted to refer to here was: Given NL perspective, the objective and inner process of each component is clear from the mathematical point of view. In fact, each module is now translated into optimizing an objective with gradient descent, which means that based on the objective, we directly know what is optimized by each component. This perspective further is important for alignment, in which we can simply encode any objective into the internal process of the model, making it aligned with an arbitrary value of interest. \n\nAfter thinking more about it, we agree with the reviewer that this terminology is not very clear. Thus, in the final version of the paper, we would like to replace it with ``white-box objective’’. We would love to hear your suggestions as well regarding this terminology.  \n\n--- \n## General Note on Contributions: \n\n--- \nFinally, we would like to kindly bring some points about the contributions of our work to your attention: (1) Please note that in our submission, we not only present a new learning paradigm that can explain and unify several well-known concepts, but we also present a new architecture that provides a noticeable performance benefit over the baselines in a wide range of benchmarks. n addition,, we study and compare the in-context learning ability of different architectures in the Appendix, which alone has often been the only focus of some similar studies.  (2) While the current focus of the community in improving architectures is on scaling the parameters and data, Nested Learning presents a new dimension to enhance the model capability, which we believe is a solid contribution. This paradigm can help design more efficient architectures with fewer parameters, and inspire more research studies to build on .  (3) Due to the recent community interest in designing alternative sequence models and understanding the in-context learning ability of  different architectures, we believe that Nested Learning is a timely contribution and its publication can help the community to better design architectures capable of in-context learning and better design architecture-specific optimizers that can lead to better performance, even for current models. \n  \n\nWe hope that our above responses have addressed your concerns, and we would be more than happy to answer any remaining concern."}, {"review_id": "oNEkv0xv9d", "reviewer": "Reviewer_P97x", "summary": "Neural network architectures and optimization algorithms are traditionally treated as separate design concerns, but The authors propose Nested Learning (NL) paradigm treats them as integrated layers of nested optimization problems. they formalize NL, show how it subsumes meta-learning and mesa-learning, introduce a new optimizer family, and propose a self-referential model called HOPE that adapts internal memory and projections at multiple levels.", "questions": "See the weakness section above.\n\nextra questions: \n\n1- How scalable is NL in practice? Does adding more nested levels incur significant computational overhead or instability?\n\n2- How do you handle nested levels in backpropagation?\n\n3- Table 4 in appendix H, show that HOPE is better at memorization and selective copying rather than ICR or noisy ICR. How do you explain that?\n\n4- How does HOPE perform with extremely long context datasets? e.g., BABILong", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:** \n\n- Original Perspective: Reframing architectures and optimizers under a unified Nested Learning framework is somehow a novel idea, particularly if it can subsume both meta-learning and in-context learning (ICL).\n\n- Technical Depth:  I did not go through all the proofs in details but the theoretical grounding is rigorous. for example, studying momentum and Muon as associative memory-based NL instances is interesting.\n\n- Concrete Contributions: based on the proposed formulation of Nested Learning and the insights gained, the authors propose a new model, called HOPE, which a self-referential deep memory model. The proposed approach achieves strong results compared to competing methods.\n\n\n**Weaknesses:**\n\n- There is a heavy reliance on terms like “prime neural modules”, “value-less associative memory”, “non-decomposable modules”, etc., without sufficiently grounding them in intuition or concrete examples. For example the definition of 'Prime Neural Module' provided in L182 is not clear. can the authors provide a concrete examples of primal vs non-primal?\n\n- Even though, the nested optimization formulation is mathematically described, the distinction between levels, gradient flows, and objectives is sometimes vague.\n\n> Each problem has its own gradient flow and there is no backpropagation between different levels and blocks. \n\nThis is both a strong assumption and an architectural decision. It needs clearer justification and implications.\n\n- Reproducibility issues: the authors don't share any codes that can be used to reproduce their results.\n\n**Minor errors:**\n\nL17 “further enhancing the interpretability, and explain how...” → fix agreement (“explaining” or split sentence)\\\n\nL106 “in which a neural architecture parameterized by θ is trained...” → repetition and long clauses\nYou use\n\n*Notation: Ensure notation is consistent.* \n- For example, In L145, you define $L^{(1)}$ as the next token prediction loss but in L157, you use   $L^{(2)}$ to refer the it. \n\n- Another example, You switch between using $l$, $L$, and $\\mathcal{L}$ to refer to loss/objective functions. For instance: $l$ is used both in Equation (1) as  $l(\\theta,\\mathcal{T}\\_i,\\Phi)$  and Equation (2) as $l(f(x_i), \\mathcal{M}_{\\theta}(x_1,f(x_1),...,x_i))$. This is inconsistent. can you make it rigorous and use different notations to define different losses.\n\n- You use $M_k$ for key modules but you also define W_k. this leads to confusion. Is M_k a memory or projection matrix? same goes for $M_q$ and $M_v$", "was_revised": true, "final_justification": "the approach is novel and the empirical results are impressive. But the paper is not well-written, i.e., some concepts are not well explained etc. The authors promised to polish the writing in the final version. Hence, I am more inclined to accept. BUt I don't have a strong opinion.", "rebuttal": "We thank the reviewer for their time and valuable comments. We are glad that you found our framework a novel idea. We respond to your comments below:\n\n>  There is a heavy reliance on terms like “prime neural modules”, “value-less associative memory”, “non-decomposable modules”, etc., without sufficiently grounding them in intuition or concrete examples. For example the definition of 'Prime Neural Module' provided in L182 is not clear. can the authors provide a concrete examples of primal vs non-primal? ...\n\n**Response**: Thank you for bringing this to our attention. We understand the importance of providing examples for these new concepts and following your suggestions, we will make sure to provide additional examples and a table to summarize examples for each of our new terms. \n\nMore specifically, we will include a more comprehensive version of the following examples:\n\n**Prime neural Modules** (or non-decomposable modules) $\\rightarrow$ \n- Examples are: MLP, Gated MLP, Convolutions, Gated Convolutions, Multi-resolution Convolutions, any combination of MLPs that only fuse the information across features (not sequence), etc.\n\n**Non-Primal Modules**  $\\rightarrow$\n-  Are modules that can be decomposed into nested blocks such as linear attention: $M_t = M_{t-1} + v_t k_t^{T}$ can be decomposed into a new level, in which given $L_t = <M_{t} k_t, v_t>$ we optimize this loss function using gradient descent in the inner-loop:\n$M_t = M_{t-1} - \\nabla L_t  \\rightarrow$ since $\\nabla L_t = - v_t k_t^{T}$ we have: $M_t = M_{t-1} + v_t k^{T}_t.$\n\nTherefore, an example for non-primal modules is linear attention, or any other recurrent architecture such as DeltaNet, Titans, etc. \n\n**Value-less Associative Memory** $\\rightarrow$ \n- Associative memory that maps all keys to a single value, such as recurrent models with formulation: $h_{t} = A h_{t-1} + f(k_t)$, where $f(.)$ is an arbitrary function of input keys $k_t$. \n\n\nWe will further clarify these concepts:\n\n- **Gradient Flow**: Each box (in Figure 1), has its own gradient flow. For example, in the above example of linear attention, we can see that optimizing the objective $L_t = <M_{t} k_t, v_t>$ using gradient descent is across sequence length and is different from the gradient flow of optimizing the entire parameters of the model (i.e., during pre-training phase). Therefore, we say there are some boxes (or sub-modules), each of which with its own gradient flow.  \n- **Objectives**: Please note that the optimization process in each sub-module requires an objective to be optimized. Therefore, we refer to the loss function of each sub-module as its objective. For example, in the above example, $L_t = <M_{t} k_t, v_t>$ is the objective of that sub-module.\n- **Levels**: Each of sub-modules with their own gradient flow is inside a larger optimization problem. For example, as discussed above, the optimization of objective $L_t = <M_{t} k_t, v_t>$, which is equivalent to linear attention, is inside a larger optimization problem, in which we optimize the parameters of the entire network (for example MLP layers, normalizations, projections, embeddings, etc.). We define a partial order on these problems and so refer to each position in this ordered list as a level. For example, the largest optimization process is the optimization of the parameters of the entire neural network. This optimization will be in level one. Next, its direct inner-optimization problem is the optimizing process of $L_t = <M_{t} k_t, v_t>$, and so this will be in level 2, and so on and so forth. \n\nFollowing your suggestion, we will make sure to provide additional explanations and examples to further clarify these concepts.\n\n\n> “Each problem has its own gradient flow and there is no backpropagation between different levels and blocks”. This is both a strong assumption and an architectural decision. It needs clearer justification and implications.\n\n**Response:** Please note that this has not been an assumption or architectural decision but part of the modeling paradigm (which in fact accurately models the internal process of the architectures). Please note that if there is a backpropagation between different levels and blocks, then they are sharing a gradient flow and so has been one block from the beginning.\n\nTo further clarify: Please note that for each sub-module (box in Figure 1), the other parameters outside of it are considered hyperparameters and so backpropagation for each sub-module has its own path. Let’s consider a special case, where we have a fully nested problem, with only two levels. This case is equivalent to meta-learning, in which the parameters of the inner-loop are considered hyperparameters for the out-loop optimization and vice versa. Therefore, the outer-loop has its own backpropagation and so does not go through the parameters of the inner-loop. The same is also valid for the inner-loop. \n\nThe above example, while illustrating a special case, is valid for all NL instances. \n \n\n\n> Reproducibility issues: the authors don't share any codes that can be used to reproduce their results.\n\n**Response**: We thank the reviewer and we understand the importance of open science. We kindly want to bring to your consideration that sometimes sharing the code is not possible due to circumstances that are not even in the control of authors. However, following your suggestion, we adhere and will make sure to provide all the details that are needed for clarifying the experimental setup and reproducibility of the results. \n\n> Minor errors\n\n**Response**: Thank you very much for bringing them to our attention. We will make sure to proofread the paper and fix all typos. Also, you are right and we will make sure to fix this inconsistent use of notations in the final version of the paper. \n\n\n> How scalable is NL in practice? Does adding more nested levels incur significant computational overhead or instability?\n\n**Response**: Please note that we have scaled our results to 1.3B, which is considered relatively large for academic papers and the resulting model did not show instability. However, we expect that excessively increasing the number of levels can cause instability in the training, which is an important direction for future study to improve NL-based models. We will make sure to discuss these limitations in the paper.\n\n> How do you handle nested levels in backpropagation?\n\n**Response**: Please note that for each sub-module (box in Figure 1), the other parameters outside of it are considered hyperparameters and so backpropagation for each sub-module has its own path. For further clarifying this, let’s consider a special case, where we have a fully nested problem, with only two levels. This case is equivalent to meta-learning, in which the parameters of the inner-loop are considered hyperparameters for the out-loop optimization and vice versa. Therefore, the outer-loop has its own backpropagation and so does not go through the parameters of the inner-loop. The same is also valid for the inner-loop. \nThe above example, while illustrating a special case, is valid for all NL instances. \n\n> Table 4 in appendix H, shows that HOPE is better at memorization and selective copying rather than ICR or noisy ICR. How do you explain that?\n\n**Response:** Thank you for mentioning that. Please note that Hope achieves the best performance across all tasks, except one of them which is fuzzy ICR. While we can compare the models based on the average of their performance across tasks, there is always a possibility that for some individual tasks, mainly because of a specific data type or distribution, a model underperforms a baseline. We believe for this specific case the more complexity of Hope can make it more data hungry than a simpler model like Titans. Since the training is small in this benchmark, for this specific task, we can see a small performance difference.  \n\n> How does HOPE perform with extremely long context datasets? e.g., BABILong\n\n**Response:** Please note that the original setup in BABILong benchmark is based on the fine-tuning models and so benchmarks like RULER are more commonly used in the similar study to show the performance of models in longer contexts. Also, in our RULER experiments, we observe that even in this simple setup, most baselines achieve near zero accuracy and so the advantage of Hope over other recurrent models are already clear. \n\nHowever, following your suggestion, based on the BABILong benchmark setup, we are fine-tuning our models and will report the results when they are ready. We also are adhered to provide these results in the final version of the paper.  \n\n\n  \n\n**We hope that our above responses have addressed your concerns, and we would be more than happy to answer any remaining concern.**"}, {"review_id": "BDKM9HPX3K", "reviewer": "Reviewer_s9eJ", "summary": "This paper introduces the Nested Learning paradigm, a theoretical framework that reframes deep learning models and their training procedures as a unified system of nested optimization problems. The authors propose that the conventional separation between neural architecture design and optimizer choice can be decomposed into a hierarchy of simpler optimization tasks. The authors posit that NL provides a general language that encompasses and extends existing concepts, including meta-learning and in-context learning.   \n\nThis framework leads to a proposed family of general optimizers with high expressivity. Building upon these insights, the authors present a self-referential memory model, HOPE, designed to be an in-context learner at multiple levels of abstraction. The paper shows that HOPE achieves competitive performance on language modeling, common-sense reasoning, and long-context recall tasks, when compared to Transformers and Mamba.", "questions": "Please see above", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea to formulate ICL as an optimization algorithm brings new insights into how to conceptually understand the intuition behind ICL.\n2. Hope demonstrates good performance at a similar training scale.\n\nWeakness:\n1. The author should provide a list (even if not exhaustive) of what can be defined as a prime neural architecture. This is the cornerstone of the author's formulation; however, it is difficult to understand what can be considered as such a cornerstone for building a neural network per the definition.\n2. Similarly, it's not entirely clear how Line 192 eliminates the need to backpropagate between building blocks.\n3. The avg of HOPE is 0 in Table 1. In general, the writing needs a lot more polishing.", "was_revised": true, "final_justification": "The authors addressed my concerns about writing and experimental results.", "rebuttal": "We thank the reviewer for their time and valuable comments. We are glad that you found our work with new insights. We respond to your comments below:\n\n> The author should provide a list (even if not exhaustive) of what can be defined as a prime neural architecture. This is the cornerstone of the author's formulation; however, it is difficult to understand what can be considered as such a cornerstone for building a neural network per the definition.\n\n**Response**: Great suggestion!. We provide a list of such architectures as follows:\n**Prime neural Modules** (or non-decomposable modules) $\\rightarrow$ \n- Examples are: MLP, Gated MLP, Convolutions, Gated Convolutions, Multi-resolution Convolutions, any combination of MLPs that only fuse the information across features (not sequence), etc.\n\n**Non-Primal Modules**  $\\rightarrow$\n-  Are modules that can be decomposed into nested blocks such as linear attention: $M_t = M_{t-1} + v_t k_t^{T}$ can be decomposed into a new level, in which given $L_t = <M_{t} k_t, v_t>$ we optimize this loss function using gradient descent in the inner-loop:\n$M_t = M_{t-1} - \\nabla L_t  \\rightarrow$ since $\\nabla L_t = - v_t k_t^{T}$ we have: $M_t = M_{t-1} + v_t k^{T}_t.$\n\nTherefore, an example for non-primal modules is linear attention, or any other recurrent architecture such as DeltaNet, Titans, etc. \n\nWe will make sure to incorporate these examples in the final version of the paper to make the concept of prime neural network more clearer. \n\n\n> Similarly, it's not entirely clear how Line 192 eliminates the need to backpropagate between building blocks.\n\n**Response**:  There seems to be a misunderstanding here. Please note that if there is a backpropagation between two building blocks, then they have been in the same block in the first place as they share the gradient flow. However, as we discussed in the paper, different levels of a learning module might not share the gradient flow as the nature of their context can be different. For example, the context for the first level is the gradient of parameters, while the context for the internal gradient flow of a recurrent model is the tokens in the input sequence. \n\nIn this regard, please see our illustrative example in Sec. 2.2. The inner-loop of the recurrent model has its own gradient flow as the gradient is with respect to the tokens and the recurrence is over the sequence length. On the other hand, the outer-loop aims to optimize the parameters in the pre-training by using gradient descent on the entire data points. Therefore, the recurrence is over the data samples and is not on the tokens any more. \n\nAs another example, considering momentum term as another level, it aims to compress the gradient terms, and so its iteration is over gradients. \n\nAll in all, we do not force the model to eliminate the need to backpropagate between building blocks.  It is indeed a fact about how neural networks are decomposed into these building blocks. \n\nWe hope that this has clarified the misunderstanding. Please let us know if further clarification is needed and we would be happy to provide more examples/clarifications. \n\n> The avg of HOPE is 0 in Table 1. In general, the writing needs a lot more polishing.\n\nThank you very much for bringing this to our attention. We will make sure to fix this issue in the final version. We have provided the average in the following table:\n\n| Model Size    | Average |\n| -------- | ------- |\n| 340 M |  46.90   |\n| 760 M |   52.25   |\n| 1.3 B  |  57.23   |\n\n\n\n> General Writing:\n\nRegarding the general writing we will extensively proofread the paper and clarify the points that are unclear. Also, if there is a specific part that you would like  us to clarify further, please let us know and we would be happy to provide additional details and improve it. \n\nPlease also note that, since this study is advocating for a new learning paradigm, it is expected and possible that several terms and concepts are new and might not be fully familiar to what we often read in deep learning literature. This would also make the paper more challenging to read and comprehend, which comes from the nature of our work. We kindly ask you to consider this point that this is the initial work in this direction. We will do our best to improve the writing in the final version of the paper and make it easy to follow.\n\n\n---\n## General Note on Contributions:\n--- \n\nWe want to kindly bring some points about the contributions of our work to your consideration: (1) In our submission, we not only present a new learning paradigm that can explain and unifies several well-known concepts, but we also present a new architecture that provides a noticeable performance benefit over the baselines in diverse benchmarks. In addition, we study and compare the in-context learning ability of different architectures in the Appendix, which alone has often been the only focus of the similar studies.  (2) While the current focus of the community is  on improving scaling of the architectures parameters and data, Nested Learning presents a new dimension to enhance the model capability, which is its internal computations. We believe this paradigm can help to design more efficient architectures with fewer parameters, and inspire more research studies to build on .  (3) Due to the recent community interest in designing alternative sequence models and understanding the in-context learning ability of  different architectures, we believe that Nested Learning is a timely contribution and its publication can help the community to better design architectures capable of in-context learning and better design architecture-specific optimizers that can lead to better performance, even for current models. \n  \n  \n\n**We hope that our responses have addressed your concerns. Please let us know if there are additional concerns and we would be more than happy to engage more and clarify any remaining concerns.**"}]

Author rebuttal:

Meta-review: The paper proposes a novel learning approach, albeit with some similarities to meta-learning, deep declarative networks, etc., that demonstrates some promise in advancing the state-of-the-art and hence is worthy of communicating to the community. While the reviewers were on the balance positive about the paper, there were a number of issues raised that the authors have responded to, and promised to address in the final version of the paper, in particular with respect to clarity. Release of open-source software code, as promised, would also be highly valuable in communicating and understanding the method. The Area Chair agrees with the consensus of the reviewers and recommends acceptance.
Decision: Accept (poster)