Title: Nested Learning: The Illusion of Deep Learning Architectures
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neural Learning Module

Deep Learning

Memory

Recurrent Neural Networks
Abstract: Over the last decades, developing more powerful neural architectures and simultaneously designing optimization algorithms to effectively train them have been the core of research efforts to enhance the capability of machine learning models. Despite the recent progresses, particularly in developing Language Models (LMs), there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improved, and find ''effective solutions,''. In this paper, we present a new learning paradigm, called Nested Learning (NL), that coherently represents a model with a set of nested, multi-level, and/or parallel optimization problems, each of which with its own ''context flow''. NL reveals that existing deep learning methods learns from data through \emph{compressing} their own context flow, and explain how in-context learning emerges in large models. NL suggests a path (a new dimension to deep learning) to design more expressive learning algorithms with more ''levels'', resulting in higher-order in-context learning abilities. In addition to its neuroscientifically plausible and mathematically white-box nature, we advocate for its importance by presenting three core contributions: (1) Deep Optimizers: Based on NL, we show that well-known gradient-based optimizers (e.g., Adam, SGD with Momentum, etc.) are in fact associative memory modules that aim to compress the gradients with gradient descent. Building on this insight, we present a set of more expressive optimizers with deep memory and/or more powerful learning rules; (2) Self-Modifying Titans: Taking advantage of NL's insights on learning algorithms, we present a novel sequence model that learns how to modify itself by learning its own update algorithm; and (3) Continuum Memory System: We present a new formulation for memory system that generalizes the traditional viewpoint of ``long-term/short-term memory''. Combining our self-modifying sequence model with the continuum memory system, we present a learning module, called Hope, showing promising results in language modeling, continual learning, and long-context reasoning tasks.


Peer reviews:
comments: SUMMARY:
This paper introduces the Nested Learning paradigm, a theoretical framework that reframes deep learning models and their training procedures as a unified system of nested optimization problems. The authors propose that the conventional separation between neural architecture design and optimizer choice can be decomposed into a hierarchy of simpler optimization tasks. The authors posit that NL provides a general language that encompasses and extends existing concepts, including meta-learning and in-context learning.   

This framework leads to a proposed family of general optimizers with high expressivity. Building upon these insights, the authors present a self-referential memory model, HOPE, designed to be an in-context learner at multiple levels of abstraction. The paper shows that HOPE achieves competitive performance on language modeling, common-sense reasoning, and long-context recall tasks, when compared to Transformers and Mamba.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The idea to formulate ICL as an optimization algorithm brings new insights into how to conceptually understand the intuition behind ICL.
2. Hope demonstrates good performance at a similar training scale.

Weakness:
1. The author should provide a list (even if not exhaustive) of what can be defined as a prime neural architecture. This is the cornerstone of the author's formulation; however, it is difficult to understand what can be considered as such a cornerstone for building a neural network per the definition.
2. Similarly, it's not entirely clear how Line 192 eliminates the need to backpropagate between building blocks.
3. The avg of HOPE is 0 in Table 1. In general, the writing needs a lot more polishing.

QUESTIONS:
Please see above

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNeural network architectures and optimization algorithms are traditionally treated as separate design concerns, but The authors propose Nested Learning (NL) paradigm treats them as integrated layers of nested optimization problems. they formalize NL, show how it subsumes meta-learning and mesa-learning, introduce a new optimizer family, and propose a self-referential model called HOPE that adapts internal memory and projections at multiple levels.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** \n\n- Original Perspective: Reframing architectures and optimizers under a unified Nested Learning framework is somehow a novel idea, particularly if it can subsume both meta-learning and in-context learning (ICL).\n\n- Technical Depth:  I did not go through all the proofs in details but the theoretical grounding is rigorous. for example, studying momentum and Muon as associative memory-based NL instances is interesting.\n\n- Concrete Contributions: based on the proposed formulation of Nested Learning and the insights gained, the authors propose a new model, called HOPE, which a self-referential deep memory model. The proposed approach achieves strong results compared to competing methods.\n\n\n**Weaknesses:**\n\n- There is a heavy reliance on terms like “prime neural modules”, “value-less associative memory”, “non-decomposable modules”, etc., without sufficiently grounding them in intuition or concrete examples. For example the definition of 'Prime Neural Module' provided in L182 is not clear. can the authors provide a concrete examples of primal vs non-primal?\n\n- Even though, the nested optimization formulation is mathematically described, the distinction between levels, gradient flows, and objectives is sometimes vague.\n\n> Each problem has its own gradient flow and there is no backpropagation between different levels and blocks. \n\nThis is both a strong assumption and an architectural decision. It needs clearer justification and implications.\n\n- Reproducibility issues: the authors don't share any codes that can be used to reproduce their results.\n\n**Minor errors:**\n\nL17 “further enhancing the interpretability, and explain how...” → fix agreement (“explaining” or split sentence)\\\n\nL106 “in which a neural architecture parameterized by θ is trained...” → repetition and long clauses\nYou use\n\n*Notation: Ensure notation is consistent.* \n- For example, In L145, you define $L^{(1)}$ as the next token prediction loss but in L157, you use   $L^{(2)}$ to refer the it. \n\n- Another example, You switch between using $l$, $L$, and $\\mathcal{L}$ to refer to loss/objective functions. For instance: $l$ is used both in Equation (1) as  $l(\\theta,\\mathcal{T}\\_i,\\Phi)$  and Equation (2) as $l(f(x_i), \\mathcal{M}_{\\theta}(x_1,f(x_1),...,x_i))$. This is inconsistent. can you make it rigorous and use different notations to define different losses.\n\n- You use $M_k$ for key modules but you also define W_k. this leads to confusion. Is M_k a memory or projection matrix? same goes for $M_q$ and $M_v$\n\nQUESTIONS:\nSee the weakness section above.\n\nextra questions: \n\n1- How scalable is NL in practice? Does adding more nested levels incur significant computational overhead or instability?\n\n2- How do you handle nested levels in backpropagation?\n\n3- Table 4 in appendix H, show that HOPE is better at memorization and selective copying rather than ICR or noisy ICR. How do you explain that?\n\n4- How does HOPE perform with extremely long context datasets? e.g., BABILong\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper proposes a conceptual framework that treats a model and the optimiser that trains it as a stack of nested optimisation problems. The authors formalise "prime" modules (that decompose to themselves) and a general nesting relation, as well as interpret popular optimisers as associate memory learners. They then prove a generic theorem that a neural network represented as an instance of NL can at least learn one-step gradient descent. Finally, they propose HOPE, a neural learning module that is dynamically updating its key/value/query projections using a single internal objective.

STRENGTHS AND WEAKNESSES:
**Strenghts:** 
- The conceptual framework of considering architectures and optimisers jointly is intellectually appealing  
- HOPE achieves small but consistent gains over strong recurrent baselines 

**Weaknesses:** 
- The induction proof for theorem 1 is largely intuitive. It assumes that nesting objectives automatically provides a valid inner-gradient update without articulating convergence conditions under which this is guaranteed. 
- The paper was overall hard to follow

QUESTIONS:
- How is NL different from bi-level/meta-learning formalisms? 
- What does it mean for a model to be "mathematically white-box" (see abstract)? How does this yield claimed interpretability gains?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
The paper proposes a new paradigm called Nested Learning (NL), which aims to unify the long-separated fields of neural architecture design and optimization algorithms in deep learning. The authors argue that this separation is unnatural and demonstrate how a complete learning system can be viewed as a set of nested optimization problems. This framework considers meta-learning as a two-level special case and can be extended to more levels and more complex structures.

The HOPE dynamizes the traditional key and value projection matrices, allowing them to adapt at each step of sequence processing, thereby achieving in-context learning at all levels. Experimental results show strong performance on language modeling, commonsense reasoning, and long-context benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
The HOPE model performs exceptionally well on multiple benchmarks.

Weaknesses:
To achieve efficient training, the HOPE model appears to use a block-wise hybrid training method, which is inconsistent with its theoretical description.

QUESTIONS:
Have the authors considered a "relaxed" nested version that allows a small amount of gradient information to flow between levels?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-14