Barriers for Learning in an Evolving World:  Mathematical Understanding of Loss of Plasticity
Authors: Amir Joudaki
Giulia Lanzillotta
Mohammad Samragh
Seyed Iman Mirzadeh
Keivan Alizadeh
Thomas Hofmann
Mehrdad Farajtabar
Fartash Faghri
Venue: ICLR 2026 Poster (2026)
Keywords: loss of plasticity
deep learning theory
continual learning
Primary area: learning theory

Abstract: Deep learning models excel in stationary settings but suffer from loss of plasticity (LoP) in non-stationary environments. While prior literature characterizes LoP through symptoms like rank collapse of representations, it often lacks a mechanistic explanation for why gradient descent fails to recover from these states. This work presents a first-principles investigation grounded in dynamical systems theory, formally defining LoP not merely as a statistical degradation, but as an entrapment of gradient dynamics within invariant sub-manifolds of the parameter space. We identify two primary mechanisms that create these traps: frozen units from activation saturation and cloned-unit manifolds from representational redundancy. Crucially, our framework uncovers a fundamental tension: the very mechanisms that promote generalization in static settings, such as low-rank compression, actively steer the network into these LoP manifolds. We validate our theoretical analysis with numerical simulations and demonstrate how architectural interventions can destabilize these manifolds to restore plasticity.

Published at ICLR 2026
BARRIERS FOR LEARNING IN AN EVOLVING WORLD: MATHEMATICAL UNDERSTANDING OF LOSS OF PLASTICITY
Amir Joudaki', Giulia Lanzillotta', Mohammad Samragh Razlighi", Iman Mirzadeh Keivan Alizadeh", Thomas Hofmann',, Mehrdad Farajtabar", Fartash Faghri ETH Zürich AApple
ami joudak @en.iiiiiii.fe... wiinf.tth..c.. ch, (fartash, [_farajtabarr@app com
ABSTRACT
Deep learning models excel in stationary settings but suffer from loss of plasticity (LoP) in non-stationary environments. While prior literature characterizes LoP through symptoms like rank collapse of representations, it often lacks a mechanistic explanation for why gradient descent fails to recover from these states. This work presents first-prinon priscprrnciples investigation grounded in dynamical systems theory, formally defining LoP not merely statistical degradation, but as an entrapment of gradient dynamics within invariant sub-manifolds of the parameter space. We identify two primary mechanisms that create these traps: frozen units from activation saturation and cloned-unit manifolds from representational redundancy. Crucially, our framework uncovers fundamental tension: the very mechanisms that promote generalization in static settings, such as low-rank compression, actively steer the network into these LoP manifolds. We validate our theoretical analysis with numerical simulations and demonstrate how architectural interventions can destabilize these manifolds to restore plasticity.
Code:
1
INTRODUCTION
The extraordinary success of back pack-propagation training deep neural networks often relics on two implicit assumptions. First, stationarity is assumed: the data distribution encountered during training is similar to the distribution faced during deployment, implying that post-training adaptation is minimal. Second, a single random initialization of network parameters is the main source of diversity and exploration potential-a resource that is progressively consumed by optimization and rarely replenished. These assumptions falter when an artificial agent must operate and learn continuously within an environment characterized by changing dynamics or evolving task distributions. This scenario, commonly referred to continual or lifelong learning, presents the
dilemma (Abraham and Robins 2005) Chaudhry et al.) 2018), demanding a system stable enough to retain acquired knowledge yet plastic enough to integrate new information.
Empirically, standard deep networks subjected to long sequences of tasks or drifting data streams exhibit a sharp decline in learning capability (Dohare et al.) 2024| Berariu et al. [202 Nikishin et al.) [2022). This phenomenon, termed Loss of Plasticity (LoP), distinct from catastrophic forgetting: while forgetting erases old knowledge, LoP prevents the acquisition of new knowledge. Common symptoms are well-documented: exploding weight magnitudes (Nikishin et al.) (2022), the emergence of "dead" units (Sokar et al. 2023 Lyle
et al., 22022), and a collapse in the effective rank of representations (Papyan al. 2020 Huh al. 2023
Kumar et al. 2020).. While recent works have linked LoP to back propppaaagttiion mechanics (Dohare et al. 2024)
NTK rank degradation (Lyle et al. 2023), these observations largely descriptive. They characterize the symptoms of the failing network but stop short of formal mechanistic explanation for the persistence of the non-plastic state.

---

Published at ICLR 2026
Independent tasks
Sequential tasks
0o
do
0a
8c
A
Oc
M
Figure 1.1: Conceptual illustration of LoP as a topological trap. The high-dimensional parameter space contains -ineeemnssioaal, invariant LoP manifolds M θ Left (Independent tasks): During standard learning, trajectories starting at 0g can explore the full space (e.g.. Op, (9,0c), though implicit biases may eventually drive them toward simpler representations on . (0x). Right (Sequential tasks): In continual learning, early tasks drive parameters onto the manifold (0p - AA). Once trapped, gradients for subsequent tasks become strictly tangent to M, constraining future dynamics to this restricted subspace ^ OB Ic - ...) and preventing the recovery of plasticity needed for new data distributions.
A fundamental question remains: Why exactly does gradient descent fail to recover from these states? If LoP is merely bad configuration, why do gradients steer the model back toward useful regions? For example, while many works have identified low rank as hallmark of LoP, why is the network unable to recover feature diversity under new task distribution? Our work revisits the dynamics of gradient descent through the lens of dynamical systems theory to answer this. We posit that LoP is not simply a degradation of statistics, like rank or weight norm, but topological entrapment under gradient descent. We seek to identify the specific geometric structures underlying the optimization landscape that act as "sinks" for gradient trajectories, restricting the model from accessing its full degrees of freedom.
The central contribution of this work is the formalization of LoP manifolds: invariant sub- mabbooassslds of the parameter space that act as "traps" for gradient descent (Fig. [.T). Once the optimization process steers parameters into these regions, the gradient flow becomes strictly tangent to the manifold, rendering escape impossible without external intervention. This formulation fills a critical theoretical gap: while prior literature has extensively cataloged the symptoms of plasticity loss (e.g., weight explosion, rank collapse), framework identifies the mathematical and geometric principles that provably make these states persistent and irreversible. Furthermore, tension. We show that the very mechanisms that promote generalization in static settings, specifically, the compression of features toward low-rank structures, are the primary forces that steer the network into these LoP manifolds. Thus, the dynamics that maximize performance on the current task inadvertently construct the barrier adaptability for future tasks.
show the driver towards this entrapment fundamental rank-plasticity
Summary of Contributions. In this work, we formalize the mechanics of plasticity loss and validate theory across varied architectures. Our specific contributions are:
A Dynamical Systems Definition of LoP (Sec.2). We move beyond symptomatic definitions propose a formal definition of LoP as the entrapment of optimization trajectories within specific invariant sub- manifolds of the parameter space.
2

---

Published at ICLR 2026
Identification of Trap Mechanisms (). We theoretically characterize two primary classes of invariant manifolds: Frozen-Une Manifolds (Mş), caused by activation saturation, and Cloned- ConeddUUitt Manifolds (Mc), caused by representational redundancy. We prove that standard gradient-basee optimization cannot escape these manifolds once entered.
The Rank-Plasticity Tension (Sec. We derive theoretical connection between feature rank dynamics and plasticity. We show that properties widely considered beneficial for generalization (e.g., neural collapse) act as attractive forces toward LoP manifolds.
Empirical Validation and Mitigation (Sec.T). We validate our framework through extensive experiments on MLPs, ResNets, and ViTs. We further analyze how architectural interventions (normalization) and targeted perturbations (noise injection, dropout) affect the stability of these manifolds, providing principled basis for recovering plasticity.
1.1 BACKGROUND AND RELATED WORK
Loss of Plasticity (LoP). A network is said to suffer a loss of plasticity when, after some period of training, it can longer acquire information as effectively
freshly-initialised model of the same architecture. LoP has been documented in a variety of continual- -eaaninng and reinforeement-lemming settings (Dohare ct al.
2024 Nikishin al. 2022). Crucially, LoP is distinct from catastrophic forgetting: performance on past tasks may remain intact while the ability to learn future tasks degrades (Lyle et al. 2023). Typical symptoms
include exploding weight norms, growing numbers of dead (saturated) units, and collapse of the effective rank of hidden representations (Dohare et al. 2021 Papyan et al. 2020).
Previous explanations. Early accounts linked LoP to individual pathologies, e.g. weight-norm growth or activation sparsity. However, these factors alone failed to consistently explain the phenomenon (Lyle et al. the network's neural tangent kernel (NTK)
2022). A more recent view connects LoP to a degeneration that is once the NTK becomes low-rank, many directions in function space receive negligible gradient and can no longer be learned (Lyle et al.) 2023). This perspective suggests that LoP is multi-faceted with diverse surface-level defects (e.g.. dead units, duplicated features) sharing the common consequence of reducing the network's effective degrees of freedom.
Geometric Singularities and Learning Dynamics. The connection between overparameterrizatoo. reduced dimensionality, and learning difficulties has deep roots in the analysis of neural network geometry. Hierarchi- cal models exhibit singularities that are regions in parameter space where the mapping from parameters to function is not unique (e.g., due to unit duplication or vanishing). Foundational work by Fukumizu and Amari (2000) and and Amari et al. (2066) demonstrated that these singularities cause the Fisher Information Matrix to degenerate, leading to slow learning dynamics (plateaus) gradient descent is attracted to these regions.
Implicit Bias and Stochastic Dynamics. Recent work highlighted how the optimization algorithm itself contributes to the collapse towards simpler representations. Chen et al.) (2023) analyzed the implicit bias of Stochastic Gradient Descent (SGD), showing that gradient noise induces an attractive force towards these singular regions (termed "Invariant Sets"). This "Stochastic Collapse" suggests that the tendency towards LoP states is exacerbated by the stochastic nature of the optimization process, even if the regions are unstable under deterministic gradient descent. Furthermore, /Wang et al. (2024) empirically demonstrated that maintaining trainability (ability to fit new data) does not guarantee generalizability (performance on unseen data), emphasizing the need for methods that restore genuine plasticity.
2 LoP MANIFOLDS: TRAPS FOR GRADIENT DESCENT
In this section, we lay the groundwork for our analysis by defining Loss of Plasticity (LoP) within a dynamical systems framework and recalling the standard definitions for feed-forward neural networks and
3

---

Published at ICLR 2026
back-propagation. The stability of LoP manifolds, a crucial concept for understanding their persistence, will be discussed later in Sec. 2 Let 0 E ④ C RP" represent the parameters of a neural network. We consider ) using gradient descent or its stochastic variants. The objective typically to minimize a loss Σi=1 (z)),
training on a stream of data { ys fi=1
which we can succinetly refer to as L()). This allows us define LoP based on the trajectory of parameters O(t) in the parameter space θ as driven by the negative gradient in the loss landscape. As illustrated in the right panel of Fig. [.] we are interested in scenarios where this trajectory becomes confined a lower-dimensional subspace M O.
Definition 2.1 (LoP Manifold). A manifold M C. θ induces LoP if the gradient of the loss function is tangent to the manifold at every point on the manifold. That is, VoL(O) E TaM for all A E M. where TaM denotes the tangent space of M at 0. This tangency condition ensures that once the gradient flow enters M, (oo((
il remains within M under the dynamics of gradient flow de(t)
ds
Remark 2.1. Definition 2.1 is stated in terms of continuous-time gradient flow Ô = -VV VoC( the tangency condition (QC( E M ensures that the flow starting in M remains in M. For a general curved manifold this need not imply invariance under discrete updates 0k+1 02 Th V ((fk), as discretization step can lead to an escape from the manifold. However. for affine LoP manifolds, both gradient flow and (stochastic) gradient descent dynamics are invariant once they enter M, as the discretization step cannot lead to an escape.
Remark 2.2. the conditions in Desinition 2.7 hold irrespective of the specific data distribution generating the loss f, which we can think of as functional LoP, and is our primary area of interest. Such LoP arises from the network architecture and gradient descent dynamics alone and is particularly relevant as it persists even if the task or data distribution evolves.
Given these definitions, we can formalize existence of these LoP manifolds, restricting subsequent learning. We present a central theorem that jointly addresses LoP arising from frozen and duplicate units. The intuition is that once units become unresponsive (frozen) or perfectly redundant (cloned), they tend to remain so under standard gradient- based optimization.
Theorem 2.1. Let G (V, E) be the network's 's computational DAG and let 0 =fuw (u u) E E) E e denote the edge parameters.
1. Frozen-unit manifold My. Assume there exists F C such that, for all finite inputs encountered,
each E F is persistently saturated (J'(z.) = 00) Then the gradients wi.t. all incoming parameters to e vanish on any mini-batch, so those coordinates remain fixed; writing the linear constraints as
(v) = const for all v E F, the affine subspace MM := 10 : 0m)(v) const Vv E F} satisfies v VL() E TaMe and GD/SGD updates initialized in MM remain in Mr.
2. Cloning manifold Mc. Assume partitioning of nodes into disjoint blocks (Si,..., S exists with following properties. For every ordered block pair (S,, ST), we have the linear equalities Eves, Ouv Eves, for all u, u E St (equal row-sums) and Eves, Our Eues, Oue for all v, v' E S, (equal column- -umns). Let Mc be the affine subspace of consisting of all A satisfying these constraints. If E MMc, then (i) all units within any block share the same forward values on any input, (üi) all units within any block share the same backpropugated errors on any input, and therefore (iii) the per-edge gradients are constant across edges connecting the same block pair, Le..for any (u, v) and (u', v') with u. ul E S, and v, v' E Sy. ac/00.. = O0/00 Hence v L(()) E ToMo and GD/SGD updates initialized in Mc remain in Mc.
Note that both LoP manifolds ME and Mc are defined as linear LoP manifolds in the sense of Definition 2.1 Formal proofs and further details are provided in Appx. A.3
Proof idea. Frozen units. If unit stays in a regime with f'(=.) = 0 for all finite inputs (e.g.. tanh with very large ||0mn (v)]l or ReLU with large negative bias), then aL/00,, (v) 0; its incoming parameters are
4

---

Published at ICLR 2026
fixed, so updates are tangent to Mp. Cloning via redistribution. The key idea the row/column-sun equalities mean total incoming/outgoing weight from/to any block is redistributed within each block pair (S,.S)) Thus, the total contribution to the forward and backward of each unit within a block remains identical, implying the forward and backward cloning (properties (i) and (ii) within blocks. Thus, per-edge gradients dC/d0... = h(u) 8(v), are therefore by forward and backward symmetries across the blocks the gradients will be constant for any two units in these blocks (u, v) E S, x S,. These block-wise constant gradients trivially satisfy the row-sum and column-sum equalities, and hence are tangent to Mc. and first-order updates remain on both manifolds.
Remark 2.3. It is important to note that the Duplicate Manifold MD (defined formally via Incoming and Outgoing Equitable partitions in Appx. A3 represents a significant generalization of the cloning concepts rypically discussed in literature. Prior analyses of singularities (Fukumizu and Amari 2000) invariant sets CChn et al.) ) generally define cloned units by requiring their associated weights to be strictly identical (the block-wise constant condition in our terminology). Our framework proves that invariance under gradient descent holds even under the relaxed condition of equitability, where individual weights may differ as long as specific incoming and outgoing sums are maintained. This significantly broadens the class of structures identified as LoP manifolds.
Remark 2.4. The cloning LoP manifold naturally lends itself to gradient descent and stochastic gradient descent, regardless of the order which we process the samples, will remain strictly within the manifold. This extends to virtually all variations of gradient descent based optimizations. namely Stochastic Gradient Descent (SGD), SGD with momentum, and Adam, as long as the optimizer is initialized at the onset of cloning. The only exception to this is weight decay which could break some symmetries. This fact can be empirically observed across our cloning experiments, showing that across a wide range of optimization schemes the model remains trapped onto to the LoP manifold.
Remarkably, the theorem admits a modular version, which allows us to create practical cloning certificate for modern architectures (see Appx. A.4 for more details).
Theorem 2.2 (Modular Cloning (informal)). This cloning property can be decomposed modularly. If a network is composed of individual modules (e.g., layers blocks), and each module locally sutisfies the cloning invariance properties- nnamlly, (1) cloned inputs produce cloned outputs (Forward Invariance), (2) cloned backward signals the outputs produce cloned backward signals at the inputs (Backward Invariance), and (3) gradient updates preserve these invariances (Persistence) then the entire network resides on a cloning manifold, provided the cloning profiles (partitions) are consistent at the interfaces between modules.
To empirically test the validity of the cloning manifold and their potential escape mechanisms conduct cloning experiments. First, base model (e.g., an MLP) is trained on a specific task. Subsequently, a larger model is constructed by expanding the base model. This expansion involves increasing the width of the model for MLPs, the number of channels for CNNs and ResNets, and the feature dimension for ViTs. The weights of the cloned model are initialized in such a way that its activations identical to those of the base model. This effectively creates blocks of units that have identical activations. Next, we train both the base and cloned models on the same task and monitor their training progress through the loss curve, the effective rank presents of representations, and the cloning R²2 score. Figure preesensss the results of such experiments on MLPs, shedding light on the dynamics within and escapes from these LoP manifolds. The empirical validation of these claims, such as demonstrating perfect cloning under specific initializations or the persistence of dead units, can be found in Fig. 2.1 2.1 and Appx. [ Notably, Remark [2.4] predicts that Adam, despite fundamental differences with SGD, also fails to escape the manifold, which is validated by our experiments.
Escaping the LoP manifolds with perturbations. While a comprehensive theoretical analysis of the stability of empirically observed LoP manifolds is beyond the scope of this work, our empirical investigations indicate that these manifolds are frequently unstable or resemble saddle-like shapes. Certain types of noise symmetry-breakee operations can help models escape these manifolds. We highlight two common
5

---

Published at ICLR 2026
perturbations: (1) Noisy SGD is a modification of SGD that adds Gaussian noise the computed gradients before parameter updates. The magnitude of this injected noise is usually proportional to the norm of the gradient, with its initial relative strength gradually decreasing over successive steps. By applying this noise after cloning, can determine whether the model can escape the LoP manifold or if it will fall back. (2) Dropoul introduces stochasticity the forward and backward passes by randomly zeroing activations. For cloned units, this breaks the symmetry because different clones might be active in different dropout masks, leading to divergent gradient updates. This is supported by experiments where dropout helps a model escape an artificially induced cloning manifold (see Fig. 2.1
MLP
MLP
MLP
1.0
1.6
Cloning
Base
Cloming
Cloning
14
+ Dropout
Noise
Effectivv Rank
80
Cloning
0.8
0.7
Cloned
Dropout
1.2
1.0
Cloned
+ Dropout
Nosse
-0
Nuise
0.8
60
0.5
0.4
50
0.2
1e5
2x5
365
465
1e5
2e5
3e5
4e5
105
2e5
365
405
Step
Step
Step
duplicate manifold LoP. The cloned network dynamics remain confined in the base network manifold when using SGI,, however using Noisy SGD Dropout the dynamics can escape the manifold. Left: Cloning R²2 score quantifies the proportion of variance in individual unit activations within a cloned block that is explained by the mean activation of that block. An R² score of indicates perfect cloning (units in a block are nearly identical), while a 0 score indicates no explained variance. See Appx. B.1.2 (Appendix B for the precise formula and calculation details. Middle: Training loss comparison. Cloned loss refers the loss of the cloned model during its training phase, while base loss refers to the loss of the original base model, which continues training for comparison. Right: Effective rank evolution showing representational diversity.
Figure 2.1: Cloning MLPs experiments. The empirical data validates Theorem 2.
Both noisy SGD and dropout act as symmetry-breakin operations. In the case of dropout, both forward and backward passes are asymmetric for cloned units. For noisy SGD, the backward pass (gradient update) becomes asymmetric. This asymmetry causes the parameters of notionally cloned units to slowly diverge. Remarkably, in our MI.P experiments, even a small amount of gradient noise, e.g., a single step with noise magnitude 0.01 relative gradient norms, suffices to initiate escape from an LoP manifold, though stronger noise generally leads to faster escape. In contrast, in settings such as Vision Transformers, while the model could escape from the manifold with a small perturbation, it did not move very far from it. More experimental studies into this direction would be vital to better understand the stability of these LoP manifolds.
3 EMERGENCE OF Loss OF PLASTICITY FROM LINEAR-NONLINEA RANK DYNAMICS
Having established the existence of LoP manifolds, we now discuss the mechanisms within standard training that drive their formation. The optimization process naturally follows trajectory beginning with an expan- sion of representational diversity, as features propagate through nonlinear layers and become increasingly decorrelated (Poole et al. 2016).. However, this expansion is counterbalanced by a drive toward geometric simplicity in the final phase of training, captured by neural collapse (Papyan et al., 2020). where within- class variability vanishes and features converge to a low-rank subspace spanned by class means. We argue that this geometric collapse- while beneficial for generalization in static settings inherently steers the model
6

---

Published at ICLR 2026
toward the low-rank LoP manifolds we identified. Thus, the very dynamics that maximize separability on the current task provide the direct mechanism for the emergence of LoP.
To diagnose whether features are diversifying or compressing during training, we track a smooth surrogate of the rank of the feature correlation matrix. Exact rank is numerically unstable, because it is not a continuous nor a differentiable map from the matrix space, Therefore, we use differentiable proxies to rank such as Rényi-2 rank, (r2(M)= (ir M)3/IIM]] or the Shannon effective rank, er( exp( exp((H (A(M)) /tr M)). Both increase when the eigenmass is evenly distributed and decrease when dominated by few values. Note that both of these surrogates are maximized when matrix M has equal eivenvalues and minimized when it a rank-1 1 matrix. The following theorem offers an insight into how nonlinear layers contribute to the formation of features.
Theorem 3.1 (Rank gain and decorrelation potential). Assume o E LN 1)) is nonlinear and normalized such that [[](2)) = 0 and Var()(2)) 1. Let C be pre-activation correlatton matrix. For an activation o define the correlation kernel K KK(r) = Corr(p(z), Φ(U)) where (I. y) are jointly Gaussian with correlation
T. The nonlinearity acts entrywise on correlations, producing Ka( We define the decorrelation strength
Yo - 1- (0)
[+ (0) and the decorrelation potential V(C) = Eiij C2 - Ctt). The Rényi-2 effective rank is
non-decreasing according to the bound below:
er2 er2(Kx(O))
(C)
2 + Ta
(C)
ICP
The ratio is strictly larger than 1 when the potential V (C) vanishing (all off off-diagonal Cig E {0, =1}) and the the activation is linear (7h is zero).
Note that the theorem implies that any correlations within range |Cyyl E (0, 1) provide a potential (CC > 0 that the nonlinearity consumes to increase rank. The rate of this increase is governed by the scalar % E [0, 1]. depends on the activation and quantifies the nonlinearity's 's ability to de-correlate inputs and thereby increase rank. Larger To guarantees larger rank gain for the same input spectrum. For rigorous derivation, see Appx. A.
Implication for emergence of frozen units. In practice, the first and second moments of pre-activations drift during training, modulating the effective activation function. This modulation directly impacts the decorrelation strength "g- We find that % is monotonic with respect to the saturation parameters: for ReLU, making the effective bias more negative increases Yo: for lanh, increasing the effective gain does the same. Consequently, optimizing for feature diversity (high decorrelation strength) creates a pressure push units into regimes where the derivative vamishes on most inputs. This explains why training that effectively restores rank also creates "frozen" or "dead" units.
Implication for creation of duplicate or cloned units. Neural collapse is a widely observed endpoint in which the penultimate features are low-rank, and the class means form simplex Equiangular Tight Frame (ETF) structure (Papyan et al., 2020). The theorem makes a precise prediction regarding this state: for the rank stabilize (ratio 1) while retaining nonlinearity (7 >> 0), the decorrelation potential (C( must vanish. This forces correlations to converge to the fixed points 0 (orthogonal features) or #1 (aligned or anti-aligned features). Thus, the dynamics of rank expansion naturally drive the network toward a mixture of orthogonal subspaces and cloned units, matching the geometry of LoP manifolds.
The two implications above provide theoretical perspective on why duplicate features and frozen units frequently emerge at or near convergence, and why the resulting representation lies close to LoP manifolds, as proven in Theorem 2.1]
Remark 3.1. Our analysis explicitly links LoP the "Rich" " or feature learning regime. In the "Lazy" learn- ing regime (associated with the Neural Tangens Kernel), weights remain close to initialization, maintaining hightrank connectivity and avoiding feature compression. Conversely, the "Rich" regime is characterized by
7

---

Published at ICLR 2026
significant weight evolution and the emergence of low-rank structures (Chizat al. 2019 Woodworth et al. 2020). The frozen units and cloned manifolds we identify are essentially symptoms of this feature learning process carried to an extreme. Thus, LoP can be viewed as a pathology specific the Rich regime: the same mechanism that enables deep networks to learn efficient representations (data compression) eventually traps them in low-rank manifolds that hinder future adaptability.
3.1 EXPERIMENTAL VALIDATION
We validate our theory with experiments on MLP, CNN, ResNet, and ViT architectures, training them continually on a sequence of 40 5-class tasks derived from Tiny ImageNet. We track the emergence of LoP symptoms, including dead units, duplicate units, and effective rank degradation. Full experimental details are provided in Appx. B The experimental evidence confirms our intuitions (see Appx. B and Figs. [311 3.2 B.4 and B.6): depending on the architecture, we observe that degradation in the model performance is concomitant with the emergence of duplicate or frozen units, and a corresponding decrese in representational diversity. Our inquiry so far highlights two key pathways to LoP common symptoms: (I)
VIT
MLP
CNN
RESNET
1.0
1.0
0.88
0.9
0.9
0.6
0.8
0.8
0.5
Jtr de de
0.7
0.7
0.4
0.6
Oilline dead
accuracy
Online
0.0
Onliise traiitiing accuracy
0.5
Fraction d:od
Praction
Prection duplicatte units
0
20 Task
40
20 Task
40
20 Task
40
0
20 Task
40
Figure 3.1: Causes and symptoms of Loss of Plasticity emerging during continual learning. The plots illustrate (across different architectures like MLP, CNN, ResNet, and VIT from left to right) an increase in the fraction dead or duplicate units during training, coincidental with a decrease in training accuracy. These are key indicators of LoP. (Details of experimental setup in Appx.
B
MLP
Fraction of duplicated units
CNN
Praction of dupllr pllcatted linits
RESNET
Fraction of duplicated aliiated units
Fraction of duplicated units
Effective Rank
500
60
a.5
Effective Hank
70
0. 70
0.75
Effective Raak
100
50-
6.76
70
Elfective Rank
150
200
40-
70-
0.4
0.3
10.0 .o
60-
50
0.2
0.55
0.0
30
10.50
Task
Task
Figure 3.2: Co-evolution of Effective rank and LoP symptoms, such dead or duplicate units in the network during continual training. (Experimental details in Appx. B).
Emergence of duplicate features, where distinct computational units, groups of units, within network layer effectively learn to become identical or highly correlated, as a potential consequence of attempting to lower representational rank, (2) Emergence of frozen dead features, where weights and biases of a unit stop learning, as a result of attempting to maximize rank increase (leading to saturation) or to flatten the loss landscape around the current parameters.
8

---

Published at ICLR 2026
4 MITIGATION AND RECOVERY STRATEGIES
Having discussed the emergence of LoP symtoms and the existence of LoP manifolds, we now turn to strategies for preventing their formation or recovering from them if they have already occurred.
Preventing LoP with Normalization. As established in Sec. 3 one primary cause for activations becoming frozen is their pre-activations drifting into saturated regions. is therefore natural to expect that normalization layers like Batch Normalization (BN) or Layer Normalization (LN) can help prevent this. By standardizing pre-activation statistics, these layers can keep activations operating in their more dynamic, non-linear range. Even with learnable affine parameters (7, 3) after normalization, these parameters often act to maintain pre- activations within a "healthy" range. rather than pushing them into extreme values that cause saturation (e.g.. consistently negative for ReLU). This is widely supported by empirical evidence (see Appx. B Figures like 3 in Sec. 3 and Fig. B.3) ). BN and LN generally help maintain higher effective rank of representations throughout training (as seen in Fig. (.2) and concurrently prevent frozen/dead features and excessive feature duplication from becoming dominant.
MLP
CNN
RESNET
VIT
100
140
Effective Rank
Effective Rank
Effective Rank
Effective Rank
60
80
120
150
100
60
100
40
80
50
40
60
20
40
20 Task
40
Task
Task
Task
Figure 4.1: Evolution of the Effective rank during training for architectures with and without normalization layers. Dotted lines represent normalization with affine parameters. (Experimental details in Appx. B).
Recovery from LoP via Perturbations. What if LoP conditions, such widespread frozen units extensive feature cloning, have already set in? In such cases, mitigation strategies like normalization, which act proactively, may no longer be sufficient to reverse the state, indicated by cloning experiments where normalization alone doesn't 't break perfect, established clones. However, similar to our discussion on manifold stability (Section [ injecting noise into the training process be viable recovery strategy. The principle is that if the LoP manifold is unstable or saddle-like, perturbations can allow the optimizer to find an escape route. Noisy SGD and the more sophisticated Continual Backpropagation (Dohare et al. 2024) are examples of such mechanisms. We test recovery from LoP on the "bit-flipping" benchmark, an online regression task with a non-stationary target function designed to challenge model's adaptability. A detailed description of the task is in Appx. B.I.I In order to demonstrate the recovery potential of noise injection, we design an experiment where the first half of 5M samples is processed by plain Stochastic Gradient Descent (SGD) and in the second half we switch the learning rule to Continual Backpropagation (CBP). Figure [B]] clearly shows a reversal in trend when the switch happens: whereas SGD causes the representations' rank to drop and the online training loss to increase, CBP amplifies the features' rank and reduces the online training loss, effectively recovering plasticity. Additionally Fig. B.2 illustrates how aspects like rank and feature duplication are affected by the dimensionality of the model. The disparity between SGD and CBP is only increased by the model size, hinting that the model scale might aggravate the symptoms of LoP. For details see Appx. B.I.I
An interesting distinction arises when comparing artificially induced LoP (like explicit cloning) with naturally emerging LoP symptoms in challenging scenarios like continual learning. In controlled cloning setups (e.g..
9

---

Published at ICLR 2026
conceptualized in Fig. 2.1 dropout can be effective in breaking the artificially imposed symmetry and allowing units to diverge. In contrast, in our continual Bit Flipping experiments, the role of dropout can be mixed or even detrimental. While it might prevent some forms of LoP,, it can also hinder the consolidation of new knowledge or exacerbate forgetting if it too aggressively discards learned information relevant to the new task. This suggests that the optimal strategy for maintaining or recovering plasticity might be context-dependent.
5 CONCLUSION
This work has presented a mathematical framework to understand Loss of Plasticity (LoP) in deep neural networks through the lens of dynamical systems. We formally defined LoP manifolds regions in parameter space that trap gradient-based optimization. We identified two primary mechanisms for their formation: the saturation of activations leading to frozen units, and representational redundancy manifesting as cloned-unit manifolds. Our analysis reveals that these LoP states are frequently characterized by a reduction in the effective rank of representations. We investigated how architectural choices, such normalization, can mitigate the emergence of LoP,, and how perturbations, like noise injection, can facilitate escape from these restrictive manifolds, depending on their stability.
A key finding from our investigation is the inherent tension between learning objectives in static and dynamic environments. While properties conducive to good generalization on a fixed dataset, such as the emergence of low-rank features or simplicity biases, appear to be beneficial, they can also lead to a loss of adaptability when the learning process extended over time or across changing tasks. This suggests that continual learning necessitates mechanisms that actively preserve regenerate representational diversity.
This study raises several intriguing questions and suggests directions for future research. From theoretical perspective, our analysis has primarily focused on linear affine LoP manifolds. However, it remains an open question whether non-linear LoP manifolds exist and could potentially arise in practical network training scenarios. Additionally, more comprehensive theoretical understanding of the stability conditions for various types of LoP manifolds is essential. Specifically, we need to determine the precise architectural or data conditions that lead to one type of stability over another.
Numerically, the curvature of the loss landscape in directions normal to an LoP manifold is critical. Even for unstable manifold, if the negative curvatures are very slight (near flat), escaping might necessitates significant perturbations or many training steps. Characterizing these curvatures and their impact on escape dynamics would be valuable. While we have demonstrated that models can escape artificially cloned LoP manifolds with interventions like dropout or noise, the question remains: can a model, once recovered from such a state, explore the parameter space as effectively and find solutions as generalizable as model trained from a fresh, random initialization? This question is of significant practical importance, as it explores whether we can fully restore exploratory capacity after falling into a highly restricted subspace.
One of the most intriguing outcomes of this work is the connection between unit cloning, a phenomenon often studied in model compression or network analysis, and LoP in continual learning. These have been largely treated as separate fields of inquiry. However, our theoretical framework, particularly the theorems regarding cloned units, reveals a deep link, suggesting that insights and tools can be transferred between these domains. This raises questions about whether techniques from continual learning, such as noisy backpropagation or methods like Continual Backpropagation (CBP) (Dohare et al., 2024). could be beneficial in the context of model expansion or escaping cloned states in other scenarios.
Ultimately, understanding and overcoming LoP is crucial for building Al systems that can learn continuously and adapt robustly in ever-changing world. By providing mathematical characterization of some fundamental barriers to such adaptation, we aim to pave the way for the development of new architectures and learning algorithms that can sustain plasticity indefinitely, leading truly lifelong learning agents.
10

---

Published at ICLR 2026
REFERENCES
Wickliffe C Abbaham and Anthony Robins. Memory retention- the synaptic stability versus plasticity dilemma. Trends in Neurosciences, 28(2):73-78, 2005.
Shun-ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Singularities affect dynamics of learning in neuroman- ifolds. Neural computation, 1865):1007-1055, 2006.
Tudor Berariu, Wojeiech Czarnecki, Stefano De, Jörg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.0000 000.0022, 2021.
URL https: w/w.xvvaavvvv/..///vv///aaaaawiv org/ abs 2006. 00042
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr Rie- mannian walk for incremental learning: Understanding forgetting and intransigence.
In Pro- Computer Vision (ECCV). pages 532 547. Springer, Setnntett__cy_____ 2018 22008/hha //tml hwwht/twmmmmlaaeeeeeee/eeaaerraa
ceedings of the European Conference
2018. URL Chaudhry Rww.anninnnnn.nn an Wa ECCV (B_____CC________ 2018 _.018_paper.... html
Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli.
Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks. Advances in Neural Information Processing Systems, 36:35027-35063, 2023.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neurul information processing systems, 32, 2019.
Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint urXiv:2108.06322 2021. URL https://arxiv org #tt//abs/2108.0533
Shibhansh Dohare, J Fernando Hernandez-Gariia. Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8006):768-777777 -774, 2024.
Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural networks, 13(3):317-333 2000.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. International Conference on Learning Representations (ICLR), 2023.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit ander-parameteriizatin inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010. 144998 2020.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204 09560, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Under- standing plasticity in neural networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 23190-23211. PMLR, 2023. .hmll
URL www.ttsss/pprrccccnnnee mlr wttpss/wwe/////eeeeeeepp
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning- In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18828-16841 PMLR, 2022. URL wRL.htttps//pprcceciii
mlr www.spressssve. https///wwi2////hiiihiahacaa htmll

---

Published at ICLR 2026
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117740):24652-24400 226222,2433 2020.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phenomenon in deep reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 22145-32168 PMLR, 2023. URL https::/proceesings. ml mtt.....eees//2222//orr2........ html
Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635-3673. PMLR, 2020.
12

---

Published at ICLR 2026
A THEORETICAL APPENDIX
This section contains detailed proofs of theorems and lemmas, further theoretical derivations, and discussions extending the concepts presented in the main paper.
A.1 FORMAL PROOF OF THE RANK GAIN AND DECORRELATION THEOREM
This section provides the rigorous derivation of Theorem 3.1 establishing quantitative lower bound on the increase in Rényi-2 effective rank across a nonlinear layer.
Preliminaries: Hermite Expansion and Kernel Structure We consider an activation function e E L2 where (0, 1). Let {hk hazo be the orthonormal Hermite basis. The activation admits the expansion:
a (2),
G2 = E[o(Z)ha(2)]
We follow the assumptions of Theorem
3.1 where & is standardized such that E[p(Z) = (implying do 1).
and (Z)( = 1 (implying Σx21 ax
The correlation kernel (r(r) is defined for jointly Gaussian (x, Y) with correlation r:
Kr(r) Corr Corr(o(X), (()))
Using Mehler's identity, the kernel is given by:
20
Ko(r)
Wi work
(1)
where = d.. Key properties derived from this structure are:
1. wk 2 0 for all k.
2. Σx=1 wi = 1, thus (1) = and (0) 0.
3. The linearity coefficient as K% (0) W1. If ø nonlinear, as <
4. [ S |r| for E [-1,
The Universal Quartic Gap Bound The key to the quantitative rank gain theorem establishing a rigorous lower bound on the decorrelation gap G(r)= p2 (r)²
Lemma A.1 (Universal Quartic Gap Bound). Let = K' (0). The decorrelation strength is defined as
I-a
For any correlation E [] T+a
2 Ko(r)² 2 No r2 )²).
(2)
Proof.
We aim to find the kernel that maximizes (r)(r) for a fixed a, subject to the constraints 2 0 and
Σ W/
Let x = |r| E [0, 1]. Due to the non-negativity of wx:
[r) = > War <
W/|r/* Kx(x).
A=1
1=1
13

---

Published at ICLR 2026
We now seek an upper bound for Kx(x). Since x [0, 1}, have x* x² for all k 2
Kx(x) = 1 +
Wkr
0c
S or 2
W/
k=2
oz + x²(
(1)
Let Q()) az (1 a)x². This quadratic function is the extremal kernel that maximizes the magnitude for given a under these constraints.
Now we bound the gap G(r):
G(r) = = -k(r) 2 -Qx))
We must verify that x² Q(x)² 2 76.2(11 x²²)
- (2 x²). 1
22 lox (1 a)x²)²² >
For E (0,1), we divide both sides by the positive term x²² - rr):
[a + (1 a)æ]²²
1 1-a
(1 + a.)
1
We factor the LHS numerator difference of squares 1 - y = (1 Y) Y), where Y a (1 a)x.
Note that - Y- =1 - a - (1 - a) = (1 - a1 - z). The LHS simplifies to:
LHS (1 - (1- x +
(1 (1 (lta))laatt((0000
x
The inequality becomes:
1-a (1+2).
(1- (I=a)l+t+=<11000 2 I+a
Assuming ø is nonlinear (a < 1), we can divide by a). We then multiply by (1 + a);
( 21
(2+ 2 1+..
1 2a + a² + 2
d²) 2 1+
1+ 2a + a² + a²x x > 1+
2a + a² x) 2 0.
Since a 2 0 (as wi = at) and x E [0, 1), the inequality holds true. The boundary cases 0, = =1,a hold trivially.
Proof that rank increases under non-linearity We now connect the gap bound (Lemma A.1 to the increase in Rényi-2 rank.
14

---

Published at ICLR 2026
Proof of Theorem 3.1 (Rank Gain and Decorrelation Potential). Let C be d x d correlation matrix. The Rényi-2 effective rank is cr²(C) = (tr / Since C and (C(C) are correlation matrices, tr(C) tr( (C)) d.
1. Rank Ratio: The ratio of the effective ranks is:
er2 (Tz(Ka(C) crz(C)
d2 &
ICI
R(C)
d//IC
2. Total Decorrelation Gap A(C): We define the total reduction in the squared Frobenius norm:
A(C) |C || (
Since ||M1] d d Eizs M3 for correlation matrix M:
A(O) = - (o(C,))))
itj
3. Applying the Bound: We apply the Universal Quartic Gap Bound (Lemma A.1
to each term:
A(C) 2
Σ Co
Cit) TC
itj
4. Bounding the Rank Ratio: We express the rank ratio using the gap (C))
|CC|
R(C)
|
AC))
Substituting the lower bound for ACC):
|
R(C) >
C
Ya (C)
V(C) Yo |C
v(C) We use the inequality (1 (1-u)-1 1 valid for u E [0, 1). Let No TCP
We know 2 0. We verify
u 1. Since ACC) 2 768 (C), we have a S Thus < 1.
A(C) As K (C( 2 > |
0, we have A(C) < ||C|]
Applying the inequality:
(C)
R(C) 2 1+ 7
|C
5. Equality Conditions: The rank ratio equals and only (CC) = This implies yo (C) = This occurs if Vo (the activation is linear, 1) if (C) = 0 (all correlations Cig E {0, =1}).
Validation of the Quartic Gap Bound. Figure A.I (Top Row) validates the Universal Quartic Gap Bound (Lemma A. For representative activations, we plot the actual decorrelation gap A(c) = c² (c)
against the thcoretical lower bound 70c²(1 c²) over the correlation domain c E 1].n all cases, the actual gap strictly dominates the lower bound. The gap vanishes only at the fixed points E (0, +1,+1 or for linear functions (% = 0), confirming that nonlinear activations strictly reduce correlation magnitude (and thus increase rank) for any non-trivial input correlation structure.
15

---

Published at ICLR 2026
(Bottom Row) tests the Rank Gain Theorem (Theorem 3.I). We generate a continuum of correlation matrices C()) ranging from the identity matrix to dense random matrix. We compute the actual Rényi-2 rank expansion ratio R(C) cr²(K&(C)) /cr((C) and compare it the derived lower bound 7a V (C)//IC The results show that the actual rank gain consistently exceeds the theoretical guarantee. The gain is positive whenever the decorrelation potential (CC) is non-zero, demonstrating that the nonlinearity actively consumes correlation structure to generate effective rank, with an efficiency bounded by ?
Validation of Rank Gain. Figure A.1
Validation of Decorrelation Gap Rank Gain Theorams (Analytical Integration)
RetU 1ym 0.15)
(ym0,6
SILU 0.131
Tanh.cy- 0.04)
0.8
07
Bound
2.5-
3.000
2.5
3.05
3.00-
0.05
0.000
0.5
0.5
05
1.5
35
5.0
1-0
1.0
0.5
0%
1.0
0.5
0.5
1.152
132
100
1.25
1.0
1130
2.75
1.04
1300
1.000
1.00
1.00
Boaro Potartia (vib
Matra Posentil Eart
Hark Actentil tnch
Matrix Potertil 140
Figure A.1: Decorrelation and rank-gain under non-linearity Empirical analysis activations theoretical predictions about quadratic lower bound for decorrelatioin, and for rank gain potential.
A.2 RANK-PLASTICIIT TENSION: AN EMPIRICAL VALIDATION
This section shows that maximizing the de-correlation strength %o. necessitates driving the activation function into regimes where the gradient vanishes (freezing).
particular, we consider modulating activations by scale a and bias b: vb2) = ó(az + 6). and analyze the de-correlation strength y(a, b)
(E(w) (Z)))²
Var (2)) as a function shape parameters {a. b).
Note that (a, b) = [E[ (2))) Var (Z)) coincides with the derivative of the corresponding correlation
kernel at 0, i.e. >(a, b) = (0) after normalization. Thus, ((a b) and the T used in Theorem 3.1 are related by simple monotone reparameterization.
Here we empirically validate that the bias ReLU and scale for Tanh under standard Gaussian input to measure the trade-off between decorrelation and gradient availability. As shown in Figure A.2 the decorrelation coefficient % is strictly monotonic with respect to the modulation parameters for both activations. Crucially, the analysis reveals necessitates driving the activation into a "frozen" regime (dead zone for ReLU or saturation for Tanh) with probability approaching 1.0, thereby effectively eliminating the gradient flow required for training.
rank mlaktilaticctty tension: approaching maximum decorrelation (Ys
16

---

Published at ICLR 2026
To extend this beyond ReLU and Tanh, Figure A.33 visualizes the fundamental trade-off described in Sec. 3 For all activations we plot the decorrelation strength %o against the probability that the unit is frozen (P() < where 0.01). The results confirm that achieve high rank expansion capabilities (large Ye), in most cases the parameters must push the activation into regime where the gradient vanishes for the vast majority of inputs, demonstrating the tension between feature learning and trainability.
ReLU: Coefficient vs. Bias
Tanh: CoefTicient vs. Scale
ReLU: varying bias
Tanh: varying scale
0.20
1.0
strorgm Ve
0.5
0.6
Prb. PUT
0.5
0.10
0.05
0.2
-2
20
30
0.00
0.25
0.50
0.75
1.00
0.00
0.05
0.10
0.15
0.20
Blas
Scale 3
%
Figure A.2: The rank-plasticity tradeoff for ReLU and Tanh. Empirical analysis of ReLU and Tanh activations confirms that increasing the decorrelation coefficient (to improve rank) strictly requires increasing the probability that the unit is frozen (dead saturated), approaching 1.0 as 76
The Rank-Plasticity Tension: Ye P(Frozen) across Activations
ReLU
Tanh
Sigmoid
Prot Froaen PU/1 (000
Probe Fregzn PVT +3.02
0.6
Proh. Frogen PUTT
0.6
04
0.4
0.2
0.0
0.0
0.4
strength
Vp
GELU
SILU
SELU
P-bb. Fraanyy 00.01
PrBb. Foa 0.041
0.8
0.6
Frozen PNMT
0.0
(Dark=Line arrlayy, Brihht-khisaaurate)
Figure A.3: The rank-plasticity tradeoff for general activations. Empirical analysis of all activations %% vs the probability that the unit is frozen, when vary the pre-activations mean and bias.
17

---

Published at ICLR 2026
A.3 LoP MANIFOLDS: FORMAL STATEMENT AND PROOF
This section provides the formal definitions, statement, and proof for the LoP manifold theorem. Since the frozen manifold argument is self explanatory, we will only prove the cloning manifold result that is more non-trivial. First, let us introduce our neural network formalization. A feed-forward neural network is defined by directed acyclic graph G = (V, E, w). where V is the set of nodes (neurons), E is the set of directed edges (connections), representations the structure of the computational graph of the network, and
w: IE - IR, is the weight parameters of the network, which will be denoted w(u, v) for each edge (u, v) E E. Furthermore, Vn C V are the input nodes, and Vour C. are the output nodes. The post- post-activation h(v) of a node v E V is computed
Zui
if E Vin: otherwise.
h(6)=
=1 fo ) Wu.w huu)
pre-activation =(v):=
Here, Ty is the input value for input node v, Ju is the activation function associated with node v, in(v) is the set of nodes with edges towards The network output is the vector of activations h( B(Vou). With the formal pass formally define, we can now define our backward passes. Given loss function L(h( ((Voo Vou): y) comparing the network output h (Vo.t) to a target y, the back-propagation algorithm computes gradients via error signals 5()). The error signal is defined recursively:
@L(h(Von)) u)/
for output nodes,
8(0)
Σ (Eont(v)
8(u) w(n, u) f((=())): v E Vauts
where out(r) is the set of nodes receiving input from v, and fü is the derivative of the activation function fw- The gradient of the loss with respect to a weight w(u, v) is then given by aL/Ow(u, u) = 5(v) f(((()) h(u).
Network partition and base network definitions. refers to a partitioning of nodes defined as:
Let G (V, E, w) be the main network. A partitioning
- S, S: = 0 for j
Given the partitioning, we define the base network Ō = (V, E, w) where each partition is a node, the edges are union of edges between two corresponding partitions, and weights are the sum total sum of edges divided by the number of rows:
V = (SE
E E= ((), x Syn 0 + 0)
Wij
Σ
Was-
Si
VES,
We can view the base graph as "meta" graph, whose nodes are of nodes, and its edges correspond to set of edges of the main graph. While the node and edge definitions are standard, the weight definition is slightly deviating from one might expect from standard quotient graph definitions, where the weights are total sum construction and is there to ensure similarity of
without averaging. The reason for this is more specific the cloned and base networks forward and backward passes.
Definitions of Weight Manifolds Given network partitioning St.... Sp. and the corresponding base graph G (V, E, W), here are the manifold definitions:
18

---

Published at ICLR 2026
The Row-wise Equitable (RE) manifold consists of all cloned weight matrices w such that for every connection (i,j) in the base network, each block wr w[SS, Sj] all row-sums are equal:
MRE RR
v(t.j) E E, and Vr, St, holds
Σ
Wru Σ
Wy's
NES,
wES;
The Column-wise Equitable (CH) manifold, consists of all cloned weight matrices w such that for partitioned block w[S,, Sy] column sums are equal:
McE
E IR
v(i.j) E, and Ve, d' SJ. it holds
Σ
Wue
Σ
Wae'
mES;
NES,
The wise Constant (BC) manifold consists of all cloned weight matrices w such that for every block w|S; Sj], all its elements equal:
MBC E R!El V(i.j) e E, and Vu, E S,, Ve, v' E Ej, it holds Wav -Ww'et
Finally, define the family of all duplicate manifolds, that affine sub-spaces of the parameters. For any matrix with row and column equitability, w E EME n MCEE they shift the block constant manifold Mp. Formally:
D = (MD(w) | w E MRE McE))
MD(w) = {w + T MBc)
Note that all the manifolds defined above are linear or affine sub-spaces, as their constraints are all linear. There are two important facts worth mentioning that will shed more light on the upcoming theorem.
Remark A.1. Note that the dimensionality of manifolds in the family Mip are given by the number of blocks W, as opposed to number of its elements. Thus, for example if the partitioning of units forms blocks of size we would roughly expect 1/n² fewer dimensions in MDp than in the original full parameter space.
Furthermore, the following remark clarifies why define these networks as cloned networks. Because when we are on these manifolds, the clone network units form perfect copies of the base network units. Remark A.2. If W MRRE any unit in block v E Sy, the forward activations will be identical to the corresponding base unit lu(v) h(ō),, where the corresponding unit in the base network to block Sk. If we further assume W E &MRE O MME, we will have a similar property for the backwards ō(v) = ō(ū).
Let us re-state the theorem on cloning to make this section more self-contained.
Theorem A.1 (Cloned- Unit Manifold (Re-stated)). Let G= (V, E, W), denote a network that is partitioned with Si...., Sk- Furthermore, units within single block share the same activation function Vu, S₁ So Ju, and there are no edges within a block. For any input and label (x. y):
1. W E MME, then all units in the same cluster u, E Sg have identical forward activations h(u) = h(()).
2. W E MR O MOE, then all units in the same cluster u, U E Sy have identical backward activations 6(u) = 5(v). Furthermore, the gradients 0L/OW will have block-wise constant structure, such that gradients between any two units in two blocks will be equal. i.e., for any u, ud St and v' E S,, have aC/oW... aC/OW.... w.
3. If the model weights initialization any point in training touch, if they lie manifold from the family W E MD where MD E MDD given any arbitrary batches of input label pairs used to obtain subsequent model parameters Wt),,, any subsequent training parameter trajectory constrained to the same manifold:
( E MD
W(t) E MD
MD Mp, t gradient steps
19

---

Published at ICLR 2026
Proof of T'Thorremm... A. Cloned-Unit Manifold). The proof will be done as a series of inductions. First, let us assume that we have sorted the units in a topological order U1..... Un which exists because the network is a directed acyclic graph. Let us further assume that input nodes appear first in this list, and that outputs as the last edges in the list. Finally, because we assume no edges inside each block between the units, let us assume that the units in the same block are adjacent in our topological sort. Thus, for any two distinct blocks S, f Sj, we either have all nodes in S before Sj or vice versa, but cannot have mix.
Forward cloning. Jolumn-equitabiligg assumption implies identical forward for units in the same block. The induction hypothesis is that for all k, any preceding unit p S k, that belongs same partition up: Ug E S,, will have identical forward h(up)) - h(wx). Because cloning does not apply to input units, meaning that every unit is separate block, the hypothesis trivially holds for all input units k L..... d where is input dimension. Now, let us prove the induction step, assuming step k. Let S k correspond to a unit in the same block Up: Ug E S. Now, consider all the units that have incoming edges to these two units, which necessarily must appear before p. Let's consider all such units within the same block Sj. Because these units appear before k the induction hypothesis tells us that they have identical forward. Thus, the total contribution from these units to pre-activations z zUp)) and z(ux) will be proportional to sum of edge weights from units in Sj. Because of our construction of the ordering. all the units in S, that feed into S, must occur before them. Now, the column-equee assumption implies that the sum of weights from all these units to up and uy must be equal weight sum. Thus, we have proven that pre-activation contribution from units in S will be identical for Up and ug. Because we chose Sj arbitrarily and it could have been any block, we have proven that pre-activation these units must be identical z(up) = z. Since they also have identical activation function, they will have identical outputs h(up) = h(ux). This completes the induction hypothesis for forward pass cloning.
Backward cloning, We want to prove that column and row-equitability assumption implies identical backward for units in the same block. The proof strategy will be highly similar to the forward cloning case, with the key difference that our induction will be backward in our ordering, starting from latest output units and then moving in backward in the list. The induction hypothesis for step & is that, for al q > k. if they are in the same block Ug, U, E S,, they will have identical backwards d(ug) d(u,). Because output units are not themselves cloned, the induction step holds trivially for the last output nodes. Now let us prove the induction hypothesis for k assuming that it holds for all higher steps. Now, for some arbitrary block S that units in S; feed into, consider all outgoing connections from uk, Uh to the units in this block. Because of our construction of the ordering, all the units in S, that S, feeds into must occur after Sj. Thus, by induction hypothesis, all these units must have identical backwards. Furthermore, from our row-equitane eewiititabil assumption we know that total edge weights from Uj: My to these units must be identical. Thus, the summation formulas in the backward of ug and uy are similar. Finally, since S, was chosen arbitrarily, this summation is identical for all subsequent blocks, which implies the overal sum is also identical. To conclude the proof, note that because of row-equitability condition we already inherit the proof from the forward case, implying f' (((((u)) = ((()) Thus, both parts to the backward formula for ux: Ug will be identical, which proves they have identical backwards. This finishes the induction step.
Gradient cloning. This step is a straightforward consequence of the forward and backward cloning steps, and the formula that gradient of an edge from u to is simply h h()))(v). Thus, the cloning structures in forward and backward, manifest themselves as a block structure in the gradients.
Constrained training trajectory. Here, the key induction step is over the gradient steps. For step 4, the induction hypothesis is that ((t) E Mcc n MRR,, and that W(t) W()) E MBc. This trivially holds for initial step f = O. Let us prove the induction step t + 1 assuming that it holds for t. Suppose gradient at this step AW(t) is defined over the loss arbitrary number of samples {(x,, 9:)]. Because of
the induction hypothesis (tt) E MME n MEE, our earlier results imply that the gradients for each
sample @t,) will have block-wise constant structure @L:/OW(t) (t) E MBCC Thus, the sum of
20

---

Published at ICLR 2026
these gradients will also have a biock-wise constant structure (t Ow MBC. Because block-wise matrices are also row- and column--qquitable this implies that the new weights will inherit those (t + 1) (tt) + (t E McE n MRE. Finally, our parameter shift can be written as (0) = (L) + (()) (0), where (I) W(0) is block-wise constant matrix and thus ( + 1) - (O) becomes sum two block-wise constant matrices, which is itself block-wise constant
+ (L
W(L + 1) - (O)) ..... This finishes our induction step.
A.4 MODULAR CLONING PROFILES AND A COMPOSITION THEOREM
(see also Appx. A..3) This subsection formalizes a modular extension the cloning theorem in Theorem 2.1 and proves that local, modulc-level cloning certificates glue to yield cloning for the entire composed architec- ture.
Modules, interfaces, and profiles. A module a feed-forward sub-DAG GM = (VMr Fas. WM) together with disjoint sets of interface nodes IM (inputs) and Our (outputs). We allow internal nodes Vy := Var (IM U Om) and edges that connect interface nodes to internal nodes or to other interface nodes as permitted by the DAG. Let Gas = (Var. EM. WM) denote smaller base module.
A cloning profile for M relative M consists of surjections
in IM IMs. MM
out M
On Ôm:
inducing partitions Pir = (=) i Ias } and PMr interface units the same block are clones of the corresponding base port.
( : O ÖM }. Intuitively, all
We say two wired modules A + B have matching profiles if their shared interface partitions coincide after applying the wiring map W OΛ - IB, i.e.
WA-B (Paut) Pha as partitions of TB,
and dually for the reversed wiring used by backpropagation. More generally, a whole network has matching profiles if this holds on every inter-module edge set.
Module-level cloning manifold. Fix a module M with profile (Pin, ). Extend these interface partitions a partition of all nodes VM by assigning each internal node of M to the block of its corresponding base-node
in the collapsed base graph G defined in Appx. A.3 On M, define the (affine) module cloning manifold
MD
WMr: : WM E MRE O NME with respect to the induced partition of VM }
i.e., each inter-block weight submatrix is row- and column-equitable (block-wise constant up to redistribution), reusing the notation of Appx. AA.3 This generalizes the block-constant manifold MC by allowing intra-block redistribution while preserving equal in/out block-sums.
Definition A.1 (Module-level cloning certificate). A module M endowed with profile (Pin, Part) admits a cloning certificate if the following for every batch:
(MC1) Forward interface preservation. If inputs in the same block of Pin carry identical values, then for any Was E MD(M) all outputs in the same block of Poat are identical (forward cloning).
(MC2) Backward interface preservation. If the output adjoints (backprop signals) are blockwise identical on Post, then for any WM E (MDMM) the input adjoints are blockwise identical on pin (backward cloning).
21

---

Published at ICLR 2026
(MC3) Gradient closedness. Under (MC1)((MC). the per-edge gradient OC/0WM M is block-wise constant on each inter-block submatrix, hence < (. ) is tangent to MM(M) and first-order parameter updates initialized on MD(M) remain on MM (M(M).
Remark A.3 (Optimizers covered). (MC3) implies closure under any first-order optimizer whose update is (possibly stateful) scalar multiple of the local gradient each parameter and whose internal state is identical across clones at initialization (e.g, SGD, momentum, RMSProp, Adam with tied clone states). Weight decay that acts per-parameter independently may break exact symmetry; see Appx. A.3 Dropout violates (MC1) because independent masks destroy blockwise equality in the forward/backward signals.
Lemma A.2 (Module certificate from MRR M.ME). IfWas E MMR O MME for the induced partition of VM, then M satisfies (MC1))(M3).
Proof. This is the restriction of Theorem 2.1 to the subgraph Gι with its node partition: row-equitability yields identical forward values within blocks, column- -quuiaeiuitabllity yields identical backward adjoints within blocks, and dC/ WC/dWMr h 8 is block-wise constant across inter-block submatrices. Tangency of the gradient MRR O MMEE follows exactly as in Appx. A.3
Theorem A.2 (Composition theorem for modular cloning). Let a feed-forward network be formed by wiring modules (M&) with matching profiles at every interface. Suppose each Mr admits cloning certificate (Def. A. and that parameters are initialized on the product manifold IIe MD Then:
1. Global forward cloning. If the external imputs respect the imput profile of the first modules, then all internal interfaces and the final oulputs are blockwise identical according to the propagated profiles. Equivalently, the composed network is a cloned enlargement of the composed base network.
2. Global backward cloning. For any loss, if the final output adjoints blockwise identical, then all internal interface adjoints and the external input adjoints are blockwise identical according to the propagated profiles.
3. Persistence under training. The network gradient is kangent to II (M))), hence any first-order parameter update that preserves (MC3) the module level preserves the global cloning manifold and items 1-2 continue to hold at all subsequent steps.
Proof. Forward. Order modules topologically. Assume the external inputs are blockwise identical on the first-layer profiles. Applying (MC1) to the first module yields blockwise-identical outputs its output profile. By profile matching, these outputs equal the input profile of the next module, so (MC1) applies again. Induction over modules yields blockwise equality at every interface and the final outputs.
Backward. Reverse the topological order. Start from blockwise-identice adjoints at the final outputs. By (MC2) for the last module, the incoming adjoints its inputs are blockwise identical. Profile matching identifies these with the previous module's 's output profile, so (MC2) applies again. The inductive step propagates back to the external inputs.
Persistence. By (MC3). in each module the gradient is block-wiss constant on inter-block submatrices, i.e., tangent MD The product of affine manifolds is an affine manifold with tangent equal to the product of tangents, so the global gradient is tangent to Ie MD. Thus first-order updates initialized on this product manifold remain on it, and the previous two items re-apply at every step.
Remark A.4 (Coverage: modern architectures). The certificate (Lemma A.2 is satisfied by the standard width/channel/heads expansions used in practice:
22

---

Published at ICLR 2026
MLPs / Linear layers: Duplicate hidden units; enforce RE/CE by tiling weights with appropriate 1 1/iiput expansion) scaling; duplicate biases. Matches the implementation in cione_linear.
CNNs / Conv layers: Duplicate channels (inJout); file kernels with /iniput expansion) scaling: duplicate biases ceclone_covvll d, clone_conv2d) Spatial pooling is per-channel and thus profile-preserving
Normalization: BN/LN/GN with duplicated (7r, B3 and running stats per clone are profile-preserving Aalone_normalizeee
Activations and elementwise ops:
Elementwise maps are profile-preserving, wciinn_activattiee parameter-free ops are trivially preserved ccione_apreeeeeeee______________ pww.meter...... ameter_free)
ResNets: Residual addition preserves cloning provided both branches use the same profile; block-level expansions meet RE/CE at each addition.
embeddingj; (ii) Multi-head atlention via head duplication; per-head linear maps satisfy RE/CE; concatenation is prefile-preserving reshape; (iii) MLP sub-blocks as MLPs; (iv) LayerNorm is profile presele-preserving. The CloneAwareFlattee operator ensures profile-preserving reshapes between conv/linear stages.
Transformers/Vits. (i) Embbedding/pptcc-ppooo projecttiono expansions via tiling (clone_embeddingg
By contrasi, Dropout with independent masks across clones breaks (MCI)-MM2) and thus is excluded from this corollary (see also discussion in the main text).
Remark A.5 (Minimal check-list for new module). To certify a new module M:
1. Choose interface partitions (Pin Pont and extend them to VM-
2. Verify WM E MREnMcs for the induced partition (row/column equitability per inter-block submatrix).
3. Conclude (MCI)-MC3) by Lemma A.2
4. Ensure adjacent modules use matching profiles shared interfaces.
Under these conditions, Theorem A.2 guarantees network-level cloning and its persistence under training.
the implementation). The functions c clone_ (linear,, convid, conv2d, normal iizztiion on, embeddi ni,, acti ietivation model clone implement the RE/CE tiling and profile-preserving reshapes described above, while iest_accivattoo on calniicoonnnngggnneenee empirically verifies (MCI)(MC2) layer-wise via forward/backward R operator profilepreserving connector that keeps duplicated channels adjacent, ensuring that profiles match across CNN+FC boundaries.
Observation
A.1 (Connection
and
The CioneAwareFles
A.5 STABILITY OF LoP MANIFOLDS.
While Theorem 2.1 establishes the existence of LoP manifolds under exact conditions (perfect saturation, perfect cloning), in practice, these conditions might only be approximately reached during training. This leads to the question of whether near-LoP states will move back closer to the LoP manifold under gradient descent dynamics, or will they move away from it. To address this, we introduce the notion of the stability of an LoP manifold.
Definition A.2 (Stability of LoP Manifold), Let M be an LoP manifold and N&M be the normal space to M at 0 e M. The stability of M is characterized by the Hessian V2,((0) in directions normal to M:
Stable LoP: Yo N&M) 10) T V3C(O)u
(Perturbations revert LoP)
23

---

Published at ICLR 2026
Unstable LoP: Yu E N&M {0): VzC(O)o 0. (Perturbations escape LoP)
Saddle LoP: 301,00 12 E N&M s.t. of ViLee > and 0n ViLuz < 0. (Escape is direction-
dependent)
Remark A.6. Stability in the normal space to the manifold (convexity of the loss in these directions) does not imply that the loss is convex general (i.e., also within the manifold or other directions). These conditions are local characterizations of the loss landscape geometry around the manifold.
To understand the practical implications of these stability types, consider injecting small perturbation A that pushes the parameters slightly off the manifold M. If M is stable, the subsequent gradient steps -C A0) will tend to project back towards M. If M is unstable, these steps will tend to move further away. For a saddle LoP manifold, escape depends on the direction of the initial perturbation relative to the stable
cigenvectors of the Hessian in the normal space. Therefore, the strongest form of LoP corresponds LoP manifold, as it actively resists escape. An unstable manifold is the easiest to escape. A saddle manifold presents a mixed scenario, where random perturbations may or may not escape depending on the perturbation vector being in a positively or negative space orientation.
B EMPIRICAL APPENDIX
This section provides comprehensive details of the experimental setups, additional empirical results, figures supporting claims made in the main text, and visualizations.
B.1
EXPERIMENTAL DETAILS
This section outlines the experimental setup, methodologies, and general procedures employed for the empirical analysis of Loss of Plasticity (LoP) in neural networks.
B.1.1 OVERVIEW OF EXPERIMENTAL. PARADIGMS
Our investigation into LoP encompasses three primary experimental paradigms.
Continual Learning Experiments These experiments involve training models sequence of temporally independent tasks where data from previously learned tasks is unavailable. Tasks are typically formulated by partitioning the output classes of standard datasets, and for any given task f, the model is trained exclusively on its assigned class subset C. We trained our models on Tiny ImageNet, which consists of 200 classes, by creating sequence of 40 tasks, each containing disjoint subset of 5 classes. Each task is trained for 500 steps, and validation is performed periodically, resulting in 20,000 total training steps. The training protocol included optional reinitialization of model output layer weights and biases are reset to zero before starting each new task to mitigate interference.
Neural Network Cloning Experiments These experiments study the effects of neuron duplication using a two-stage training protocol. Initially, a base model is trained on target task to establish baseline performance. Subsequently, this base model is expanded by a specified factor (always fixed to two), using the cloning procedures detailed later. The expanded (cloned) model is then trained. To compare the base and cloned model, we also keep training base model the same time during this second phase. The results presented here all on the CIFAR-10 dataset, and we used 20 epochs to train the base model and 500 epochs to train the cloned model. Functional equivalence post-cloning is verified by ensuring the cloned model produces activations identical to its base, assessed via R² scores between corresponding layer activations. R2 scores, computed for each layer, measure if the mean of cloned units can explain the variance all units that block.
24

---

Published at ICLR 2026
Bit Flipping Experiments These experiments simulate slowly-changing regression problem to evalu- ate network adaptability to gradually drifting input distributions. An illustrative benchmark for studying adaptability is the 'it-flipping' experiment, an online regression task where the model receives an m-bit input vector z and must predict output y. The environment is non-stationary: a subset of f input bits are designated "flipping bits,' and at regular T-step intervals, one of these bits is randomly inverted. The remaining m f input bits are randomly sampled at each step. The target output y is generated by fixed (but unknown to the learning model) two-layer network, and two-layer MLP is trained to learn this continuously drifting target function. The complexity of the learning model is typically designed to be less than that of the data-generating process, thereby creating challenging scenario for maintaining plasticity. A target network with Linear Threshold Units (LTUs) implements h; LTU (w: x 0,) and y w + b A Linear Threshold Unit operation is defined as LTU(2)- = 1 if 2 0, and otherwise (a Heaviside step function). For the target network, the specific thresholds are 0: (m 3) S,, and S; =Ejrons 1 0.5 Wi,m4+1- Input consists of m bits plus a bias bit; f of these bits are "flipping bits" changing every T time steps (one randomly selected flipping bit is inverted), while the remaining
f bits are randomly sampled each step. A two-layer MLP with a configurable activation function is trained online to learn this target.
SGD Cest 25.0%, CBP second 75 0% stops
SCD first half. CBP second half
SCIP flest 75.0% CBP sccund 5% staps
Lons & &
Onlon Toiiiing Laua
Edfectin Builk
Towming
Edivettiv
E.
144
Stepp
106
Step
Figure B.I: Bit Flipping experiment on 5M samples, switching from SGD to CBP 2.5M samples. Low rank structures emerge during training with standard Backpropagation (SGD), but after the switch Continual Backpropagation (CBP) is able to recover representational diversity, suggesting that CBP-like training could be effective for cloning too. (Experimental details in Appx.
Effective Bank
10.0
Duplacate Praction
0.4
1.00
0.3
0.75
Loss
0.2
Eirstt Hinl ISGGDI
0.1
Secoad balf (CBP)
0.0
0.0
0.00
10
50
100
1.0 50
100
5
1.0
50
100
Width
Width
Width
Figure B.2: Bit Flipping experiment 5M samples, switching from SGD to CBP at 2.5M samples. For each of the two phases, we show the average over the last 100K steps. Duplicated structures (indicated by fraction of duplicate features at different layers/scales) emerge during training with standard Backpropagation (BP) but Continual Backpropagation (CBP) is able to decouple the cloned units. As the model width is increased more duplicate features emerge. The size of the data generating function is 100. (Other experimental details in Appx. B)
B.1.2 CORE METHODOLOGIES AND IMPLEMENTATIONS
Several core methodologies underpin our experiments.
Cloning Implementation. Our cloning implementation is modular. For each architecture, we first need to decide the "free" parameter to expand. This is the feature dimension for MLP and ViT, and channels for CNN and ResNet. After creating base and expanded model, our cloning implementation proceeds in
25

---

Published at ICLR 2026
modular fashion. The key implementation idea that allowed this modular design is the principle that the cloning profile of inputs and outputs of different modules must be consistent. For example if inputs A and B to module are assumed to be cloned, and if these are output by different modules, that module must ensure this cloning. We can think of this this design in mind, for linear layers, weights and biases are replicated according to input/output expansion factors; weights connected to cloned input neurons are scaled (e.g.. by 1/ain for an input duplication factor of ann) to maintain activation magnitudes. Convolutional layers see similar expansion of input/output channels, with kernels tiled and appropriately scaled while preserving spatial dimensions. For normalization layer, if affine features are learned, their cloning will be a simple duplication for different cloned units. The same applies modules such as patch embeddings, which require a simple duplication. Parameterized activations (e.g., PReLU) have their parameters correspondingly duplicated or broadcast. Any other units that does not have parameters, such as softmax layer or activations without parameter, will not require any particular treatment, because it has the potential to create cloning profiles that do not match. To fix this, we implemented a clore-aware flattening operation CNNs ensures duplicated channels remain adjacent after flattening to preserve structure for subsequent colle--coneeted layers.
matching cloning profile between connected modules. With
Noisy SGD optimizer introduces Gaussian noise é, ~ (0, of [l::1121) gradients gt. where the noise scale og ou X decays over time t from an initial value do. The values of do and À are hyperparameters of the optimizer. Later, we show the effect of varying them on the cloned model dynamics.
Continual Backpropagation (CBP) implemented in sre/ ut i iitt/is///e httss/cbp_oooee coopppoeptmiieer.ooom py following the Generate-ane -aiite--n---esss -es- framework, aims to maintain plasticity by selectively replacing low-utility neurons. Utility tracking involves measures like Contribution Utility ((vimait)) = |n!²] ( and Adaptable (0)
Contribution ((ugs)) 16(--R(") - HMmo.s)))) ), where h, is activation, T.")) is its running average, and terms are mean weight magnitudes. Instantaneous utilities are smoothed using an exponential moving average (p is decay rate, de is neuron age): u. put (t-1) + (1 p) with a bias-corrected version a&t) = u.9) /(1 - -pa""). ). Neuron replacement occurs for eligible mature neurons (a; > -mmaturity) with the lowest utility (a fraction Treplace of layer neurons N2). Selected neurons are reinitialized (incoming weights via Kaiming, outgoing to zero, utility/age reset). A bias correction (duext f- bnext + Woul:; i]. h2) is applied to the subsequent layer.
Winas
Metries for Analysis Our comprehensive metric suite quantifies various aspects of network behavior and plasticity loss. Single and pair feuture metrics include the fraction of "dead" neurons, identified
when N Σi 1[ < 10-7] > Taead for neuron j across N samples, with Taend = 0.95. "Duplicate" neurons are detected through cosine similarity patterns, with neurons j. k are considered duplicates if Mt H > Tcor 0.95, where activations are normalized by feature H, H.//||H..))
"Saturated" neurons are identified when the ratio of gradient magnitude mean activation magnitude |Gsgl/ max(u;; e) falls below Tsa 10-4 for more than Psat 99% of samples in batch. Representation diversity metrics include effective rank, computed as cxp(- - Σ, P; log where ps = =o,/ o,/E, oJ are normalized singular values from the activation matrix SVD; stable rank, calculated as [//////r(( iT)2 for mean-centered activations A: Cloning quality is assessed by R² scores between base and cloned model activations, computed R = 1 (ar(residuals) //aralls)//aa(taalll where the predictor is the mean of N cloned units and we measure explained variance across individual units relative to the total variance in that layer. This is done for both forward and backward activations across all layers, and numbers presented here are averages across all layers and both forward and backwards for the fixed batch that we are measuring the metrics. We also keep tracking all metrics for both base and cloned model after training to provide a comparison between the two.
26

---

Published at ICLR 2026
1.1.3 GENERAL SETUP AND PROCEDURES
Model Architectures include Multi-Layer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), ResNets, and Vision Transformers (ViTs), with configurations (depth, width, activations, normalization layer, dropout). The default configurations are as follows: Our Multi-Layer Perceptron (MLP) consists of hidden layers with 128 units each, employing ReI.U activations, batch normalization applied before activation, and 20% dropout. The Convolutional Neural Network (CNN) architecture comprises 3 convolutional layers with [64, 128, 256] channels respectively, using 3 3 kernels with stride I and padding 1, followed by 2 x max pooling operations. The convolutional features are processed by single fully connected layer with 512 units, with ReLU activations, batch normalization, and 10% dropout throughout. For ResNet, we implement a ResNet-18 variant with [2, 2, 2, 2] residual blocks per stage, starting with 64 base channels that double at each stage, using ReLU activations, batch normalization, and 10% dropout. The Vision Transformer (ViT) architecture divides input images into 8 x 8 pixel patches, which are projected to 384-dimensional embeddings and processed through 6 transformer layers with 6 attention heads each. The ViT employs an MLP ratio of 4.0 (yielding hidden dimensions of 1536), GELU activations, layer normalization, and 10% dropout for both general operations and attention mechanisms. All normalization layers include Icarnable affine parameters (7, (),, unless stated otherwise, and bias terms enabled where applicable. Default hyperparameter configurations for each architecture can be adjusted per experiment as described in the experimental setup.
Datasets and Preprocessing involve standard image classification benchmarks: MNIST (28 x 28 grayscale), CIFAR- 10 and CIFAR- 100 (32 x 32 RGB with standard augmentations like random crops and flips), and Tiny ImageNet (64 x 64 RGB). Standard train/test splits are used, For all the figures and results reported here, we used tiny ImageNet dataset for continual learning experiments, while for cloning. CIFAR- 10 was used.
Training Configuration involves optimizers like Adam or SGD without momentum and no weight decay with otherwise parameters in torch. The learning rates for the continual experiments where set to 0.001 using Adam for all architectures except for Vision Transformer, which was set to 0.0001. For cloning experiments with dropout, we varied the learning rate on a grid 0.01, 0.001, 0.0001.
Experimental Control maintained through comprehensive random seeding, which controls the random- ness across all relevant libraries (Python. NumPy, PyTorch) and CuDNN deterministic mode, We used 5 seeds for all experiments to calculate confidence intervals. Experiments utilize GPUs when available, falling back to CPUs otherwise. Metrics are typically computed at fixed epoch intervals (e.g.. every 5 epochs), often on consistent fixed data batches for reproducibility. Computationally intensive metrics like SVD may use subsampling of the features or samples to make them less expensive.
Computational Resources. For the continual learning and cloning experiments, our experimental grid consisted of approximately 2,000 individual runs (counting each random seed separately). These experiments were executed on a cluster of NVIDIA A100 GPUs, utilizing heterogeneous mix of 40GB and 80GB memory variants. The total computational cost for these experiments was approximately 10,000 GPU-hours. The bit flipping experiments and additional theory validation experiments were conducted on a more diverse set of hardware, utilizing lower-end computational nodes equipped with NVIDIA RTX 3090, V100, and RTX 2080 GPUs. This heterogeneous setup was sufficient for these less computationally intensive experiments, and the overall compute amounted to under 100 GPU-hours on these nodes. The theoretical validation figures and numerical simulations presented in the theory appendix (Appendix A.2 were generated on a MacBook. using CPU computation only.
27

---

Published at ICLR 2026
Figures details. Unless stated otherwise, all our figures report standard deviations over 5 experiment randomization, by the use of different seed. Additionally, to reduce the number of points in the plot, in Figs.
3.2 B.1 and B.2 we plot the average over time windows of 1000 steps.
B.2 ADDITIONAL FIGURES AND EMPIRICAL SUBSTANTIATION
This subsection includes placeholder figures for concepts discussed in the main text, for which specific existing figures were not available or suitable for direct inclusion in the main body.
CNN
BESNET
VIT
Saturaee Fractinn
Situanted Stuctins
g
D008
layse
D088
Dayne
D008
layer
D088
6.00
DapSosia Froction
0.00
Guma
Furce
Zome
Batch
logerr
layse
2008
bwitch
laye7
1xyer
1.4
Truus Accurvcy
Truss Accurce
S S
0008 Diyne
D088
Day
8208
layer
0008
88yer
Figure B.3: Normalization reduces the number of dead/saturated units (top row) and duplicated units (middle row), and its impact training accuracy (bottom row) across different architectures. The training accuracy displayed is calculated as the average online accuracy over the entire training length. These results highlight the role of normalization in mitigating LoP symptoms.
VIT
CNN
RESNET
0.7
Saturated Fraction
Saturated Fraction
Saturated Praction
0.9
0.00
0.3
0.8
Saturated
%
0.65
0.06
0.02
0.005
0.50 Duplicate Fraction
Duplicate Praction
Duplicate
Duplicate
Figure B.4: Evolution of duplicate/dead unit fractions and training accuracy. The colors correspond to training steps (lighter is earlier) and the points size the Training Accuracy (bigger is higher). This figure illustrates the correlation between the increase in LoP symptoms (duplicate/dead units) and training dynamics.
28

---

Published at ICLR 2026
MLP Duplicate fraction
CNN Duplicate fraction
RESNET Duplicate fraction
VIT Duplicate fraction
Duplicate fruction
0.15
Layee
Duplicate fraction
0.00
Lotre
Looer
Duplicate fraction
0
Loser
Daplicate fraction
1.0
0.8
Layoe
0.04
Lagse
Lower
Daser
0.4
0.02
0.00
0.0
Lawec
20
-0.02
30
2.0
3.0
20
30
Task
Taask
Task
Task
Figure B.5: Emergence of duplicate units layer-wise during training without normalization and no dropout.
This figure shows the increasing fraction of duplicate units as training progresses, a symptom of LoP.
MLP Dead fraction
CNN Dead fraction
RESNET Dead fraction
VIT Saturated fraction
1.0
1.0
1.0
Saturated fraction
Dead fraction
Dead fraction
Dead fraction
layee
0.6
Lager
9.6
Laove
Lager
0.6
0.4
0.2
Eapue
0.0
0.0
1.0
20
40
1.0
200
30
10
20
1.0
20
30
Task
Task
Task
Task
Figure B.6: Emergence of dead or saturated units layer-wise during training without normalization and no dropout. This figure shows the increasing fraction of dead units as training progresses, symptom of LoP.
29

---

Published at ICLR 2026
MLP
MLP
1.0
1.6
Conning
90
Cloning
Cloning
Dropout
14
Noise
Effective Rank
80
Cloning R2
0.8
0.7
Cloned (x2)
Drogout
1.0
Closed (x2))
Noise
Dropout
70
Noise
0.8
80
0.5
0.4
50
0.2
1e5
2e5 Step
3e5
4e5
le5
Ze5
3e5
4e5
265 Step
465
Step
CNN
CNN
1.00
Cloning
Cloni
0.8
Dropout
0.95
Noise
180
0.90
Cloaing
Clozed (x2)
Cloning
0.85
Dropout
(x2)
Loss
Nalse
Dropout
Effective
1:0
0.80
Noise
0.75
0.2
100
0.70
0.0
60
100000 200000 300000 400000
Step
Step
Step
RESNET
RESNET
RESNET
1.00
Base
0.8
Dropout
Cloning R22
0.95
0.6
Closed (12)
Nolse
Effective Rank
250
Cloxing
Cloned (22)
Loss
Dropout
Dropout
Noise
200
0.85
Noise
0.2
0.80
100
2e5
3e5
165
265
365
465
100000
200000 300000
Step
Step
Step
1.0
VIT
VIT
VIT
400
Cloming
1.50
Cloning
Dropout
Cloning R22
1.25
Noise
Effective Rank
0.8
Closed (x2)
1.00
Clozed (X2)
Dropout
300
* Dropoat
Noise
0.75
Notso
250
0.7
150
0
1eb
265
365
4e5
1.5
2m5 Step
365
4e5
100000
200000 300000 400000
Step
Step
Figure B.7: Cloning experiments across architectures. Configurations details: SGD with LR=0.01, Noisy SGD with 0.01 and À 0.999, and Dropout with probability 0.1. Normalization used: Batch Norm for all architectures, except ViTs, where we use Layer Norm.
30

---

Published at ICLR 2026
MLP 0.9
MLP A 0.99
MLP A= 0.999
1.75
Clloning
1.75
Cloing
1.75
Cloning
1.50
1.50
1.50
1.25
1.25
1.25
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
le5
2055 Step
365
4e5
0
165
265 Step
305
465
165
265 Step
365
465
Cloced SGD
Cloned
Clonad
Cloned 0.02
Cloned a
Rase SGD
Base 0.02
Base
Rase o- 0
SGD
Base a
Cloned = 0.01
Cloned = 0.05
Cloned d= 0,01
Closed = 0.05
Cloned == 0.01
Cloned a 0.05
Figure B.8: Effect of noise scale parameter in Noisy SGD for the Cloning MLP Experiments.
MLP
MLP 0.02
MLP 0.05
1.75
Cloning
1.75
Clocing
1.75
Cloning
1.50
1.50
1.50
1.25
1.25
1.25
Loos
1.00
LOSS
1.00
1.00
0.75
0.75
0.50
0.50
0.50
0.25
0.25
165
265 Stop
3w0
165
Ze5 Step
4
3e5
465
Step
Cloned SGD
Cloned - 1.0
Cloned SGD
Ckoad A 1.0
Cloned SGD
Cloned ) 1.0
Bose 1.0
Base 1.0
SGD
Base 1.0
Clooed
A
Base m 0.9
Base A 0.999
Base A 0.9
Base A 0,99
..... Bose A 0.9
Base = 999
Cloned
Dase 2 0, 99
Base Am
Base A 0.99
Base
..... Base A 0.99
Base = 0.0
Figure B.9: Effect of noise decay parameter in Noisy SGD for the Cloning MLP Experiments.
VIT A = 0.9
VIT A = 0.99
VIT = 0.999
Cloning
1.75
Kloning
3 Cloning
1.25
Losss
Loos
1.00
0.50
0
1e5
2e5 Step
3e5
165
0
14.5
2655 Step
365
4
1c5
2e5 Stop
365
4e5
Cloned SGD
Cloned a 0.01
Cloned
Cloned 0.01
Claned SGD
Cloned a= 0.01
Base SGD
Jase e= 0,
Base
Cloaed = 0.05
Cloned = 0.02
Cloned dre 0.05
Clozed a 0.02
Cloned a 0.05
Cloeed om 0.02
0.02
Figure 1.10: Effect of noise scale parameter in Noisy SGD for the Cloning ViT Experiments.
31

---

300000 Published at ICLR 2026
VIT o- 0.01
Coning
VEI 0.0..2
Cloning
VIT o = 0.05
1.75
Cloning
1.50
1.5
1.25
1.0e5
1.0
Loss
0.75
0.50
2
0.25
0.00
1.65
2e5 Step
3r6
465
1e5
2n5 Step
365
4e5
Step
Cloned SGD
0.0
SCD
Cloned SGD
Cloned
Reow SGD
Base = 0.0
Base SGD
Base A 0.00
.... Base SGD
Rase 2
Cloned
Cloned
Cloned 0.9
Closed A 0.39
Cloned A 0.9
Cloned A 0.99
Base A 0.9
Base A 0.99
Base
Cloned 1.00
Cloned - 0.999
Cloned A 1.0
Cloned A 0.999
Cloned A 1.0
Cloned A 0 0.999
0.909
Figure B.11: Effect of noise decay parameter in Noisy SGD for the Cloning ViT Experiments.
ML.P
CNN
VII
1.8
Cloning
Claand Despour p-d.1
Despout p-0.1
1.0
Cloning
Ciozed
2,00
Cloning
1.6
Imppott poob
1.75
0.8
Dropect p-
Loss
1.4
0.6
1.50
1.25
Ubaand Drogost peo
Baso Despect p=t
1.2
0.2
0.50
0.0
0.25
Step
Step
Step
Figure 1.12: Effect of dropout probability parameter for the Cloning MLP, CNN and ViT Experiments. Batch norm is used for the MLP and CNN models, and Layer norm for the VIT model.
MLP
MLP
MLP
75
Bono Gread spp sp!
1.50
Clored adam spd
70
1.4
Claning R2
1.02
18
1,088
3.2
Effeetiiee
80
50
100000
200000
500000
100000
200000 Stap
300000
000000
100000
200000
200000
400000
Step
Step
Figure 131 Differences between SGD and Adam optimizers in the MLP Experiments. Like SGD, Adam cannot escape the base sub-manifold, although the dynamics are different.
32

---

Published at ICLR 2026
VIT
VIT
VIT
1.00
1.5
Cloning R2
L,083
Bone
Cloand otlors
1.00
0.96
0.0
0
300000
200000
303000
400000
100003
200000
300000
400030
300000
200000
300000
400000
Step
Stop
Stap
Figure B.14: Differences between SGD and Adam optimizers in the VIT Experiments. Like SGD, Adam cannot escape the base sub-manifold, although the dynamics are different.
33


Reviews:
[{"review_id": "UAAKCxY0vz", "reviewer": "Reviewer_29Cb", "summary": "This paper introduces a dynamical-systems framework for loss of plasticity (LoP) in gradient-based learning. The authors define LoP manifolds, subsets of parameter space invariant under gradient flow, corresponding to frozen or cloned units. The key result (Theorem 2.1) shows that certain linear constraints (row/column-sum equalities) define manifolds on which gradients remain tangent, preventing escape from such manifolds. Empirical experiments use cloned networks to demonstrate these manifolds and show that noise injection or dropout can help escape them. Further experiments and analyses connect LoP to representational compression and duplicated units, and test normalisation and perturbation strategies to prevent LoP.", "questions": "- Could the authors clarify how the mitigation experiments in Section 4 directly relate to the LoP manifold framework of Theorem 2.1? Specifically, which theoretical assumptions or predictions of the theorem are being tested or illustrated by the normalization and noise-perturbation experiments?\n- Line 164: Typo. “weigh(t) decay.” \n- Line 186: Why is Adam said to violate the symmetry conditions in Theorem 2.1 if Remark 2.3 claims the opposite?\n- In Fig. 2.1, did you vary dropout rates or noise scales to examine stability/escape times from LoP manifolds?\n- Line 271: The statement “training that enhances decorrelation also creates units that are nearly always inactive or saturated” should be empirically supported or cited.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work addresses an important unsolved topic and is of high relevance to the ICLR community. The paper is well written and presented, with extensive experiments and analysis. \n\nWhile the general principle (group symmetries resulting in an invariant subspaces for GD) is known. The specific algebraic characterisation via row/column-sum constraints across block pairs, within a unified LoP-manifold framework, appears novel and is a clear, well-stated contribution.", "weaknesses": "The connection between theoretical results in section 2 and mitigation strategy experiments in section 4 is weak. The perturbations are an obvious way to recover from loss of plasticity, and the normalisation strategies seem largely heuristic. They do not directly test the assumptions of theorem 2.1 (tangency of the gradient to the manifold).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m0eyhohy3p", "reviewer": "Reviewer_bx6B", "summary": "The authors begin their work by asking what structural properties of gradient flow lead to loss of plasticity, and in response, what sorts of algorithms or architectures can be designed to mitigate this. The authors proceed by defining a notion of a loss of plasticity manifold, which is a manifold in weight space where for each point in the manifold the gradient of the loss evaluated at that point is tangent to that manifold. Or in other words, a loss of plasticity manifold is a subset of possible network from which gradient flow cannot escape. The authors prove that for manifolds characterized by dead (saturated) neurons or those characterized by duplicated features, gradient flow cannot escape, resulting in the persistence of neuron death or redundant features under gradient descent or SGD. Empirically, the authors show that noisey GD/SGD and dropout can escape these loss of plasticity manifolds and that these manifolds can arise during continual learning.", "questions": "In the bit-flipping benchmark, you switch from SGD to CBP at half-point and see recovery. Can you show sensitivity to the switch point?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "They give a mathematical formalism of loss of plasticity due to neuron death or redundant units via the lens of stable manifolds. Specifically, the authors prove that gradient flow cannot escape a loss of plasticity manifold, and this modeling of plasticity is one way of unifying two well known-correlates: neuron death and rank collapse. This also introduces a new framework for modeling plasticity loss which future researchers may build upon.", "weaknesses": "While the introduction of stable manifolds is novel to this field, the results are not necessarily as novel. For instance, the implications of Theorem 2.1 are often stated in most papers in this space, notably those focused on neuron resets or spectral regularization. Moreover, the implications of this theorem are easily observed via a simple analysis of the gradient of the loss when neuron death or duplicate features are present, which many existing works explicitly state. Similarly, the empirical results appear to reproduce many of the phenomena already observed. At some level, the paper reads like it is restating existing results and observations under a more abstract formalism of stable manifolds. \nThe paper does not analyze theoretically the stability of empirically observed LoP manifolds.\nWhile the paper is generally well written, it could make use of better sign-posting or a clear summary of contributions in the introduction. The motivating question is quite vague and only half way through the main body does the reader begin to understand the scope of contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rlG59fk5JD", "reviewer": "Reviewer_J4Px", "summary": "This paper studies loss of plasticity (LoP), the gradual inability to learn as training progresses in continual learning settings. The authors define LoP in terms of manifolds and show theoretically that GD/SGD updates that occur in frozen-unit manifolds (from saturated units) and cloning manifolds (from duplicate/redundant units) will remain in their respective manifolds, preventing subsequent learning. They also show that if a network is composed of individual “modules” (layers or blocks) that satisfy some properties, then the entire network resides on a cloning manifold. Experimentally, they compare a base network and the same network with cloned units and show that they exhibit the same behavior as training progresses, indicating an inability to escape the cloning manifold. However, perturbations like dropout and noisy SGD can help to escape the manifold because they act as symmetry-breaking. Studying CL tasks, they show that as training progresses, the fraction of dead units and fraction of duplicated units tends to increase and effective rank and performance decrease. They also show how batch norm and layer norm can help to prevent the saturation of units (and subsequent LoP), preserving effective rank, and that training with a noisy learning rule like continual backpropagation (CBP) can help networks recover from LoP.", "questions": "1. In Figure 2.1, why does adding dropout worsen performance if it manages to reduce cloning and increase effective rank?\n\n2. I understand that dropout provides symmetry breaking, but I’m wondering if you can discuss why this increases rank and decreases cloning? Dropout increases redundancy and is used as a form of regularization, so it’s surprising to me that it would have this effect. Is it because dropout is applied after cloning? Would LoP be worsened if it were applied at the onset of training?\n\n3. In Figure B.3, batch and layer norm only appear effective in removing saturated units for the MLP and duplicated for the MLP and VIT. Can you discuss why this behavior is not consistent across all architectures?\n\nSuggestion: you could perhaps use weight decay to prevent saturation of units/formation of dead units?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper formalizes and studies an interesting explanation for LoP. Indeed, low-rank connectivity is generally associated with better representational quality in static settings, but the paper shows how this breaks down in continual learning settings. It further highlights two ways this behavior can happen through frozen, saturated units and through cloned units. The paper is well-contextualized in the literature. The paper complements theory with several experiments in different architectures, showing the application of their results in explaining behavior in standard models and providing methods to prevent or escape LoP based on their analysis.", "weaknesses": "The specific contributions of the paper could be outlined more, such that specific results are highlighted. There are many simulations but it’s somewhat difficult to remember and follow what is or isn’t shown in them, especially as many figures are in the appendix and experiments are interspersed with theory. I found some parts (like Theorem 3.1) difficult to follow. Minor point, but there are many in-text typos.\n\nIn the paper, it’s mentioned that “initial growth is followed by a compression phase” and that this is “consistent with the information bottleneck principle.” However, the existence of a compression phase in neural network training is disputed and Saxe et al., 2018 showed that the central claims of Schwart-Ziv and Tishby, 2017 did not hold true in the general case. I’m thus skeptical of the claim of a compression phase in training.\n\nRelated, although the authors briefly mention the NTK and rank, the paper could link its results and study to feature learning more explicitly as it is very related to their analysis. For example, low-rank connectivity is typically related to the so-called rich learning regime, while higher-rank connectivity is related to lazy learning. Hence, the presence of frozen units or cloned manifolds may be a symptom of rich learning, and might be prevented by a network in the lazy learning regime. This could further highlight whether and how learning regime plays a role in LoP.\n\nI acknowledge that the authors do show experiments of cases where there are duplicate or dead neurons and correlate this to worsened learning. However, I’m somewhat skeptical that GD/SGD always pushes the network to these manifolds, and whether they might be other underlying factors related to LoP? For example, in Figure B.5 and B.6, the behavior across different architectures and layers varies dramatically and the fraction of duplicate units is quite small. Nevertheless, I think the contribution of the paper still stands on its own but there is more to be learned in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lQ96UDtDgd", "reviewer": "Reviewer_kAhz", "summary": "This paper provides a theoretical foundation for understanding and subsequently tackling the LoP phenomenon encountered in learning in dynamic settings. The paper pinpoints properties of solutions that inhibit the ability of the model to adapt and effectively learn in the future.", "questions": "I have the following questions regarding your work:\n\n1. Do you expect other types of manifolds to exist that cause the loss of plasticity? Apart from the frozen unit and the cloning manifold, especially given that the addition of the dropout layer is not effective consistently in non artificial experiments.\n2. My second question revolves around neighboring points to the manifolds you describe. While it is true that if an algorithm reaches an exact cloning manifold then it will be impossible to escape, what happens when we are in the points around it?\n3. Furthermore as a continuation to my previous question. What happens when you use SGD in these scenarios of inexact solutions. It seems hard for me to believe that if you reduce the batch size to 1 (even though this goes against general practice) there is not enough noise from the individual and new samples to help the model escape the neighborhood of the manifold. I am curious about this question since SGD with Gaussian noise manages to escape these manifolds.\n4. Is the fact that we arrive at a neighborhood of the manifold and not the exact manifold a potential reason for the differentiation between the behavior of dropout layers in the artificial experiment and the non-artificial one?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, has a clear structure and is easy to follow.\n\nThe paper identifies a geometrical property of the loss of plasticity manifolds that prevents gradient based methods from escaping them. Namely that the gradients are tangent to the manifold. Then they proceed to identify to categories of such manifolds that might occur while training, as the networks effective dimension decreases.\n\nThe discussion about escaping them through noise, dropout or some form of randomness is an interesting idea and apparently according to the experiments an effective one as well.\n\nThe paper also provides an explanation for the emergence of these manifolds through training, which is interesting.", "weaknesses": "The paper has no significant weaknesses in my opinion. As in the sense that it provides an explanation for the LoP in practical settings, from a theoretical standpoint. The explanation might be somewhat incomplete. Mostly I have some concerns which I put in the questions, as I would prefer to discuss them.\n\nFor the writing. In **line 109** there is a double the and in the appendix there is a double network in **line 646** and it should be in my opinion a represents in **line 647**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)