FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability–Plasticity Tradeoff
Authors: Isaac Han
Sangyeon Park
Seungwon Oh
Donghu Kim
Hojoon Lee
KyungJoong Kim
Venue: ICLR 2026 Oral (2026)
Keywords: stability-plasticity tradeoff
continual learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We present FIRE, a principled reinitialization approach that balances stability and plasticity through constrained optimization.

Abstract: Deep neural networks trained on nonstationary data must balance stability (i.e., retaining prior knowledge) and plasticity (i.e., adapting to new tasks). Standard reinitialization methods, which reinitialize weights toward their original values, are widely used but difficult to tune: conservative reinitializations fail to restore plasticity, while aggressive ones erase useful knowledge. We propose FIRE, a principled reinitialization method that explicitly balances the stability–plasticity tradeoff. FIRE quantifies stability through Squared Frobenius Error (SFE), measuring proximity to past weights, and plasticity through Deviation from Isometry (DfI), reflecting weight isotropy. The reinitialization point is obtained by solving a constrained optimization problem, minimizing SFE subject to DfI being zero, which is efficiently approximated by Newton–Schulz iteration. FIRE is evaluated on continual visual learning (CIFAR-10 with ResNet-18), language modeling (OpenWebText with GPT-0.1B), and reinforcement learning (HumanoidBench with SAC and Atari games with DQN). Across all domains, FIRE consistently outperforms both naive training without intervention and standard reinitialization methods, demonstrating effective balancing of the stability–plasticity tradeoff.

Published as a conference paper at ICLR 2026
FIRE: FROBENIUS-LSOMETRE REINITIALIZATION FOR BALANCING THE STABILITY-PLASTICITY TRADEOFF
Isaac Han'
Sangyeon Park!
Seungwon Oh'
Donghu Kim?
Hojoon Lee2
Kyung-Joong Kim
'wanngju Institute of Science and Technology (GIST)
Korea Advanced Institute of Science and Technology (KAIST)
Equal advising
{1s sac7777,,, tkadus0421, sw980907) 6gm. gist.
ABSTRACT
Deep neural networks trained on nonstationary data must balance stability (i.e., retaining prior knowledge) and plasticity (i.c., adapting to new tasks). Standard reinitialization methods, which reinitialize weights toward their original values, are widely used but difficult to tune: conservative reinitializations fail to restore plasticity, while aggressive ones erase useful knowledge. We propose FIRE, a principled reinitialization method that explicitly balances the stability-plasticiii tradeoff. FIRE quantifies stability through Squared Frobenius Error (SFE), measur- ing proximity to past weights, and plasticity through Deviation from Isometry (Dfl), reflecting weight isotropy. The reinitialization point is obtained by solving strained optimization problem, minimizing SFE subject to Dil being zero, which is contin- efficiently approximated by Newton-Schut Schulz iteration. FIRE is evaluated ual visual learning (CIFAR-16 10 with 18). language modeling (OpenWebTexx WebTeet with GPT-0. IB), and reinforcement learning (HumanoidBench with SAC and Atari games with DQN). Across all domains, FIRE consistently outperforms both naive training without intervention and standard reinitialization methods, demonstrating effective balancing of the stability-plasticity tradeoff. Explore codes and videos at project page: https://isaac777 11t777.tttubiooffrree
INTRODUCTION
Deep neural networks are typically trained under fixed, stationary data distribution (Brown et al., 2020; Podell et al., 2024). However, many real-world applications require models to adapt continually as new data and shifting distributions emerge. In computer vision, tonomous driving systems must recognize unseen traffic signs, road layouts, weather conditions that were absent during training (Verwimp et al., 2023). Large lan- guage models, trained once and deployed with fixed knowledge cutoff date, quickly become outdated unless continually updated (Ke et al., 2023). Likewise, robots deployed in dynamic physical environments must adjust their perception and control policies as the environment changes (Wotczyk et al., 2021). !). In all of these domains, a central challenge reliable adaptation to nonstationary data while preserving prior knowledge.
FIRE (Ours)
Parameter Reset
W
Wo
High Stability (Faster convergence)
High Plasticity adaptabilit))
Figure Illustration of FIRE. Solving constrained optimization problem, FIRE places weights at the intersection of high- stability and high-plasticity manifolds.
This challenge often framed as balance between two competing properties: stability, the retention of learned knowledge, and plasticiry, the ability to incorporate new information (Mermillod et al., 2013). Different research communities emphasize these properties to varying degrees. Conventional continual learning assumes limited access past data, prioritizing stability mitigate catastrophic

---

Published as a conference paper at ICLR 2026
forgetting (Kirkpatrick et al., 2017; Rebuffi et al., 2017: Rusu et al., 2016). In contrast, most foundation models and robotic agents are trained on expanding datasets where past data remain accessible (Achiam et al., 2023; Team et al., 2025), making plasticity loss central challenge (Lyle et al., 2023; Berariu et al., 2021). In this regime, stability is less about preserving past knowledge and more about accelerating adaptation new tasks by leveraging prior representations. Motivated by these real world scenarios, we study the stabilitypplasticin tradcoff under the assumption of access to past data during continual learning.
Existing approaches to mitigating plasticity loss fall broadly into two categories: regularization- based and reinitialization based. Regularization-basee methods constrain parameters or features near their initialization (Kumar et al., 2025;; Lyle et al., 2022), or enforce weight orthogonality (Chung et al., 2024). While these methods can preserve a favorable geometry for future learning, overly strong constraints slow convergence, and overly weak ones fail to prevent plasticity degradation. Reinitialization- based methods instead reset weights to earlier checkpoints when new data arrive (Ash & Adams, 2020; Nikishin et al., 2022; Lee et al., 2024a; Shin et al., 2024). Their advantage lies in avoiding interference with current optimization, often yielding faster adaptation with lower overhead. However, they also suffer from tuning dilemma: aggressive resets erase useful knowledge, while conservative ones provide little plasticity gain.
We aim to resolve this dilemma by treating reinitialization as a principled constrained optimization problem. Our approach relies on two complementary measures that capture the core dimensions of the stability-plasticit tradeoff. First, we define stability using the Squared Frobenius Error (SFE) between current and past weights, which measures the sum of squared differences across all weight entries. A smaller SFE indicates greater similarity, meaning the model remains closer its previous representations. For plasticity, prior work has linked plasticity loss to sharp loss curvature (Lyle et al., 2023), dormant neurons (Sokar et al., 2023), and low rank features (Kumar al., 202 la). but these metrics depend on incoming data and are non differentiable, limiting their use for optimization. We instead propose the Deviation from Isometry (Dfl) (Pennington al., 2017; Xiao et al., 2018), which measures how close weight matrices are to orthonormal. We show that reducing DfT simultaneously decreases curvature, prevents neuron dormancy, increases feature rank, and remains differentiable, making it practical measure of plasticity optimize. A formal proof is provided in Section 3.
We propose FIRE (Frobenius-Isometry REinitialization), which minimizes the SFE subject to the Dil being O. As illustrated in Figure 1, FIRE avoids the pitfalls of either overly conservative or aggressive reinitialization by projecting weights onto the isotropy manifold while remaining close to their previous subspace. While directly solving this constrained optimization is costly, it can be implemented efficiently with the Newton-Schulz iteration, adding less than 1% to training time.
We evaluate FIRE continual learning benchmarks in vision, language, and reinforcement learning, assuming access to past data. For vision, we split CIFAR-10, CIFAR-100, and Tiny- Tiny-ImageNet into chunks under random or class-incremental protocols using ResNet (He et al., 2016) and Vision Trans- former (Dosovitskiy et al., 2020). For language, we use a warm-start setup where GPT-0.1B (Karpathy, 2023) is pretrained on WikiText- 103 and then continually trained on a mixture of Open WebText and WikiTexx- 103. For reinforcement learning, we test continuous control with SAC (Haarnoj et al., 2018) on HumanoidBench (Sferrazza et al., 2024) and discrete control with DQN (Mnih et al., 2015) on Atari (Bellemare et al., 2013). For vision and language tasks, reinitialization is applied whenever new data arrive, while in reinforcement learning it is applied once at the midpoint of training. Across all domains, FIRE consistently outperforms naive training and standard reinitialization, showing its effectiveness as a unified solution to the stability plasticity tradeoff.
RELATED WORK
2.1
MAABLIIY-PLLSTICITT TRADEOFF
The stability-plasticii tradeoff (Mermillod et al., 2013; Kim Han, 2023) is fundamental challenge in continual learning. Stability refers to the ability of a model to preserve previously acquired knowledge and avoid catastrophic forgetting when exposed to new data. Plasticity refers to the ability of model to adapt flexibly and effectively to novel tasks. These two properties often conflict with each other since strong stability can make the model rigid and resistant to new learning while excessive plasticity can lead to the loss of past knowledge.
2

---

Published as a conference paper at ICLR 2026
Research in continual learning has therefore focused methods that balance these competing requirements such as constraining parameter updates through regularization (Kirkpatrick et al., 2017), revisiting earlier data through replay (Rebuffi al., 2017; Rolnick et al., 2019; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018; 8; Aljundi et al., 2019), or designing architectures that separate parameters across tasks (Rusu et al., 2016; Mallya & Lazebnik, 2018; Mallya et al., 2018; Yoon et al., 2017; Wortsman et al., 2020). Their aim is to develop models that can maintain previously learned skills while remaining adaptive to new experiences.
2.2 LOSS OF PLASTICITY
Deep learning has traditionally been studied under stationary datasets (Glorot & Bengio, 2010; He et al., 2015), yet real-world applications often involve non-stationary streams (Shen et al., 2024; Kumar et al., 2025a). Training in such environments leads to a loss of plasticity (Lyle et al., 2023; Dohare al., 2024; Kumar et al., 2025b), where models fail to adapt to new distributions. Prior work has identified potential indicators of this phenomenon, including dormant neurons (Sokar et al., 2023; Xu et al., 2023), shifts in pre- prevactivtsions (Lyle al., 2024), feature rank collapse (Kumar et al., 2021a), and diverging weight magnitudes (Lyle et al., 2024).
Loss of plasticity hinders not only the ability fit the training data, but also the ability to generalize to unseen data. Models trained incrementally often generalize worse than those trained from scratch (Ash & Adams, 2020; Berariu et al., 2021: Lyle et al., 2025), due to factors such as diminished gradient norms (Ash & Adams, 2020), weak feature changes (Lyle et al., 2025), and the compounding effects of small pretraining datasets or noisy labels (Leee al., 2024a).
To counteract plasticity loss, ciintializationn-aseee strategies such as S&P (Ash & Adams, 2020), DASH (Shin et al., 2024) reinitialize weights into an intermediate checkpoint, weight regularizers constrain parameters to initialization or specific subspaces (Kumar et al., 2025b; Elsayed et al., 2024; Lewandowski et al., 2024a), and spectral or rank- based approaches explicitly maintain representation quality (Kumar et al., 202 He et al., 2024). Another approach proposed reinitializing at the neuronal level, based on the utility of each neuron (Sokar et al., 2023; Dohare et al., 2024; Elsayed & Mahmood, 2024). Recent work further leverages the fact that linear networks do not suffer from plasticity loss (Dohare et al., 2024; Lewandowski et al., 2024b; Park et al., 2025).
METHOD
In this section we explain how we frame the stability-plaste pialsiiityy tradeoff as constrained optimization problem. To do this we first need two metries: one for stability and one for plasticity loss. With these two pieces place the optimization problem naturally emerges, and from this formulation we introduce our method FIRE, which provides an efficient approximation to the solution.
3.1 MEASURE FOR STABILITY
To measure stability, we propose simple yet effective metric, the Squared Frobenius Error (SFE). SFE provides a natural way to quantify the preservation of learned information by comparing an original weight matrix W with its modified counterpart W. Formally,
SFE(W, W) = I|W WIF.
which captures the element wise squared deviation between the two weight configurations. However, it remains unclear whether SFE can be used as a metric that can meaningfully capture similarity establish theoretical link between SFE and
of feature representations. To clarify this point, the normalized feature covariance, a metric widely used in prior work to measure representation similarity (Lyle et al., 2025; Yang al., 2022). In particular, show that SFE provides an upper bound on the discrepancy between the normalized feature covariances of two distinct neural networks' output features (Theorem 1).
Theorem (SFE bounds output feature covariance between two deep neural networks). Ler {W', ....... w'1 and θ = (W',... ..... W be the parameters of two depth-L feedforrwad networks with elementwise activations ae (Lipschitz constants Teae)
3

---

Published as a conference paper at ICLR 2026
For input batch Z E Raxdo we denote the layer output recursively by HQ(Z) Z and Hg(Z) = oe(H1
Let Be max [| [12] be the maximum spectral norm of the weights in layer l and By II By the product across all layers. We further define m min ( (| (Z)
> O. The normalized feature covariances of the two networks (g(Z) = H2(Z)
( Then the difference between the oulpu feature covariances of the two networks is bounded as follows:
are given by (g(2)= = (
E
16
(IC)(Z) - (Z)
Bi
Σ B
SFF(O,
m
In particular, if each activation 1-Lipschitz (Lou S 1) and each spectral norm is bounded
(By S S), then
4 Z Vese-1 1 SSE((F),G)) θ).
- -
mg
Theorem I shows that the discrepancy between normalized feature covariances is bounded by four factors: the input norm (1))) the Lipschitz constants of activation functions (Loz), the spectral norms of the weight matrices (Bn and B,), and SFE. In practice, inputs are typically normalized and have a fixed upper bound, and commonly used activation functions have small Lipschitz constants (e.g., 1 for ReLU and Tanh). Thus, the contributions from || and Lg, can be ignored. The remaining dominant factors are the spectral norms of the weight matrices (Вn and B,) and SFE. This has two implications. First, for any fixed architecture and weight scale, minimizing SFE monotonically tightens the upper bound, and thus is an effective way to preserve feature similarity between two networks. Second, the slope of the bound with respect to SFE is proportional to the spectral norms: larger spectral norms make the worst-case discrepancy more sensitive to changes in SFE. In such regimes, reducing SFE becomes even more important for stability, since even a small increase in SFE can, in principle, lead to substantial change in the resulting feature representations.
3.2 MEASURE FOR PLASTICITY LOSS
In this work, we view plasticity loss as the situation where weights learned from previous data fail to serve favorable initialization point for new data. This perspective is particularly useful when designing reinitialization strategies that partially reinitialize weights before the arrival of new data. Prior analyses of plasticity loss suggest that one of its main causes is the increasing sharpness of the loss landscape curvature with respect to new data during training on the current task, which destabilizes optimization (Lyle al., 2023). In addition, an increase in dormant neurons (Sokar et al., 2023) and a collapse in effective rank (Kumar et al., 2021a) la) have also been known to indicate plasticity loss.
However, as these measures are data-dependent and non-differentiable, they are inappropriate for the optimization objective. Instead, we propose Deviation from Isometry (Dfl) (Pennington et al., 2017; Xiao et al., 2018), an optimizable metric that closely connected to previous plasticity measure.
DII(W) = || W - -1
(2)
Our theoretical analysis reveals that minimizing Dil results in a smoother loss landscape curvature (Theorem 2), smaller number of dormant neurons (Theorem 4), and higher effective rank (Theorem 3). This supports the use of Dfl as a suitable measure for plasticity measure. In addition, minimizing Dfl also makes the weights close to isotropy, which is known favorable neural network initializations, termed as dynamical isometry (Xiao et al., 2018).
key property of
Theorem (Hessian spectral norm bounded by layerwise Dils). We assume that
the inputs are whitened, so that the empirical covariance Σz = 1 TT Z is approximately the identity: Σz 1z m
for every sample i and relevant parameter vector u. the Hessian norm satisfies [V(4(u)22 3, while the gradient norm is bounded by V V.ll(u)]2 %.
4

---

Published as a conference paper at ICLR 2026
finally, we focus a fixed ReLU activation pattern at the point of interest, that the network is piecewise linear in that region. In particular, each diagonal gating matrix arising from the ReLU has operator norm S L.
Let Uh = =+ / Df)(W&)), then the Hessian spectral norm can be bounded as follows:
V) & 8
&III UJ k=1søk
27
EI Un-
Theorem 3 (Dfl controls effective rank). Let Z E Raxa and W E RRaxb be input and weight matrix, respectively, & ZW be a feature matrix. Let Σn = + "ZZZ Z be the empirical covariance of the inputs and let W QS be the right polar decomposition of W, where Q E Raab has orthonormal columns. Then rh d =rank(Q) (Q" E2Q), and (rank,(T)) () be defined from the nonzero singular values ( by
2 Da > denote the positive eigenvalues of Ez with
sranks( min k: (Eiz ( (1-0))21-8)) 21-
Then /DII(W) gives lower bound the srank left inequality below. If additionally Σz = I, the bound simplifies to the right inequality below.
8)
(1 (1-8)d
sranks($)
s
2
(3)
8
+1
8 1+e +1
Theorem (Minimizing Dil increases neuron activity score). Let RRabb and let a be positie-hoomoeenoous (a = aa((t for 2 0). Assume the input vector (O, la) (isotropic Gaussian). For neuron j with column wj of W, define activity score of neuron j as E. [a lffe Df] < 1, then for all
8j t E:. (s(lo((s.w)))
1+E
1+E 8 S V 1-E"
Sokar et al. (2023) classified neurons with s, of dormant neurons requires increasing the activity scores sy. Theorem 4 states that minimizing Dfl
dormant neurons. Hence, reducing the number
increases the lower bound Iife < 8z.. Note that minimizing Dfl also decreases the corresponding
upper bound. This is particularly meaningful because the neuron activity score sj a relative measure, defined as an activation normalized by the mean activation within the same layer. Consequently, is impossible to increase sy simultaneously, since they are normalized by their average. To reduce dormant neurons, is therefore critical to reduce the discrepancy in activations across neurons rather than uniformly scaling them. Theorem 4 supports this perspective: minimizing Dfl tightens both the lower and upper bounds on 8y, thereby limiting the score discrepancy between neurons and effectively reduces dormant neurons.
We provided detailed proofs in Appendix A. Theorem 2, 3, 4 demonstrates that reducing DfI is directly associated with maintaining smoother loss landscape, a high effective rank, and large number of active units.
3.3 BALANCING BETWEEN STABILITY AND PLASTICITY
To achieve low DII while minimizing the loss of information, we formulate the problem as a constrained optimization. Specifically, minimize the SFE between the original weights W and their orthogonalized counterpart W, subject the orthogonality constraint W TW I. which is equivalent to requiring DII(W) = W W 1| 0.
min |W W s.t. W WI W
(4)
This formulation is mathematically equivalent to the well-studied Orthogonal Procrustes Problem (Schönemann, 1966), whose solution can be expressed in closed form via the polar decomposition:
W W W(W (T)= w)-
(5)
5

---

Published as a conference paper at ICLR 2026
While the optimization itself is classical, our contribution lies in leveraging this operation principled mechanism to balance stability and plasticity in neural networks. In particular, we reinterpret equation 5 projection that simultancously drives the spectrum of W toward isotropy (low Dff) while maintaining stability (low SFE). We provide a derivation of Equation in the Appendix A.
3.4 APPROXIMATING THE SOLUTION
Directly computing W exactly can be expensive, we efficiently approximate it using the Newton- Schulz iteration, making the approach scalable to large networks.
Our method, FIRE, orthogonalizes neural network weights after training on the current dataset but be- fore learning on new data, using the Newton-Schulz iteration (Shown in Algorithm 1.) to efficiently approx- imate the above solution. Specifically, given weight matrix W E Rm we apply the Newton-Schulz up- date defined X a. + X&) with con- stants = 1.5 and b = - 0.5, where Xo W/||W [[. This iterative process progressively drives the singular values of W toward 1, thereby enforcing approximate orthonormality.
Algorithm 1: (Pytorch-like)
Iteration
X: swo-dimensional matrix number of iteration
a, b (1.5, -0.5)
x - x x.nerr
for in range (N)
A X.T = = (XOA)
return X
In convolutional layers, the update is applied kernel-wise along the spatial dimensions, ensuring that each convolutional filter is orthogonalized independently.
4 EXPERIMENTS
To demonstrate the effectiveness of FIRE, we evaluated it on three settings: continual visual learning, continual pretraining LLMs, and reinforcement learning.
4.1 CONTINUAL VISUAL LEARNING
In continual visual learning experiments, we evaluated FIRE on various the CIFAR-10 dataset with ResNet-18, the CIFAR- 100 dataset with ViT-Tiny, and the Tiny ImageNet dataset with 16.
pairs:
To evaluate both the ability of FIRE to recover plasticity and its capacity to restore performance after a reset, we compare against two representative reset-based baselines: S&P (Ash Adams, 2020), and DASH (Shin et al., 2024). Also, to evaluate the impact of regularization speed, we adopt Parseval regularization (Chung et al., 2024), which constrains the weights to remain close to orthogonal, representative regularization based method. Node-resetting methods such as Continual Backprop (CBP) (Dohare al., 2024), Seff-Normalized Resets (SNR) (Farias & Jozefiak, 2024), and Recycling Dormant Neurons (ReDo) (Sokar et al., 2023) are employed as baselines. In addition, we use the Muon optimizer (Jordan al.) as a baseline, as it also employs the Newton-Schulz iteration. While FIRE minimizes the Deviation from Isometry (DIT) when resetting, Parseval regularization enforces the same constraint continuously throughout training. Thus, both FIRE and Parseval regularization share the same underlying optimization objective. Detailed experiment settings for continual visual learning are provided in Appendix E.
the convergence
baseline. We also used L2init (Kumar et al., 2025b) a baseline. which is
Following Ash & Adams (2020) and Lee et al. (2024a), we evaluate our approach in the warm-start setting, where the model is first trained on a subset of the dataset and then on the full dataset. Since plasticity loss is most severe when the subset ratio is small (Lee et al., 2024a), we warm-start with only 10% of the data before continuing on the entire dataset. As shown Figure (a), FIRE provides consistent performance gains across all three benchmarks. In particular, and Tiny ImageNet with VGG-16, FIRE outperforms all baselines, demonstrating a strong ability to recover plasticity. On CIFAR-100 100 with ViT- Tiny, the improvement is less pronounced. However FIRE still outperforms all other baselines except DASH, and competitive to S&P. In this setting, DASH, which employs dependep shrinking strategy, proves especially effective, suggesting
CIFAR- 10 with ResNet-18

---

Published as a conference paper at ICLR 2026
Parseval
DASH
ReDo
L2inZ
Base
CIFA--1 (Benhan-10)
Trny Imagammet (WGG-16)
0.85
(Put
JUl wtwp:r/ww......... pocll
Accurs))
Tist RCCUPDCY
Accuracy
(a)
0.30
0.30
MATE starl
PUT2E start (8
0.75
100
25
50
75
100
0.10
50
100
Epnch
Epoch
Epech
CIFAR-1D (Reshlet-
CIPAR-LOG (WITTinn)
Thny Imaperet (VGG
0.80
0.70
Nost Accurocs
Accurccy
(b)
0.60
0.20
0.40
100
200
300
400
500
600 700 800 9003020
0.10
300
200
300
400
500
600
700
800
9003000
0.10
100
200
300
400 500
660 700 800 9001000 E2007
Epech
Epoch
CIRAA 100 MVTliny)
Tiny imagenet IVGG 161
0.40
Acuuncy
0.25
0.30
(c)
0.10
0.03 200
400
600
800
1003
1200
1400
1600
1800
2000
200
400
600
800
1000
1200
1400
1600
1800
2000
Spoch
Epech
Figure 2: Continual visual learning results. Warm-start setting (a): training begins with only 10% of the data before continuing on the full dataset. Continual setting (b): the dataset is revealed in ten stages, expanding from 10% to 100% in 10% increments. inasemnnreeeetal setting (c): new classes introduced over 20 phases, with equal number of classes added at each phase.
that guidance from data can be beneficial when reinitializing transformer architectures. Notably, Parseval regularization and 12iiit converges more slowly than FIRE, particularly on CIFAR-10 with ResNet-18 18 and Tiny ImageNet with VGG-16. This suggests that continuously enforcing the constraint during training can hinder convergence, leading to longer training and incurring additional computational cost.
To examine whether these findings hold in setting where data continuously added, which is more realistic and natural setting, we evaluate FIRE in the continual setting (Lee et al., 2024a). Here, training is divided into stages, starting with 10% of the dataset and adding additional 10% each stage. In this way, data gradually expand from 10% to the full 100%. As shown in Figure 2 (b), FIRE delivers consistent gains across all datasets. The improvements are particularly pronounced on CIFAR-I0 with ResNet-18 and Tiny ImageNet with VGG- while on CIFAR-100 with ViT-Tiny it achieves performance comparable the best alternatives. In contrast, full reset and DASH suffer sharp drop immediately after each reset, and although S&P avoids such drops, its performance remains suboptimal compared FIRE. In contrast, FIRE incurs only slight negligible drop. suggesting that it successfully balances stability and plasticity, thereby achieving high performance with minimal drop performance.
To assess the effectiveness of FIRE under large distribution shifts, we conducted experiments in class-incremennal learning scenario, which is widely used setup in the continual learning literature (Rebuffi et al., 2017; Dohare et al., 2024; Lewandowski et al., 2024b). New classes were gradually introduced at regular intervals. The training process was divided into 20 phases, with an equal number of classes added in each phase. Since CIFAR-10 does not contain sufficient number of classes for this setting, we exclude it in this experiment. Figure (c) reports the results in the class-incrementee setting. Consistent with earlier findings, FIRE shows strong performance without exhibiting
7

---

Published as a conference paper at ICLR 2026
FIRE (Ours)
Shrink Perturb
Full Reset
Base
Best checkpoint
30k iterations
60k iterations
Validation Loss
Validation Loss
Validation Loss
3.00
3.0
3.00
5k
10k lterations
15k
10k lterations
15k
10k
15k
lterations
Figure 3: Continual pretraining of GPT-0.1B. Models are first pretrained on WikiText- 103 and then continually trained From left to right, results correspond to models initialized from the best checkpoint during pretraining, from 30k pretraining iterations, and from 60k pretraining iterations.
dataset consisting of mixture of Open and WikiText- 1033.
a performance drop after resets by effectively balancing stability and plasticity, and full reset and DASH show sharp drop after reset while S&P show suboptimal performance.
Node-resetting methods such as CBP, SNR, and ReDo show poor overall performance. This is consistent with Lee al. (2024a), which found that methods aiming to improve plasticity by maintaining trainability provide only limited gains in generalization. The Muon optimizer also performs poorly overall, suggesting that periodically reinitializing weights using the Newton-Schut iteration (FIRE) is substantially more effective than applying this iteration to the gradients (Muon).
4.2 CONTINUAL PRETRAINING OF LLMS
Setup. We also tested FIRE in the continual pretraining of LLMs. We first pretrained a GPT-0. IB model on WikiText-109 103 and then trained on combination of the second phase, we used the best, 30k, and 60k checkpoints from initial pretraining to examine how plasticity loss worsens beyond the best checkpoint and how effectively FIRE mitigates this degradation at different stages. We present the detailed settings for the LLM experiments in Appendix E.2.
and WikiTextt 103. For
Results. As shown in Figure 3, the gap between the base model and full reset narrows pretraining progresses, since the base model's validation loss increases with longer training. This aligns with prior findings that plasticity loss becomes more severe as pretraining duration grows (Ash & Adams, 2020). While S&P improves performance by moving parameters toward intermediate trade-off points between stability and plasticity, it remains suboptimal compared FIRE, which achieves a more principled balance. Notably, FIRE was applied without any tuning, using a fixed 5 iterations, whereas S&P was carefully tuned over varying reinitialization degrees. Moreover, while the performance of the base model deteriorates with longer pretraining, FIRE maintains strong performance even when initialized from the 60k checkpoint. This demonstrates that FIRE can effectively balance the plability-llastiiiyy trade-off even under severe plasticity loss.
In addition, unlike in continual visual learning (Section 4.1), full reset performs poorly in this setting. The main reason is the lack of stability inherent full reset. Consequently, the full resetted model cannot outperform the base model, even though the base model itself already suffers from plasticity loss. In other words, the instability introduced by erasing all past information outweighs the potential benefit of restoring plasticity. These findings indicate that full parameter resetting is not an effective strategy for mitigating plasticity loss in continual pretraining of L.Mss. Instead of providing stable improvement, it wastes useful prior knowledge and leads to extreme inefficiency, making it an impractical choice in this setting.
4.3 REINFORCEMENT LEARNING
Setup. Finally, evaluated FIRE in reinforcement learning. We evaluated the effectiveness of FIRE a high Replay Ratio (RR) setting (Nikishin et al., 2022; Sokar et al., 2023), where loss of plasticity severe and acts critical bottleneck for sample efficiency. For comprehensive evaluation,
8

---

Published as a conference paper at ICLR 2026
DON FIRE (Ours)
DON Shrinz Perturb
DON Plasticty Injection
DON Reset
DON
Asterix-y4
BeamRider-ve
Demonnttaackvv
2300
5000
5000
4000
Average rdum
Bae Avergge retum
1500
1000
3000
Averg0
2000
2000
500
1900
0.0
2.5
2.0
0.0
EnN stop
1ub
Eny S160
1wb
Eny step
100
SAC FIRE (Ours]
SAC Shrink Pertura
SAC Plasticity injection
SAC
Reset
SAC
h1 11l-walievU
h1 n1-balance simple-v0
h2-runivü
800
600
200
600
Arengg retum
Aevenge wetum
150
200
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
0.0
0.2
0.6
0.8
Erw stop
stop
Erty stop
Figure 4: Reinforcement learning results. Discrete control with DQN on three Atari environments that suffer from severe plasticity loss (a) and continuous control with SAC on three HumanoidBench tasks (b). The black dashed line indicates the point at which reinitialization is applied.
consider both continuous and discrete control environments. For discrete control, we focus on three Atari 2600 (Bellemare al., 2013) games (Asterix, BeamRider, and DemonAttack), which have been reported to suffer from severe plasticity loss (Sokar al., 2023). We use standard nature CNN with DQN algorithm (Mnih et al., 2015). For continuous control, we choose three primary tasks from HumanoidBench (Mnih et al., 2015); balance, walk, and run. We use SimBa (Lee et al., 2024b) with SAC algorithm (Haarnoja et al., 2018) our baseline, whose replay ratio has failed to scale beyond 1 without resets (Leee et al., 2025). We considered three baselines: full reset, Shrink and Perturb (S&P) (Ash & Adams, 2020; D' Oro et al., 2022), and Plasticity Injection (Nikishin et al., 2023). To climinate performance differences caused by randomness before reinitialization, we reinitialized the network using the same checkpoint and replay buffer.
Results. As shown in Figure 4, FIRE achieves superior or competitive performance across environ- ments compared S&P, Plasticity Injection, and full reset. In DQN, FIRE consistently outperforms S&P, surpasses full reset in Asterix, and remains competitive in other environments. Although S&P provides a slight improvement in convergence speed, it is still suboptimal relative to both full reset and FIRE. In continuous control tasks, S&P performs competitively, but it falls short of FIRE in all Atari environments. Plasticity Injection, which introduces additional parameters balance stability and plasticity, shows poor performance across discrete and control tasks. These results suggest that manually tuning hyperparameters to balance stability and plasticity is less effective in visual reinforcement toaminggwwerr plasticity loss is particularly severe than our principled approach, FIRE, which explicitly balances the two.
4.4 ABLATION STUDY
To better understand the underlying factor of FIRE's strong performance, we conducted an ablation study. To verify whether FIRE indeed effectively balances stability and plasticity, we evaluated the stability metric (SFE) and the plasticity metric (DfI), and compared FIRE against reinitialization baselines. In addition, we measured the loss landscape curvature with respect to upcoming data immediately after reset, to examine whether theoretical findings also reflected in practice.
As shown in Figure 5 (b), FIRE achieves the lowest Dfl while maintaining the lowest SFE, which suggests that FIRE successfully balances stability and plasticity in practice. Moreover, FIRE produces a smoother loss landscape compared to S&P, while still preserving a lower SFE. This indicates that theoretical insights on Dfl and loss curvature indeed manifested in practice. Although DASH

---

Published as a conference paper at ICLR 2026
40% WL
Tiad Acturey
Riai Toxt Accursco
0.35
(a)
start
0.20
0.75
10
20
30
20
20
30
40
0.30
10
20
Nur Eieratoon
Number Berations
Number Iterations
"quuree Normalbed froe cnta
1.0
20
(b)
600
400
0.4
200
R2
Figure 5: Ablation study results. Final performance of FIRE with varying numbers of iterations for Netwon- Schoul algorithm (a). Comparison of FIRE and baselines in terms of loss curvature (maximum eigenvalue of the Hessian), plasticity (Dff), and stability (normalized SFE) (b).
is particularly effective in smoothing the loss landscape, it also exhibits the highest SFE, which may contribute an erasure of useful learned knowledge, thereby leading instability after reset.
In addition, we evaluated FIRE with various hyperparameters in the warm-start setting to assess its sensitivity. The only hyperparameter FIRE is the number of iterations for the Newton-Schulz iteration. As the number of iterations increases, we obtain a more accurate estimate of the solution to the constrained optimization problem discussed in Section 3.3. Therefore, our interest is identify the minimum number of iterations that provides sufficiently accurate estimate of the solution to yield performance benefits. shown in Figure 5 (a), FIRE highly robust to the number of iterations and already provides strong performance gains even with as few as five iterations.
CONCLUSION
In this work, we addressed stability-plasticity trade-off, which is the long-standing problem in continual learning, by introducing FIRE. By approaching stability-plasticity tradeoff constrained optimization problem, FIRE enables a principled reinitialization without heavy hyperparameter tuning. Across continual visual learning, reinforcement learning, and language learning bench- marks, FIRE achieved superior or competitive performance, underscoring the importance of effective stability-plasticity management for advancing continual learning.
The main limitation of our work is the assumption of access past data. Since our focus is on balancing stability and plasticity when such access is available, we did not evaluate FIRE under restricted data scenarios. Future work should, therefore, examine FIRE under restricted access past data. In addition, we only used relatively small models for continual pretraining of LLMs. Evaluating FIRE on the larger models and applying FIRE not only pretraining, but also continual fine-tuning of LLMs can be a promising direction for future works.
REPRODUCIBILLTY STATEMENT
We provide hyperparameter configurations and implementation details of our experiments in Ap- pendix E. The core algorithmic part of our method is described in Algorithm The proofs and assumption of theoretical works provided in this paper are described in Appendix A.
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) BRS-2025-169002990 This research was financially supported by the
10

---

Published as a conference paper at ICLR 2026
Ministry of Trade, Industry and Energy (MOTIE) and Korea Institute for Advancement of Technology (KIAT) through the "International Cooperative R&D program (Grant No. PO028435). This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) N0.2201--0-0144 Artificial Intelligence Graduate School Program (GIST)). We appreciate the high-performance GPU computing support of HPC-AI Open Infrastructure via GIST SCENT.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.08774, 2023.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances neural information processing systems, 13:3884-3894, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of artificial intelligence research, 47: 253-279, 2013.
Tudor Berariu, Wojeiech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity neural networks. arXiv preprint arXiv:2 006400 2021.
Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning, In Forty-second Interna- tional Conference on Machine Learning, 2025. URL https: https::t//ppenaeeeeee http::::////peeeeee...... www.eviwwwwweeeeeeeeeeeeeeeeeeee nwt/forumm forum?
iddhErdff FTs.u.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint 1rXiv:1812.00420 2018.
Wesley Chung, Lynn Cherif, David Meger, and Doina Precup. Parseval regularization for continual reinforcement learning. Advances in Neural Information Processing Systems, 37:127937-127967, 2024.
Shibhansh Dohare, Fernando Hernandez-Gareia, Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-7744 068,774,22 2024.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Mohamed Elsayed and A Rupam Mahmood. Addressing loss of plasticity and catastrophic forgetting in continual learning. arXiv preprint arXiv:2404.0078. 2024.
Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. arXiv preprint 24X10:2407.01704 2024.
11

---

Published as a conference paper at ICLR 2026
Vivek F Farias and Adam D Jozefiak. Self nelf-normalized resets for plasticity in continual learning. arXiv preprint arXiv:2410.20099, 2024.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, JMLR Workshop and Conference Proceedings, 2010.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic; Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Qiang He, Tianyi Zhou, Meng Fang, and Setareh Maghsudi. Adaptive regularization of representation rank an implicit constraint of bellman equation. arXiv preprint arXiv:2404.1275: 2024.
Peter J Huber. Robust estimation of location parameter. In Breakthroughs in statistics: Methodology and distribution, pp. 492-518. Springer, 1992.
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordann github. iofposts/muon, 6.
Andrej Karpathy, NanoGPT. https: /ttp://gttttbbtt http://gggtuuu...ooaaaaaaaaaaaat cww./kaappaattyyaaaG.... 2023.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In The Eleventh International Conference Learning Represen- lutions, 2023.
Dongwan Kim and Bohyung Han. On the stability-plasticity dilemma of class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20196-20204, 2023.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barr Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parrameterizaionn inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021a.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint 12Xi::2122004116 2021b.
Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, Ben- jamin Van Roy, et al. Continual learning as computationally constrained reinforcement learning. Foundations and Trends0 in Machine Learning, 18(5):913-1053 2025a.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization. In Conference on Lifelong Learning Agents, pp. 410-430. PMLR, 2025b.
12

---

Published as a conference paper at ICLR 2026
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In International Conference Machine Learning, pp. 26416-26438 PMLR, 2024a.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024b.
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. arXiv preprint (ll.:252..12800 2025.
Alex Lewandowski, Michal Bortkiewicz, Saurabh Kumar, András György, Dale Schuurmans, Mateusz Ostaszewski, and Marlos C Machado. Learning continually by spectral regularization. arXiv preprint arXiv:2406.000 168/ 2024a.
Alex Lewandowski, Dale Schuurmans, and Marlos C. Machado. Plastic learning with deep fourier features. arXiv preprint arXiv:2410.20634 2024b.
David Lopez-Par Papez-Pazzeee and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference Machine Learning, pp. 23190-23211. PMLR, 2023.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18766 18762, 2024.
Clare Lyle, Gharda Sokar, Razvan Pascanu, and Andras Gyorgy. What can grokking teach us about learning under nonstationarity? arXiv preprint arXiv:2507.20057. 2025.
Guozheng Ma, Lu Li, Sen Zhang, Zixuan Liu, Zhen Wang. Yixin Chen, Li Shen, Xueqian Wang. and Dacheng Tao. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages, arXiv preprint 10.00:2310.07418, 2023.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European conference on computer vision (ECCV), pp. 67-82, 2018.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The gating the continuum from catastrophic forgetting to age-limited learning effects, 2013.
dilemma: Investi-
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540):599-333 2015.
Exgenii Nikishin, Max Schwarzer, Pierluca DrO,o, Pierre-Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp- 16828-16847 PMLR, 2022.
Evgenii Nikishin, Junbyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36:37142-37155 2023.
13

---

Published as a conference paper at ICLR 2026
Sangyeon Park, Isaac Han, Seungwon Oh, and KyungJoong Kim. Activation by interval-wise dropout: A simple way to prevent neural networks from plasticity loss. In Forty-second International Conference on Machine Learning, 2025.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, Advances neural information processing systems, 30, 2017.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.
Peter H Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10, 1966.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation, 2024.
Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, and Jose M Alvarez. Step out and seek around: On warm-start training with incremental data, arXiv preprint wxXvv2246.004488 2024.
Backrok Shin, Junsoo Oh, Hanseul Cho, and Chulhee Yun. Dash: Warm-starting neural network training in stationary settings without loss of plasticity. Advances in Neural Information Processing Systems, 77:43300-43340 2024.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, pp. 32145-32168 PMLR, 2023.
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean- Caan-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025.
Eli Verwimp. Kuo Yang, Sarah Parisot, Lanqing Hong, Steven McDonagh, Eduardo Pérez-Pellitero Matthias De Lange, and Tinne Tuytelaars. Clad: A realistic continual learning benchmark for autonomous driving. Neural Networks, 161 (11::6696999 669, 2023.
Maciej Wolczyk, Michal Zając, Razvan Pascanu, Lukasz Kuciñisk, and Piotr Miloš. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 14:28496-28510 2021.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances neural information 2020.
processing systems, 33:15175
Lechao Xiao, Yasaman Bahri, Jascha Sohl- Diek-Dicstten,, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and mean field theory of cnns: How to train 10,000-layer vanilla convolu- tional neural networks. In International conference on machine learning, pp. 5393-5402. PMLR, 2018.
14

---

Published as a conference paper at ICLR 2026
Guowei Xu, Ruijie Zheng, Yongyuan Liang. Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, et al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668 2023.
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2033.03466 2022.
Jaehong Yoon, Eunho Yang. Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint wxXiv:/708.0154,, 2017.
15

---

Published as a conference paper at ICLR 2026
A
PROOF OF THEOREMS
Proof of Theorem
Proof. Unless explicitly subscripted, || spectral (operator) norm. We also recall
denotes the Frobenius norm
|| Ind and
||
||| denotes the
SFE(O,
Σ j=1
|W - W 1/2.
We will use two elementary facts.
For any F, F with m = min FI. [1]
> 0,
F
F
2 m
|F
Fll.
||
|
If arbitrary two matrix U and V satisfies
|U]]|
[
then
|IUT
VV || U VIl.
Combining these two facts with U = F|| and V = F/ F|| yields
FFT FFT
I m
|| Fl1
(6)
|1 |11
Applying equation with (z) and F (7). and me m= [
gives
[[Cg(Z) - c2 < me
(7)
Introduce the hybrid outputs H ((s)) (Z): the first j layers use θ and the remaining + 1,...,, layers
Ô. Note that H (50)(() (Z) (Z) and (( (Z) ( Then
(Z).
HZ(Z)- - Hg Σ
H (H)))()) H (()))))(2))
1=1
so by the triangle inequality,
H
H (()))()) H ((s)-1)(Z)l)
(8)
Each summand differs in only the j-th layer weights. Let X,-1 H 1-1 (Z) denote the shared input fed to layer in both hybrids. Consider the backend subnetwork
T = oz i
(Y W 141)...... W 9)
For arbitrary Y1 and Y2, its iaput-Lipschin constant is
-, s (
Low [lW*|12) ||8 - -Y2l-- (9) II k=j+1
Hence
[ H ( <
II
Lov || (l2) La, |X II W
(10)
k=j-1
16

---

Published as a conference paper at ICLR 2026
Mcanwhile,
| = | [Z)||
|
II
Lon
||
(11)
k=1
Combining equation 8, equation 10, and equation
yields
Lou
Σ
Bk)
|
W'
Right
By Cauchy-Schwarz,
1/2
BI
II kgj
Bx |
W1
s
Bi
Σ
VSFE(G,
B3)
j=1
Therefore,
i
1/2
(H6(Z)- H1
I ||
II
Loz
Bi
Σ
1
V SFE(O,
(12)
B3
j=1
Substitute equation 12 into equation 7:
1/2
4 || mg
[[Cg(2) - Cg (Z))
TIoN
B
Σ
/SFF(O, 6).
By
j=1
Squaring both sides gives
16 1|
1C& -
- 0gll <
Lox
(B
Σ B²
SFE(O, ²).
m?
If Log < and B, S S for all j. then (Bf)
2 Σラ BJ²² < 2 Therefore
4 | V v SFF(O,
C[g C/ll
m
Network, Loss, and Notation. Let Z E xdo be the input matrix and Wi E Rd 1xds (k 1, 1...., L) be the weight matrices. Define
A HA-IWE E Raxd,
H = p(At) (p= RDRR), U: A₁ gaxds,
with Ho = Z. The empirical risk is
1 Σ ((us),
((WI.)) n
u; E Ed.
Let 0 vec(Wi,.... W,) E R and Hg V2C RPXP, We use || ||2 for spectral norm and
|| - || F for the Frobenius norm.
We denote the maximum eigen values Xmax
17

---

Published as a conference paper at ICLR 2026
Deviation From Isometry (DfI). For matrix W, DiI(W) | W I... For each layer, set
P 1 DDI(W)), ah :- Vik.
First, let us examine the lemmas required for the proof of Theorem
Lemma (Dfl controls the spectral norm), For each layer k, |W S uy and || I ak.
Proof. | Amax (W w) S (W W +11++11 W W 1 + W W - 11 1 1=1+ (D)(W).
Lemma (Covariancc/speectaal growth through layers). Let En. 1 H Hg. Then
n
Amax (E11,)) S Amax (EH, 1 I Amax (EtHe-)) vk-
Consequently, with EHo Ez.
k Amax (EHH)) S Amax(Ez) II
Vj- j=1
In particular, under Ez Ne I we have Xmax ( S 11:=11 Vij
Proof. ReLU is 1-Lipschitz (applied elementwise), hence ||Ha||2 S ||Ae|la S HH-1 |2
Therefore 1 H S -11 i.e., Amax (.Ha) S Amax (EH. W Apply
Lemma 1.
Lemma 3 (Block-Jacobian bound). Let J E ((nd))xp be the Jacobian of vec(U) w.r.t 0 and J1 J2 Jz] the block-colunns corresponding to vec(Ws). Then
A-1 (II ())
II 53)
II
jmm+1
k
Consequently.
|
<
1 S
ΣI k=1
Proof. Consider a perturbation AW,- Without loss of generality, we may assume || |WWellr = 1, since the operator is defined by the supremum unit perturbations. With fixed ReLU gates (op. norm < 1), the output perturbation over all n samples satisfies
AU
H AWr BETI:L,
BR+1&L
D& k+ De+1 DI-1 WL.
diag gates. |1-1|a51
Thus ||U|| < H A& | B 1. ll., so the operator norm of the linear map AWa +
AU is at most H H |2| Bx+1. 1l2. Hence || ll2 I || [|2|| Be41:2 l|2 and
11 < H B B = Amax (EHH |1
Using Lemma Amax (HH,.,) S I1 my. Also |B+11LL S I I|Wrlla S Ti/ aj. thus |B | Mg. Multiplying the two bounds yields the claim. Finally, since J is horizontal concatenation of blocks, || E. ||
Lemma 4 R. By (A2), 44l22 I B, hence
part). For each sample, V3A = (T + R, with some remainder
Σ J. (V
8 |
< 3 III'% Vi
k=1
where the last inequality uses Lemma 3.
18

---

Published as a conference paper at ICLR 2026
Lemma 5 (Remainder term), R := , + R4. Under (A2) and (A3),
||||2 S 27
E II Vj-
iskctSLjE(k.))
Proof. Fix the ReLU gates locally (piecewise linear region) and Ex || 1, since the operator norm is defined by the supremum over unit perturbations. Then the network output U is For l, the mixed second derivative block maps (AWk, AWe) to multilinearr in {W*]k=1- (
Hk-1 A & 1:- AW, BE+1IL,
where Ck+1-8-1 is the product of intermediate gated weights, and Be+1.L the tail product as in Lemma 3. By submuliiplicatiitty.
ABeiill & 1| |lWell |C (1WWel |
Using Lemma and Lemma
k-1
P-1
I.
H S vn
II ag, I|C441e11122 - j=1
II oj j=k+1
| & II oj.
j=f-1
Therefore, after dividing by (from the prefactor 1/n in L) and summing the symmetric contribution (E, k), the bilinear remainder contributes at most
| 1if--1121110 [2][CC++1:e (l22| n k<E
2
S
Σ KAE
II ot) ||WWWl=|AWWlll jE(K,E
6² and Σ. 11 (unit parameter direction), the operator norm of the second-derivative map is bounded by Exce I Vj. Multiplying by V/ 222 S 7 and averaging over gives the claim.
Finally, by 2ab < a² +
Proof of Theorem
Proof. Combine Lemma 4 and Lemma and use ||A + B1I2 S ||A|12 + ||BB|2
Corollary (Near-interpolation small-gradient regime). If the training gradients at the outputs are small so that ry Re 0, then
S 8 EII +1 DN(WW)).
k=1 j
Next, we present the lemma required for the proof of Theorem 3. Lemma (A hasic spectral lemma from Dff). Let e = VDI((W). Then
|| TWTP W - 1112 I || W 1||F = s,
hence every eigenvalue μ of W W 52 satisfies
S μ $ e. Equivalently. vi-eI S vI+eI.
Proof. By the definition of e, we have
|| W Ill2 s | w Ale E.
Therefore, all eigenvalues f TyTW TW lie within the interval
- e < / < 1 + Since W T W= = S2 with S 2 0, this is equivalent to the spectral bound
v1 = s = VI+el.
Using this Lemma, we provide proof of Theorem below:
19

---

Published as a conference paper at ICLR 2026
Proof of Theorem 3
Proof. Let E RRxxa be the input matrix and W E Raxb weight matrix. The resulting feature matrix is 4 = ZW E _x and the empirical covariances are
Σg = ZER
Σg - W ΣnW W E R
Let W = QS denote the right polar decomposition of W, where Q E Raxb has orthonormal columns
(Q' Q = Ib) and S = (W w/ E hg is positive definite. Then
V = T W = Ez))..
Write M == E = O, and let its positive eigenvalues be 7n1 2 ... > TM 0, where
(M) S min(b, r=2). Let () 2 ... 2 oa(Φ) > 0 denote the nonzero singular values of @
For any x E R with ||||| = 1,
Exx = x SMSr (Sx) M(x)).
Let Sx. Then r Esz = y My and, by Lemma 6,
1 | E [1-E, 1+
Therefore
Esp S (1+ \max(M),
where X (M) 17 denotes the smallest positive eigenvalue of M (the lower bound is interpreted on the subspace where My * 0). Taking the maximum over unit yields
Amax (Ex) < (1 ++) 171
and taking the minimum Rayleigh quotient over the orthogonal complement of ker(E.) yields
X (Ex) 2 (1 - E) Ma.
Since muax (D)2 Amnaxe (E+) and (omin. (T)) 2 (t (E+), below inequality holds with d = rank
VnVI- TTd
min (+)
<
max
① S VnVI+EVT
Here ttin (1) denotes the smallest positive singular value of d (defined only when d 2 1).
Therefore, if d 2 then
max $
I+E 1-6
P4
(13)
o min
na
Consider the worst-case allocation of the nonzero singular values that maximizes the cumulative ratio Ek_ro El= 0i given fixed condition number bound ps: the top singular values all equal Tmax and the remaining d equal o at Then
Σ = (T)
Tmax
1: po
(14)
M
(T) k Tmax + (d (d-k) k) o
k Ps + (d k)
To achieve coverage level of 1 â with l singular values, it is necessary that
P4
(1 8)
2 1-8
1o Pg + (d k) Taking the ceiling and substituting the bound on pe from (13), establishes the left inequality in (3)
PΦ (1 -0-) 5)
When Σz we have M = TQQQQQQ IQ I, so th2
no and Then,
W {.) S Vovi+s (vi).
20

---

Published as a conference paper at ICLR 2026
Which leads to
Ps S
Substituting Pa in first inequality of (3) leads to the right inequality (3), Without whitening, the achievable flattening is limited by the compressed input spectrum M = "T ΣzQ.
Next, we present the proof for Theorem 4.
Proof of Theorem 4
Proof. By isotropy and positive homogeneity, there exists constant eo > 0 such that
Eg (x wy))il = ca ||wll for all j. Hence 8y = ||, Ex || Let uy = ||,0 ||
[W T Whij Since
DI((W) |ww W T1:
Σ j=1
U1 - 1)2 + Σ
(wi, w2)² 2
U 1)²
iej
j=1
we obtain My - 1/ v VE (uk - -1 I s,, 1- c ||w3|1 2 s +1+ c for all j. The same bounds
imply vI- - S F = Σx ||wx|l S v1+ Therefore
V--
vl+c S8j |wyl|
VT+c
V1-
Corollary 2 (Absence of r-dormant neurons). Fix E (0, DiW s (z then sj 2 T for all j.
Note that Sokar et al. (2023) measured neurons with a dormancy score of 0.025 or lower as dormant. In this threshold, based on theoretical analysis, DfI(W) < 0.9975 eliminate dormant neurons from the network.
Derivation of Equation
Here we provide derivation of Equation 5.
First expand the norm:
W W 13
| + | - 2 2tr(W W).
From the constraint W W = I, we have || 13 = tr(T)= = n, so
min W W=1
I| 3
max tr( W). W W
Let S TTW W > and define Q = WS- 1/2 Then
Q = S-12 ws = 5-11/25--12 =
so Q is feasible, and the (column) polar decomposition of W.
W Q/1/2
Now take any feasible W and set
Z = TTQ
Then
tr W) tr(W QS2/2) tr(2s)/2).
21

---

Published as a conference paper at ICLR 2026
Because W and Q have orthonormal columns, one show z < I, so all singular values (Z( satisfy S (2) I 1. By von Neumann's 's trace inequality,
n
tr(2s
o(
Σ -)
with equality when Z = i.e. when W =Q.
Therefore the solution is:
W Q = W(W W(WTW)-E
22

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RESULTS
B.1 COMPUTATIONAL EFFICIENCY
To prove computational efficiency of FIRE, we provide wall-clock time and GPU memory usage in Table 1.
Table I: Wall-Clock Time and GPU memory footprint FIRE and baseline methods
Method
Wall-Clock Time
GPU Memory
Shrink Perturb
0.002 sec
27 MB
FIRE
0.06 sec
55 MB
DASH
69 sec
2834 MB
As shown in the table, FIRE introduces negligible computational cost and memory usage similar to Shrink Perturb, while significantly efficient compard to DASH.
The result is averaged across 10 trials, VGG16 architecture with TinyImageNet dataset. We used a machine consists of TITAN RTX 24GB GPU and AMD Ryzen 7 5800X 8-Core Processor, with 64GB RAM.
B.2 NUMBER ITERATIONS FOR NEWTON-SCHULL ITERATION
In this section, we provide more detailed analysis which illustrates how SFE and Dfl evolve across FIRE iterations.
100racoo-11 11ll::
Imagenet
Squieed Froberiuu Ernor 15123
Deviatonn from lommety (DP)
40F
Bose
6000
4000
0.10
0.1
20
10a
10
15
25
Iteration
Iteration
Figure 6: Effect of number of FIRE iterations. Test accuracy of FIRE with single iteration and 10 iterations (left). Change of SFE during FIRE iterations (middle). Change of Dfl during FIRE iterations (right).
As shown in Figure (right), DfI decreases substantially after only single iteration. This suggests that using small number of iterations (< 5) is sufficient to bring performance benefits. However, as shown Figure 6 (middle), SFE reaches its peak at the first iteration and then decreases as the number of iterations increases, indicating that using only few iterations (< 5) can introduce instability and ultimately lead to performance degradation.
We validate this result in continual visual learning (VGG-16 with ImageNee, Figure (left) shows comparison between FIRE with 10 iterations and with single iteration. The result shows that with single iteration still achieve comparable performance with 10 iterations, but show significant drop after reinitalization, which supports aforementioned findings.
B.3 COFFFICIENTS FOR NEWTON-SCHUIE ITERATION
Our orthogonalization strategy builds on the Newton-Schulz Schulz (NS) iteration, which has also been adopted in recent works such as Muon. However, the exact recurrence used in Muon differs from ours. Muon employs tuned quintic polynomial of the form
(p(x)=
bx + cr.
23

---

Published as a conference paper at ICLR 2026
with optimized coefficients such (a, b) (3.4445, -4.7750, 2.0315). chosen to accelerate convergence so that only a few iterations are needed in practice. However, this sacrifices accuracy for speed, since the singular values do not converge I. but oscillate near it. Since interest accuracy rather than speed, we adopt the standard coefficients (a, b, c) = (2, 115,, 0.5), which correspond to well-known rectangular variant of NS:
X 2X, 1.5x, Xx) + (X, Xx)
Although this more slowly increases small singular values than Muon's 's tuned version, accurately converges to orthogonal matrix.
Empirically, we did not observed significant difference in performance when we tested both coeffi- cients in the warm-start setting (results are shown in Figure 7).
coefficients
1/0 restt
2/2 Hout (54 (1r5)
0:35
Thne: Accursty
Accuracyy
0.30
0.30
WOrT start (Bare)
00 201 18aos)
0.25
0.75
Tpach
Epach
Figure 7: Effect of Newton Schulz iteration coefficients on FIRE. FIRE and FIRE with Muon's ss coefficients are evaluated on warm-start setting under CIFAR- 10 with ResNet- 18 (left), CIFAR-100 with ViT-Tiny (middle), and Tiny ImageNet with VGG-16 (right).
B.4 TRAIN ACCURACY
(Ours)
Perseval
DASH
ReDo
12mnt
Base
CIP40-200 NNt-iwy)
1.00
Imagenst (192-18)
CIPAR-20 (Resteet-111
(a)
Then
0.00
0.00
0.00
Esoch
Egech
Eanch
1.00
OPAR L0 (Resttct 15)
1.00
CHAR LOO EVT Tiny)
1.00
Tiyy imagenet (VGC 16)
0.00
0.80
0.80
0.00
0.60
0.69
(b)
0.40
Then
0.40
0.49
0.20
0.20
0.29
0.00
200
100
600
000
1300
0.00
200
400
602
800
1000
200
400
000
000
1000
Epach
Epoth
1pach
0.60
0.30
(c)
0.40
6.00
0.00
Fooch
Figure& 8: Train accuracy of continual visual learning experiment. Warm-start setting (a), Continual setting (b), and Class-incremental setting (c).
24

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS OF FIRE
Here we describe how FIRE is applied in practice to different modules of the network.
Linear layers. For colly-connected weights W R we first normalize and then apply NS iteration to approximate an orthogonal matrix. Since orthogonalization alone changes the scale of the outputs, we multiply the result by
scale
dun
di
This factor is motivated by the Modular Duality framework (Bernstein & Newhouse, 2025), which shows that taking the ratio of output to input dimension is sufficient to preserve stable signal variance. In short, the orthogonalization ensures the weights are well-conditioned, and the scaling factor restores the right magnitude.
Convolutional layers. For convolutional filters W RCuxxCakkakk Kkw, we apply the same proce- dure slice by slice over the spatial indices. Here the scaling factor additionally accounts for the size of the kernel:
/Cou/ Ci
scale
kg ki
Intuitively, the larger the kernel, the more input values contribute to each output, so we divide by the kernel area to prevent the output variance from exploding.
Note that for each spatial location (i,j), the slice W[a,;, E Coun is orthogonalized indepen- dently by applying the Newton-Schule iteration.
Attention modules. In Vision Transformers (ViTs), we restrict orthogonalization to the query (Q) and key (K) projections. Empirically, applying it to the feedforward MLP layers or the output projections does not provide clear benefits and may even reduce performance. Because the dot- product QKT is the part most sensitive to poor conditioning, orthogonalizing Q and helps improve the stability of similarity scores while leaving the value (V), output (O), and MLP weights unchanged.
25

---

Published as a conference paper at ICLR 2026
D BASELINE METHODS
Shrink & Perturb. Shrink & Perturb (S&P) is Reset-bused method that shrinks weight parameters and injects noise (Ash & Adams, 2020). This method has proven particularly beneficial for warm-start training. Following the setup in prior work Lee et al. (2024a), we control both the noise level and the shrinkage strength using a single hyperparameter. Formally, letting f denote the learnable parameters, 8o the initial parameters, and À the S&P coefficient, the update rule is: A + (1 X)0 + 200-
DASH. Direction-Aware SHrinking (DASH) (Shin et al., 2024) is a Reinitialization based method that selectively shrinks network weights according their directional alignment with the loss gradient, measured by cosine similarity. This method suppresses parameters that contribute to noise memorization while preserving weights that encode task-relevant features. This method enhances training efficiency and preserves model plasticity. leading to improved generalization under stationary data distributions. In our experiments, we applied DASH when new data was added.
Parseval Regularization. Parseval Reg. introduces a regularization term that enforces the weight matrices of neural network layers to remain approximately orthogonal (Chung et al., 2024). Formally, letting W denote a weight matrix, I identity matrix, and || ||x the Frobenius norm. The loss term is W||WWT - 1/. where & > 0 is a scaling factor and the regularization strength. penalizes the deviation of WWT from scaled identity matrix, encouraging the rows of each weight matrix W to be orthogonal and have controlled norms. This constraint keeps the singular values of W close to a constant, preventing gradient explosion or vanishing and leading to more stable and efficient optimization. We used s = 1 in all experiments and only swept X.
Continual Backpropagation. Continual Backpropagation (CBP) selectively reinitializes low-utility hidden units using a contribution-utilit measure (Dohare al., 2024). Contribution-uttin scores are computed as an exponential moving average of the unit's activation magnitude multiplied by the summed magnitude of its outgoing weights. Units with persistently low contribution utility considered uninformative and are periodically reset. CBP is controlled by two hyperparameters: the maturity threshold m, which protects units from reinitialization for at least m update steps allow stable utility estimation, and the replacement rate which determines the expected fraction of units to reset at each update step via fractional accumulation.
Recycling Dormant neurons. Recycling Dormant neurons (ReDo) (Sokar et al., 2023) is another unit-reimitialization method that assigns a neuron score to each hidden unit in every layer, and resets units whose scores fall below the hyperparameter т. The neuron score is computed as the ratio between a unit's average activation magnitude and the average activation magnitude of all units in the same layer, formally defined of
HT Eees (x)] where h (x)) denotes the activation of
FeeD (h((2)]
neuron i in layer for input x E D, and H is the number of neurons in layer f.
L2 Init. L2 Init (Kumar et al., 2025b), as known Regen (Regenerative regularization), is weight regularization method designed to mitigate plasticity loss by leveraging the property that the initial network exhibits the highest plasticity. 1.2 Init regularizes the weights to stay close the initial weights by adding term \|| Wo 1. to the loss function, where À is the regularization strength, W is the current weight matrix and Wo is initial weight matrix.
Self-Normalized Resets. Self-Normalized Resets (SNR) (Farias & Jozefiak, 2024) is a reset-based method that detects inactive neurons by monitoring their firing statisties and statistically testing whether a neuron's 's activity is consistent with its past behavior. For each neuron, SNR maintains an empirical distribution of inter-firing times (the number of consecutive updates with zero activation). If the computed probability falls below a threshold 1 the neuron is classified as inactive and reset. This procedure adaptively replaces neurons whose activity has effectively vanished, mitigating plasticity loss without relying on a fixed, hand-tuned inactivity window.
Muon. Muon (Jordan et al.) is an optimizer that augments SGD with momentum by orthogonalizing its update matrices. Concretely, Muon first forms the usual SGD-momentum update G' for each weight matrix and then applies a small fixed number of -carton iterations to approximate the closest semi-orthe semi-orthogonal matrix Ortho(G),, effectively replacing G by matrix with singular values near one while staying close in Frobenius norm. Following the reference implementation, in our experiments we apply Muon only the middle weight matrices of hidden layers, while scalar
26

---

Published as a conference paper at ICLR 2026
and vector parameters, well input and output layers, optimized with AdamW. We set the momentum to 0.95 as recommended by the authors.
Plasticity Injection. Plasticity injection (Nikishin et al., 2023) restores neural network's plasticity by adding a fresh, randomly initialized copy of the prediction head while leaving current predictions unchanged at the moment of the change. The original prediction head is frozen, and two identical new heads are created, one that is allowed to learn and one that always stays fixed. At the start, the learned and fixed same. As training continues, the learnable new head adapts to new data, giving the agent renewed flexibility, while the original and the fixed new head act as a stable reference. For DQN, we applied plasticity injection to MLP layers, and for SAC, we applied it to whole critic network which is known suffer from severe plasticity loss (Ma al., 2023).
heads cancel each other out, the overall output of the agent stays exactly the
27

---

Published as a conference paper at ICLR 2026
E DETAILED EXPERIMENT SETTINGS
E. CONTINUAL VISUAL LEARNING
For continual visual learning, we report the results with 3 seeds.
Table 2: Detailed settings in continual visual learning.
Parameter
Value
Optimizer
Adam (Kingma & Ba, 2014)
Learning Rate
1e-3
Learning Rate Scheduler
Warmup
Gradient norm clipping
0.5
Batch Size
256
Epochs per Chunk Data Augmentation
100
False
In this section, we describe the detailed settings for conducting continual visual learning. We note that most of the hyperparameters we used were adopted from Lee et al. (2024a).
For the Warmup scheduler, the learning rate is increased linearly from 0 to the target learning rate during the first 10% of training on each dataset. In other words, in the case of Table 2, the learning rate is gradually raised from to 1e-3 over the first 10 epochs of each data chunk.
In the warm-start scenario described in Section 4.1, we trained for 1000 epochs before new data arrived and for 100 epochs after its arrival, order to balance the total number of gradient updates before and after the introduction of new data.
E.2 CONTINUAL PRETRAINING OF LLMS
For continual pretraining of LLMs, we report the results with 3 seeds.
Table 3: Detailed settings in continual pretraining of LLMs.
Parameter
Value
Optimizer
AdamW (Loshchilov & Hutter, 2017)
Weight Decay
le-1
Learning Rate
6e-4
Minimum Learning Rate
6e-5
Learning Rate Scheduler
Warmup Linearly Decaying
Gradient norm clipping
1.0
Batch Size
480
We used implementation and hyperparameters of nanoGPT from Karpathy (2023).
During first phase, the learning rate linearly increases from to the target learning rate (6c-4) during 2,000 steps, then annealed to minimum learning rate (6e-5) until the end of the phase. In the second phase, the learning rate linearly increases from the target learning rate (6c-4) during 10% of training iterations of second phase. Then, it decreases linearly to minimum learning rate (6e-5) until the end of the phase.
E.3 REINFORCEMENT LEARNING
For reinforcement learning, report the results with 5 seeds.
For S&P method, we apply S&P to the encoder and Reset to the fully connected layers (D'Ore et al., 2022) for discrete control, and S&P with À 0.8 to whole parameters for continuous control tasks. We perform a single intervention (Full Reset, S&P, FIRE) at the midpoint learning. We followed the hyperparameter configurations used in prior work (Sokar al., 2023).
28

---

Published as a conference paper at ICLR 2026
Table 4: Hyperparameters used in the ALE environment with DQN algorithm.
Parameter
Value
Optimizer
Adam (Kingma & Ba, 2014)
Optimizer:
1.5e
Optimizer: Learning rate
6.25e
Minimum c for training
0.01
Evaluation
0.001
Discount factor 7
0.99
Replay buffer size
10a
Minibatch size
32
Initial collect steps
20000
Training iterations
10
Training environment steps per iteration
250
Updates per environment step (Replay Ratio)
Target network update period
2000
Loss function
Huber Loss (Huber, 1992)
For continuous tasks, the hyperparameter setting is followed by Lee et al. (2024b).
Table 5: Hyperparameters used in HumanoidBench environments with SimBa.
Parameter
Value
Optimizer
AdamW (Loshchilov & Hutter, 2017)
Optimizer: Learning rate
1c
Optimizer: Weight decay
0.01
Actor hidden dim
128
Actor num blocks
Critic hidden dim
512
Critic num blocks
2
Discount factor ?
0.99
Clipped Double-Q (Fujimoto et al., 2018)
True
Replay buffer size
106
Minibatch size
256
Initial collect steps
5000
Updates per environment step (Replay Ratio)
Soft target update factor 7
0.005
E.4 HYPERPARAMETER SEARCH SPACE
Table 6 presents the hyperparameter search space, and Tables 7-10 present their optimal values.
29

---

Published as a conference paper at ICLR 2026
Table 6: Hyperparameter search space for all experiments.
Experiment
Method
Hyperparameters
Search Space
Warm-Starting
S&P
0.2, 0.6,0.6,, 0.8
DASH
0.1,0.3 0.3 0.05, 0.1,
Parseval Reg.
le-3, 1e-4, 1e-5
CBP
le-4, le-5
100, 1000
ReDo
0.01, 0.1.
1.2 Init
1e-3, le-4, 1e-5
SNR
0.01, 0.04, 0.08
Continual Learning
S&P
0.2, 0.4,0,6,0.8
DASH
0.1, 0.3
0.05,0.1,0.
Parseval Reg.
le-3, 1e-4, le-5
CBP
le-4,le-5
m
100, 1000
ReDo
0.01,
1.2 Init
le-3, 1e-3, le-5
SNR
0.01,002, 0.04, 0.08
Class-Incrementas Learning
S&P
0.2, 024.6.4006,,.3
DASH
0.1,0.3 0.05, 0.3
Parseval Reg.
le-3, le-4, le-5
CBP
le-4, le-5
100, 1000
ReDo
0.01, 0.05, 0.1,0.5
L2 Init
X
1e-3, 1e-4, 1e-5
SNR
0.01,0.02 0.02, 0.04, 0.08
Continual pretraining of GPT-0.1B
S&P
0.2, 0.5, 0.8
30

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for Warm-Start setting.
Dataset
Method
Value
CIFAR-10
S&P
À 0.8
(ResNet-18)
DASH
= 0.3, À = 0.05
Parseval Reg.
À le-3
CBP
1e-4, m 1000
ReDo
T 0.5
L2 Init
À 1c-3
SNR
T = 0.01
FIRE
iter 10
CIFAR-100
S&P
À 0.8
(ViT-Tiny)
DASH
= 0.1, À = 0.05
Parseval Reg.
À le-5
CBP
T 1c 4, m = 100
ReDo
T 0.5
L2 Init
À le-3
SNR
T = 0.01
FIRE
iter 10
Tiny ImageNet (VGG-16)
S&P
À = 0.8
DASH
a = 0.1, À = 0.05
Parseval Reg.
À 1e-3
CBP
T 1e 4, = 1000
ReDo
T 0.5
L2 Init
À le-3
SNR
T 0.08
FIRE
iter 10
31

---

Published as a conference paper at ICLR 2026
Table 8; Hyperparameters for Continual Setting.
Dataset
Method
Value
CIFAR-10
S&P
0.8
(KesNet-18)
DASH
= 0.3, À = 0.05
Parseval Reg.
À le-3
CBP
1e-5, 1000
ReDo
T 0.05
L2 Init
À 1c-5
SNR
T = 0.08
FIRE
iter 10
CIFAR-100
S&P
À 0.8
(ViT-Tiny)
DASH
= 0.1, À = 0.05
Parseval Reg.
le-5
CBP
T 1c-5, 1000
ReDo
T 0.01
L2 Init
À le-4
SNR
T = 0.04
FIRE
iter 10
Tiny ImageNet (VGG-16)
S&P
À = 0.8
DASH
= 0.1, À = 0.1
Parseval Reg.
À 1e-4
CBP
T le- 5,
= 1000
ReDo
T 0.01
L2 Init
À le-5
SNR
T 0.01
FIRE
iter 10
Table 9: Hyperparameters for Class-Incremental Setting.
Dataset
Method
Value
CIFAR-100
S&P
A= 0.8
(ViT-Tiny)
DASH
0.3, = 0.3
Parseval Reg.
À = 1c-5
CBP
T le-5, 1000
ReDo
T 0.01
L2 Init
À le-5
SNR
T = 0.08
FIRE
iter
Tiny ImageNet (VGG-16)
S&P
À = 0.8
DASH
0.3, = 0.1
Parseval Reg.
À = 1e-3
CBP
1c
4, 1000
ReDo
T = 0.05
L2 Init
À le-4
SNR
T 0.02
FIRE
iter 10
32

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameters for Continual pretraining of LLMs.
Method
CKPT
Value
S&P
Best
0.5
30k
À = 0.8
60k
0.5
FIRE
Best
iter 5
30k
iter 5
60k
iter 5
33


Reviews:
[{"review_id": "ywWBXN2Z0H", "reviewer": "Reviewer_EWuU", "summary": "This paper proposes the FIRE algorithm for training Neural networks continually that is able to better balance stability with plasticity. The way it does so by doing a constrained optimization of the Squared Frobenius Error (SFE) between the weights and past weights, subject to the Deviation from Isometry (DfI) property being 0. The DfI property measures how close the weights are to orthonormal, which has been linked to high plasticity, and the SFE ensures the weights don’t deviate far when trying to achieve the plasticity objective. The paper evaluates the algorithm across continual visual learning, warm starting LLM pretraining, and RL.", "questions": "- Do you apply FIRE only at the start of distribution shifts, or is it a regular intervention? In RL it seems reinitialization only happens once, what happens when you reinitialize more often? \n- I am assuming that before the reinitialization point, the different RL algorithms should be the same, so why are the results for the different algorithms different before the reinitialization point? If it’s just the result of randomness in the run, then maybe try an experiment where you take the same checkpoints/replay buffers and reinitialize them with the different strategies, rather than starting the runs from scratch. This would help disentangle the effect of what was potentially in the replay buffer with the effect of the reinitialization strategy.\n- In Figure 5, what happens when you decrease the number of iterations even more? What is the breaking point?\n- What is the type of change that happens with this reset? For example, on average, how much do the parameters move, is it generally a low loss path/is the reset to the same basin?\n- This is lower priority, but I’d be curious to see how Hare and Tortoise does in your experiments as well, since it can also be thought of as a type of resetting baseline.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a clearly written paper and the algorithm is very intuitive. The results are fairly strong across a variety of domains, showing the generality of the idea. The suite of experiments and ablations is fairly comprehensive. The theoretical results in the provide a further justification for why the objective used (DfI) is appropriate for plasticity, as it addresses several of the issues commonly associated with loss of plasticity.", "weaknesses": "- Given the similarity of the DfI optimization to the Muon optimizer, I feel like the paper should have more of a discussion on the work beyond what they've provided. The paper claims that the difference is in the coefficients chosen for the Newton-Schulz iteration, and that muon favors speed over stability, but it is not clear if that would actually matter. It seems that muon would also be a good baseline for this paper.\n- I have a few questions which I've outlined below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VejxMjrhtB", "reviewer": "Reviewer_Ts7T", "summary": "In this paper, the authors address the challenge of balancing plasticity and stability in continual learning by proposing a reinitialization-based method called FIRE. This approach is grounded in theoretical analysis and demonstrates its effectiveness across various learning paradigms, including continual visual learning, language modeling, and reinforcement learning.", "questions": "In Theorem 1, it can be directly inferred that minimizing the SFE leads to a reduction in the variance difference. However, a more specific analysis is needed to explain how a smaller variance difference translates into strong stability for the model. For Theorem 4, the upper bound of $s_j$ is 1, but since $1<\\sqrt{\\frac{1+\\epsilon}{1-\\epsilon}}$ always holds, the derivation of the upper bound does not provide any meaningful insight. Meanwhile, the lower bound of $s_j$  increases as $\\epsilon$ decreases, which actually implies an increase in the number of dormant neurons—contradicting the paper’s claim that the number of dormant neurons is reduced. Moreover, in Theorems 3 and 4, the assumption $\\epsilon<1$ appears rather strong,  and its validity in practice should be justified. Additionally, the authors should provide a derivation for Equation (5).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed method is strongly motivated and comes with theoretical guarantees (although I have not thoroughly checked the correctness of the proofs). Moreover, the authors validate the effectiveness of their approach across multiple learning paradigms, demonstrating its broad applicability through extensive experiments.", "weaknesses": "the authors only state the theorems without providing detailed analysis of the theorems and their corresponding implications, especially for Theorem 1 and Theorem 4. In addition, the choice of baseline methods for comparison is rather limited, which to some extent reduces the strength of the empirical validation of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P3QafWHc0P", "reviewer": "Reviewer_iykC", "summary": "The paper presents FIRE, a novel and principled method for reinitializing neural network weights to balance stability and plasticity in continual learning scenarios. The core idea is to formulate reinitialization as a constrained optimization problem: minimizing the Squared Frobenius Error (SFE) to the previous weights (stability) subject to the constraint of zero Deviation from Isometry (DfI) (plasticity). The solution is efficiently approximated via the Newton-Schulz iteration. The method is evaluated extensively across vision, language, and reinforcement learning tasks, demonstrating consistent superiority over naive training and baseline reinitialization methods like S&P and DASH.", "questions": "please ref to the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper introduces the novel use of Deviation from Isometry (DfI) as a unified and direct measure of plasticity loss, providing compelling theoretical evidence that minimizing DfI is mathematically equivalent to achieving multiple desirable plasticity properties, including flattening the loss surface, preventing neuron dormancy, and increasing feature rank.\n- The solution, corresponding to the nearest orthogonal matrix via polar decomposition, is efficiently computed using the Newton–Schulz iteration, rendering the method practical with negligible computational overhead.\n- The robustness of FIRE is convincingly demonstrated across diverse domains, including continual visual learning (ResNet, ViT), large language model pretraining (GPT-0.1B), and reinforcement learning (SAC, DQN), suggesting its broad applicability.", "weaknesses": "- The Newton–Schulz iteration used in FIRE is implemented with a fixed number of steps and lacks any adaptive stopping criterion or convergence check. For highly ill-conditioned weight matrices, the fixed step may be vulnerable to divergence, which could compromise the isometric constraint.\n- While the theoretical claim of low time overhead is promising, providing concrete measurements would further strengthen its credibility. Additionally, a comparative analysis of memory footprint against baseline methods is lacking. \n- The study does not examine the interaction of FIRE with other effective continual learning strategies, such as parameter-efficient fine-tuning (PEFT) methods like LoRA, which also target the stability and plasticity tradeoff. Besides, how does FIRE perform with other regularization or sharpness-aware landscape techniques?\n- Which stage or task is evaluated in Figure 2c, and do the later tasks perform differently from the earlier ones?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dzcNvjTU8x", "reviewer": "Reviewer_9wQe", "summary": "The authors introduce FIRE which views resetting a network in continual learning as a constrained projection that keeps weights close to the previous solution (minimize squared Frobenius error) while restoring isometry (zero deviation-from-isometry), yielding the nearest orthogonal weights via an Orthogonal Procrustes step implemented efficiently with a few Newton–Schulz iterations. This method is motivated by the need to balance plasticity and stability. Theoretically, the authors demonstrate that this limits representation drift, smooths the loss (via a Hessian bound), raises effective rank, and bounds neuron dormancy, thus improving plasticity without sacrificing stability. Over a series of continual learning experiments in vision, language, and RL, the authors show that using FIRE at task boundaries (or once mid-training in RL) consistently maintains plasticity, attaining similar or superior performance to the benchmark methods that the authors evaluate.", "questions": "- Given the noted similarity with Parseval, could the authors compare and contrast their contributions with that of the Parseval paper?\n- What are the with reset and without reset dashed lines in row (a) of Figure 2?\n- Have the author's found any limitations to FIRE? For instance, many continual learning methods can be sensitive to the choice of hyper parameter. Could there be data streams that could be particularly challenging for FIRE to train on?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well written.\n- The paper utilizes a good mix of continual learning benchmark problems: vision, language, and RL along with a good mix of network architectures.\n- The derivation of FIRE is very well done and posing the stability-plasticity tradeoff as a hard constraint optimization problem leads to an algorithm that does not necessitate an explicit $\\lambda$ weight that requires the user to tune how much the method should prioritize stability vs plasticity. Many continual learning methods are sensitive to hyperparameter choice and require careful tuning, and outside the well structured academic benchmarks, it's not clear how robustly these methods will perform.\n- The theory is solid and well justifies FIRE.", "weaknesses": "- While the experiments are thorough, the empirical conclusions would be more convincing had more competitor methods been evaluated. Given that FIRE is partially motivated by parameter resets, it would be useful to compare FIRE against reset methods such as Self-Normalized Resets, Continual Backdrop, and ReDO which have been shown to be effective at maintaining plasticity and against the regularization based method of L2 Init.\n- Tangential to the above point, FIRE appears to show similar performance to its competitor methods on the class incremental and data incremental settings. While in the warm start setting FIRE does perform the best, given the scale the performance in marginally better. Had additional baseline methods been evaluated it is not clear if FIRE would be consistently outperforming these baselines.\n- While test error is the metric of interest, many papers on plasticity loss consider trainability and minimizing training error as a diagnostic of plasticity loss. It would be beneficial to also report training error for the experiments as plasticity loss and generalization error can confound with regularization and the number of training epochs, see appendix A.2 in Self Normalized Resets for Plasticity in Continual Learning. \n- While FIRE is motivating by preserving stability, I don't quite see how the vision and language experiments evaluate stability, these look to me to be purely problems that evaluate plasticity. For instance in the vision problems, the network sees progressively more data, whether it is warm starting or class/data incremental. For the language problem, correct me if I am mistaken, but if the validation set is the second training set, then again I do not see how stability is measured. \n- Additionally for the language problem, I don't quite see how plasticity is present. It appears that every method is able to reduce the validation error, and a plateau begins to appear roughly at the same iteration count for each method. It would be helpful, for instance, to see an experiment where the baseline's error (training or test) increases over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)