Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models
Authors: Zekun Wang
Anant Gupta
Zihan Dong
Christopher J. MacLellan
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
diffusion models
catastrophic forgetting
image generation
elastic weight consolidation
generative replay
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: In diffusion, per-sample gradients become collinear in the low SNR, yielding a rank-1 Fisher. We propose a rank-1 EWC and pair it with replay. On continual learning tasks, it improves FID and nearly eliminates forgetting on MNIST and FashionMNIST.

Abstract: Catastrophic forgetting remains a central obstacle for continual learning in neural models.
Popular approaches---replay and elastic weight consolidation (EWC)---have limitations: replay requires a strong generator and is prone to distributional drift, while EWC implicitly assumes a shared optimum across tasks and typically uses a diagonal Fisher approximation.
In this work, we study the gradient geometry of diffusion models, which can already produce high-quality replay data.
We provide theoretical and empirical evidence that, in the low signal-to-noise ratio (SNR) regime, per-sample gradients become strongly collinear, yielding an empirical Fisher that is effectively rank-1 and aligned with the mean gradient.
Leveraging this structure, we propose a rank-1 variant of EWC that is as cheap as the diagonal approximation yet captures the dominant curvature direction.
We pair this penalty with a replay-based approach to encourage parameter sharing across tasks while mitigating drift.
On class-incremental image generation datasets (MNIST, FashionMNIST, CIFAR-10, ImageNet-1k), our method consistently improves average FID and reduces forgetting relative to replay-only and diagonal-EWC baselines. In particular, forgetting is nearly eliminated on MNIST and FashionMNIST and is roughly halved on ImageNet-1k.
These results suggest that diffusion models admit an approximately rank-1 Fisher.
With a better Fisher estimate, EWC becomes a strong complement to replay: replay encourages parameter sharing across tasks, while EWC effectively constrains replay-induced drift.

Published as a conference paper at ICLR 2026
AVOID CATASTROPHIC FORGETTING WITH RANK-1 FISHER FROM DIFFUSION MODELS
Zekun Wang"
Anant Gupta"
Zihan Dong
Christopher J. MacLellan
College Computing
Georgia Institute of Technology
Atlanta, GA 30332, USA
[zekun, agupta88p.pp8888 zdong312, (aatonngn.,.mmclelle.. cmaclelll @gatlchhgaccced
ABSTRACT
Catastrophic forgetting remains central obstacle for continual learning in neural models. Popular approaches-repla and elastic weight consolidation (EWC)- have limitations: replay requires strong generator and is prone to distributional drift, while EWC implicitly assumes shared optimum across tasks and typically uses a diagonal Fisher approximation. In this work, we study the gradient geometry of diffusion models, which can already produce high-quality replay data. We pro- vide theoretical and empirical evidence that, in the low hignal-to-nosse ratio (SNR) regime, per-sample gradients become strongly collinear, yielding empirical Fisher that is effectively rank-1 and aligned with the mean gradient. Leveraging this structure, we propose rank-1 variant of EWC that is cheap as the diagonal approximation yet captures the dominant curvature direction. We pair this penalty with replay-based approach encourage parameter sharing across tasks while mitigating drift. On lnasemnncremttall image generation datasets (MNIST, Fash- ionMNIST, CIFAR-10, ImageNet-1k) our method consistently improves average FID and reduces forgetting relative to replay-only and diagonal-EWC baselines. particular, forgetting nearly eliminated on MNIST and MashionMNIST and is more than halved on ImageNet- These results suggest that diffusion models admit an approximately rank-1 Fisher. With a better Fisher estimate, EWC becomes strong complement to replay: replay encourages parameter sharing across tasks, while EWC effectively constrains ipplay- drift.
INTRODUCTION
The task of continual learning aims train neural models on stream of tasks without revisiting the full past data. A long-standing obstacle is catastrophic forgetting- whenn learning new tasks drasti- cally degrades performance on earlier ones (McCloskey Cohen 1989; French 1999; Parisi et al.) 2019). In contrast, humans exhibit striking robustness to interference, partly due to systems-level mechanisms such as memory consolidation and replay the hippocampal--neccorica loop (Mc-
Clelland et al.) 1995 McGaugh 2000 Foster & Wilson (006). These observations have inspired family of continual learning methods that explicitly encode consolidation or replay in training deep networks continuous tasks.
Two representative approaches are elastic weight consolidation (EWC) (Kirkpatrick et al. 2017) and generative replay (Shin et al., 2017). EWC constrains parameter changes with quadratic penalty weighted by an estimate of the Fisher information at a previously learned task, constraining updates remain near parameter directions that support old tasks (Kirkpatrick al. 2017 Schwarz et al. 2018). Intuitively, EWC behaves like an online consolidation process that selectively "stiffens" important parameters. Generative replay maintains a generator and distills samples from past tasks while learning new ones, thereby approximating rehearsal without storing original data (Shin et al.) 2017 van de Ven & Tolias 2018). However, replay inherits the generator's imperfections and can amplity distributional shift. EWC in practice relies on diagonal Fisher approximation, which
"Equal contribution.
I Code is availahle at: https://githbb..n/r//chhhe

---

Published as a conference paper at ICLR 2026
neglects roos-parametter correlations and struggles to find shared parameter space between tasks in overparameterized models, particularly when tasks have disjoint optimum. These observations suggest complementary pairing: replay can encourage parameter sharing by exposing shared data support, while a stronger approximation to Fisher information can constrain updates and mitigate replay's residual shift.
In this work, we study the gradient behavior of diffusion models (Ho et al. 2020 Nichol & Dhariwal 202I), which are already capable of generating high quality replay data. Our starting point is the observation that diffusion models admit a tractable gradient structure when the signal-to-noise ratio (SNR) is lower (at later timesteps). As a model converges, the per-sample gradients g become approximately collinear with their mean u. This makes the empirical Fisher F effectively rank-1;
F [[gg'] ~ auut, u E[g]
This yields consolidation penalty that captures the dominant curvature direction "for free" from model gradients. In contrast, the commonly used diagonal Fisher approximation captures almost no curvature when SNR is low. We leverage this structure to instantiate a rank-1 EWC that complements generative replay: replay encourages cross-task parameter overlap while our proposed rank-1 EWC constrains updates along the principal sensitive direction to the shared optimum across tasks.
Our main contributions are: (1) We provide both theoretical and empirical characterizations of Fisher information geometry in diffusion models, showing that low SNR induces a near rank-1 Fisher aligned with the mean gradient. (2) We propose a practical rank- EWC penalty that as cheap as a diagonal penalty but captures more curvature information for diffusion models. (3) We demonstrate that combining rank-1 EWC with distillation-based replay substantially reduces forgetting in continual image generation tasks, improving generation fidelity and stability across long horizons. On MNIST, FashionMNIST, CIFAR-10. 10, and ImageNet- Ik, our method consistently outperforms replay-only and diagonal Fisher baselines. In particular, forgetting is nearly eliminated on MNIST and FaShionMNIST and is more than halved on ImageNet- kk, the longest-horizer setting, relative to baselines.
2 BACKGROUND AND RELATED WORK
2.1 DIFFUSION MODELS
Diffusion models are family of generative models that define fixed forward noising process and learn a reverse (denoising) process that maps Gaussian noise to data (Sohl-Dickstein et al. 2015 Ho
et al. 2020). The forward chain corrupts data To 9o via Markov process
axx: 11-1) N(VI - 8, 21-1; B.I), q(x, mo) M(Vā zo. (1 -1-a0 (1-ā))I) a.)
where 0₁ 1 Be, ā. -T ag B, (0, is the noise level, and E [O...N} is the discrete (zx) | are parameterized by a neural network Eq and trained by maximizing a variational lower bound (ELBO). practice, the ELBO reduces to a reweighted denoising loss (Ho et al. (020)):
forward noising process timestep. The reverse transitions po
Siimple ( = E, au.
|| co (x),,,))) 112]
St / Vo V1 04 E.
2
This surrogate loss can be interpreted as bcore-based generative modeling that estimates the score V.. log qe (x+) at each timestep, where ge(xx) = q(a; zo)yn ) and then integrates a reverse-time SDE/ODE to sample (Song & Ermon, 2019] Song et al.] 2022bb |Vicent] 2011] Hyvärinen 2005 V log (zx) VI-a,
E ze]. and the model's score estimate follows from
S0 (2 t)
sp(x2, t), where 50 t) = E[e 2x]] at model optimum.
Denoising Diffusion Implicit Models (Song et al. 2021a) further show that one can define non- Markovian, deterministie sampling path that preserves DDPM's per-time marginal distributions while allowing far fewer steps. In this work, a DDIM sampling process for faster image generation.
2.2 ELASTIC WEIGHT CONSOLIDATION
Elastic weight consolidation Kirkpatrick et al. 2017) mitigates catastrophic forgetting by regularizing parameter updates using information about how important each weight to previously learned
2

---

Published as a conference paper at ICLR 2026
tasks. It casts continual learning as approximate Bayesian updating: for tasks 1:7, the posterior factors as p(0 DI.)) & p(Dr 0) p(0 D1. EWC approximates the previous-task posterior p(0 1 with a Laplace approximation around the prior optimum - yielding log p(0 DIT-I) 2 (0 - 07-1) (0-00-11)
(0 - 01-1); where F(I-1) is the Fisher information evaluated
at 07-1 1- Plugging this quadratic surrogate into the negative log-posterior for task k gives the EWC objective
T-1 (owc( (T( + (0- (( F( 0%)
(I)
k=1
with À trading off plasticity and stability. Intuitively, parameters with high curvature under the previous task are penalized more strongly, discouraging changes that would degrade old performance. In the original work, (r()) is the negative log-likelihood for classification model. For latent- variable generative models such as diffusion models, one can replace this term with a tractable variational surrogate such as the negative ELBO.
In practice, a diagonal Fisher is often used as an approximation to the full Fisher. this work, we show that diffusion models approximate rank-1 Fisher for free for more effective application in continual learning.
2.3 CONTINUAL LEARNING WITH GENERATIVE MODELS
Continual learning trains model sequence of tasks, posing the dual challenge of adapting to distribution shift while preserving knowledge from earlier tasks. Broadly, continual learning approaches fall into (i) egguaaizattion-basee constraints on parameter drift (c.g.. EWC) and (ii) replay-based strategies that rehearse past knowledge (Kirkpatrick et al. [2017] van de Ven & Tolias 2018). For generative models, especially diffusion models, replay particularly natural because the model can synthesize
samples for rehearsal (Ho et al. 2020).
DDGR uses diffusion generator with class conditioning to synthesize exemplars of prior tasks, and diffusion-based replay has been adapted to dense prediction (segmentation. detection) using task- specific guidance or pseudo-labels (Gao et al. 2023 Chen et al. 2023 Kim al., 2024). However, naïve replay with continually updated diffusion models can degrade denoising because the reverse process itself drifts across tasks. Recent work on generative distillation mitigates this by distilling the entire reverse chain (0--1 + 0,) across timesteps, aligning noise predictions so the continually trained model retains both sample quality and coverage of past tasks (Masip et al.) 2025). In this work, employ generative distillation along with approach to EWC for more effective learning.
3 DIFFUSION MODELS APPROXIMATE THE RANK-1 FISHER
The Bayesian view of the continual learning suggests that p(0 Di:r) x p(D 0) p(O D where Dr is the dataset for task T and 9 represents the model parameters. EWC Kirkpatrick et al. 2017) proposed a Laplace approximation of the posterior distribution p(0 D1:T-1) for previous tasks as - log information matrix for the previous task. However, forming a full Fisher is impractical and a diagonal approximation is widely used. In this work, we show that the Fisher of diffusion model can be approximated as a rank-l structure, which better captures important weights associated with previous tasks. We theoretically analyze why diffusion models approximate a rank-1 Fisher in Section 3.1 along with empirical analysis in Section 3.2 We then derive the practical EWC loss using rank-T Fisher in Section 3.3 and complement it with generative distillation in Section 3.4
DD) s (0- 07 .) (
07-1) 1). where F(T-1) is the Fisher
3.1 PER-SAMPLE GRADIENTS ALIGN WITH THEIR MEAN
Setup and notations.
We use the standard pariance-preservve forward process
Zt ãa Zo + v1 āi E, E ~~ (O, e (1....}}
If we let co f) be the noise prediction network, the per-sample surrogate loss (Ho et al.
2020 is:
(simple(0:x.)) =
= =/11==
- 1)
(2)
3

---

Published as a conference paper at ICLR 2026
From denoise score-matching perspective, let model score sø( t) Va, po and the true marginal score s (zz) := vz, log qe (2z), where qi denotes the noisy data distribution at time t and let Pg be the model with parameters 0. The connection to per-sample Eximple can expressed as follows (Ho et al. 2020 Vincent 2011 Song et al., 2021b):
(21, -v------- ai t)
E ma] --v1- āi 8t (I1e)
(3)
Substituting Equation
3 into Equation 2 yields the per-sample denoising score-matching loss:
1 1 at s0 (a( - sf (zxe) -xxr))
(0) = 2
Additionally, we define vāz as the signal level and 1 1- oe as the noise level at time t. The signal-to noise ratio SNR :- vār((1 (it) decreases with Intuitively, noise will dominate the signal in the later diffusion timesteps.
Proposition 1. Let si (11),, variance-preseerving diffusion process. As SNR decreases, 8 (2x) Rs -Ia/(1 .is).
qb, be the score of the noisy data distribution at time & in
We refer Appendix for the proof. Proposition shows that in the low SNR region (i.e, at later diffusion timesteps), the score function is approximately predicting its scaled input. In other words, the score function behaves like a scaled identity map from Ie to -2a/(
āi) when the SNR is low.
(a t) approximates a linear function sø( t) ss Agæz, where Ag is a linear operator, when the model learns to perform scaled identity mapping such that sp(x,, t) Tt 7t R.
Assumption 1.
Proposition 2. As the SNR decreases and the model converges, for any It qt. the per-sample gradient V VoLDs (0; v.) becomes collinear with its population mean under Assumption 7
VOLDSM (0:TsM((;:)) (4) E (0
Proposition 1 suggests that when the SNR is low, st (2zz) R Ie, where 7/t -1/(1 0z). which a scalar independent of input Ty. Near convergence, (x,,,t) t) (24) R Στ²r. Assumption ! hypothesizes that so s0 f) could be linear and takes the form of sp (xx,,)) t) Aạzz. This is plausible for an autoencoder with nonlinearities optimized with the MSE objective, such as the Laimple in Ho et al. (2020), these autoencoders are still "likely" to lie in PCA linear subspace (Vincent et al. 2010). We leave additional discussion and analysis on alternative autoencoding objectives Appendix C At given / such that the SNR is low, so (Izz, L) Agæ, & Tt As result, the gradient of so (Ft, t) can be understood as the direction that moves Ag to 7. Since sg (ate, () Rs Agat linear, the directional change in Ag each Fe is collinear Ag- %el independent of 2₁, so the per-sample gradients are collinear with each other, and hence collinear to their mean. We leave the complete proof in Appendix B.2
Theorem Under Proposition and Proposition 2 the empirical Fisher information matrix at time t: Fo) = Eex g(04;1 0) g(x));;)1))) 0) g 0) VoLDSM xz), is approximately rank-1 when the SNR is low with eigenvector (c( En (g(a 0)}, and eigenvalue (s(0)" F.(0) p.10) [|((0)]
Proof. Let v0) Eej L 0)] be the mean of the gradients. By Proposition 2 g(z;;0) xe c(zx)v((0) for some scalar function c(-). Then
(() Ezirq, g( ((2(:)))((0))) 0( ~ Erg
(5)
Therefore, (() is approximately rank-1 with eigenvector (0) = m4(0) and eigenvalue
Eug-an (xt)). Multiplying by n. the left and μ on the right, we get
#e (()).()))(()) Exg (e (x(x)))) = Engra 1e2 (21)) (e()) ( F((() (0) 1| [1)]1
4

---

Published as a conference paper at ICLR 2026
3.2 EMPIRICAL VALIDATION
Theorem suggests that, under model convergence, diffusion models approximate rank-1 Fisher information matrix for their gradients as the SNR decreases (i.e., as the forward pro- cess timestep increases). To empirically verify this result, we train small diffusion model on MNIST (Lecun et al. 1998), whose full Fisher matrix fits in GPU memory, and analyze its gradient behavior across timesteps í 100, 200, ..... 900. For each t, sample 1024 data points and compute their gradients. Experiment and model details can be found Appendix D We first empirically validate Proposition]] IT, which states that when SNR low, the denoising network behaves as scaled identity map. Let em(wxi
be the denoising network, and by Tweedie's iden- tity (Efron 201 1). ff, = v1 Ór Figure plots the mean-square error between input 2; and prediction zt. As timestep increases (SNR decreases), the model begins to perfectly predict its scaled input with error approaching 0.
1.2
MSE(R. Xel
We then empirically validate Proposition that the per-sample gradi- ents (8; .r,) are collinear with their mean (0) Egg (g(6; x()] at low SNR when the model converges. For each gradient, we com- pute the absolute cosine similarity to population mean. Figure 2 shows these similarities at selected timesteps where each pixel represents a per-sample similarity. We find that per-sample gradients mostly collinear with their mean at each timestep, with stronger collinearity in the mid-to-late timesteps (i.e. deeper red). This also provides empirical support for Assumption I that (0 t) = xz such that the per-sample gradient becomes scaled directional change in AO - 0- that is not dependent on Zt. We additionally plot the pairwise cosine similarities of (0) across timesteps in Figure Sa We find that the je (0) for different timesteps highly align with each other. This suggests a practical benefit where can Monte- Carlo sample timesteps instead of constructing a separate Fisher (0) at each timestep.
100
700
900
Timestep
Figure I: MSE between model input æ4 and the scaled predic- tion 24 at each timestep.
To probe the rank of the Fisher, we perform eigen decomposition on the empirical Fisher (()0) from the gradients of the 1024 samples and collect top 5 eigenvalues A 2 ... Às If F( is nearly
Figure 2: Absolute cosine similarities between per-sample gradient g00; and their expectation u()) different diffusion timesteps. Each pixel represents a per-sample similarity. Higher values (deeper red) indicate stronger collinearity with p(0).
40-4
Del. Foberiuu Norm
limestep
300
Rigervaluue
9.8
9.6-
500
10-4
0.4
0.1
10
100
300
700
900
100
300
500
700
300
350
700
Timestep
Timesteps
(a)
(b)
(c)
(d)
Figure 3: (a): Pairwise cosine similarities of (0) across each forward process timestep. (b): Top cigenvalues of (() across timesteps in log-scale. (c): The ratio r, = Ap/A1 across timesteps. (d): Relative Frobenius norm between (() and diagonal and rank-1 approximations across timesteps.
5

---

Published as a conference paper at ICLR 2026
rank-1, then X1 >> Ag. In Figure 3b
we plot the top eigenvalues at each timestep in log-scale. We observe that Å1 is typically one or two orders of magnitude larger than the remaining eigenvalues and their overall magnitude decreases timestep increases. To quantify the dominance of the leading eigenvalue, we compute the ratio rt = λ2/ A1/ Smaller r; indicates a larger eigengap and stronger rank-1 behavior. Figure 3e indicates that X1 > X2 all timesteps with the lowest ratio (r, 0.022) achieved at = 700, suggesting a sharper singlc-eigenvalue dominance in the low SNR timesteps.
Since Fisher is empirically near rank-1, we compare two approximations each timestep: (i) the rank-1 reconstruction from Equation 5 and (ii) the diagonal ( For each, we report the relative ]| (1) F,l0, F, E (Franki, Frliie) ). [[F(()]
Frobenius error with respect the full Fisher: err(Fi)
where || || is the Frobenius norm. Figure 3d) shows that rank-1 I approximation achieves a lower error the mid-to-late timesteps, suggesting that the rank-1 reconstruction close to the true Fisher. This result aligns with the above two empirical results that the rank-1 structure is clearer in the low SNR timesteps. It worth noting that the diagonal approximation yields an error near 1.0 every timestep. This behavior indicates that the Fisher information matrix in diffusion models has most of its curvature concentrated the off- off-diagonal terms, making the diagonal approximation particularly inadequate.
Our empirical results collectively suggest that the per-sample gradients are mostly collinear with their mean as SNR decreases and hence approximates a rank- I structure that captures most of the curvatures in Fisher where a diagonal approximation fails.
3.3 EWC PENALTY WITH RANK-1 1 FISHER
With the rank-1 1 approximation using Equation 5 derive practical EWC penalty term without forming a full Fisher matrix at each timestep and only use the gradients from the model. For simplicity of notation, let y g0;; zra) and μ = p(() Elg]- In implementation, this expectation is taken over the joint sampling process used during diffusion training: we draw a datapoint xo ~ Pdatt sample a timestep ~ p(t) (uniform in our experiments unless stated otherwise), construct the noised input Te from (Io, t), and then compute g(0; xz, f) = V Vol(O; 2z, t). Thus, μ should be understood as (0)) Ex... (g((; xt,
This comes from the observation that the SNR decreases sharply at early timesteps via Tweedie s identity, and therefore remains low for the majority of diffusion steps (see Figure [ Consistent with this, the empirical Fisher is strongly dominated by its leading eigen-direction across timesteps. For instance, we observe X2/Ar /A < 0.4 already at = 100 Figure 3cl and the mean gradient at different timesteps is highly collinear with one another in Figure 3a This indicates that the rank-1 approximation captures a large portion of the curvature even outside the latest timesteps. Consequently, averaging gradients over timesteps yields a practical surrogate that dilutes potentially residual higher-rant rank structure from the earliest steps and is dominated by the nearly rank-1 gradients from the remaining timesteps. With F(6) = E [gg ]. we have
c² -EC)( = F(F0)))
E (1 9)
J T F(0)μ= = J T [gg
TE
E (a [g)
|110|14
Thus, plugging in e* and Equation 5 into Equation I the EWC penalty with the rank-1 Fisher is:
CRank-1 (0) -O)(0)
ch (pk (0- 02))
(6)
3.4 PROMOTING PARAMETER SHARING ACROSS TASKS VIA GENERATIVE DISTILLATION
EWC is most effective when the task optima lie in a shared parameter subspace, so that its quadratic penalty can steer gradient descent to a region that remains good for all tasks. In overparameterizee models, however, different tasks can converge to disjoint basins, in which case single parameter vector is simultaneously optimal, and the EWC penalty alone may fail.
To mitigate this limitation and strengthen our evaluation, we promote parameter sharing by adding a generative distillation term computed on replayed inputs from earlier tasks (Masip et af.) 2025). Concretely, we keep frozen teacher model co: from the previous task and sample replay inputs

---

Published as a conference paper at ICLR 2026
from it. Generative distillation then encourages the current model Eg to match the teacher model's denoising behavior:
(GD D 1 (ll - = (=)l
where D denotes the replay distribution. Our full objective becomes:
Lona( = 1 (0) + (G)
(7)
Intuitively, the generative distillation term pulls Eg to remain compatible with past task behaviors on their ipput manifolds, guiding gradient descent toward regions that overlap with previous optima, thereby complementing EWC's curvature- based constraint.
4 CONTINUAL LEARNING WITH RANK- 1 FISHER
As the Fisher of diffusion models is approximately rank-1, we validate the effectiveness of our findings mlas--nncrmmnnal continual learning tasks.
4.1 DATASETS FOR CLASS-INCREMENTAL CONTINUAL. LEARNING
In cass-incremental continual learning, a dataset is partitioned into T' tasks, where each task Ta contains n class labels and (T) = 0. We evaluate our approach on four image datasets commonly used in generative modeling: MNIST (Lecun et al.] 1998), Fashion MNIST (FMNIST) (Xiao) jet al, 2017), CIFAR-I0 10 (Krizhevsky) 2009), and ImageNet-Ik (Deng et al. 2009). We use the down-sampled ImageNet-Ik (Chrabaszez et al. 2017) such that each image has the dimension of 3 x 32 x 32 for faster training and evaluation while preserving similar performance characteristics to the full-size ImageNet- Ik. We additionally pad MNIST and FMNIST 32 x 32 to be consistent with CIFAR-10 10 and down-sampled ImageNet-Ik, For each of MNIST, FMNIST, and CIFAR-10, 10, we partition the dataset into 5 tasks with 2 classes per task. For ImageNet- 1k, partition it into 20 tasks with 50 classes per task, simulating a much longer horizon continual learning task. We use the same class label ordering as in the original dataset. We refer to Appendix E.3 for additional details.
4.2 BASELINES AND METRICS
We compare our proposed EWC approach with rank-1 1 Fisher approximation (Rank-1) against the widely used diagonal Fisher approximation (Diag), with both methods augmented by generative distillation suggested in Section 3.4 For the ablation study, compare both EWC approaches without generative distillation. In addition, we evaluate the generative distillation only (GD) approach to assess whether EWC provides complementary benefits beyond distillation. Finally, compare these approaches to the non-continual learning setting, which serves as an upper bound for the performance. We hypothesize that with parameter sharing across tasks encouraged by generative distillation, a better Fisher approximation approach will lead to better continual learning performance and will complement generative distillation.
Metrics. To evaluate continual learning performance for generative models, we compute the Fréchet Inception Distance (FID: Heusel et al. 2018) between generated samples and each task's held-out test set. Let m& be the model after training on tasks 1:k, and let FID,(m;) denote the FID on task i's test set when evaluated with mg- We report two primary metrics:
Average FID through task k. AFID@k
1 Σ FID,(m&), AFID@T summarizes
overall performance at the end of training.
Final average forrgetiing. F $ ΣK= (FIDx(mr) - (ID)(m))) the mean change in each task's FID immediately after it learned (m²) as compared after training on all tasks (mr).
4.3 IMPLEMENTATION DETAILS
We the label conditioning UNet implementation from Huggingface with default hyper- parameers as our denoising network. We use ResNet blocks with 128 output channels in the first down sample block and 256 output channels in the rest. For sampling, use DDIM scheduler with 50 sampling
7

---

Published as a conference paper at ICLR 2026
steps and 1000 noising steps. We train our UNet with 100 epochs on ImageNet- 1k and 200 epochs on the rest of the datasets for each task with a batch size of 128 and a learning rate of 2 x 10 using Adam (Kingma & Ba 2017). We use EWC penalty weight of 15000 for all tasks. For generative distillation, we create a replay huffer of 1300 images per class for ImageNet-Il and 5000 images per class for other datasets. We refer Appendix for additional training details.
5 RESULTS AND DISCUSSIONS
5.1 CONTINUAL LEARNING PERFORMANCE
Table 1:
Average FID at the final task and average forgetting across methods and datasets. Standard errors are reported over random seeds.
Methods
MNIST
FMNIST
CIFAR-10
ImageNet-1K
AFIDI
F4
AFID,
F
AFID4
F4
AFIDI
F
Non-continual
2.600.1
5.740.8
23.3
11.740.1
Diag
62.242.9
51.1+4.2
99.9+3.6
81.7=4.7
128.644.66
74.428.5
44.2
34.2+33.0
Rank-labe
65.211.8
58.3g44
96.913.
82.1
120.0g10.2
77.4ga.4
74.311.8
41.321.8
GD
10.1
2.3
19.1
3.920.5
61.2ga.2
16.620.6
69.012.2 a
46.22112.
Diag
1..3
5.2g1.2
27.7
9.1
72.623.2
17.921.8
73.8
25.8
Rank-l (ours)
7.6
0.6
15.4.0.6
0.9
50.5_1.2
7.4
1..5
15.2
EWC complements generative distillation. We report the average FID at the final task as comprehensive measure of performance and forgetting for each dataset and method in Table Our results show that without generative distillation, EWC alone struggles to maintain a shared optimum across tasks, leading to degraded continual learning performance on all datasets. The consistently high forgetting indicates that the EWC penalty pulls the model toward the previous task's optimum while moving it away from the current task's, suggesting little to no overlap between task optima.
Using generative distillation alone substantially improves both average FID and forgetting all datasets compared to EWC-only by encouraging the model to move toward an optima that performs well across all (including replayed) tasks. When combined with our rank-1 EWC, we observe further improvements on all datasets. On MNIST and FMNIST, catastrophic forgetting is nearly eliminated with forgetting F 0.6 1 and F 0.940.3. respectively. On the long horizon ImageNet- 1k dataset, rank-1 EWC more than halves forgetting relative to generative distillation alone (F = 15.2 vs. F 46.2 And across all datasets, image generation quality improves substantially, further narrowing the gap to the non-continual upper bound. In contrast, while combining diagonal EWC with distillation improves the diagonal EWC-only approach, the improvement stems largely from distillation, and performance often matches or even degrades compared to distillation alone, suggesting the ineffectiveness of the diagonal EWC constraint. Overall, these results demonstrate that rank-1 EWC effectively complements generative distillation by enforcing parameter constraints within the shared optimum that distillation provides. We provide additional analysis low-rank Fisher estimation (ranks 1-5) via power iteration, as well as continual fine-tuning baselines, in Appendix E.4
Rank-1 EWC is stable and robust on long horizon tasks. To study the learning dynamics during continual learning on evaluated datasets, we plot the average FID at each task in Figure 4 Our results show that rank-1 1 EWC with distillation consistently reaches lower average FID during learning on all datasets than distillation-only and the diagonal variant. Surprisingly. on MNIST and FMNIST, average FID even decreases as the model learns new tasks. Given the near-zero positive forgetting from Table IT Figure 4 suggests that the generation quality some tasks improve when learning new tasks. This improvement in generation quality on early trained tasks can also be found in CIFAR- 10 where the average FID decreases at the final task. This finding implies that our rank- 1 EWC approach not only effectively constrained the model updates to preserve knowledge on the old tasks, but also refining old knowledge based new tasks.
In addition, on the long horizon ImageNet- 1k dataset, both generative distillation-onle and the diagonal variant begin to diverge around task 10 while the average FID only gradually increases for
8

---

Published as a conference paper at ICLR 2026
17.
30
Dlog CD
75
Rank:
Average FID
25
67-
Average FID
65
Awerage
Average
12
57-
20
45
0
10
15
Task
Task
Task
Task
(a) MNIST
(b) FMNIST
(c) CIFAR- -0
(d) ImageNet- Ik
Figure 4: Average FID at each task during continual learning on evaluated datasets. Standard errors averaged over random seeds.
(a) Hombill
(b) Ruffed grouse
(c) Convertible
(d) Digital Watch
Figure 5: Examples of generated images from selected classes in ImageNet- 1k over continual learning
tasks. (a) Hornbill class sampled from models trained on task to 19. (b) Ruffed grouse class from
task 1 to 19. (c) Convertible class from task 10 to 19. (d) Digital watch class from task 10 to 19. Top row: generative distillation-only; middle row: diagonal; bottom row: rank-1.
our rank-1 approach. In particular, the generative distillation-only approach reached a plateau around task 6 to 1 before diverging. This suggests that distillation suffers from distribution shift due to errors that have accumulated from the previous imperfect replay samples, such that the model optima moves away from early tasks. We refer Appendix E.3 for additional results.
5.2 QUALITATIVE ANALYSIS
To visually inspect how image generation quality evolves during continual learning, we fix label class and, after each task, sample images from that class and plot them in sequence. F ImageNet-Ik examples: the hornbill and the ruffed grouse classes sampled from models trained on task 1-19 (Figs. and and 5b). the convertible and the digital watch classes sampled from models trained on tasks 10-19 Figs. [dd. Our results show that both generative distillation-onl. (top row) and the diagonal variant (middle row) progressively generate noisier sample as continual learning progresses to later tasks. On the other hand, rank-1 FWC. with generative distillation maintains the sharpness of the objects. For example, both ruffed grouse and digital watch remain intact while other approaches almost distort the central object in the later tasks. Our results show that the proposed rank-I I method consistently preserves image quality across tasks, whereas images generated by the diagonal variant and generative distillation-only progressively become noisy and less recognizable.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION, LIMITATIONS, AND FUTURE WORK
In this paper, we investigate gradients in diffusion models and show that per-sample gradients become approximately collinear with their population Fisher information matrix. Leveraging this structure, we hypothesized and validated that rank-1 Fisher provides better approximation for EWC than the commonly used diagonal Fisher approxima- tion. When paired with generative distillation, which encourages cross-task parameter sharing that EWC assumes, our method improves continual learning performance. On class ctass-incrementall image generation, rank- 1 EWC with generative distillation outperforms both generative distillation-ons -olllttooooonyyyyy and diagonal Fisher EWC across MNIST, FMNIST, CIFAR- 10, and ImageNet-Ik in terms of lower average FID) and reduced forgetting. In particular, forgetting is nearly eliminated on MNIST and FMNIST and is roughly halved on ImageNet-INe Ik, the longest-horizon setting, relative to generative distillation-only. These findings indicate that diffusion models admit an effective rank-1 Fisher; with this better Fisher approximation, EWC complements replay by constraining replay-induced distribution drift toward shared parameter region that supports all tasks.
when the SNR is low, inducing effective rank-1
Limitations and future work. While the main contribution of our work lies in analyzing the gradient geometry of the diffusion model's mathematical framework, important direction for future work is to conduct a more in-depth empirical and theoretical study of Assumption across multiple model architectures. Such analysis would help clarify which architectural components (e.g., skip connections, attention mechanisms, etc.) or training regularizations (e.g., L2 norms) make Assumption more or less likely to hold.
REPRODUCIBILITY STATEMENT
We used open-source datasets and model implementations described in Section 3.2 and Section 4 and additional experiment details, including empirical validation experiment implementations, class-
incremental dataset splits, and syper-parameters in Appendices D and E.3
REFERENCES
Jingfan Chen, Yuxi Wang, Pengfei Wang, Xiao Chen, Zhaoxiang Zhang, Zhen Lei, and Qing Li. DiffusePast: Diffusion-based generative replay for class incremental semantic segmentation. arXiv preprint arXiv:2308.01 127, 2023.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as alternative to the cifar datasets, 2017. URL
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
doi: 10. 2000CCCVPR.20955006665
Bradley Efron. Tweedie's s formula and selection bias. Journal of the American Statistical Association, 106(496):1602) 602-96661660-1114 1614, 2011.
David J Foster and Matthew A Wilson. Reverse replay of behavioural sequences in hippocampal place cells. Nature, 440(7084):680-668 683, 2006.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, (4):128-135, 1999.
Ruikang Gao al. DDGR: Continual learning with deep diffusion-baseee generative replay. In Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 2023. URL http::tttts:///rceeeiiii .maiggssmlrrr mml. www.s..peessv222.
gao23e. .htll
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. https:://roceet wwt.semwrr..reeeeeem wwwpss/sv7.... hhwp//a 19w.tmm
PMLR, 09-15 Jun 2019. URL
10

---

Published as a conference paper at ICLR 2026
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium, 2018. URL
/ttpps://arxiv.. https://aas arxiv. org/
htapg//bb///e abs 1766. 08500
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, In Advances in Neural Information Processing Systems, 2020. arXiv:2006.1 11399.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching, Journal of Machine Learning Research, 6:695-709, 2005.
Junsu Kim, Hoseong Cho, Jihyeon Kim, Yihalem Y. Tiruneh, and Seungryul Baek. SDDGR:
Stable diffusion-basse deep generative replay for class incremental object detection. In
Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. http:////ppppeaaaai.a /ttpa//cce/eeeooooooe///////////
URL https: httpp//ppaaacccccpooooooooeeeaap the.ff.hecc..cc. com
SDDGR /
[norementaa_OOjjecc_CCLLLL oatp:eeee__cccor2222aaaeeeeee..e pdf
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
wwwpxxxwwwr//a////111aa1 abs/ htapgg/bs/4422 6980
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the
International Conference on Learning Representations (ICLR), 2014. URL httpp://aaxxx
org/abs/ 1312. 6114
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kicran Milan, John Quan, Tago Ramalho, Agnieszka Grabska Barnska-BBriikkk,, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521 March doi: 00007//paa..61183355
2017. ISSN 1091-6490,
URL http: doi org/10
1073/ 6611835114
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Y. Lecun, L. Bottou, Y. Bengio, and Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( (111:12278-22222 1998. doi: 10. 10.1109/5.22691.
Sergio Masip et al. Continual learning of diffusion models with generative distillation. In Proceedings of Machine Learning Research, 2025. URL ltttps::/proceecinngg mmr...m......................... htt.://dddda.a.......rrres/////
masi 12.5a. masip2.5... html
James L McClelland, Bruce L McNaughton, and Randall C O'Reilll, Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models. Psychological Review, 102(3):419-45 1995.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989.
James L McGaugh. Memory-a century of consolidation. Science, 287(5451):24--55,, 2000.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
German I Parisi et al. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015. httphhttss///axii..... org./ abs/ 1505. 04597
Jonathan Schwarz et al. Progress & compress: A scalable framework for continual learning. In ICML, pp. 4528-4537, 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NewrIPS, 2017.
11

---

Published as a conference paper at ICLR 2026
Jascha Sicll-ickstten, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning. 2015.
pp.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a. arXiv:2010.02502.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-basee generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b. Seameviie..e/foommmdddddi.iiiii
Gido M van de Ven and Andreas S Tollias. Generative replay with feedback connections general Strategy for continual learning. In NeurlPS, 2018.
Paseal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- lion, 23(7):1661-1674 2011. doi: 10. 10.116//NECO___000 L00000000_0001222222
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. Journal of Machine Learning Research, 1 111110)3371-3400 2010. URL
ptttp://jmlr.org///
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017. URL httpp:://arxiaa org/ wwtgg/b///////// 1170000777777
A THE USE OF LARGE LANGUAGE MODELS
We primarily use LLMs improve wording and sentence structure, check grammar, and reorganize or document code.
B ADDITIONAL PROOFS
B. PROOF FOR PROPOSITION
Proof. By Tweedie's 's identity, write (2x) T-dx Eizo [xc]] x1/((1 (1-d.). Gz). As ar decreases with timestep due to DDPM's linear noise scheduler, SNR= decreases faster than L/(1 ā.). (xx) -z/ Cz..
vai I-6
decreases with t. And because var
1-a.
B.2 PROOF FOR PROPOSITION 2
Proof. Let g(0 V VoLDsM(O;zz) uu() Bri [g(0; x/)] By Proposition
I s* (x z aete
where -1/(1 āi) is a scalar independent of x4. Near convergence, the denoising network approximates the score function: so(x,,t) t) s sxxx;) a17-- By Assumption sp (zx, t) takes
linear form as s0 t) AgZz Substitute back to x(0:zz))
6
so t) s* s*xx (moi)12)
g(0:zz) = Vo
(8)
2
1
- 6t
Agott 2 112
Vo
(9)
2
(1 a1) (Agx's II((eT) Too
(10)
(1 d ||, |AA 7.1).
(11)
Let c(x;) = az]] be a scalar function dependent on 24, and u = Ag 1 n be a vector in ø,. Then o(0:2,) c(x²) ). By the definition of collinearity,
parameter space doesn't 't depend g(0; 22) collinear with v for any 2, ~ q.. And since per-sample gradient collinear with each other, collinear with the population mean p()).
12

---

Published as a conference paper at ICLR 2026
C ADDITIONAL DISCUSSION ON ASSUMPTION 1
The theoretical rationale behind Assumption T comes from the analysis of autoencoders. Let So be an autoencoder trained with the standard MSE objective to approximate the identity mapping, i.e., fo(x) R z, As stated in Proposition ! in the late diffusion steps the denoising network behaves approximately like such autoencoder fo-
Vincent et al. (2010) report that an autoencoder with MSE is doing PCA-like operation when there no nonlinearity. When nonlinearities are introduced, they argue that the learned representation is still "likely" to lie in a PCA subspace. For alternative reconstruction losses such as cross-entropy, Vincent al. (2010) note that the autoencoder is no longer the same that of PCA and will likely learn different features. In our setting, this suggests that the observed approximately rank-1 behavior primarily a consequence of the specific DDPM parameterization: non--oarmmaen non--paameterized, variance- preserving forward process that reduces the ELBO to simple MSE objective. In particular, this rank-1 behavior need not hold when the training objective is not purely MSE.
To illustrate this, we also experimented with a VAE (Kingma & Welling 2014) trained with an MSE + KL objective. To quantify rank- I dominance, we measure the percentage of total variance explained by the largest eigenvalue of the empirical Fisher information matrix:
A
Σ
When analyzing the gradients of this VAE, found that the top eigenvalue accounts for only about 50% of the total variance. In contrast, for DDPM in late timesteps, the top eigenvalue explains roughly 99% of the variance.
Moreover, when we down-weight the KL term by a factor of 10-3, the top eigenvalue in the VAE setting explains about 85% of the variance, suggesting that the near rank-1 behavior is closely tied to the autoencoding regime with an MSE-dominated loss.
This is consistent with our theoretical analysis: for some affine parameter Ag. the gradient of a combined MSE + KL objective cannot in general be factored as VoL(x) = c(x)) v, where c(x) is a scalar depending on x and is parameter-spaee vector independent of æ. Such a factorization is precisely the condition for gradient collinearity established in Appendix [.2
even if fa( = Aaæ as it works in a PCA subspace
Although rigorous mathematical proof of this behavior for large-scale nonlinear autoencoder architectures currently out of reach, we assume that the skip connections in the U-Net (Ronneberger et al., 2015) make it even more likely that the large-scale network effectively operates in such PiA-like subspace (Ghorbani et al.) 2019). Intuitively, these skip connections allow information to bypass some nonlinear activations, thereby preserving approximately linear structure in the mapping, in line with the observations of Vincent et al. (2010).
D ADDITIONAL EMPIRICAL RESULTS
This section provides implementation details for the empirical study presented in the main text.
We use the label-conditioned UNet implementation from HuggingFace as the denoising backbone. The base model consists of three ResNet blocks per downsampling stage, each with 16 output channels and two layers per block.
Training is performed with batch size of 128 and learning rate 2 10-1 using the Adam optimizer (Kingma & Bal (211)). Each task is trained for 200 epochs on the full dataset. On MNIST (Lecun) al. 1998), training the base model required approximately 97 minutes on a single Nvidia A100
(40GB) GPU.
D.1 ADDITIONAL. MODEL. VARIANTS
To assess whether the observed Fisher structure depends on model capacity, we trained two reduced variants of the UNet under identical hyperparameters:
1. Small-1: one ResNet block with 16 output channels and single layer per block;
13

---

Published as a conference paper at ICLR 2026
2. Small-3: three ResNet blocks with 16 output channels each, but only one layer per block.
Both models were trained for 200 epochs per task with batch size 128 and learning rate 2 x 10-4. On MNIST, training required 56 minutes for Small-1 and 78 minutes for Small-3, compared to 97 minutes for the base model.
While all variants display the same qualitative crends--igensseectuum decay (Figs, 6a and 66c and dominance of single cigenvalue (Figs, [bb and 6d)-we observe that the empirical rank-1 behavior becomes more pronounced as model size increases. This suggests that our theoretical predictions are not artifacts of small networks, but rather strengthen with scale. Theoretically, this follows from our optimality assumption: larger models are expected to achieve solutions closer to the optimal point, thereby aligning more closely with the conditions under which the Fisher reduces to rank-1 structure. Consequently, one should expect the rank-1 Fisher approximation to hold even more robustly in larger, real-world diffusion models.
10-3
0.8
10-3
Eivenvalue
20-4
0.6
Eigenvalue
10-1
0.2
10-3
22
10
10
0.2
10
0.1
250
500
750
0.0
400
600
800
250
500
750
0.0
400
800
800
Timesleps
Timesteps
Timesleps
Timesteps
(a)
(b)
(c)
(d)
Figure 6: Eigenvalue analysis of the empirical Fisher for reduced UNet variants. Panels (a) and (b) show the top 5 eigenvalues (log-scale) and eigengap ratios vz A2/ À across timesteps for the Small-1 model, while panels (c) and (d) report the same quantities for the Small-3 model. In both cases, the Fisher remains nearly rank-1, with the leading eigenvalue A₁ dominating and the eigengap widening as model size increases.
E ADDITIONAL CONTINUAL LEARNING RESULTS
E.1 ADDITIONAL IMPLEMENTATION DETAILS
We use the label-conditioned UNet implementation from Huggingface as our denoising network. The UNet employs three ResNet blocks with 16 output channels in each downsampling block. For training. we use a batch size of 128 and a learning rate of 2 x 10 with the Adam optimizer (Kingma Ba 2017), and train for 200 epochs per task for each dataset.
Training was performed on a single Nvidia A100 40GB GPU, taking total of 157 minutes.
E.2 ADDITIONAL IMPLEMENTATION DETAILS FOR CONTINUAL. CEROIAA..EEAANN LEARNING IMENTS
We employ label-conditioned UNet implementation from HuggingFace as our denoising network. retaining the default dyper-parameterd The architecture consists of four ResNet blocks, with 128 output channels in the first wommssan block and 256 output channels in the remaining blocks. For sampling, we use DDIM scheduler with 50 sampling steps and 1000 noising steps.
We summarize our training hyperparameters in Table 2 Unless otherwise stated, the same settings are used across all datasets and tasks. We also provide the average runtime (including FID evaluation) for each dataset and method Table
E.3 ADDITIONAL DATASET DETAILS
All datasets resized padded 32 32 for consistency. Table summarizes their configurations.
MNIST. A dataset of handwritten digits (0-9) with 60,000 training and 10,000 test images. Each task contains two digit classes.
Task splits: Ti (0.1) 1), T2 (2,3), T3 (4,5). T4 (6. 7), Ts (8, (8.9).
14

---

Published as a conference paper at ICLR 2026
Table 2: Training configurations used across datasets.
Setting
Value
Optimizer
Adam (Kingma & Ba 2017)
Learning Rate
2 10
Batch Size
128
Training Epochs
200 per task
EWC Penalty Weight
15000
Replay Buffer (ImageNet- 1k)
1300 images per class
Replay Buffer (others)
5000 images per class
Table 3:
Average training runtime (hours) and GPU used per dataset/method.
Dataset
Method
Hours
GPU
MNIST
Diag
5
NVIDIA L40S
MNIST
Rank-1
5
NVIDIA L40SS
MNIST
GR
NVIDIA L40SS
MNIST
Diag + GR
NVIDIA L,40S
MNIST
Rank- + GR
NVIDIA L40S
FMNIST
Diag
NVIDIA L40S
FMNIST
Rank-1
NVIDIA L40S
FMNIST
GR
8
NVIDIA L,0SS
FMNIST
Diag GR
NVIDIA L40S
FMNIST
Rank- + GR
NVIDIA L40S
CIFAR-10
Diag
NVIDIA L40S
CIFAR-10
Rank-1
4
NVIDIA L40S
CIFAR-10
GR
~
NVIDIA L40S
CIFAR-10
Diag GR
NVIDIA L40SS
CIFAR-10
Rank-1 + GR
NVIDIA L40S
ImageNet- lk
Diag
25
NVIDIA H200
ImageNet- lk
Rank-1
NVIDIA H200
ImageNet- 1k
GR
64
NVIDIA H200
ImageNet-1k
Diag + GR
~ 71
NVIDIA H200
ImageNet-1k
Rank- + GR
70
NVIDIA H200
Fashion MNIST.
A dataset of 10 grayscale clothing categories (e.g., shirts, shoes, bags) with 60,000 training and 10,000 images. Each task contains categories.
Task splits: Ti {0,1), 1}, T2 = (2,3), T3 [4, 5). Ta (6, 7}, T3 = (8,9).
CIFAR-10.
A dataset of 32 32 RGB images across 10 classes (e.g., animals, vehicles) with 50,000 training and 10,000 test images. Each task contains two classes.
Task splits: Ti = 10, T2 (2,3), T3 (4,5). T. (6. 7), T3 (8,9).
ImageNet-1k (downsampled).
A large-scale dataset with 1,000 object categories and 1.28 million training images. We use the 32 32 downsampled version for computational efficiency. Each task contains fifty classes. Task splits: Ti (0.... 49), T2 (50. .5......99 .,9.....))....... T20 {950.... ... 999)--
E.4 ADDITIONAL BASELINES
E.4.1 ABLATIONS ON LOW-RANK FISHER SURROGATES
A key practical benefit of our analysis is that yields an efficient and scalable estimate of the leading Fisher eigen-ding eigen-direction using only the empirical mean gradient E[g], avoiding explicit formation of the Fisher and the costly iterative procedures typically required to extract top eigenvectors in high-
15

---

Published as a conference paper at ICLR 2026
#Training Images per Task
Dataset
#Takk
Description of Each Task
MNIST
Lecun al 1998
12,000
Generation of classss of hundwrilten digils
Fashion
MNIST Kia cial
2017
12,000
Generation of 2 classss of fashion products
CIFAR-10 10 Krizhevsky 2015)
10,000
Generation classes common items
ImageNot-f & ( Chrabasz e al.
2017
~-64,000
20
Gencration of 50 classes of ImageNet objects
Table 4: Detailed dataset configurations and task partitions used in our experiments.
dimensional parameter spaces. To contextualize this design choice, we compare against low-rank baselines that estimate the top-k Fisher subspace via stochastic power iteration and then construct rank-k approximations (rank-1 through rank-5). Across both CIFAR-10 10 and FashionMNIST, these iterative low-rank baselines do not improve performance in many cases perform worse, consistent with the difficulty of reliably converging power iteration under limited compute. Moreover, increasing the approximation rank beyond yields little benefit, supporting the view that the curvature is already well-captured by the dominant direction in this regime.
mean-granne irean-gradient rank-1 I estimate, and
Table 5: Low-rank ablations using rank-k Fisher approximations (estimated via stochastic power iteration) versus our mean-gret bean-gradieet rank-1 1 estimate. We report final FID and average forgetting (mean standard error).
CIFAR-10 10
FashionMNIST
Method
Final FID
Avg Forgetting
Final FID
Avg Forgetting
GD rank-1 (ours)
50.5 1.2
7.4 + 1.2
15.4 + 0.6
0.9 + 0.3
GD rank-2
57.5 2.6
15.9 1.8
18.1 1.6
3.8 + 1.4
GD + rank-3
60.4 5.4
1.0
16.8 0.9
2.9 0.6
GD rank-4
55.7 3.4
1.0
1..0
2.9 1.2
GD rank-5
55.6 1 2.3
12.6+1.6
17.1 1.2
2.9 + 1.0
E.4.2 CONTINUAL HINE-TUNING WITHOUT REGULARIZATION
We additionally report the performance of plain continual fine-tuning with regularization (i.e., setting the penalty weight to = 0). This baseline isolates the extent of catastrophic forgetting in the absence of any stabilization mechanism, and serves sanity check that regularized baselines are not benefiting from overly conservative hyperparameters. As shown in Table removing regularization yields substantially worse final generation quality and severe forgetting across datasets.
Table 6: Continual fine-tuning with no regularization (A forgetting.
0). We report final FID and average
Dataset
Final FID
Avg Forgetting
CIFAR-10
115.410
72.496
MNSSI
103.955
88.009
ImageNet32
265.902
67.80 67.864
MNIST
102.642
93.700
E.5 CONTINUAL LEARNING CURVES FOR EACH TASK AND DATASET
We provide comprehensive of FID evaluation plots for all tasks across the datasets considered in our experiments. These plots illustrate the model's performance on individual tasks and complement the quantitative results presented in the main paper.
16

---

Published as a conference paper at ICLR 2026
2
2
19th Boc
Tran Do
Trxn bra
Ton In
19ah boc
Task 0
Task
Task
Task 3
Task 4
Figure 7: FID plots for MNIST (generative replay).
290
50-
2
Task 0
Task
Task 2
Task 3
Task
Figure 8: FID plots for MNIST mnon-generative replay).
20-
12-
17-
15-
15
18
14
19
Toth Ro0
Tran BOA
Trxe TuAID
Tean Taak
Toth Boc
Task 0
Task
Task 2
Task 3
Task 4
Figure 9: FID plots for Fashion-MNIST (generative replay).
11-
0
0
10-
29-
12
Ta
Teah Tar.10
3ut
Teai
Task 0
Task
Task
Task 3
Task
Figure 10: FID plots for Fashion-MNIST inon-generative replay).
40-
Por Roc
Task 0
Task
Task 2
Task 3
Task 4
Figure 11: FID plots for CIFAR-10 (generative replay).
17

---

Published as a conference paper at ICLR 2026
305
340-
17
331
159
127
Task 0
Task
Task 2
Task 3
Task 4
Figure 12: FID plots for CIFAR-10 (non-generative replay).
c0
10
28
25
10
Baa
Tran 3ec
Task 0
Task
Task
Task 3
Task
215-
Se
15
Task 5
Task
Task 7
Task 8
Task
55
45-
Belk
True
Tran
Boc
Task 10
Task 11
Task 12
Task 13
Task 14
B
Tut Rac
Than Tast
Then TD
Tea TARIDD
Tuats Mac
Task 1.5
Task 16
Task 17
Task 18
Task 19
Figure 13: FID plots for Imagenet- 1k (generative replay).
18

---

Published as a conference paper at ICLR 2026
190
29
55
Task 0
Task
Task 2
Task 3
Task
48
Teah
Trtn
bock
Task 5
Task
Task 7
Task 8
Task 9
P
40
Than Ta
Tecn TakiD
Ton Malc
Tutn Bac
Tran T0
Task 10
Task 11
Task 12
Task 13
Task 14
1104
TD
Tark
Teah Taak/D
Task 15
Task 16
Task 17
Task 18
Task 19
Figure 14: FID plots for Imagenet-Ii 1k (non-generative replay).
19


Reviews:
[{"review_id": "7TzdEGrcR9", "reviewer": "Reviewer_rvLS", "summary": "The paper studies the gradient geometry of diffusion models and argues that, at low SNR timesteps, per-sample gradients align strongly with their mean, making the empirical Fisher approximately rank-1. Building on this, the authors propose a rank-1 EWC penalty that is as cheap as the diagonal approximation but better aligned with the dominant curvature direction. They pair it with diffusion-based generative distillation/replay. On class-incremental image generation (MNIST, FMNIST, CIFAR-10, downsampled ImageNet-1k), the method improves AFID and reduces forgetting versus diagonal-EWC and replay alone; forgetting is nearly eliminated on MNIST/FM and roughly halved on ImageNet-1k.", "questions": "- Is this the first work about continual learning for diffusion model? If it is not, I expect the author to compare against other related work. The main table result looks like an ablation studies for Fisher approximation approaches.\n\n- Can you conduct ablation studies on rank-k (k=2–8) variants to test diminishing returns beyond rank-1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation example is well positioned, the story of the paper is clear and easy to follow.\n- Clean connection from low SNR → gradient collinearity → rank-1 Fisher, with both theory and measurement\n- Rank-1 EWC that is drop-in and cost-comparable to diagonal while capturing dominant curvature. \n- Consistent AFID/forgetting improvements; notably near-zero forgetting and significant improvement on different benchmarks.", "weaknesses": "- Ablations missing/limited: (i) k-rank (>1) Fisher vs rank-1 vs diagonal; (ii) Sensitivity to λ, μ-estimation schedule, and the SNR/timestep sampling strategy; (iii) replay buffer size; teacher choice (EMA vs last checkpoint) in distillation.\n\n- No wall-clock/memory comparisons vs Diag-EWC and low-rank baselines (e.g., diagonal + small K-FAC block, diagonal + momentum/EMA Fisher smoothing).\n\n- Main long-horizon test uses 32×32 ImageNet-1k; higher-res or domain/task-incremental setups would strengthen claims.\n\n- I suspect that other baselines are not well tuned. Figure 4 shows that the proposed method is consistently better than others in adapting to new task, which means the method can obtain both stability and plasticity overall. I am wondering how the result would look like if λ=0 (no regularization).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PnaUgLUebz", "reviewer": "Reviewer_AxKy", "summary": "This paper concentrates on the continual learning of diffusion models with an improved EWC method (i.e., rank-1 EWC). The main contributions are summarized as:\n\na. This paper provides both theoretical and empirical characterizations of Fisher information geometry in diffusion models, showing that low SNR induces a near rank-1 Fisher aligned with the mean gradient.\n\nb. This paper proposes a practical rank-1 EWC penalty that is as cheap as a diagonal penalty but captures more curvature information for diffusion models.\n\nc. This paper provides extensive experiments to prove the effectiveness of the proposed method.", "questions": "a. In Section 3, it is mentioned that \"This is plausible in practice because a trivial solution for UNet is to directly route the inputs to the output due to the skip connections\". Please theoretically discuss Assumption 1 according to this example in details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a. This paper is well written and easy to follow.\n\nb. This paper is technically solid.\n\nc. It is interesting and significant to prove that the empirical Fisher matrix of diffusion models is effectively rank-1.", "weaknesses": "I am generally satisfied with the content presented in the paper; however, I still have the following concerns:\n\na. I believe that the most critical premise supporting this paper is Assumption 1. However, its explanation is overly intuitive and lacks rigor. Please use a concrete example and demonstrate, from a mathematical perspective, how this example is connected to Assumption 1. This is the major concern.\n\nb. The Theorem considers the low SNR region (i.e, at later diffusion timesteps). However, SNR will be high at most diffusion timesteps. How does the paper account for this situation?\n\nc. How to determine that a SNR is low?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dEnmRqWdCQ", "reviewer": "Reviewer_mxSL", "summary": "This paper investigates the problem of catastrophic forgetting in the continual learning of generative diffusion models. The authors identify a key limitation of Elastic Weight Consolidation (EWC), a popular regularization method, which typically relies on a diagonal approximation of the Fisher Information Matrix (FIM) that fails to capture important parameter correlations. The paper's main contribution is a theoretical and empirical analysis showing that for diffusion models in the low signal-to-noise ratio (SNR) regime, the empirical FIM is effectively rank-1 and aligned with the mean gradient. Leveraging this insight, the authors propose a \"Rank-1 EWC\" penalty that is computationally efficient yet captures this dominant curvature direction. This regularizer is combined with generative distillation (a form of replay) to create a synergistic approach that encourages a shared parameter space while mitigating drift. Experiments on class-incremental image generation tasks (MNIST, FashionMNIST, CIFAR-10, and ImageNet-1k) show that the proposed method significantly reduces forgetting and improves sample quality (FID) compared to replay-only and diagonal-EWC baselines.", "questions": "1. The effectiveness of your Rank-1 EWC is clearly dependent on the generative distillation component. Could you elaborate on the precise role of the penalty?\n2. Your theoretical analysis highlights the emergence of the rank-1 structure in the low-SNR (late timestep) regime. However, the EWC penalty in Equation 6 appears to be based on a mean gradient $\\mu$ that is averaged over all timesteps. How does the inclusion of gradients from high-SNR timesteps, where the Fisher may not be rank-1, affect the validity and performance of your proposed penalty?\n3. The analysis in Figure 3a shows that the mean gradients $\\mu_t(\\theta)$ are highly aligned across timesteps but are not perfectly collinear. This suggests that a single rank-1 approximation for the entire FIM might be discarding some useful information.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and Insightful Analysis of Fisher Geometry: The core of the paper is a novel and non-obvious finding about the gradient structure of diffusion models.\n2. Practical and Computationally Efficient Algorithm: The proposed Rank-1 EWC penalty, as formulated in Equation 6, is highly practical.\n3. Strong Empirical Performance and Validation: The method demonstrates impressive empirical results, substantially reducing catastrophic forgetting and improving generation quality (FID) across four different benchmarks.", "weaknesses": "1. Heavy Reliance on Generative Distillation: A significant weakness is the method's apparent dependence on the replay component.\n2. Strength of Theoretical Assumptions: The theoretical argument hinges on Assumption 1, which posits that the score network $s_{\\theta}(x_t, t)$ can be approximated as a linear function of its parameters, $s_{\\theta}(x_t, t) \\approx x_t\\theta$, in the low-SNR regime.\n3. Limited Scope of Fisher Matrix Analysis: The detailed empirical analysis of the FIM in Section 3.2 is conducted on a small-scale diffusion model trained on MNIST", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4crlX9aklV", "reviewer": "Reviewer_HQm2", "summary": "This paper studies the gradient geometry of diffusion models in low-SNR timesteps and argues that per-sample gradients become nearly collinear, making the empirical Fisher effectively rank-1. Building on this, the authors propose a rank-1 EWC penalty (cheap to compute) and pair it with generative distillation for continual image generation. Across MNIST, FMNIST, CIFAR-10, and downsampled ImageNet-1k (32×32), the method improves average FID and notably reduces forgetting versus diagonal-Fisher EWC and distillation alone (e.g., near-zero forgetting on MNIST/FMNIst and roughly halved on ImageNet-1k). Theory (Propositions/Theorem) and empirical analyses (eigenspectra, cosine similarities) support the rank-1 claim.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear theory → practice bridge: formal propositions/theorem (low-SNR → gradient collinearity → rank-1 Fisher) with convincing empirical validation (eigenvalue dominance, Frobenius error vs. diagonal). \n\nSimple & efficient: rank-1 EWC captures dominant curvature at cost comparable to diagonal EWC; easy to add to standard UNet diffusion setups. \n\nConsistent gains: with generative distillation, improves AFID and reduces forgetting on four datasets; forgetting ≈ 0 on MNIST/FMNIst and ~halved on ImageNet-1k. \n\nRobust learning dynamics: smoother curves over long horizons; qualitative samples maintain object sharpness compared to baselines that drift.", "weaknesses": "Scope/scale limits: experiments are at 32×32 resolution (even for ImageNet-1k) and on UNet backbones; no results on larger resolutions or Transformer/DiT diffusion models, which the paper itself flags as future work. \n\nHeavy reliance on distillation: EWC alone underperforms; the strongest results require rank-1 EWC + generative distillation, so the standalone benefit of the rank-1 penalty is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)