Heads collapse, features stay: Why Replay needs big buffers
Authors: Giulia Lanzillotta
Damiano Meier
Thomas Hofmann
Venue: ICLR 2026 Poster (2026)
Keywords: continual larning
neural collapse
deep learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We analyze continual learning in the long-training limit, showing via Neural Collapse that replay preserves feature separability but causes head–feature misalignment, explaining why deep forgetting is mitigated while shallow forgetting persists.

Abstract: A persistent paradox in continual learning (CL) is that neural networks often retain linearly separable representations of past tasks even when their output predictions fail. We formalize this distinction as the gap between *deep* (feature-space) and *shallow* (classifier-level) forgetting. We reveal a critical asymmetry in Experience Replay: while minimal buffers successfully anchor feature geometry and prevent deep forgetting, mitigating shallow forgetting typically requires substantially larger buffer capacities.
To explain this, we extend the Neural Collapse framework to the sequential setting. We characterize deep forgetting as a geometric drift toward out-of-distribution subspaces and prove that any non-zero replay fraction asymptotically guarantees the retention of linear separability. Conversely, we identify that the ``strong collapse'' induced by small buffers leads to rank-deficient covariances and inflated class means, effectively blinding the classifier to true population boundaries. By unifying CL with out-of-distribution detection, our work challenges the prevailing reliance on large buffers, suggesting that explicitly correcting these statistical artifacts could unlock robust performance with minimal replay.

Published as a conference paper at ICLR 2026
HEADS COLLAPSE, FEATURES STAY: WHY REPLAY NEEDS BIG BUFFERS
Giulia Lanzillotta," Damiano Meier* & Thomas Hofmann ETH Al Center & Department of Computer Science, ETH Zürich
ABSTRACT
A persistent paradox continual learning (CL) is that neural networks often retain linearly separable representations of past tasks even when their out- put predictions fail. We formalize this distinction as the gap between deep (feature-space) and shallow (classifier-level) forgetting. We reveal critical asymmetry in Experience Replay: while minimal buffers successfully anchor feature geometry and prevent deep forgetting, mitigating shallow forgetting typically requires substantially larger buffer capacities. To explain this, we extend the Neural Collapse framework to the sequential setting. We characterize deep geometric drift toward out-of-distribution subspaces and prove that any non-zero replay fraction asymptotically guarantees the retention linear separability. Conversely, we identify that the "strong collapse" induced by small buffers leads to rank-deficient covariances and inflated class means, effectively blinding the classifier to true population boundaries. By unifying CL with out-o--distributtin detection, our work challenges the prevailing reliance on large buffers, suggesting that explicitly correcting these statistical artifacts could unlock robust performance with minimal replay.
forgetting
INTRODUCTION
Continual learning (Hadsell al. 2020) aims train neural networks on sequence of tasks with- out catastrophic forgetting. It holds particular promise for adaptive Al systems, such as autonomous agents that must integrate new information without full retraining or centralized data access. The theoretical understanding of optimization in non-stationary environments remains limited, particu- larly regarding the mechanisms that govern the retention and loss of learned representations.
A persistent observation in the literature is that neural networks retain substantially more information about past tasks in their internal representations than their output predictions. This phenomenon, first demonstrated through linear probe evaluations, shows that a linear classifier trained on frozen last-layer representations achieves markedly higher accuracy on old tasks than the network's own output layer (Murata et al. 2020 Hess et al. 2023). In other words, past-task data remain linearly separable in feature space, even when the classifier fails to exploit this structure. This motivates distinction between two levels of forgetting: shallow forgetting, corresponding output-level degradation recoverable by linear probe, and deep forgetting, corresponding to irreversible loss of feature-space separability.
In this work, we show that replay buffers affect these two forms of forgetting in systematically differ- ways. Replay-the the practice of storing small subset of past samples for joint training with new data-is among the most effective and widely adopted strategies in continual learning. However, the requirement to store and repeatedly process substantial amounts of past data limits its scalabil- ity. Our analysis reveals a critical efficiency gap: even small buffers are sufficient to preserve feature separability and prevent deep forgetting. whereas mitigating shallow forgetting requires substan- tially larger buffers. Thus, while replay robustly preserves representational geometry, it often fails to maintain alignment between the learned head and the true data distribution.
To explain this phenomenon, we turn to the geometry of deep network representations. Recent work has shown that, at convergence, standard architectures often exhibit highly structured, low-
"Equal contribution, email at (glanzz 11c darmei er cht

---

Published as a conference paper at ICLR 2026
Task pnsett
Shalloow Fergenting
Tisks
bonddyy
Good posuiaton bearday
optinol decision
The data is OOD, Blzre is class information the fostrr
squarbli
Clases are seperable the boundary misliinnd
Figure 1: Evolution of decision boundaries and feature separability. PCA evolution of two Cifar10 classes (1% replay). Replay samples are highlighted with black edge. While features retain separability across tasks (low deep forgetting), the classifier optimization becomes under- determined: multiple "buffer-optimal" boundaries (dashed brown) perfectly classify the stored sam- ples but largely fail to align to the true population boundary (dashed green), resulting in shallow forgetting.
dimensional feature organization. In particular, the Neural Collapse (NC) phenomenon (Papyan 2020) describes a regime in which within-class variability vanishes, class means form an equiangular tight frame (ETF), and classifier weights align with these means. Originally observed in simplified settings, NC has now been documented across architectures, training regimes, and even large-scale language models (Súkeník et al. 2025 Wu & Papyan 2025). making it a powerful framework to analyze feature-head interactions.
In this work, extend the NC framework to continual learning, providing characterization of geometry of features and heads under extended training. Our analysis covers task-, class-, and domain-increment settings and explicitly accounts for replay. To account for this, propose two governing hypotheses for historical data absent from the buffer: (I) topologically, forgotten samples behave out-of-distribution (OOD) entities, and (2) enlarging the replay buffer induces smooth interpolation between this OOD regime and fully collapsed representations. These insights allow us construct simple yet predictive theory of feature-space forgetting that lower-bounds separability and captures the influence of weight decay, feature-norm scaling, and buffer size.
In summary, this paper makes the following distinct contributions:
1. The replay efficiency gap. We identify an intrinsic asymmetry in replay-based contin- ual learning: minimal buffers are sufficient to anchor feature geometry (preventing deep forgetting), whereas mitigating classifier misalignment (shallow forgetting) requires dis- proportionately large capacities.
2. Asymptotic framework for continual learning. We extend Neural Collapse theory continual learning, characterizing the asymptotic geometry of both single-head and multi-head architectures and identifying unique phenomena like rank reduction in task- incremental learning-
3. Effects of replay on feature geometry. We demonstrate that shallow forgetting arises because classifier optimization on buffers is mnder-determined- condition structurally exacerbated by Neural Collapse. The resulting geometric simplification (covariance defi- ciency and norm inflation) blinds the classifier to the true population boundaries.
4. Connection to OOD detection. We re-conceptualize deep forgetting as geometric drift toward out-of-distribution subspaces. This perspective bridges the gap between CL and OOD literature, offering a rigorous geometric definition of "forgetting" beyond simple ac- curacy loss.
1.1 NOTATION AND SETUP
We adopt the standard compositional formulation of neural network, decomposing it into a feature map and a classification head. The network function is defined as fo(x) = h(o(z)), where h(z) = Wa by, with parameters 0 {á, W.).
2

---

Published as a conference paper at ICLR 2026
We refer to the feature map, to its image the feature space and to ((x) as the features or representation of imput z.
We consider sequential classification problems subdivided into tasks. For each class dataset of labeled examples (X Yz) is available for training. Given any sample (æt y), the network prediction is obtained via the maximum-logit rule
ÿ arg max(us; (b(x)),
where wi denotes the k-th column vector of W.. Network performance is evaluated after each task on all previously seen tasks.
Saglloing[LLoee-PPaar & Ranzato 2017), shallow forgetting is quantified as the difference A.j Agj- where Aij denotes the accuracy on task j measured after completing learning session i. In contrast, deep forgetting defined as the difference Až, - -A where A represents the accuracy of linear probe trained on the frozen representations of task at the end of session i.
We adopt the three continual learning setups introduced by van de Ven al. (002)) described in detail in Section 3 dask-incrementar learning (TIL), insseenncremntal learning (CIL), and nomain--ncremment learning (DIL).
For the experimental analysis, we train both ResNet and VIT architectures, from scratch and from a pre-trained initialization. We train on three widely used benchmarks adapted to the continual learning setting: Cifar/00 (Krizhevsky & Hinton 2009), Tiny-ImageNet (TTrralba et 22088) and CUB200 (Wah et al.) 2011). A detailed description datasets and training protocols, including linear probing, is provided in Appendix A.1
2
EMPIRICAL CHARACTERIZATION OF DEEP AND SHALLOW FORGETTING
Cifar100, ResNet
Cifarl00, ViT
TinyIMG, ResNet
CUB200, ResNet
100
80
stallow
80
dlopp
Forgettigg
60
60
CIL
40
40
20
30
20
2
10 100
8
100
6
10
100
10
100
Buffer size (%)
Buffer size (%)
Buffer size (%6)
Buffer size (%)
Figure 2: Replay efficiency gap. Forgetting decays at different rates in the feature space and the classifier head, producing a persistent gap between deep and shallow forgetting. Increasing the replay buffer closes this gap only gradually, with substantial buffer sizes required for convergence. See Appendix A.2 for details.
We first present our main empirical finding. We evaluate forgetting in both the network output layer and a linear probe trained on frozen features across varying buffer sizes, datasets, and architec- tures (randomly initialized and pre-trained). Our results, summarized in Figure 2 reveal a robust phenomenon: while small replay buffers sufficient to prevent deep forgetting (preserving feature separability), mitigating shallow forgetting requires substantially larger buffers. This extends prior observations of feature-output discrepancies (Murata al. 20200; Hess al., 2023) by demonstrating that replay stabilizes representations far more efficiently than il maintains classifier alignment. The gap persists across settings, vanishing only near full replay (100%).
We highlight three specific trends:
1. Head architecture.
The deep-shallow gap is pronounced in single-head setups (CIL, DIL) but significantly smaller in multi-head setups (TIL).
2. Replay efficacy DIL. Contrary to the assumption that CIL is the most challenging bench- mark, DII. exhibits high levels of deep forgetting, converging to levels similar to CII..
3

---

Published as a conference paper at ICLR 2026
3. Pre-training robustness. Corroborating Ramasesh al. (2021), pre-trained models exhibit negligible deep forgetting. Their feature spaces remain robust even with minimal replay, yielding nearly flat deep-forgetting curves.
In the following section, we present theoretical model explaining this asymmetric effect of replay via the asymptotic dynamics of the feature space.
3 NEURAL COLLAPSE UNDER SEQUENTIAL TRAINING
3.1 PRELIMINARIES ON NEURAL COLLAPSE
characterizes the geometry of repre- sentations in the terminal phase training (TPT) the regime in which the training loss has reached zero and features stabilize. this regime, features converge to highly symmetric configuration known Neural Collapse (NC), which is provably optimal for standard supervised objectives and emerges naturally under a range optimization dynamics (Tirer Bruna 2022; Súkeník et al. 2025).
Recent work (Papyan al. 2020
Steinerberger 2022)
We denote the feature class means by (t(t) = lo Pe(t) the centered means, and the matrix of centered means by (t). We focus on first three properties defining NC:
NC1 (Variability Collapse). Within-class variability vanishes as features collapse to their class means: &e (z) - (e(t), V Xx, implying the within-class covariance approaches 0.
NC2 (Simplex ETF). Centered class means form simplex Equiangular Tight Frame (ETF). They attain equal norms and maximal pairwise separation:
B
if
lim (P)()), Fo (1)) t-0
cf
K-I
.C3 (Neural Duality). Classifier weights align with the class means up scaling, i.e., W,T (t) (t).
3.2 NEURAL COLLAPSE IN CONTINUAL LEARNING
Standard evaluation in continual learning measures performance strictly the completion of each task. Thus, while forgetting arises from optimization dynamics, its magnitude defined effectively by the network's configuration the end of training. We leverage the Neural Collapse framework to rigorously characterize this terminal geometry, modeling the stable structures that emerge in the limit of long training times [
While prior work focuses on stationary settings, we extend the NC framework to continual learning. We empirically verify its emergence in domain- (DIL), class- (CIL), and task-incremental (TIL)
settings (Figure 3 see Appendix C.4..
Observed vs. population statistics. NC emerges on the training data (current task + buffer). We must therefore distinguish between observed statistics î (computed on available training samples) and population statistics μ (computed on the full distribution). The following empirical analysis concerns fr; in subsequent sections, we develop a theory for μ to quantify forgetting.
3.2.1 SINGLE--EAD ARCHITECTURES
In domain-incremente learning (DIL), all tasks share fixed label set, with each task introducing a new input distribution. Consequently, while the estimated class means fae and global mean fáo evolve throughout the task sequence, the asymptotic target geometry remains invariant: the number of class means and their optimal angles are constant. We find that the NC properties established in
'Our analysis focuses structures at convergence; however, observe that Neural Collapse emerges quickly in practice standard architectures (cf. Figure
4

---

Published as a conference paper at ICLR 2026
NC1
NC2
.C3
Task-IL Domain-IL Class-II -I
10
0.00
1.00
0.05
0.50
-0.10
0.00
10
1.00
Opira
0.00
Optman Kal
0.50
0.10
0.00
1.00
0.05
between tasks
0.50
TE
0.00
157
472
157
314
472
0
157
314
472
Iterations (x100)
Iterations (x100)
Iterations (x100)
Figure 3: NC metrics in sequential training (Cifar100, ResNet with 5% replay). NC emerges across all tasks. In DIL, the ETF structure (NC2) remains stable; in CIL, it evolves as class count increases; in TIL, it arises per-head with variable cross-task alignment. Highlighted in green is the asymptotic limit of the NC metrics. See Appendix A.2 for details.
the single-task regime (Definitions 4 to 6 persist under DIL. When replay buffer is employed, the class means are effectively computed over the mixture new data and buffered samples.
In classincrrmmntal learning (CIL), each task introduces a disjoint subset of classes. The asymp- totic structure of the feature space therefore redefined after each task, governed by the relative representation of old versus dataset, they act as minority classes: their features collapse toward a degenerate distribution centered the origin, and their classifier weights converge to constant vectors (Fang et al. 2021 Dang eet al. 2023). This phenomenon, known as Minority Collapse (MC), occurs sharply below a critical representation threshold. Without replay, MC dominates the asymptotic structure past classes absent from the loss. However, we observe that replay mitigates this effect when buffers are sampled in lask-balanced manner. This strategy ensures that all classes both and old-are equally represented in each training batch, thereby preserving the global ETF structure and preventing the marginalization of past tasks (Figure
classes. When past classes under-represented in the training
3.2.2 MULTI-HEAD ARCHITECTURES
Neural Collapse has not previously been characterized in multi-head architectures. In task- incremental learning (TIL), the network output partitioned into separate heads, each associated with a distinct task. This ensures that error propagation is localized to the assigned head (see Fig- ure 25). While this local normalization prevents Minority Collapse even without replay, the resulting global geometry across tasks is non-trivee trivial.. Specifically, investigate the relative angles and norms between class means belonging to different tasks.
We measure standard NC metrics including within-class variance, inter-task inner products, and feature norms, Our findings reveal a clear distinction between local and global structure TIL:
1. Local collapse. NC emerges consistently within each head. Each task- head satisfies NC1-NC3 locally.
2. Global misalignment. A coherent cross-task NC structure absent. Across tasks, class means display variable scaling and alignment (Figure 3 Figures 12 to 14
3. Rank reduction. We find that local normalization induces a dimensionality reduction in the feature space. The global feature space attains a maximal rank of n(K for n tasks, which is strictly lower than the nk rank observed in single-head settings (Figure [T).
These empirical observations sppecfficaly that task-balanced replay restores global NC in single- head setups while TIL lacks global alignment- serve as the foundation for the theoretical model of class separability developed in the next section.
5

---

Published as a conference paper at ICLR 2026
4 ASYMPTOTIC BEHAVIOUR OF DEEP AND SHALLOW FORGETTING
4.1 PRELIMINARIES
Linear separability. To analyse deep forgetting, we require a mathematically tractable measure of linear separability in feature space. Formally, linear separability between two distributions Pi and P3 is the maximum classification accuracy achievable by any linear classifier. Given the first two moments (mi Σ④) and (μ22 Σ2), the Mahalanobis distance is standard proxy, Here, the signal-to-noise ratio (SNR) between class distributions, defined as
|1| 22 Tr(2; M2)
SN 02)
Higher SNR values imply greater separability. In Appendix C.2 we show that this quantity lower- bounds the Mahalanobis distance, and thus linear separability itself. Accordingly, we focus on the first- and second-order statistics of class representations (means and covariances), these directly govern the SNR.
Asymptotic notation. We use (.)) and (-)) to characterize the scaling of time-dependent quan- tities (t), suppressing constants independent of t. When bounds depend controllable quantities such as the buffer size b, we retain these dependencies explicitly. This notation highlights scaling behaviour relevant training dynamics and experimental design choices.
4.2 ANALYSIS OF DEEP FORGETTING
4.2.1
FORGOTTEN =
The Neural Collapse (NC) framework characterizes the asymptotic geometry of representations for training data. For- getting, however, concerns the evolution of representations for samples of past tasks that are longer part of the optimization objective. We bridge this conceptual gap through the following hypothesis:
10
Norm in s
T1
T2
T3
157 315 Iterations (x100)
473
Hypothesis 1. Forgotten samples behave analogously to sam- ples that were never learned, i.e., they are effectively out-of- distribution (OOD) with respect to the current model.
Figure 4: Projection of (a(t) onto S. (Cifar100, no replay). The population means of past and fu- ture tasks exhibit equivalent (near- zero) norms when projected onto the active subspace S..
This perspective motivates our analysis of forgetting as form of shift to out-of-distributtin in feature space. Specifically, in the absence of replay, data from past tasks exhibits the same geometric behaviour as future-task (OOD) inputs. To formal-
ize this correspondence, we adopt a feature-space definition of OOD based on the recently proposed ID/OOD orthogonality property (NC5, Ammar et al. 2024
Definition 1 catt-of-distrbbution (OOD)). Let X denote the samples of class and let (x) be the feature map of a network trained on dataset with classes. Denote by S, = (t), ..b......puu fix (t)} the active subspace spanned by the centered class means of the training data at time We say that X, is out-of-distribuu for di if the average representation of X, is orthogonal to S..
In Appendix C.6 (Proposition] 2 we show that, under the NC regime, the empirical observation that OOD inputs yield higher predictive entropy than in-distribution (ID) inputs is mathematically cquivalent to this orthogonality condition thus establishing a formal connection between predictive uncertainty and the geometric structure of NC5.
We validate our hypothesis by monitoring the projection of centered class means Aa. onto the active subspace S. As shown in Figure 4 (and Figures I7 19. shortly after task switch, the projection of tast means collapses sharply, indistinguishably matching the behavior of unseen (OOD) tasks.

---

Published as a conference paper at ICLR 2026
4.2.2 ASYMPTOTIC DISTRIBUTION OF OOD CLASSES
Leveraging the connection between forgetting and OOD dynamics, we now characterize the asymp- totic behavior of past-task data. We find that the residual signal of past classes is confined to the inactive subspace S making susceptible to erasure by weight decay.
Theorem 1 (Asymptotic distribution of OOD data). Let X be OOD inputs (Definition 7 for feature map de trained with a sufficiently small learning rate ij and weight decay X. Let Se denote the observed centered class-mean norm by Definition [ In the terminal phase (t 2 to), the feature distribution of X has mean P.c and variance o² given by:
(11m))) pe,st (tu), 02(1) ( 111))
(1)
Corollary (Collapse to null distribution). If > O, the OOD distribution converges to degener- ate null distribution: the mean decays zero, and the variance limits depend Bt-
The proof (see Theorem relies on the observation that, once NC3 (alignment between class fea- ture means and classifier weights) emerges, optimization updates become restricted to the active subspace St. Consequently, components of the representation in the orthogonal complement S are frozen-oo or decay exponentially under weight decay- yielding the dynamics above.
Notation. For brevity, let e = 17>, and note that S. = SSo = for all 2 l0-
Theorem (Lower bound on OOD linear separability). For two OOD classes c, in the TPT, let -- ma.. The Signal-to-Noise Ratio (SNR), which lower-bounds linear separability, satisfies:
(1 ((>00-00) +1)
)1)"))
SNR(c, E E
Discussion. Theorem 2 does not imply that separability necessarily vanishes; consistent with empirical findings (Figure 2). residual signal persists S' However, this signal is fragile. The result reveals the dual role of weight decay: it accelerates the exponential decay of the signal in S (reducing the numerator), yet simultaneously prevents the explosion of the class-mean norm 3 (constraining the denominator). Thus, weight decay both erases and indirectly preserves past-task representations.
We empirically observe that tends increase upon introducing new classes (Appendix A.3.3 which amplifies forgetting, as by Theorem 2 We hypothesize this is an artifact of classifier head initialization in sequential settings. Preliminary experiments, discussed in Appendix [A.3.3] lend support to this hypothesis; however, we leave a comprehensive investigation of this finding to future research.
4.2.3 ASYMPTOTIC DISTRIBUTION OF PAST DATA WITH REPLAY
Having that, without replay, past-task data behaves like OOD inputs drifting into S now consider how replay alters this picture. Replay provides a foothold in the active subspace 5, prevent- ing the collapse of old-task representations and preserving linear separability. Intuitively, the effect of replay should interpolate between the two extremes: no replay (DooD) and full replay (DNc).
The class structure in feature space emerges smoothly as a function of the buffer size, with past-task features retaining progressively larger component in S.
Hypothesis 2.
To formalize this intuition, we introduce a mixture model for the asymptotic feature distribution under replay. Let Te E [0, 1] denote a monotonic function of the buffer size, representing the fraction the NC-like component retained in Then, in the terminal phase of training, the feature distribution of class can be expressed mixture
((x) n. DNC (1 ne) DOOD-
This model is exact in the extremes (me 0 or 1) and interpolates for intermediate buffer sizes.

---

Published as a conference paper at ICLR 2026
Validation. Figure confirms that increasing replay transfers variance from S' to S, improving separabilityy We observe that stronger weight decay reduces norms and within-class variability glob- ally. Notably, find inverse relationship between buffer size and centered feature norms: while population means gravitate toward the global mean, representations small-buffers are subject to distinct repulsive force, pushing partially collapsed features outward. Finally, centered feature norms are consistently lower in DIL than in TIL or CIL.
This mixture model yields lower bound on the Signal-to-Noise Ratio (SNR), proving that replay guarantees separability asymptotically.
Theorem (Lower bound on separability with replay). Let c, d' be past-task classes and π E (0,1) the busfer mixing coefficient. In the TPT,
,² B. + 2((
π
SNR(c, c') E
0² + 3t + (
where 1² (1 x)²
Corollary 2. If > O (non-empry buffer), the SNR does not vanish: SNR(c,c')) E O(r²) as cc.
The corollary formalizes the intuition that any non-empty buffer anchors features in S. The anchor- ing strength r2 grows with buffer size; empirically, this growth is superlinear in single-head models (CIL, DIL) but sublinear in multi-head TIL.
SNR
WV
Norm norm
Centered mean norm
1.5
Class-11 1I
0.0001
0.3
1.0
0.2
5
0.0
1.5
Domain-11
1.0
0.2
0.1
4
2
0.0
Task-IL
0.4
0.5
observed
population
0.0
2
10
100
2
6
10
100
10 100
2
10 100
Buffer size [%6]
Buffer size [%)
Bufter size [%)
Bufter size (%)
Figure 5: Empirical validation for the theoretical model of feature space structure (Cifar100, ResNet with 5% replay). Plot shows the average over all past tasks after training the last task for four metrics. Results are shown for different buffer sizes and weight decay parameters (different lines). Details Appendix A.2
Discussion. These results rigorously establish replay as an anchor within the active subspace S. While the absence of replay forces representations into - c'uususing exponential signal decay- any non-empty buffer guarantees persistent signal proportional to r², ensuring asymptotic separability. Crucially, the efficiency of this anchoring varies by architecture: empirical trends (SNR, Figure ()) indicate sublinear growth of n, in single-head settings (CIL, DIL) versus superlinear growth in melti-head TIL, suggesting fundamental differences in how shared versus partitioned heads utilize replay capacity.
4.3
THE REPLAY EFFICIENCY GAP
We have established that even modest replay buffers suffice to anchor the feature space, preserving -ani-iiisiin Signal-to-Now Noise Ratio (mitigating deep forgetting). This resolves the first half of the puzzle. We now address the second half: why does this preserved separability not translate into classifier performance (shallow forgetting)?
8

---

Published as a conference paper at ICLR 2026
The under-determined classifier. Shallow forgetting arises from the fundamental statistical divergence between the finite replay buffer and the true population distribution. This divergence is structurally amplified by Neural Collapse. As noted by Hui et al. (2022), small sample sizes induce a "strong" NC regime where samples collapse aggressively to their empirical means (yielding smaller NC1 values, Figure [5]. Geometrically, this projects the buffer data onto
Mechanism:
low-dimensional subspace SB C S (rank s K - 1). However, the true population retains variance in directions orthogonal SB (specifically within S-).
This geometric mismatch renders the optimization of the classifier head an ill-posed, under- determined problem. Let W be the classifier weights. Since the buffer variance vanishes in di- rections orthogonal to SB, the cost function is invariant to changes in W along these directions. Consequently, the optimization landscape contains manifold of "buffer-optimal" solutions that achieve near-zero training error. However, these solutions can vary arbitrarily in the orthogonal complement of SB. leading to decision boundaries that are misaligned with the true population mass (as visualized in Figure D}. The classifier overfits the simplified geometry of the buffer, failing to generalize to the richer geometry of the population.
pepelation
115
Covariance gop
16
Covarinnee rank
000
observed ad
0144
Mean gou
15 18
0.3
Full popelation
Ciazi10 Tey
-0
Observe mears cov)
Full dleevel
10
100
100
2
100
Buffer (%)
Buffer (%6)
Buffer size (%6
Buffer sizze (9)
Figure 6: Deconstructing the statistical gap. Left and center-left: Gap (measured as L2 distance) between population and observed metrics. Center-right: Rank of the population (light shade) and observed (dark shade) covariance, the gap persists as the buffer size increased. Right: Synthetic Linear Discriminant Analysis (LDA) on TinyIMG. We replace true statistics (μ, Σ) with buffer esti- mates (ú, ε) to isolate error sources. Details in Appendix A.2
Mechanistic analysis of statistical divergence. We quantitatively decompose this divergence into two primary artifacts, validated via synthetic Linear Discriminant Analysis (LDA) counterfactuals (Figure [ methodological details in Appendix (A.2). First, covariance deficiency: the buffer's is em- pirical covariance Σg is rank-deficient and blind variance in S The criticality of second-order statistics evidenced by the sharp accuracy drop observed when replacing the true population co- variance with the identity matrix in LDA (gray line). Second, mean norm inflation: buffer means exhibit inflated norms relative to population means due to repulsive forces. Our LDA analysis con- firms that replacing population means with buffer estimates (olive line) causes a distinct, additive performance degradation. Notably, when relying on both observed estimates of mean and covari- ance (cyan line) the performance of the LDA classifier drops below the original network's 's perfor- mance.Metrics such as mean and covariance gap (Figure 6 Left) further confirm that these discrep- ancies particulalll the covariance rank deficiency- persist until the buffer approaches full size.
Implications. These findings mechanistically explain the replay efficiency gap: the feature space retains linear separability, yet the classifier remains statistically blinded to it. Consequently, simply increasing buffer size is inefficient, brute-force solution. Instead, our results suggest that to bridge the gap between shallow and deep forgetting. one must explicitly counteract the effects of Neural Collapse chansseeseecfficaly by preventing the extreme concentration and radial repulsion appearing in small buffers. We further elaborate on these implications the discussion of future work.
5 RELATED WORK
Our work intersects three main research directions: the geometry of neural feature spaces, out-of- distribution (OOD) detection, and continual learning (CL). A more detailed overview is provided in Appendix B
Below highlight the most relevant connections and contributions.

---

Published as a conference paper at ICLR 2026
Deep shallow forgetting. Classical definitions of catastrophic forgetting focus output degradation (shallow forgetting). More recent studies show that internal representations often retain past-task structure. recoverable via probes forgettin))) (Murata et al. 2020
Ramasesh
iet al. 2020, Fini et al. 2022 Davari et al. 2022; Zhang et al. 2022] Hess et al. 2023). Replay is
known mitigate deep forgetting in hidden layers (Murata et 20.00 Zhang et al. 2022). To our
knowledge, we are the first to demonstrate that deep and shallow forgetting scale fundamentally differently with buffer size.
Neural Collapse. NC describes the emergence of an ETF structure in last-layer features at conver-
gence (Papyan et al. 2020, Mixon et al.) 2022; Tirer& & Bruna 2022] Jacot et al.) 2024 Sūkeník
et al. 2025). Extensions address class imbalance (Minority Collapse) (Fang et al. [202]] Dang et al.,
2023
Hong & Ling 2023) and overcomplete regimes (Jiang al. 2024 Liu et al., 2023; Wu & Papyan 2024). In continual learning, NC has been leveraged to fix global ETF heads to reduce
forgetting (Yang et al., 2023 Dang et al.] 2024, Wang et al.) 2025). Our approach distinct: we ap- ply NC theory to the asymptotic analysis of continual learning and introduce the multi-head setting, common in CL but previously unexplored in NC theory.
OOD detection. Early work observed that OOD inputs yield lower softmax confidence (Hendrycks & Gimpel, 2018), while later studies showed that OOD features collapse toward the origin due to low-rank compression (Kang et al. 2024). Recent results connect this behavior to NC: L2 regu- larization accelerates NC and sharpens ID/OOD separation (Haas et al.) 2023), and ID/OOD or- thogonality has been proposed
additional NC property, with OOD scores derived from ETF subspace norms (Ammar et al. (0024). Our work extends these insights by formally establishing orthogonality, clarifying the role of weight decay and feature norms, and crt-icccllilllooviiiiiiii the first explicit link between OOD detection and forgetting in CL.
6 FINAL DISCUSSION & CONCLUSION
Takeaways. This work has shown that: (1) replay affects network features and classifier heads in fundamentally different ways, leading slow reduction of the replay efficiency gap as buffer size increases; (2) the Neural Collapse framework can be systematically extended to continual learning, with particular emphasis on the multi-head setting a case not previously addressed in the NC literature; (3) continual learning can be formally connected to the out-of-distribution (OOD) detection literature, and our results extend existing discussions of NC on OOD data. We further elucidated how weight decay and the growth of class feature norms jointly determine linear separability in feature space. Our analysis also uncovered several unexpected phenomena: (i) class feature norms grow with the number of classes in class- and inskeincremental learning; (ii) multi-head models yield structurally lower-rank feature spaces compared to single-head models; and (iii) weight decay exerts double-edged influence feature separability, with its effect differing across continual learning setups.
Limitations. Our theoretical analysis adopts an asymptotic perspective, thereby neglecting the tran- sient dynamics of early training, which are likely central to the onset of forgetting (Lapacz et al., 2024.. Moreover, our modeling replay buffers interpolations between idealized extremes sim- plifies the true distributional dynamics and may not fully capture practical scenarios. Finally, many aspects of feature-so space evolution under sequential training parttccularllyy the nature of cross-task in- teractions in multi-head architectures-renne poorly understood and require further investigation.
Broader Implications. By establishing a formal link between Neural Collapse, OOD represen- tations, and continual learning, our findings highlight key design choices- includinc buffer size, weight decay, and head structure-that shape the stability of past-task knowledge. These results raise broader questions: What constitutes an "optimal" representation for continual learning? Is the Neural Collapse structure beneficial or detrimental in this context? Our results suggest that while NC enhances feature organization, it also exacerbates the mismatch between replay and true distributions, thereby contributing to the replay efficiency gap. Addressing these open questions will be essential for designing future continual learning systems.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Mouin Ben Ammar, Nacim Belkhir, Sebastian Popescu, Antoine Manzanera, and Gianni Franchi.
NECO: NEural Collapse Based Out-of- detection, February 2024. URL http://
arxiv. org.v.org/aae httg//bb///e
mrXiv:2310.06822 [stat].
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDER- ARA. Dark experience for general continual learning: a strong, simple baseline. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neu- ral Information Processing Systems, volume 33, pp. 15920-15930. Curran Associates, Inc., files/paperx/202//
2020. URL httpps//ppoccecdass...urrrssssss
file (tt.://0ee222777700006777760eee0
Hien Dang, Tho Tran, Stanley Osher, Hung Tran-The Nhat Ho, and Tan Nguyen. Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data, June 2023. URL/attp://arxiv.
org/abs/ 2301 00437
arXiv:2301 .00437 [es].
Trung-Anh Dang, Vincent Nguyen, Ngoc-Son Vu, and Christel Vrain. Memory-efficient Contin- ual Learnnng with Neural Collapse Contrastive, December 2024. URL TRRLittp://axxiv.oo
abs/2412 02865 Siii:2212.02866 [cs].
MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Fugene Belilovsky, Prob- ing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 167 16712-16721 2022.
Cong Fang. Hangfeng He, Qi Long, and Weijie Su. Exploring Deep Neural Networks via Layer-
Peeled Model: Minority Collapse in Imbalanced Training. Proceedings of the National Academy of Sciences, (144))::213001110
October 2021. ISSN 0027-8424, 1091-6490. doi: 10.1073/ org/ /tt. 12699 arXiv:2101. 12699 [cs].
pnas.2103091118 18. URL htt.tttp:/
Enrico Fini, Victor G Turrisi Da Costa, Xavier Mlameda-Pinedaa Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual leamerr. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9621-9630, 2022.
Jarrod Haas, William Yolland, and Bernhard Rabus. Linking Neural Collapse and 1.2 Normalization with Improved Out-of-Disttributin Detection in Deep Neural Networks, January 2023. URL
http::/ httt:://axxivoor//bbb/sss
www./rgg/......... 2209 08378 arXiv:2209.0833 [cs].
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual 2020. doi: 10.
learning in deep neural networks. Trends in Cognitive Sciences,
/016//..icc.2022..oo 004. URL https: / wttp://///...r///////// wttp://....1111......... 2020 09. 004
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and outtaaaf Examples in Neural Networks, October 2018. URL http:://arxxvv.org/abb////
02136
[cs].
Timm Hess, Eli Verwimp. Gido M van de Ven, and Tinne Tuytelaars. Knowledge accumula- tion in continually learned representations and the issue of feature forgetting. arXiv preprint N.Xiv:2304.009333 2023.
Wanli Hong and Shuyang Ling. Neural Collapse for Unconstrained Feature Model under Cross- entropy Loss with Imbalanced Data, October 2023. hwR.:ttt:://aaxii.orr/////// 09725 arXiv:2309.099 09922 [stat] version: 2.
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384 2022.
Arthur Jacot, Peter Súkeník, Zihan Wang, and Marco Mondelli. Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse, October 2024. URL http:///rxxiv
hog/abs/220 2220/2210..m 04887 arXiv:2410.0488) [cs] version: 1.
Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin G. Mixon, Chong You, and Zhihui Zhu. Generalized Neural Collapse for Large Number of Classes. In Proceedings of the 41st International Conference on Machine Learning, pp. 22010 22041. PMLR, July 2024. URL ISSN: 2640-3498.
.html

---

Published as a conference paper at ICLR 2026
Katie Kang, Amrith Setlur, Claire Tomlin, and Sergey Levine. Deep Neural Networks Tend
To Extrapolate Predictably, March 2024. URL atttp://arxta httpa//wwww..gg//////..a....... 00873
arXiv:2310.00873 [es].
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers features from tiny im-
ages. https://wwl es www.nto...e.o...oooo wwtpronttoed//kkii22 http:://wwt.eeeeaaaaaaaatttteeee
pdf
2009.
Weiyang Liu, Longhui Yu, Adrian Weller, and Bernhard Schölkopf. Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap, April 2023. URRlhhttp///axxi org/ [es] version: 2.
abs/2303. 06484 arXiv:2303.066 06484444444
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Com- putational Harmonic Analysis, 59:224-244 2022. Publisher: Elsevier.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022. Publisher: Springer.
Kengo Murata, Tetsuya Toyota, and Kouzou Ohara. What is happening inside a continual learning model? representation-based evaluation of representational forgetting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 234-235, 2020.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 14662-24663, 2020. Publisher: National Acad Sciences.
Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu, Anatomy catastrophic forgetting: Hidden representations and task semantics. arXiv preprint arXiv:2007.0744 2020.
Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, October 2021.
Peter Súkeník, Christoph H. Lampert, and Marco Mondelli. Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers, May 2025. URL http://axxiv.org/baa arXiv:2505. 15239 [cs].
2505. 15239
Tom Tirer and Joan Bruna. Extended Unconstrained Features Model for Exploring Deep Neural Collapse, October 2022. URL httt:://aaxxxi/ org. wwtp://wwwww.rr/ab////22/.. 08087 arXiv:2202.08087 [cs].
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine 2008.
intelligence, 30(11):1958-11)
Gido M van Ven, Tinne Tuytelaars, and Andreas Tolias. Three types of incremental learning. doi: 00.1038/44225--02-000666 URL https:://da 222 022-00568-2
Nature Machine Intelligence, 4:1185-1197, 2022.
Roman Vershynin. High- DighenDimesionall Probability: An Introduction with Applications in Data Sci- ence. Cambridge University Press, Cambridge, UK, 2018. ISBN 9781 108415217.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-200 dataset. 2011.
Zheng Wang, Wanhao Yu, Li Yang, and Sen Lin. Rethinking Continual Learning with Pro- gressive Neural Collapse, May 2025. URL http::/ /ttp//rriiv./...////v///..//..il wttgabs//205..202 24254 arXiv:2505.2423 [es].
12

---

Published as a conference paper at ICLR 2026
Robert Wu and Vardan Papyan. Linguistic Collapse: Neural Collapse in (Large) Language Models, November 2024. URL http://aawx....///////000.0..0
arXiv:2405.17767 [cs]
version: 2.
Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. Advances in Neural Information Processing Systems, 37:137432-137773, 2025.
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural col- lapse inspired feature-classffier alignment for few-shot class incremental learning. arXiv preprint wrXiv:2302.03004, 2023.
Xiao Zhang, Dejing Dou, and Ji Wu. Feature forgetting in continual representation learning- arXiv preprint arXiv:2205. 2022.
Wojciech Lapacz, Daniel Marezak, Filip Szatkowski, and Tomasz Trzciński. Exploring the Stability Gap in Continual Learning: The Role of the Classification Head, November 2024. URL http: 7/arxiv. wttpvv/..gg/bbb///11..... 04723 2Xiv:2411.04722 [cs] version: 2.
13

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
Empirical Appendix
15
A.1 Experimental ectails
15
A.2 Figure details
16
A.3 Ablations
17
Additional higures and empirical substantiation
B
Overview of related work
26
C
Mathematical derivations
28
C.1
Setup
28
C.2
Lincar Scparability
29
C,3
Terminal Phase of Training (TPT)
30
Neural Collapse continual learning Setup
31
C.5
Main result stabilization the training feature subspace.
35
C.6
Another definition of OOD]
36
Asymptotics of OOD data
37
Main results 3: feature space asymptotic structure with replay.
39
14

---

Published as a conference paper at ICLR 2026
A EMPIRICAL APPENDIX
A.1 EXPERIMENTAL EETAILS
We utilize the benchmark codebase developed by Buzzega et al. (2020 To accommodate our experiments we performed several changes to the default implementation.
Training configurations. Table I summarizes the configurations used in our main experiments. All models are trained in an offline continual learning setting, where each task's ' dataset is trained for a specified number of iterations before transitioning to the next task. Models are trained to reach error convergence on each task and more training does not improve performance. For all experiments, the random seeds were set to [1000, 2000, 3000).
We define the number of tasks as specified in Table[] The classes of each dataset are then distributed across these tasks. the DIL setting, the same class labels are reused for every task. The class or- dering was randomized in each run, meaning that specific tasks consist of different classes in each run. This was done to ensure that the results are not biased by a specific class sequence. However, observed that this increases the variance when metrics evaluated task wise compared to using a fixed class assignment.
Dataset
Tasks
Epochs first task
Network
Batch Size
C0
10
200
ResNet18 (11M)
64
C0
10
40
ViT base, pretrained on ImageNET (86M)
256
TinyIMG
10
200
ResNet18 (11M)
64
CUB200
10
80
ResNer50, pretrained on ImageNET (24M)
32
Table 1: Experiment configurations.
Hyper parameters Our hyper parameters were largely adapted from (Buzzega et al. 2020) and are listed in T We use a constant learning rate. For all buffer sizes the same hyper parameters are used. Finally to study the effects of weight decay, vary the weight decay strength in our experiments while keeping all other factors constant.
Dataset
Method
Optimizer
Hyper Parameters
Cifar100, ResNet
ER
SGD
Ir 0.1, wd 0.0001
Ci ResNet
DER
SGD
Ir : 0.03, a = 0.3,
C0 ResNet
FDR
SGD
Ir 0.03, a 0.3,
Cifar1 00, ResNet
iCaRL.
SGD
Ir : 0.1, wd 0.00005
Cifar1001 VIT
ER
AdamW
Ir 0.0001, wd 0.0001
TinyIMG, ResNet
ER
SGD
(r 0.1, wd 0.0001
CUB200, ResNet
ER
SGD
Ir 0.03, wd 0.0001
Table 2: Hyper parameters
Datasets and preprocessing. We adopt publicly available image classification benchmarks: Ci- far100 (32 x 32 RGB, 50000 samples across 100 classes), TinyIMG (64 x 64 RGB across 100000 samples, 200 classes) and CUB200 (224 x 224 RGB, 12000 samples across 200 classes). Standard train/test splits used. We apply standard augmentations like random crops and flips, without increasing the dataset size.
Experience Replay (ER). In our implementation of ER, we adopt a balanced sampling strategy in which each task contributes equally to the mini-batches. While this strategy would normally require
Thhei codebase is publicaly availihle at: hhtp://ghhha..coommmaaggaaamammm
15

---

Published as a conference paper at ICLR 2026
more iterations for later tasks, we avoid this by fixing the total number of iterations across all tasks to match those performed on the first task effectively reducing the number of epochs for later tasks.
To maintain precise control over the buffer composition, employ an offline sampling scheme to populate the buffer. Samples (together with their labels) from a task are added to the buffer only after training on that task is completed. This guarantees balanced number of stored samples per task and class. Because the buffer size specified as a percentage of the task-wise dataset, the number of stored samples per task remains constant and equal the chosen buffer percentage. As more tasks are encountered, these fixed per-task allocations accumulate, resulting in a steadily increasing overall buffer size. Table B] specifies the buffer sizes used in the experiments. When setting the buffer size to O, ER naturally reduces to standard SGD.
Buffer Sizes (% of task-wise dataset)
0, 1, 3, 4, 5, 6, 8, 10, 100
Table 3: Buffer sizes expressed as percentages of each task-wise dataset. The same values are used consistently across all experimental configurations.
Measures of superficial and deep forgetting. accuracy past tasks after learning ones, defined as
Shallow forgetting quantifies the drop in output
F shalloow b
= AAj= - Anj.
where Aij is the accuracy on task j measured after learning session i.
Deep forgetring measures the loss of discriminative information in the features themselves, inde- pendent of the head. To measure it, we train logistic regression classifier (scikit-learn' Logisti- cRegression, default settings, C=10) on frozen features extracted from the full dataset after learning session The resulting accuracy, evaluated the end of session j, is denoted by Aij- Formally,
pdeep- A - Arz-
For single-head models, one probe is trained over all classes; for multi head architectures, one probe per task-specific head is used.
A.2 FIGURE DETAILS
This subsection details the computations behind the figures presented in the main text.
Forgetting metrics are evaluated after the final training session, following the procedure described in Appendix A.I] and across buffer sizes specified in Table Different line styles corre- spond to distinct continual learning settings (TIL, DIL, CIL).
Figure 2
Figure 3 Neural Collapse (NC) metrics are computed for each task every 100 steps during training of a ResNet from scratch on CIFAR100 in both CIL, DIL and TIL settings. Metrics are evaluated on the available training data, which includes the current task's 's dataset plus the replay buffer which contains 5% of the past tasks' dataset. In TIL, for NC2 the within-elass-pai values for each task are shown in the standard task colors, while the values across class pairs from different tasks are highlighted in violet. The brown vertical lines indicate the task switches.
Note that DIL each task contains same set of classes. If class means were computed naively across tasks, differences specific to each task would be obscured, preventing us from identifying task-wise trends. Therefore Figure evaluates the NC metrics separately for each task. In con- trast Figure 15 reports the NC metrics computed jointly over all tasks in DIL. Importantly, both approuches produce consistent results.
Figure 4 Norm of (t)(t) projected S,, averaged over all classes belonging task, is computed every 100 steps during training of a ResNet from scratch on CIFARIO0 under CIL. The brown vertical lines indicate the task switches. In DIL (t())) is computed separately for every task.
Figure 5
Measurements are collected after the final training session on Cifar 100 using a ResNet trained from scratch, averaged over all past-task classes. In DII. again calculate the class means
16

---

Published as a conference paper at ICLR 2026
task wise. The buffer sizes correspond those listed in Table 3 The signal-to-noise ratio (SNR) computed as described in Section [.I] The second panel displays the normalized variance ratio, where the within-clad variance is defined as Tट Deec (r(Cov(((2)) I E X.)), and the between- class variance is defined as Tr( [r(Cov( (fuetcec)) with #e the population feature mean vector of class The third panel displays the average ratio: ||S] (e(t)]l [|,]ac(t]]] And the fourth panel displays the average norm of (t)(t) and (t))))
Figure 6 (first three plots). At the end of the last training session, the network evaluated on multiple datasets under CIL protocol. For CUB200 do not report bufter sizes which smaller then 4%, as at least two samples per class are needed to calculate the covariance matrix. For each buffer size reported in Table 3 we collect the class-wise mean vectors Ae((t) and covariances E.(t) from the buffer, as well the corresponding population statisties (e((t and Ee(t). The following metrics are computed and averaged across past classes:
Mean gap: (t)e(t) (t)(2)
Covariance gap: [EE(()) Et(t) the Frobenius norm of the difference between covari- ances.
Covariance rank: both the rank of the population covariance matrix (t()) and the observed covariance matrix (t)t) are reported. Note that the rank upper bounded by the number of samples which are used to calculate the covariance matrix.
These quantities quantify the discrepancy between the buffer and class distributions in feature space, which drives shallow forgetting.
Figure 6 (right-most panel). Same experimental setup as the first three panels. We evaluate differ- ent linear classifiers on TinyIMG using class-wise feature statistics. Specifically, we construct linear discriminant analysis (LDA) classifiers. For class c, the LDA decision rule
((x) = arg max (x
AAc(t),
where Aec denote the estimated class mean and E is shared covariance matrix. We vary the estimates used for each class as follows:
Full population: both mean (t()) and covariance (t) are measured population, M(t) is pooled across all classes.
Population means (ID cov): mean taken from population pa((), but covariance is fixed to the identity.
Observed means (ID cov): mean is taken from observed samples (c(t), but covariance is fixed to the identity.
Full buffer: both mean (t) and covariance E(t are computed from the observed sam- ples, ε(() is pooled across all classes.
This evaluation highlights how errors in buffer-based mean and covariance estimates contribute to shallow forgetting, and quantifies the impact of each component on linear decoding performance.
A.3
ABLATIONS
A.3.1 EFFECT OF PRETRAINING
Our results in Iigureademonsrrall that models trained from scratch indeed undergo Neural Collapse (NC) in a continual learning setting. However, when comparing this to pre-trained models, we find that while both settings converge to the same asymptotic feature geometry, the pre-trained models do substantially accelerated rate. This difference in convergence speed is illustrated in Figure 7 A side-by-side comparison of the initial 1500 iterations confirms that pre-trained models rapidly achieve the high NC scores that their de novo counterparts only reach much later in training.
17

---

Published as a conference paper at ICLR 2026
from scratch
0.75
WC1
10
pretrained
0.5
0.50
mm
0.0
0.00
Iterations (x100)
Iterations (x100)
Iterations (x1 00)
Figure 7: Convergence Neural Collapse (NC) for pre-trained versus from-scratch models under CIL on CUB200. Pre-trained models achieve asymptotic NC scores significantly faster than their de novo counterparts.
A.3.2 FEATURE BOTTLENECK; WHEN d & K
In the main paper, considered settings where the feature dimension exceeds the number of classes. However, in many practical applications, such as language modeling, the number classes (e.g.. vocabulary size) typically much larger than the feature dimension. Recent work by (Liu et al. 2023) explored this regime. To examine how our framework behaves under these conditions, conducted additional experiments by modifying the Cifar100 with ResNet setup. Specifically, we split Cifar100 100 into four tasks of 25 classes each and inserted bottleneck layer of dimension 10 between the feature layer and the classifier head. All other components are left unchanged.
As illustrated in Figure variability collapse .C11 and neural duality NC3 remain in this con- strained setting. However, the equiangularity NC2 exhibits significant degradation. While the mean pairwise cosine similarity aligns with theoretical expectations, its standard deviation increases sub- stantially to 0.3 compared to the typical convergence levels (Figure 12 and similarly observed by Papyyan et al. (2020). Therefore, even though the mean ap- pears correct, the underlying structure is not, as the standard deviation far to high. This high variance indicates failure to converge to rigid simplex, suggesting that in the d < K regime al- ternative geometric structures must be considered, such as the Hyperspherical Uniformity explored by] Liu et al. (023).
0.1 in our standard setting
NC1
.C2
NC3
7.5
T1
0.6
var
1.0
Class-IL
5.0
T2
0.4
meun
2.5
0.2
0.0
7.5
0.6
Domain-Il
5.0
0.4
2.5
0.2
0.0
7.5
0.6
between tasks
Task-IL
5.0
0.4
0.5
2.5
0.0
382
382
Iterations (x100)
Iterations (x100)
Iterations (x100)
Figure 8: NC metrics in the bottleneck regime (d=10). Same setup Figure Results for Ci (4 tasks, 25 classes, 5% replay). While variability collapse (NC1) and duality (NC3) persist, the rigid ETF structure (NC2) degrades, exhibiting high variance in pairwise angles.
Crucially, we find that the replay efficiency gap persists (Figure 8 despite this geometric shift. This implies that the decoupling between feature separability and classifier alignment is not contingent
18

---

Published as a conference paper at ICLR 2026
the specific ETF geometry. Rather, the gap is a fundamental phenomenon that emerges when the learned representations follow alternative geometric structures, provided they remain collapsed.
Dataset
Learning paradigm
Shallow forgetting
Deep forgetting
C0
CIL
52.27 + 2.41
26.15 + 1.92
C0
DIL
46.38 + 1.88
35.07 = 0.51
Cifar100
TIL
18.45 + 2.78
14.89 + 1.77
Table 4: The deep-shallow forgetting gap persists in the low feature- dinture-immenion regime (Cifar100,
ResNet with 5% replay).
A.3.3 EFFECT OF HEAD INITIALIZATION
We analyze the empirical evolution of the centered class-mean norm 3r across tasks. As illustrated in Figure 9 we observe a distinct architectural split: 3r increases monotonically in setups with increasing number of classes (CIL and TIL), whereas it remains asymptoticallyy stable in DIL.
We attribute this drift to a weight norm asymmetry induced by the sequential expansion of the net- work outputs. In CIL and TIL, new head weights are typically instantiated using standard schemes (e.g.. Kaiming Uniform), which initialize weights with significantly lower norms than those of the prready-converee's heads from previous tasks. This creates recurrent initialization shock. In trast, DIL employs a fixed, shared head across all tasks, inherently avoiding this discontinuity.
Class-IL
Domain-IL.
Task-IL
Class-IL
15.0
observed
12
12.5
populstion
Norm
10.0
0.0001
0,0003
7.5
5.0
6
2
4
6 Task
8
10
2
4
10
2
4 6 Task
8
10
6
10
Task
Task
Figure 9: Average (over all classes norm of the centered observed class means (i) and population class means (t) after training each task on Cifar100, with varying weight decay coef- ficients. The three panels on the left correspond progressively increasing norm in both CIL and TIL. The rightmost panel shows the results when each new head is initialized with the same norm as the previously trained heads. This adjustment prevents the norm from growing.
the default head initialization, which results in
To validate this hypothesis, we performed an ablation using a norm-matching initialization strategy- In this setup, the weights of new tasks are scaled to match the average norm of existing heads while preserving their random orientation.
Results in Figure (right) confirm that this intervention effectively suppresses the progressive growth of St. recovering the stationary norm behavior observed in DIL. Interestingly, while this adjustment stabilizes the geometric scale of the representation, we found it yields negligible impact on final forgetting or test accuracy metrics.
A.4 ADDITIONAL FIGURES AND EMPIRICAL SUBSTANTIATION
This subsection includes placeholder figures for concepts discussed in the main text, for which specific existing figures were not available or suitable for direct inclusion in the main body.
19

---

Published as a conference paper at ICLR 2026
Cifar100, ResNet
Cifar100, VIT
TinyIMG, ResNet
CUB200, ResNet
shallow
80
80
deep
Accuracy
60
60
60
60
CIL
40
40
TIL
20
20
20
20
2
100
2
8 100
2
100
10 100
Buffer (%)
Buffer size (%)
Bufter size (%)
Buffer size (%)
Figure 10: Same setup as Figure
This plot reports test accuracy.
DER
FDR
iCaRL
80
80
shallow
Forgetting
deep
DIL
40
40
10
Buffer Size (%)
Bulfer Size (%)
Buffer Size (%)
Figure 11: Deep- shallow forgetting gap for Dark Experience Replay (DER), Functional Distance Relation (FDR) and Incremental Classifier and Representation Learning (iCaRL) on Cifar100 with ResNet. Note that iCaRL. does not support DIL nor training without buffer.
NC1
NC2
NC3
T1
Var
Class-IL
T2
menn
5
0.0
10
0.6
Domain-IL
0.4
5
0.6
between tasks
Task-IL
0.4
0.2
0.5
0.0
137
Iterations (x100)
Iterations (x100)
Iterations (x100)
Figure 12: Same setup as Figure
This plot shows the NC metrics on Cifarl00 100 with 5% replay. For NC2, both the mean and standard deviation are shown.
20

---

Published as a conference paper at ICLR 2026
NC1
A/2
NC3
0.6
1.0
11
var
Class-Il IL
T2
0.4
meen
0.5
0.0
0.0
0.6
Domain-ll.
0.5
0.2
0.0
0.0
0.66
between tasks
Task-Il
0.4
0.5
0.2
0.0
Iterations (x100)
Iterations (x100)
Iterations (x100)
Figure 13: Same setup Figure
This plot shows the NC metrics on TinyIMG with 5% replay. For NC2, both the mean and standard deviation are shown.
AC1
NC2
NC3
1.0
var
Class-IL
0.2
meea
0.5
0.0
1.0
Domain-Il
0.2
0.5
0.0
0.0
between tasks
Task-IL
0.2
0.5
0.0
Iterations (x100)
Iterations (x100)
Iterations (x100)
Figure 14: Same setup as Figure This plot shows the NC metrics on CUB200 with 10% replay. For NC2, both the mean and standard deviation shown.
21

---

Published as a conference paper at ICLR 2026
Class-IL
Class-IL
Class-IL
1.25
0.2
1%
1.00
0.1
0.75
10%
0.50
0.0
mean
0.85
0.25
0.1
6
8
10
2
4
6
10
2
4
6
8
10
Task
Task
Task
Domain-IL
Domain-IL
Domain-II IL
1.0
.22
0.1
0.0
0.85
0.2
-0.1
0.80
10
4
6
4
6
Task Task-IL
Task
Task
Task-IL
.C22 (belween tasks)
Task-IL
Task-IL
1.5
./C2 (within task)
0.925
0.2
0.2
0.900
0.875
0.5
0.0
0.1
0.850
-0.1
0.0
0.825
10
2
Task
Task
Task
Task
Figure 15: Evolution of Neural Collapse metrics over all tasks in sequential training (Cifar100,
ResNet) varying the replay buffer size. Neural Collapse is stronger for smaller buffers.
C00 ResNet
Cifar100, ViT
TinyIMG, ResNet CUB200, ResNet
100
100
200
200
CIL
75
DIL
75
150
150
Rank
TIL.
50
50
100
100
25
25
50
50
2
6
6
10
10
4
10
Task
Task
Task
Task
Figure 16: Rank the centered observed class mean matrix (1). CIL and TIL rank increases (at different speeds) more tasks are learned, whereas in DIL the rank remains remains constant.
Class-IL
Domain-IL.
Task- ILL
10
Norm in s
15
10
10
5
T1
72
157
315
473
157
315
473
157
315
473
Iterations (x100)
Iterations (x100)
Iterations (x100)
Figure 17: Same setup as Figure for CIL, DIL and TIL on C0 with replay.
This plot shows the average norm of (e(()) when projected to S,
22

---

Published as a conference paper at ICLR 2026
Class-IL
Domain-IL
Task-1 IL
7.5
7.5
Norm in
5.0
5.0
2.5
2.5
Iterations (x100)
Iterations (x1
Iterations (x1 00)
Figure 18: Same setup as Figure This plot shows the average norm of (a(()) when projected to S. for CIL, DIL and TIL on TinyIMG with replay.
Class- IL
Domain-IL
Task-IL
20
Norm in
15
15
10
T2
T3
45
29
Iterations (x100)
Iterations (x100)
Iterations (x100)
Figure 19: Same setup Figure This plot shows the average norm of ji(()) when projected to S,
for CIL, DIL and TIL on CUB200 with no replay.
SNR
WV / BW
Norm $ /norm S
Centered mean
0_0003
0.3
Class-11 1L
DO01
0.1
0.1
Domain-1l
0.2
0.4
2
0.2
0.0
0.3
10
3
Task-IL
2
0.1
observed
0.4
0-0
population
0.0
10 Buffer size [%]
100
2
100
2
10
100
4
100
Buffer size [%)]
Buffer size [%]]
Buffer size [%]
Figure 20: Same setup as Figure
5 This plot displays the results for TinyIMG.
23

---

Published as a conference paper at ICLR 2026
SNR
WV / BW
Norm s norm
Centered mean norm
0.0003
0.8
0.4
Class-Il II
0000
0.6
0.3
0.2
0.1
0.2
2.0
Domain 11
2.0
1.5
0.3
1.5
0.1
0.5
0.5
0.4
Task-
4
0.3
20
0.3
observed
0.1
16
populaatio
0.0
8 10
100
2
10
100
2
4
10
100
0
2
10
100
Buffer size [%]]
Buffer size [%6]
Buffer size [%6]
Buffer size [%]
Figure 21: Same setup Figure
This plot displays the results for CUB200.
Cifarl 00,
Class-IL.
Tice aui Lia Date
-00
30
Class
10
12
110
15
Class
Figure 22: Class-wise inner products of the centered population class means Ao on Ci
under CIL after the second task. Classes O to belong to the first task, while 10 to 19 belong to the second task. The classes belonging to task are structured according to the NC regime, while classes belonging to task 1 show structure.
24

---

Published as a conference paper at ICLR 2026
Cifar100, Task-IL.
40
Class
20
15 12--0-
0028
38 6C
127
so
102 13c 01.2
13.8
Class
Figure 23: Class-wise inner products of the centered population class means (t( Cifar100 under TIL after the second task. Classes 0 to belong to the first task, while 10 19 belong to the second task. The classes belonging task 2 structured according to the NC regime, while classes belonging to task show no structure.
Cifar 00,, Domain-IL. Task 1
Cifarl 100, Domain- Task
400
Class
Class
20
20
0AS
340
137
40
40
524
0.0
30:11
3M0 20
Class
Class
Figure 24: Class-wiss wise inner products of the centered population class means (t) on Cifar 100 under DIL after the second task. The left plot shows the results for samples belonging the first task, while the right plot shows results for samples from the second task. Task 2 is structured according to the NC regime, while task I shows no structure.
25

---

Published as a conference paper at ICLR 2026
B OVERVIEW OF RELATED WORK
Our work intersects several strands of research. First, it builds on the literature studying the geomet- ric structures that emerge in neural feature spaces, extending these analyses to the sequential setting of continual learning and accounting for the additional challenges introduced by different head ex- pansion mechanisms. Second, it connects to the out-of-distribution detection literature, where we reinterpret forgetting as feature drift and broaden existing insights to more general framework. Finally, it contributes to the continual learning literature that disentangles knowledge retention at the representation level from that at the output level, highlighting the systematic mismatch between the two in replay.
Deep and shallow forgetting Traditionally, catastrophic forgetting is defined as the decline in network's performance on a previously learned task after training on a new one, with performance measured at the level of the network's outputs. We refer to this notion as shallow forgetting. In con- trast, MMrrata et al. (2020) highlighted that forgetting can also be assessed in terms of the network's 's internal representations. They proposed quantifying forgetting subsequent layers (+1 to L on past data and comparing the resulting accuracy to that of the original network. Applied last-layer features, this procedure coincides with the widely used linear probe evaluation from the representation learning literature, often complemented by kNN estimators, to assess task knowledge independently of a task task-specific head. In this work, we refer to the loss of information at the feature level as deep forgetting. This probing- probing-based approach has also been adopted in continual learning studies (Ramasesh et al.) [2020] Fini et al. 2022). Multiple works have since reported a consistent discrepancy between deep and shallow forgetting across diverse settings (Davari et al. 2022, Zhang et al. 2022 2| Hess et al.] (023). Of particular relevance to our study are the findings of Murata et al. (2020) and Zhang et al. (2022), who observed that replay methods help mitigate deep forgetting in hidden representations. To our knowledge, however, we are the first to demonstrate that deep and shallow forgetting exhibit categorically different scaling behaviors with respect to replay buffer size.
hidden layer by retraining the
Neural Collapse and continual learning Neural Collapse (NC) was first introduced by Papyan jet al.) (2020) to describe the emergence of a highly structured geometry in neural feature spaces, namely a simplex equiangular tight frame (simplex ETF) characterized by the NCI-NC4 prop- erties. Its optimality for neural classifiers, as well as its emergence under gradient descent, was initially established under the simplifying unconstrained featur feature model (UFM) (Mixon al. 2022)
Subsequent theoretical work extended these results to end-to-end training of modern architectures with both MSE and CE loss standard classification tasks (Tirer & Bruna 2022 Jacot al., 2024] Súkeník et al. 2025). Generalizations of NC have been proposed for settings where the number of classes exceeds the feature dimension, precluding simplex structure. In such cases, the NC2 and NC3 properties are extended via onc-vs-all margins (Jiang et al.j 2024) or hyperspherical uniformity principles (Liu et al., 2023] Wu & Papyan 2024). Another important line of work concerns the class-imbalanced regime, which arises systematically in continual learning. Here, the phenomenon of Minority Collapse (MC) (Fang et al.
(02T) has been observed, in which minority-clase features
are pushed toward the origin. Dang et al. (2023); |Hong& & Ling (2023) derived an exact law for this collapse, including a threshold on the number of samples below which features collapse to the origin and above which the NC configuration is gradually restored. Because class imbalance is inherent in innseeinrremetall continual learning. NC principles have also been leveraged to design better heads. Several works (Yang al. 2023 Dang et al. 2024: Wang et al., (025) impose fixed global ETF structure in the classifier head, rather than learning it, to mitigate catastrophic forgetting. To our knowledge, we are the first to use NC theory to analyze the asymptotic geometry of neural feature spaces in continual learning. In doing so, we introduce the multi-head setting, which is widely used in continual learning but has not been formally studied in the NC literature. While full theory of multi-head NC lies beyond the scope of this paper, our empirical evidence provides the first steps toward such framework.
Out-Of-Distribution Detection Qutttf--ditriiuutio (OOD) detection is a critical challenge for neural networks deployed in error-sensitive settings. Hendrycks & Gimpel 2018 first observed that networks consistently assign lower prediction confidences to OOD samples across a wide range of tasks. Subsequent work has shown that OOD samples occupy distinct regions of the represen-
26

---

Published as a conference paper at ICLR 2026
tation space, often collapsing toward the origin due to the in-distribution filtering effect induced by low-rank structures in the backbone (Kang et al. 2024 Haas et al. (2023) connected this phe- nomenon Neural Collapse (NC), demonstrating that L2 regularization accelerates the emergence of NC and sharpens OOD separation. Building on this, Ammar et al. (2024) proposed an additional Orthogonality- which postulates that in-distribution and out-of-disriibttion
NC proppetyy-1//O0 features become asymptotically orthogonal. They further introduced a detection score based on the norm of samples projected onto the simplex ETF subspace S, which closely parallels the analysis in our work. Our results extend this line of research by providing formal evidence for the ID/OOD orthogonality hypothesis, offering precise characterization of the roles of weight decay and fea- ture norm, and, to our knowledge, establishing the first explicit connection between catastrophic forgetting and OOD detection.
27

---

Published as a conference paper at ICLR 2026
C
MATHEMATICAL DERIVATIONS
Notation
Da
Dataset of task
D
Training dataset during session n -may include buffer
D
Datasets of all tasks combined
Xe
Instances of class in all tasks
[(0, D)
Average loss function over
A
Weight decay factor
7
SGD learning rage
fo
Network function R44 + IR..
6
Feature map Rdi - Rd
h
Network head Rd TR
Wa
Network head weights
W7
Network head weights for classes of task n (only multi- head)
11, E, 0²
The mean, covariance and variance of a distribution
3e = Ee [uc]
Centered class c mean
S span( ................................ pK)
Centered mean span
Ú [iI,..., [ūx]
Centered mean matrix
Pa
Projection onto the space A
B. ||((1)112
Training class (squared) norm
C.I SETUP
Consider a neural network with weights 0, divided into a non-linear map Ø Rdd + Rds and linear head h RR - IR The function takes the form:
fo(Ø) Wn. <(x) + bb.
We hereafter refer the map ((x) as features or representation of the input and to f(x) as output. The network trained to minimize classification loss E((x, y), fo) on a given dataset D. We denote by L(D,0 0) the average f((2,8) y). ffe over D, and where clear we leave D implicit. The loss assumed be convex the network output f.(.).
For each task n new dataset D2 is provided, with K classes. We denote by D, = Unst Dn the union of all datasets for tasks 1 to f and simply D the union of all datasets across all tasks. Moreover, we denote by D, the training data used during the session t- which may include buffer. For a given class e we denote by XD.c the available inputs from that class, i.c. XD.c {2 (z, y) D and y c). We use X_ X D.c. the set of inputs for class e across all learning sessions. We assume the number of classes K be predicted be the same for each task.
For given class data X. the class mean feature vector is:
(e(D) = Ex, [d(x)].
We call Pe (c(D) the population mean, to distinguish it from the buffer mean (B( If given class appears in multiple training session, we additionally distinguish between (D( and Pe(D,) where the latter is the observed mean. For set of classes {1...., K} in dataset D the global mean feature vector is:
(D) E2 Ex, [d(x)) E [o(x)),
28

---

Published as a conference paper at ICLR 2026
which we call population global mean distinguish it from the buffer global mean (B Finally, the centered class mean feature vector is:
Pc(DD = ( - (G(D)
and similarly (B) = (B) - (B( When clear, we may omit D and B from the notation.
C.2 LINEAR SEPARABILITY
In our study we are interested in quantifying the linear separability of the old tasks' classes feature space. In this section discuss the metric of linear separability used and derive lower bound for it.
Definition (Linear Separability), Consider the two distributions Pi and P2. The linear separability of the two classes is defined the maximum success rate achievable by any linear classifier:
E(Pi, Po) == max (w b + Pp, x+b ++b<00] w.
Equivalently, c(P,, = Cmin- where Cuin is the minimal misclassification probability over all linear classifiers.
Definition 3 (Mahalanobis Distance). Consider two distribution in the feature space 122, and covariances Σ1, E2. The Mahalanobis distance between the distributions defined as
dz (M1, M2, Et, M2) (#1 p2) T (Ei
222 -22)
For two Gaussian distributions with equal covariance the Mahalanobis distance determines the min- imal misclassification probability over all linear classifiers:
Cysin 4
(-2
/er dis
In this study take the Mahalanobis distance be a proxy for the linear separability of two distributions in feature space. When only the first two moments of the distributions are known, this the best proxy for linear separability. the following lemma we derive handy lower bound for the Mahalanobis distance which we will be using throughout.
Lemma 1 (Lower Bound to Mahalanobis Distance). Let Pl, u2 Rd and Σ,, En E Raxd be positive semidefinite covariance matrices. Then the squared Mahalanobis distance satisfies
11 u2ll Tr(E, + Σ2)
(11, 11, E2) = (m p2) (E1+ + -2 - p22)
Proof. Let A Σ Σg and == P1 u22 Let X be the eigenvalues of and u, the corresponding orthonormal eigenvectors. Write
Σ a.44 so that
AA
A
By Jensen's inequality for the convex function we have
f(x) 1/2, x + and the fact that E. =||0|12,
a ||0|1 Σ
Σ
A
Tr(A)
Applying this to v M1 42 and A = Σ1 + Ee gives the claimed inequality.
In this work we use the lower bound to the Mahalanobis distance as a proxy for linear separability. This quantity is also related the signal noise ratio, and thus hereafter the following
29

---

Published as a conference paper at ICLR 2026
notation:
|| p2l SNR(c1, 02) = Tr(E1 Σ2)
R 02) and E(C1, 02) are directly proportional, although the latter is bounded while the former not. Therefore an increase in SNN R((),02) e2) corresponds to increase in linear separability, within the applicability a Gaussian assumption.
C.3 TERMINAL PHASE OF TRAINING (TPT)
The terminal phase of training is the set of training steps including and succeeding the step where the training loss is zero. Given our network structure, a direct consequence of TPT is that the class- conditional distributions linearly separable in feature space.
Starting from Papyan et al. (2020), several works have studied the structures that emerge in the network in this last phase training (see Appendix for an overview). In particular, Papyan al. (2220) has discovered that TPT induces the phenomenon of Neural Collapse (NC) on the features of the training data. This phenomenon composed of four key distinct effects, which outline in the following definitions. Notably the definitions below apply exclusively to the training data, which we denote generically by D here. Thus, the class means and the global means in Definition are all computed using the training data (i.e. Me (D()), and ire = (D))..
Definition (NCI or Variability collapse). Let be the training step index and o, the feature map at step f trained on data D. Then, the within variation becomes negligible as the features collapse their class means. In other words, for every x E X,ee. with e the training data:
Exp,e [1(b((x) [=(())]3)== = 5z,
lim 6 = O +
(3)
Definition (NC2 Convergence to Simplex ETF). The vectors of the class means (after centering by their global mean) converge to having equal length, forming equal-sized angles between any given pair, and being the maximally pairwise-distaneed configuration constrained to the previous two properties.
lim [1Ae())12 + 3r Vc ++
(4)
lim (os( (1)) - if td
(5)
Definition 6 (NC3 Convergence Self-duality). The class means and linear classi- fiers althhughh mathematically quite different objects, living dual-vector spaces converge to each other, up to rescaling. Let Üt) = [t)(t)........ .000......A... AA..AAxx (f)]:
WT (t) Ú(t)
(6)
( 10(IIII
As consequence, rank(W,(C ran)((((t) = KK
Definition 7 (NC4 or Simplification to NCC). For given deepnet activation, the network classi- fier converges choosing whichever class has the nearest train class mean (in standard Fuclidean distance).
EST Notation. In all the following proofs we denote by S, = span(( (t)}) and by S its orthogonal components, and similarly by Ps,, Ps. the respective projection operators. Note that the reference to the training data is implicit. We might signal explicitly when necessary.
30

---

Taakk-nnremmntann Published as a conference paper at ICLR 2026
Lemma 2 (Feature classes gram matrix), Ler Ü. the training data). Then there exist to the TPT such that, for all >> ig the gram matrix U,7 Ü, has the following structure:
[tr(t)..... Pk (t)] (computed with respect to
0, = B ((xTTTTTT K 1111)
(U, (Ú) 3-1 (Ix
2K 11
(8)
Proof. Let (t(() = [e(t)) - Na (t) be the centered class mean given by Øi on the training data. Then by Definition 5 we know that for all t to for some tq-
Be, c= d' ---I. c#d,
(Le(t), (t))
Also, denote by U, (t) 7x (t) the matrix of centered class means. Gram matrix U, U. has the following structure:
Then the centered
Ú,ú - -3e (I--
which is a rank-one perturbation a diagonal matrix. In fact, the matrix a projection matrix onto the space orthogonal to 1, scaled by 3.. It has cigenvalues 3e with multiplicity K - and 0 with multiplicity 1. Since it's a projection matrix, it idempotent (up to the scaling factor 5²). Its inverse is well-defined.
does not exist but the pseudo-invers
Be (1 [K =17
1 1 - Br
C.4 NEURAL COLLAPSE CONTINUAL. LEARNING SETUP
Depending on the continual learning setup, the number of outputs in the network may be increasing with each task. Therefore the Neural Collapse definitions need to carefully revisited for different continual learning scenarios.
W
W.
W
W,
W.
W
W.
Domain-Incremente (single head)
Class-Incremental (single-head)
(multi-head)
Depiction of continual learning Setups and corresponding head structures. Different colors indicate different gradient information propagated through the weights.
Figure 25:
In the case of task-incremental and class-incremental learning, where each task introduces new classes, we distinguish between the tasks heads as follows:
f (x) w/ 0(z) 6
(9)
fe(x) ()).,... (x))
(10)
where only heads from the first to the current task are used in the computation of the network function. For brevity, hereafter we will denote by W.r. by the concatenation of active heads at any task: for example, for task n, W.A. [Wx ...... w and fo(x) WAA((a)) + by In order to unify the notation, the same symbols will be used for Nomain-incremeen
learning where W/A W₁, and 6% = bn The difference between task- and dlass-incrementa learning is whether
the residuals depend only on the cureent task output p(n) (x) or on the entire output fo (x). as we will explain shortly.
31

---

Published as a conference paper at ICLR 2026
C.4.1 DOMAIN-INCREMENT LEARNING (DIL)
In
Learning (DIL), the classification head is consistently shared across all tasks, as each task utilizes the same set of classes, Consequently, NC is expected to induce a fixed number of clusters in the feature space, corresponding to the total number of classes. Given that the same class appears in multiple tasks, we must distinguish between the population mean Pe (D) and the observed mean (c(D), where D is generic training set. Generally, we expect NC properties (Definitions
to emerge on the training data D. If the training data includes buffer, all class means and the global mean will be computed including the buffer samples. Accounting for this, the NC characteristics emerge analogously to those in single-task training.
Emergent Task Wise Simplex Structure Curiously, our experiments also observe an emergent within-task simplex structure. When features are centered by the task-wise feature means (taking, for each task, all samples included in the buffer), we also observe the characteristic NC structure within the task. This finding is non-trivial, because the task-wise mean and the global mean not the same. It seems, then, that during continual learning in DIL, Neural Collapse emerges on two distinet levels simultaneously. This dual emergence creates highly constrained feature mani- fold, which substantially limits the degrees of freedom available for learning subsequent tasks. Our observations suggest that a significantly more constrained version of NC emerges under the DIL. paradigm compared to standard single-task training.
C.4.2 LASS-INCRRMENTAL LEARNING (CIL)
In Gaass-nncremenal Learning (CIL), each task introduces a new set of classes (for simplicity, we assume the same number K per task). For task n, the classification head is expanded by adding
wi(n) to form
(t)(t) (t) (t). w (t)].
Nevertheless, training proceeds as in a hingle-task setting: residuals are shared across all outputs,
a0((z, S6)
fo(x) ú.
Ofe
where both fe(x) and ÿ are vectors of dimension x K. For instance, fo(x) fa (z) for MSE loss, and fz( softmax(fe(x)) for cross-entropy loss, while ÿ corresponds to the one-hot encoding of y.
In CII., the composition and relative proportion of classes in the training data affect the asymptot- ically optimal feature structure. If all classes are present in equal proportion, the Neural Collapse (NC) structure for task consists of x K clusters with vanishing intra-cluster variance, which increases to (n + 1) x K clusters when the next task is introduced. By Definition the resulting rank of the weight matrix is n x K 1 after n tasks.
However, if the training dataset is imbalanced-i.e, the number of samples per class is not equal-the network is pushed, during the TPT, toward a variant of NC known as Minority Collapse (MC) (Fang et al.) (02T). For this reason, in our experiments we use datasets with equal numbers of samples per class and buffers of uniform size across tasks. Assuming all tasks' datasets have the same size, for dataset D and buffer B, the degree of imbalance can be quantified by
B
P
D
identify a critical threshold for pt below this value, the heads of minority classes (i.e., buffer classes) become indistinguishable, producing nearly identical outputs for different classes. Above the threshold, the MC structure is gradually restored standard NC configuration, with class mean norms and angles increasing smoothly.
Dang al. (2023) : Hong & Ling 2023)
As noted by Fang et al. (202I), MC can be avoided by over-sampling from minority classes to restore class balance. In continual learning, this is implemented by sampling in a task-balanced fashion from the buffer, ensuring that each batch contains an equal number of samples per class. Under task- balanced sampling, the inasemnnremettall setup reproduces the standard NC characteristics observed in single-tar task training. In contrast, in the absence of replay, class- incremental learning is inherently prone to Minority Collapse
32

---

Published as a conference paper at ICLR 2026
C.4.3 MASK-INCREMENTAL LEARNING (TIL)
In Insk-Incrrmental Learning (TIL), each task introduces K new classes, as in the CIL case. The crucial difference lies in the treatment of the residuals: they are computed separately for each task. For sample x belonging task n, we have
(l((z. f0) f/ (x) g(m),
0f
where both (x) and g(m) are K-dimensional vectors. For instance, under MSE loss f(r) (x) = 1 (z), while under cross-entropy =/ (x) softmax(So (z)), and g(m) denotes the one-hot encoding of E (0, (0....,K-11,
partitioned across tasks, logits corresponding to inactive heads do not tribute to the loss. That is, for z E D,, the terms W 6(z) + b with j remain unconstrained. In contrast, in CIL such logits explicitly penalized, the residuals are shared across all heads. Consequently, the TIL multi-head setting imposes fewer explicit constraints on the relative geometry of weights and class means across tasks.
Since the outputs
Our empirical results indeed reveal that there is structure within each task, but the relative geometry across tasks is more variable and does seem exhibit clear pattern. Within each task, the features exhibit the standard Neural Collapse (NC) geometry, consistent with Definitions 4 to 6 However, the class means of different tasks can overlap arbitrarily, as there are no explicit constraints linking them.
Motivated by these observations, formalize the emergent structure as follows.
Proposition (Neural Collapse in Multi-Head Models). Let μ.2 denote the mean feature of class frorm task n. at time l. In the terminal phase of training, under balanced sampling, the following hold:
1. NCI (Variability collapse). Within each task, features collapse to their class means, i.e.,
lim Ezexa [1)))(z) - P.: 1. (t)11)]] =.
++000
2. NC2 (Convergence to simplex ETF within each task). Centered class means within each task converge to an Equiangular Tight Frame (ETF):
lim [1 Bh
VCE {1,..., K)
(11)
+
{1
[1
ifc=
lim (t), (t)) ++0
(12)
ifc
&d,
K-I
where (t) (t) and p& (t) is the task mean.
3. NC3 (Convergence to self-duality). The classifier weights for each head align with the centered class means of the corresponding task, up to rescaling:
(T) (1)
((()(()
(m) (1)l [000
where (t)) (t)......
... Pr. (t)] Consequently, rank(W (t)) rank(ū((m)t))
In summary, each task forms an ETF simplex in the feature space (NC2), with variability collapse (NC1) and classifier self-duality (NC3) holding in the single-task case.
A key implication is the difference in rank scaling compared to CIL. In CIL, the rank of the head weights after n tasks is n x K - 1, whereas in TIL is upper bounded by n x (K (K1), as confirmed empirically (Figure 16). Thus, the multi-head structure imposes a strictly stronger rank limitation.
33

---

Published as a conference paper at ICLR 2026
Replay replay. When training without replay, i.e., relying solely on the current task's data, the TIL setup reduces to an effective single-task regime: earlier heads receive no gradient signal, and NC emerges only within the most recent task, in standard single-task training.
Lemma (Gram Matrix in TIL). Let Ü. = 10, (n be the matrix of centered class means (1) (i())()
time I, where (mm) [ (1) (I),.... "K
(m)) (1), with fic
(m)) (1) PG
(m)) (t). Suppose that in
pee
the terminal phase of training Proposition Tholds. Then for all sufficiently large I, the Gram matrix
U, Ug is:
0 0
T ((1)7
GP
T o(2)
0,Ü,-
T( o 0.00
where each block (m)) satisfies
Glm)
Bm
[K - 11)).
a(m)
= =33²l [kk 1 111)).
Thus, the inverse Gram matrix satisfies (U, U;:
Proof. By Proposition 1 in the terminal phase of training each task satisfies NC2 (within-task ETF) and task subspaces are orthogonal.
Diagonal blocks: Each G. (m)) ( ((())) ETF, its columns sum zero:
is an ETF matrix of size K x K. By definition of
G(m)) O. Off-diagonal blocks: For B(ú)- 0(00 (1) ((()) we have
B = 0 ((6) 1 = (1)) -0
since the columns of 0+60 are centered.
Global null vector: For the full block Gram matrix Ü, Ür. the i-th block-row acting on 1, is
1= 1 1 0+
Σ 50=0.
1=1
574
j&t
Hence, U7 U,Ü,,
= so 1, lies in the null space of the Gram matrix.
Inverse / pseudoinverse: Since U,T Ü, is singular, the Moore-Penrose pseudoinverse exists, and ker(U, Ú.) implies 1 ker((U, U.)+). Thus, is eigenvector of both Ü, Ü, and its pseudoinverse.
C.4.4 FINAL RESULTS AND TAKEAWAYS
The preceding analysis allows us to draw several unifying conclusions regarding the asymptotic feature geometry in continual learning.
first key takeaway that, the absence replay, continual learning effectively reduces re- peated single-task training. In this regime, only the current task is represented in feature space with Neural Collapse (NC) geometry, while features from previous tasks degenerate. This observation is formalized follows.
Finding (Asymptotic Structure without replay). When training exclusively on the most recent task, irrespective the continual learning setup, the asymptotically optimal feature representation for the current task coincides with the Neural Collapse (NC) structure observed in the single-task regime. In the CIL case, this further implies that the feature representations of all classes from previous tasks collapse to the zero vector, while only the features of the current task organize according to NC.
34

---

Published as a conference paper at ICLR 2026
A second key takeaway is that this setting, the replay buffer restores balanced exposure to all classes, preventing the degeneration of past representations. Consequently, in single-head setups (DIL and CIL) the network converges to global NC structure over all observed classes (measured on the training data). In contrast, the multi-head setup of TII continues decouple the heads across tasks, yielding NC geometry within each task but leaving the relative geometry across tasks unconstrained.
replay fundamentally alters the asymptotic structure. In
Finding 2 (Asymptotic Structure of the Feature Space with task- lak-balancedd replay). When training on tasks with task-balanced replay, the single-head setups converge to Neural Collapse over all classes represented in the training data (K classes for DIL and n classes for CIL). For TIL, each task head individually exhibits Neural Collapse within its K classes, but the relative positioning of class means across tasks is unconstrained, leading to a blockwise NC structure in feature space.
Taken together, these results highlight fundamental distinction between single-head and multi- head continual learning: while replay suffices to recover global NC geometry in single-head settings, in TIL the absence of cross-task coupling in the loss function enforces only local NC structure within each task.
C.5 MAIN RESULT 1: STABILIZATION OF THE TRAINING FEATURE SUBSPACE.
Theorem 4 (Subspace stabilization in TPT under SGD.). Let (x) = (A)()) Φe(x) + by be the network at step t in the optimization of task with P classes and dataset D, and let St = span( (aan(((11 (t), (p(()) (Re = pc(D)). Assume NC3 holds on D for all 2 to, i.e. (p)n( = S.. Then, for all z to, the gradient VoL(0,) is confined directions parame- ter space that affect features in S,, and, consequently, S. Sta and S St.:
Proof. Let (x) be the feature representation of x at time f, and let [x) = Vo)((z) be its Jacobian with respect to parameters 0,. Consider an infinitesimal parameter change 20, ec, with Ja))v( = 0 for all x in the training data, i.e., this change only affects the feature component in ST. By a first order approximation the corresponding feature change is:
( J=()) Je(()) e =es J(Zzo
Now, consider the effect of this change on the loss:
[(O, + cv) [(O,) ~ V VoL(0.) (d)(x)
(13)
@C 0f
(x)(x)
(14)
0f 06)
QC wA(l)) Ad(())
(15)
0f
By NC3, for any > lo. span (A))() S., and since Adr(x) St:
()(() Ad. (x) = 0 = C(O, + + - L(0.) =
Dividing by and taking the limit 0,
VoL(O,) Le v for all such that 13
This shows that the loss gradient lies entirely in directions that affect S, and consequently the St component of the input representation not changed. It follows that, after NC3 gradient descent cannot change the subspaces S,, S,,, since all changes in the features for t > to will lie in St,, We conclude that S. = St, and S St
Ear Notation. Hereafter we denote by S the subspace spanned by the centered class means after its stabilization at the onset of NC3, i.c. S = St.. Note that the centered class means may still change, but their span doesn't.
35

---

Published as a conference paper at ICLR 2026
Lemma 4 (Freezing and decay of S-1 in TPT under SGD.). Let fe) (t) (t) be the network at time t, where (x) is the feature representation and W (t) the final layer weights. Suppose the training loss includes weight decay with coefficient X > 0, i.e.
Linal [()) + 11012
and that for all 2 to. NC3 holds, ie.. span (W = S. and n sufficiently small. Then the component of (x()) in St, denoted by Øt,s1 (x), evolves as follows:
01, (x) v dio.s! s (I)
Proof. By gradient descent the parameter update is:
A = - (Vo((()) + A0,)
and, for small enough 17 we can approximate the feature update as
0++1(x) - (x( x (x(r) A0, = -1 VaC(O.) - yu (e))) 02
Decompose this into components in S and S By Theorem for all t > to and all x E D [((2) VnL(0) S. Then:
0141,54 (x:) - 2e.s (x) = m/ Jx))),
Noticing that = makes (xx) = for any x, by a first order approximation we have that (x Rd J+(x) 0₂ and thus:
0141.-- (I) 0r.s- nA))
for all to. Unrolling this sequence over time, starting from to. we get our result.
Remark. The results presented in this section hold for both single-head and multi-head training. When training with more than head, the subspace corresponds to the span of the class means of all heads combined, and by Proposition I it has lower rank than in the single-head case.
C.6 ANOTHER DEFINITION OF OOD
Definition 8 (ID/OOD orthogonality property of Ammar et al. (2024)). ) Consider model with
feature map (x), trained on dataset D with K classes. Denote by S, = span faa (t), (p).... .A....Ax (t)} the subspace spanned by the centered class means of the training data time t. The set of data X is said to be OOD
cos (Ex (Bx(d
ACE [K]
Definition 9 -Outrof-distributtin (OOD)). Let X, be a set of samples from class c. Consider network with feature map, Φe(x), trained on dataset D with K classes, such that X. D 0. Denote by S, = (f), Au (f)} the subspace spanned by the centered class means of the training data at time t. We say that X. of distribution for fa, (trained on D) if
Ps, Fx. [Ø(x)) =
This definition restates the ID/OOD orthogonality property of Ammar et al. 2024) in different form.
Next, we show that the observation, common in the OOD detection literature, that old tasks data is maximally uncertain in the network output coherent with these definitions of OOD when there is Neural Collapse.
Proposition 2 (Out Of Distribution (OOD) data is maximally uncertain.). A set of samples X from the same class is out of distribution for the model fo with homogeneous head and Neural Collapse if and only if the average model output over X is maximally uncertain, i.e. the uniform distribution.
36

---

Published as a conference paper at ICLR 2026
Proof. By definition of S being the span of (x.(t)....,,xxx)))
write
(t) (x) =
(<(x))
where Ü the matrix whose columns are the centered class means fr. By Definition 6 we have, for all > to. (t) = a U. where a = [[W,tt))l and therefore, for an homogenous head model, the
(1)l
network outputs are fa(=) (t) (z)) Finally, to complete the proof that by the structure of the gram matrix (Lemma its null space is one-dimensional along the 1 direction. Therefore must be that
Ú(t) Ex. [x]xx()) (x())xx,l(z)) =0
(16)
(17)
are always true concurrently.
Remark (Old task data behaves as OOD without replay). When training on task n without replay, samples from previous tasks m effectively behave out-of-distribution for the active subspace corresponding to task n, in the sense of Definition g For single-head models, a similar effect occurs in CIL due to Minority Collapse, which guarantees that the representations (x(x) old task data simply converges to the origin, which is trivially orthogonal to St. Consequently, the theoretical re- sults we derive for OOD data this section also apply to old task data under training without replay.
Corollary (The OOD class mean vector converges in TPT under SGD with weight decay.). In the TPT, with weight decay coefficient À > 0, OOD class inputs Xe are all mapped to the origin asymptotically
lim Ex. [0e(=)] = 10000
C.7 ASYMPTOTICS OF OOD DATA
Notation To simplify exposition, we introduce the notation e = 1 n,a Additionally, this section we use W,, and Ü to refer in general to the head and class means used in the current training. Note that, since we don't consider replay for now, this is equivalent the current task's classes' head and features.
Theorem (OOD class variance after NC3.). Let be(x) be the coefficients of the projection of the input the centered training class means space S. In the terminal phase training, for OOD inputs, if be(x), E X has covariance Ee with constant norm in t, then the within-class variance in feature space for X. satisfies
Varx, (o)(z)) e + (1 (11
(18)
where 3r accounts for the contribution of all active heads: the single-head case 3 Br, and in the multi-head case B.a. Em= 13th with n the number of active heads.
Proof. Consider representations of inputs from OOD class X.. By Theorem 4 for any to. we can decompose
<x(x) = (x) + $t (x),
where (1,)(x) lies in the span of the centered training class means. We can express this component
(x,)(x) = (x),, b((z) (()) (z).
From Definition g Ex. (or 0. Hence, the within-class variance in feature space is
Varx. (0((z)) = Ex. (x) [x
(19)
Ex. (1((()) by (x) Varx. (x)(
(20)
37

---

Published as a conference paper at ICLR 2026
The orthogonal component S shrinks remains constant due to Lemma 4
Varx, S (de(x)) m)))(((f() Varx,, (0to (x)).
The variance in the S component depends the covariance E, of by (x)), which is assumed constant
in t
Thus,
Varx (() (00
where -= (t) has the structure described Definition 5 and Proposition
Single-head case.
For P classes, an ETF matrix with P vertices
B,
Ask Bt, Ash
j k.
P-1
so that
P P
P
Bt
(r(())) X1 S (r(AEe)
S 3r
P-1 tr(Ee).
Clow
Chigh
Multi-head case. For n heads, A has the block structure described in Lemma block having K cigenvalues equal to 8mh and one zero eigenvalue. Hence,
3 with each diagonal
K
Σ m=1
(t(()(m)) (1(2(())) & tr(AAe) <
m
tr (m())
K
K
Clow
m=1
S Chigh
Denoting by Br 1
Σn Pt get:
P
P
Clow S (tr(AE.) S 3 K-1 Chigh.
Thus, recognising that the only dynamic variable in t is 3,a for both cases, obtain
sarx.,,(o((z)) E O(3A), Varx...s (dr(z)) E @((1 77) (t)))))
completing the proof.
Notation When we are not considering replay, there is only one active head in multi-headed models. In this cases we use Br to denote the feature norm of the active head. The results of this section are presented a more general way, using 3, to denote the contribution of all active heads.
Theorem 6 (Linear separability of OOD data with Neural Collapse.). Consider two OOD classes with inputs X Xez. During TPT of the model fo, (x) trained on a dataset D, the SNR between the two classes has asymptotic behaviour:
SN 62) EO (((
3
where BA, is the class feature norm, averaged across the active heads.
Proof. Let Px,, (d)())), Px., (z)( be the distributions of the two OOD classes in feature space. Let Pl, A2 and Σg. Σg be the respective mean and covariances in feature space. By Definition 9 we know that μ4 = Ex,, [or (x)) = 2). Therefore the SNR lower bound is:
RN 02) ||Ex,, [ (x)) Ex., (0,.s (x))]l²
Tr(E, + M2)
38

---

Published as a conference paper at ICLR 2026
where |[Ex,, (0r,s- (x)] Ex., (or, s4 (=]]]l E @ ((1 2)))) Notice that the trace decom-
poses across subspaces as well and therefore:
Tr(21 + 22) Tr(E1, + 1 + 22.5 + E2,s+)
In the proof of Theorem 5 we have that Tr(E,,)) ((3)) and (h E ((1 (1-11)10 -7)))ttm)))
Thus from simple asympiotic analysis we get that the linear separability of OOD data grows as:
S2 E (a1 8 A ((--AApr-o)) +1
Remark. By Theorem 6 when learning a new task without replay, if class from previous task becomes out-of-distribution (OOD) with respect the current network (and its active subspace), an increasing class means norm 3r or weight decay leads to deep forgetting, with the class information degrade over time.
Remark. The SNR also depends on the degree of linear separability of the classes in the orthogonal subspace S at the onset NC. Consequently, in the absence of weight decay or without growth of the feature norms, the old classes may retain nonzero level of linear separability asymptotically,
C.8 MAIN RESULTS 3: FEATURE SPACE ASYMPTOTIC STRUCTURE WITH REPLAY.
We now turn our attention to training with replay, to explain how replay mitigates deep forgetting.
Notation. We denote by D, the datasets of task and by B; the buffer used when training on task i. Further, let p; /|D]| be the percentage of the dataset used for replay and assume that there is balanced sampling, iec each task is equally represented in each training batch. We again the training data in TPT, which in this case the
look at the case where there is Neural Collapse current task data D, and the buffers Bi...., Bn-1. Finally, for DIL we denote by X the data of class in task and by X. the data of class in all tasks, i.e. X U X%
Modeling the distribution of data from old tasks with replay μ (B)., the mean computed on the buffer samples. Define
Hereafter, we denote by j =
(t) = (t) + Et())
where (t) is the difference between the population mean and the observed mean. For CIL and TIL this is the buffer B., while for DIL this is the union of all buffers B U B, and the current task class data X We know (t) [t]]] decreases with the buffer size and, in particular, it's zero when B.- X
Let DN be the distribution of the representations when training on 100% of the training data X We know that this distribution has NC, each class e has mean flc and decaying variance d,. Also let DOOD denote the OOD data distribution which we observe in the absence of replay (mean in S and larger variance governed by 3, and the decay factor -==fo). Based on these observations, we model the distribution of (1(x)) as the mixture of its two limiting distributions with mixing weight (b) [0, 1] which a monotonic function of b:
(x(x) Te DNC (1-xe) DOOD no)
According to this model, the mean and variance for the distribution of class e asymptotically are:
(t) (Mo(() + (c,s(1) + (1 To) (u-- Me,s (lo))
(21)
o² θ (.6 + (1 (1= (s
+ 20((-
(22)
Note that in Equation 21 S is defined based ne. and absorbed the S- component of (l) in Pe,sa (40). In the variance expression we used the results of Theorem 5 for the OOD component and the fact that the variance of Dv is ò,. In the TIL case, 3r is the average of the class feature means across all active heads.
39

---

Published as a conference paper at ICLR 2026
Remark (Interpretation of the buffer-OOD mixture model). The proposed model interpolates be- tween two limiting regimes smoothly, and is based on our hypothesis regarding the evolution of the feature representation of past tasks the buffer size is gradually increased. For small buffer size b, the representation distribution is dominated by the OOD component DOOD, which contributes variance in the orthogonal subspace 5 and acts structured noise with respect to the span S of the current task. As b increases, the mixture weight grows monotonically, and the replayed samples increasingly constrain the class means inside S. In the limit 6 = [Xe] Te and the representation collapses to the Neural Collapse distribution DNc with vanishing variance. For intermediate b, the replay buffer introduces signal in S through the term (t), while the residual OOD component adds noise. The evolution of T. therefore captures how replay gradually aligns the buffer distribution with the NC structure, while modulating the relative strength of signal (from in-span replay) versus noise (from OOD drift).
Proposition 3 (Concentration of buffer estimates). Let De be the feature distribution of class at time t, with mean (t) ) in the active subspace S and covariance De Let Be C. DD denote a replay buffer of size b obtained by i..dd sampling. Then the buffer statisties jee and Ee satisfy
Tr(22)
Tr(2)) b
E [lllA. -o.1 =
B[IE - Eolli]] O
In particular, the standard deviation of both estimators decays as 0(b-1/2)
Proof. Let (1:)/=1 ~ D. be i..dd samples with mean μ = Mr. and covariance = E.. The sample ( p4), so by independence ( (Verrshynn 2018)),
mean satisfies fie
[l - [=]l]] = 1
[1((z;) - m|] Tr(2) = 0 Tr(E))
(TTEE)
Similarly, for the buffer covariance Eg we have
Tr(2)) E B[I - E|E = 0 b
Thus the standard deviations of both estimators decay as O(b 1/2).
In the above result we have hidden many other constants as they are independent of training time.
Remark. This bound should be interpreted as a heuristic scaling law rather than a formal guarantee. The key caveat that feature evolution (x)) is coupled to the buffer B, through training, violating independence. Nevertheless, the i.i.d. assumption is reasonable if buffer-induced correlations are small relative to the intrinsic variance of the features. In this sense, the bound captures the typical order of fluctuations in (t), even if the exact constants may differ in practice.
Theorem 7 (Linear separability of replay data under Neural Collapse). Let C₁, C2 be two replay- buffer classes decoded by the same head, and let jaa(t) denote their observed class means with deviation (t,)(t), from the population mean inside the NC subspace S. Assume that the old classes features follow the mixture model
Φe(x)) n; DNC + (1-71) TT.) DOOD:
with mixing proportion Ti, and that the class means norms for each lask m follow the same growth pattern 3Thh E (B.)). Then the signal-to-non ratio between ei and 02 satisfies
7² 3², + (0)
SN R(eRe,,oo (2) E e
2 de + (3,1 ,211-110))
1 = (m))
[1(x)++22)
Proof. Let (t), Et) be the mean and covariance of class i in feature space. If there is replay we assume they follow the mixed distribution described above with mixing proportion æ1, m2 respec-
40

---

Published as a conference paper at ICLR 2026
tively. Therefore, for each of them know the following:
(t(t) ni ((:(() + (.,((t) + (1 mi) (u'-to pi,s (to))
(23)
(t n. -N (t) + mz)² x (t)
(24)
Moreover, by Theorem we know that
tr (EOO E + (8, + ()((
and by Definition 4 we also know that (ENC ó, - with +00. Using this, we can write the SNR lower bound:
SNR(c. 02) - + |1 (t) (2.5 (t)l?
Tr(2, E2(())
where by definition of (t):::
(()) (2,5(() = || (1) -2 + - - 2
|| (t) |22,-- (+)ll = (m1 (2) + t [[(1 - T1) -1,5= (to) - - (1- (12) (2,s=(tt)ll)
Using the linearity of the trace and the fact that it decomposes across subspaces:
(r)),(t) .. Tr(E, (1))+(11))) Tr(E (t)) E 12 + (1 - -T,²² (g5 + (2(e-10))
The mean difference in the S component expands into
||, sll + [ 22 2(T T2)2 T1E1. T2E2,s)
and the first term
ر| (B) T2j (B) = m (B)]1 + ( 2 (B), (2
By Definition 5 and Proposition [ and by the fact that C2 belong to the same head m, also in multi-headed models, we know that [|² 1 ~ 3g and (Mez Mez) = K-I Then:
8
(B) T2 (मी + m 2 मं R- ①((m 32))
(25)
Define the per-class ratios
|ks.
1 [
71
[
1)2
Notice that the deviations in S must behave in norm the variance in the S component, which by Proposition 3) (t)) E @ (3:1). Thus the coefficients satisfy 71, = @(1). By the Cauchy-
Schwarz inequality, have
( T2ūa(B), TIEI, - I || (B) T2 (B) Ei 2262,
Bound the second factor:
|| || - 2252ll S || + |2 || = 7 [TTAI + m
Therefore, the magnitude of the cross-term bounded by
2 Putting everything together, we obtain
TIč1,s-T263
(B) (aa
[2(T (B) T2pa BB), TILL, T252,5)1 0((m + m
Since the cross-product is signed, it could contribute negatively to the mean difference. However, by the same argument cannot exceed the leading term in magnitude. Unless the two terms perfectly cancel each other, the scaling with è is dominated by the positive norms:
(())((())))))))))))))))))) #2)s(1))² O((TI + m2)²² B(),
41

---

Published as a conference paper at ICLR 2026
Putting everything together obtain the asymptotic behaviour of the SNR lower bound:
(Ti + T2) 36 (1 - (T1 + T2)) 22((-00)
SN E
(T1 + T2) 8z (1 (T1 + T2))² (36 + 22(1-t0))
To write it more clearly, define by p² (m++t2)²
(1-(
r² 36 + 2 ((
SN E +
y2 8, + (3, + 2 to))
For + O we recover the asymptotic behaviour of OOD data. For O, the SNR is guaranteed not vanish in the TPT.
Corollary 4 (Asymptotic SNR with replay). Under the conditions of Theorem 7 let r? (x) (1- +m1+ denote the buffer-weighted ratio of signal to residual OOD contribution. Then:
the limit > (corresponding to no replay), the SNR asymptotically reduces to the OOD case, and old-task features
features remain vulnerable drift in S+.
For any 0 (non-zero buffer fraction), the SNR is guaranteed not to vanish in the TPT as long as task-balanced replay is used. In particular, with increasing Br or weight decay, the limiting SNR satisfies
lim to
SNR(s 02) E @
ensuring that replay effectively preserves linear separability between old-task classes in the NC subspace.
42


Reviews:
[{"review_id": "fHPE64IK8v", "reviewer": "Reviewer_5K6E", "summary": "The paper analyzes how the replay buffer in continual learning scenarios influences model forgetting, distinguishing between shallow and deep forgetting. It further investigates these phenomena within the Neural Collapse framework, examining the geometric structure of the feature space and supporting the analysis with empirical results.", "questions": "- Could the authors clarify the rationale behind choosing ResNet and ViT as reference architectures, and discuss how this choice might influence the observed behaviors and results?\n\n- How were pretrained models adapted to the continual learning setting, and what motivated the comparison between pretrained and from-scratch training approaches?\n\n- Beyond weight decay, have the authors examined the influence of other hyperparameters or architectural choices on the empirical results, particularly in relation to deep forgetting?\n\n- What motivated the adoption of the Neural Collapse framework, and could the authors discuss potential alternative frameworks or justify why NC is particularly suitable for this analysis?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and clearly organized, with a pleasant and coherent flow of discussion. The topic addressed is novel and engaging. Moreover, the theoretical analysis is insightful and clearly explained.", "weaknesses": "While the theoretical discussion is sound and convincing, I have some concerns regarding the empirical analysis. First, it is unclear why the authors chose ResNet and ViT as reference models. It seems that the selected architectures could significantly influence the observed behaviors and results. If this is the case, the authors should explicitly discuss this aspect. Otherwise, a justification of why the chosen architectures do not affect the outcomes should be provided.  Along the same lines, the rationale behind considering both pretrained and from-scratch models is not entirely clear. In the case of pretrained models, it would be important to explain how the initialization was adapted to the continual learning setting, as mentioned in Section 1.1. Additionally, the discussion in Section 3.3.2 highlights the effect of weight decay, but the influence of other hyperparameters and architectural choices remains unexplored. Given their potential impact, especially in the context of deep forgetting, this omission seems non-negligible. The authors should include a discussion addressing this point to provide a more comprehensive understanding of the empirical results. Another aspect that would benefit from clarification is the adoption of the Neural Collapse (NC) framework. The authors should briefly discuss possible alternative frameworks and justify the choice of NC in this context.\nFinally, the discussion on the distinction between multi-head and single-head settings could be improved by adding a short introductory explanation earlier in the paper to help readers unfamiliar with these concepts. In addition, Section 3 contains some citation issues, where “?” symbols appear instead of proper references, and these should be corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fy5cZsyJo4", "reviewer": "Reviewer_wGRs", "summary": "This paper presents an asymptotic analysis of replay-based continual learning under the neural collapse phenomenon. The authors study how replay buffers affect shallow and deep forgetting. Empirically, they find that replay-based continual learning effectively mitigates deep forgetting but still suffers from shallow forgetting even when the replay buffers are large. They then use Neural Collapse theory to analyze the limiting geometry of features and heads in three continual learning setups. They also identify a connection between continual learning and OOD detection, showing that under weight decay, the distribution of OOD inputs converges to a degenerate null distribution. They also show the effect of replay in their framework, demonstrating that deep forgetting is not mitigated by the model with small replay buffer because of the approximation error when using the buffer distribution to approximate the true class distribution.", "questions": "1. How to understand the theoretical results when $\\eta \\lambda \\geq 2$?\n2. Why are experiments conducted only for buffer sizes varying from $0\\\\%$ to $10\\\\%$?\n3. What is the precise meaning of \"balanced replay\"?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work proposes a novel framework for replay-based continual learning. The results and implications are meaningful and helpful to the community. \n2. The authors provide a sufficient and comprehensive theoretical analysis for replay-based continual learning, considering three different setups and showing the effect of replay.\n3. The empirical study is consistent with the theoretical findings, across both real-world and simulated datasets.\n4. The work is well structured and easy to follow for the readers.", "weaknesses": "1. In Theorems 1, 2, and 3, it seems that $\\nu = 1 - \\eta \\lambda$ is required to be non-negative or greater than $-1$, but I do not find any explicit condition on $\\nu$.\n\n2. The explanation of why replay cannot strongly mitigate shallow forgetting is not convincing to me. The authors argue that the approximation error is the key reason, but there is no formal result to support this claim, which limits the contribution of this paper.\n \n3. In the experimental results, the authors only vary the buffer size from $0\\\\%$ to $10\\\\%$. This seems insufficient to support their theoretical findings.\n\n4. There are some typos and inconsistencies:\n   1. Lines 139, 141. Two citations are missing.\n   2. Line 315, \"Theorem 6\" should be \"Theorem 2.\"\n   3. Line 987, \"class-il\", \"domain-il\", and \"task-il\" should be written as \"CIL,\" \"DIL,\" and \"TIL,\" consistent with other figures. Moreover, the task indices in Figure 12 should be positive integers.\n\n5. The term “balanced replay” is unclear. I think “balanced replay” refers to the replay buffer being sampled in a balanced manner from the training set, rather than the buffer size being equal to the size of the training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UuQqJb7DAj", "reviewer": "Reviewer_hg1i", "summary": "The author developed an asymptotic framework to analyze feature geometry with and without replay buffers. They demonstrate that replay can reliably alleviate deep forgetting (loss of feature separability), but does not alleviate shallow forgetting (misalignment between classifier weights and features). This work extends NC theory to multi head CL settings, characterizes the effects of buffer size and weight decay, and establishes a theoretical connection between CL and OOD. The empirical results of CIFAR100, Tiny ImageNet, and CUB-200 validate the theoretical findings.", "questions": "Why do we focus on discussing Multi Head Models? Is this model commonly used in modern continuous learning and multi task learning? Do you analyze whether the purpose of this model is to increase workload or has practical significance.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Extends Neural Collapse analysis to continual learning and multi-head architectures—an unexplored direction. Uses asymptotic analysis and connects NC with OOD theory in a rigorous manner. Establishes a bridge between NC, CL, and OOD detection, enriching all three research domains.", "weaknesses": "While conceptually strong, it provides limited actionable guidance for improving CL performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)