Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning
Authors: Edoardo Urettini
Antonio Carta
Venue: ICML 2025 poster (2025)
Keywords: Online Continual Learning
Second-order optimization
Deep Learning
Primary area: deep_learning

Abstract: Online Continual Learning (OCL) models continuously adapt to nonstationary data streams, usually without task information. These settings are complex and many traditional CL methods fail, while online methods (mainly replay-based) suffer from instabilities after the task shift. To address this issue, we formalize replay-based OCL as a second-order online joint optimization with explicit KL-divergence constraints on replay data. We propose Online Curvature-Aware Replay (OCAR) to solve the problem: a method that leverages second-order information of the loss using a K-FAC approximation of the Fisher Information Matrix (FIM) to precondition the gradient. The FIM acts as a stabilizer to prevent forgetting while also accelerating the optimization in non-interfering directions. We show how to adapt the estimation of the FIM to a continual setting, stabilizing second-order optimization for non-iid data, uncovering the role of the Tikhonov damping in the stability-plasticity tradeoff.
  Empirical results show that OCAR outperforms state-of-the-art methods in continual metrics, achieving higher average accuracy throughout the training process in three different benchmarks.

Online Curvature-Aware Replay:
Leveraging 2nd Order Information for Online Continual Learning
Edoardo Urettini 2
Antonio Carta
Abstract
Learning (CL) approaches cannot be applied (Aljundi et al., 2019b; Mai al., 2022). Many specific OCL methods have been developed, nearly all replay-based (Yoo et al., 2024), but several shortcomings still exist:
Online Continual Learning (OCL) models con- tinuously adapt to nonstationary data streams, usually without task information. These settings are complex and many traditional CL methods fail, while online methods (mainly replay-based) suffer from instabilities after the task shift. To address this issue, we formalize replay-based OCL as second-order online joint optimization with explicit Kl--dieegeence constraints on replay data. We propose Online Curvature- -wwrre Re- play (OCAR) to solve the problem: a method that leverages second-order information of the loss using K-FAC approximation of the Fisher Infor- mation Matrix (FIM) to precondition the gradient. The FIM acts as stabilizer to prevent forget- ting while also accelerating the optimization in non-interfering directions. We show how to adapt the estimation of the FIM to a continual setting, stabilizing second-order optimization for non-iid data, uncovering the role of the Tikhonov damp- ing in the stability-plasticity tradeoff. Empirical results show that OCAR outperforms state-of-the- art methods in continual metrics, achieving higher average accuracy throughout the training process in three different benchmarks.
et al., 2023) shows that most OCL methods have high forgetting and fail to beat a simple replay baseline on some metrics; (De Lange et al., 2023; Caccia et al., 2022) discovered the stability gap, a sudden drop in performance at task bound- aries; (Dohare et al., 2024) even observed loss of plasticity, limiting the ability to learn in time.
These results suggest fundamental failures of the algorithms. Most methods focus on preventing forgetting at the end of a learning task, an approach that does not ensure stable optimization throughout the entire stream (stability gap); furthermore, stability is often optimized the expense of plasticity. We argue that a proper CL optimizer should seek to maximize both stability and plasticity, assuming that the model is large enough, at every step in time, interpreting it as a continual filtering process (filter intended as in nonsta- tionary time series literature (Durbin & Koopman, 2012)).
This paper proposes Online Curvature Aware Replay (OCAR), a novel method designed for replay- on- line continual learning, aiming at tackling the challenges in both plasticity and stability inherent in this scenario with continual optimization approach at every step in time. We formalize OCL as the joint optimization on past and new data, with past data approximated using a limited replay buffer, and adding an explicit constraint on the variation of KL--divegeece on previous information. The Fisher Infor- mation Matrix (FIM) is used to capture the loss function's curvature, providing both plasticity and stability constraints in the model distribution space. When the KL divergence is used as a metric, the FIM has the additional value of describing the curvature of the parameter space itself, be- ing the Riemannian metric tensor of that space (Amari, 2016). We can directly adapt our gradient to the geometry of the space in stark contrast with traditional CL meth- ods that use the FIM as penalization term (Kirkpatrick et al., 2017). fronecker-factorce Approximate Curvature (K-FAC) (Martens & Grosse, 2015) is used to efficiently ap- proximate the FIM, with some critical adjustments to make it work in OCL settings.
1. Introduction
Online Continual Learning (OCL) models are trained contin- uously on nonstationary data stream. The goal is to obtain model that is accurate any point in time, quickly adapts to new data (i.e. plasticity), and does not forget old informa- tion (i.e. stability), without any information about task start, end, or identity. In this setting, many standard Continual
Universitt of Pisa, Pisa, Italy "Scuola Normale Su- periore, Pisa, Italy. Correspondence tini <edoardoo Codoaado.urcttinieee urettiniissssit>, Antonio Carta <anto- nio.carta uni
Edoardo Uret-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Online Curvature-Aware Replay
The main contributions of this paper are: the design of On- line Curvature-Awar Replay (OCAR) as a combination of replay, second-order optimization, and information geom- etry; the analysis of the Tikhonov damping and its ratio with the learning rate in the stability-plasticiyy tradeoff; an improvement of state-of-the-art performance in standard computer vision benchmarks across all the stability and plasticity metrics.
in the null space of the latent activations. (Mirzadeh et al., 2020) discusses the relationship between the curvature of the first task and the forgetting, proposing hyperparameter schedule that implicitly regularizes the curvature. More recently, LPR(Yoo et al., 2024) exploits proximal optimiza- tion in the L2 space of latent activations, and it is the only projection- baaeed optimizer compatible with replay. (Hess et al., 2023) proposes a combination of GEM and replay as potential mitigation for the stability gap.
2. Related Work
Natural Gradient and FIM in CL: Natural gradients can be used to train neural networks thanks to efficient approxi- mations of the Fisher Information Matrix (FIM) (Martens & Sutskever, 2012), such as the K-FAC (Martens, 2020) and E--FAAC (George et al., 2018). Interestingly, Benzing (2022) showed that K-FAC seems to work better than the full FIM in some empirical settings, which is connected to form of gradient descent on the neurons. In continual learn- ing, the FIM is typically used to approximate the posterior of the weights with a Laplace approximation. The result is a quadratic regularizer that is combined with the loss on new data, as introduced by EWC (Kirkpatrick et al., 2017) and its several extensions (Chaudhry et al., 2018; Liu et al., 2018; Huszár, 2017). Magistri al. (2024) proposed to the FIM only for the final layer, which can be computed efficiently with a closed-fome equation. As an alternative, FROMP (Pan et al., 2020) computes a Gaussian Process pos- terior, which is also used to estimate the importance of sam- ples in the replay buffer. Daxberger et al. (2023) proposes a method that combines EWC, replay, and knowledge distil- lation. More relevant our work, NCL. (Kao et al., 2021) proposed modified natural gradient step with a quadratic posterior as in EWC. Here the Fisher is computed only at the boundaries and the method does not support rehearsal, making it difficult to implement in OCL.
Continual Stability and OCL: Most continual learning methods assume that stability must be kept at the expense of plasticity (Lange et al., 2022; Masana et al., 2023), the so-called plasticity-stabilityy tradeoff, and therefore are de- signed to preserve knowledge about previous tasks to mit- igate catastrophic forgetting (French, 1999). However, re- cent evidence in (Lange et al., 2023; Caccia et al., 2022) suggests that even the methods with high "stability" mea- sured at the end of tasks suffer from high instability and forgetting immediately after the task switch, and they re- cover the lost performance over time. (Kamath et al., 2024) found evidence of the stability settings. In OCL settings, gap even in incremental i.i.d. et al., 2023)
showed that some state-of-the-art methods are unable to outperform a simple reservoir sampling baseline on some fundamental stability metrics. Furthermore, CL methods also fail at keeping plasticity, and (Dohare et al., 2024) pro- vides evidence of the loss of plasticity in deep continual networks. Overall, the literature suggests that CL methods fail at both stability and plasticity due to instabilities in the learning dynamics. Recently, some methods such as On- Pro (Wei et al., 2023) and OCM (Guo et al., 2022) proposed novel self-supervised auxiliary losses and prototype- based classifiers, two approaches orthogonal to our optimization- based method. Regarding the latest works on OCL with pre-trained models, (Moon et al., 2023) proposed Si-Blurry, a more realistic scenario with stochastic and blurred bound- aries, and introduced MVP, prompt-based tuning method that keeps the backbone frozen. Advancing on this, (Kang et al., 2025) presented MISA, which introduced a forgetting- aware initial session adaptation and a non- non-parametric logit masking to alleviate forgetting. In our work instead, we focus on the full training, not only finetuning.
Comparison with our work Most CL methods use the FIM to compute a Laplace approximation of the posterior. This is not possible in OCL because the model is never assumed to be at a local minimum, not knowing task boundaries or length. In general, in the CL literature, the FIM is often restricted to its use in quadratic penalties. While quadratic regularizers are easier to use, we show that the use of the Fisher as a gradient preconditioner is promising direction, improving the optimization path. Many of the limitations found for EWC and similar methods may be due some suboptimal choices in the use of the FIM, such as the use of the empirical FIM, popular in the CL literature but with different properties from the FIM (Kunstner et al., 2019). Another difference with the literature is that our optimizer is compatible with replay, unlike most projection- based meth- ods. Furthermore, while most methods penalize plasticity indirectly to prevent forgetting, our approach is designed to improve both on learning speed and forgetting.
Most CL optimiza- tion algorithms are designed to prevent forgetting by re- moving interfering updates. GEM (Lopez-Paz & Ranzato, 2017: Chaudhry et al., 2019a) models interference using the dot product of the task gradients and constrains the model updates to have positive dot products with the gradients of previous tasks. Subsequent work explored orthogonal pro- jection methods (Saha al., 2021; Farajtabar et al., 2020) that either extend the idea of interfering gradients or project
Optimization in Continual Learning:
2

---

Online Curvature-Aware Replay
3. Online Continual Learning
chine learning implicitly assume stationarity, which justifies the gradient estimate from the minibatches (e.g.: ADAM (Kingma & Ba, 2015), SGD), Instead, in OCL the distribu- tion can change at any point in time (non i...d). At each step, the method can only use the current minibatch and, in replay-baseh methods, an additional minibatch sampled from a small buffer of old data.
In continual learning (CL), the model learns incrementally from nonstationary data. In most CL settings. we can identify a sequence of tasks Ti..... .N. each one with its own distribution. For example, in learning (van de Ven & Tolias, 2019), each task has dif- ferent classes, and tasks are seen sequentially during train- ing, The goal of the model is to learn all the tasks seen during training. Given the model parameters wy learned at time 1, a loss function L, and a test set for each task Di...., Dv. we can evaluate the model with the average task loss (t) Σi= L(w,, D;).
We define our learning process as a sequence of local optimization problems (as in (Martens, 2020), section 7). Unlike in stationary settings, these prob- lems can greatly change from one step to another, preventing the use of more "global" approaches (e.g., learning rate de- cay and momentum (LeCun et al., 2015)). For example, the previous gradient directions can be of little use after a task change, making momentum detrimental to fast adaptation to new data ( (Yuan et al., 2016). On the other hand, learning rate decay would forcibly decrease plasticity in time, reach- ing point where no new task can be learned at all. Each single step must be meaningful by itself. What is forgotten or not learned could be lost forever.
First-order optimization:
Online continual learning (OCL) requires some additional constraints and desiderata (Soutif- Couuif-Cormerais et al., 2023; Yoo et al., 2024; Mai et al., 2022): (DI-Online training) at each step, we do not have access to the whole training dataset for the current task, but only a small minibatch, which can be processed for limited amount of time: (D2- Anytime inference) the model should be ready for inference at any point in time; Consequently, (D3-Continual stability) the model must be stable at any point in time, instead of only at the task boundaries; (D4-fast adaptation) the model must also be able to learn quickly from new data.
The Hailbbcck-Libble (KL) divergence (Thomas & Joy, 2006; LeCun al., 2015) is used as our objective, aiming to minimize the "distance" between the pre- dicted and the real distribution. The KL is esti- mated on the current batch of data (L(DDllf((DD) = (ED.))) In our
There are similarities with the traditional Online Learning (OI.) approach: at each time a new observation is made from an unknown task, new loss is experienced, and the final objective is to minimize the cumulative loss on stream of urbitrary length. The difference in OCL is that the aim is also to minimize forgetting, hence having a model able to perform well also on past data of the stream. To get this, usually, access to some constrained past information (for example, a small collection of past examples) is allowed.
ΣS N KT(P(yD.)) (er.://)/ll notation D represents the batch of data, i the specific sample, y the target variable dependent on the observed x. fw is the model parametrized by w that is supposed to represent the reality when w = w empirically observed distribution is p(yp,ilf (x).))
Experience Replay (ER) (Chaudhry et al., 2019h) keeps a small buffer B of past data and solves joint optimization on batch N, sampled from the unknown real current data distribution and a batch B, sampled from B:
Following the literature, we focus replay- based methods, which use a limited buffer of previous data for rehearsal by combining new minibatches with samples from the buffer at each iteration. This work assumes no access to knowledge about task identities, boundaries, or length. Each single observation in the stream can be sampled from different distributions/tasks.
min 6. subject to 1
KLLUN, Sw (x.)) + KL(Nn, ll (Fn))
11101 Se
where 6, we wt-1- Here and throughout the paper we write a sum of the two KI. divergences for visual ease. Usually, and also in our experiments, the mean is taken, combining the data in a single batch with no distinctions. The KL divergence can be approximated by first-order Tay- lor expansion:
4. Online Curvature-Aware Replay
We now present the building process of our method, starting from the standard ER optimization and expanding it to a second-order method, which is then approximated using FIM and followed by some final adjustments to enhance its CL. performance.
KLD. fl = KL(YD. 1l (I + V.. 8
where VD, V.r fw (FD)) is the gradi- ent with respect to the weights w of the KL between the targets and the model predictions, with V representing the gradient operator and A the resulting vector. When comput- ing this on newly observed data D, = N₂, while for buffer
4.1. The optimization problem
Online continual learning (OCL) is an online learning prob- lem in nonstationary setting. Common optimizers in ma-
3

---

Online Curvature-Aware Replay
data D, Bt- After approximation, the solution for prob- lem 1 is 6-- - + Vn.) where À is the Lagrange multiplier of the constraint. ER actively optimizes the buffer and the current data distributions with no distinctions and no forgetting constraint. The only stability require- ment is 1 A (the learning rate), limiting the movement of the weights in all directions, trading stability for plasticity.
The term KL(Sw, (FB,) ) fe, (xg,)) measures the varia- tion of the model predicted distribution on buffer data, and its expansion around the pre-update parameters W;-1 will have zero zeroth and first-order terms. The remaining term is controlled by the Hessian of the KI.-divergence evalu- ated at w = wt-1. which is exactly the Fisher Information Matrix (Ollivier et al., 2017) (Martens, 2020):
Failure of ER with first-order optimization: Unfortu- nately, the first-order information can be poor approxima- tion in CL. Often, information about the curvature is neces- sary to avoid catastrophic forgetting, For example, at task boundaries, the model is often close to a minimum for the previous task. In that case, buffer gradients VB R 0, while new gradients V N can potentially be much higher, domi- nating the update direction. This issue partially explains the stability gap (De Lange et al., 2023). Second-order methods can solve this problem by enlarging the "sight" of our optimizer with the local variation of the variation. The second-order Taylor expansion of the KL divergence is:
8 @KK.mm (F (11B)lllec I1 (xB.)
F B
Ow, Owy
The optimization problem becomes
min VT, + 2
H
2
subject to
F"FF SP
_11811 s,
2
(2)
which is solved by
KLDYD. (zDD.)) Ne KL 1
1 V.. de +
67 = -HHN + HB. + AF +TI) + +VB,,
50 HD
2
where À controls the strength of the stability constraint, ensuring that the predictions of the model remain stable, while r is the Tikhonov damping controlling directions of maximum acceleration. As done in many second-order methods (Martens, 2020: Martens Sutskever, 2012), a learning rate a can be used uniformly control the step size in all directions, making the final update be adõ As we will show, the relation between a and has an important impact on the learning dynamics and the tradeoff.
where now HD, L.D, |l fu==.. Hessian of the KL computed on data Dr, with H. represent- ing the operator to compute the second-order derivatives and H the resulting matrix. again, D, it can be both N₂ or B. Using this expansion, the problem is solved by
(xD.) is the
8 - (HN, + HB, + +II--'(VN, + BB)),
where the Lagrange multiplier v, still related to the L2 regu- larization imposed on the update, now acts as a Tikhonov damping term (Martens & Sutskever, 2012). Unlike the learning rate in SGD, r has a different effect on the eigendi- rections of the Hessian, which depends on their eigenvalues (see Sec. 5).
4.2. Estimations and approximations
Approximating the Hessians: and one Fisher Information Matrix would be impractical and inverting them unfeasible. Luckily, in the second-order opti- mization literature, has been shown how for some distri- butions (including the multivariate normal and the multino- mial), the FIM is equivalent to Generalized Gauss-Newton (GGN) matrix, both approximating the Hessian of the KL- divergence (Martens, 2020). They are equal to the Hessian when the model describes well the data (near optimum) and the work of Martens (Martens, 2020) illustrates some gen- eral advantages of these approximations over the Hessian also when far from optimum.
Computing two Hessians
Stability constraint: By using the Hessian matrix to pre- condition the gradient, we expect faster learning on both new and past data, in a Newton-like fashion. If we "stretch" the plasticity requirements to include new information in both new and past data, we can consider this approach to directly answer the fast-adaptation desiderata. What still lacks is an explicit stability constraint necessary for non-i.i.d. settings to obtain the Continual Stability requirement. Com- bining experience replay, second-orde methods, an explicit constraint for stability on past data, and L2 regularization on the update, the problem becomes:
Moreover, the use of the FIM has the additional interpre- tation of describing the curvature of the parameter space itself. In this space, each point represents a distribution, requiring the use of the KL divergence as "distance". The KL-divergence of infinitesimal displacements corresponds to 2 Frydw;dw, giving rise to the Fisher Information as the
L'NNN fan.)) + KLVVE, ll fe, (z),) min subject to
fl <
| 2
<
4

---

Online Curvature-Aware Replay
metric tensor used in this manifold (Amari, 2016). More than simple approximation of the Hessian, the FIM can be used to correctly measure distances in the manifold where we are performing the gradient descent if the steps are small enough. Assuming a small learning rate, our method would use the preconditioner not as in a Newton method (that usually requires a learning rate of 1 to directly go the the approximate optimum) but as a metric adaptation.
for large networks. While diagonal approximations are fre- quently used (Kirkpatrick et al., 2017; Kingma & Ba, 2015), given the particular challenges of OCL more informative approximation is needed. We rely on the mronecker-factorey Approximate Curvature (K-FAC) in its block-diagonal ver- sion to approximate the FIM (Martens & Grosse, 2015);
F = (Ao.o & G..1,..,AA-11--- @ Gr,).
We believe these two results justify the use of the Fisher in- formation in nonstationary settings as OCL. Following this, we approximate the two Hessian matrices of our solution with the FIM, greatly simplifying the computations. The new optimal update becomes (including the learning rate):
where each block corresponds to an approximation of the covariance matrix of the score of a specific layer, obtained by the Kronecker product of A = E[a,a?] and G.2 = Elg.9"] a, is the vector of the layer activations (with an additional appended for the bias) and 9i the gradient of the prediction with respect to the output of the layer before the activation function. The expected values of both are computed with an Exponential Moving Average (EMA) of past values. The efficiency of this method relies on the property of the Kronecker product (A@B)-1 = & allowing us to invert much smaller matrices for each layer, ignoring layer interactions.
82 - =a + (1 + A)F + -I) N. + B.).
This implies the second-orde order information is now taken with respect the model prediction instead of the real targets as also F F H [f (Ix,) fl The gradient of our method corresponds to the usual gradient obtained from the loss function when the loss is derivation of the KL divergence (negative log-likelihood, cross-entropy, etc.... The weighted sum (or mean) of the two FIMs can now be obtained by computing single FIM on the batch, giving more weight to the buffer data.
4.3. Nuts and Bolts for OCL
To make the method work in practice, we found some ad- justments are needed.
There is a deep connection between our method and Natural Gradient (NG) (Amari, 1998). We have to underline that our building process is much different from the one used for the original NG, thought for stationary settings. Moreover, our preconditioner is not really the FIM of the model, but a modified and regularized version stemming from our OCL second-order optimization with the use of replay. We then cannot assume all benefits of NG would apply to our case.
Hyperparameter optimization and scheduling: Usu- ally, hyperparameter selection is done only on the first K tasks of the stream, but it must generalize to longer streams during training. Our method uses three hyperparameters: the learning rate o, the Tikhonov damping strength т, and the parameter y for the EMA used for Kronecker factors estimation. In hyperparameter selection, we found benefi- cial to search for value for the increase of instead of itself. T is then initialized at the same value of the learning rate and increased by the selected value at each optimization step improving long-term stability.
The Empirical FIM: The Fisher is also defined as the expected value of the squared score:
F- Σ Ey-pislfu(ea)) V pog((vlfiu(m)))
Estimate of K-FAC factors at boundaries: In class- incremental settings, the shape of the classifier will grow over time as new classes are observed. This means the K- FAC factor G1, of the last layer will change shape, also breaking the relations with the previously observed gra- dients. To avoid keeping track of errors, if G1, changes shape, the EMA of this factor is reset. This is not done for the other factor of the last layer A We assume the model has consistent representations, and that the majority of instability is happening on the classifier.
V ((
One could argue that better preconditioner for the gradi- ent would be the Empirical Fisher (EF) matrix, computed using the real target y instead of the one sampled from the predicted distribution, in particular for approximating the Hessian of new data H Besides getting us outside the theory about Natural Gradient and Fisher/GGN equivalence, has been shown is a questionable choice, even when the model is not a good description of the data (Kunstner et al., 2019). For these reasons, OCAR uses the "real" Fisher, unlike other traditional CL. approaches (Kirkpatrick et al., 2017).
Scheduling of A: The stability constraint of problem is estimated on the current batch extracted from the buffer, but it should represent the whole buffer. As new experiences are encountered in time, the information content of the buffer will grow, with less redundancy. At the limit, we can get a buffer where each example represent a different class or domain. Being À the Lagrange multiplier, it is directly
K-FAC: This approach has thcoretical advantages, but it re- quires the inversion of an extremely large matrix, unfeasible
5

---

Online Curvature-Aware Replay
Task (ACC-0.72)
Task 2 (ACC-0.73)
AVG (ACC 0.84)
across all analyzed loss landscapes (Task Task 2, and Average loss), which results in consistent improvements over time (plasticity) and mild forgetting (stability). Second- order information moves the optimization directly toward the next minimum (black stars in the plots), in fewer steps. On the other hand, ER always suffers from instability at the task boundaries (right after black stars in the plots) which results in an abrupt deviation from the optimal path. The learning curves on the first task (Fig. 10b) and the average of all tasks (Fig. 10a), available in Appendix G, confirm the result. OCAR maintains a smoother learning path without experiencing any stability gap on the first task and ending the stream with higher overall accuracy. ER instead, while still performing well in the basic MNIST setting, suffers from much more instability during training.
1.0
0.6
0.4
0.0
(a) OCAR 2D projection of the learning trajectory.
Task (ACC-0.56) Task (ACC-0.61)
AVG (ACC-0.75)
0.8
0.6
0.2
0.0
(b) ER 2D projection of the learning trajectory.
Role of the Eigenvectors and Hyperparameters: One approach to understand OCAR's 's effect is to study how the eigenvalues of the matrix a Fx + (1 + X) +T)- are related to the hyperparameters a, X, and T. We can diag- onalize F = FN + (1 + F) = QIQT, where Q is a unitary matrix whose columns are the eigenvectors and E diagonal matrix with the eigenvalues o+, Therefore, we find that the eigenvalues of a(Fx + (1 + X)F + +/)-1 are of o. In the new coordinate system defined by the eigenvectors Q, we can interpret OCAR as slowing or accelerating directions depending on their curvature. Direc- tions with or < 1 (o; > 1) correspond to directions with high (low) curvature for some tasks. The learning rate a and the Tikhonov damping strength rescale these eigen- values (as shown in Figure 9 in Appendix). In particular, learning rate < 1 decreases the step size. Conversely, mitigates the acceleration caused by small eigenvalues. When o; + 0, of + s, limiting the maximum accelcration in each direction. When o; >> T. we have of x a which is approximately independent of T.
Figure 2D projections of the training trajectories for ER and OCAR on Split MNIST (5 Tasks). Loss surface on the first task (left), second task (middle), and the average loss on all the 5 tasks (right). The black stars highlight the task boundaries. More details on the 2D projections and additional plots are available in the Appendix.
connected with the constraint p. By decreasing p to have a stronger constraint and save buffer information, we get an increase in A. For class-incremental problems, À increases with the number of different classes encountered. In domain- incremental settings, it grows in time (as done with r).
The composition of all these pieces together forms the On- line Curvature-Aware Replay (OCAR): at each step. the method is the optimal step of a second-order replay-based optimization problem, approximated with the FIM made computationally viable via K-FAC, with some fundamental adjustments to make it work in practice. The algorithm can be found in the Appendix A.
Empirically, we find that the spectrum spans several orders of magnitude, with the smallest eigenvalues close to zero and a small set of very large values around (10*, 10%) (con- sistently with the common intuition behind methods such as EWC, which expect few important parameters).
5. Continual Stability in OCAR
In this section, we provide a qualitative analysis in a sim- plified setting, showing how OCAR results in smoother continual optimization compared to ER and how a, 7, and their ratio can be used to control stability and plasticity.
Stability-Plasticee tradeoff of OCAR hyperparameters: An interpretation of OCAR hyperparameters in the stability- plasticity tradcoff can be given. While higher À gives more importance to the FIM of the buffer (stability) and higher values of a allow larger learning steps (plasticity), the role of and is less intuitive. First, higher values of are needed for the introduction of new classes, that creates instability due to the FIM being the variance of gradients (See the Appendix C for detailed explanation). Figure shows the results of a grid search on and (Complete figure in Appendix fig 11). We can make some empirically- based speculations: (1) the ratio a controls the learning
Loss landscape and model trajectories: We can visualize the improvements in the optimization trajectory of OCAR in simple continual learning setting. We train a small feed- forward network with ER and OCAR on Split MNIST (5 Tasks). Given the small size of the model, we can store the entire training history of the model, which allows us to plot 2D projections of the model trajectory in the loss landscape (Figure 1) (details in Appendix F).
Looking at their wise and joint loss surfaces in figures aa and 1b, OCAR shows a much smoother model trajectory
6

---

Online Curvature-Aware Replay
Taskk Forgetting
Task Accuracy (Hesticity
tations in OCL when no task boundaries are provided and it's not possible to select the previous task's "best" weights (Mai et al., 2022), breaking the fundamental assumption be- hind the Laplace approximation. Additionally, regularizing the loss does not directly speed up non-important directions, a non-optimal approach in OCL. On the other hand, outside CL, the Natural Gradient Descent (NGD) (Amari, 1998) uses the FIM as a preconditioner for the gradient, slowing it down in the direction of high curvature and accelerating it in others. We believe NGD can be well-suited for OCL problems. The problem is that raw NGD is derived for i.i.d. settings. OCAR, on the other hand, is an adaptation for non-i..d. problems. To underline the differences, we tested the three approaches in combination with ER in an online stream of 10 small convex tasks (all details in Appendix
10a
00 .00
10
89
52
08
10f
-57
-52 -59
-8
39
29
10--
.41
-66
39
29
10:7
16-3
32
.90
-1.0
10-110
10-1100"10-11000
10
Figure 2. Grid search over a and :: (left) forgetting on the first task, (right) plasticity measured as the accuracy the final task. Metrics are computed on the test stream at the end of training.
ER
EWC
NGD
OCAR
x10
x10
-8
D). We measure the cumulative loss experienced on single batches during the training Lp Σ, L(v, 9t) to measure the ability to adapt to current data and the cumulative loss experienced on all previous data L, = Σ, L(voir No:s) to measure the stability of the model. The results in figure shows that, while NG is much more adaptable than EWC, it slightly less stable. EWC performance OCL is very similar to the one of basic ER, a result aligned with (Mai et al., 2022). OCAR, thanks to its explicit memory con- straint, and its dynamic hyperparameters is able to improve both on the speed and on the stability, showing a slight optimization superiority already in this very basic setting.
1.0-
1.0
0.5
0.5
0.0
Time
0.0
Time
Figure 3. Left: Lp Cumulative loss of single batches. Right: L. Cumulative loss measured on all previous data of the stream.
plasticity. As a result, hottom-left elements in Figure 2 show less plasticity; (2) & controls the effective step size in the eigendirections with small eigenvalues, problematic at task boundaries. If the ratio is too low, OCAR may use updates that are too large, resulting in instability (top-right); (3) Decreasing & consistently increases forgetting; (4) After sufficient number of steps after task drift, the FIM becomes more stable, and a becomes less important in the learning dynamics. At this point, small a will result in conservative updates, with low plasticity and high stability. In the figure, we notice that for equal values of the ratio, models may have similar accuracy but different sability-pllasticn tradeoff, depending on their (low=stability, high=plasticity)
6.2. Literature benchmarks and SOTA comparison
To ensure reproducibility and a fair comparison we use the same code and experimental setup in (Soutif-Cormerais et al., 2023), later used by LPR (Yoo et al., 2024), an ICML.24 paper, that represents the SOTA in OCL and our main "competitor". We use Avalanche (Carta et al., 2023) and nngeometry (George, 2021). In line with the litera- ture Split-CIFARION 100 (20 tasks) and Split-Tinylmageen (20 tasks), are used as task incremental benchmarks. Fol- lowing (Yoo et al., 2024) we experimented also on On- line CLEAR (10 tasks) (Lin et al., 2021), a domain in- cremental scenario fundamentally different from the other two datasets. All the methods use a reservoir sampling buffer with 2000 samples for Split-CIFAR 100 and online CLEAR and 4000 for Split-TinylmageNee More details about the experimental setting are available in the Appendix B with specific information for CLEAR in Appendix E. Our entire code for the experiments can be found at https: / //github. com /ddo wrettinniic__staiiiii
6. Experiments
6.1. Comparison Between FWC,, NGD, and OCAR in a Convex Setting
First, we compare OCAR with alternative uses of the Fisher Information, that are not commonly considered for OCL, in scall-scale convex setting. In CL, EWC (Kirkpatrick et al., 2017) generated cascade of derived methods (Chaudhry et al., 2018; Liu et al., 2018; Huszár, 2017) based on the idea to add a quadratic regularization term to the loss, penalizing the movement of the parameters from an optimal configura- tion, weighted by the FIM. This approach finds some limi-
We compare our approach with a large set of CL baselines (Table 1), evaluated on 4 fundamental OCL metrics: WC- Ace (De Lange et al., 2023): the worst-case accuracy is a metric of the stability of the model, measuring trade-off between the model accuracy and the minimum accuracy among all tasks encountered. It is measured as a weighted
7

---

Online Curvature-Aware Replay
Method
Split-Cifaria (20 Tasks) AM 1
Oplit-Tinymmag (20 Tasks)
Acc 1
WC-Ace va
Probed Acc
Acc t
AAAT
WC-Ace val
Prohed Acc 1
Lidd
35.3 1.5
45.8 + 0.6
26.5 0.6
34.3 + 0.5
HR(Chaudhry et al., 2019h)
28.2 + 1.2
36.6 2.0
12.5 0.6
44.9 + 0.9
21.2 + 0.6
33.9 + 1.7
15.2 + 0.5
35.6 + 0.6
GDumb(Prabhu et ul., 2020)
18.5 + 0.5
13.1 + 0.4
AGEM(Chaudary et al., 2019a)
3.1 0.2
10.4=00 0.6
0.8
2.6 0.2
7.3
0.2
23.3 - 0.6
ER + LwF(Li& & Hoiem. 2017)
30.4 = 0.8
39.2 2.0
0.9
4.8
2
314
1.7
33.8
MIR(Aljundi et al., 2019a)
29.1 = 1.
332
11
0.7
0.8
310
1.3
3..0
RAR(Kumuri et al., 2022)
28.2 + 1.4
38.2 1.6
1.7
4.9
0.9
278
10.1
29.8
DER+ (Buzzega et al., 2020)
29.3 0.9
37.5= 2.5
1.74
4..0
0.5
3..2
0.3
315
ER-ACE (Caccia al., 2022)
29.9 0.6
38.5
1.99
4.6
0.7
3..0
1.78
342+
SCR(Mui et al., 2021)
28.3
2.1
0.4
0.3
1..9
1..7
1.3
2.
OnPro Wei
2023)
31.7 + 1.2
36.6 2.5
12.2
1..1
24.2+U44
0.0
OCM(Guo et al., 2022)
30.9 0.7
33
14
0.66
248
1..
LPR (Yoo al., 2024)
33.3 0.6
42.5
42.5=0.5 0.5
1.3
2.2
34.9
16.2
OCAR (ours)
34.9 0.6
482 1.2
1.10
2 0.6
21.7
38
1.
38.3 + 0.6
OCAR-DER+ (ours)
34.3 1.1
46.8 1.7
0.8
46.0
OCAR-ACE (ours)
35.6 + 1.2
48.7
1.7
26.5 0.4
44.1 44.1
25.6 0.4
39.8 2.0
21.5 + 0.9
34.7 + 0.3
Table 1. Results on Split CIFAR100 (20 Tasks) and Split Tiny ImageNet (20 Tasks). Best in bold for the base methods. Best underlined for all methods included OCAR+OTHER.
sum between the accuracy of the current model on the cur- rent task and the min-ACC. Min-ACC is the average of the absolute minimum accuracy that has been reached on previ- ous tasks after they have been learned. Hence, the metric is worse the stronger the stability gap is. AAA (Caccia et al., 2022): The Average Anytime Accuracy is a metric devel- oped specifically for OCL, measuring the mean accuracy of the model in time all the encountered tasks. If we evaluate the model on all encountered tasks at each time step (as shown in the plots in Appendix G). the AAA is the mean of these evaluations. It is measure of the model's 's overall accuracy history. Acc (Lopez-Paz & Ranzato, 2017): The final average accuracy a snapshot of the accuracy of the model the end of the entire stream. If again we evaluate each step the model on all encountered tasks, Acc would be equal to the last of these evaluations. Probed Ace (Davari et al., 2022): The metric is the final accuracy after freezing the feature extractor and retraining only the linear classifier on all the training data. It is a measure of the representation quality of the model.
stability at every point in time. Both metrics have shown significant improvement, with a jump of several points.
End of training accuracy: While in Split-Cifar 100 OCAR is able to achieve the best Acc among base models, ER-ACE takes the crown in Tinyimagenet (Table 1). However, we must point out that Acc is a very poor evaluation metric for the OCL performance. Since it is evaluated at the very final iteration, it badly represents the whole training and can be affected by noise or overly stable methods. We believe that ER-ACE and other methods have artificially lower plasticity, learning less than OCAR from each task, and that this approach can give an advantage at the very end of the stream as we approach the "learning limits" of the model. Of course, this works only for a specific stream length, while OCAR showed increased performance on the overall stream (AAA and WC-Acc).
OCAR obtains the best results on linear probing in both benchmarks, underlying the optimization improvements also on the feature extractor. This is sig- nificant result when compared to the i.i.d. case: while the classifier suffers from forgetting, OCAR can learn better feature representation if compared to the i.i.d. case, confirm- ing how it learns efficiently in nonstationary settings.
Probed Accuracy:
6.3. Results
Using the same code and setup, all results of other methods are taken from (Soutif-Cormerai et al., 2023) and (Yoo et al., 2024) (for LPR), except for OnPro (Wei et al., 2023) and OCM (Guo et al., 2022), where we reused the original code and run the experiments following the setup in (Soutif- Cormerais et al., 2023; Yoo et al., 2024). Everything is evaluated on 5 runs (except LPR in 10).
Integration with other methods: Being ER-ACE the method with the best accuracy on Tinyimagenet, we try a combination with it to see if OCAR can improve the re- sults. Even if ER-ACE uses a slightly modified loss, the integration with OCAR works very well. The combina- tion OCAR-ACE beats all other methods (including base OCAR) on Acc, AAA and WC-Ace. What is surprising is that this combination is able to overcome even the i.i.d. accuracy for Split-Cifar 100 while being very near to it for Split-Tinylmagenet. Both ER-CCE and OCAR-ACE show a
Continual metrics:
When evaluated at every step in time, OCAR obtains the best AAA and WC-Ace among the meth- ods in both benchmarks. This confirms OCAR as a robust, continual optimization approach with high accuracy and
8

---

Online Curvature-Aware Replay
lower probing aceuracy, which suggests again that ER-ACE may be tuned to prefer stability plasticity, learning less transferable features despite the high accuracy. Pos- sibly, the robustness of the classifier given by ER-ACE slightly interferes with deeper feature learning. Following this experiment, we also tried the combination with DER++ Unfortunately, we have found OCAR-DER to be much less stable (failing optimization on Tinyimagent), which we con- jecture to be caused by the entropy regularization implicit in the DER loss, which results in loss that is "too different" from the KI. divergence and breaks the assumption for the use of the Fisher.
incremental and domain-incremental the previous SOTA results, including the ICML.24 paper LPR (Yoo et al., 2024), using the same code, setting, and benchmarks. All of this is done efficiently: in table 3 the first-task training times are presented, showing how OCAR even improves on speed with respect to some previous SOTA approaches (Appendix B for more information). The method showed the possibility of being combined with ER-ACE, obtaining even stronger results.
7. Conclusion
In this paper, we revisit replay- OCL as a sequence of second-order optimizations with hard stability constraints and information geometry rooting. The resulting method OCAR shows consistent improvements in plasticity and continual stability both in class incremental and domain incremental scenarios, with clear hyperparameter interpreta- tion in the stability-plasticity tradeoff. The combination of OCAR and ER-ACE obtained even stronger results, being also capable of reaching similar final accuracies as the i.i.d. training in the class incremental setting. Future research directions include the comparison of different approxima- tions of the curvature (e-g. George et al. (2018)), alterna- tive derivations for the optimization problem (e.g. Benzing (2022)), and feasible dynamic adaptation of a and т. We believe OCAR can be the starting point for further improve- ments towards a deeper understanding of continual learning dynamics and the design a flexible and general continual optimizer.
Online CLEAR The Online CLEAR benchmark is fun- damentally different: no classes are added from task to task, but all classes evolve in time, in a domain incremental fashion. This puts much more importance forward and backward transfer and less catastrophic forgetting due to the similarities between tasks. For this reason, the final ac- curacy is higher than AAA (see figure 8 in Appendix). This setting, being more similar to the standard iid one, makes the estimation and the stability of the FIM much easier, making OCAR remarkably better than the previous SOTA LPR (see table 2). This confirms OCAR robustness also on domain incremental settings, underlying its optimization improvements.
Method
Online CLEAR (10 Tasks)
Acc f
AAA
WC-Ace val
ER
63.1 0.7
58.9 + 0.8
47.7 + 1.6
L.RR
65.2 + 0.9
63.5 + 1.0
62.6 + 0.7
OCAR(Ours)
75.3 + 0.8
73.9 0.5
70.3 + 0.5
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Table 2. Results on Online CLEAR (10 Tasks) domain incremental setting. 2000 Buffer size. Best in bold.
Method
Training Time (seconds)
A Acknowledgements
ER
14
ER LWF
15
This paper has been partially supported by the CoEvolu- tion project, funded by EU Horizon 2020 under GA n 161569.
MIR
31
ER-ACE
17
DER
17
RAR
72
References
SCR
131
LPR
213
Aljundi, R., Belilovsky, E.,, Tuytelaars, T., Charlin, L., Cac- cia, M., Lin, M., and Page-Caccia. L. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019a.
OCAR (Ours)
38
Table Training Time for the First Task Split-IIFAR-100.
Aljundi, R., Kelchtermans, K., and Tuytelaars, T. Task- free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
Final Comment: OCAR showed remarkable performance across all continual metrics, improving both on task-

---

Online Curvature-Aware Replay
pp. 11254-11263, 2019b.
Davari, M., Asadi, N., Mudur, S., Aljundi, R., and Belilovsky, E. Probing representation forgetting in super- vised and unsupervised continual learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16712-16721, 2022.
Amari, S.-I. Natural gradient works efficiently in learning. Neural computation, 10(2):251-217 2766, 1998.
Amari, S.-i. Information geometry and its applications, volume 194. Springer, 2016.
Daxberger, E., Swaroop, S., Osawa, K., Yokota, R., Turner, R. E., Hernández-Looato, J. M., and Khan, M. E. Im- proving continual learning by accurate gradient recon- structions of the past. Transactions on Machine Learning Research, 2023.
Benzing, F. Gradient descent on neurons and its link to approximate second-order optimization. In Chaud- huri, K., Jegelka, S., Song, I.., Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference Ma- chine Learning, ICML 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 1817-1853. PMLR, 2022. URL http:: htapttt:::/ppoooceieii /ptp//eerceeeeesseeoeeeeeee..sse mwr..rreess 62. http///ezznnna....mm htmml.
De Lange, M., van de Ven, G., and Tuytelaars, T. Continual evaluation for lifelong learning: Identifying the stability gap, 2023.
Dohare, S., Hernandez-Garcia. J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nat., 632(8026):768-77777 2024. doi: 10.1038844156--00000711117 URL https: wtd.ii.oo/.......00 http:////.................--00-0
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learn- ing: strong, simple baseline. Advances in neural infor- mation processing systems, 33:15920-15998 2020.
Caccia, L.. Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J.. and Belilovsky, E. New insights reducing abrupt representation change in online continual learning. In The Tenth International Conference on Learning Repre- sentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. http::/:: openreview. net met/fooumm? forum? id- NaMBB
Durbin, J. and Koopman, S. J. Time series analysis by state space methods, volume 38. OUP Oxford, 2012.
Farajtabar, M., Azizan, N., Mott, A., and Li, Or- thogonal gradient descent for continual learning. In Chiappa, S. and Calandra, R. (eds.), The 23rd In- ternational Conference on Artificial Intelligence and Statisties, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pp. 3762-3773, PMLR, 2020. URL http: httt:://proceedingg milrppress v1 08 fa Tarratttaaarra html.
Carta, A., Pellegrini, L.., Cossu, A., Hemati, H., and Lomonaco, V. Avalanche: A pytorch library for deep continual learning. Journal of Machine Learning Re- search, 24(363): 2023.
Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H. S. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Fer- rari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.). Computer Vision ECCV 2018 15th Euro- pean Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XI, volume 11215 of Lecture Notes in Computer Science, pp. 556-572. Springer, 2018 doi: 11tp://00011007700---00-000----0 doi. foo//org/10. 1007 /0000/997788--8
French, R. M. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 344):128-135, 1999.
George, T. NNGeometry: Easy and Fast Fisher Infor- mation Matrices and Neural Tangent Kernels in Py- Torch, February 2021. URL /ntttss//daae org/10. 5281/zenodo 4532597.
33. URL https://
George, T., Laurent, C., Bouthillier, X., Ballas, N., and Vincent, P. Fast approximate natural gradient descent in a kronecker factored eigenbasis. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018. December 3-8, 2018, Montréal, Canada, pp. 9573- 9583, 2018 URL https: http::/:/p/occeiings.....
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with A-GEM. In 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019a. URL ptttps:://peerevviiiie forum? 1d=kkf_aaC5P
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P., Torr, P., and Ranzato, M. Continual learning with tiny episodic memories. In Workshop on Multi-Task and Lifelong Reinforcement Learning, 2019b.
neurips.cc/ppn hewrrpps.cc/ppper///////eeeeee 4800GGTT315566600e913caaa33333 html.
ractt
10

---

Online Curvature-Aware Replay
Guo, Y., Liu, B., and Zhao, D. Online continual learning through mutual information maximization. In Interna- tional conference on machine learning, pp. 8109-8126. PMLR. 2022.
Kunstner, E,, Hennig, P., and Balles, L. Limitations of the empirical fisher approximation for natural gradient de- scent. Advances in neural information processing systems, 32, 2019.
Hess, T., Tuytelaars, T., and van de Ven, G. M. Two comple- mentary perspectives continual learning: Ask not only what to optimize. but also how. CoRR, abs/2311.00 11.3111.4000 104898, URL https: :/doi.org/1000 4885048850/iiii 2311. 04898.
Lange, M. D., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G. G., and Tuyte- laars, T. A continual learning survey: Defying forget- ting in classification tasks. IEEE Trans. Pattern Anal. Mach. Intell., 44(7):3366-3385, 2022. doi: 10.1109/ TPAMI,2021 IPA.M.202133057444 33074446. URL https::: /ttp//ooo//.../////////////// 1109/ 9099T 2021 3057446.
2023. doi: 10.48550/ARXI..311...85
Huszár, F. On quadratic penalties in elastic weight con- solidation. CoRR, abs/1712.03847, 2017. URL http:
//arxiv.org.//ooo/......oooooooe httgg/abs/// 1111/171..000 03847.
Lange, M. D., van de Ven, G. M., and Tuytelaars, T. Con- tinual evaluation for lifelong learning: Identifying the sta- bility gap. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: / //openrevi ew. net / forum? dd=2yy50csstcc
Kamath, S., Cormif-C A., van de Weijer, J., and Raducanu, B. The expanding scope of the stability gap: Unveiling its presence in joint incremental learning of homogeneous tasks. CoRR, abs/2406.051 151 2024. doi: 1044850/ARXXIV2440666111 URL pectps://ol.. org/10. 4 405548855//iiii/ 2406. 05114.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436 444, 2015.
Kang. Z., Wang, L., Zhang, X., and Alahari, K. Advancing prompt-based methods for eepla--ndependent general continual learning. arXiv preprint arXiv:2503.00677, 2025.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Kao, T., Jensen, K. T., van de Ven, G., Bernacchia, A., and Hennequin, G. Natural continual learning: success is a journey, not (just) a destination. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 28067-
Lin, Z., Shi, J., Pathak, D., and Ramanan, D. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2), 2021.
Liu, X., Masana, M., Herranz, L., Van de Weijer, J., Lopez, A. M., and Bagdanov, A. D. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recogni- tion (ICPR), pp. 2262-2268. IEEE, 2018
28079, 2021. URL hitps:/ http://:://pccceeiggg....
neurips.. peurips.cc/papeee hctps//cccc/ppee/////aaaaas//// 22550007840022221559900000222222 html.
Lppe-Pazz D. and Ranzato, M. Gradient episodic memory for continual learning. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6467- 6476, 2017. URL https://proceedaie
Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv. org/ abs/ /11111111112 6980.
meurips.cc/papeeeepe /ttp://scc///peeee//////aaaa//
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351 -3266, 2017.
html.
Magistri, S., Trinci, T., Soutif-Cormerais, A., van de Weijer, J., and Bagdanov, A. D. Elastic feature consolidation for cold start exemplar-free incremental learning. In The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL pettts://oppnzeeet net/ font/forumii.c wDtpmmmmmmdd0097mmmm0mmmm
Kumari, L., Wang. S., Zhou, T., and Bilmes, J. A. Retrospec- tive adversarial replay for continual learning. Advances in neural information processing systems, 35:28530-28544 2022.

---

Online Curvature-Aware Replay
Mai, Z.,, Li, R., Kim, H., and Sanner, S. Supervised con- trastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3589-3599 2021.
boundary via mask and visual prompt tuning. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 11731-11741, 2023.
Ollivier, Y., Amold, L., Auger, A., and Hansen, N. - optimization algorithms: A unify- ing picture via invariance principles. Journal of Machine Learning Research, 18(18)1166, 2017.
Mai, Z., Li, R., Jeong, J., Quispe, D., Kim, H.,, and Sanner, S. Online continual learning image classification: An empirical survey. Neurocompptting, 469:28-511 21,8-1,,2 2022. doi: N0..0016/NEEECOO..222.....0
Pan, P., Swaroop, S., Immer, A., Eschenhagen, R., Turner, R., and Khan, M. E. E. Continual deep learning by func- tional regularisation of memorable past. Advances in neural information processing systems, 13:4453-4466 2020.
URL https: doi www.ooggg//oo. 1016/ wtt.///...ee.mmmmmmmeeeee neucom. 2021 10. 021.
Martens, J. New insights and perspectives the natural gradient method. Journal of Machine Learning Research, 21(146):1-76, 2020.
Prabhu, A.. Torr, P. H., and Dokania, P. K. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 524-540, Springer, 2020.
Martens, J. and Grosse, R. Optimizing neural networks with kronecker- cronecker-factored approximate curvature. Interna- tional conference on machine learning, pp. 2402-448220 2417. PMLR, 2015.
Saha, G., Garg, I., and Roy, K. Gradient projection mem- ory for continual learning. In 9th International Con- ference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:/ http//pppeeeeeeeeeeeeeeeeeee.eep net forum? id=3A0JORCNC2.
Martens, J. and Sutskever, I. Training deep and recurrent networks with hessian-free optimization. In Neural Net- works: Tricks of the Trade: Second Edition, pp. 479-535. Springer, 2012.
Masana, M., Liu, X., Twardowski, B., Menta, M., Bag- danov, A. D., and van de Weijer, Class incremental learning: Survey and performance evaluation on image classification. IEEE Trans. Pattern Anal. Mach. Intell., 45(5):5513-55333 2023. doi: 10.1109/PPMM..222.. 3213473. URL https::/doo http::://doi.oo attps://do..org/1... TPAMI 3211233..
Soutif-Cormerais, A., Carta, A., Cossu, A., Hurtado, J., Lomonaco, V., van de Weijer, J., and Hemati, H. A comprehensive empirical evaluation on online continual learning. In JEEE/CVF International Conference on Com- puter Vision, ICCV 2023 Workshops, Paris, Frunce, October 2-6, 2023, pp. 3510-3520. IEEE, 2023, doi: 1L11099/CCVW66673222233.. doi.orrg/1..109//CC
1109/
URL https: // 00378.
Mirzadeh, S., Farajtabar, M., Pascanu, R., and Ghasemzadeh, H. Understanding the role of train- ing regimes in continual learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: /ttp://////./ccecegg..... neurips oc/
Thomas, M. and Joy, A. T. Elements of information theory. Intey--nterscience, 2006.
van de Ven, G. M. and Tolias, A. S. Three scenarios for continual learning. CORR, abs/1994.00734 2019. URL www.abs.///a...oooooo.o........t 1904. 07734.
51 wttpapppeee/2///asttttt html.
Wei, Y., Ye, J., Huang, Z., Zhang, J., and Shan, H. Online prototype learning for online continual learning. In Pro- ceedings of the IEFF/CVF International Conference on Computer Vision, pp. 18764-18774, 2023.
Mirzadeh, S., Farajtabar, M., Görür, D., Pascanu, R., and Ghasemzadeh, H. Linear mode connectivity in multi- task and continual learning. In 9th International Con- ference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
Yoo, J., Liu, Y., Wood, F., and Pleiss, G. Layerwise proxi- mal replay: a proximal point method for online continual learning. In Proceedings of the 41st International Confer- ence on Machine Learning. ICML'24. JMLR.org, 2024.
Yuan, K., Ying, B., and Sayed, A. H. On the influence of momentum accelcration on online learning. Journal of Machine Learning Research, 17(192): 2016.
Moon, J., Park, K.-H., Kim, U., and Park, G.-M. On- line class incremental learning on stochastic blurry task
12

---

end for Online Curvature-Aware Replay
A. Algorithm
Algorithm 1. Online Curvature-Awant Aware Replay (OCAR)
Input: network parameters w, learning rate a, per batch gradient steps count S, Tikhonov increase Ar, EMA parameter
.ЕMA-
Output: trained network parameters
Initialize replay buffer: B 0-
2:
3: A+1
4: for (1,...,00} do
5: Obtain new data batch N₁.
6: Sample buffer data batch Br
7: for s {1.....S] S} do
8: Compute loss ((w) using N₂ and B,
9: Compute loss gradient VL(w).
T T+ + AT
11:
if Class Incremental then
12:
Update known classes list with No Increase Ả with new classes
13:
14:
else
15:
À A+ AT
end if
ifs then
18:
Compute K-FAC factors A and G with N₂ and B, (B, influence weighted by X) for for I,... L do
19:
20:
AEMAI < (1 aεMA) + AEMAA,
21:
GEMA, < (1 GEMA) + OEMAG
22:
if L and changed shape then
23:
GEMAJ <- G1
end if
25:
end for
26:
FEMA + AEMA, GEMA
27:
FINV < (FEMA+ +1)-
28:
end if
29:
VL(w) + FINV V((w)
30: w < w - aVL(w)
end for
32:
Reservoir.upaate(B.. Nt, maxsize)
33:
The K-FAC computations are done after weighting with À the data related to the buffer.
B. Experimental Setup
Code The entire code used to perform the full experiments (CIFARI00, TinylmageNet Clear) can found in anonymous repository at https: github. con/ /tte//douurrete ietooouretini///e CL__ stability. The anonymization process could have removed some important parts of the code but this is highly unlikely. The code was built combination of the repository for the OCL survey Cortif-Cormerai al., 2023) and the angeometry repository (George, 2021).
The experiments were performed a Linux cluster equipped with Nvidia Tesla V100 16GB GPUs and Intel Xeon Gold 6140M CPUs. The training was not parallelized between the GPUs.
Hardware
13

---

Online Curvature-Aware Replay
Main training setup All the experiments have been performed in (Soutif- Soomir-Coormrris al., 2023). We only report the most important aspects here, referring to the original paper for all the additional details. The datasets used for the main Both are class-incremental benchmarks, divided in 20 tasks,
experiments are Split-CIFARI00 and Pedit-TinylmageNee with a number of new classes encountered at each new experience (5 in Split- Aplit-CIFARIOO 100 and 10 in Seiit-TinImmageNee The model used for the main experiments is a Slim-ResNetl 8 trained using SGD. SGD can help the model adapt faster when new data are encountered after a task boundary, while ADAM (Kingma & Ba, 2015) will need some time to update its running statistics. The batches are tiny, with 10 examples from the current portion of the stream and 10 from the replay buffer. The replay buffer for Split-CIFAR100 was kept at 2000 samples, while for Selit-TinImmageNe at 4000. The metries computed at the end of the training (cumulative accuracy and linear probing accuracy) are computed on test stream. The continual metrics (average anytime accuracy and worst-case accuracy) are computed after the training on each batch on heald-out validation set. The only additional information OCAR has access to is the number of classes on each experience (5 for one and 10 for the other). This information is only used to increase the weight À for the computation of the Fisher Information on the buffer data. If the buffer data contains n classes and the number of classes observed in the current portion of the stream is k, then the FIM of the buffer will weight E. While this information is useful, if it is not accessible, other weighting procedures can be performed, or. can simply be estimated by watching how many classes are observed in the first few steps of the training. Usually few steps are enough to observe all the different classes present in the task.
The same setup has been used in (Yoo et al., 2024), with the only difference they used 10 seeds for their experiments instead of only 5. There is also possible difference in the hyperparameter selection. Given the equality of the training and validation setup, we compared our methods' performances directly with the results of (Yoo et al., 2024) and (Soutif-Cormerais et al., 2023). This minimizes the probability of errors or bugs and uses the results obtained in the best conditions.
Training time We report the training time to complete a single task on Split-CIFARIO of different methods. Since OCAR and other OCI, methods do not increase their computational cost over time, we report the training times on the first task avoid unnecessary retraining of all the methods. We train on the first task without the continual evaluation to remove the evaluation overhead and provide fair comparison between the methods.
Method
Training Time (seconds)
ER
14
ER LWF
15
MIR
31
ER-ACE
17
DER
17
RAR
72
SCR
131
LPR
213
OCAR(Ours)
38
Table 4. Training Time for the First Task on Split-CIFAR-100
OCAR is about 3 times slower than standard ER, which does not perform any additional operations apart from the basic SGD loop. It is also faster than other sophisticated methods such as MIR, SCR, and LPR. We underline that in our tests LPR resulted much slower than what the authors found in their original paper (Yoo et al., 2024). We did not identify the cause of this. It can be something related to our hardware or the implementation details of our code.
Hy pyperaaraeterss selection As with everything else, the hyper-optimization is performed as in (Soutif-Cormerra Cormerais et al., 2023). The best configuration is selected as the best performing in the cumulative accuracy metrics on the validation set at the end of the fourth experience. This approach is suboptimal when working on nonstationary data. We notice that in our model, this shorter stream would select a combination of parameters prone to having high plasticity and lower stability. For this reason, instead of selecting fixed Tikhonov regularizer T, we instead selected a speed of increase for T. In this way, we found a much more robust approach. Differently from (Soutif-Cor Coutif-Cormerais et al., 2023), all our hyperparameters for OCAR have been selected after trying 100 combinations using the tree-structured Parzen estimator algorithm, instead of 200 trials used in the original survey. This can give our method small disadvantage, but we found it to be sufficient to get good
14

---

Online Curvature-Aware Replay
results. The only exception was OCAR-ACE on Split-TinylmageNe where 200 trials were needed due to the particular complexity of using loss function not derived from KI. divergence complex dataset as TinylmageNet. We want to highlight that in the LPR paper (Yoo et al., 2024), it seems the hyperparameters selection is done by training on all the experiences and selecting the best final accuracy. It is possible this can result in an advantage, avoiding the problem of the short and partial stream used (Soutif-Cormerais et al., 2023) and for our method.
C. Fisher Matrix Computations
The Fisher Information Matrix can be computed both the variance of the score (the gradient of the log-likelihood) as the negative expected value of the eurvature of the log-likelihood For classification problem assumee the model fw is predicting a probability vector probability vector p (f .Pk] where each Pk is the probability that ya 1, such that = f(z; w). The distribution assumed is then a categorical distribution. On a single example æ the FIM can be computed as:
Fww) Ey Cy-Cat(y) ? logCatly w)] By-Cattyla:u) V
Σ
yklogfu (2)x
Me Σn Fo(
(f)())
Σ
Th (w)
(tw(
k
Bhn
Owi
By-Cat(ylkuswnl
Σx 7 10)
We (fa(
Ee [al Jhes
OwH
Σx k Fi (z)
yk
(z)) )2
Ex (a(a)
vi
(fw(
(fw(a))
Ow
Owa
= EyxCCatyyknnsn)
Σx (a
(f)( (f)()))
dwn
Owg
En Tu(2)a (5-(a)) (ar) (a))
(mm(2)) )2
Σn Tw)
(fr (f)())
Oun
Own
Σn
T=( Own
Ow2
Note how each element is sum of the contribution from each class, weighted by the inverse predicted probability for that class. The Fisher computed for gradient composed the sum (or mean) gradients from multiple examples the sum (or mean) of the individual Fishers. This due the required assumption compute the Fisher: being the optimum of the log-likes ligllikel optimization, assuming the predicted parameters are the true ones. A consequence of this assumption is that the correlations between gradients of different examples will be zero: (VV w) + V(y, Jay; w)] V(V(y Jng w]] + V (y w)]. Then, our À to give more weight to the buffer data is easily implemented. For the same assumption about the optimum, the variance of the score is computed as the expected value of the squared gradients: the squared expected value would be equal to zero.
Last Layer FIM Focusing on the FIM of the last layer and ignoring all the correlations between the weights for moment, we can compute the diagonal element of classifier's 's weight. Assuming the loss C is standard cross entropy loss:
ac
h,(ps - -y:)
Ovey
where is the prediction for the class i and yi an indicator function for the real value. The variance of the score for this weight is:
(
ac Sw,, )
27 )}
E
- h13 Ps)
If we transform the partial derivative with the inverse variance:
0c
OC
p:
E
Ourza
owy.
h p:
pt)
15

---

Online Curvature-Aware Replay
Vebka
64
148
Pate Jukt
12
200
1500 2003
3lazs
Figure 4. Ratio between the norm of the gradient after being transformed with OCAR and the norm of the original gradient when a small T is used
In inassinncremenal settings, task boundaries, we can assume the probability predicted for the classes will be pretty low due to the random initialization of the weights. In this case, the gradient of the weights connected to this new unit will be high while the variance of the gradient will be very low, greatly accelerating the gradient, and contributing to possible instabilities due to the assumption of the Fisher of predicting the "true" distribution while, particularly at task boundaries, this is not true. Clearly this analysis ignores all the complex cross- cross-correlations between the weights. To visualize the complete effect, we plot in Figure the ratio between the norm of the gradient after being transformed with OCAR and the norm of the original cross-entropy gradient. We show only subset of the training on Split-CIFAR (lr/100). All task boundaries are clearly visible, showing the great acceleration the FIM provides when new classes arrive. This underlines the importance of the Tikhonov regularization (7) in class-incremental settings, where the Fisher can exacerbate some instabilities when the predictions of the model are completely off.
using a small
D. Qualitative Experiment
The code of the qualitative experiment is included in our repository. We used a basic linear model with 10 inputs and a single output trained solve a simple linear regression problem with MSE loss. 10 tasks with 1000 samples per task randomly generated. Each task has different multivariate normal from which input data are sampled and different real weights that should be estimated by the mode. The data are accessed as a stream, with single pass per batch. The Standard Vitter algorithm is used to keep a buffer of old data. Each method uses batch composed of 10 new data and 10 data sampled from the buffer.
Natural Gradient is applied directly by estimating the full Fisher information of the model with an EMA of the FIM computed on the single batches. This FIM after the EMA is used to precondition the gradient.
OCAR uses the exact same approach but uses À to give Tikhonov regularize) to increase them both in time. EWC is more tricky to implement in its basic form for online problems. Our approach is very similar to what is done in (Mai et al., 2022), an online extension of the EWC++ strategy (Chaudhry et al., 2018). The difference is that we use also replay data for computing the gradient. Namely, the loss is computed on both old and new data, but the Fisher (being a penalization) is computed only on old data (an EMA is kept also in this case for estimation). Given that no task boundaries are accessible, the penalization of the movement of the weights is done with respect the weights at the previous step in time. In this way, more regularized descent should be followed, penalizing the displacement from step step using the Fisher Information of old data. The loss is basic MSE but for analysis purposes, the figure D shows the cumulative loss (simply the loss experienced in each batch accumulated in time) and the cumulative loss on all previous data (at each step the loss on all previous data is computed after the optimization step and accumulated in time). The first measure shows a general capability of fast adaptation while the second an ability to find an optimum stable for the entire previous stream.
weight to old data and uses scheduling of both À and 7 (the
hyperparameter selection is performed to select the best setting on the sum of the final value of both these stability and
16

---

Online Curvature-Aware Replay
Task 1 (ACC=0.72
Task 2 (ACC=0.73 73)
Task (ACC=0.96)
Task 0.CC=
Task (ACC=0.88)
(a) OCAR TI.
(b) OCAR T2.
(c) OCAR T3.
(d) OCAR -T4
(e) OCAR TT.
Task 1 (ACC=0.56)
Task 2 (ACC=0.61)
Task 3 (ACC=0.84
Task 4 (ACC=0.76)
Task (ACC=0.99)
(1) ER TI.
(g) ER T2.
(h) ER T3.
(i ER TT.
(i) ER TS.
1.0
1.0
1.0
1.0
10
0.5
ACC
0.5
ACC
0.5
ACC
0.5
ACC
0.5
ER
ER
ER
ER
ER
OCAR
OCAR
OCAR
OCAR
OCAR
0.0
0.0
2500
5000
2500
5000
2500
5000
2500
5000
2500
5000
step
step
step
step
step
(k) TI.
T2.
(m) T3.
(n) T4.
(0) TS.
Figure 2D projections of the training trajectories ER and OCAR on Split MNIST (5 Tasks). The black stars highlight the task boundaries, the red star the final model. We also show learning curves each task separately.
plasticity measures. Then, on the same data, using the best hyperparameters, the results are averaged across 10 random seeds test fro training stability with different optimization paths.
E. Online CLEAR
Following the very recent work of (Yoo et al., 2024), we tested our method also on the Online CLEAR benchmark (Lin et al., 2021), a domain incremental learning scenario. This scenario is fundamentally different from the class-incremental, with the same classes that undergo some sort of evolution. The CL aspect is then less impactful, with much more forward/backwart transfer and less catastrophic forgetting due the similarity of the tasks in different domains. We followed the same settings in LPR paper (Yoo al., 2024), that are very similar the standard approach used for our main experiments, but with some differences: the use of the full ResNet18 instead of the Slim version and 10 gradient steps per batch. Unfortunately, we encountered some bugs when their exact code was used in ours, requiring some slight modifications. Not being able to ensure the exact same conditions, we rerun all the experiments, We decided to compare the baseline of Experience Replay, the very recent L.R approach, and our method. ER-ACE is tailored for inkkinncreeetal settings, so we avoid its use. In this scenario, we increase À not with the number of classes in the buffer (as in class incremental), but we increase it in time as new batches arrive. LPR was tested using 100 samples from the buffer to estimate its preconditioner.
F. Learning Trajectory on Split MNIST
The model is small feedforward network with two hidden layers of 100 units and ReLUU activations. The model is trained online with passes for each mini-batch with a small replay buffer of 100 elements, corresponding to 10 samples per class by the end of training.
To plot the 2D learning projections Figure 5, we follow procedure similar to (Mirzadeh et al., 2021).
17

---

Online Curvature-Aware Replay
We consider the 2D plane that intersects the model's initialization wg- the model after the first task wi. and the final model "iz. Many other possibilities
considered before this choice (e.g. using different tasks random directions), but they all and 5 wh. obtaining ù z [لग and
resulted in qualitatively similar plots. The coordinates system is obtained by orthonormalizing the directions
w w Te- tulll Given model w, its coordinates in the 2D space are the unique (xr v) such that w = zu + yö + wy- Notice that each method has different values for wi and w.z, and therefore different 2D planes and coordinate systems are chosen for each method. Since the coordinates are not meaningful and cannot be compared across plots, we do not show them in Figure 5.
u-cossu.t)e
G. Additional Figures
Accuracy over time on validation set
0.8
OCAR
0.7
Accuracy
0.3
0.2
Time index
Figure 6. Accuracy time the validation for Split-CIFAR100 experiment.
Accuracy over time validation
OCAR
0.7
Accuracy
0.4
0.3
4000
6000
Time Index
Figure Accuracy over time the validation set for Split- Tinglmageet experiment.
18

---

Online Curvature-Aware Replay
Accuracy over time validation set
OCAR
ER
Accuracy
0.4
0.2
500
1600
1500
2000
2500
3000
Time index
Figure 8. Accuracy over time on the validation set for Online CLEAR experiment.
Effective Step Size vs. Eigenvalues
104
10 10
1= 1-
Effective Step Size (A8))
a= 10
a- 1--10-1--00--
a- 10 3,-10
10
a= 101
r=10-
10-4
10-1
10 5, 1
10-6
10
10
10
102
Eigenvalue
Figure 9. Effective step size against the FIM eigenvalues. The step size in the directions with small eigenvalues is regularized via r, while large eigenvalues unaffected by it. The learning rate affects all the directions equally.
1.0
1.0
OCAR
ACC
ACC
0.5
0.5
ER
OCAR
0.0
2500
5000
2000
4000
6000
step
step
(a) Average of all tasks.
First Task.
Figure 10. Learning curves on the first task (10b) and the average accuracy for all tasks (10a).
19

---

Online Curvature-Aware Replay
Avg Final Accuracy
Task 1 Forgetting
Task 5 Accuracy (Plasticity)
1.0
1.0
10
77
.60
.10
.10
.10
.10
10
101
.89
.99
.50
.50
.50
.50
0.8
0.5
0.8
10a
84
.73
700
.66 .63
.61
109
27
10a
.88
.99
.99
-99
.99
98
0.6
0.6
10
.86
.84
.77
.75
.74
.75
10
-2
.39
442
-41
-41
.40
0.0
'00 1
.76
.98
.99
.99
.99
.99
0.4
0.4
10-2
.69
.88
.83
80
.78
.75
10-2
-.5
-21
-32
10-2
.10
.95
.98
.98
.99
.99
-0.5
0.2
0.2
10-3
.1
.69
84
82
.78
.75
10-3
23
-12
-22 -28
-37
4.40
10
.00
.15
.92
.95
.99
.99
0.0
-1.0
0.0
10-1
10-0100010000000
10-1
10
10
$10 410 P10 210
100
10-5
10-510-111
10-10-11000110
10
10-1
10"
Figure Grid search over P: (left) average accuracy, (middle) forgetting the first task, (right) plasticity measured the accuracy on the final task. Metrics are computed on the test stream the end of training.
20


Reviews:
[{"review_id": "eMK6ShbkMJ", "reviewer": "Reviewer_RSCF", "summary": "The paper addressed the online continual learning setting. The authors proposed Online Curvature-Aware Replay (OCAR), a replay-based method that leverages an approximated Fisher Information Matrix to help both the stability and plasticity. The authors further proposed specific adaptation for online continual learning. Experiments showed improved performance with the proposed method.\n\n## update after rebuttal\n\nThe authors have addressed all my questions. Therefore I retain my initial rating as Accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you very much for your review and your time. We are happy to answer all your questions:\n\n1) **Additional references**:  Thank you for the suggested papers. They are relevant, and we will add a discussion about them in section 2 in the final version.\n\n2) **Classifier Instability**: Our assumption about the stability of the representation is based on previous works that showed that the representation forgetting is usually small, and often comparable between methods that do not have any control for forgetting and methods that instead try to control it [1]. In our experiments, the *Probed Acc* metric is obtained by freezing the feature extractor after training, and retraining the classifier on all data, to show how good the representations were. The results show that the simple ER has a *Probed Acc* on par with iid training. Our method is even higher due to the improved optimization. This means that the majority of the forgetting happens on the classifier. From this, we can infer that using a pretrained backbone would get a final accuracy between our results of the *Acc* and the *Probed Acc*. \n\n3) **OCAR without Replay**: OCAR fundamentally requires Replay data to estimate the Fisher Matrix (FIM) required for the stability constraint. Unfortunately, removing the Replay data would hinder the scope of the method. On the other hand, without the Replay, the remaining effect would be to accelerate learning, as with standard second-order methods. Within this scope, the combination with [2] is possible and beneficial. In particular, during the Forgetting-Aware Minimization, the FIM matrix can be useful to penalize directions of high curvature, increasing the chances of finding a flatter minima, hence strengthening the relation with Sharpness Aware Minimization.\n\n4) **Computation Efficiency**: The single-task training times for CIFAR-100 are shown in a table in Appendix B, showing that our method is more efficient than other SOTA approaches. Given the importance for online scenarios, we will move the table into the main paper. \n\nWe thank you again for the suggestions and ideas. We hope to have satisfied all questions and concerns. We will be happy to discuss this further if any doubts remain. \n\n***\n\n**References**:\n[1] Davari, M., Asadi, N., Mudur, S., Aljundi, R. and Belilovsky, E., 2022. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16712-16721).\n[2] Kang, Z., Wang, L., Zhang, X. and Alahari, K., 2025. Advancing Prompt-Based Methods for Replay-Independent General Continual Learning. arXiv preprint arXiv:2503.00677.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Proposed a better method for replay-based online continual learning.", "essential_references_not_discussed": "There is a missing discussion with the latest works of online continual learning with pretrained models, such as [a, b].\n\n[a] Moon, Jun-Yeong, et al. \"Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[b] Zhiqi K, Wang L, Zhang X, et al. Advancing Prompt-Based Methods for Replay-Independent General Continual Learning[C]//The Thirteenth International Conference on Learning Representations. 2025.", "other_strengths_and_weaknesses": "Strengths:\n\n- The method is well motivated, well developed\n- The paper is easy to follow\n- The visualization in figures helps improve readability\n\n\nWeaknesses:\n\nThere are no serious weaknesses in the paper, I have only some remarks in the Question section.", "other_comments_or_suggestions": "No.", "questions_for_authors": "- About the assumption of instability\n\nIn lines 245-246, the authors claimed that the majority of instability happens on the classifier. I am wondering if using a frozen pretrained backbone can help clarify this point.\n\n- About the replay buffer\n\nSince most of the improvement is due to the optimization process, is it possible to design a version of OCAR that works without the replay buffer? For instance, the recent work [b] proposed a replay-independent approach to deal with online continual learning. Do you think OCAR can also boost its performance?\n\n[b] Zhiqi K, Wang L, Zhang X, et al. Advancing Prompt-Based Methods for Replay-Independent General Continual Learning[C]//The Thirteenth International Conference on Learning Representations. 2025.\n\n- Computation efficiency\n\nIt would also be interesting to see the computation efficiency of the proposed method, as it is important for online scenarios.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IP16qPW7Je", "reviewer": "Reviewer_BrQe", "summary": "This paper formalizes replay-based Online Continual Learning (OCL) as a second-order online joint optimization with explicit KL-divergence constraints on replay data. It proposes Online Curvature-Aware Replay (OCAR) to solve the problem: a method that leverages second-order information of the loss using a K-FAC approximation of the Fisher Information Matrix (FIM) to precondition the gradient. Extensive experiments are conducted.\n\n## update after rebuttal\n\nBased on the response of authors, I will keep my scores.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you very much for your review. We hope to be able to solve your doubts:\n\n1) **OL vs OCL**: We agree that a deeper comparison between Online Learning and Online Continual Learning can be beneficial. In Section 3, we will add a paragraph explaining the main points: both OL and OCL requires the model to update with full access only to the last seen data-points, without the whole training dataset; both OL and OCL aim to minimize the cumulative loss (regret) experienced over the whole stream, without knowing the length of it *a priori*; OCL, unlike OL, aims to tackle forgetting, building models that are robust also when tested on past data of the stream; OCL, unlike OL, equips the model with additional information (for example a Replay Buffer) to stabilize the model on old data. \n\n2) **Time Cost**: The table with the time cost of the different models is presented in Appendix B. Given its importance, we will move it to the main paper in the final version. Thanks to its approximations and the efficient use of K-FAC, our method is more efficient than other SOTA approaches tested on the same setting.  \n\nThank you again for your review and your suggestions. We hope we tackled all your doubts. If that is not the case, we are happy to engage more.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\na. This paper is well-written and easy to follow.\n\nb. I appreciate the extensive experiments.\n\nc. I think the idea is novel.\n\nWeaknesses:\n\na. I am a little bit worried about the time cost of this proposed method.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "a. In my view, there should be more discussion about the difference and similarity between online learning and online continual learning in Sections 2 and 3.\n\nb. It is better to show the time cost of the proposed method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HY6hXarZyL", "reviewer": "Reviewer_cTJP", "summary": "This paper proposes Online Curvature-Aware Replay (OCAR) that leverages second-order information of the loss\nusing a K-FAC approximation of the Fisher Information Matrix (FIM) to precondition the gradient in OCL.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we thank you for your review and your time. We try to address all your concerns:\n\n1) **Other gradient-altering strategies**: In our experiments, we tested A-GEM and LPR, both examples of gradient projection methods. LPR (current SOTA) is a variant of preconditioned SGD, preconditioning the gradient to penalize update directions that interfere with replay-data activations. A-GEM instead is a \"pure\" projection method: when the gradient violates the constraint $\\tilde{g}^\\top g_{ref} \\geq 0$, hence interfering with a previous reference gradient, it is projected via $\\tilde{g} = g - \\frac{g^\\top g_{ref}}{g_{ref}^\\top g_{ref}}$. Our method is different from both these approaches: we work on the parameter space, using the Fisher Information (FIM) as a preconditioner, penalizing directions that would interfere with replay-data predictions and accelerating other directions, instead of projecting the gradient only when interfering. \nWe will explain these differences better in the final version of the paper, and we will expand the related work section, which already briefly mentions some works on gradient projection, with additional references, such as [1-5].\nIn general, our goals and those of most gradient projection methods are different. Gradient projection is often used to prevent forgetting in a replay-free setting (with some notable exceptions like LPR and A-GEM), often requiring expensive SVDs or inner optimizations, making them harder to apply to OCL (see [7]).\n\n2) **First Order + Regularization**: In our paper, we compared with some methods that indirectly alter the gradient with a regularization term on the loss. One notable example is EWC, which adds a penalization for the change of parameters important for previous tasks. Unlike our approach, it uses the FIM in the penalization and not as a preconditioner. In section 6, we compared ourselves directly to EWC, with further explanations in Appendix C. We will expand the section, explaining how our method can achieve both stability and plasticity, while EWC can only slow some directions. Moreover, in our experiments, we also compared with other methods that use regularized losses, like DER and ER-ACE. \n\n3) **OCAR worse on Acc**: We agree. We should expand the explanation of the metrics. The *Acc* measures only the final accuracy at the end of the stream, a single point that can be affected by noise and by the length of the stream. From an OCL perspective, the *AAA* metric is more meaningful, measuring the average performance over the entire stream (more similar to the regret in Online Learning). Our method always shows better stability (*Worst Case Acc*) and overall performance (*AAA*) than others, but it is not the best in *Acc* for TinyImagenet. We believe it has something to do with different plasticities in different parts of the stream, with OCAR more efficient at the beginning and, for example, ER-ACE better in the final part. Corroborating this, the combination of OCAR and ER-ACE is dominant on all metrics. \n\n4) **$2^{nd}$-order methods accuracy**: In fact, the opposite happens. While the curvature information penalizes the optimization along the directions important for previous tasks, it also speeds up learning in the other directions, something that most other methods cannot do. In our experiments, the probed accuracy shows that OCAR is learning better representations than the other methods. This improved optimization is based on the information geometry theory, which shows how the FIM can help the optimizer to converge faster to better minima [6]. \n\nAgain, we thank you for your review and suggestions. We hope we addressed all your concerns. We are happy to engage more if any doubt persists. \n\n***\n**References**:\n[1] Yichen Wu, Hong Wang, Peilin Zhao, Yefeng Zheng, Ying Wei, Long-Kai Huang: Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization. ICML 2024\n[2] Saha, Gobinda, and Kaushik Roy. \"Continual learning with scaled gradient projection.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 8. 2023.\n[3] Lin, Sen, et al. \"Trgp: Trust region gradient projection for continual learning.\" arXiv preprint arXiv:2202.02931 (2022).\n[4] Yang, Enneng, et al. \"Data augmented flatness-aware gradient projection for continual learning.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[5] Zhao, Zhen, et al. \"Rethinking gradient projection continual learning: Stability/plasticity feature space decoupling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[6] Amari, S.I., 2016. Information geometry and its applications (Vol. 194). Springer.\n[7] Yoo, J., Liu, Y., Wood, F. and Pleiss, G., 2024. Layerwise proximal replay: A proximal point method for online continual learning. arXiv preprint arXiv:2402.09542.", "claims_and_evidence": "Overall the claimed information are well supported. By using second-order methods, it can alleviate the CF problem by enlarging the ”sight” of optimizer with the local variation of the variation.", "methods_and_evaluation_criteria": "I think the evaluation criteria is reasonable", "theoretical_claims": "I think the theoretical claims are correct", "experimental_designs_or_analyses": "1. Missing discussions and comparisons with other OCL methods that aiming at alleviate the CF problem from gradient aspect. \n\n2. Missing analysis and comparisons of 'second order method' and 'first-order + gradient regularization strategy'", "relation_to_broader_scientific_literature": "I think it misses comparisons and discussions with other OCL methods that focus on modifying the gradient to alleviate the CF problem.", "essential_references_not_discussed": "Missing references to methods that focuses on modifying the gradient to alleviate the CF problem.", "other_strengths_and_weaknesses": "1. Missing comparisons with other graident precondition methods in OCL", "other_comments_or_suggestions": "1. From Table 1, sometimes the proposed method performns worse than existing methods, I suggest provide more discussions on why this happens.\n2. I understand second-order methods can alleviate the CF, will it degrade the learning accuracy?", "questions_for_authors": "I understand changing first-order methods to second-order methods can be effective in some cases, however, I am wondering the comparsions with first-order methods+some gradient altering strategies.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kcenCKhKtL", "reviewer": "Reviewer_KWqw", "summary": "The paper combines experience replay with a second-order optimizer derived from a KL constraint to tackle online continual learning. The paper improves SOTA results on three vision-based online continual learning benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you for your insightful review. We are happy to address your comments:\n\n1) **Line 155**: We consider our stream to be arbitrary (we can have a different task for each step). If the objective has changed (task boundary), and we are at a different point in the space, the previous gradient history is not beneficial (see [1] for OL setting). On the other hand, when the length of the stream is not known, learning rate decay diminishes plasticity in all directions, stopping the learning at the limit. We will clarify these points in the paper.\nThe idea of casting the first order optimization as a sequence of local optimizations can be found in [2], Section 7. We will refer to this. \n\n2) **Line 170**: The Hessian matrix increases the convergence speed as in a Newton-like step, improving fast-adaptation w.r.t. first-order methods. Here we are stretching the definition of fast-adaptation to include also buffer elements that are not fully learned (note that the stability condition is defined w.r.t. the model and not the data). We will rewrite the sentence to be more transparent about our \"stretch\".\n\n3) **Line 133**: Unfortunately, there is a typo on the Taylor expansion, missing the $\\frac{1}{2}$ term. We will correct it. \n\n4) **Confusing sentence**: You are right! The sentence about the first-order solution is confusing because we misplaced the second-order Taylor expansion in the first-order section. We will correct it. \n\n5) **Notation**: You are right again. We tried to lighten the notation as much as possible, but it is now confusing. We will distinguish the function symbol from its resulting object (e.g. $\\mathcal{H}$ and $\\mathbf{H}$ for the Hessian; $g$ and $\\nabla$ for the gradient). On the other hand, the $\\hat{KL}$ function is defined in line 110 as the sample estimation of the real KL computed on probability distributions.\n\n6) **Tikhonov damping**: We happily welcome the suggestion of renaming the \"Tikhonov regularization\" to \"Tikhonov damping\".\n\n7) **Different OCAR derivation**:  This is an interesting idea. Our work aimed to propose a standalone approach leveraging the natural gradient (NG) theory, which shows how the KL generates the FIM as Riemannian metric. To connect to NG, we needed to use the KL as the objective function to derive the FIM matrix that is shown to accelerate learning (see [3]). With a general loss function, we would lose this connection. Moreover, a non-KL-derived loss can be tricky. In our experiments, OCAR-DER++ was very unstable, failing the optimization on Imagenet. We believe this idea can be a valuable future research direction. \n\n8) **Hessian of KL**: On this point, we have to disagree. The Hessian of $KL(p_{\\theta'} || p_\\theta)$ is equal to the FIM only when evaluated at $\\theta' = \\theta$, near the minimum of the KL (see [4], [3]). In our case, this true only for the constraint $\\hat{KL}(f_{w_{t-1}}(x_{B_t})  ||  f_{w_t}(x_{B_t}))$, but false for $\\hat{KL}(y_{N_t}  ||  f_{w_t}(x_{N_t}))$ and $\\hat{KL}(y_{B_t}  ||  f_{w_t}(x_{B_t})$. Our approximation, to compute a single FIM instead of a FIM and an Hessian, is to assume the model is \"the correct\" model for both new and buffer data.\n\n9) **Additional passes**: We need to compute only a single FIM with a weight $(1+\\lambda)$ on the buffer data. Using a log-likelihood loss, the FIM can be computed in closed form, requiring only an additional forward pass for the entire batch, and a backward pass for each output, when an update is needed. \n\n10) **Scaling**: OCAR is efficient w.r.t. other SOTA approaches (see training times in Appendix B). The most intensive operation is the K-FAC computation and inversion. For fully-connected layers, K-FAC stores and inverts, for each layer, a factor $n_{in} \\times n_{in}$ and a factor $n_{out} \\times n_{out}$. [5] shows that training large models with K-FAC is possible. Moreover, one of our future directions is to explore the use of E-KFAC to make the method more efficient. \n\n\nYour review has been invaluable, and we are profoundly grateful for this. We hope we addressed all the concerns. We are happy to engage more if you like so!\n\n***\n\n**References**:\n[1] Yuan, K., Ying, B. and Sayed, A.H., 2016. On the influence of momentum acceleration on online learning. Journal of Machine Learning Research, 17(192), pp.1-66.\n[2] Martens, J., 2020. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146), pp.1-76.\n[3] Amari, S.I., 2016. Information geometry and its applications (Vol. 194). Springer.\n[4] Ollivier, Y., Arnold, L., Auger, A. and Hansen, N., 2017. Information-geometric optimization algorithms: A unifying picture via invariance principles. Journal of Machine Learning Research, 18(18), pp.1-65.\n[5] Pauloski, J.G., Huang, L., Xu, W., Chard, K., Foster, I.T. and Zhang, Z., 2022. Deep neural network training with distributed k-fac. IEEE Transactions on Parallel and Distributed Systems, 33(12), pp.3616-3627.", "claims_and_evidence": "The main claims on OCAR improving online continual learning optimization and performance are supported by convincing evidence. They report improved scores on standard benchmarks, have toy experiments to contrast learning trajectories of OCAR vs baselines, and investigate stability-plasticity tradeoff of OCAR hyperparameters.\n\nHowever, the paper would significantly benefit from expanding on some of its methodology derivation-related claims in Section 4.\n- Line 155: Why does a sequence of local optimization problems solved at each step prevent the use of \"global approaches\" like learning rate decay and momentum? In addition, it would be good to put references on why first-order optimization can be cast as a sequence of local optimization problems.\n- Line 170: Where is fast adaptation constraint (D4) achieved?\n- Line 133: How does the 2nd order KL divergence Taylor approximation formula hold?\n\nIn addition, the sentence \"after approximation, the solution for problem 1 is $\\delta_t^* = -\\tfrac{1}{\\lambda}(\\nabla_{N_t} + \\nabla_{B_t})$\" is very confusingly introduced, since the previous equation gives a second-order Taylor expansion of the KL divergence whereas the sentence refers to the parameter update solution of the first-order Taylor expansion of the KL divergence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experimental designs and analyses are valid.", "relation_to_broader_scientific_literature": "The paper investigates alternative optimization methods aside from commonly used SGD to improve online continual learning. Their method is highly related to LPR [1], which uses a block diagonal preconditioner to improve online continual learning accuracy and optimization efficiency.\n\n[1] Yoo, J., Liu, Y., Wood, F., & Pleiss, G. (2024). Layerwise proximal replay: A proximal point method for online continual learning. arXiv preprint arXiv:2402.09542.", "essential_references_not_discussed": "To my knowledge, relevant references are discussed.", "other_strengths_and_weaknesses": "The paper's strength is its originality since not many works investigate how to better optimize neural networks during online continual learning. In addition, the use of natural gradient descent-like approaches, as the paper states, draws interesting connections with information geometry (though the authors do not expand on this much).\n\nThe paper's greatest weakness is its lack of clarity arising from highly unintuitive notations and derivations. For example, $\\nabla, \\mathbf{H}$ are used as both operators and vectors/matrices. In addition, $\\hat{KL}$, which is suggestive of KL divergence that accepts probability distributions as inputs, accepts random variables as inputs. For the SGD derivation for experience replay, the paper gives the second-order Taylor expansion of the inner optimization objective and provides a parameter update solution for the first-order Taylor expansion of the inner optimization objective. Adjusting issues like this would make the paper much stronger.", "other_comments_or_suggestions": "I suggest renaming \"Tikhonov regularization\" in the paper to \"Tikhonov damping\" as done by Martens and Sutskever [1] to minimize confusion from the fact that the paper's Tikhonov regularization applies to inner level optimization objective of Eq 1, not to the outer level experience replay optimization objective.\n\nIn addition,\n\n[1] Martens, J., & Sutskever, I. (2012). Training deep and recurrent networks with hessian-free optimization. In Neural Networks: Tricks of the Trade: Second Edition (pp. 479-535). Berlin, Heidelberg: Springer Berlin Heidelberg.", "questions_for_authors": "- Have you considered deriving OCAR parameter update purely from $KL$-constrained inner optimization problem of a first-order Taylor expanded objective? This seems possible and would be a much simpler way of deriving OCAR update that does not require strictly working with $KL$ objective function. This would also let OCAR-DER++ and OCAR-ACE an instantiation of the paper's framework, since OCAR would no longer be constrained to a specific form of experience replay loss.\n\n- In addition, the Hessian of KL divergence is simply a Fisher information matrix, so there's no reason to state \"we approximate the two Hessian matrices of our solution with the FIM, greatly simplifying the computations\".\n\n- For computing the Fisher information matrix, how many additional forward/backward passes does OCAR have to make?\n\n- How well would OCAR scale to larger models, compute and memory wise?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper formalizes replay-based OCL as a second-order online joint optimization with explicit KL-divergence constraints on replay data. And it proposes Online Curvature-Aware Replay (OCAR) to solve the problem: a method that leverages second-order information of the loss using a K-FAC approximation of the Fisher Information Matrix (FIM) to precondition the gradient. The proposed method is novel. The writing of this paper is clear. Thus, I recommend accepting this paper. However, the authors should modify this paper carefully according to the given comments.
Decision: Accept (poster)