Meta Continual Learning Revisited: Implicitly Enhancing Online Hessian Approximation via Variance Reduction
Authors: Yichen Wu
Long-Kai Huang
Renzhen Wang
Deyu Meng
Ying Wei
Venue: ICLR 2024 oral (2024)
Keywords: Continual Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We provide a new perspective to Meta-continual learning and propose a Variance Reduction Meta-CL based on the novel understanding.

Abstract: Regularization-based methods have so far been among the *de facto* choices for continual learning. Recent theoretical studies have revealed that these methods all boil down to relying on the Hessian matrix approximation of model weights. 
However, these methods suffer from suboptimal trade-offs between knowledge transfer and forgetting due to fixed and unchanging Hessian estimations during training.
Another seemingly parallel strand of Meta-Continual Learning (Meta-CL) algorithms enforces alignment between gradients of previous tasks and that of the current task. 
In this work we revisit Meta-CL and for the first time bridge it with regularization-based methods. Concretely, Meta-CL implicitly approximates Hessian in an online manner, which enjoys the benefits of timely adaptation but meantime suffers from high variance induced by random memory buffer sampling. 
We are thus highly motivated to combine the best of both worlds, through the proposal of Variance Reduced Meta-CL (VR-MCL) to achieve both timely and accurate Hessian approximation.
Through comprehensive experiments across three datasets and various settings, we consistently observe that VR-MCL outperforms other SOTA methods, which further validates the effectiveness of VR-MCL.

Published as a conference paper at ICLR 2024
META CONTINUAL LEARNING REVISITED: IMPLIC- ITLY ENHANCING ONLINE HESSIAN APPROXIMATION VIA VARIANCE REDUCTION
Yichen Wu Long-Kai Huang?; Renzhen Wang', Deyu Meng Ying Weil.5t
City University of Hong Kong, "Tencent Lab, 'Xi'an Jiaotong University. Paahoo Laboratory (Huangpu), 'Nanyang Technological University
ABSTRACT
Regularization-basee methods have so far been among the de facto choices for continual learning. Recent theoretical studies have revealed that these methods all boil down to relying on the Hessian matrix approximation of model weights. However, these methods suffer from suboptimal trade-offs between knowledge transfer and forgetting due to fixed and unchanging Hessian estimations during training. Another seemingly parallel strand of Meta-Continual Learning (Meta- CL) algorithms enforees alignment between gradients of previous tasks and that of the current task. In this work we revisit Meta-CL and for the first time bridge it with regularization-be based methods. Concretely, Meta-CL. implicitly approximates Hessian an online manner, which enjoys the benefits of timely adaptation but meantime suffers from high variance induced by random memory buffer sampling. We are thus highly motivated to combine the best of both worlds, through the pro- posal of Variance Reduced Meta-CL (VR-MCL) to achieve both timely and ac- curate Hessian approximation. Through comprehensive experiments across three datasets and various settings, we consistently observe that VR-MCL outperforms other SOTA methods, which further validates the effectiveness of VR-MCI!!
INTRODUCTION
Continual learning (CL) is promising approach to constructing learning systems that can contin- uously process information streams, adapt to changing environments, and acquire new knowledge while consolidating and retaining previously learned knowledge (Parisi et al., 2019). In contrast to traditional supervised learning, which trains on independent and identically distributed (i.i.d) samples, continual learning involves models trained on non-stationary data distributions, where dif- ferent classification tasks are presented sequentially. The violation of the i.i.d. assumption in CL can lead to catastrophic forgetting, causing a significant drop in test performance on previously learned tasks (French, 1999; McClelland et al., 1995; McCloskey & Cohen, 1989).
In recent years, numerous methods in the field of CL have been proposed with the aim of tackling the catastrophic forgetting problem. Regularization-batt et al., 2018; Huszár, 2017: Zenke et al., 2017), being an important branch of these methods, aim to discern and retain the important parameter weights associated with prior tasks as sustain their performance. Specifically, they assess the importance of different parameter weights through the second-order Hessian of previous tasks computed at the end of each task training. The differences among these regularization-bate mating the Hessian. For example, EWC (Kirkpatrick et al., 2017) and Online-EWC (Huszár, 2017) employ the diagonal Fisher information matrix, whereas Kronecker factored Laplace approximation (KFLA) (Ritter al., 2018) utilizes more accurate Kronecker factorization approximation, con- sidering off-diagonal elements. Similarly, Intelligent Synapses (IS) (Zenke et al., 2017) adopts an approximate diagonal matrix, but its elements take into account the importance of a task throughout the entire training trajectory. Despite these methods striving to improve the Hessian approximation
methods (Kirkpatrick et al., 2017; Ritter
methods primarily reside in their distinct approaches to approxi-
Parr of the work done when the author interned in Tencent AI Lab (wuyichen.am977 gmail.com). 1 'Corresoondigg author: Long-Kai Kai Huang (hlongkai @ gmail.com) and Ying Wei (ying.wei@ ntu.cdu.sg)
"ooee is available https://githuh.com wow.e//occo//o/////e..

---

Published as a conference paper at ICLR 2024
to preserve important weights for previous tasks, their estimated Hessian matrices of previous tasks remain unchanged and fail to update during subsequent training. As the model parameters gradually deviate from the point at which the Hessian initially computed, the unaltered estimated Hessian progressively loses accuracy due to the increasing truncation error of the Taylor expansion.
Different from regularization-basee based methods that explicitly approximate the second-order Hessian at the end of training for each task, as will be detailed in Sec. 3, Cecta-Continual Learning (Meta-CI) CL) methods utilize the second-order Hessian information implicitly through the hypergradient obtained via bi-level optimization (Riemer et al., 2019; Gupta et al., 2020; Von Oswald et al., 2021). The computation of hypergradient involves a batch of samples drawn from the memory buffer, which stores examples from previous tasks, enabling Meta- CL to leverage up-to-date second-orde, infor- mation from previous tasks. However, it is noteworthy that the utilization of samples drawn from the memory buffer may not offer a complete representation of previous tasks, and may even lack data pertaining to certain tasks. This can result the presence of erroneous information the Hes- sian estimation. As consequence, there may be considerable variation in the implicitly estimated Hessian, ultimately leading to decline in performance for previous tasks.
To leverage the benefits of Meta-CL, specifically the timely updating of second-order Hessian infor- mation, while mitigating the negative impact of variation, we propose a momentum moomentum-based Variance- Reduced Meta-CL (VR-MCL). Through our theoretical analysis, demonstrate that the integra- tion of the momentum- hased variance-reduced technique into Meta-CL. is equivalent to imposing a penalty its online estimated Hessian. This penalty effectively suppresses excessive model up- dates, thereby better preserving crucial weights for previous tasks and their performance. Our main contributions are summarized as follows:
We introduce a new perspective on Meta-CL by characterizing it as a technique that approximates the Hessian matrix, thereby establishing a connection between Meta-CL. and regularized methods. To address the issue of high variance in Meta-CI., we propose Sariance-Reducee Meta-CI. (VR- MCL), which incorporates hypergradient variance reduction technique into Meta-CL.
Theoretically, we demonstrate that our proposed momentum-based hypergradient variance reduc- tion technique is equivalent the inclusion of effective penalty term within the implicitly estimated Hessian. Additionally, we provide the regret bound of VR-MCL to further explain its superior performance in the context of continual learning.
We conduct comprehensive experiments covering three datasets and multiple continual learning settings, providing empirical evidence for the effectiveness of the proposed algorithm.
2 THE UNIFIED FRAMEWORK OF REGULARIZATIONNBBANN
CL METHODS
In this section, we will first derive the unified iterative updating framework used by regularization- based algorithms following the analysis of Yin et al. (2020) and then show the impact of Hessian approximation accuracy on the performance of these algorithms.
Analysis of Regularizationn-asee methods. Continual learning considers a sequence of N tasks [T), T2 ..... ".,.,,,,,,,,.... where the i-th task consists of N samples, i.e., T (X,Y") ( v 1N Here, T represents the current training task, Thas refers to all tasks the model have seen so far, and (²) denotes the empirical risk of the parameter 9 on T'. The objective of CL to learn a model with parameters 0 that minimizes the empirical risk of all seen tasks 1 i.e., 1 (')). (0)). However, in the context of regularization-basee CL, we have no access to the samples of previously learned tasks 11:-111, The model parameters 9 cannot be directly optimized to minimize the cor- responding empirical risks, i.e., El= (0). Therefore, regularized CL methods choose to approx- imate the empirical risk of previous tasks during training on T (Yin et al., 2020). Concretely, in methods is to minimize 1 (CCrrxx+ + L²), where 2T (0) is an approximation of L (0) derived through Taylor
the simplest case where j=2, when training on T?, the objective of regularization-
V (0') (0 Ø²). Here, 0 represents the model parameters at the end training of T1 For a more general setting where j > 2,
expansion at 01 as L.rox C²(0') + (0- @*) (O') + [0
we can approximate the empirical loss of the previous (j 1) tasks by,
C (0)
(0 (0-
0 VoL (0
(0- 0' H (0 0¹)
(b)
2

---

Published as a conference paper at ICLR 2024
Table 1: Summary of continual learning methods under the unified iterative update rule with various instantiations of Hessian approximation. Here F and F are the full and diagonal Fisher information matrices, respectively. HD denotes diagonal approximation of the Hessian matrix, and HM and HVR are the online Hessian estimated on previous tasks and its regularized version, respectively.
Method
Hessian Approxination
Iterative Update Rule
Off Off-diagonal elements
Online update
Penalty
EWC/ On-EWC
X
x
0 = 0 a x Pb) (V,C
KFLA
X
x
0 = 0 - a (EE P1) V.C100
x
= a Hb) (V.C(0)
La-MAML
x 0:
VR-MCL
V 0: (V
where O and H = (0') denote the model parameter at the end of training T and the Hessian matrix, respectively. For the concrete analysis of 43-r(0) (0) in Eqn. (1), the first term (a) is not related to 0 and thus can be discarded. Furthermore, since the training process at the end of each task is usually converged, the gradient (Q.) is near zero and the term (b) can be ignored in practice. Therefore, the empirical loss approximation 2 (0) is almost equivalent to approximating the Hes- sian matrix in item (c). Based on this approximation and the analysis, we can derive the iterative update rule of regularized methods in Proposition 1. (See Appendix A.1 for proof.)
Proposition 1. In regularization-basee continual learning, if the model parameter is searched within the neighborhood set U with N² = (0:: 8*) < 6'), then the iterative update rule of approximately is,
θ m θ o
+
H²
H H (8).
(2)
The Effect of Hessian in Regularization-basee CL. Let H = Σラ
si=1 H' denote the Hessian part in Eqn. (2). This update rule implies that the Hessian H plays a crucial role in the optimization process. Specifically, by multiplying the inverse of H with the gradient VoL (f), the gradient is suppressed along the large curvature directions, i.e., the directions along the eigenvectors corresponding to large eigenvalues of H, and amplified along the small curvature directions. In regularization based CL algorithms, the large curvature directions of the Hessian align with the important weights storing the knowledge of the previous tasks, which suppresses the update scale to reduce forgetting, while the small curvature directions facilitate updating the model to fit the current task.
Regularization-basse CL in a Unified Framework. Given the critical role of the Hessian in model updates, various regularization-batt based methods primarily focus on enhancing their approximation of the Hessian. By integrating Proposition 1, encapsulate existing regularization-ben based CL methods within a unified iterative update framework as summarized them in Table 1. Concretely, EWC (Kirk- patrick al., 2017) and On-EWC (Huszár, 2017) utilize the diagonal Fisher information matrix F; to approximate the H' in Eqn. (2). In contrast, Intelligent Synapses (IS) (Zenke et al., 2017) em- ploys diagonal matrix H. with elements signifying the importance of a task throughout the entire training trajectory. To enhance the approximation of the Hessian, Kronecker factorrdd Laplace ap- proximation (KFLA) (Ritter et al., 2018) opts to use the Kronecker factorization approximation F', taking into account off-diagonal elements, to approximate H'
3 REVISITING META-CL FROM A HESSIAN APPROXIMATION PERSPECTIVE
In this section, revisit the Meta-CL from Hessian approximation perspective. Specifically, while regularization-batt based methods explicitly approximate the second-order Hessian, Meta-CL uti- lizes second-order information through the computation of hypergradient. Consequently, it can also be scrutinized within the iterative update framework presented in Eqn. (2).
Meta-Continual Learning (Meta-CL). As of the earliest works of Meta-CL. Riemer et al. (2019) propose a meta-learning formulation for minimizing the empirical risk of all seen tasks 70s)) In this approach, the data in Tils- are partially accessible via memory buffer M, which stores
3

---

Published as a conference paper at ICLR 2024
Inner Loop
Ooop
Inputs
Weights
Outer Gradient
(x) (x(0,
bb
M UT
ming (2:1 (0
Bb
PD)
DbK
Bdb
0
Figure I: Iterative update process of Meta-CL for the b-th iteration. The notation drawn from .M refers to the memory buffer and 80, is the update gradient.
Tike means samples
fs
data from previous tasks. Concretely, the optimization problem is formulated as,
ming L (0(33) s.t. (kk) (0;
inner steps
where - (0 (0); Too)) U(O(s)) ((a) -00) -Ve (0)) 00)10=-
the (())) characterizes the empirical risk of 8ix) on all j tasks 7P which in practice is estimated T' and the data in memory buffer M,, f()) denotes the parameters of the k-th inner step of learning for task T and (.) means one-step SGD update in the inner loop optimization.
We illustrate the iterative update process for the model parameters for the b-th itcration, denoted by 0g. in Fig. 1. In the inner loop, the model is initialized with doço) 0s and iteratively updated via K steps of SGD updates, where Ti is the data randomly sampled from To in the k-th step. Following the acquisition of (uck), the outer loop, we randomly sample data 6, from T and the memory buffer M. Then we compute the loss ch> (Du(()) and obtain the hypergradient w.r.t. as Ke, = (0((0()) ( /80, We finally update 6, using the hypergradient. It should be noted that dif- ferent Meta-CL CL methods typically employ different approximations of the update gradient ga; For instance, MER (Riemer al., 2019) utilizes a first-order approximation, while La-MAML (Gupta et al., 2020), following MAML (Finn et al., 2017), directly computes the hypergradient.
Revisiting from the Hessian approximation perspective. To solve the bi-level optimization prob- lem shown in Eqn. (3), we need to compute the hypergradient of cl2:5l (0(K)) w.r.l. 0, i.e.
(ch:)((0)))
act: (0()) (0(K)
(0(K) where the Taylor approximation of the first term around 0 is,
00
(0 Vo(x (B(r)) xo (
(0(x)- - 0)²² (0(())) 0).
(0(K)
Note that H3 Vi (o))(()) and T denote the Hessian matrix and the third-order symmetric tensor, respectively, and @ represents the Kronecker product. Through the above approximation and assuming single inner step in optimizing T (i.e., K=1) for simplicity,
have Proposition 2. Proposition 2. For Meta-CL CL with single inner step adaption, suppose that ((K) is located in the c-neighborhood (6 é) of the optimal model parameter 0* =argmin C. (l(c))) f. is p-smooth, and 3 < V (0)) where 8 is small number. Let a = 52, then the iterative update rule approximately is.
= ar (H M -V
We defer the proof and analysis with K K-step (K > 1) adaption to Appendix A.2. Proposition 2 demonstrates that the use of hypergradient by Meta-CI. is equivalent implicitly employing the Hessian (H²,) and adheres to the unified iterative update rule of regularization-bast
techniques. By comparing Proposition and Proposition 1, we can deduce that Meta-CL utilizes the samples drawn from M to implicitly compute Hessian HA in Eqn (2). In contrast to regularized methods that estimate and fix each individual H', Meta-CL. implicitly computes one Hessian H enabling it to have the up-to-date second-orde information.
to approximate (H' H² + H²-1)
Despite the advantages of Meta-CL in utilizing up-to-date Hessian information via M, it can also introduce erroneous information. For example, the samples drawn from M may not adequately
4

---

Published as a conference paper at ICLR 2024
represent previous tasks may lack instances from specific tasks. This inadequacy may lead the estimated Hessian mistakenly having a small curvature direction along important weights these previous tasks, which in turn fails to suppress updates along this direction. Consequently, this could result in the loss of previous knowledge from those tasks and lead to forgetting.
VARIANCE REDUCTION ON META-CL
To tackle the issue of high variance in Meta- CL caused by the random sampling strategy in the mem- ory buffer, we propose Variance- in the hypergradient of Meta-CL. Our theoretical analysis demonstrates that this variance reduction in the hypergradient is equivalent to the addition of regularization to the implicitly estimated Hes- sian matrix, resulting in a decrease the variance the Hessian matrix and an improvement in its estimation accuracy. Morcover, also provide theoretical guarantee of the regret bound.
Meta-CL. (VR-MCL) that effectively reduces the variance
4.1 VARIANCE- REDUCED META-CL (VR-MCL)
VR-MCL aims to reduce the variance of its implicitly estimated Hessian by diminishing the variance of its hypergradient. While there various variance reduction approaches, such as SAG (Gower et al., 2020), SAGA (Defazio et al., 2014), and SVRG (Johnson & Zhang, 2013) for convex opti- mization as well as the non-convex variants (Allen-Zhu & Hazan, 2016; Nguyen et al., 2017; Fang et al., 2018), directly applying these methods to online continual learning is challenging due to the requirement of computing the full-batch gradient for all samples in task. However, in the online continual learning setting, samples from a task are received in mani-batches, and the full sample set remains inaccessible, which poses the challenge to the implementation of these techniques.
On this account, we propose momentum-based rediueee-reduedd Meta-Continual Learning (Meta- CL) method, inspired by previous works Cutkosky & Orabona (2019); Khanduri et al. (2021). Instead of computing the full-batch gradient which is im- practical for online CL, our method, VR-MCL, re- tains the parameters from the previous iterative step. The b-th iteration of VR-MCL is illustrated in Fig. 2, where _e,_; represents the momentum component and cs and K denote the hypergradients on pre- vious 6/_ and fr, respectively, computed using data Eb. In VR-MCL, the final update gradient with re- duced variance is shown in Eqn. (4), where rep- resents the momentum ratio. For clarity, present the pseudo-codes of the algorithm in Appendix E.
Imvor Loop
Outer Loup
(13
29
Imputs
Paca
Racey
Wiights
Eo
OuterGradient
tig Si + relo
Figure 2:
The iterative update process of VR- MCL for the 6-th iteration, where g5, is the final update gradient to obtain the 0s+1-
g g r(88 )
To comprehend why Eqn. (4) delivers smaller gradient variance, we initially examine A ge; Go,- This term measures the error incurred when we use $0; as the gradient instead of the truc, but unknown full-batch gradient direction Go,- Through demonstrating that the term E||| || decreases over time, we can show the effectiveness of the VR-MCL in variance reduction Cutkosky & Orabona (2019). Substituting Eqn. (4) into A,, we can obtain the following expression,
(5) The second term (1-r) Go.) can be modulated by adjusting the ratio of r, and the third term - (Ge. - Go. ,) is expected in the order of O 0p- I|] ((x) g8; [I) given the f- smooth loss function L. Thus, we have | = +||A++ where S is number that influenced by r and the learning rate a. As | keeps decreasing until reaches (/11r), choosing appropriate values of and that render 5/(1 -r) as small as possible ensures variance reduction
A r + +11 - (B Ge.) r - g - (Ge. Ge...)).
expect.
Remark. As previously analyzed, most variance reduction methods, such as SAG (Gower et al., 2020), SVRG (Johnson & Zhang, 2013) and their nonconvex version, not applicable in online CL settings. Therefore, we design the momentum- based variance reduction technique specifically for Meta-CI.. We also discuss other potentially viable variance reduction methods in Sec.
5

---

Published as a conference paper at ICLR 2024
4.2 THEORETICAL ANALYSIS OF VR-MCL.
In this subsection, we will demonstrate that the reduction of hypergradient variance by VR-MCL is equivalent imposing penalty on its implicitly estimated Hessian. As result, it effectively miti- gates the adverse effects of significant Hessian variation, as detailed in Proposition 3. Additionally, we also provide the regret bound of VR-MCL in Theorem
further explain its effectiveness.
Proposition 3. Assume that the batch size for inner step adaptation is sufficiently large and the b-th step trains on TV. Let H,, = =V L(0b; 6b) = [bi, hi...... hy denote the Hessian at Og calculated
on tb where bg is the d-th column vector of Ha,, Similarly, H H -hb--1"'. -,hp- denotes the Hessian for the momentum term the 1b-1)-th iteration. If (E ) dahig_)) 1) 2.2 holds for any da o d= (1,2, ,D,, then we have the following iterative update rule for VR-MCL,
0 =e 0 agin
0 a
where (Har) (H, (M 1 JH 7*) with Has, denoting the Hessian at 0-11 calcu- M
+
lated on Eb- The eigenvalues of (Hv are approximately given by 1/b where Ub. 0--1, and U--1 denote the eigenvalues of Hag H and H respectively.
(Th- 1),,
Please kindly refer to Appendix A.3 for the detailed proof. Proposition develops the relationship between the Hessian H used in the update rule of VR-MCL and that used in the update rule of Meta-CL (i.e., HM,) Similar to the analysis of the effect of Hessian in Secs. 2 and 3, we reach the following conclusions.
For those wrongly estimated low-curvature directions by Meta-CI., i.e., fo-13>0 Ub, Ub-1 (some potential crucial weights are erroneously viewed as unimportant), we have v, + ((6-1)) (p.-1)=")==((6-111 + S0,²" This signifies that VR-MCL prevent excessive updates triggered by the wrongly estimated low-curvature direction of the Hessian, thereby retaining the critical weights for previous tasks and improving the accuracy of its iterative updating formula.
For the high-curvature directions with large eigenvalues, i.c., 0--1 R vy R Up-1, we have 2n + r(((6-1)~' (To-))-) = o5'.. This suggests that VR-MCL can effectively ensure cautious updates along these high-curvature directions of Meta-CL.
To sum up. Meta-CL is susceptible to inaccurate estimations of Har, due to the random samples drawn from M. In contrast, the proposed VR-MCL incorporates regularization term H which effectively limits the model from making large updates and improves the precision of the iterative update. As a result, VR-MCL can effectively preserve the important parameters for previous tasks, thereby maintaining their performance.
Theorem (Regret Bound of VR-MCL). The regret in online continual learning follows CR,- = F(0) F(0*), where F(0) = El= ('0)), F(6*) = ming Eí L' (B). Assuming F with ip- Lipschitz hessian is p- p-strongly convex, G-Lipschitz, smsoohhh and grounded on the four assumptions in Appendix A.4, and M are two large constants in Assumption 3 and 4 respectively, and T denotes the training iteration. Then with probability at least 8 for any (0, 1), we can get,
CR2 S (log T 1)(F(0)) F(0*)) LDD T + 1)²
2
(log 1)² + (16LD² + 16oD + 4M) log =
Theorem illlustrates that, under mild assumptions in stochastic setting, VR-MCL attains nearly optimal Õ(VT) regret bound w.h.p. for the online optimization problem. This theoretically demon- strates the effectiveness of the proposed VR-MCL algorithm. Detailed proof see Appendix A.4.
5 EXPERIMENTS
Datasets and Settings. To verify the effectiveness of the proposed VR-MCL, we conduct compre- hensive experiments commonly used datasets CIFARIO,,,, well as the longer task sequences

---

Published as a conference paper at ICLR 2024
Table 2: Performance of Seq-CIFARIO and longer task sequences Seq-CIFARI00, Seq- TinyImageNet with 95% confidence interval on reduced ResNet-18. The memory buffer size is set as 1000 (i.e., | All reported numbers are the average of runs. Shaded areas are our methods, and "-' indicates the result is omitted due to high instability.
Seq- CIQ-CIFARIO
Scq-CIFARION
Seq-Tinn Tennyiinmagaa ImageNet AAA Acc
Method
AAA
Acc
AAA
Ace
SGD
34.85 1.71
16.96.10.66
11.63 1:038
5.27 10028
8.99 1:.39
3.86
LWF
35.31 +0.98
18.84
11 1.98 +0040
+0.37
9.21 #0.37
4.01 +:029
A-GEM
38.66 4..79
18.46
13. +0023
6.02
9.81 +0.31
4.07 =0.188
GEM
38.67 1:077
18.49 1:015
15,18 1:0.38
8.30
10,99 1.032
5.15 4:031
ER
55.53
43.83:44.8
23.19 1:038
16.07
19.45 20.40
11.13
DER
45.85 1.66
29.87 2:95
13.35 4:036
6.12
10.35 0.033
4.08 +:0.33
DER+
+0.70
52.29 1.86
19.88
11.79 +:065
14.75 +0.15
8.264
CLSER
63.02
52.80
1.66
25.46 1:0.5
17.88 1:0669
19.43 1.037
11.09 1:0244
OCM
66.14 0.995
53.39
1.00
22.54 4:079
14.40
10.45 10665
+0.54
ER-OP ER-OBC
65.82 1.091
54.85 2:1.66
25.54 1:0.25
17.21 4:0.92
20.11 1:031
11.51 4:0.18
On-EWC
38.44
17.12.00.5
11.81 1:0.42
5.88 1:0.31
9.38 10.17
3.65 1:0.22
IS
37.33
17.39 1:0.19
12.32 4:0.2
5.20
8.73 4.025
3.33 4:.28
MER
50.99 1:065
36.92
La-MAML
42.98 1.60
33.43
12.55 +:039
11.78 4:0.6
11.10
+0.70
6.74 4:0.6
VR-MCL
66.97 1.58
56.48 1.79
27.01 1:0.48
19.49 0.066
21.26 1.553
13.27 0.0.9
Ceq-CIFAR100 and Seq- Beg-TinylmageNet (Buzzega et al., 2020). Specifically, the Seq-CIFARIO dataset comprises 5 tasks, with each task containing classes. In contrast, Seq-CIFAR 100 consists includes 20 tasks, each encompassing
of 10 tasks, each with 10 classes, while 10 classes. In this paper, we focus on the following settings of continual learning:
Online CL: the data is fed to the model in small batches and trained in a single pass.
Class Incremental: the model cannot get the oracle task index during testing.
Our evaluation includes the metrics and experimental settings following the previous works line CL with a single-head (Caccia et al., 2021; Ji et al., 2020; Shim et al., 2021). We choose the final Averaged accuracy (Acc) across all tasks after sequential training metric for comparing approaches. Morcover, under online CL, we use the Averaged Anytime Ac- curacy (AAA) Caccia et al. (2021) to evaluate the model through the stream tasks. Let AA, denote the test average accuracy after training on Tj. then the evaluation metrics of AAA and Acc are:
each task the main
AAA = NN N. ()A)))) Acc = AAN
Baselines and Training Details. To more effectively validate the efficacy of VR-MCI, which improves the Hessian component and thus better utilizes second-order information, we select key regularization-basde CL methods (Huszár, 2017; Zenke et al., 2017) and Meta-CL methods (Riemer et al., 2019; Gupta et al., 2020). We also opt for other representative SOTA CL methods (Chaudhry et al., 2018; Lopez-Paz & Ranzato, 2017; Rolnick al., 2019; Buzzega et al., 2020; Arani et al., 2021; Guo et al., 2022: Chrysakis & Moens, 2023) as baselines to further highlight the supe- rior performance of VR-MCL. For detailed information each baseline method, please refer to Appendix C. For fair comparison among different CL methods, we train all the models using the Stochastic Gradient Descent (SGD) optimizer under the online continual learning. Follow- ing (Chrysakis & Moens, 2023; Lopez-Paz & Ranzato, 2017) and (Gupta et al., 2020) we utilize two backbones: reduced ResNet-18 and a smaller network named PeCNN which has only conv- layers. For the reduced ResNet18, we set both the batch size and the replay batch size (i.e., the batch size sampled from the memory buffer M) as 32. For the smaller network PcCNN, we set both the batch size and the replay bath size as 10. The momentum ratio and the learning rate a of VR-MCL are both set as 0.25 for all experiments. For other training details please check Appendix C.
To assess the effectiveness of VR-MCL, we perform comprehensive set of experiments and con- duct ablation studies to address the following six key questions:
Question 1: How does VR-MCL perform on online CL benchmarks? We report results with a 95% confidence interval on Seq-CIFAR10 and the longer task sequences Seq-CIFAR 100, Seq- TinylmageNet in Table 2. The methods in the bottom information while the top row are the methods of other representative CL methods. It is evident that the proposed VR-MCL substantially enhances the performance of CL methods employing second-
are the methods utilizing second-order

---

Published as a conference paper at ICLR 2024
Table 3: Performance of Seq-CIFARI00 with 95% confidence interval (CI) on different memory buffer sizes [M] The backbone is reduced ResNet- 18, and all numbers are the average of 5 runs. Shaded areas are our methods, and '_' indicates the result omitted due to high instability.
LM
200
|M
LM|=1001
Method
AAA
Acc
AAA
Acc
AAA
Acc
SGD
11.63 20.38
5.60
11.63 20.88
5.60 0028
11.63 4:.38
5.60
LWF
12.01 #0.42
5.84
12.12 #0,41
5.97
12.01 4:443
5.77 +0.32
A-GEM
13.45=022
6.25 +0.25
13.36 =0.21
00.2
6.00
13.15 4:036
6.12
GEM
15.41 +021
+.92
14.92
7.89
15.18 +..38
8.30
ER
18.52=003
10.15:0044
21.73
13.63 0.66
23.19++0.38
16.07 =0.88
DER
13.44=2031
6.12
13.57 0.22
6.23=0011
13.35 +:036
6. 18
DER++
17.91
9.59 20,42
19.13
10.45 0.33
19.88 1:0.43
11.79 2.065
CLSER
00.35
11.35 00336
23.63 =0,33
16.55 0.33
25.46 +..55
17.88 +0069
OCM
18.73
8.40=0.3
22.54 +:079
14.40 +082
ER-OBC
10.38 19
0.337
23.96
-
25.54-+0025
17.21 +0002
On-EWC
11.81
1..31
11.81 10,42
5.88 =0.31
11.81 4:0.42
5,88 10.31
IS
12.32
5.20 20.18
12.32 =0.22
5.20
12.32 4:022
5.20 20.18
La-MAML
8.48 #0.34
6.95 +0.27
10.07 11110
9.05 =0.58
12.55 4:0.9
11.78 #0.65
VR-MCL
22.41 +0.59
13.27 0.440
25.97
17.15
27.01 +0.48
19.49 =0.69
order information, and also outperforms other representative SOTA CL methods by nearly 2%.
is noteworthy that the performance of VR-MCL is superior to both MER (Riemer et al., 2019) and La-MAML (Gupta et al., 2020), and significantly better than IS (Zenke et al., 2017) and On- EWC (Huszár, 2017). This observation aligns with our analysis in Table 1.
How does the performance gain from VR-MCL vary with buffer size |M|? In large-scale CL, it is impractical to store large number of examples from previous tasks due to storage constraints. Therefore, an ideal CL method should still perform well even with a modest memory buffer. To investigate if VR-MCL is still effective with a smaller buffer size, conduct experiments on Seq-CIFAR100 with different [M as shown in Table 3. The results show that the performance of most methods improves with the increase of memory buffer size [M) And our VR-MCL consistently outperforms other baselines, suggesting that VR-MCL is effective even with modest buffer sizes. Note that GEM (Lopez-Paz & Ranzato, 2017) and A-GEM (Chaudhry et al., 2018) are insensitive to the buffer size complying with the results in Shim et al. (2021).
Question 2:
Question 3: How effective is VR-MCL in dealing with increasingly non-stationary Settings (i.e., imbalanced CL)? In the imbalanced CL setting, where the number of samples varies across tasks, an imbalance issue arises in the samples stored in M. This introduces a high variance during the calculation process of the hyper-gradient, thereby negatively affecting the impact on the implicit Hessian. To assess the effectiveness of VR-MCL in addressing these challenges, we choose differ- ent imbalanced settings and explore the performance of various methods under this setup in Table 4. Specifically, we choose three imbalanced settings. The Normal means the total number of samples per streaming task from high to low, and the Reversed is the opposite. The Random setting, on the other hand, represents a situation where there is no specific pattern in the number of samples per streaming task. From Table 4, we can that the performance of most models degrades in this challenging setup. However, VR-MCL significantly outperforms other basclines and even surpasses methods such as CBRS (Chrysakis & Moens, 2020) and Coresets (Borsos et al., 2020), which are specifically designed for imbalanced CL. This suggests that the VR-MCL has excellent performance under the challenged imbalanced CL setting. The 95% CI for Table 4, as well as additional experi- mental results for different datasets and various imbalance ratios, can be found in Appendix D.4.
Question 4: Whether the proposed method VR-MCL is still effective using different back- bones? To verify whether the improvement of VR-MCI. is related to the backbone network, we change the reduced ResNet- 18 as shallow network PcCNN following Gupta et al. (2020). The outcomes are provided in Table 5. It can be observed that VR-MCL shows nearly 3% improve- ment over the other best method in terms of both the AAA and Ace metrics. This indicates that the proposed VR-MCL remains effective across different backbone architectures.
Question 5: How does VR-MCL compare to other variance reduction methods? As dis- cussed in Sec. 4. most existing popular variance reduction methods like SAG (Gower al., 2020), SAGA (Defazio et al., 2014), and SVRG (Johnson & Zhang, 2013) cannot be directly applied since they need to compute the full-batch gradient which is not accessible in online CL. Therefore,
8

---

Published as a conference paper at ICLR 2024
compare VR-MCL with other techniques that promise reduce the variance: 1) VR-MCL', which simply increases the replay batch size; 2) VR-MCL?, which incorporates SGD with the naive mo- mentum term. The outcomes shown in Table 6 indicate that: 1) Other variance reduction techniques are also effective, supporting our analysis in Sec. 4; 2) Our proposed momentum-based VR-MCI. significantly outperforms other variance techniques, further demonstrating its effectiveness.
Table 4: Performance on the imbalanced Seq-CIFAR10
Table 5: Performance of Seq-CIFARI0 with 95% confidence interval on the small network PoCNN with |MM=2000. The results are the average of 5 runs.
([M|=1000) with imbalance ratio 7=22 The Reversed means the total number of samples per streaming task from low to high. Full table see Appendix D.6.
=2 (Normal)
7=2 (Reversed)
=2 (Random)
Method
AAA
Acc
Mothods
AAA
Acc
AAA
Acc
AAA
Acc
SGD
10.84
17.58
1.020
SCD
36.64
16:46
35.57
17.34
34,02
17.33
A-GEM
20.84
18.50 10338
A-GEM
38.50
17.64
36.72
17.43
37.79
17.85
GEM
45.07 20.52
22.52 1.00
GEM
38,46
39.63
18.00
ER
54.6410.09
32.46 1.158
ER
52.50
33.10
46,58
28.49
43.78
31.10
DER
58.06
38.21 =0.20
DER
46.95
16.82
40.62
18.63
41.44
18.78
DER++
56.07=0.x
34.28 2..79
DER+
44.14
58.22
60.25
42.83
CLS-ER
58.03
39.38 1.44
CLS-ER
61.37
47.75
54,92
40.51
56,60
47,48
On-EWC
38.85
16.92
37.41
17.79
37,26
14.35
ER-OBC
56.75=0.6
38.10
CBRS
43.81
57.57
58.
On-EWC
38.72
17.05 +0.05
Coresets
61.11
45.37
58.12
45,80
58.56
45.63
MER
5..50
32.01 #1.22
La La-MAMI.
36.64
29.17
32.08
31.17
42.52
31.24
La-MAML
46.36 0..98
29.45 +0051
VR-MCL
51.36
61.91
50.74
VR-MCL
60.88 0.22
43.41 00330
Table 6: Performance with 95% CI using various variance reduction techniques, where |M = 1000.
Seq CIFARIO
Seq CIFARI00
Seq-Tiny TinylmmaeNNe
Method
AAA
Acc
AAA
Acc
AAA
Acc
La-MAML
42.98 1.60
33,43
12.55 20.39
11.78
6.74
VR-MCL
66.74
55.02
26.15
19.02
20.39:1.17
12.09:1.30
VR-MCL2
62.87
53.20
20.15:11.5
16.05
2:004
15.96
9.82 20:4
VR-MCL(ours)
66.97 1.58
56.48 1.79
27.01 0.48
19.49 0.66
21.26 #0.53
13.27 00039
Question 6: Whether the proposed method VR-
Oralen Varimee
1.25
MCL can effectively reduce the variance of gra- dients? To explore this problem, following (Yang & Kwok, 2022), we choose the relative variance metric (i.e.. El Elg", )ll", ), which can eliminate the influ- [[
1.00
0.75
0.50
Mew CL
ence of the gradient value on the variance. The out- comes are shown in Fig. 3, where the Meta-CL refers to La-MAML (Gupta et al., 2020). It can be seen that the proposed VR-MCL indeed has less variance dur- ing training, complying with the analysis in Eqn. (5).
VR-MCL
0.00
500
1000
1500
2000
Figure The relative variance of Meta-CL and the proposed VR-MCL.
Other Results. Due to space constraints, additional comprehensive experiments have been pro- vided in Appendix D, including other evaluation metrics (i.e., forgetting measure), hyperparameter selection (i.e., the selection of the momentum ratio and learning rate a), and training time analysis.
CONCLUSION
In this paper, we revisited Meta-CL and first bridged it with regularization based methods from the Hessian matrix approximation perspective. Specifically, Meta-CL, through the use of hypergradient, implicitly approximates the Hessian in an online manner. While this approach benefits from timely adaptation, it also grapples with high variance resulting from random memory buffer sampling. Building on this understanding, we proposed the VR-MCL method, which effectively reduces the variance of the hypergradient and exhibits superior performance under the online continual learning setting. We provide theoretical proof that VR-MCL imposes a regularization term on the implicitly estimated Hessian matrix. This prevents updates from moving excessively in the wrongly estimated low-curvature directions and thus has a more accurate iterative update rule shown in Table 1. More- over, to enhance comprehension, also provide the regret bound of our VR-MCI..

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank all the anonymous reviewers for their constructive suggestions on improving this paper. This research was sponsored by the National Key R&D Program of China (020YFA0713900)) the RMGS 9229073, the China NSFC projects under contract 62272375, 12226004 and 62306233, the Young Elite Scientists Sponsorship Program by CAST 2023QNRC00..
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances Neural Information Processing Systems, 32, 2019.
Zeyuan Allen-Zhu and Elad Hazan. Variance reduction for faster non-convex optimization. In International Conference on Machine Learning, pp. 699-707. PMIR, 2016.
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general contin- ual learning method based on complementary learning system. In International Conference on Learning Representations, 2021.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances Neural Information Processing Systems, 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe-
rience for general continual learning: strong, simple baseline. Advances in Neural Information Processing Systems, 33:15920-15930, 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. International Conference on Learning Representations, 2021.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. International Conference on Learning Representutions, 2018.
Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In International Conference on Machine Learning, pp. 1952-1961. PMLR, 2020.
Aristotelis Chrysakis and Marie-Francine Moens. Online bias correction for task-free continual learning. International Conference on Learning Representations, 2023.
Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in Neural Information Processing Systems, 32, 2019.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems, 27, 2014.
Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non-convex optimization via stochastic path- inttgrrteeeted differential estimator. Advances Neural Information Processing Systems, 31, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1op.1126-62 1126-1135. PMLR, 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference Machine Learning, pp. 1920-1923 PMLR, 2019.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4): 135, 1999.
Robert M Gower, Mark Schmidt, Francis Bach, and Peter Richtárik. Variance- methods for machine learning. Proceedings of the IEFF, 108(1 (08(11)11988119 1983, 2020.
10

---

Published as a conference paper at ICLR 2024
Yanan Gu, Xu Yang, Kun Wei, and Cheng Deng. Not just selection, but exploration: Online class- incremental continual learning via dual view consistency. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 7442-74525 2022.
Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. In International Conference on Machine Learning, 2022.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. Advances in Neural Information Processing Systems, 33:1 13:11588-111 11788811158 11598, 2020.
Ferene Huszár. On quadratic penalties in elastic weight consolidation. arXiv:1712.033877, 2017.
arXiv preprint
Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Xu Ji, Joao Henriques, Tinne Tuytelaars, and Andrea Vedaldi. Automatic recall machines: Internal replay, continual learning and the brain. arXiv preprint arXiv:2006.1232. 12323, 2020.
Ric Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in Neural Information Processing Systems, 26, 2013.
Robert Keener. Theoretical statistics: Topics for a core course. Springer, 2010.
Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in Neural Information Processing Systems,
2021.
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partition- ing reservoir sampling. In Computer Vision-ECC ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 411-425 Springer, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barde Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-332 2017.
Hyunseo Koh, Minhyuk Seo, Jihwan Bang. Hwanjun Song, Deokki Hong, Seulki Park, Jung- Woo Ha, and Jonghyun Choi. Online boundary-free continual learning by scheduled data prior. In The Eleventh International Conference on Learning Representations.
Zhengfeng Lai, Chao Wang, Sen-ching Cheung, and Chen-Nee Chuah. Sar: Self-adaptive refine- ment on pseudo labels for multiclass-imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4091-4100, 2022a.
Zhengfeng Lai, Chao Wang, Henrry Gunawan, Sen-Ching S Cheung, and Chen-Nee Chuah. Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data. In International Conference Machine Learning, pp. 11828 11843, PMLR, 2022b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-22 222477777-272,.... 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems, 30, 2017.
James L McClelland, Bruce L McNaughton, and Randall C O'Rellyy Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
11

---

Published as a conference paper at ICLR 2024
Aryan Mokhtari, Hamed Hassani, and Amin Karbasi. Stochastic conditional gradient methods: From convex minimization to submodular maximization. The Journal of Machine Learning Re- search, 21(1):4232 044422228800 2020.
Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Takáč. Sarah: A novel method for machine learning problems using stochastic recursive gradient. In International Conference Machine Learning, pp. 2613-2621. PMLR, 2017.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11::54-71, 2019.
Iosif Pinelis. Optimum bounds for the distributions of martingales in banach spaces. The Annals of Probability,, pp. 1679-1706, 1994.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13588 13588-13597, 2020.
Jathushan Rajasegaran, Chelsea Finn, and Sergey Levine. Fully online meta-learning without task boundaries. arXiv preprint XXiv:2202.002633 2022.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. In International Conference on Learning Representations, 2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31, 2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap. and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang, On- line glass-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9630-9638, 2021.
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias. Information- theoretic online memory selection for continual learning. In International Conference on Learn- ing Representations.
Johannes Von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where to learn: Gradient sparsity in meta and continual learning. Advances in Neural Information Processing Systems, 2021.
Ling Xiao Wang, Kevin Huang, Tengyu Ma, Quanquan Gu, and Jing Huang, Variance-reducce first-order meta-learning for natural language processing tasks. In Proceedings of the 2021 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2609-2615, 2021.
Quanziang Wang, Renzhen Wang, Yichen Wu, Xixi Jia, and Deyu Meng. Cha: Improving online continual learning via continual bias adaptor. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 19082-19092, 2023a.
Renzhen Wang, Xixi Jia, Quanziang Wang, Yichen Wu, and Deyu Meng. Imbalanced semi- supervised learning with bias adaptive classificr. In 11th International Conference on Learning Representations (ICLR 2023), 2023b.
Yichen Wu, Jun Shu, Qi Xie, Qian Zhao. and Deyu Meng. Learning to purify noisy labels via meta soft label corrector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10388-10396. 2021.
Yichen Wu, Long- Kai Huang, and Ying Wei. Adversarial task sammanngnng for meta-learning. Ad- vances in Neural Information Processing Systems, 35:31 15:31102-33115, 2022.
12

---

Published as a conference paper at ICLR 2024
Hansi Yang and James Kwok. Efficient variance reduction for meta-learning International Confer- ence on Machine Learning, 2022.
Yang Yang, Da-Wei Zhou, De-Chuan Zhan, Hui Xiong, and Yuan Jiang. Adaptive deep models for incremental learning: Considering capacity scalability and sustainability. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 74 82, 2019.
Yang Yang, Da- Wei Zhou, De-Chuan Zhan, Hui Xiong, Yuan Jiang, and Jian Yang. Cost-effective incremental deep model: Matching model capacity with the least sampling. IEEE Transactions on Knowledge and Data Engineering, 2021.
Da-Wei
Dong Yin, Mehrdad Farajtabar, Ang Li, Nir Levine, and Alex Mott. Optimization and generaliza- tion of regularizzaion--asee continual learning: a loss approximation viewpoint. arXiv preprint arXiv:2006. 10974, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995 PMLR, 2017.
13

---

Published as a conference paper at ICLR 2024
A DETAILED PROOF
A. PROOF OF PROPOSITION 1
Proposition 1. In regularization-basel continual learning, if the model parameter is searched
within the neighborhood set U with N = d(; 8*) < 6'), then the iterative update rule of approximately is
0 ; 0 - a(H H +.. H 1 (0)
Proof. The empirical loss L on T1:ss can be approximate as,
L
(0) + ()) ~ 4- (0) L,(0)
(
+ (0 8¹) VoL
(A 89 H +C,
(a)
(b)
(c)
(0 8¹) T 0*) + (0)
Here, (1) is expand each loss Le using Taylor series point O' to compute (0), where i 1, j-. Step (2) is due to the term (a) is not related to f and thus can be discarded. Furthermore, since the training process at the end of each task (i.e., 0') is usually converged. the gradient (o') is near zero and the term (b) can be ignored in practice. Then we can derive 3c 36 as,
ac 00
j-1
Σ
- (0) Voc,(0).
the parameters are searched the neighborhood set -i where N d(0, 09 < 8%), then we can make the approximation Σ ( 0³) ~ (E H 01-1) Huszár (2017).
Let dc 0, we can obtain the iterative update formula of T as:
0 : (H + H² + ........................ H
where & = 1 to maintain formal consistency with other iterative rules in Table 1, (H + H +... H are the Hessian matrices of previous tasks computed the end of training of each T', and V C (0) is the gradient of the j-th task.
A.2 PROOF OF PROPOSITION 2
Proposition 2. For MCL with single inner step adaption, suppose that 0(K) is located in the e- neighborhood N(6".c, e) of the optimal model parameter 0" = argming (O(K)) L is p-smooth, and 3 < V (0) - (V)() where 8 is small number. Thus, the iterative update rule approximately is
e = e
Proof. (Single inner step adaption) For simplicity, we analyze the MCL with single inner step adaption (i.e., K 1) first. The objective function of one step inner-loop MCL could be formu- lated as, l" = argming cl 135) (0(k),
s.t. 0(k) = 0- (V)
Then the derivative of loss g()s (B(K)) over f is,
act:)((000) act::i((0 00(K)
00
00(K)
14

---

Published as a conference paper at ICLR 2024
act):) we take the Taylor expansion at 0,
For the first term
(0())
Dc VoiNs (0(K)) Vony ( (0)+H (0(x) -0) (0())) 8), (K)
(2)
where Ha = V2 (0) and denote the Hessian matrix and the third-order symmetric tensor, respectively, and @ represents the Kronecker product.
Since we hope to find the optimal parameters H* so as achieve the lowest Cllu], which means (0:5)(00::) 09
According Eqn. (1), that approximately to let ac(=a)(0 0.
80(x)
Suppose 8 is small number, 8< ( (0)3) and there exists an e-neighborhood (-)) such that parameters 8 E VL(0)) ll12 I pl10 0 112.
e). Since L is p-smooth, which means || (())
Then get the following equations according to Eqn. (2),
V (200
00
Ve-
C (0)
(U)) -) -000000-000000000000000000000100
Vauo C c1::((0)
V e (0)(0000)+((0)0)))0)))00)))))))0 =0000=
VoLK (R
V
(0)(0) - 0) o
o(uC) + V VOLKA L (0) 0) + +V
o(UE) + V VO(K) C (0) 0) (v +
(0)(00) -) ()
+ o(ue) 0,
where C means constant vector, O denotes the element-wise multiplication operator. Then it is easy to get = 0 - (Vi vV) = 0 -HHH,,)VV,, where a = B and
H Bik c(aa)(0)
(K inner steps adaption) For MCL with K steps of inner-loop updating, 0(K) = 0 3VL -
22 BV c( BVC
(0(K-1)). Similar to Eqn. (3), can get
K
0= 0 - aH
1=0
where Ory denotes the model parameters after the i-th inner loop adaptation and (()) = 0. is worth noting that in MCL with multiple updating steps. the hypergradient incorporates more gradients from the inner loop, which is the main difference from the one-step inner loop MCL. However, this difference does not affect the variance reduction analysis presented in Proposition
A.3 PROOF OF PROPOSITION 3
A.3.1 PROOF OF LEMMA
Lemma 1. (The linear combination of two invertible matrices is invertible under finite conditions.)
Let A = [01, ....... an] Raxn and B = by.., bn] E RExn. denote two square matrices, where a, E R", b, E Ra are the i-th column vector of the matrix A and B. Suppose: are invertible 2) for any Xa # 0, = 21,2,..n), (En Xax)) Di= Xabi) 2 0, then the linear combination of A and B (i.e., r B)) invertible, r EEE (0,
and
Proof. (By contradiction.)
(Step-1) We assume r + (1 (1-r)B, where r E (0, 1), not invertible, which means the square matrix A (1 B (ra: rrbb, (i01,,.. ran r))..] is singular.
15

---

Published as a conference paper at ICLR 2024
(Step-2) Then there must exists X, + 0, {1,2, such that Σ (irr,+(1-r00)) = That is equivalent to r Ei= A + (1 -r) Ei= X,b; O. According to condition 2, it can deduce that Di=1 A;a; = - A,b = 0. This is contradict with the condition that A and B are invertible.
(Step-3) Since assuming r (1 B is not invertible leads contradiction, therefore, r.A + (1 ---r) B is convertible.
A.3.2 PROOF OF PROPOSITION
Proposition 3. Assume that the batch size for inner step adaptation is sufficiently large. Let H
Vouce gl::A) (0p) -[h.. hi,.,- .......hh hng]
denote the Hessian O, calculated on eh, where hg the d-th
column vector of M Similarly, H Mo-1 -[ -bp_ denotes the Hessian for the momentum
term at the (b 1)-th iteration. If (Ed=1 )ah Xahg_1) 2 0 holds for any da + 0, d =
21,2,-.,, D,, then we have the following iterative update rule for VR-MCL.
: 0
+r where (H ) (ri ) (M,)))) with Hag denoting the Hessian at f-- cal-
M
culated on Eb. The eigenvalues of (H (RR) are approximately given by 0/ 1 (r(((()))- 11), where Ub, 06-1, and Ub-1 denote the eigenvalues of Hy,, H HM and H M respectively.
Proof. Let Be, denote the gradient of the b-th outer step. In proposition 2, the updating formula of MCL = - which means the gradient g0; can be rewrite the form of (H M. VoL' When adopt the variance reduction method, let denote the momentum part, then we can get.
51 = (H Vo
mi = Bi
-r(m1 80,)) = (H ~HoM)) (H)) (H (
m2 80g
m6 = 80, 5o
r(mb-1 B
where (M))) and VoL. (0) are the Hessian and corresponding gradient in g When there
has sufficiently large batch size on the current task, then VoLi(02) ~ Vol (0,) ~ (0))
Therefore, rewrite m2 as,
m2 = _ + r(mm go; = (H Mr ) ((H 1H0---HM,)~-Ja
Since HH , ) is highly related with (H's)) we can get,
(H) Mr ) + r(H 1- HH* M )
r( M, )-1
For m2, H [h], h.. hP] and H Ma 112 h2 h2 b., hb]] are also highly related. Based the assumption that for any Xa # 0, d = (1,2, D). (ED Xah Xahş) 2 0. According to Lemma 1, can get r r(H M, )1 (1-r)
(H) Mr ) invertible, and thus there exist matrix
(fi,,)-1 r(HH, - +
(1
For my, the previous assumption also holds, which is for H [i. h....,,,],. H My
"b.....,,,,,, for any Aa * 0, = (I, (22 Aa (2D -0 >ahgg1)
2 0. Ac-
cording this recurrence relation,
get (HM,)) r(H ) 1 + (1 (H,,)) Let
H VR M. we can get the following iterative update rule of VR-MCL,
0 0:20 0 aHH HR VoL
16

---

Published as a conference paper at ICLR 2024
A.4 REGRET BOUND OF VR-MCL
This section aims establish the regret bound of VR-MCL. Our method involves three key parts: Firstly, we define the objective form of Regret Bound in continual learning. Secondly, we prove that updating the objective of VR-MCL is equivalent to updating the latent optimal loss function, by bounding the gradient of VR-MCL with the latent optimal gradients. Finally, we derive the regret bound of VR-MCL based on the aforementioned steps.
(Part 1) We first define the Regret Bound in continual learning.
Definition 1 (The Regret Bound in CL). Consider an online learning scenario where an agent makes decision by choosing action 0g RR each time step 1...., T, and experiences corresponding loss (e(B)). The primary objective of the agent is to minimize the difference between its cumulative loss and that of the best action in hindsight, which is commonty known as the regret.
RT RT=Et=
Et Li (0t) mingee E L((0)
Similarly, in MCL, for the current task T the goal is to minimize CR, = F((0) - F(O"), where
= = (01) and F(0*) = ming Ei-
This proof is grounded on certain assumptions, which are listed below:
Assumption 1. The compact convex set C C gd has diameter D, i.e, v0,0' E EC. 110 0'1 S D.D.
Assumption 2. The function F(-) is G-Lipschitz, p-strongly convex, sp-smooth (i.e., has s--iipscitz gradients) and has ss-Lipschitz hessian. Then, according to the Theorem in Finn et al. (2019), when the step size a < {2 SkG Jht gradient 4 F(x, ()) is L-Lipschitz (L = 8 C, that is
9:2 over the constraint set
(VE(0.e)--ViC Eell L110 0'11 92 0 11 0 EC
Assumption 3. Let Fr denote the t-th training step, then the distance between our computed (f, a) and the gradient derived from the optimal function is bounded over the constraint set C, i.e. for any 0 E C, (I,..., T), there exists 6² co such that with probubility
Assumption 4. Let F((O) mean the optimal t-th training step. and the difference of (() and F(0) bounded over the constraint set C, iee. 40 E C, E {1,..., T), there exists M² < 0o such that with probability 1,
[F((0) - F(O) S M
(Part II) In the following, we will demonstrate that updating the objective function of VR-MCI. can be regarded as updating the latent optimal loss function. This is achieved by bounding the gradient of VR-MCL with the latent optimal gradients. Specifically, the updating gradient can be expressed F ct)). Let denote the small learning rate and define A = My (f))), our aim is to prove that A, is bounded. Here, we suppose the momentum ratio (1 Pt), the learning rate nhe and P. nt 1/(t + 1)", E (0, 1].
m, (F ct) + r(m--1
L Lemma& 2. If Assumptions 1,2,3 are satisfied, Vt 2 1, 60 (0, 1), we have with probability at least
80.
| < 2 22LD
3 3-
(t (++1) log (4/8))
Lemma 2 demonstrates that the gradient approximation error ||А||| is bounded with high probability increases.
17

---

Published as a conference paper at ICLR 2024
Proof. We derive the formulation of A at first.
A. = m - VF(O.)
= -
( -
II (1-Pk) + (LL-://VVE(ccc------------------ k=2
k=
+ Σ Pr II (1- (a) -VF(0,))
k=7+1
Let A. = D St.r, where G.1 - and for > 1, St, (V (0,, - P. I (1 (VF, (F)))) Recall A VF(02, 61) V We observe that E [St.- F = 0 when Fr-1 is the o- o-flld generate by E1, 1, 1 1. Therefore. { is a martingale difference sequence. Then, will derive upper bounds ||G.,- Il. We initially proved the following results,
6, (VF(0))
(1) II (
(k= (
II
(k (k+
II
ky
1)7
(t+ 1)7
k=y
k=
where the inequality holds from the concavity of h(x) xx for any x 2 0. Based on this inequality we can bound || || as follows,
|| S
(1 11 (0)(0, VF(O)|| (F(0)))))
2"
<
2" def
Ct.1
(1 + 1)a
where the second inequality holds according to Assumption 3. When r 2 1, we can get,
||G,=l1 s [1 (11=-pol0 > (VF-,))(VF((0...G) 11 [1 -
VF,(0--1,
k=
II (1 (x- -VF(()))
+ Pr
k=
(4)
<2L|10, - 0,-1
(1 - pk) opr II (1 pk)
k=T
k=T+1
2LDP
II (1-Pk) + op,
II (1-Pk)
k=
kmT+1
We can further derive the final term as follows,
Pr
(1- px)
Pr
Pr-1
(1 pk)
1
Pr-1
(1 pxb)
P--1 (1
pr)
1- Pr
k=++1
k=r
k=T
(5)
1
Pr-1
pk)
S
3
P7-1 II (1 ph)
1 1/37
37 -
k=r
By plugging Eqn. (5) into Eqn. (4, we can get, VT 2 1
[[Crrll < (2LD
ad
pr
(1 - px) def
Ct.r
3
R=T
According to Theorem 3.5 in Pinelis (1994), we can get AA > 0.
x²
IP (I 2 A) < 4exp
(6)
- 2
18

---

Published as a conference paper at ICLR 2024
where Ct,1 is defined in Eqn. (A.4) and Ct,r, (T 1) is defined in Eqn. (4). Then get,
Σ 1=1
Ct,1 +
22
+
2LD +
3
2
Σ
Pr-1
II
(1 - pk)
2
(t (++1)27
37 -
T=2
7=2
kmT
)²
2
2
2LD 3'0
30-1
(()2)))
2LD +
37-1
3"a
(2LD + 3'0
30
+
(1 1)27
(1+ 1)7
11+
(L 1)7
(1+ (7)
where the last inequality is because (v2)² S 33/ - vy E (0,1). Substituting Eqn. (7) into 3"a
Eqn. (6) and setting À 2 22D + 3--1 with probability at least 80.
(t+1) 7/2 /2 (4/6)) for some 5n E (0, 1), we have
| S 2LD +
3 3-
(l (1+ / log (4/6o).
which is the expected result.
(Part III) Next, will establish the regret bound CR₃ for continual learning.
Theorem 1 (Regret Bound of VR-MCL). If F is convex and all aforementioned four Assumptions are satisfied. then with probability least for any 8 E (0, 1),
- 2 1)²
CR₁ S
2
(Do 7 + 1)2 + (16LD2 + 16o + 4M) 2T log(T = +
2
Proof. We initially define a sequence s; = - observed that [s F(-1] O, where F is the o-aleerra generated by {Fi, 6,. Fi 6t-1) }.
T. It can be
It means that { is martingale difference sequence. According to Assumption 4, we have
| || = ((.) - F.((") - (F(O.) - F(0 $ 2M
According to Theorem 3.5 in Pinelis (1994), we can get
X
P
St
A
< 4exp
16T
where À > By setting À 4M / Tlog(8/8)) we have with probability least 8/2,
Σ 8: = Σ ) Σ (F((.))
F(OO)) 4M / TToog(8/8).
By rearranging the above terms, we get
T CR, F(( - F(O)) = (f.00))) < > (F(())) + 4M (Tog)(/0)). (8)
T
1=1
19

---

Published as a conference paper at ICLR 2024
Then according to Lemma 2 in (Mokhtari al., 2020), derive the first term in Eqn. (8) as follows,
L.Dom
F(O+) -((0) - F(O)) + 7p 1 1|
[-1
LD
II - (F)()) +
7/ (D 4 112
(1 -
(1))))
k=T-1
LD² T+1
(F(0)) F(O)))
|r| + 2(T 2(T
1 (F(0)) F(()))
LD
Σ
D +
2(T+1)
$ 1 (F(()) (F((1)-F(00)
D Σ
LD logt
| 11?
(9)
Summing Eqn. (9) from
T, we obtain,
T
LD log
- F(O)) S
(F()))-F(0') F(O*))
| 1
Σ
21
1=1
LD2
log T
T
$ (F(0))- - =(())) + Σ
| 11
2
1=1
T 1-1
LD T + 1)
S (log T ()(F(()) - F(O))) +
||
+
2
(10)
According to Lemma 2, we can obtain the second term is,
T D | s 4(LD + 200
T t-1
ΣΣ =1
WT+1
1=
2
(11)
4(LD2 A(LD*++DD) 2og(88/
t
1=1
$ 16(LD + +oD) 2T (8T
Substituting Eqn. (11) into Eqn. (10), we have with probability at least 5/2,
Σ F(0.)-F(0)) < (o LDd (log T
(log + 1)2
+16(LD&+oD) 2T log
(8g(81/0)
(12)
Combining Eqn. (12) with Eqn. (8) we have with probability least 1 5,
CR² ( 1)(F(0)) F(OO)) (log T 1)2
2
+ L.D T + 1)2
2 + (16LD² + 16o + 4M) (
B RELATED WORKS
Regularization-baser methods (or parameter regularization methods) usually construct quadratic regularization term to slow down the learning of weights that are important to prior tasks. As we know, the Fisher information matri is equivalent the Hessian matrix if the loss function
20

---

Published as a conference paper at ICLR 2024
is negative log- likeliieeoodd and we get a ground truth probabilistic model (Chapter 4.5 of (Keener, 2010)). In this way, EWC (Kirkpatrick et al., 2017) and on-EWC (Huszár, 2017), as the first work applying parameter regularization, utilize the diagonal Fisher information matrix to approximate the Hessian matrix, while Kronecker factored Laplace approximation (KFLA) (Ritter et al., 2018) exploits Kronecker factored Laplace to compute the Fisher information matrix with off-diagonal el- ements. To account for the trajectory of model training. IS (Zenke et al., 2017) proposes an online estimation of a diagonal matrix and assigns weights based on its contribution loss decay. On the other hand, in class- incrementlt learning, IADM (Yang et al., 2019) and CE-IDM (Yang et al., 2021) report that different layers have varying characteristics. Shallow layers with limited representation tend to converge faster, while deep layers with powerful discrimination abilities tend to converge more slowly. Therefore, based on EWC (Kirkpatrick et al., 2017), IADM (Yang et al., 2019) learns an online layer-wise importance matrix.
Rehearsal- methods address catastrophic forgetting by replaying previous task samples from memory buffer. Experience Replay (ER) (Rolnick et al., 2019) directly sample data from previous tasks and put them jointly train with current data. On the basis of ER, recent studies have further extended this idea. Meta Experience Replay (MER) (Riemer et al., 2019) views replay learning problem for maximizing the transfer from previous tasks and minimizing the interference. Gradient-based Sample Selection (GSS) (Aljundi al., 2019) pays attention to the samples stored in the memory module and hopes to increase the diversity of samples in the gradient space. Gradient Episodic Memory (GEM) (Lopez-Paz& & Ranzato, 2017) and its lightweight variant Averaged- (A-GEM) (Chaudhry et al., 2018) formulate optimization constraints such that gradients between current and old training data are aligned further determine the final direction of optimization. Dark Experience Replay (DER++) (Buzzega et al., 2020) adds the knowledge distillation part on top of ER to regularize the logits of samples stored in the memory buffer. CLSER (Arani et al., 2021) introduces an innovative approach to episodic replay (ER) by presenting dual memory framework. This framework incorporates both short-term and long-term semantic memories, allowing for their interaction with the episodic memory. CBA (Wang et al., 2023a) proposes to address the issue of recency bias by introducing a specially designed bias attractor, guided by the memory buffer M.
meta-
Meta-Continual Learning (Meta-CL) is class of methods that apply meta-learning (Finn et al., 2017; Wu et al., 2022; 2021; Wang et al., 2023b) in continual learning. From the li-le optimiza- tion perspective to analyze meta-learinn, its inner-loop (or lower-level) optimization usually fits the training data, while the outer-loop (higher-level) centralized test data aims to increase the model's generalization ability. Therefore, meta-learning is well suited for continual learning, as we want the model to fit the current task and still perform well on all observed tasks (i.e.. previous and cur- rent tasks). Meta Experience Replay (MER) (Riemer et al., 2019), inspired by GEM (Lopez-Po & Ranzato, 2017), utilizes replay to incentivize the alignment of gradients between old and new tasks to further maximize the transfer from previous tasks and minimize the interference. Online- aware Meta Learning (OML) (Javed & White, 2019) adopts a pre-training algorithm to learn an optimal representation offline, which is frozen when training in the downstream tasks. However, this offline training manner and assumption that the training data is correlated data stream limit its applicabil- ity to practical scenarios. Besides, MER (Riemer et al., 2019) and OML (Javed & White, 2019) use a single sample at one time during the inner-loop optimization, which is time-cor consuminn and not suit- able for large-scale CL. To alleviate these problems, Look-aheaad Meta learning (La-MAML) (Gupta et al., 2020) proposed an online gradient-based meta-learning algorithm and proved it is equivalent to a simplified version of gradient alignment. On the other hand, some Meta-CI. algorithms (Ra- jasegaran al., 2020; 2022) focus on combining meta-learning with task-agnostic continual learning that does not require task information during training and testing. Besides, the variance reduction has also been integrated with first-order meta-learning (Yang & Kwok, 2022; Wang et al., 2021). I). In this work, we focus on the online continual learning setting and provide variance reduction Meta-CL CL (second-order) based on our novel understanding of Meta- CL, it also links Meta- -CL with regularization-bane based methods.
Online and Imbalanced Continual learning, different from the traditional offline CL setting, are more challenging CL settings that have not been explored as much. For online CL scenarios, the initial methods still have been developed based on similar ways in offline CL (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018; Aljundi et al., 2019). Recently, there has been increasing interest in developing online CL methods that can fully utilize the single-pass data stream to decrease the performance difference between online and offline CL. OCDVC (Gu et al., 2022) advocates for
21

---

Published as a conference paper at ICLR 2024
the dual view consistency strategy, while SDP (Koh et al.) utilizes knowledge distillation to fully leverage the data stream. OBC (Chrysakis & Moens, 2023) proposes a simple approach to mitigate the bias of online CL by changing the optimization manner of the output layer model. OCM (Guo et al., 2022) combines these motivations and aims to reduce feature bias and fully utilize streaming data through mutual information maximization. However, none of these methods directly focus on the performance drop between online and offline CL and analyze the underlying reasons, which is a motivation of this work.
For imbalanced CL. where the sample number of each task is different (Lai et al., 2022b;a), most works aim directly design a sampling strategy that can balance the distribution of samples re- trieved from the memory buffer that stores data from previous tasks. CBRS (Chrysakis & Moens, 2020) introduced a novel mechanism for class-balancing reservoir sampling based on the sample distribution within the buffer. PRS (Kim et al., 2020) utilizes current data stream statistics to deter- mine the sample-in and sample-out process, maintaining a balanced replay memory. Additionally, InfoRS (Sun et al.) introduces a stochastic information theoretic reservoir sampler to select and store the informative samples. Coresets (Borsos et al., 2020) employs the bi-level optimization technique for the same purpose. Different from these methods, our proposed approach takes different per- spective. It aims to conduct a comprehensive analysis of the factors that contribute to the substantial performance drop observed in both imbalanced and online settings compared to offline learning. By understanding these underlying factors, we propose the VR-MCI. to the model performance in imbalanced CL scenarios.
Variance Reduction is a widely used technique for reducing the variance in stochastic gradients and accelerating the optimization process. Early research on variance reduction methods, such as SAG Gower et al. (2020), SAGA (Defazio et al., 2014), and SVRG (Johnson & Zhang, 2013), was primarily focused on strongly convex optimization problems. However, in recent years, these meth- ods have been extended to general non-convex optimization problems (Allen-Zhu & Hazan, 2016; Nguyen et al., 2017; Fang et al., 2018). Nonetheless, these methods typically require the expensive computation of the full-batch gradient by processing all samples in a task, which is impractical in the context of online continual learning where samples are processed in mini-batches and the com- plete sample set is unavailable. To alleviate this expensive computation problem,
cantenttumbbsse variance reduction algorithms (Cutkosky & Orabona, 2019; Khanduri et al., 2021) are proposed. Specifically, these methods incorporate momentum term compute the stochastic gradients at two consecutive iterations on the same set of stochastic samples, thereby replacing the conecconuuming computation of the full-batch gradient. Additionally, these methods have the same asymptotic con- vergence rate as other variance reduction methods.
C IMPLEMENTATION DETAILS
C.I EXPERIMENTAL SETTINGS
Imbalanced Settings. We conducted imbalanced online continual learning experiments on Seq- c0far10, consisting of five tasks, each with two classes. As shown in Fig. 4, when the imbalance ratio 7 = the number of samples for each class across all tasks is [5000, 4629, 4286, 3968, 3674, 3401, 3149, 2916, 2700, 2500]. We also considered the reversed version, where the number of samples for each class (2500, 2700, 2916, 3149, 3401, 3674, 3968, 4286, 4629, 5000J. Without loss of generality, we additionally sampled a random version with [2700, 2500, 5000, 4286, 3674, 3968, 3149, 3401, 2916, 4629] samples per class across all tasks.
Baselines. We compared the proposed VR-MCL to various baseline approaches, including regularization-base methods, rehearsal-basee methods, meta-continual learning methods, and spe- cific CL methods designed for imbalanced and online settings.
Regularization-basee methods:
On-EWC: (Huszár, 2017). It approximates each Hessian matrix by a diagonal Fisher matrix calculated the end of each task training. The larger Fisher information value means the corresponding weights are more important to the old tasks and thus should undergo less change in future tasks.
22

---

Published as a conference paper at ICLR 2024
5000
5000
9300
Sanple atr
4003
4000
+000
3002
2000
2000
2300
1000
1000
1000
Class Indox
Class Indox
Class index
(a) Traditional imbalance.
(b) Reverse imbalance.
(c) Random imbalance.
Figure 4: Different imbalance settings of Seq-CIFARIO, where the imbalance ratio = 2.
IS:
(Zenke et al., 2017). It introduces the intelligent synapse. Each synapse accumulates task-relevant information over time so to rapidly store new memories without forgetting the learned ones. From the Hessian approximation perspective, it constructs a diagonal regulariza- tion matrix (i.e., generalization form of the Hessian matrix integrating over time) estimate the Hessian matrix.
LWF: (Li & Hoiem, 2017). It is similar to joint training, but the key difference is that LWF does not require data and labels from the old tasks, instead employing distillation techniques.
Rehearsal methods:
GEM and A-GEM (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018). They formulate optimization constraints such that gradients between current and old training data are aligned to determine the final direction of optimization.
ER (Rolnick et al., 2019). Training current task with data sampled from the memory buffer which stores the examples from previous tasks.
DER/DER+ (Buzzega et al., 2020). Adding the distillation loss of logits on the top of ER. CLSER (Arani et al., 2021). The dual memory experience replay (ER) method maintains short- term and long-term semantic memories that interact with episodic memory.
CL methods focus on imbalance and online settings:
CBRS (Chrysakis & Moens, 2020). It introduces a novel mechanism for class- balas-bllnncing reser- voir sampling based the sample distribution within the buffer to ensure more balanced sam- ples are stored.
Coresets (Borsos et al., 2020). It presents novel coreset construction technique through constrained bilevel optimization, which effective the imbalanced CL setting. OCM (Guo et al., 2022). It proposes to reduce feature bias and fully exploit streaming data through mutual information maximization. Note that, for fair comparison, we choose its OCM (no local augmentation) for evaluation.
ER-OBC (Chrysakis & Moens, 2023). It proposes a simple approach to mitigate bias in online CL by modifying the optimization process of the output layer model.
Meta-CL methods:
MER (Riemer et al., 2019). utilizes replay examples to incentivize the alignment gradients between old and new tasks to further maximize the transfer from previous tasks and minimize interference.
La-MAML (Gupta et al., 2020). It proposed an online gradient- based meta-learning algorithm and proved equivalent
simplified version of MER. Note that in the online scenario of La-MAML (Gupta et al., 2020), the mini-batch samples instead of the whole task could be trained multiple times, which is not the strict online continual learning. our experiments, all methods follow the standard online setting where all samples can only be trained once.
plyper-parameters The hyperparameters used for each experimental setting are listed in Table 7. Let Ir denote the learning rate, bs denote the batch size and màs means the minibatch size (i.e., the batch size drawn from the buffer). It is worth noting that used the same hyperparameters across different datasets and buffer sizes, which shows our method has good generalizability without requiring extensive hyperparameter tuning for each specific setting.
23

---

Published as a conference paper at ICLR 2024
Table 7: Hyperparameters on all the three datasets including Seq-Cifarl0/100 and Seq-
TinylmgaeNet.
Backbone
Epochs
Buffer size
Inner Ir (3)
Outer Ir (a)
r
inner-bs bs
mbs
PcCNN
200
0.1
0.25
0.25
10
10
PeCNN
1
600
0.1
0.25
0.25
10
10
PcCNN
I
1000
0.1
0.25
0.25
10
10
Reduced ResNet-18
200
0.1
0.25
0.25
8
32
32
Reduced ResNet- 18
600
0.1
0.25
0.25
32
32
Reduced ResNet-18
1000
0.1
0.25
0.25
8
32
32
C.2 THE MASK TRAINING OF THE PROPOSED VR-MCL
For the implementation of the VR-MCL, adopt mask training different from La-MAML (i.e., one mask training strategy on the final output layer). In this subsection, we will first introduce two mask training approaches: Mask-LM, which is the original mask training in La-MAMI, and Mask-VR, which is the proposed mask training used VR-MCL. We will then provide both mathematical and experimental analysis compare the mask training methods.
Predict probability of one class
(not masked] Preserved predicted probability
T
T1
J.
T
DM
D 2's
DE"LLTT
D 3'
24
Tz
Inner loop
Outer loop (Mask-UM LM)
Outer loop (Mask VR)
Figure 5: The illustration of the two mask training approaches, where the current training task is T
Introduction of the two Mask training methods As the output layer shared among tasks, meta-continual learning methods have the problem of mismatch between the limited rehearsal data from previous tasks [1:-1]] and the larger incoming data from the current task T To solve this problem, masked softmax loss is applied by La-MAML Gupta et al. (2020). As shown in Figure 5, La-MAML applies mask (i.e., Mask-LM) that only keeps the logits of the corresponding classes within a task to both the loss in the inner-loop adaptation and outer-loop optimization. The Mask-LM improves the inner-loop adaptation for the current tasks and mitigates the interference between the limited data sampled from the memory buffer M and the larger incoming data from the current task. However, the employment of the Mask-IM in the outer-loop still has limitation. The left part of Mask-LM concentrates solely on minimizing the intra-task loss, without consider- ing the interdependencies between different classes from various tasks. This oversight significantly reduces the model's s efficacy in the class incremental setting. To address this issue, we propose modification (i.e., Mask-VR) inspired by (Caccia et al., 2021), where we mask all seen classes that appeared in the memory buffer M. This modification enables the model to distinguish samples belonging different tasks, which is shown the left part Mask- VR. Mathematical and Ex- perimental Analysis. The main idea of Mask VR to enlarge the preserved logits is to introduce the negative gradients so as get more concise class prototypes and thus achieve better performance. conducted experiments and demonstrated
To investigate the impact of the proposed Mask-VR, empirically that it can effectively reduce the gradient variance. Furthermore, our findings suggest that the Mask-VE -VR and the proposed variance reduction method can be synergistically combined to effectively reduce the gradient variance.
We will begin by conducting the mathematical analysis. Note that the following mathematical symbols are solely for the purpose of illustrating the Mask-VR, they are not aligned with the math- ematical symbols used in the main text.
24

---

Published as a conference paper at ICLR 2024
Let Cseen refer to all seen classes, and |Cseen is the number of classes the model has seen. In Figure 5, the current training task is T and [Ceeen = 6. Let the model architecture consists two parts: feature extractor bgg and classifier fa and the output of the model before the softmax operation o(2; 0, Φ) = {x()) Then the cross-entropy loss in continual learning Lee is,
ICucul Σ lo.log(ft),
cap((oa)
(o,(0(300,0)
Da
D.Crrol emp(oe.)
where is the training sample, ly 10,1) is the one-hot lahel of class ca, and o(;;0, 0, (De, Oegs Oclenen is the corresponding logit values of x. Let x mean the sample class Ci. then the gradients on the classifier fo is,
dccr (o (0 = =pa -1, Dce (o (0 .0)
= p
8oa.
Doc,
dce (0 (0))
dco (0 0, 0)))
(x';
doc: (pf 1) hg ho(x'),
[Positive gradient]
Doe
0o..
Doa
dc (0 )))
dce (o (x"; ②)) docs
= pS ho("), Doc,
[Negative gradient)
Oo.,
doc,
Let W denote the matrix with all class prototypes (EeJoe then W o ha = (hol())) Ac- cording the above equation, can get in Mask-LM, only We: receives a positive gradient from x², and there are no negative gradients due to preserving only portion of the logits. However, in Mask-VE VR, both positive and negative gradients are present, similar to the concept of positives and negatives in contrastive loss. Consequently, the class prototypes W learned with Mask- VR exhibit superior performance compared those learned with Mask-IMM.
Relative Gradient Variance
1.25
1.00
La-MAML
0.50
MCL
0.25
VR-MCL1
VR-MCL,
0.00
500
1000
1500
2000
Iterations
Figure 6: Relative gradient variance of update during training. The subscript represents training with Mask-LM, while the subscript represents training with Mask-VR
Ablation studies of the Mask- Training. To better understand the effectiveness of the Mask-VR and the proposed VR-MCL, we compare the performance of La-MAML, MCL(La-MAML with Mask-VR), well VR-MCL, (i.e., VR-MCL with Mask-LM) and VR-MCL2 (i.e., VR-MCL with Mask-VR,, as shown in Table 8. Moreover, we also plot the relative gradient variance of La- MAML. MCL, VR-MCL, and VR-MCL2, as shown in Fig. Based on the fact that the curves with variance reduction (VR) are all lower than those without VR. it can be concluded that: 1) the proposed VR technique indeed can reduce the variance, 2) the effects of VR and Mask- MR can be superimposed and work together.
Table 8: Performance of La-MAML, MCL, VR-MCL, and VR-MCL, on Seq- AIg-CIFAR1O...
Method
La-MAML
MCL
VR-MCL,
VR-MCL
Acc
33.43
52.40+2.13
35.66
56.48
AAA
42.98=1.60
65.87 +1.6
50.02
66.97 +.58
25

---

Published as a conference paper at ICLR 2024
D OTHER EXPERIMENTAL RESULTS.
D.1 THE TRAINING TIME ANALYSIS OF VR-MCL
To analyze the time complexity the proposed algorithm, we compared the training time of the proposed VR-MCI. with relevant Meta-CI. methods on the Seq-CIFAR-10 dataset using reduced ResNet-18, as shown in Table substantial reduction in training time compared to MER. To assess the value of the additional training time required by VR-MCL, we conducted experiments by extending the training epoch of La-MAML from to 2. Surprisingly, despite the comparable training time with VR-MCL, La- MAML achieved a performance of only 38.89%, which is extremely lower than the performance of VR-MCL (i.e., 56.48%). This outcome
is evident that our proposed VR-MCL method demonstrates
strong evidence for the superiority of VR-MCL in terms of both training efficiency and performance on Seq-CIFARIO.
Table 9: Training time comparison of different meta-continual learning methods on Seq-CIFARIO under the online setting (i.e., the training epoch of each task set as one.)
Method
La-MAML
MER
VR-MCL
Training Epochs
1
Training Time (s)
750.61
20697.70
1297.10
Training Epochs
2
1
Training Time (s)
1511.54
20697.70
1297.10
AAA
53.21%
50.99%
66.97%
Acc 38.89%
36.92%
56.48%
D.2 OTHER EVALUATION METRICS
To further evaluate the effectiveness of the proposed VR-MCL, provide the backward transfer (BWT) metric in Table 10, which measures the performance degradation in subsequent tasks. It can be observed our VR-MCL achieves the best results. is excluded from the analysis due to the extremely negative impact of Mask-LM utilization on its newest training task (i.e., T performance (Caccia et al., 2021). This impact results in large and even a positive backward transfer (BWT) value for La-MAML.
worth noting that in Table 10, La-MAML
Table 10: The Backward Transfer of Ceq-CIFAR10 and longer task sequences Seq- CI--CIIARROO,, 100, Seq- TinylmageNer with 95% confidence interval on reduced ResNet-18 under the online setting. The memory buffer size is set as 1000 (i.e., |M|=0000). All reported numbers are the average of runs. Shaded areas are our methods, and = indicates the result was omitted due to high instability.
CIFAR-10
CIFAR-100 BWT
Tiny-ImageNet BWT
Method
BWT
SGD
-61.48
.44.53
-34.59 +0.35
On-EWC
-65.94
-43.45
-33.05 +0.56
-63.38
-39.54
-23.76
A-GEM
-67.06
.48.74
-37.93
GEM
-62.72+2.00
-42.27+1.99
-36.21 40079
ER
-33.97
-31.24
-31.07
DER
-62,17
-50.21 11.2
-39.50
DER++
-24.65+2.21
-46.74
-39.16+1.34
CLSER
-23.61
-37.02 +135
-34.58 +0.85
OCM
-22.02
-17.91 10.44
-11.99 10.86
ER-OBC
-34.71 10.11
-5.39 1188
-6,20 11.27
MER
-28.77
VR-MCL
-20.92+194
-0.16 0.100
-2.86 0.883
26

---

Published as a conference paper at ICLR 2024
D.3 HYPERPARAMETER SELECTION
We utilized grid search technique to select the optimal hyperparameters, including the learning rate and the momentum ratio r, as illustrated in Table 11 and Table 12. The results demonstrate that our proposed method achieves the best performance when both the momentum ratio r and learning rate or are set 0.25. Moreover, is worth noting that there small performance difference observed when different hyperparameters are chosen. This suggests that the proposed VR-MCL method exhibits robustness and is not highly sensitive to the selection of hyperparameters.
Table 11: Reduced ResNet18 performance with different momentum ratios Seq-CIFARIO, using a fixed learning rate (a = 0.25).
Table 12: Reduced ResNet18 performance with different learning rates Seq-CIFARIO, us- ing a fixed momentum ratio (r 0.25).
0.15
0.25
0.35
a
0.15
0.25
0.35
Ace
55.46:0044
56.48 1.79
54.14 11.50
Acc
54,466 1.94
56.48 1.79
55.91
1.52
AAA
16.46
66.97 1
63.92 00889
AAA
66.06
66.97 1:58
66.33 1.33
D.4 OTHER IMBALANCED CL EXPERIMENTS
Due to the page limit, the full results of Table are shown in Table 13. To further explore the performance of VR-MCL in imbalanced settings, we select the more realistic random imbalanced scenario and present the results for a higher imbalance ratio, specifically 7 = in Table 14. For simplicity, we select the SOTA methods from Table 13 as the comparable methods. The results clearly demonstrate that the proposed VR-MCL surpasses other methods in the case of y = 5, providing further evidence of the effectiveness of the variance reduction technique in addressing imbalanced CI..
To further demonstrate the effectiveness of the proposed VR-MCL under imbalanced settings, performed additional verification on various imbalance settings using the Seq-CIFARIO0 100 dataset. The results of these experiments are presented in Table 15.
Table 13: Reduced ResNet-18 performance on the imbalanced Seq-CIFARIO (IM|=1000) with imbalance ratio 2=2.
7=2
=2 (Reversed)
Y == (Random)
Methods
AAA
ACC
AAA
ACC
AAA
ACC
SGD
36.64 #0.42
16.46
#0.31
35.57
=0.70
17.54=0.11
34.62=131
17.33
On-EWC
38.85 10018
16.92
1031
37.41
1026
17.79
10.17
37.26 0049
14.35 1.85
A-GEM
18.50
17.64
36.72=0.3.
17.43
10.47
$7.79=0.5
17.85 +0.22
GEM
41.29 10.4
17.90
38.46
18.37
2025
39.63
18.00 10.73 10.73
ER
52.50
33.10
46.58 2229
28.49
1346
$3.78=255
31.10 15.38
DER
46.95 1.54
16.82
10.8
40.62
18.63
1061
41.44-066
18.78 10.23
DER++
62.02 +0.52
42.7
58.22
39.21
#381
60.25
42.83 +2.60
CLSER
61.37 +0.69
47.75
+0.70
54.92
40.51
11.1
56.60=151
47.48 +0.70
CBRS
59.07 11.78
43.81
10.13
57.57
1.60
44.20
1.96
58.16
44.66 13.21
Coresets
61.11 11.34
45.37
10.8
58.12
11.37
45.80
1.89
58.56 2117
45.63 1
La-MAML
36.64
29.17
+091
32.08
+237
31.17
42.52=281
31.24
VR-MCL
65.06 +0.85
49.82
+ 1.133
61.16
51.36
61.91 =1.7
50.74 + 1.
Table 14:
Reduced ResNet-18 performance under the random imbalanced scenario with larger im- balance ratio, i.e., 7 = 5 (Random).
Method Acc
CLS-ER
DER++
La-MAML
VR-MCL
35.66 1.56
40.93 10669
27.46 14.65
41.55 13115
AAA
53.93 #0.61
49.98+254
36.85 +133
54.52
27

---

Published as a conference paper at ICLR 2024
Table 15: Reduced ResNet-18 performance on the imbalanced Seq-CIFARI00 ([M]=1000) with imbalance ratio y=2.
Setting
Method
CLSER
DER++
La-MAML
VR-MCL
Acc
13.8
9.19 1.22
15.12 11.19
7.
AAA
2.004
18.78
19.34
21.1
Acc
+0.89
9.66
+1.66
14.98 1166
7 = 2 (Reverse)
AAA
1.22
15.60
16.04
+1.6
11.16
Acc
13.42 1.48
1.609
11.32
14.93
10.27
= (Random)
AAA
19.10
15.55 20.25
16.22
20.13 0.557
D.5 OTHER RESULTS USING THE PCCNN BACKBONE
In Table 16, we further investigate the performance gain the memory buffer size the small network PeCNN. Combined with the results on reduced ResNet18, it is evident that the proposed VR-MCL has consistent and significant performance improvement in different buffer sizes and various network structures.
Table 16: Performance of Seq-CIFARIO with 95% confidence interval PcCNN. All numbers the average of 5 runs. |M denotes the memory buffer size.
200
|
Method
AAA
Acc
AAA
Acc
AAA
Acc
SGD
+0.03
17.58 +0.20
37.84 =0.33
0020 17.58
37.84 +.00
17.58
+0.20
On-EWC
38.72+0058
17.05
38.72
17.05
+0.08
38.52
17.05
+0.08
A-GEM
41.84.10.4
18.50
41.54.0.14
18.45
10.37
41.73 1061
18.78 1.038
GEM
45.07 10.52
22.5211.00
45.11
10.34
22.45
1089
44.92 1:020
22.10 1.060
ER
54.64
3..4
58.85 =082
40.29
59.26
41.71
DER
58.06
38.21 +0029
58.40
38.46
57.90
38.88 +0.39
DER+
56.07
34.28
62.13
43.89 +0.0
63.55
47.20
CLSER
58.03 +0.75
39.38 +0.75
61,53-0.3
44.94
62.39
10.34
48.24 1.88
MER
55.50
32.01
11.2
63.20
44.69
20.43 20.43
65.75 +0.6
$1,04:077
La-MAML
46.36
10.9
29.45
10.51
47.22 1087
32.78 1.53
47.22
1087
32.51
& L188
VR-MCL
60.88 10.22
43.41
0.30
63.86
48.85
66.02 10311
52.67 10.25
D.6 OFFLINE CL PERFORMANCE
As shown Table 17. also provide the performance of different methods under the offline CL setting where we set the training epochs as 5.
D.7 INFLUENCE OF THE INNER BATCH SIZE
To investigate the practical implications of the inner batch size, we conducted experiments using various inner batch sizes. The corresponding results presented in Table 18. The findings indicate that a larger inner batch size consistently leads to improved performance. However, it is worth noting that the performance gains tend to plateau as the inner batch size increases. This can be attributed to the online nature of the setting, where each sample is observed only once. A larger inner batch size reduces the number of outer loop steps, which may potentially compromise the overall performance improvement.
E ALGORITHM.
As presented in Algo. 1, Steps 1-9 present the update of in the first iteration, where momentum term is available. Steps 2-6 present the inner loop and steps 7-8 present the estimate of hypergradient in the outer loop. Steps 10-28 represent the update of 0 in b-th iteration (b 1). Steps 12-17
28

---

Published as a conference paper at ICLR 2024
Table 17: Offline CL (i.e., training epochs) performance of Seq-CIFARI0 and longer task se- quences CIFARIIIOOO Boq-TinylmageNer with 95% confidence interval on reduced ResNet- 18, The memory buffer size is set as 1000 (i.e., |||=1000).
CIQ-CIFARIO
Seq-CIFARII 100
Seq-TinylmageNet
Method
AAA
Acc
AAA
Acc
AAA
Acc
SGD
42.26
18.87 10.66
18.73
7.73 10.66
16.89
6.03
On-EWC
42.13
1..76
16.99
5.34
14.89 #0.14
4.85
IS
39.84
17.49
15.56
10.92 4.98
2.65
A-GEM
42.13
19.12
18.69
1.44
16.73
6.50
GEM
48.28 1.83
23.75 1:391
21.56
10.71 1:2.11
18.12 20111
7.98
ER
74.64.1..8
63.97 10.27
35.37 1:336
1.774
26.49 10.38
12.47 1023
DER
1.54
$9.95
18.73
7.67
16.96
6.68 +022
DER+
75.50
1..1
39.15
24.03
25.91 #0.63
11.54
CLSER
+5.45
67.36
4..1
26.16
26.24
12.98 10.10
ER-OBC
+.99
1.42
41.15
29.92
29.01 #0.33
16.27 =0.44
La-MAML
60.55
1.0.48
43.58
10.56
31.05
10.28
19.67 1.46
23.07 10.71
11.81 1..55
VR-MCL
77.15 1.7.5
67.34
40.02 1..99
29.32 0.94
30.43
19.60 0.58
Table 18: Reduced ResNet-18 performance under different inner batch sizes.
Inner hatch size
6
8
10
Acc
53.80
2.36
56.48 + 1.79
56.81 1
AAA 65.73
3.06
66.97 1 1.58
67.26
the inner loops. Steps 18-19 show the computation of the hypergradient for 0g. In steps 20-25, we perform the inner loop to obtain (o(K) and compute the hypergradient for 0, g g During the update of memory buffer A (step 28), we employ the reservoir sampling strategy to ensure that M is updated in way that the stored examples are uniformly sampled from the tasks during online training.
E. VARYING TASK LENGTH ANALYSIS
To further demonstrate the effectiveness of the proposed VR-MCL across varying task lengths, adjust the split mechanism of Seq-CIFAR 100 and Seq-TinylmageNet to increase their original task numbers. Specifically, we restructure CIFARI0O,,, originally composed 10 tasks with 10 classes each, into 20 tasks, each with 5 classes. Similarly, we modify Seq- TinyImageNet, initially consisting of 20 tasks with classes each, into 40 tasks, each comprising classes. For elarity, we refer to these new division datasets as Seq-CIFAR100 100' and Seq- Tiny ImageNer respectively. The performance of various baselines on these two datasets is presented in Table 19.
F.2 EMPIRICAL RESULTS OF THE OUTER-LOOO Loss
The cuter-lass during triining
The oaben loss during training
The outeriies during traihing in r
The otter-ines during training
Outeeloop
Datre hoop
25
1M
Traning slo33
Training Jtpps
Teannig stegs
Tratinn
Figure 7: The outer-loop training loss for each task T in Seq-CIFARIO (j = 2, 3,3,4,5)
29

---

Published as a conference paper at ICLR 2024
Algorithm The Algorithm of the proposed VR-MCL.
Require:
Initial model weights 0, initial inner-loop learning rate 3, outer-loop learning rate number of inner-loop steps K, memory buffer M, and the task number total N.
1: Sample training data T(e) (s I.... K) from current task T!
2: ((0) 0, 0u-
3: for i = 1 to K do
4: obtain samples 7
5: (s)) = ( VV (0))
6: end for
7: sample en from M U
8: 8%, (o)(
9: 01 0o 0gg,
10: for j = to do
11: for (j-1) x
to jxB do
sample training data T/o (s (s= L, ... K) from current task T
13:
Ou(o) - 0,
14: for 1 to K do
15:
obtain samples T/d)
16:
(o(e) = Boti 1) 3VC (0)( (0)(tt-))))))
17:
end for
sample e, from MUTi gø; ()) ())
19:
20:
(N)) = 0--
21:
for 1 to K do
22:
the same sampled training data To
23:
0bee) 0 - - LV (0
24:
end for
25:
go
VL((0k))
26:
g. = go
+ r(
(6-
86h_1
27:
5 0-1 0, agh
28:
Update the memory buffer M with Tid using reservoir sampling strategy.
end for
30:
end for
Table 19: Experimental results under varying task lengths with [ = 1000 and a 95% confidence interval. The Notation N denotes the task length.
Seq-CIFAR-1001 (N=20)
Beq-TinylmageNer (N==0) Aec
Method
AAA
Ace
AAA
SGD
9.23 10.26
3.34
4.95 10.16
1.19 10.27
A-GEM
10.84 +0023
3.56
6.10
1.67 +0.07
GEM
16.0442.55
7.01 1.95
7.9240.11
2.93 10.38
ER
20.46 1:0.48
12.71 40.28
13.75 10.12
6.82 20.11
DER+
16.32
8.24
+0.07
3,76
CLSER
20.9 +096
15.39
14.93
7.74
ER-OBC
21.04 10.65
15.87 1.22
14.92 .10.20
8.45 1.229
La-MAML
17.42 1.0.9
10.27 1..46
10.83 1.0.9
5.29 10.28
VR-MCL
24.29 11.0
17.44 +0.97
18.28 10.14
10.54 00.30
Fig. illustrates the value of (((s)(( during the training of each task T CIF-C1 (j=2,3,4,5). The loss (( exhibits a significant decrease within an average of outer loop update steps, suggesting close proximity between (c)) and 0* stated in Proposition 2.
30


Reviews:
[{"review_id": "jDqahNjYrv", "reviewer": "Reviewer_6aKN", "summary": "The paper introduces a novel approach called VRMCL (Variance Reduced Meta Continual Learning), integrating a hyper-gradient variance reduction technique for Meta Continual Learning (CL). Furthermore, it offers theoretical regret bounds for the proposed method. The paper extensively evaluates the VRMCL method across three datasets, with diverse continual learning scenarios.", "questions": "1. Regarding the algorithm, the paper mentions that the memory buffer is updated to ensure a balanced storage of tasks. Could you provide more details on how this task-balancing process is implemented within the algorithm?\n2. It would be valuable to include additional experiments as mentioned earlier, especially those assessing the method's performance under scenarios involving varying task lengths across each datasets.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clarity: The paper is well written and easy to follow.\n2. Technical Proficiency: The paper showcases a highly technical.\n3. Originality and Novelty: The paper introduces a novel concept focused on diminishing variance in gradient computations concerning memory buffers in online settings.\n4. Comprehensive Empirical Validation: The paper includes extensive experiments and comprehensive ablation study which support the claims made in the paper.", "weaknesses": "1. Limited Comparison:\n    1. While the authors have made comparisons with recent baselines, the paper could benefit from a more extensive comparison by including well-established methods such as FTML[1] and LFW[2]. A broader comparison would provide a more comprehensive evaluation of the proposed method's strengths and weaknesses.\n2. Limited Experimental Width:\n    1. Although the authors have conducted evaluations on popular datasets like CIFAR10, CIFAR100, and TinyImageNet, it would be good to test the effectiveness of the proposed method on larger datasets, such as ImageNet-1K. This would offer insights into the algorithm's performance in handling catastrophic forgetting in longer sequences.\n    2. Additionally, the experiments could be enhanced by varying the number of tasks on each dataset, thereby showcasing the adaptability of VR-MCL under different task configurations.\n3. Lack of Memory Update Strategy Explanation:\n    1. The paper could benefit from a more thorough explanation of the memory update strategy employed in the VR-MCL algorithm. Given the algorithm's reliance on the Memory Buffer, a clearer and more detailed description of the update mechanism is essential to provide a comprehensive understanding of the methodology.\n\n[1] Finn, C., Rajeswaran, A., Kakade, S., & Levine, S. (2019, May). Online meta-learning. In International Conference on Machine Learning (pp. 1920-1930). PMLR.\n\n[2] Li, Z., & Hoiem, D. (2017). Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12), 2935-2947.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Gg9o4uHE6", "reviewer": "Reviewer_1xci", "summary": "This paper focused on the branch of Meta-Continual Learning (Meta-CL) methods in the context of Continual Learning (CL). By characterizing the Meta-CL algorithms as a new perspective of up-to-date Hessian matrix approximation, the authors tried to bridge the gap between the Meta-CL and the regularized-based CL methods. Under this viewpoint, Meta-CL implicitly approximated the Hessian in an online manner through the use of hypergradient in the bi-level optimization process. To address the erroneous information during the Hessian estimation due to the sampling process from the random memory buffer, the authors proposed Variance Reduced Meta-CL (VR-MCL) to control the high variance of the hypergradient under online continual learning. With a theoretical analysis, the authors showed that the proposed VR-MCL is equivalent to the inclusion of a penalty term within the implicit Hessian estimation in Meta-CL. The experimental results on three benchmarks indicated that the proposed method outperformed the regularization-based and Meta-CL baselines.", "questions": "1. I wonder whether the assumptions during the mathematical derivation always hold in the practical scenarios. For example:\n   - In Proposition 2, the authors assumed that $\\theta_{(K)}$ is lolcated in the $\\epsilon$-neighbourhood of the optimal model parameter. Is it too strong?\n   - In Proposition 3, the authors assumed that the batch size of the inner step adaptation is sufficiently large. I wonder how large is enough to make the following analyses hold. And how did the authors set it during the practical training?\n2. The motivation for the evaluations under the imbalance CL setting was not clear to me. I did not get the relationship between the superior performance under this setting and the main objective of this paper. Or does the author just intend to show that the proposed method could still perform well under this challenging setting? Besides, it was disappointing to see that the authors did not provide further analyses about why the proposed method could address this challenging setting.\n3. In Proposition 2, the authors mentioned the assumption of $\\beta$. However, it was not contained in the final main conclusion.\n4. After Eqn.(4), $G_{\\theta_{b}}$ appeared without further explanations, which made the reader fail to have a straightforward comprehension of the meaning of $\\Delta_{b}$.\n5. How about the time and memory complexity of the proposed method compared to the baseline approaches, especially the Meta-CL methods, like LA-MAML? Could the authors provide quantitative comparisons? I believe such a comparison will help the readers to better understand the superiority of the proposed VR-MCL.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this work is clear and easy to follow.\n2. It is interesting to see that an inherent connection can be built between the regularization-based methods and Meta-CL methods via the roles of the Hessian information in these two methodological streams.\n3. This work provided theoretical analyses and empirical verifications to help to better understand the motivation.", "weaknesses": "1. Most parts of the mathematical derivations are easy to follow. However, some detailed notations are not clear in the context, which reduces the readability.\n2. The motivation of some experimental designs was not too clear, such as the imbalance CL setting.\n3. It seems the math derivation process needs some strict assumptions. I doubt the gap between the theoretical findings and the empirical applications. \n\nSee the Questions part for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ruFBZjAAUd", "reviewer": "Reviewer_Z5So", "summary": "In this paper, the author revisited the methodology of Meta-Continual Learning (Meta-CL) and, for the first time, provides a formal connection between meta-continual learning and seminal regularization-based methods (like Elastic Weight Consolidation (EWC)) which mainly exploits the empirical Hessian matrix to provide the regularization to counter forgetting. The main finding is that Meta-CL methods implicitly utilize the second-order Hessian information through the hypergradient obtained by bi-level optimization for meta-learning. From this new perspective, the author further points out the issue existing in the methodology of Meta-CL, i.e., the presence of erroneous information in the Hessian information due to insufficient memory data. To resolve the problem, the author correspondingly proposes a momentum-based Variance-Reduced Meta-CL (VR-MCL) method and provides extensive theoretical analysis to demonstrate how the proposed method can impose a penalty on the online estimated Hessian such that the model can be updated with caution to preserve crucial parameters. Extensive experiments are conducted on standard continual learning benchmarks, and the proposed theoretical method outperform both representative and state-of-the-art (SOTA) continual learning methods.", "questions": "Please refer to the Weaknesses for more details.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The reviewer really enjoys reading this paper. This should be the first paper that formally and clearly dissects the relationship between seminal regularization-based methods and the methodology of meta-continual learning. The key message and insights are conveyed smoothly in the whole paper, and the author does a really good job of presenting them in a decent way. Table 1 provides a very precise and clear summary and comparison of the seminar and the state-of-the-art regularization-based method for the reader to get their main idea in common, making it easier for the reader to comprehend the novelty and contribution made by the present paper. Figures 1 and 2 are also compact and reduce the difficulties for the reader to understand the technical details of the iterative update process, which also highlights the difference made in this paper.\n\nAs the Hessian information is widely used not only in continual learning but also in many different areas of deep learning (e.g., meta-learning and flatness-aware optimization), the reviewer believes that the theoretical findings provided by this paper may not only motivate novel methods on Meta-CL but may also motivate novel methods for other areas in general.\n\n2. The unification of the Meta-CL and Regularization-based method is sound. Although there exist papers that try to unify different regularization-based CL methods in a unified framework, the CL methods they considered are mainly for CL in a fully-supervised setting, to the best of my knowledge, this paper should be the first one to connect the regularization-based CL methods with the methodology of Meta-CL, which may stand as a new research direction in the future.\n\n3. The reviewer also appreciates the understanding provided by the author in Section 4.2 after Proposition 3. It is refreshing to see that the variance-reduce method can ensure cautious updates such that the model can prevent excessive updates triggered by the wrongly estimated low-curvature direction of the Hessian, which may mitigate the partiality and erroneousness in the insufficient memory data, which should also be a desideratum about the kind of model update we should purse for. The insight may also motivate future work in continual learning and may also in areas like parameter-efficient finetuning.\n\n4. The extensive comparison with state-of-the-art methods in both CL and Meta-CL further demonstrates the significance and effectiveness of the proposed method. The questions listed in each subsection of the Experiments section provide good guidance for the reviewer to focus on and reason about the results. It is also great to see that the author also conducts many empirical analyses in both the main paper and supplementary to validate the correctness of the proposed theorem.", "weaknesses": "1. In Proposition 3, the author assumes that the batch size for inner step adaptation is sufficiently large. How do we quantify the term \"sufficiently large\" in reality? Is there any principle we can obtain from the proposed theorem to guide us in choosing the batch size?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)