Title: Bilevel Coreset Selection in Continual Learning: A New Formulation and Algorithm
Venue: NeurIPS 2023
Accepted: yes
Keywords: Coreset Selection

Continual Learning

Bilevel Optimization
Abstract: Coreset is a small set that provides a data summary for a large dataset, such that training solely on the small set achieves competitive performance compared with a large dataset. In rehearsal-based continual learning, the coreset is typically used in the memory replay buffer to stand for representative samples in previous tasks, and the coreset selection procedure is typically formulated as a bilevel problem. However, the typical bilevel formulation for coreset selection explicitly performs optimization over discrete decision variables with greedy search, which is computationally expensive. Several works consider other formulations to address this issue, but they ignore the nested nature of bilevel optimization problems and may not solve the bilevel coreset selection problem accurately. To address these issues, we propose a new bilevel formulation, where the inner problem tries to find a model which minimizes the expected training error sampled from a given probability distribution, and the outer problem aims to learn the probability distribution with approximately $K$ (coreset size) nonzero entries such that learned model in the inner problem minimizes the training error over the whole data. To ensure the learned probability has approximately $K$ nonzero entries, we introduce a novel regularizer based on the smoothed top-$K$ loss in the upper problem. We design a new optimization algorithm that provably converges to the $\epsilon$-stationary point with $O(1/\epsilon^4)$ computational complexity. We conduct extensive experiments in various settings in continual learning, including balanced data, imbalanced data, and label noise, to show that our proposed formulation and new algorithm significantly outperform competitive baselines. From bilevel optimization point of view, our algorithm significantly improves the vanilla greedy coreset selection method in terms of running time on continual learning benchmark datasets. The code is available at https://github.com/MingruiLiu-ML-Lab/Bilevel-Coreset-Selection-via-Regularization.

Bilevel Coreset Selection in Continual Learning: A New Formulation and Algorithm

Abstract

Coreset is a small set that provides a data summary for a large dataset, such that training solely on the small set achieves competitive performance compared with a large dataset. In rehearsal-based continual learning, the coreset is typically used in the memory replay buffer to stand for representative samples in previous tasks, and the coreset selection procedure is typically formulated as a bilevel problem. However, the typical bilevel formulation for coreset selection explicitly performs optimization over discrete decision variables with greedy search, which is computationally expensive. Several works consider other formulations to address this issue, but they ignore the nested nature of bilevel optimization problems and may not solve the bilevel coreset selection problem accurately. To address these issues, we propose a new bilevel formulation, where the inner problem tries to find a model which minimizes the expected training error sampled from a given probability distribution, and the outer problem aims to learn the probability distribution with approximately K (coreset size) nonzero entries such that learned model in the inner problem minimizes the training error over the whole data. To ensure the learned probability has approximately K nonzero entries, we introduce a novel regularizer based on the smoothed topK loss in the upper problem. We design a new optimization algorithm that provably converges to the ϵ -stationary point with O (1 /ϵ 4 ) computational complexity. We conduct extensive experiments in various settings in continual learning, including balanced data, imbalanced data, and label noise, to show that our proposed formulation and new algorithm significantly outperform competitive baselines. From bilevel optimization point of view, our algorithm significantly improves the vanilla greedy coreset selection method in terms of running time on continual learning benchmark datasets. The code is available at https://github.com/MingruiLiu-ML-Lab/ Bilevel-Coreset-Selection-via-Regularization .

1 Introduction

Deep Neural Networks (DNNs) have achieved tremendous successes in various domains, including computer vision [41, 30], natural language processing [72, 15], generative modeling [26] and

Kaiyi Ji

Task 1

Task 2

Task

T

…

…

Buffer

…

1.0

0

③

②

④

⑤

①

①

⑥

…

games [68]. However, in continual learning, where DNNs are trained on a sequence of tasks with possibly non-i.i.d. data, the performance will be degraded on the previously trained tasks. This is referred to as catastrophic forgetting [53, 52, 60]. To alleviate catastrophic forgetting, one of the effective ways is rehearsal-based continual learning , where a small replay buffer is maintained and revisited during the continuous learning process. There is a line of works studying how to efficiently maintain the replay buffer using the coreset selection approach [6, 78, 83], in which a small set of data is selected as representative samples to be used in continual learning.

The coreset selection in continual learning is formulated as a cardinality-constrained bilevel optimization problem which is solved by incremental subset selection [6]. This greedy approach is computationally expensive and hence is not scalable when the coreset size is large. To address this issue, Zhou et al. [83] propose a relaxation of the bilevel formulation in [6], which drops the nested nature of bilevel formulation and actually becomes two sequential optimization problems. Tiwari et al.proposed a gradient approximation method in [71], which selects a coreset that approximates the gradient of model parameters over the entirety of the data seen so far. Yoon et al. [78] proposes an online coreset selection method by maximizing several similarity metrics based on data pairs within each minibatch, and sample pairs between each minibatch and coreset. These approaches do not directly address the algorithmic challenges caused by the nested nature of the bilevel optimization problem, and may not solve the original bilevel coreset selection problem efficiently.

The key challenges in the bilevel coreset selection problems are two folds. First, the bilevel formulation in [6] needs to directly perform optimization over cardinality constraint, which is a nonconvex set and greedy approaches are expensive when the coreset size is large. Second, the bilevel formulation in [6] has a nested structure: one problem is embedded within another, and the outer and inner functions both have dependencies over the same set of decision variables. It remains unclear how to design efficient algorithms to solve constrained bilevel optimization algorithms for the coreset selection problem with provable theoretical guarantees.

Our proposed solution addresses these challenges with a novel bilevel formulation and provably efficient optimization algorithms. The proposed new bilevel formulation is referred to as Bilevel Coreset Selection via Regularization (BCSR). The main differences between our approach and the standard bilevel approach in [6] are: (i) unlike the standard bilevel formulation which requires performing optimization based on a cardinality constraint, we propose to solve a bilevel optimization on a probability simplex over training examples; (ii) to make sure the probability distribution lies in a low dimensional manifold, we propose to add a smoothed topK loss as a regularizer to the upper-level problem; (iii) due to our new formulation, we are able to design a simple and effective first-order method to solve this new bilevel problem with provable non-asymptotic convergence guarantees. The first-order method is easy to implement and much faster than the greedy approach as in [6]. Our main contribution is listed as follows.

Algorithm 1 PyTorch-style pseudocode for BCSR

We propose a new bilevel formulation, namely BCSR, for the coreset selection in rehearsalbased continual learning. Instead of directly learning the binary masks for each sample, the new formulation tries to learn a probability distribution in a low-dimensional manifold by adding a smoothed topK loss as a regularizer in the upper problem. This formulation is designed to satisfy two important features in continual learning with DNNs: (i) keeping the nested structure in the coreset selection; (ii) being amenable to first-order algorithms, which makes it easy to implement in modern deep learning frameworks such as PyTorch and TensorFlow. Based on the new formulation, we propose an efficient first-order algorithm for solving it. The main workflow of our algorithm is illustrated in Figure 1, and the corresponding Pytorch-style pseudocode is presented in Algorithm 1.

We have conducted extensive experiments among various scenarios to verify the effectiveness of our proposed algorithm, including balanced, imbalanced, and label-noise data. Our algorithm outperforms all baselines for all settings in terms of average accuracy, and it is much better than all other coreset selection algorithms. For example, on imbalanced data of Multiple Datasets, BCSR is better than the best coreset selection algorithm by 4 . 65% in average accuracy. From bilevel optimization point of view, our algorithm significantly improves the vanilla greedy coreset selection method [6] in terms of running time on continual learning benchmark datasets.

Under the standard smoothness assumptions of the loss function, we show that our algorithm requires at most O (1 /ϵ 4 ) complexity for finding an ϵ -stationary point in the constrained case 2 . Notably, the O (1 /ϵ 4 ) complexity consists of O (1 /ϵ 2 ) backpropagations and O (1 /ϵ 4 ) samplings from Gaussian distribution, where the latter cost is computationally cheap.

2 Related Work

Continual Learning There are different classes of continual learning methods, including regularization-based approaches [39, 81, 10, 1, 59, 64, 17], dynamic architecture methods [61,

79, 62, 51, 76, 44, 77], and rehearsal-based methods [48, 57, 11, 58, 31, 3, 18, 28, 80, 6, 83, 78, 84]. In the rehearsal-based continual learning, the memory is either reproduced experience replay [48] or generative replay [67]. Our work focuses on the aspect of the coreset selection on the replay memory and can be flexibly integrated into rehearsal-based methods in continual learning.

Coreset Selection The coreset selection methods were used frequently in supervised and unsupervised learning, such as k -means [19], Gaussian mixture model [49], logistic regression [33] and bayesian inference [9]. They were also used frequently in the active learning literature [74, 63]. The coreset selection in continual learning is related to the sample selection [34, 2, 3]. Nguyen et al. [55] introduce variational continual learning which is combined with coreset summarization [5]. Borsos et al. [6] proposed the first bilevel formulation for the coreset selection in continual learning, which is later improved by [83, 78]. Compared with these works, our work focuses on improved bilevel coreset selection: we provide a better bilevel formulation than [6] and design a provably efficient optimization algorithm.

Bilevel optimization Bilevel optimization is used to model nested structure in the decision-making process [73]. Recently, gradient-based bilevel optimization methods have broad applications in machine learning, including meta-learning [20], hyperparameter optimization [56, 22], neural architecture search [46], and reinforcement learning [40, 32]. These methods can be generally categorized into implicit differentiation [16, 56, 45, 4] and iterative differentiation [50, 21, 20, 65, 27] based approaches. Recently, various stochastic bilevel algorithms have been also proposed and analyzed by [12, 37, 25, 32, 29, 4, 14]. A comprehensive introduction can be found in the survey [47]. In this work, we propose a novel stochastic bilevel optimizer with very flexible parameter selection, which shows great promise in the coreset selection for continual learning.

3 New Bilevel Formulation for Coreset Selection in Continual Learning

In this section, we first introduce our new bilevel formulation, namely Bilevel Coreset Selection via Regularization (BCSR). The key idea of this approach is to learn a probability distribution over the whole dataset such that the best model parameter obtained by minimizing the loss on the sampled dataset (i.e., the minimizer for the lower-level problem) is also the best for the whole dataset (i.e., the minimizer for the upper-level problem), and then a coreset can be sampled based on the learned probability distribution. In addition, the learned probability distribution is expected to lie in a low dimensional manifold (i.e., with K nonzero entries where K is the coreset size). To achieve this, we added a smooth topK loss as a regularizer to promote the probability distribution to have K nonzero entries. Specifically, the objective function of BCSR is:

where n is the sample size, θ is the model parameter, w is the sample weights, ℓ i ( θ ) denote the loss function calculated based on i -th sample with model parameter θ , w ( i ) is the i -th coordinate of w , w [ i ] is the i -th largest component of w , λ > 0 is the regularization parameter, w + δz denote to adding δz on each coordinate of w where z ∼ N (0 , 1) . Note that R ( w,δ ) := -λ ∑ K i =1 E z ( w + δz ) [ i ] denote the smoothed topK regularization. We add this regularization to make sure the summation of the topK entries of the learned probability vector is large, such that we can confidently choose a coreset with size K . The goal of employing Gaussian noise to the regularizer is for the ease of algorithm design: this Gaussian smoothing technique can make the regularizer to be smooth such that it is easier to design efficient first-order bilevel optimization solvers. Otherwise, the upper-level problem would become nonconvex and nonsmooth, and it would be difficult for algorithm design under this case.

Discussion and Comparison with Prior Works In this part, we illustrate how this new formulation addresses the drawbacks of the previous approaches. The work of [6] does not use any regularizer and regards the weight of each sample as a binary mask. This formulation needs to solve a combinatorial

Algorithm 2 Bilevel Coreset Selection via Regularization (BCSR)

Input

: Dataset D

Initialize

: model parameter θ 0 , memory M = {}

1: for batch B t ∼ D do

2: Compute coreset S t = Find-coreset( θ t - 1 , B t )

3: M = M∪S t

4: end for

Algorithm 3 Find-coreset ( θ, B )

Input

: the current model parameter θ , the current batch B , the iteration parameters J , N , Q

Initialize : coreset size

K , n = |B| , and v 0

1: w 0 = [ 1 n , . . . , 1 n ] (uniform probability initialization)

2: θ 0 1 = θ

3: for j = 0 , 2 , . . . , J - 1 do

4: Set initialization θ 0 j = θ N j - 1 if j > 0 and θ 0 1 otherwise

5: for k = 1 , . . . , N do

6: update θ k j according to eq. (2).

7: end for

8: Set initialization v 0 j = v Q j - 1 if j > 0 and v 0 otherwise

9: Compute estimate v Q j by solving QP in (4) by GD with stepsize η and inital v 0 j = v Q j - 1 .

10: Compute hypergradient estimate in (3)

11: Update w j +1 and project it onto simplex by (5)

12: end for

13: S ← a set of K data points sampled from B according to the probability distribution w J

14: Return S

optimization problem and their approach of incremental subset selection is computationally expensive. The work of [83] relaxes the bilevel formulation in [6] to minimize the expected loss function over the Bernoulli distribution s , i.e., min s ∈C Φ( s ) , and develops a policy gradient solver to optimize the Bernoulli variable. Their gradient ∇ s Φ( s ) = E p ( m | s ) L ( θ ∗ ( m )) ∇ s ln p ( m | s ) does not include the implicit gradient of L ( θ ∗ ( m )) in terms of s . However, θ ∗ ( m ) actually depends on the mask m , and m depends on the Bernoulli variable s . In contrast, our bilevel optimization computes the hypergradients for the coreset weights w ( 0 ≤ w ≤ 1 and ∥ w ∥ 1 = 1 ), which considers the dependence between θ ( w ) and w 3 . In addition, Zhou et al. [83] assume that the inner loop can obtain the exact minimizer θ ∗ ( m ) , which may not hold in practice. In contrast, we carefully analyze the gap between the estimated θ ∗ ( w ) and itself by our algorithm and analysis.

4 Algorithm Design

Equipped with the new formulation, the entire algorithm is presented in Algorithm 2, which calls Algorithm 3 as a subroutine. Each time the algorithm encounters a minibatch B , a coreset is selected within this minibatch by invoking Algorithm 3. Algorithm 3 is a first-order algorithm for solving the bilevel formulation (1). In Algorithm 3, the model parameter θ and the weight distribution w are updated alternatively. We first perform N steps of gradient descent steps to find a sufficiently good θ for the lower-level problem (lines 5-7) by the update rule:

where θ k j denotes the model parameters at the j -th outer loop and the k -th inner loop. To update the outer variable w , BCSR approximates the true gradient ∇ ϕ ( w ) of the outer function w.r.t w , which is called hypergradient [56]. BCSR constructs a hypergradient estimator:

where R ( w,δ ; ˜ z ) := -λ ∑ K i =1 ( w + δ ˜ z ) [ i ] and ˜ z ∼ N (0 , 1) . Solving the Hessian-inverse-vector product in eq. (3) is computationally intractable. We denote v ∗ := ( ∇ 2 θ L ( θ N j , w j )) -1 ( ∑ n i =1 ∇ θ ℓ i ( θ N j )) in eq. (3), where v ∗ can be approximated by solving the following quadratic programming problem efficiently by Q steps of gradient descent (line 9):

Next, the hypergradient estimate (line 10) is computed based on the output of approximated quadratic programming. Note both model parameters and sample weights need to use warm start initialization (line 4 and line 8). Then the weight is updated and projected onto simplex (line 11):

where ∆ n := { w ∈ R n : 0 ≤ w ( i ) ≤ 1 , || w || 1 = 1 } . In terms of other experimental hyperparameters, we allow very flexible choices of hyperparameters (e.g., N , Q ) as shown in our theory to achieve polynomial time complexity for finding a ϵ -stationary point.

The selected coresets for each task are stored in a memory buffer with a fixed size m . There is a separated memory slot for each task with size [ m/i ] when the task i comes. After each task i , the memory slots before i will randomly remove some samples to adjust all the memory slots to [ m/i ] . That means the memory size for each task decreases as the task ID increase to maintain the total buffer size m unchanged. The same memory strategy is also used in the greedy coreset approach [6].

5 Experiments

We conduct extensive experiments under various settings, including balanced data, imbalanced data, and label-noise data. The empirical results demonstrate the effectiveness of our method in rehearsal-based continual learning.

5.1 Experimental Setup

Datasets We use commonly-used datasets in the field of continual learning, including Split CIFAR100, Permuted MNIST, Multiple Datasets, Tiny-ImageNet, and Split Food-101. We follow the experimental settings as that in prior work [78] and [83]. Each dataset is processed with three approaches: balanced, imbalanced, and label-noise. Please refer to Appendix L for more details about data processing and settings.

Baselines We compare our algorithm BCSR with other continual learning methods based on coreset strategy, including k -means features [55], k -means embedding [63], Uniform Sampling, iCaRL [57], Grad Matching [9], Greedy Coreset [6], PBCS [83], GCR [71], and OCS [78]. We also compare with non-coreset reply method, SPR [38], MetaSP [70]. All algorithms are built upon episodic memory, which stores coreset selected from stream data. Then a model, such as ResNet-18 [30], is trained over the data from the current stream and the episodic memory.

Metrics Average accuracy and forgetting measure [10] are two primary evaluation metrics that are used in continual learning literature. AVG ACC ( A T ) is the average accuracy tested on all tasks after finishing the task T : A T = 1 T ∑ T i =1 a T,i , where a T,i is the test accuracy of task i after training

the task T . FGT T evaluates the performance drop on the past tasks after training on the task T : FGT T = 1 T ∑ T j =1 [max j ∈ 1 ,...T -1 ( a j,i -a T,i )] .

Implementation Details Following the implementation of the coreset-based algorithm [6], we use two models for the purposes of model training and coreset selection respectively, where the first model is denoted as M tr for model training, and the second model (also known as a proxy model) is denoted as M cs for coreset selection. In the section of the model training, we adopt a single-head MLP with two hidden layers for Permuted MNIST, and a ResNet-18 for Split CIFAR-100 and other datasets. In the section of the coreset selection, M cs adopts the same architecture as M tr . This is the key difference between our work and the Greedy Coreset [6]: they use Neural Tangent Kernels (NTK) [35] while we use a specific deep neural network. Note that NTK performs learning based on fixed features, which is shown to be limited [66]. In contrast, our algorithm allows hierarchical feature learning during the training process. In addition, our algorithm does not rely on discrete decision variables as in [6]: we use first-order methods with gradient and Hessian-inverse-vector product information with convergence guarantee and hence is more efficient in practice. For noncoreset reply methods SPR and MetaSP, we set the size of the method buffer as that in coreset-based methods. Especially for SPR, there are two memory buffers: the delayed buffer D temporarily stocks the incoming data stream, and the purified buffer P maintains the cleansed data. To satisfy the

0.4

0.6

0.8

0 1 2 3 4 5 6 7 8 9 10111213141516171819

Tasks

50

60

70

80

AVG ACC

(a) Permuted MNIST

0 1 2 3 4 5 6 7 8 9 10111213141516171819

Tasks

20

30

40

50

60

70

AVG ACC

(b) Imbalanced Permuted MNIST

0 1 2 3 4 5 6 7 8 9 10111213141516171819

Tasks

40

50

60

70

80

AVG ACC

(c) Label-Noise Permuted MNIST

K-means Features

K-means Embedding

Uniform

iCaRL

SPR

MetaSP

Greedy Coreset

GCR

PBCS

OCS

BCSR

1.0

Figure 2: The average accuracy during the continual learning. After each task, the training model is tested on all encountered tasks. Due to the nature of forgetting, the average test performance of all the methods tends to decrease.

requirement of coreset experiments, we keep the size of P and D buffers the same. The detailed hyperparameter settings can be found in Appendix C.

5.2 Results

We report the results of average accuracy (AVG ACC) and forgetting (FGT) for all the algorithms on the balanced, imbalanced, and label-noise benchmarks, and the results are presented in Table 1, Table 2, Table 3, Table 4, and Table 9 (Appendix E) respectively. We have the following observations. (i) In the balanced setting, our method outperforms other baselines significantly in terms of A VG ACC. For example, compared with the best coreset selection methods, our BCSR shows 2 . 21% , 4 . 24% , 2 . 68% , 2 . 11% , and 1 . 7% improvements in AVG ACC on five benchmarks, respectively. (ii) In the imbalanced and label-noise setting, our method also demonstrates relatively higher performance. For example, on imbalanced data of Multiple Datasets, BCSR is better than the best coreset selection algorithm by 4 . 65% in average accuracy. An interesting observation is that Greedy Coreset [6] does not perform very well, especially in imbalanced and label-noise settings. The reason is that the inner optimization in Greedy Coreset is conducted by an NTK, where only a fixed feature is used but not the learned feature. (iii) From bilevel optimization point of view, our algorithm significantly reduces the running time of the vanilla greedy coreset selection method [6] by at least 58% on continual learning benchmark datasets. For detailed comparison, please check Table 8 in Appendix D. (iv) In addition, the test A VG ACC (Figure 2 and Figure 4 in Appendix F) of the BCSR during the training process shows that our algorithm alleviates catastrophic forgetting and it is comparable to other best baselines. For example, BCSR enjoys the lowest forgetting for almost all datasets under the label-noise setting (except for Multiple Datasets in Table 2). For most experiments on other settings, BCSR achieves a forgetting performance that is comparable to the best methods with at most 1% gap.

5.3 Ablation Studies

We conduct ablation studies to inspect the effectiveness of individual components in our proposed approach, including the effect of the bilevel optimizer, the smoothed topK regularizer, and the coreset size K (Appendix J) respectively.

Random initialization with and without bilevel optimizers. Our new bilevel optimization algorithm (Algorithm 3) is of vital importance for finding a good distribution w for selecting the coreset. To demonstrate this, we design comparative experiments: one experiment initializes w randomly without updating it, and the other one adopts the same initialization strategy but updates w by the proposed bilevel optimizer. Then we sample coresets based on w from two methods respectively. As you can see, the random initialization method is equivalent to Uniform Sampling. We report the experimental results of average accuracy and forgetting on three balanced datasets in Table 5. It can be observed that the sampling strategy based on bilevel optimization significantly outperforms random sampling. In particular, the strategy with bilevel optimization shows 3 . 23% , 2 . 89% , and 4 . 39% AVG ACC improvement on three benchmarks. Meanwhile, it also obtains 2 . 20% , 1 . 60% , and 3 . 20% FGT reduction on three benchmarks.

1

3

5

7

9

11 13 15 17 19

Task

0.5

1.0

1.5

2.0

2.5

Upper loss

BCSR

Greedy Coreset

(a) Split CIFAR-100

1

2

3

4

5

Task

0

1

2

3

4

Upper loss

BCSR

Greedy Coreset

(b) Multiple Datasets

To verify the effectiveness of our bilevel optimizer, we compare the loss curve with Greedy Coreset that uses NTK. The result is presented in Figure 3. There are a number of stream mini-batches in each task. The bilevel optimizer trains over each mini-batch, where the upper loss is plotted in the figure. In the experiment, we plot the loss value for every 5 mini-batches. Within each task, the loss from BCSR gradually decreases with slight fluctuations, and it increases only when encountering a new task. In contrast, the loss value of the Greedy Coreset approach always stays large. It indicates that our bilevel optimizer is more effective than the Greedy Coreset approach.

Effectiveness of the regularizer. In our algorithm, the bilevel formulation has a smooth topK loss as a regularizer in the objective function to promote the probability distribution to have K nonzero entries. The goal is to make sure that the summation of topK entries of the learned probability vector is large, which increases confidence in the coreset selection. The hyperparameter λ is used to balance the cross-entropy loss and the regularization term. We explore the effects on the performance with different λ , and list the results in Table 6.

This ablation experiment is performed on our framework BCSR, and average accuracy and forgetting on three balanced benchmarks are reported. When λ = 0 . 10 , BCSR can reach the best performance (The highest AVG ACC and lowest FGT) on Split CIFAR-100 and Permuted MNIST. While on Multiple Datasets, BCSR performs the best when λ = 1 . 0 . This dataset contains more categories, and hence it is more challenging to select the appropriate representative samples. In this case, larger λ would emphasize more on maximizing the topK probability and help select better representative samples. In addition, we observe that λ set as a too large or too small value will damage the performance, which is in line with the observations in standard regularized empirical risk minimization problems such as overfitting and underfitting. Please refer to Appendix H for further analysis.

6 Theoretical Analysis

In this section, we provide a theoretical analysis for our proposed method. In particular, we establish convergence rate for our algorithm BCSR.

Theorem 1. Suppose standard assumptions hold in bilevel optimization (see Assumptions 1, 2 and 3 in Appendix M) . Choose parameters λ, α, η and N such that (1+ λ )(1 -αµ ) N (1+ 8 rL 2 ηµ ) ≤ 1 -ηµ , where r = C 2 Q ( ρM µ + L ) 2 and C Q = QρMη µ + η 2 Q 2 ρM + ηQL . Furthermore, choose the stepsize β such that 6 ωβ 2 L 2 < 1 9 ηµ and β ≤ 1 4 L ϕ , where the constant ω is given by Equation (16) in Appendix A.

Then, we have the following convergence result.

where G j := 1 β ( w j - P ∆ n ( w j -β ∇ ϕ ( w j ))) denote the generalized projected gradient, D ϕ := ϕ ( w 0 ) -min w ϕ ( w ) > 0 , D 0 = ∥ θ 0 0 -θ ∗ 0 ∥ 2 + ∥ v 0 0 -v ∗ 0 ∥ 2 , L ϕ is the smoothness parameter of the total objective ϕ ( w ) whose form is proved in Appendix A.

Theorem 1 provides a general convergence result for the proposed bilevel algorithm, which allows for a very flexible selection of subloop lengths N and Q as long as the inequality (1 + λ )(1 -αµ ) N (1 + 8 rL 2 ηµ ) ≤ 1 -ηµ holds given proper stepsizes λ, η, α . For example, in most of our experiments, the choice of N = 1 and Q = 3 works the best. Then, in this case, we further specify the parameters in Theorem 1, and provide the following corollary.

Corollary 1. Under the same setting as in Theorem 1, choose N = 1 , Q = 3 and set λ = αµ 2 , η ≤ µ 2 α 4608 L 2 and α ≤ 1 L . Then, to make sure an ϵ -accurate stationary point, i.e., 1 J ∑ J -1 j =0 E ∥ G j ∥ 2 ≤ ϵ 2 , the number of iterations is O ( ϵ -2 ) , each using |B| = O ( ϵ -2 ) of samples from the standard Gaussian distribution N (0 , 1) .

Corollary 1 shows that the proposed bilevel algorithm converges to an ϵ -accurate stationary point using only O ( ϵ -2 ) iterations and O ( ϵ -2 ) samples drawn from N (0 , 1) per iteration. Note the the large batch size |B| = O ( ϵ -2 ) is necessary here to guarantee a convergence rate of O (1 /T ) , which matches the results in solving single-level constrained nonconvex problems [24]. In addition, it is computationally tractable because sampling from a known Gaussian distribution is easy and cheap.

7 Conclusion

In this paper, we advance the state-of-the-art bilevel coreset selection in continual learning. We first introduce a new bilevel formulation with smoothed topK regularization and then design an efficient bilevel optimizer as a solver. We conduct extensive experiments in continual learning benchmark datasets to demonstrate the effectiveness of our proposed approach. We also show that the bilevel optimizer can efficiently find ϵ -stationary point with O (1 /ϵ 4 ) computational complexity, which matches the best complexity of projected SGD for a single-level problem.

Acknowledgments and Disclosure of Funding

We would like to thank the anonymous reviewers for their helpful comments. Jie Hao and Mingrui Liu are both supported by a grant from George Mason University. Computations were run on ARGO, a research computing cluster provided by the Office of Research Computing at George Mason University (URL: https://orc.gmu.edu).

References

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 139-154, 2018.

Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems 32 , pages 11849-11860. 2019.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems , 32, 2019.

Michael Arbel and Julien Mairal. Amortized implicit differentiation for stochastic bilevel optimization. In International Conference on Learning Representations , 2022.

Olivier Bachem, Mario Lucic, and Andreas Krause. Coresets for nonparametric estimation-the case of dp-means. In International Conference on Machine Learning , pages 209-217. PMLR, 2015.

Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems , 33:1487914890, 2020.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision , 2014.

Yaroslav Bulatov. Notmnist dataset. Google (Books/OCR), Tech. Rep.[Online]. Available: http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html , 2, 2011.

Trevor Campbell and Tamara Broderick. Automated scalable bayesian inference via hilbert coresets. The Journal of Machine Learning Research , 20(1):551-588, 2019.

Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 532-547, 2018.

Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420 , 2018.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. A single-timescale stochastic bilevel optimization method. arXiv preprint arXiv:2102.04671 , 2021.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9268-9277, 2019.

Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. arXiv preprint arXiv:2201.13409 , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics (AISTATS) , pages 318-326, 2012.

Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertaintyguided continual learning with bayesian neural networks. arXiv preprint arXiv:1906.02425 , 2019.

Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. arXiv preprint arXiv:1910.07104 , 2019.

Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing , pages 569-578, 2011.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1126-1135. JMLR. org, 2017.

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning (ICML) , pages 1165-1173, 2017.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning , pages 1568-1577. PMLR, 2018.

Camille Garcin, Maximilien Servajean, Alexis Joly, and Joseph Salmon. Stochastic smoothing of the top-k calibrated hinge loss for deep imbalanced classification. arXiv preprint arXiv:2202.02193 , 2022.

Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming , 155(1):267-305, 2016.

Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680, 2014.

Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation. In Proc. International Conference on Machine Learning (ICML) , 2020.

Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory-based lifelong learning. Advances in Neural Information Processing Systems , 33, 2020.

Zhishuai Guo and Tianbao Yang. Randomized stochastic variance-reduced methods for stochastic bilevel optimization. arXiv preprint arXiv:2105.02266 , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Xu He and Herbert Jaeger. Overcoming catastrophic interference using conceptor-aided backpropagation. 2018.

Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint arXiv:2007.05170 , 2020.

Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. Advances in Neural Information Processing Systems , 29, 2016.

David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Kaiyi Ji, Mingrui Liu, Yingbin Liang, and Lei Ying. Will bilevel optimizers benefit from loops. arXiv preprint arXiv:2205.14224 , 2022.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning (ICML) , pages 4882-4892. PMLR, 2021.

Chris Dongjoo Kim, Jinseo Jeong, Sangwoo Moon, and Gunhee Kim. Continual learning on noisy data streams via self-purified replay. In Proceedings of the IEEE/CVF international conference on computer vision , pages 537-547, 2021.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning , pages 3925-3934. PMLR, 2019.

Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, and Richard Zemel. Reviving and improving recurrent back-propagation. In Proc. International Conference on Machine Learning (ICML) , 2018.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. ICLR , 2019.

Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems , pages 6467-6476, 2017.

Mario Lucic, Matthew Faulkner, Andreas Krause, and Dan Feldman. Training gaussian mixture models at scale via coresets. The Journal of Machine Learning Research , 18(1):5885-5909, 2017.

Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning (ICML) , pages 2113-2122, 2015.

Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 7765-7773, 2018.

James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review , 102(3):419, 1995.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv:1710.10628 , 2017.

Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International conference on machine learning , pages 737-746. PMLR, 2016.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 , 2018.

Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems , pages 3738-3748, 2018.

Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science , 7(2):123-146, 1995.

Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016.

Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning , pages 4528-4537. PMLR, 2018.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017.

Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning , pages 4548-4557. PMLR, 2018.

Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated backpropagation for bilevel optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1723-1732, 2019.

Zhenmei Shi, Junyi Wei, and Yingyu Liang. A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features. arXiv preprint arXiv:2206.01717 , 2022.

Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems , 30, 2017.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484, 2016.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks , pages 1453-1460. IEEE, 2011.

Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. Exploring example influence in continual learning. Advances in Neural Information Processing Systems , 35:27075-27086, 2022.

Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Gcr: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 99-108, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Luis N Vicente and Paul H Calamai. Bilevel and multilevel programming: A bibliography review. Journal of Global optimization , 5(3):291-306, 1994.

Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning , pages 1954-1963. PMLR, 2015.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Ju Xu and Zhanxing Zhu. Reinforced continual learning. Advances in Neural Information Processing Systems , 31, 2018.

Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. International Conference on Learning Representations , 2020.

Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085 , 2021.

Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 , 2017.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems , 33:5824-5836, 2020.

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 3987-3995. JMLR. org, 2017.

Guodong Zhang, James Martens, and Roger B Grosse. Fast convergence of natural gradient descent for over-parameterized neural networks. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Probabilistic bilevel coreset selection. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 27287-27302. PMLR, 17-23 Jul 2022.

Xiangyu Zhu, Jie Hao, Yunhui Guo, and Mingrui Liu. Auc maximization in imbalanced lifelong learning. In Uncertainty in Artificial Intelligence , pages 2574-2585. PMLR, 2023.

A Proof of Theorem 1

Let ∆ ( n ) := { w : 0 ≤ w ( i ) ≤ 1 , || w || 1 = 1 } denote the constraint set of the upper-level problem. We first provide some important inequalities.

Recall that v q j be the q th GD iterate in solving the linear system ∇ 2 θ L ( θ N j , w j ) v = ∑ n i =1 ∇ ℓ i ( θ N j ) at iteration j via the following process:

which, by telescoping Equation (6) over q from 0 to Q -1 , yields

Let v ∗ j be the solution of the linear system ∇ 2 θ L ( θ ∗ j , w j ) v = ∑ n i =1 ∇ ℓ i ( θ ∗ j ) , and then we have

Combining Equation (6) and Equation (7), noting that v 0 j = v Q j -1 and using Assumption 2 that ∥ v ∗ j ∥ ≤ ∥ ( ∇ 2 θ L ( θ ∗ j , w j )) -1 ∥∥ ∑ n i =1 ∇ ℓ i ( θ ∗ j ) ∥ ≤ M µ , the difference between v Q j and v ∗ j can be bounded as

We next bound ∆ q := ∥ ( I -η ∇ 2 θ L ( θ N j , w j )) q -( I -η ∇ 2 θ L ( θ ∗ j , w j )) q ∥ in Equation (9) as:

which, by telescoping Equation (10) and in conjunction with Equation (9), yields

which, combined with ∥ v ∗ j -v ∗ j -1 ∥ ≤ ( L µ + Mρ µ 2 )( L µ + 1 ) ∥ w j -w j -1 ∥ and using the fact that (1 -x ) Q ≥ 1 -xQ for 0 ≤ x ≤ 1 , yields

where the constant C Q := QρMη µ + η 2 Q 2 ρM + ηQL .

The next step is to characterize the error induced by the lower-level updates on θ .

Note that θ ∗ j = arg min θ L ( θ, w j ) . Using Assumptions 1 and 2, we have

which, in conjunction with θ 0 j = θ N j -1 and Lemma 2.2 in [25], yields

Recall the definition that r = C 2 Q ( ρM µ + L ) 2 . Then, combining Equation (12) and Equation (14) yields we can obtain

where the constant ω is given by

Recall that we choose (1 + λ )(1 -αµ ) N (1 + 8 rL 2 ηµ ) ≤ 1 -ηµ . Then, we obtain from Equation (15) that

Let δ j := ( 1 + ρ 2 M 2 L 2 µ 2 ) E ∥ θ N j -θ ∗ j ∥ 2 + E ∥ v Q j -v ∗ j ∥ 2 . Then, incorporating the update that w j = P ∆ n ( w j -1 -βφ j -1 ) into Equation (17) yields

We next bound the last term in the above Equation (18). Based on the definition of the hypergradient estimate φ j = ∇ w R ( w,δ ; B ) -∇ w ∇ θ L ( θ N j , w j ) v Q j , we have

where ( i ) follows because ∇ R ( w j , δ ; ˜ z ) is an unbiased estimate of ∇ R ( w,δ ) and ( ii ) follows from Proposition 1. Substituting Equation (19) into Equation (18) yields

Let τ := 1 -ηµ +6 ωβ 2 L 2 . Then, telescoping Equation (20) yields

Based on Equation (21), we are ready to provide the final convergence result. First, based on the Lipschitz continuity in Assumption 1, Assumption 2 and Assumption 3, we have

where the constant L ϕ = √ Kn δ + L 2 + ρM 2 µ + 2 ρLM + L 3 µ 2 + ρL 2 M µ 3 is the smoothness parameter. Then, this inequality further implies

To analyze the second term at the right hand side of the above Equation (22), we note that

= ⟨ w j -βφ j -P ∆ n ( w j -βφ j ) , P ∆ n ( w j -βφ j ) -w j ⟩ + ∥P ∆ n ( w j -βφ j ) -w j ∥ 2 , which, in conjunction with the property of projection on convex set that ⟨ x -P ∆ n ( x ) , y -P ∆ n ( x ) ≤ 0 for any y ∈ S and the fact that w j = P ∆ n ( w j -1 -βφ j -1 ) ∈ S , yields

Then, substituting Equation (23) into Equation (22) yields

where we use the notation that ̂ G j = 1 β ( w j -P ∆ n ( w j -βφ j ) ) , and the non-expansive property of projection. Then, taking the expectation and incorporating Equation (19) into Equation (24), we have

Then, substituting Equation (21) into the above Equation (25) yields

which, by taking the telescoping over j from 0 to J -1 , yields

Rearranging the above inequality, we have

Recalling the definition that τ = 1 -ηµ +6 ωβ 2 L 2 , and noting that we choose the stepsize β such that 6 ωβ 2 L 2 < 1 9 ηµ and β ≤ 1 4 L ϕ , we can simplify Equation (26) as

From the gradient descent based updates, we have δ 0 = ( 1+ ρ 2 M 2 L 2 µ 2 ) E ∥ θ N 0 -θ ∗ 0 ∥ 2 + E ∥ v Q 0 -v ∗ 0 ∥ 2 ≤ ( 1 + ρ 2 M 2 L 2 µ 2 ) ∥ θ 0 0 -θ ∗ 0 ∥ 2 + ∥ v 0 0 -v ∗ 0 ∥ 2 < + ∞ . Then, the proof is complete.

B Proof of Corollary 1

Based on the definition of C Q = QρMη µ + η 2 Q 2 ρM + ηQL and noting that η ≤ 1 L ≤ 1 µ and Q = 3 , we have C Q ≤ 12 η ( ρM µ + L ) , which, combined with the definition that r = C 2 Q ( ρM µ + L ) 2 , yields r ≤ 144 η 2 . Then, based on the choice of N = 1 and λ = αµ 2 , we have

which, in conjunction with η ≤ µ 2 α and α ≤ 1 , yields

This implies that the inequality (1 + λ )(1 -αµ ) N (1 + 8 rL 2 ηµ ) ≤ 1 -ηµ required by Theorem 1 is satisfied. Then, treating µ, η, L, ρ, M, α, β, K , ∥ θ 0 0 -θ ∗ 0 ∥ 2 and ∥ v 0 0 -v ∗ 0 ∥ 2 as constants independent of the total number J of iterations, we have

Then, to ensure an ϵ -accurate stationary point, the number of iterations is ϵ -2 with a batch size |B| = O ( ϵ -2 ) .

C Experiment Hyperparameters

The experimemtal hyperparameters are list in Table 7, including the memory size: |M| , coreset size: |S| , stream batch size: |B t | , learning rate for M tr : lr t , learning rate for M cs : lr p , learning rate for sample weights: lr w , regularization efficient: λ , epochs for training model: E , outer loops in bilevel optimization: J , inner loops in bilevel optimization: N , loops for estimating the Hessian-inversevector product: Q , a factor of Gaussian noise: δ .

D Running Time Comparison

We evaluate the running time (wall-clock time) of baseline methods on Split Tiny-ImageNet and Split Food-101 in Table 8. All the training sections keep the same among these algorithms. Since there is no advanced coreset selection procedures in K-means Features, K-means Embedding, Uniform sampling, iCaRL, and Grad Matching, their running costs are pretty low but with worse performance (i.e., accuracy and forgetting) compared with our approach BCSR. PBCS and GCR, SPR and MetaSP don't involve bilevel formulations, they cannot guarantee accurate coreset sampling though with lower time cost against BCSR. Compared with OCS, BCSR takes 23% and 35% running time reduction on Tiny-ImageNet and Split Food-101, respectively. Because similarity computing based on data pairs is computationally expensive for OCS. Greedy Coreset takes much more time than BCSR due to the usage of NTK.

E Experiments on Permuted-MNIST

The experiment results on Permuted-MNIST is presented in this section. BCSR outperforms other baselines significantly in A VG ACC on all the data settings, while showing relatively low forgetting.

0.4

0.6

0.8

F Evolution of Average Accuracy

We present the learning process of different methods on Multiple Datasets in Figure 4. There are five tasks, where the average accuracy is tested after training each task. BCSR shows better performance than other baselines in different settings, including Balanced, Imbalanced, and label-noise.

0

1

2

3

4

Tasks

60

70

80

90

AVG ACC

(a) Balanced Multiple Datasets

0

1

2

3

4

Tasks

20

30

40

50

60

70

80

AVG ACC

(b) Imbalanced Multiple Datasets

0

1

2

3

4

Tasks

40

50

60

70

80

AVG ACC

(c) Label-Noise Multiple Datasets

K-means Features

K-means Embedding

Uniform

iCaRL

SPR

MetaSP

Greedy Coreset

GCR

PBCS

OCS

BCSR

1.0

G Possibility Distribution of Candidate Coreset

To explore the effect of topK regularizer, we observe the possibility distribution of sample weights after each bilevel optimization, where sample weights of candidate coresets are initialized uniformly. The novel topK regularizer makes sure that the summation of the topK entries of the learned probability vector is large, such that we can confidently choose a coreset with size K . We show the weight distribution after optimization in Figure. 5, where you can find that topK weights are much higher than others (with a margin 2% ), which easily distinguishes the topK core samples from candidates.

0

10

20

30

40

50

Candidate sample index

0.00

0.02

0.04

0.06

0.08

0.10

Possibility distribution

0

2

4

6

8

Iteration

0.01

0.03

0.06

0.08

0.10

0.12

Average norm of Gradient

H The Effect of TopK Regularizer

To further analyze the effects of topK regularizer, we conduct the ablation study with different values of regularizer coefficient λ on balanced Split CIFAR-100. The performance results with different λ are shown in Table 10 and the corresponding average topK summations of coreset weights are in Table 11. In our experiment, there are 50 candidate samples in each mini-batch data, and the summation of 50 coreset weights is equal to 1 . 00 . TopK summation of weights increases as λ increases, which imposes higher probabilities on topK entries and lower probabilities on the rest candidates. The best performance is achieved when λ = 0 . 1 , which means λ balances the trade-off between the loss function and regularizer strength: if λ is too large, the algorithm primarily focuses on choosing the important samples instead of updating the model parameter, and vice versa.

I Hypergradient Evolution

To demonstrate the efficiency of bilevel optimization, we illustrate the evolution of hypergradients for a bilevel optimization on Split CIFAR-100 in Figure. 6. We observe the average norm of hypergradient reduces from 10 -1 to less than 10 -2 in each round of coreset selection with loops equal to 10 . The hypergradient curves show that our designed bilevel optimization provides both theoretical and practical convergence guarantees.

J The Effect of Coreset Size K

Coreset size K also plays an important role in the experiment. We compare our BCSR with other coreset-based algorithms on different K . Note that coreset is selected from the current stream mini-batch B t , so the coreset size K satisfies K ≤ |B t | . In the main experiment results, K = 10 is fixed in all the algorithms for a fair comparison. Here, we set K = 10 , 20 , 40 to conduct the continual learning experiments, respectively. The result is represented in Table 12. Note that all the results presented here are based on balanced Split CIFAR-100.

We mainly compare with the other four methods, including Uniform Sampling and three coresetbased methods, Greedy Coreset, PBCS, and OCS. We can observe that the performance of almost all methods becomes worse when coreset size is large. The reason is that if coreset size is larger and closer to B t , more redundant or noisy data are selected from the current stream mini-batch, and the coreset would not be representative anymore. In contrast, the smaller coreset could reduce the probability that redundant data are selected. Compared with other methods, BCSR shows the best performance (both accuracy and forgetting) and better robustness on different K .

K The Effect of Inner Loops N and Hessian-inverse-vector Product Loops Q

We conduct ablation studies to explore the sensitivity of hyperparameters ( N and Q ) on Split CIFAR100. The results are presented in Tables 13 ( N ) and Table 14 ( Q ). The model performance remains relatively stable when increasing inner loops ( N ) while fixing Q . But too large N ( N ≥ 15 ) leads to performance degradation due to overfitting. The Q loops show similar properties that a few Q loops (e.g., Q = 3 ) are enough to approximate the Hessian-inverse-vector product. Too small Q and too large Q will hurt the performance due to possible underfitting (e.g., Q = 1 ) and overfitting (e.g., Q = 20 ).

L Datasets

Spit CIFAR-100 . Balanced split CIFAR-100 is based on the original CIFAR-100 and is split into 20 tasks, each consisting of 5 disjoint classes. The imbalanced setting and label noise are also applied to this dataset to make the task more challenging. We follow [13] to transform the original dataset to imbalanced long-tailed CIFAR-100. In the label-noise setting, we randomly select 20% data in each task and randomly change their labels to an arbitrary label of 10 classes.

Permuted MNIST . Balanced MNIST is a handwritten digits dataset [43] containing 20 tasks, where each task applies a fixed random permutation to the image pixels. For the imbalanced setting, we randomly select 8 classes over 10 and sample 10% of the selected classes for training. We also conduct the experiments on the label noise scenario, where symmetric label noise with 20% noise rate is imposed on the data. In particular, to make the problem setting more challenging, each task only retains 3000 training data randomly sampled from the original data.

Multiple Datasets . This dataset [78] contains a couple of totally different datasets, including MNIST [43], fashion-MNIST [75], NotMNIST [8], Traffic Sign [69] and SVHN [54]. There are 5 tasks, and each task is constructed by randomly selecting 1000 training samples from a different dataset. The procedure of creating the dataset in the imbalanced and label-noise settings is the same as that in Split CIFAR-100.

Split Tiny-ImageNet . This dataset [42] contains 100000 images of 200 classes (500 for each class) downsized to 64 × 64 colored images. Each class has 500 training images, 50 validation images, and 50 test images. We construct the task sequences by splitting data into 20 tasks, where each task consists of 10 disjoint classes.

Split Food-101 . This dataset [7] is a challenging data set of 101 food categories, with 101'000 images. All the images are resized to 64 × 64 pixels to be fed into models. To build continual learning tasks easily, We discard the last category and split data into 20 tasks, with 5 categories within each task. Other data settings, including imbalanced and lable-noise, are the same as Split CIFAR-100.

M Assumptions and Properties

We first provide standard definitions and assumptions for the convergence rate analysis of bilevel optimization [25, 36]. For notational convenience, we define ℓ ( θ ) := ∑ n i =1 ℓ i ( θ ) .

′ ′ ′

Definition 1. A mapping f is L f -Lipschitz continuous if for ∀ z, z , ∥ f ( z ) -f ( z ) ∥ ≤ L f ∥ z -z ∥ . Assumption 1. The lower-level function L ( θ, w ) is µ -strongly-convex w.r.t. θ .

Assumption 1 is a necessary geometric assumption in analyzing the convergence rate of bilevel optimization algorithms, as also widely adopted existing theories in [25, 37, 32]. We also note that this condition is satisfied for overparameterized neural networks [82]. The following assumption imposes some Lipschitz continuity conditions on the upper- and lower-level objective functions.

Assumption 2. The gradients ∇ θ L ( θ, w ) , ∇ w L ( θ, w ) and ℓ ( θ ) are L -Lipschitz continuous w.r.t. θ and w . In addition, the gradient norm ∥∇ ℓ ( θ ∗ ( w )) ∥ ≤ M .

Note that we do not impose any conditions on the regularization function R ( w,δ ) . The following assumption imposes the Lipschitz continuity on the second-order derivatives of the lower-level functions.

Assumption 3. The second-order derivatives ∇ w ∇ θ L ( θ, w ) and ∇ 2 θ L ( w,θ ) are ρ -Lipschitz continuous.

Then, we use the following proposition to characterize the properties of the smoothed topK regularizer R ( w,δ ) , based on the results in [23].

Proposition 1. The smoothed regularizer R ( w,δ ) and its sampled version R ( w,δ ; ˜ z ) satisfy the following two important properties: (i) The gradient ∇ R ( w,δ ) exists and is √ Kn δ -Lipschitz continuous; (ii) The gradient norm ∥ R ( w,δ ; ˜ z ) ∥ is bounded by √ K for any sample ˜ z .

Proposition 1 shows that the regularizer R ( w,δ ) is smooth and its stochastic version R ( w,δ ; ˜ z ) is bounded. These two properties are important to guarantee the non-asymptotic convergence of our proposed bilevel method.

Peer reviews:
comments: SUMMARY:
Past works have limitations in terms of scalability, formulation approximation, or performance. This paper offers an efficient coreset selection problem with provable theoretical guarantees. That is, the authors solve a bilevel optimization on a probability distribution over the dataset with loss minimization on the selected dataset that is guided into a low-dimensional manifold via a smoothed top-K loss as a regularizer on the probability distribution.

STRENGTHS:
The approach is simple, yet well-motivated by the limitations of previous works.
The writing addresses the limitations well and is easy to follow.

WEAKNESSES:
(1) Further analysis into the top-K loss and its causal effects would aid in understanding the mechanics of probability distribution being regularized into a low-dimensional manifold. 

(2) comparisons with state-of-the-art non-coreset replay methods (e.g., using stability plasticity scores [1], contrastive representation based selection [2]), as it would be meaningful to see the prospect of this direction. 

(3) [minor] main illustration figure 1 could include more technical information (embedding some important eqns for example) or include an additional figure to guide the reader better.



[1] Sun et al, Exploring Example Influence in Continual Learning, NeurIPS 2022
[2] Kim et al, Continual Learning on Noisy Data Streams via Self-Purified Replay, ICCV 2021

QUESTIONS:
if Weaknessses (1) could be further analyzed and shown, as well as comparison with (2) other state of the art rehearsal baselines comparison would be informative.

LIMITATIONS:
Not mentioned in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new approach to coreset selection in rehearsal-based continual learning. The authors claim that traditional methods optimise over discrete decision variables, resulting in computationally expensive processes. To address this, they propose a new bilevel formulation where the inner problem finds a model that minimizes expected training error, and the outer problem learns a probability distribution with approximately K nonzero entries through adding a regularizer and ensuring convergence to the ϵ-stationary point with O(1/ϵ^4) complexity. The authors also perform extensive experiments demonstrate superior performance compared to existing methods in various continual learning settings.\n\nSTRENGTHS:\n1. The availability of provided code facilitates reproducibility, making it a straightforward process.\n2. The authors present experimental results that demonstrate a pretty good performance in comparison to the selected baselines.\n3. The inclusion of the added regularizer seems promising, but it requires a more detailed and elaborate explanation to justify its contribution adequately.\n\nWEAKNESSES:\n**Weaknesses**\nMy main concern is that this work appears to be primarily an incremental extension of prior work [2] by adding a regularizer. While this is not necessarily negative, I do not see any significant new improvements from an algorithmic perspective. Specifically, in lines 46-52, the authors assert that the key challenges of bilevel coreset selection are (1) the expensive nature of optimization over cardinality constraints and (2) the nested nature of bilevel optimization. However, these weaknesses pertain to [1] rather than the coreset selection problem itself.\nIn order to address these challenges, [2] was proposed. The contributions that the authors claim in this paper seem to stem from [2] rather than the proposed method itself.\n\n**Suggestion:**  \nCertainly, enhancing the motivation of the paper and clarifying the advantages of the proposed method compared to [7] is crucial for improving the paper. The author should allocate more space to address these aspects.\n\n[1] Borsos, Z., Mutny, M., & Krause, A. (2020). Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33, 14879-14890.\n\n[2] Zhou, X., Pi, R., Zhang, W., Lin, Y., Chen, Z., & Zhang, T. (2022, June). Probabilistic bilevel coreset selection. In International Conference on Machine Learning (pp. 27287-27302). PMLR.\n\nQUESTIONS:\nCould you please clarify the differences and advantages between your method and [7]? In lines 135-139, the author simply states that \"so this formulation [7] oversimplifies the coreset selection problem.\" However, this statement does not highlight any specific disadvantages of [7].\n\nI will consider improving my score if my question and concern could be well-addressed.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the coreset selection problem in rehearsal-based continual learning, focusing specifically on the application of bilevel optimization. The authors identify limitations in existing bilevel optimization-based coreset selection methods for continual learning, including high computational costs resulting from greedy search and the loss of bilevel optimization nature due to single-level equivalence. To overcome these drawbacks, the authors propose a new formulation that incorporates the probability simplex and a smoothed top-K regularizer. The latter enforces the K most important elements have larger weights. The authors develop a new stochastic bilevel method for this continuous and smooth loss. The effectiveness of the proposed method is demonstrated through comprehensive experiments, which also include various ablation studies. The authors establish the properties of the new loss function and provide guarantees on convergence.\n\nSTRENGTHS:\n1. Both continual learning and bilevel optimization are timely topics.The exploration of the benefits of bilevel optimization in continual learning is an under-explored area, making the studied topic in this work both interesting and important.\n2. The paper is well written and easy to follow. The inclusion of Figure 1, illustrating the process of training bilevel algorithms in continual learning, along with PyTorch-style pseudocode, greatly aids in understanding the entire process.\n3. The introduction of the new loss function and bilevel algorithms is well-motivated. The replacement of the nuclear norm with a smoothed top-K regularizer, as proposed in the paper, is a good idea that leads to faster and improved bilevel optimization algorithms. The empirical evidence provided in Figure 3 serves as a strong justification for the efficacy of the new loss function and algorithms.\n4. The experiments conducted in the paper are comprehensive. The evaluation encompasses various datasets, including multiple datasets, split Cifar100, and large-scale datasets like Tiny-ImageNet and Food-101. Additionally, various ablation studies are also included.\n5. The paper establishes guarantees on convergence rate and explores the properties of the new loss function.\n\nWEAKNESSES:\nThe proposed algorithm needs to compute the Hessian-vector products, which may be computation expensive in large-scale cases. This computational expense may pose challenges and limit the applicability of the algorithm. Is it possible to improve the efficiency of the proposed method by utilizing Hessian-free bilevel algorithms based on recent advancements (e.g., Liu et al. [1], Sow et al. [2])？\n\n[1] Liu, Bo, et al. \"Bome! bilevel optimization made easy: A simple first-order approach.\" Advances in Neural Information Processing Systems 35 (2022): 17248-17262.\n\n[2] Sow, Daouda, Kaiyi Ji, and Yingbin Liang. \"On the convergence theory for hessian-free bilevel algorithms.\" Advances in Neural Information Processing Systems 35 (2022): 4136-4149.\n\nQUESTIONS:\n1. How to select the parameters such as N, Q and $\\delta$ in Algorithm 3 for continual learning? It would be beneficial to provide suggestions or guidelines for choosing these parameters and to discuss the sensitivity of the algorithm's performance to such parameters.\n\n2. What are the main challenges of the bilevel optimization analysis in the continual learning setting?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a better approach for grasping the coreset-based bi-level optimization procedure used in the field of continual learning. The proposed method relies on the use of the proxy model to obtain a coreset, which in turn will affect the training of the original model.  With that being said, the idea relies on relaxing the hard cardinality constraints usually used in the field of coreset construction for continual learning tasks, to a softer version that allows the use of first-order methods.\n\nSTRENGTHS:\nIn what follows, a list of the strengths of the presented paper is given:\n\n * The paper draws back from the hard cardinality constraint and follows a relaxed version of it by resolving to the use of probability simplex and the top-K smoothed regularization to mimic the effect of the lost cardinality constraint.\n * The method is easy to follow, as well as it relies on the use of a proxy model.\n * While the proposed model is among the top 5 slowest from the list of competitors (referring to Table 8 in the appendix), it is by far the best method in terms of accuracy and forgetting almost across the board. \n * The new formulation of the problem allows for dropping the use of combinatorial optimization techniques while exposing the field to first-order optimizers. \n * Finally, the paper is well-written and easy to follow.\n\nWEAKNESSES:\nOne problem that I have in mind is the use of the proxy model, which might be a limitation in the presence of large models, i.e., models containing $> 100M$ parameters.\n\nQUESTIONS:\nIs it possible to circumvent the need for a proxy model in the case of working with large models? How worse would the results be if one would use some quantized or pruned version of the original model $M_{tr}$?\n\nLIMITATIONS:\nThe authors listed the limitations associated with their theoretical analysis in the form of assumptions usually used in the field of coresets for continual learning.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}