SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left(1 + \frac {\rho^ {2} M ^ {2}}{L ^ {2} \mu^ {2}}\right) \mathbb {E} \| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \| ^ {2} + \mathbb {E} \| v _ {j} ^ {Q} - v _ {j} ^ {*} \| ^ {2} \\ \leq (1 + \lambda) (1 - \alpha \mu) ^ {N} \left(1 + \frac {\rho^ {2} M ^ {2}}{L ^ {2} \mu^ {2}}\right) \left(1 + \frac {8 r L ^ {2}}{\eta \mu}\right) \mathbb {E} \| \theta_ {j - 1} ^ {N} - \theta_ {j - 1} ^ {*} \| ^ {2} \\ + (1 - \eta \mu) \mathbb {E} \| v _ {j - 1} ^ {Q} - v _ {j - 1} ^ {*} \| ^ {2} + \omega \mathbb {E} \| w _ {j} - w _ {j - 1} \| ^ {2}, \tag {15} \\ \end{array}
$$
where the constant  $\omega$  is given by
$$
\begin{array}{l} \omega := \left(1 + \frac {1}{\lambda}\right) (1 - \alpha \mu) ^ {N} \left(1 + \frac {\rho^ {2} M ^ {2}}{L ^ {2} \mu^ {2}}\right) \frac {L ^ {2}}{\mu^ {2}} \\ + \frac {8}{\eta \mu} \frac {L ^ {4}}{\mu^ {2}} \left(1 + \frac {\rho^ {2} M ^ {2}}{L ^ {2} \mu^ {2}}\right) \left(\frac {4}{\mu^ {2}} + r (1 - \alpha \mu) ^ {N} \left(1 + \frac {1}{\lambda}\right)\right). \tag {16} \\ \end{array}
$$
Recall that we choose  $(1 + \lambda)(1 - \alpha \mu)^{N}\left(1 + \frac{8rL^{2}}{\eta\mu}\right)\leq 1 - \eta \mu$ . Then, we obtain from Equation (15) that
$$
\begin{array}{l} \left(1 + \frac {\rho^ {2} M ^ {2}}{L ^ {2} \mu^ {2}}\right) \mathbb {E} \| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \| ^ {2} + \mathbb {E} \| v _ {j} ^ {Q} - v _ {j} ^ {*} \| ^ {2} \\ \leq (1 - \eta \mu) \left(1 + \frac {\rho^ {2} M ^ {2}}{L ^ {2} \mu^ {2}}\right) \mathbb {E} \| \theta_ {j - 1} ^ {N} - \theta_ {j - 1} ^ {*} \| ^ {2} \\ + (1 - \eta \mu) \mathbb {E} \| v _ {j - 1} ^ {Q} - v _ {j - 1} ^ {*} \| ^ {2} + \omega \mathbb {E} \| w _ {j} - w _ {j - 1} \| ^ {2}. \tag {17} \\ \end{array}
$$
Let  $\delta_{j} := \left(1 + \frac{\rho^{2}M^{2}}{L^{2}\mu^{2}}\right)\mathbb{E}\|\theta_{j}^{N} - \theta_{j}^{*}\|^{2} + \mathbb{E}\|v_{j}^{Q} - v_{j}^{*}\|^{2}$ . Then, incorporating the update that  $w_{j} = \mathcal{P}_{\Delta^{n}}(w_{j-1} - \beta \varphi_{j-1})$  into Equation (17) yields
$$
\begin{array}{l} \delta_ {j} \leq (1 - \eta \mu) \delta_ {j - 1} + 2 \omega \beta^ {2} \mathbb {E} \bigg \| \underbrace {\frac {1}{\beta} (w _ {j - 1} - \mathcal {P} _ {\Delta^ {n}} (w _ {j - 1} - \beta \nabla \phi (w _ {j - 1})))} _ {G _ {j - 1}} \bigg \| ^ {2} \\ + 2 \omega \beta^ {2} \| \varphi_ {j - 1} - \nabla \phi (w _ {j - 1}) \| ^ {2}. \tag {18} \\ \end{array}
$$
We next bound the last term in the above Equation (18). Based on the definition of the hypergradient estimate  $\varphi_{j} = \nabla_{w}R(w,\delta ;\mathcal{B}) - \nabla_{w}\nabla_{\theta}L(\theta_{j}^{N},w_{j})v_{j}^{Q}$ , we have
$$
\begin{array}{l} \mathbb {E} \| \varphi_ {j} - \nabla \phi (w _ {j}) \| ^ {2} \\ \leq 3 \mathbb {E} \| \nabla R (w _ {j}, \delta ; \mathcal {B}) - \nabla R (w _ {j}, \delta) \| ^ {2} + \frac {3 \rho^ {2} M ^ {2}}{\mu^ {2}} \mathbb {E} \| \theta_ {j} ^ {*} - \theta_ {j} ^ {N} \| ^ {2} + 3 L ^ {2} \mathbb {E} \| v _ {j} ^ {*} - v _ {j} ^ {Q} \| ^ {2} \\ \stackrel {(i)} {=} \frac {3}{| \mathcal {B} | ^ {2}} \sum_ {\widetilde {z} \in \mathcal {B}} \| \nabla R (w _ {j}, \delta ; \widetilde {z}) - \nabla R (w, \delta) \| ^ {2} + 3 L ^ {2} \Big (1 + \frac {\rho^ {2} M ^ {2}}{\mu^ {2} L ^ {2}} \Big) \mathbb {E} \| \theta_ {j} ^ {*} - \theta_ {j} ^ {N} \| ^ {2} + 3 L ^ {2} \mathbb {E} \| v _ {j} ^ {*} - v _ {j} ^ {Q} \| ^ {2} \\ \stackrel {(i i)} {\leq} \frac {6 K}{| \mathcal {B} |} + 3 L ^ {2} \left(1 + \frac {\rho^ {2} M ^ {2}}{\mu^ {2} L ^ {2}}\right) \mathbb {E} \| \theta_ {j} ^ {*} - \theta_ {j} ^ {N} \| ^ {2} + 3 L ^ {2} \mathbb {E} \| v _ {j} ^ {*} - v _ {j} ^ {Q} \| ^ {2} = \frac {6 K}{| \mathcal {B} |} + 3 L ^ {2} \delta_ {j}, \tag {19} \\ \end{array}
$$
where  $(i)$  follows because  $\nabla R(w_j,\delta ;\widetilde{z})$  is an unbiased estimate of  $\nabla R(w,\delta)$  and  $(ii)$  follows from Proposition 1. Substituting Equation (19) into Equation (18) yields
$$
\delta_ {j} \leq (1 - \eta \mu + 6 \omega \beta^ {2} L ^ {2}) \delta_ {j - 1} + \frac {1 2 \omega K \beta^ {2}}{| \mathcal {B} |} + 2 \omega \beta^ {2} \mathbb {E} \| G _ {j - 1} \| ^ {2}. \tag {20}
$$
Let  $\tau \coloneqq 1 - \eta \mu +6\omega \beta^2 L^2$  . Then, telescoping Equation (20) yields
$$
\delta_ {j} \leq \tau^ {j} \delta_ {0} + \frac {1 2 \omega K \beta^ {2}}{(1 - \tau) | \mathcal {B} |} + 2 \omega \beta^ {2} \sum_ {t = 0} ^ {j - 1} \tau^ {t} \mathbb {E} \| G _ {j - 1 - t} \| ^ {2}. \tag {21}
$$
Based on Equation (21), we are ready to provide the final convergence result. First, based on the Lipschitz continuity in Assumption 1, Assumption 2 and Assumption 3, we have
$$
\| \nabla \phi (w _ {1}) - \nabla \phi (w _ {2}) \| \leq L _ {\phi} \| w _ {1} - w _ {2} \|,
$$
where the constant  $L_{\phi} = \frac{\sqrt{Kn}}{\delta} + \frac{L^2 + \rho M^2}{\mu} + \frac{2\rho LM + L^3}{\mu^2} + \frac{\rho L^2M}{\mu^3}$  is the smoothness parameter. Then, this inequality further implies
$$
\begin{array}{l} \phi (w _ {j + 1}) \leq \phi (w _ {j}) + \langle \nabla \phi (w _ {j}), w _ {j + 1} - w _ {j} \rangle + \frac {L _ {\phi}}{2} \| w _ {j + 1} - w _ {j} \| ^ {2} \\ \leq \phi (w _ {j}) + \frac {1}{\beta} \left\langle \beta \varphi_ {j}, \mathcal {P} _ {\Delta^ {n}} \left(w _ {j} - \beta \varphi_ {j}\right) - w _ {j} \right\rangle + \left\langle \nabla \phi (w _ {j}) - \varphi_ {j}, \mathcal {P} _ {\Delta^ {n}} \left(w _ {j} - \beta \varphi_ {j}\right) - w _ {j} \right\rangle \\ + \frac {L _ {\phi}}{2} \| w _ {j + 1} - w _ {j} \| ^ {2}. \tag {22} \\ \end{array}
$$
To analyze the second term at the right hand side of the above Equation (22), we note that
$$
\begin{array}{l} - \left\langle \beta \varphi_ {j}, \mathcal {P} _ {\Delta^ {n}} \left(w _ {j} - \beta \varphi_ {j}\right) - w _ {j} \right\rangle \\ = \left\langle w _ {j} - \beta \varphi_ {j} - \mathcal {P} _ {\Delta^ {n}} \left(w _ {j} - \beta \varphi_ {j}\right), \mathcal {P} _ {\Delta^ {n}} \left(w _ {j} - \beta \varphi_ {j}\right) - w _ {j} \right\rangle + \| \mathcal {P} _ {\Delta^ {n}} \left(w _ {j} - \beta \varphi_ {j}\right) - w _ {j} \| ^ {2}, \\ \end{array}
$$
which, in conjunction with the property of projection on convex set that  $\langle x - \mathcal{P}_{\Delta^n}(x), y - \mathcal{P}_{\Delta^n}(x) \leq 0$  for any  $y \in S$  and the fact that  $w_j = \mathcal{P}_{\Delta^n}(w_{j-1} - \beta \varphi_{j-1}) \in S$ , yields
$$
- \left\langle \beta \varphi_ {j}, \mathcal {P} _ {\Delta^ {n}} \left(w _ {j} - \beta \varphi_ {j}\right) - w _ {j} \right\rangle \geq \| \mathcal {P} _ {\Delta^ {n}} \left(w _ {j} - \beta \varphi_ {j}\right) - w _ {j} \| ^ {2} \geq 0. \tag {23}
$$
Then, substituting Equation (23) into Equation (22) yields
$$
\begin{array}{l} \phi (w _ {j + 1}) \leq \phi (w _ {j}) + \langle \nabla \phi (w _ {j}) - \varphi_ {j}, \mathcal {P} _ {\Delta^ {n}} (w _ {j} - \beta \varphi_ {j}) - w _ {j} \rangle + \frac {L _ {\phi}}{2} \| w _ {j + 1} - w _ {j} \| ^ {2} \\ \leq \phi (w _ {j}) - \frac {\beta}{2} \| \widehat {G} _ {j} \| ^ {2} + \frac {\beta}{2} \| \varphi_ {j} - \nabla \phi (w _ {j}) \| ^ {2} + \frac {L _ {\phi} \beta^ {2}}{2} \| \widehat {G} _ {j} \| ^ {2} \\ \leq \phi (w _ {j}) - \left(\frac {\beta}{4} - \frac {L _ {\phi} \beta^ {2}}{4}\right) \| G _ {j} \| ^ {2} + \left(\beta - \frac {L _ {\phi} \beta^ {2}}{2}\right) \| \varphi_ {j} - \nabla \phi (w _ {j}) \| ^ {2}, \tag {24} \\ \end{array}
$$
where we use the notation that  $\widehat{G}_j = \frac{1}{\beta} (w_j - \mathcal{P}_{\Delta^n}(w_j - \beta \varphi_j))$ , and the non-expansive property of projection. Then, taking the expectation and incorporating Equation (19) into Equation (24), we have
$$
\begin{array}{l} \mathbb {E} \phi (w _ {j + 1}) \leq \mathbb {E} \phi (w _ {j}) - \left(\frac {\beta}{4} - \frac {L _ {\phi} \beta^ {2}}{4}\right) \mathbb {E} \| G _ {j} \| ^ {2} + \left(\beta - \frac {L _ {\phi} \beta^ {2}}{2}\right) \mathbb {E} \| \varphi_ {j} - \nabla \phi (w _ {j}) \| ^ {2} \\ \leq \mathbb {E} \phi \left(w _ {j}\right) - \left(\frac {\beta}{4} - \frac {L _ {\phi} \beta^ {2}}{4}\right) \mathbb {E} \| G _ {j} \| ^ {2} + \left(\beta - \frac {L _ {\phi} \beta^ {2}}{2}\right) \left(\frac {6 K}{| \mathcal {B} |} + 3 L ^ {2} \delta_ {j}\right). \tag {25} \\ \end{array}
$$
Then, substituting Equation (21) into the above Equation (25) yields
$$
\begin{array}{l} \mathbb {E} \phi (w _ {j + 1}) \leq \mathbb {E} \phi (w _ {j}) - \left(\frac {\beta}{4} - \frac {L _ {\phi} \beta^ {2}}{4}\right) \mathbb {E} \| G _ {j} \| ^ {2} + \left(\beta - \frac {L _ {\phi} \beta^ {2}}{2}\right) \frac {6 K}{| \mathcal {B} |} \\ + 3 L ^ {2} \left(\beta - \frac {L _ {\phi} \beta^ {2}}{2}\right) \left(\tau^ {j} \delta_ {0} + \frac {1 2 \omega K \beta^ {2}}{(1 - \tau) | \mathcal {B} |} + 2 \omega \beta^ {2} \sum_ {t = 0} ^ {j - 1} \tau^ {t} \mathbb {E} \| G _ {j - 1 - t} \| ^ {2}\right) \\ \leq \mathbb {E} \phi (w _ {j}) - \left(\frac {\beta}{4} - \frac {L _ {\phi} \beta^ {2}}{4}\right) \mathbb {E} \| G _ {j} \| ^ {2} + \left(1 + \frac {6 \omega \beta^ {2} L ^ {2}}{1 - \tau}\right) \left(\beta - \frac {L _ {\phi} \beta^ {2}}{2}\right) \frac {6 K}{| \mathcal {B} |} \\ + 3 L ^ {2} \Big (\beta - \frac {L _ {\phi} \beta^ {2}}{2} \Big) \tau^ {j} \delta_ {0} + 6 \omega \beta^ {2} L ^ {2} \Big (\beta - \frac {L _ {\phi} \beta^ {2}}{2} \Big) \sum_ {t = 0} ^ {j - 1} \tau^ {t} \mathbb {E} \| G _ {j - 1 - t} \| ^ {2}, \\ \end{array}
$$
which, by taking the telescoping over  $j$  from 0 to  $J - 1$ , yields
$$
\begin{array}{l} \frac {1}{J} \sum_ {j = 0} ^ {J - 1} \left(\frac {1}{4} - \frac {L _ {\phi} \beta}{4}\right) \mathbb {E} \| G _ {j} \| ^ {2} \\ \leq \frac {\phi (w _ {0}) - \operatorname* {m i n} _ {w} \phi (w)}{\beta J} + \left(1 + \frac {6 \omega \beta^ {2} L ^ {2}}{1 - \tau}\right) \left(1 - \frac {L _ {\phi} \beta}{2}\right) \frac {6 K}{| \mathcal {B} |} \\ + \frac {3 L ^ {2} \left(1 - \frac {L _ {\phi} \beta}{2}\right) \delta_ {0}}{(1 - \tau) J} + 6 \omega \beta^ {2} L ^ {2} \left(1 - \frac {L _ {\phi} \beta}{2}\right) \frac {1}{J} \sum_ {j = 0} ^ {J - 1} \sum_ {t = 0} ^ {j - 1} \tau^ {t} \mathbb {E} \| G _ {j - 1 - t} \| ^ {2} \\ \leq \frac {\phi (w _ {0}) - \operatorname* {m i n} _ {w} \phi (w)}{\beta J} + \left(1 + \frac {6 \omega \beta^ {2} L ^ {2}}{1 - \tau}\right) \left(1 - \frac {L _ {\phi} \beta}{2}\right) \frac {6 K}{| \mathcal {B} |} \\ + \frac {3 L ^ {2} \left(1 - \frac {L _ {\phi} \beta}{2}\right) \delta_ {0}}{(1 - \tau) J} + 6 \omega \beta^ {2} L ^ {2} \left(1 - \frac {L _ {\phi} \beta}{2}\right) \frac {1}{(1 - \tau) J} \sum_ {j = 0} ^ {J - 1} \mathbb {E} \| G _ {j} \| ^ {2}. \\ \end{array}
$$
Rearranging the above inequality, we have
$$
\begin{array}{l} \frac {1}{J} \sum_ {j = 0} ^ {J - 1} \left(\frac {1}{4} - \frac {L _ {\phi} \beta}{4} - \frac {6 \omega \beta^ {2} L ^ {2}}{1 - \tau} \left(1 - \frac {L _ {\phi} \beta}{2}\right)\right) \mathbb {E} \| G _ {j} \| ^ {2} \\ \leq \frac {\phi \left(w _ {0}\right) - \min  _ {w} \phi (w)}{\beta J} + \left(1 + \frac {6 \omega \beta^ {2} L ^ {2}}{1 - \tau}\right) \left(1 - \frac {L _ {\phi} \beta}{2}\right) \frac {6 K}{| \mathcal {B} |} + \frac {3 L ^ {2} \left(1 - \frac {L _ {\phi} \beta}{2}\right) \delta_ {0}}{(1 - \tau) J}. \tag {26} \\ \end{array}
$$
Recalling the definition that  $\tau = 1 - \eta \mu + 6\omega \beta^2 L^2$ , and noting that we choose the stepsize  $\beta$  such that  $6\omega \beta^2 L^2 < \frac{1}{9}\eta \mu$  and  $\beta \leq \frac{1}{4L_{\phi}}$ , we can simplify Equation (26) as
$$
\frac {1}{1 6 J} \sum_ {j = 0} ^ {J - 1} \mathbb {E} \| G _ {j} \| ^ {2} \leq \frac {\phi (w _ {0}) - \min _ {w} \phi (w)}{\beta J} + \frac {2 7 K}{4 | \mathcal {B} |} + \frac {2 7 L ^ {2} \delta_ {0}}{8 \eta \mu J}.
$$
From the gradient descent based updates, we have  $\delta_0 = \left(1 + \frac{\rho^2M^2}{L^2\mu^2}\right)\mathbb{E}\| \theta_0^N -\theta_0^*\| ^2 +\mathbb{E}\| v_0^Q -v_0^*\| ^2\leq$ $(1 + \frac{\rho^2M^2}{L^2\mu^2})\| \theta_0^0 -\theta_0^*\| ^2 +\| v_0^0 -v_0^*\| ^2 < + \infty .$  Then, the proof is complete.
B Proof of Corollary 1
Based on the definition of  $C_Q = \frac{Q\rho M\eta}{\mu} + \eta^2 Q^2\rho M + \eta QL$  and noting that  $\eta \leq \frac{1}{L} \leq \frac{1}{\mu}$  and  $Q = 3$ , we have  $C_Q \leq 12\eta \left( \frac{\rho M}{\mu} + L \right)$ , which, combined with the definition that  $r = \frac{C_Q^2}{(\frac{\rho M}{\mu} + L)^2}$ , yields  $r \leq 144\eta^2$ . Then, based on the choice of  $N = 1$  and  $\lambda = \frac{\alpha\mu}{2}$ , we have
$$
(1 + \lambda) (1 - \alpha \mu) ^ {N} \Big (1 + \frac {8 r L ^ {2}}{\eta \mu} \Big) \leq \Big (1 - \frac {\alpha \mu}{2} \Big) \Big (1 + \frac {1 1 5 2 \eta L ^ {2}}{\mu} \Big) \leq 1 - \frac {\alpha \mu}{2} + \frac {1 1 5 2 \eta L ^ {2}}{\mu},
$$
which, in conjunction with  $\eta \leq \frac{\mu^2}{4608L^2}\alpha$  and  $\alpha \leq \frac{1}{L}$ , yields
$$
(1 + \lambda) (1 - \alpha \mu) ^ {N} \left(1 + \frac {8 r L ^ {2}}{\eta \mu}\right) \leq 1 - \frac {1}{4} \alpha \mu \leq 1 - \eta \mu . \tag {27}
$$
This implies that the inequality  $(1 + \lambda)(1 - \alpha \mu)^{N}(1 + \frac{8rL^{2}}{\eta\mu})\leq 1 - \eta \mu$  required by Theorem 1 is satisfied. Then, treating  $\mu ,\eta ,L,\rho ,M,\alpha ,\beta ,K,\| \theta_0^0 -\theta_0^*\| ^2$  and  $\| v_0^0 -v_0^*\| ^2$  as constants independent of the total number  $J$  of iterations, we have
$$
\frac {1}{J} \sum_ {j = 0} ^ {J - 1} \mathbb {E} \| G _ {j} \| ^ {2} \leq \mathcal {O} \left(\frac {1}{J} + \frac {1}{| \mathcal {B} |}\right). \tag {28}
$$
Then, to ensure an  $\epsilon$ -accurate stationary point, the number of iterations is  $\epsilon^{-2}$  with a batch size  $|\mathcal{B}| = \mathcal{O}(\epsilon^{-2})$ .
C Experiment Hyperparameters
The experimental hyperparameters are list in Table 7, including the memory size:  $|\mathcal{M}|$ , coreset size:  $|S|$ , stream batch size:  $|\mathcal{B}_t|$ , learning rate for  $M_{tr}$ :  $lr_t$ , learning rate for  $M_{cs}$ :  $lr_p$ , learning rate for sample weights:  $lr_w$ , regularization efficient:  $\lambda$ , epochs for training model:  $E$ , outer loops in bilevel optimization:  $J$ , inner loops in bilevel optimization:  $N$ , loops for estimating the Hessian-inverse-vector product:  $Q$ , a factor of Gaussian noise:  $\delta$ .
Table 7: Hyperparameters settings in experiments.
Hyperparameters	Permuted MNIST	Split CIFAR-100	Split Tiny-Imagenet	Multiple Datasets	Split Food-101
|M|	200	100	200	83	100
|S|	10	10	20	10	10
|Bt|	50	50	100	50	50
lr_t	0.005	0.15	0.20	0.1	0.15
lr_p	5.0	5.0	10	5.0	10
lr_w	5.0	5.0	10	5.0	10
λ	0.1	0.1	0.1	0.1	0.1
E	1	1	1	1	1
J	5	10	5	5	5
N	1	1	1	1	1
Q	3	3	3	3	3
δ	1e-3	1e-3	1e-3	1e-3	1e-3
D Running Time Comparison
We evaluate the running time (wall-clock time) of baseline methods on Split Tiny-[ImageNet and Split Food-101 in Table 8. All the training sections keep the same among these algorithms. Since there is no advanced coreset selection procedures in K-means Features, K-means Embedding, Uniform sampling, iCaRL, and Grad Matching, their running costs are pretty low but with worse performance (i.e., accuracy and forgetting) compared with our approach BCSR. PBCS and GCR, SPR and MetaSP don't involve bilevel formulations, they cannot guarantee accurate coreset sampling though with lower time cost against BCSR. Compared with OCS, BCSR takes  $23\%$  and  $35\%$  running time reduction on Tiny-[ImageNet and Split Food-101, respectively. Because similarity computing based on data pairs is computationally expensive for OCS. Greedy Coreset takes much more time than BCSR due to the usage of NTK.
Table 8: Running time.
Methods	Time (hours)
Tiny-ImageNet	Split Food-101
K-means Features	0.15	0.02
K-means Embedding	0.41	0.04
Uniform	0.13	0.03
iCaRL	0.41	0.04
Grad Matching	0.57	0.06
SPR	0.77	0.11
MetaSP	0.86	0.13
PBCS	1.13	0.15
GCR	0.91	0.13
Greedy Coreset	6.26	0.83
OCS	3.45	0.40
BCSR	2.65	0.26
E Experiments on Permuted-MNIST
The experiment results on Permuted-MNIST is presented in this section. BCSR outperforms other baselines significantly in AVG ACC on all the data settings, while showing relatively low forgetting.
Table 9: Experiment results on Permuted MNIST
Methods	Balanced	Imbalanced	Label Noise
AT	FGTT	AT	FGTT	AT	FGTT
K-means Features	54.30±0.93	0.064±0.012	39.46±0.11	0.023±0.003	46.71±0.64	0.094±0.011
K-means Embedding	54.78±1.83	0.056±0.009	41.97±2.02	0.013±0.005	47.36±1.58	0.082±0.015
Uniform	53.74±0.35	0.073±0.006	38.49±1.89	0.025±0.008	46.35±1.72	0.091±0.014
iCaRL	52.62±0.01	0.076±0.001	48.64±0.58	0.022±0.001	48.14±0.83	0.082±0.007
Grad Matching	54.76±1.61	0.065±0.011	33.67±1.24	0.028±0.003	48.24±0.42	0.089±0.009
SPR	54.24±0.45	0.068±0.019	40.79±0.83	0.031±0.003	48.23±0.75	0.067±0.004
MetaSP	54.63±0.31	0.059±0.006	41.32±0.94	0.025±0.006	48.84±0.77	0.061±0.007
Greedy Coreset	54.10±0.81	0.051±0.007	26.68±13.39	0.033±0.014	49.45±1.73	0.078±0.009
GCR	54.53±0.64	0.067±0.021	40.63±0.50	0.032±0.005	48.64±0.75	0.074±0.014
PBCS	51.61±1.14	0.144±0.021	41.14±0.23	0.119±0.007	39.74±1.98	0.178±0.001
OCS	54.37±0.34	0.026±0.001	50.19±0.47	0.020±0.005	48.08±1.44	0.046±0.003
BCSR	56.23±0.29	0.058±0.002	52.52±0.43	0.010±0.002	50.82±3.03	0.056±0.017
F Evolution of Average Accuracy
We present the learning process of different methods on Multiple Datasets in Figure 4. There are five tasks, where the average accuracy is tested after training each task. BCSR shows better performance than other baselines in different settings, including Balanced, Imbalanced, and label-noise.
(a) Balanced Multiple Datasets
(b) Imbalanced Multiple Datasets
(c) Label-Noise Multiple Datasets
Figure 4: Evolution of average accuracy during the continual learning process for Multiple Datasets.
G Possibility Distribution of Candidate Coreset
To explore the effect of top-  $K$  regularizer, we observe the possibility distribution of sample weights after each bilevel optimization, where sample weights of candidate coresets are initialized uniformly. The novel top-  $K$  regularizer makes sure that the summation of the top-  $K$  entries of the learned probability vector is large, such that we can confidently choose a coreset with size  $K$ . We show the weight distribution after optimization in Figure. 5, where you can find that top-  $K$  weights are much higher than others (with a margin  $2\%$ ), which easily distinguishes the top-  $K$  core samples from candidates.
Figure 5: Possibility distribution of candidate coreset for one mini-batch stream data.
Figure 6: The hypergradients in an outer-loop.
H The Effect of Top-  $K$  Regularizer
To further analyze the effects of top-  $K$  regularizer, we conduct the ablation study with different values of regularizer coefficient  $\lambda$  on balanced Split CIFAR-100. The performance results with different  $\lambda$  are shown in Table 10 and the corresponding average top-  $K$  summations of coreset weights are in Table 11. In our experiment, there are 50 candidate samples in each mini-batch data, and the summation of 50 coreset weights is equal to 1.00. Top-  $K$  summation of weights increases as  $\lambda$  increases, which imposes higher probabilities on top-  $K$  entries and lower probabilities on the rest candidates. The best performance is achieved when  $\lambda = 0.1$ , which means  $\lambda$  balances the trade-off between the loss function and regularizer strength: if  $\lambda$  is too large, the algorithm primarily focuses on choosing the important samples instead of updating the model parameter, and vice versa.
Table 10: Ablation study for the regularize coefficient  $\lambda$  .
Measure	λ=0.01	λ=0.05	λ=0.10	λ=0.50	λ=1.00
AT	59.37±0.35	60.23±0.43	61.60±0.14	59.42±1.45	58.89±1.64
FGTT	0.095±0.098	0.074±0.054	0.051±0.015	0.138±0.075	0.128±0.076
Table 11: Top-  $K$  summation/Total summation of coreset weights  $\left( {K = {10}}\right)$  .
Measure	λ=0.01	λ=0.05	λ=0.10	λ=0.50	λ=1.00
Top-K Sum/Total Sum	0.41/1.00	0.56/1.00	0.63/1.00	0.73/1.00	0.84/1.00
I Hypergradient Evolution
To demonstrate the efficiency of bilevel optimization, we illustrate the evolution of hypergradients for a bilevel optimization on Split CIFAR-100 in Figure. 6. We observe the average norm of hypergradient reduces from  $10^{-1}$  to less than  $10^{-2}$  in each round of coreset selection with loops equal to 10. The hypergradient curves show that our designed bilevel optimization provides both theoretical and practical convergence guarantees.
Table 12: The effect of coreset size
Methods	K=10AT	K=20AT	K=40AT
Uniform	58.99±0.54	53.57±2.93	53.03±1.97
Greedy Coreset	59.39±0.16	56.81±3.32	56.09±0.42
PBCS	55.64±2.26	49.84±1.76	40.95±0.32
OCS	52.57±0.37	54.87±0.58	56.46±0.07
BCSR	61.60±0.14	59.06±1.15	56.58±0.21
J The Effect of Coreset Size  $K$
Coreset size  $K$  also plays an important role in the experiment. We compare our BCSR with other coreset-based algorithms on different  $K$ . Note that coreset is selected from the current stream mini-batch  $\mathcal{B}_t$ , so the coreset size  $K$  satisfies  $K \leq |\mathcal{B}_t|$ . In the main experiment results,  $K = 10$  is fixed in all the algorithms for a fair comparison. Here, we set  $K = 10, 20, 40$  to conduct the continual learning experiments, respectively. The result is represented in Table 12. Note that all the results presented here are based on balanced Split CIFAR-100.
We mainly compare with the other four methods, including Uniform Sampling and three coreset-based methods, Greedy Coreset, PBCS, and OCS. We can observe that the performance of almost all methods becomes worse when coreset size is large. The reason is that if coreset size is larger and closer to  $\mathcal{B}_t$ , more redundant or noisy data are selected from the current stream mini-batch, and the coreset would not be representative anymore. In contrast, the smaller coreset could reduce the probability that redundant data are selected. Compared with other methods, BCSR shows the best performance (both accuracy and forgetting) and better robustness on different  $K$ .
K The Effect of Inner Loops  $N$  and Hessian-inverse-vector Product Loops  $Q$
We conduct ablation studies to explore the sensitivity of hyperparameters  $(N$  and  $Q)$  on Split CIFAR100. The results are presented in Tables 13  $(N)$  and Table 14  $(Q)$ . The model performance remains relatively stable when increasing inner loops  $(N)$  while fixing  $Q$ . But too large  $N$ $(N\geq 15)$  leads to performance degradation due to overfitting. The  $Q$  loops show similar properties that a few  $Q$  loops (e.g.,  $Q = 3$ ) are enough to approximate the Hessian-inverse-vector product. Too small  $Q$  and too large  $Q$  will hurt the performance due to possible underfitting (e.g.,  $Q = 1$ ) and overfitting (e.g.,  $Q = 20$ ).
Table 13: Ablation study for the inner loops (N) with fixed  $Q = 3$
Measure	N=1	N=5	N=10	N=15	N=20
AT	61.60±0.14	61.75±0.11	61.64±0.15	60.77±0.32	59.20±0.41
FGT	0.051±0.015	0.047±0.013	0.063±0.017	0.074±0.021	0.079±0.035
Table 14: Ablation study for the loops  $Q$  with fixed  $N = 1$ .
Measure	Q=1	Q=3	Q=5	Q=10	Q=20
AT	52.14±1.53	61.60±0.14	61.57±0.15	58.42±0.53	57.80±1.31
FGTT	0.123±0.038	0.051±0.015	0.064±0.012	0.173±0.045	0.162±0.041
L Datasets