SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Experiment results on Multiple Datasets
Methods	Balanced	Imbalanced	Label Noise
AT	FGTT	AT	FGTT	AT	FGTT
K-means Features	54.63±0.88	0.138±0.007	33.63±2.66	0.136±0.063	45.46±3.50	0.120±0.049
K-means Embedding	56.83±1.65	0.106±0.019	35.93±1.60	0.106±0.031	46.32±3.19	0.084±0.030
Uniform	55.93±0.03	0.101±0.032	35.48±2.96	0.104±0.025	48.68±0.44	0.079±0.002
iCaRL	56.19±0.32	0.130±0.012	42.18±1.59	0.057±0.022	49.22±0.54	0.067±0.010
Grad Matching	53.41±0.46	0.119±0.020	38.16±3.90	0.082±0.003	46.96±1.05	0.091±0.029
SPR	56.20±1.91	0.124±0.036	40.79±1.73	0.143±0.051	49.77±1.58	0.062±0.024
MetaSP	57.14±1.10	0.113±0.042	41.32±1.50	0.103±0.053	47.14±1.66	0.081±0.027
Greedy Coreset	53.56±0.06	0.099±0.005	22.57±1.10	0.265±0.022	41.32±1.51	0.137±0.009
GCR	54.35±0.31	0.125±0.014	35.13±2.79	0.105±0.043	47.58±1.30	0.078±0.016
PBCS	52.93±0.28	0.152±0.016	37.25±2.93	0.115±0.033	47.51±1.56	0.101±0.021
OCS	55.65±2.26	0.062±0.001	40.48±1.39	0.051±0.003	45.03±4.16	0.049±0.012
BCSR	59.89±0.95	0.096±0.005	45.13±0.54	0.046±0.008	49.97±1.14	0.064±0.031
Table 3: Experiment results on Tiny-ImageNet
Methods	Balanced	Imbalanced	Label Noise
AVG ACC	FGT	AVG ACC	FGT	AVG ACC	FGT
K-means Features	41.20±0.75	0.131±0.004	36.27±0.30	0.079±0.014	36.68±1.35	0.095±0.004
K-means Embedding	41.48±1.21	0.129±0.007	36.29±0.23	0.085±0.003	36.01±1.51	0.083±0.005
Uniform	42.11±0.52	0.129±0.002	37.07±0.53	0.083±0.009	37.14±1.05	0.099±0.003
iCaRL	43.84±0.09	0.114±0.004	37.65±0.84	0.058±0.003	38.52±0.25	0.063±0.006
Grad Matching	43.45±0.32	0.105±0.007	37.58±0.39	0.066±0.004	38.84±0.42	0.064±0.006
SPR	42.79±0.50	0.102±0.009	36.55±0.74	0.070±0.026	39.89±0.53	0.065±0.021
MetaSP	43.33±0.32	0.127±0.002	36.75±0.57	0.086±0.006	37.18±0.76	0.068±0.007
Greedy Coreset	41.02±0.33	0.119±0.017	33.43±0.86	0.103±0.002	36.37±0.16	0.079±0.006
GCR	41.45±0.35	0.125±0.008	36.08±0.62	0.072±0.017	37.46±0.40	0.115±0.011
PBCS	36.99±0.15	0.177±0.002	35.88±0.16	0.071±0.080	37.02±0.16	0.133±0.029
OCS	41.29±0.09	0.112±0.001	35.09±0.60	0.036±0.011	35.36±0.94	0.061±0.005
BCSR	44.13±0.33	0.106±0.001	38.59±0.11	0.070±0.004	40.72±0.56	0.055±0.006
the task  $T$ .  $FGT_{T}$  evaluates the performance drop on the past tasks after training on the task  $T$ :  $FGT_{T} = \frac{1}{T}\sum_{j=1}^{T}\left[\max_{j\in 1,\dots T-1}(a_{j,i} - a_{T,i})\right]$ .
Implementation Details Following the implementation of the coreset-based algorithm [6], we use two models for the purposes of model training and coreset selection respectively, where the first model is denoted as  $M_{tr}$  for model training, and the second model (also known as a proxy model) is denoted as  $M_{cs}$  for coreset selection. In the section of the model training, we adopt a single-head MLP with two hidden layers for Permuted MNIST, and a ResNet-18 for Split CIFAR-100 and other datasets. In the section of the coreset selection,  $M_{cs}$  adopts the same architecture as  $M_{tr}$ . This is the key difference between our work and the Greedy Coreset [6]: they use Neural Tangent Kernels (NTK) [35] while we use a specific deep neural network. Note that NTK performs learning based on fixed features, which is shown to be limited [66]. In contrast, our algorithm allows hierarchical feature learning during the training process. In addition, our algorithm does not rely on discrete decision variables as in [6]: we use first-order methods with gradient and Hessian-inverse-vector product information with convergence guarantee and hence is more efficient in practice. For noncoreset reply methods SPR and MetaSP, we set the size of the method buffer as that in coreset-based methods. Especially for SPR, there are two memory buffers: the delayed buffer  $D$  temporarily stocks the incoming data stream, and the purified buffer  $P$  maintains the cleansed data. To satisfy the
Table 4: Experiment results on Food-101
Methods	Balanced	Imbalanced	Label Noise
AVG ACC	FGT	AVG ACC	FGT	AVG ACC	FGT
K-means Features	46.35±1.41	0.057±0.004	33.35±0.93	0.064±0.008	34.88±0.64	0.127±0.011
K-means Embedding	49.47±0.46	0.040±0.004	34.07±0.89	0.057±0.014	35.54±0.57	0.141±0.008
Uniform	46.50±0.13	0.048±0.001	34.61±0.16	0.060±0.007	36.69±1.13	0.123±0.007
iCaRL	48.12±0.78	0.035±0.001	36.83±1.06	0.032±0.001	39.22±0.30	0.084±0.010
Grad Matching	49.13±0.44	0.026±0.002	36.51±0.04	0.043±0.003	38.86±0.31	0.086±0.019
SPR	48.34±0.54	0.042±0.003	37.41±0.68	0.037±0.006	38.63±0.34	0.069±0.016
MetaSP	48.84±0.38	0.053±0.006	38.41±0.78	0.053±0.011	38.78±0.41	0.063±0.013
Greedy Coreset	49.19±0.91	0.030±0.008	38.78±0.01	0.038±0.001	38.61±0.18	0.096±0.004
GCR	47.03±1.17	0.048±0.004	36.99±0.30	0.042±0.005	35.78±1.08	0.126±0.013
PBCS	47.98±0.42	0.098±0.006	38.00±0.16	0.046±0.001	37.77±0.40	0.123±0.013
OCS	46.80±1.05	0.103±0.013	34.44±0.29	0.067±0.001	35.50±0.04	0.135±0.004
BCSR	51.30±0.62	0.039±0.008	39.05±0.99	0.044±0.001	39.63±0.22	0.060±0.071
(a) Permuted MNIST
(b) Imbalanced Permuted MNIST
(c) Label-Noise Permuted MNIST
Figure 2: The average accuracy during the continual learning. After each task, the training model is tested on all encountered tasks. Due to the nature of forgetting, the average test performance of all the methods tends to decrease.
Table 5: Random initialization with (W) and without (WO) bilevel optimizers (BO)
BO	Split CIFAR-100	Permuted MNIST	Multiple Datasets
AT	FGTT	AT	FGTT	AT	FGTT
WO	58.37±0.37	0.073±0.004	53.34±0.74	0.074±0.009	55.50±0.80	0.128±0.027
W	61.60±0.14	0.051±0.015	56.23±0.29	0.058±0.002	59.89±0.95	0.096±0.005
requirement of coreset experiments, we keep the size of  $P$  and  $D$  buffers the same. The detailed hyperparameter settings can be found in Appendix C.
We report the results of average accuracy (AVG ACC) and forgetting (FGT) for all the algorithms on the balanced, imbalanced, and label-noise benchmarks, and the results are presented in Table 1, Table 2, Table 3, Table 4, and Table 9 (Appendix E) respectively. We have the following observations. (i) In the balanced setting, our method outperforms other baselines significantly in terms of AVG ACC. For example, compared with the best coreset selection methods, our BCSR shows  $2.21\%$ ,  $4.24\%$ ,  $2.68\%$ ,  $2.11\%$ , and  $1.7\%$  improvements in AVG ACC on five benchmarks, respectively. (ii) In the imbalanced and label-noise setting, our method also demonstrates relatively higher performance. For example, on imbalanced data of Multiple Datasets, BCSR is better than the best coreset selection algorithm by  $4.65\%$  in average accuracy. An interesting observation is that Greedy Coreset [6] does not perform very well, especially in imbalanced and label-noise settings. The reason is that the inner optimization in Greedy Coreset is conducted by an NTK, where only a fixed feature is used but not the learned feature. (iii) From bilevel optimization point of view, our algorithm significantly reduces the running time of the vanilla greedy coreset selection method [6] by at least  $58\%$  on continual learning benchmark datasets. For detailed comparison, please check Table 8 in Appendix D. (iv) In addition, the test AVG ACC (Figure 2 and Figure 4 in Appendix F) of the BCSR during the training process shows that our algorithm alleviates catastrophic forgetting and it is comparable to other best baselines. For example, BCSR enjoys the lowest forgetting for almost all datasets under the label-noise setting (except for Multiple Datasets in Table 2). For most experiments on other settings, BCSR achieves a forgetting performance that is comparable to the best methods with at most  $1\%$  gap.
We conduct ablation studies to inspect the effectiveness of individual components in our proposed approach, including the effect of the bilevel optimizer, the smoothed top-  $K$  regularizer, and the coreset size  $K$  (Appendix J) respectively.
Random initialization with and without bilevel optimizers. Our new bilevel optimization algorithm (Algorithm 3) is of vital importance for finding a good distribution  $w$  for selecting the coreset. To demonstrate this, we design comparative experiments: one experiment initializes  $w$  randomly without updating it, and the other one adopts the same initialization strategy but updates  $w$  by the proposed bilevel optimizer. Then we sample coresets based on  $w$  from two methods respectively. As you can see, the random initialization method is equivalent to Uniform Sampling. We report the experimental results of average accuracy and forgetting on three balanced datasets in Table 5. It can be observed that the sampling strategy based on bilevel optimization significantly outperforms random sampling. In particular, the strategy with bilevel optimization shows  $3.23\%$ ,  $2.89\%$ , and  $4.39\%$  AVG ACC improvement on three benchmarks. Meanwhile, it also obtains  $2.20\%$ ,  $1.60\%$ , and  $3.20\%$  FGT reduction on three benchmarks.
Table 6: The impact of different values of  $\lambda$
λ	Split CIFAR-100	Permuted MNIST	Multiple Datasets
AT	FGT	AT	FGT	AT	FGT
0.00	60.43±0.15	0.064±0.005	54.20±1.53	0.116±0.030	55.71±0.32	0.055±0.003
0.01	60.56±2.05	0.074±0.016	55.09±2.94	0.097±0.020	55.39±0.95	0.065±0.012
0.10	61.04±0.53	0.063±0.007	57.20±0.58	0.064±0.010	55.69±0.40	0.057±0.002
1.00	59.43±1.20	0.072±0.008	55.43±1.53	0.108±0.019	58.18±0.77	0.046±0.006
5.00	58.80±1.58	0.084±0.017	53.13±2.74	0.120±0.021	56.67±0.78	0.051±0.007
10.00	59.53±0.42	0.075±0.012	53.54±2.02	0.118±0.029	55.57±1.05	0.060±0.010
(a) Split CIFAR-100
(b) Multiple Datasets
Figure 3: The upper loss of bilevel optimization. Each distinct spike means the arrival of a new task.
To verify the effectiveness of our bilevel optimizer, we compare the loss curve with Greedy Coreset that uses NTK. The result is presented in Figure 3. There are a number of stream mini-batches in each task. The bilevel optimizer trains over each mini-batch, where the upper loss is plotted in the figure. In the experiment, we plot the loss value for every 5 mini-batches. Within each task, the loss from BCSR gradually decreases with slight fluctuations, and it increases only when encountering a new task. In contrast, the loss value of the Greedy Coreset approach always stays large. It indicates that our bilevel optimizer is more effective than the Greedy Coreset approach.
Effectiveness of the regularizer. In our algorithm, the bilevel formulation has a smooth top-  $K$  loss as a regularizer in the objective function to promote the probability distribution to have  $K$  nonzero entries. The goal is to make sure that the summation of top-  $K$  entries of the learned probability vector is large, which increases confidence in the coreset selection. The hyperparameter  $\lambda$  is used to balance the cross-entropy loss and the regularization term. We explore the effects on the performance with different  $\lambda$ , and list the results in Table 6.
This ablation experiment is performed on our framework BCSR, and average accuracy and forgetting on three balanced benchmarks are reported. When  $\lambda = 0.10$ , BCSR can reach the best performance (The highest AVG ACC and lowest FGT) on Split CIFAR-100 and Permuted MNIST. While on Multiple Datasets, BCSR performs the best when  $\lambda = 1.0$ . This dataset contains more categories, and hence it is more challenging to select the appropriate representative samples. In this case, larger  $\lambda$  would emphasize more on maximizing the top- $K$  probability and help select better representative samples. In addition, we observe that  $\lambda$  set as a too large or too small value will damage the performance, which is in line with the observations in standard regularized empirical risk minimization problems such as overfitting and underfitting. Please refer to Appendix H for further analysis.
6 Theoretical Analysis
In this section, we provide a theoretical analysis for our proposed method. In particular, we establish convergence rate for our algorithm BCSR.
Theorem 1. Suppose standard assumptions hold in bilevel optimization (see Assumptions 1, 2 and 3 in Appendix M). Choose parameters  $\lambda$ ,  $\alpha$ ,  $\eta$  and  $N$  such that  $(1 + \lambda)(1 - \alpha \mu)^{N} \left(1 + \frac{8rL^{2}}{\eta\mu}\right) \leq 1 - \eta \mu$ , where  $r = \frac{C_Q^2}{(\frac{\rho M}{\mu} + L)^2}$  and  $C_Q = \frac{Q\rho M\eta}{\mu} + \eta^2 Q^2\rho M + \eta QL$ . Furthermore, choose the stepsize  $\beta$  such that  $6\omega\beta^2 L^2 < \frac{1}{9}\eta\mu$  and  $\beta \leq \frac{1}{4L_{\phi}}$ , where the constant  $\omega$  is given by Equation (16) in Appendix A.
Then, we have the following convergence result.
$$
\frac {1}{J} \sum_ {j = 0} ^ {J - 1} \mathbb {E} \| G _ {j} \| ^ {2} \leq \mathcal {O} \Big (\frac {D _ {\phi}}{\beta J} + \frac {1}{| \mathcal {B} |} + \frac {D _ {0}}{\eta \mu J} \Big).
$$
where  $G_{j} \coloneqq \frac{1}{\beta} (w_{j} - \mathcal{P}_{\Delta^{n}}(w_{j} - \beta \nabla \phi (w_{j})))$  denote the generalized projected gradient,  $D_{\phi} \coloneqq \phi (w_0) - \min_w\phi (w) > 0$ ,  $D_0 = \| \theta_0^0 -\theta_0^*\| ^2 +\| v_0^0 -v_0^*\| ^2$ ,  $L_{\phi}$  is the smoothness parameter of the total objective  $\phi (w)$  whose form is proved in Appendix A.
Theorem 1 provides a general convergence result for the proposed bilevel algorithm, which allows for a very flexible selection of subloop lengths  $N$  and  $Q$  as long as the inequality  $(1 + \lambda)(1 - \alpha \mu)^N (1 + \frac{8rL^2}{\eta\mu}) \leq 1 - \eta \mu$  holds given proper stepsizes  $\lambda, \eta, \alpha$ . For example, in most of our experiments, the choice of  $N = 1$  and  $Q = 3$  works the best. Then, in this case, we further specify the parameters in Theorem 1, and provide the following corollary.
Corollary 1. Under the same setting as in Theorem 1, choose  $N = 1$ ,  $Q = 3$  and set  $\lambda = \frac{\alpha\mu}{2}$ ,  $\eta \leq \frac{\mu^2\alpha}{4608L^2}$  and  $\alpha \leq \frac{1}{L}$ . Then, to make sure an  $\epsilon$ -accurate stationary point, i.e.,  $\frac{1}{J}\sum_{j=0}^{J-1}\mathbb{E}\|G_j\|^2 \leq \epsilon^2$ , the number of iterations is  $\mathcal{O}(\epsilon^{-2})$ , each using  $|\mathcal{B}| = \mathcal{O}(\epsilon^{-2})$  of samples from the standard Gaussian distribution  $\mathcal{N}(0,1)$ .
Corollary 1 shows that the proposed bilevel algorithm converges to an  $\epsilon$ -accurate stationary point using only  $\mathcal{O}(\epsilon^{-2})$  iterations and  $\mathcal{O}(\epsilon^{-2})$  samples drawn from  $\mathcal{N}(0,1)$  per iteration. Note the large batch size  $|\mathcal{B}| = \mathcal{O}(\epsilon^{-2})$  is necessary here to guarantee a convergence rate of  $\mathcal{O}(1 / T)$ , which matches the results in solving single-level constrained nonconvex problems [24]. In addition, it is computationally tractable because sampling from a known Gaussian distribution is easy and cheap.
In this paper, we advance the state-of-the-art bilevel coreset selection in continual learning. We first introduce a new bilevel formulation with smoothed top-  $K$  regularization and then design an efficient bilevel optimizer as a solver. We conduct extensive experiments in continual learning benchmark datasets to demonstrate the effectiveness of our proposed approach. We also show that the bilevel optimizer can efficiently find  $\epsilon$ -stationary point with  $O(1 / \epsilon^4)$  computational complexity, which matches the best complexity of projected SGD for a single-level problem.
We would like to thank the anonymous reviewers for their helpful comments. Jie Hao and Mingrui Liu are both supported by a grant from George Mason University. Computations were run on ARGO, a research computing cluster provided by the Office of Research Computing at George Mason University (URL: https://orc.gmu.edu).
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte-laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139-154, 2018.
[2] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems 32, pages 11849-11860. 2019.
[3] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.
[4] Michael Arbel and Julien Mairal. Amortized implicit differentiation for stochastic bilevel optimization. In International Conference on Learning Representations, 2022.
[76] Ju Xu and Zhanxing Zhu. Reinforced continual learning. Advances in Neural Information Processing Systems, 31, 2018.
[77] Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. International Conference on Learning Representations, 2020.
[78] Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085, 2021.
[79] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
[80] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
[81] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3987-3995. JMLR.org, 2017.
[82] Guodong Zhang, James Martens, and Roger B Grosse. Fast convergence of natural gradient descent for over-parameterized neural networks. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[83] Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Probabilistic bilevel coreset selection. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szeptesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27287-27302. PMLR, 17-23 Jul 2022.
[84] Xiangyu Zhu, Jie Hao, Yunhui Guo, and Mingrui Liu. Auc maximization in imbalanced lifelong learning. In Uncertainty in Artificial Intelligence, pages 2574-2585. PMLR, 2023.
Let  $\Delta^{(n)}\coloneqq \{w:0\leq w_{(i)}\leq 1,||w||_1 = 1\}$  denote the constraint set of the upper-level problem. We first provide some important inequalities.
Recall that  $v_{j}^{q}$  be the  $q^{th}$  GD iterate in solving the linear system  $\nabla_{\theta}^{2}L(\theta_{j}^{N},w_{j})v = \sum_{i = 1}^{n}\nabla \ell_{i}(\theta_{j}^{N})$  at iteration  $j$  via the following process:
$$
v _ {j} ^ {q + 1} = \left(I - \eta \nabla_ {\theta} ^ {2} L \left(\theta_ {j} ^ {N}, w _ {j}\right)\right) v _ {j} ^ {q} + \eta \sum_ {i = 1} ^ {n} \nabla \ell_ {i} \left(\theta_ {j} ^ {N}\right). \tag {6}
$$
which, by telescoping Equation (6) over  $q$  from 0 to  $Q - 1$ , yields
$$
v _ {j} ^ {Q} = \left(I - \eta \nabla_ {\theta} ^ {2} L \left(\theta_ {j} ^ {N}, w _ {j}\right)\right) ^ {Q} v _ {k} ^ {0} + \eta \sum_ {q = 0} ^ {Q - 1} \left(I - \eta \nabla_ {\theta} ^ {2} L \left(\theta_ {j} ^ {N}, w _ {j}\right)\right) ^ {q} \sum_ {i = 1} ^ {n} \nabla \ell_ {i} \left(\theta_ {j} ^ {N}\right). \tag {7}
$$
Let  $v_{j}^{*}$  be the solution of the linear system  $\nabla_{\theta}^{2}L(\theta_{j}^{*},w_{j})v = \sum_{i = 1}^{n}\nabla \ell_{i}(\theta_{j}^{*})$ , and then we have
$$
v _ {j} ^ {*} = \left(I - \eta \nabla_ {\theta} ^ {2} L \left(\theta_ {j} ^ {*}, w _ {j}\right)\right) ^ {Q} v _ {j} ^ {*} + \eta \sum_ {q = 0} ^ {Q - 1} \left(I - \eta \nabla_ {\theta} ^ {2} L \left(\theta_ {j} ^ {*}, w _ {j}\right)\right) ^ {q} \sum_ {i = 1} ^ {n} \nabla \ell_ {i} \left(\theta_ {j} ^ {*}\right). \tag {8}
$$
Combining Equation (6) and Equation (7), noting that  $v_{j}^{0} = v_{j - 1}^{Q}$  and using Assumption 2 that  $\| v_{j}^{*}\| \leq \| (\nabla_{\theta}^{2}L(\theta_{j}^{*},w_{j}))^{-1}\| \| \sum_{i = 1}^{n}\nabla \ell_{i}(\theta_{j}^{*})\| \leq \frac{M}{\mu}$ , the difference between  $v_{j}^{Q}$  and  $v_{j}^{*}$  can be bounded as
$$
\begin{array}{l} \left\| v _ {j} ^ {Q} - v _ {j} ^ {*} \right\| \leq \left\| \left(I - \eta \nabla_ {\theta} ^ {2} L \left(\theta_ {j} ^ {N}, w _ {j}\right)\right) ^ {Q} - \left(I - \eta \nabla_ {\theta} ^ {2} L \left(\theta_ {j} ^ {*}, w _ {j}\right)\right) ^ {Q} \right\| \frac {M}{\mu} + (1 - \eta \mu) ^ {Q} \left\| v _ {j - 1} ^ {Q} - v _ {j} ^ {*} \right\| \\ + \eta M \left\| \sum_ {q = 0} ^ {Q - 1} (I - \eta \nabla_ {\theta} ^ {2} L (\theta_ {j} ^ {N}, w _ {j})) ^ {q} - \sum_ {q = 0} ^ {Q - 1} (I - \eta \nabla_ {\theta} ^ {2} L (\theta_ {j} ^ {*}, w _ {j})) ^ {q} \right\| \\ + (1 - (1 - \eta \mu) ^ {Q}) \frac {L}{\mu} \| \theta_ {j} ^ {*} - \theta_ {j} ^ {N} \|. \tag {9} \\ \end{array}
$$
We next bound  $\Delta_q \coloneqq \|(I - \eta \nabla_\theta^2 L(\theta_j^N, w_j))^q - (I - \eta \nabla_\theta^2 L(\theta_j^*, w_j))^q\|$  in Equation (9) as:
$$
\Delta_ {q} \stackrel {(i)} {\leq} (1 - \eta \mu) \Delta_ {q - 1} + (1 - \eta \mu) ^ {q - 1} \eta \rho \| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \|. \tag {10}
$$
which, by telescoping Equation (10) and in conjunction with Equation (9), yields
$$
\begin{array}{l} \| v _ {j} ^ {Q} - v _ {j} ^ {*} \| \leq Q (1 - \eta \mu) ^ {Q - 1} \eta \rho \frac {M}{\mu} \| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \| + (1 - \eta \mu) ^ {Q} \| v _ {j - 1} ^ {Q} - v _ {j} ^ {*} \| \\ + \eta M \sum_ {q = 0} ^ {Q - 1} q (1 - \eta \mu) ^ {q - 1} \eta \rho \| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \| + (1 - (1 - \eta \mu) ^ {Q}) \frac {L}{\mu} \| \theta_ {j} ^ {*} - \theta_ {j} ^ {N} \| \tag {11} \\ \leq \frac {Q (1 - \eta \mu) ^ {Q - 1} \rho M \eta}{\mu} \| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \| + (1 - \eta \mu) ^ {Q} \| v _ {j - 1} ^ {Q} - v _ {j - 1} ^ {*} \| \\ + (1 - \eta \mu) ^ {Q} \| v _ {j - 1} ^ {*} - v _ {j} ^ {*} \| + \frac {1 - (1 - \eta \mu) ^ {Q} (1 + \eta Q \mu)}{\mu^ {2}} \rho M \| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \| \\ + (1 - (1 - \eta \mu) ^ {Q}) \frac {L}{\mu} \| \theta_ {j} ^ {*} - \theta_ {j} ^ {N} \| \\ \end{array}
$$
which, combined with  $\| v_{j}^{*} - v_{j - 1}^{*}\| \leq \left(\frac{L}{\mu} +\frac{M\rho}{\mu^{2}}\right)\left(\frac{L}{\mu} +1\right)\| w_{j} - w_{j - 1}\|$  and using the fact that  $(1 - x)^{Q}\geq 1 - xQ$  for  $0\leq x\leq 1$ , yields
$$
\begin{array}{l} \mathbb {E} \| v _ {j} ^ {Q} - v _ {j} ^ {*} \| ^ {2} \leq (1 - \eta \mu) \mathbb {E} \| v _ {j - 1} ^ {Q} - v _ {j - 1} ^ {*} \| ^ {2} + \frac {4}{\eta \mu} C _ {Q} ^ {2} \| \theta_ {j} ^ {*} - \theta_ {j} ^ {N} \| ^ {2} \\ + \frac {4}{\eta \mu} \left(\frac {L}{\mu} + \frac {M \rho}{\mu^ {2}}\right) ^ {2} \left(\frac {L}{\mu} + 1\right) \| w _ {j} - w _ {j - 1} \| ^ {2} \tag {12} \\ \end{array}
$$
where the constant  $C_Q \coloneqq \frac{Q\rho M\eta}{\mu} + \eta^2 Q^2\rho M + \eta QL$ .
The next step is to characterize the error induced by the lower-level updates on  $\theta$ .
Note that  $\theta_j^* = \arg \min_{\theta} L(\theta, w_j)$ . Using Assumptions 1 and 2, we have
$$
\left\| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \right\| ^ {2} \leq (1 - \alpha \mu) ^ {N} \left\| \theta_ {j} ^ {0} - \theta_ {j} ^ {*} \right\| ^ {2}, \tag {13}
$$
which, in conjunction with  $\theta_j^0 = \theta_{j - 1}^N$  and Lemma 2.2 in [25], yields
$$
\begin{array}{l} \mathbb {E} \left\| \theta_ {j} ^ {N} - \theta_ {j} ^ {*} \right\| ^ {2} \leq (1 - \alpha \mu) ^ {N} (1 + \lambda) \mathbb {E} \left\| \theta_ {j - 1} ^ {N} - \theta_ {j - 1} ^ {*} \right\| ^ {2} \\ + (1 - \alpha \mu) ^ {N} \left(1 + \frac {1}{\lambda}\right) \frac {L ^ {2}}{\mu^ {2}} \mathbb {E} \| w _ {j} - w _ {j - 1} \| ^ {2}. \tag {14} \\ \end{array}
$$
Recall the definition that  $r = \frac{C_Q^2}{\left(\frac{\rho M}{\mu} + L\right)^2}$ . Then, combining Equation (12) and Equation (14) yields we can obtain