SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Spit CIFAR-100. Balanced split CIFAR-100 is based on the original CIFAR-100 and is split into 20 tasks, each consisting of 5 disjoint classes. The imbalanced setting and label noise are also applied to this dataset to make the task more challenging. We follow [13] to transform the original dataset to imbalanced long-tailed CIFAR-100. In the label-noise setting, we randomly select  $20\%$  data in each task and randomly change their labels to an arbitrary label of 10 classes.
- Permuted MNIST. Balanced MNIST is a handwritten digits dataset [43] containing 20 tasks, where each task applies a fixed random permutation to the image pixels. For the imbalanced setting, we randomly select 8 classes over 10 and sample  $10\%$  of the selected classes for training. We also conduct the experiments on the label noise scenario, where symmetric label noise with  $20\%$  noise rate is imposed on the data. In particular, to make the problem setting more challenging, each task only retains 3000 training data randomly sampled from the original data.
- Multiple Datasets. This dataset [78] contains a couple of totally different datasets, including MNIST [43], fashion-MNIST [75], NotMNIST [8], Traffic Sign [69] and SVHN [54]. There are 5 tasks, and each task is constructed by randomly selecting 1000 training samples from a different dataset. The procedure of creating the dataset in the imbalanced and label-noise settings is the same as that in Split CIFAR-100.
- Split Tiny/ImageNet. This dataset [42] contains 100000 images of 200 classes (500 for each class) downsized to  $64 \times 64$  colored images. Each class has 500 training images, 50 validation images, and 50 test images. We construct the task sequences by splitting data into 20 tasks, where each task consists of 10 disjoint classes.
- Split Food-101. This dataset [7] is a challenging data set of 101 food categories, with 101'000 images. All the images are resized to  $64 \times 64$  pixels to be fed into models. To build continual learning tasks easily, We discard the last category and split data into 20 tasks, with 5 categories within each task. Other data settings, including imbalanced and lable-noise, are the same as Split CIFAR-100.
M Assumptions and Properties
We first provide standard definitions and assumptions for the convergence rate analysis of bilevel optimization [25, 36]. For notational convenience, we define  $\ell (\theta)\coloneqq \sum_{i = 1}^{n}\ell_{i}(\theta)$
Definition 1. A mapping  $f$  is  $L_{f}$ -Lipschitz continuous if for  $\forall z, z'$ ,  $\| f(z) - f(z') \| \leq L_{f} \| z - z' \|$ .
Assumption 1. The lower-level function  $L(\theta, w)$  is  $\mu$ -strongly-convex w.r.t.  $\theta$ .
Assumption 1 is a necessary geometric assumption in analyzing the convergence rate of bilevel optimization algorithms, as also widely adopted existing theories in [25, 37, 32]. We also note that this condition is satisfied for overparameterized neural networks [82]. The following assumption imposes some Lipschitz continuity conditions on the upper- and lower-level objective functions.
Assumption 2. The gradients  $\nabla_{\theta}L(\theta ,w),\nabla_{w}L(\theta ,w)$  and  $\ell (\theta)$  are  $L$  -Lipschitz continuous w.r.t.  $\theta$  and  $w$  . In addition, the gradient norm  $\| \nabla \ell (\theta^{*}(w))\| \leq M$
Note that we do not impose any conditions on the regularization function  $R(w, \delta)$ . The following assumption imposes the Lipschitz continuity on the second-order derivatives of the lower-level functions.
Assumption 3. The second-order derivatives  $\nabla_w\nabla_\theta L(\theta ,w)$  and  $\nabla_{\theta}^{2}L(w,\theta)$  are  $\rho$  -Lipschitz continuous.
Then, we use the following proposition to characterize the properties of the smoothed top- $K$  regularizer  $R(w,\delta)$ , based on the results in [23].
Proposition 1. The smoothed regularizer  $R(w, \delta)$  and its sampled version  $R(w, \delta; \widetilde{z})$  satisfy the following two important properties: (i) The gradient  $\nabla R(w, \delta)$  exists and is  $\frac{\sqrt{Kn}}{\delta}$ -Lipschitz continuous; (ii) The gradient norm  $\| R(w, \delta; \widetilde{z}) \|$  is bounded by  $\sqrt{K}$  for any sample  $\widetilde{z}$ .
Proposition 1 shows that the regularizer  $R(w, \delta)$  is smooth and its stochastic version  $R(w, \delta; \widetilde{z})$  is bounded. These two properties are important to guarantee the non-asymptotic convergence of our proposed bilevel method.