SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A typical real-world application of CML is the online recommendation system, where we aim to predict the user's preference for various products. Different users' preferences are considered as different task distributions. The distribution of users can be regarded as a task environment. The recommendation system maintains a meta-model that predicts the recent preferences shared across users and adapts task-specific models that predict the user-specific preferences.
The user queries for displaying products arrive sequentially, and the system randomly distributes several (e.g.,  $m_t = 200$ ) products in response to the query. Denote the products and the corresponding preferences for  $t$ -th user as  $(X_i, Y_i) \sim \mu_t, \forall i \in [m_t]$ , which are i.i.d.. The labels are initially unknown for this setting, but a few labels can be acquired through the following interaction:
- The meta-model predicts the preferences over products and displays them according to the predicted preference order.
- Normally, the user will only view a small part ( $m_t' = 50$ ) of the distributed products, and the preference can be determined with the clicks or view time over these products, which gives the few-shot labeled samples.
- Then, the task-specific model is adapted from the meta-model with these examples. We hope the model can generalize on the rest of the unseen examples with correct preference prediction, which can provide new recommendations.
- Finally, the task-specific model of each user is used for updating the meta-model. Since the shared preferences change with the season, fashion trends, and accidental events, the task environment is shifting.
G.3 More details on DCML
Relate parameters in eq. 1 to specific base learner Since the proposed framework is valid for different base learners, the parameters like  $a$ ,  $b$ , and  $\kappa$  in the unified form of excess risk upper bounds
(cost function) are related to the specific algorithm. Since all of the experiments adopt SGD, we give here a detailed description of it.
According to Proposition D.4, the  $K$ -step SGD with learning rate  $\eta_t$  and initialization  $\phi_t$  for the  $t$ -th task has the following cost function:
$$
f (\beta_ {t}, \phi_ {t}) = \kappa_ {t} \left(\frac {1}{2} \beta_ {t} + \frac {2 \alpha \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {0} + 2 \kappa_ {t} ^ {2}}{\beta_ {t}} + 2 \kappa_ {t}\right),
$$
where  $a = \frac{1}{2}, b = 2\alpha, \kappa_t = L\sqrt{\frac{1}{K\alpha} + \frac{2}{m_t\alpha}}, \epsilon_t = 2\kappa_t^2, \Delta_t = 2\kappa_t$ . The learning rate of the base learner  $\eta_t = (\frac{\beta_t}{\kappa_t} + 2) / (K\alpha)$  is related to the sample number  $m_t$  of each task via  $\kappa_t$ , which can reflect the generalization.
Impact of the initial meta-parameter on the model's behavior The initial meta-parameter substantially affects the model's behavior. For instance, the aforementioned recommendation system suffers from a well-known cold-start problem. A well-pertained model initialization  $\phi_{1}$  can help address the problem.
Given a good initialization  $\phi_1$ ,  $\beta_{1}$  also affects the performance substantially. According to the meta-parameter adaptation, if  $\beta_{1}$  is too large, the algorithm tends to keep the prior knowledge and will not learn from the data. If  $\beta_{1}$  is too small, the algorithm will ignore the initialization and learn from scratch.
How does DCML adjust the meta-parameter The updates in Algo. 1 for the meta-parameter  $u = (\beta_t, \phi_t)$  are:
$$
\phi_ {t + 1} = (1 - \frac {2 b \kappa_ {t} \gamma_ {t}}{\beta_ {t}}) \phi_ {t} + \frac {2 b \kappa_ {t} \gamma_ {t}}{\beta_ {t}} w _ {t}, \beta_ {t + 1} = \beta_ {t} - \gamma_ {t} (a \kappa_ {t} - \frac {\kappa_ {t} (b \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {t} + \epsilon_ {0})}{\beta_ {t} ^ {2}}).
$$
We adjust the learning rate of the meta-parameter  $(\gamma_{t})$  with the following strategy:
- For  $k$ -th task inside the  $n$ -th environment (slot),  $\gamma_{t} = \gamma_{0} / \sqrt{k}$ , where the initialization is determined by the theoretical optimal value  $\gamma_{0} = \frac{\epsilon_{0}}{\kappa_{t}}\sqrt{\frac{(1 + b / a)\hat{D}^{2} + \epsilon_{t} / \alpha}{4ab^{2}\hat{D}^{2}\epsilon_{0} + a^{2}(b*\hat{D}^{2} + \epsilon_{t}^{2})^{2}}}$ .
- When an environment change is detected,  $\gamma_{t}$  is set to a large hopping rate  $\gamma_{t} = \rho$ , which is related to the path length.
How to set hyper-parameters In the experiments, we tune the following hyper-parameters  $\alpha, K, L, \hat{D}, \epsilon_0, \rho, \beta_1$  by random search on the defined space in Tab. 8. Specifically for the simple moving 2D Gaussian: since the loss function is 2-strongly convex, we have  $\alpha = 2$ .
G.4 Limitations
The proposed theory works for non-convex loss when using the Gibbs algorithm. However, We do not offer analysis for SGD with non-convex loss, where it's hard to find an easily estimable form for the excess risk upper bound. Excess risk analysis for gradient-based learning algorithms with non-convex loss is a promising but challenging direction for the deep learning community. We hope to work on this topic in the future.
In this paper, we do not provide theoretical guarantees for memory-based approaches, which can better address meta-level forgetting with additional memory cost. Future theoretical studies can be conducted to complete the framework.
G.5 Broader Impacts
In general, this work is fundamental. It aims to understand the bi-level learning-forgetting trade-off in CML and improve the model performance in real-time intelligent systems with a better trade-off balance. The potential negative impacts depend on the specific application.
Potential negative impacts Since the main objective is to obtain the optimal average excess risk over rounds, when each task relates to a person in some applications like personalized recommendation systems or medical diagnosis systems, it may cause some fairness issues for individuals, especially those who appear around the changing points.
How to address? The aforementioned potential unfairness can be mitigated by adding calibration modules as in [53-55] or providing more training data for tasks around changing points.
H Additional Related Works
H.1 Summary and comparison of related settings
Table 9: Comparison of different settings adapted from [13], where  $S$  and  $Q$  represent the train and test data from the same distribution, respectively. Train data  $S = \bar{S} \cup \tilde{S}$  is further divided into the support and query sets.  ${\mu }_{t}$  is a sub-task in CL with i.i.d. data,  $C{L}{P}_{1 : M}$  represent whole CL problems of infinite non-i.i.d. data stream and  $p\left( T\right)$  is the distribution over them.
Settings	Task Datasets	Tasks & Environments	Task Parameter (base)	Meta Updates	Evaluation
SL	S, Q ~ μ	-	w = A(S)	-	L(w, Q)
CL	S1:T, Q1:T ~ μ1:T	μ1:T ~ τ (static) μ1:T ~ τ1:T (shifting)	w = ACL(S1:T)	-	∑t=1T L(w, Qt)
ML	S1:M ~ μ1:M (train tasks), SM:N ~ μM:N (test tasks)	μ1:N ~ τ	wi = A(u, Si), ∀i ∈ [M], M < N	∇u ∑i=1M L(A(u, Ši), Ši)	∑N=M ∑L(A(u, Ši), Ši)
CML	S1:T, Q1:T ~ μ1:T	μ1:T ~ τ (static) μ1:T ~ τ1:T (shifting)	wt = A(ut, St)	∇f(t) or ∇u_t L(A(ut, Št), Št)	∑t L(A(ut, Št), Qt)
MCL	Si,t:1:T, Qi,t:1:T ~ CLPi, ∀i ∈ [M]	CLP1:M ~ p(T)	wi = ACL(u, Ši,1:T)	∇u ∑t L(ACL(u, Ši,1:T), Ši,t)	∑M i=1 ∑t L(ACL(u, Ši,1:T), Qt,i)
We present different settings related to the Continual Meta-Learning (CML) problem studied in this paper in Tab. 9. The detailed discussions in each setting can be found in the following sections.
Supervised Learning (SL) methods learn as a single task from the train data and evaluate on independent test data. The data points are assumed, i.i.d. sampled from the same distribution. Continual Learning (CL) aims to learn from a sequence of non-i.i.d. tasks, where the data points inside each task are i.i.d. Meta-Learning (ML) has many settings. Here, we present the statistical version. All the tasks are assumed to be i.i.d. sampled from the same task environment. The evaluation is conducted on test tasks to measure the generalization performance of the algorithm on new tasks.
In Meta-Continual Learning (MCL), a continual learning prediction problem (CLP) is defined as a non-i.i.d. data stream, which is a task sampled from the stationary distribution  $p(\mathcal{T})$ . Random subsequences of dependent data points of length  $k$  are sampled from each CLP task for training. The major difference between CML and MCL settings is that the non-stationarity of CML comes from the sequentially encountered non-i.i.d. task distributions. In contrast, the non-stationarity of MCL comes from the non-i.i.d. data points within a task.
H.2 Continual Learning
Except for the taxonomy w.r.t different approaches mentioned in the main paper, CL can also be divided into the following three scenarios w.r.t the different properties of the task distributions [56, 57]. Class Incremental Learning (CIL) [58] involves learning new classes over time without having access to all classes at once during training. The label space is growing over time, where  $\mathcal{Y}_t \subset \mathcal{Y}_{t+1}, \forall t$ . In the final step, the predictor is aimed to be capable of classifying all the seen classes. Domain Incremental Learning (DIL) [59] defines on the same label space  $\mathcal{Y}$  that has a fixed associated semantic meaning, and the marginal distributions on input space  $\mu_t(\mathcal{X})$  varies over time. Task Incremental Learning (TIL) [5] can define on different label spaces  $\mathcal{Y}_t \neq \mathcal{Y}_{t+1}, \forall t$  or the same label space  $\mathcal{Y}$  associated to different semantic meanings over tasks. In conventional definitions, CIL and DIL are under a weak task-agnostic setting (the task identities or task boundaries are unknown during testing but known during training), while TIL is under the task-aware setting with known task identities. However, with increasing research diversities, TIL has also been studied in task-agnostic settings recently. In this paper, we study CML within a similar TIL scenario of CL.
Key Differences between CL and CML The two settings share the common objective of addressing the stability-plasticity dilemma, which is the typical objective for learning from non-stationary data. Their key differences are:
Continual Learning:
- Standard CL methods sequentially adapt a task model and aim for a final model that performs well on all the tasks encountered. (Some memory-based approaches like dynamic architecture grow a subnet for each task.)
- The current task model is adapted from the previous task model.
- Standard CL reports the final performance of the agent on all the tasks at the end of its lifetime.
- Standard CL approaches mainly focus on minimizing catastrophic forgetting without considering quick generalization for tasks with few-shot data, e.g., the replay-based and regularization-based methods.
Continual Meta-Learning:
- CML methods sequentially adapt a meta-model and aim to recover the performance on previous tasks by adapting the meta-model with few additional samples. Therefore, it's more suitable for learning few-shot tasks. (Some memory-based approaches grow a subnet for each environment.)
- The current task model is adapted from the meta-model.
CML reports the cumulative performance of the agent throughout its lifetime.
- CML focuses on quick generalization for new tasks and fast recovering performance of previous tasks with few-shot data.
H.3 Meta-Continual Learning
Javed and White [60] and Gupta et al. [61] study a different Meta-Continual Learning (MCL) setting. A continual learning prediction problem (CLP) is defined as a non-i.i.d. data stream, which can be considered a CLP task  $\mathcal{T}$ . The CLP tasks are sampled from a stationary distribution  $p(\mathcal{T})$ , where for each CLP task, random subsequences of dependent data points are sampled from the task for training. [60] conducts offline meta-representation learning w.r.t two objectives MAML-REP and OML. MAML-REP uses batch data for inner adaptation. OML is calculated with a single data point for each inner adaptation, which can reflect the degree of forgetting in CLP tasks. Gupta et al. [61] proposes La-MAML, which adopts a replay buffer and conducts meta-initialization with optimization on the OML objective in an online manner. In addition, La-MAML adopts a modulation of per-parameter learning rates in the meta-learning updates. The optimization of learning rates in La-MAML is w.r.t empirical loss, while our learning rates optimization is based on the excess risk upper bound. So, our approach suffers less from over-fitting.
H.4 Continual Meta-Learning
Static environments Harrison et al. [32] studies continual meta-learning (CML) under task agnostic setting, where the task boundaries are unknown. It proposed an algorithmic framework MOCA that incorporates different meta-learning methods with Bayesian Online Changing-point Detection (BOCD) to identify unknown task boundaries during the meta-learning process. The task environment is still assumed static. On the contrary, we focus on the bi-level trade-off in shifting task environments with known task boundaries, where we detect the environmental distribution shift, not the task distribution shift.
Shifting environments To address the environment shift, Jerfel et al. [14] proposes a non-parametric Dirichlet process mixture of hierarchical Bayesian models that allows adaptively adding new task clusters and selecting over a set of learned meta-initialization parameters for the newly encountered task. A similar approach is also used to detect the task shift in a task-agnostic setting. For both task-agnostic and task-aware settings, [14] has tested the proposed algorithm on datasets with two times environment shifts. The proposed method requires additional  $\mathcal{O}(K)$  memory for
storing the initialization parameters for the detected  $K$  clusters. For each task, it needs to update the task-specific model and the meta-model for all the  $K$  initializations.  $K$  can grow with the time.
Zhang et al. [15] use a Dynamic Gaussian Mixture Model (DGMM) to model the distribution of the meta-parameters. Different from [14], which applies point estimation during inference, [15] derived a structured variational inference method that can reduce overfitting. The experiment is conducted with three times environment shifts of four datasets. [15] needs  $\mathcal{O}(K)$  extra memory for storing the meta-parameters.
Wang et al. [16] studies a slightly different setting where each environment has a super long task sequence. In this case, the cartographic forgetting for meta-knowledge is more obvious. To address this, they use a memory buffer to store a small number of training tasks from previous environments and a shared representation for different task environments and grow the subnets when a new environment is detected on latent space using BOCD. For deciding which cluster the task belongs to, they store the average embedding for each environment for calculating the distances. Hence, its memory complexity is  $\mathcal{O}(K + M)$ .
H.5 Meta-Learning
Statistical Learning to Learn (LTL) LTL [62] has been intensively studied with various theoretic tools. The early theoretical framework was proposed by Baxter [24], where they first defined the notion of the task environment and derived the uniform convergence bounds. And the tasks are i.i.d sampled from the environment. Denevi et al. [63, 28] provided excess risk for ridge regression and discussed the train-validation split impact. Amit and Meir [25] applied PAC Bayes theory and provided generalization bounds for stochastic neural networks. They also derived a joint training algorithm that simultaneously updates the meta-parameter and task parameters, which cannot be extended to the sequential learning scenario. Recently, Chen et al. [26] derived an information-theoretic generalization bound for the MAML-like algorithm, which provides non-vacuous bounds for deep few-shot learning and can be applied to sequential task learning.
Statistical Lifelong Learning The difference between Lifelong learning [64] and LTL is that the tasks are observed sequentially, while LTL has all the tasks in hand. Pentina and Lampert [65] first applied PAC Bayes theory to Lifelong Learning, where they also assumed that all the tasks are, i.i.d. sampled from the task environment. In a subsequent work [66], they relaxed the i.i.d. assumption with two scenarios. The first is that the tasks are dependent, but a dependency graph is known so that they can prove a statistic bound. In the second, they also consider a shifting environment change. However, they assume that the base learner output of the current task only depends on the current and the previous task data, and the expected performance of the base learner does not change with time. Under these assumptions, they obtain statistical guarantees for the gradually changing environment.
Online LTL The online LTL methods have all the tasks in hand, but at each round  $t$ , they sample a task from the  $N$  tasks in hand. Cavallanti et al. [67] works on this setting for online multi-task learning. Multi-task learning is often treated as a simplified version of LTL in previous works [68].
Online Meta-Learning Two main settings to meta-learn sequential tasks in an online manner are referred to as online-within-online (OWO) [28-31] and batch-within-online (BWO) [9, 27]. The former applies online algorithms for both base and meta learners. The latter treats task-level learning differently as a statistical batch setting. The BWO setting is close to the CML studied in this paper, while the theoretical work is rare, and none of the previous BWO methods considers shifting task environments. Khodak et al. [31] first considered shifting environments in the OWO setting, where the base learner is the gradient-based online learner. In contrast, we consider more choices of batch base learners and a fine-grained algorithm w.r.t environment change. Even though their bounds are not w.r.t AER and the results are not directly comparable, we make an intuitive comparison, which suggests the proposed algorithm in this paper has an improved rate for gradient-based base learners in slot-wise stationary environments over their related results. Moreover, we conducted a rigorous analysis of the bi-level trade-off, which is missing in the related works mentioned above. Finally, although not clearly defined in previous meta-learning literature, we note Meta Continual Learning (MCL) [60, 61] can be named in a similar way as Online-Within-Batch (OWB), where the tasks are processed in one batch, but the data within each task are processed online.
Other related works Model-Agnostic Meta-Learning (MAML) [4] that uses the higher-order gradients for meta-updates has gained tremendous success in practice. Therefore, a lot of work emerges to improve MAML. Finn et al. [69], Grant et al. [70], Yoon et al. [71] combine MAML with Bayesian methods. Rajeswaran et al. [72] improves MAML with implicit gradient calculation for the meta-updates. Since meta-learning considers learning and generalization for both seen and unseen tasks, it has not only been applied to CL for addressing catastrophic forgetting but has also been applied to Test-Time Domain Adaptation (DA) for quick generalization. For more related works in Test-Time DA or DA, please refer to [73-75].