SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qi Chen* Laval University
Changjian Shui McGill University
Ligong Han
Rutgers University
Mario Marchand  
Laval University
We focus on Continual Meta-Learning (CML), which targets accumulating and exploiting meta-knowledge on a sequence of non-i.i.d. tasks. The primary challenge is to strike a balance between stability and plasticity, where a model should be stable to avoid catastrophic forgetting in previous tasks and plastic to learn generalizable concepts from new tasks. To address this, we formulate the CML objective as controlling the average excess risk upper bound of the task sequence, which reflects the trade-off between forgetting and generalization. Based on the objective, we introduce a unified theoretical framework for CML in both static and shifting environments, providing guarantees for various task-specific learning algorithms. Moreover, we first present a rigorous analysis of a bi-level trade-off in shifting environments. To approach the optimal trade-off, we propose a novel algorithm that dynamically adjusts the meta-parameter and its learning rate w.r.t environment change. Empirical evaluations on synthetic and real datasets illustrate the effectiveness of the proposed theory and algorithm.
An essential goal in real-time intelligent systems is to balance stability (preserve past knowledge; minimize catastrophic forgetting [1]) and plasticity (rapidly learn from new experiences; generalize quickly [2]). For addressing this dilemma, a promising direction is to incorporate meta-learning [3, 4] with Continual Learning (CL) [5-7], which constitutes the Continual Meta-Learning (CML) [8-10] problem. Specifically, CML sequentially learns a meta-model from few-shot tasks as the common prior. Then, for new tasks, it quickly adapts task-specific models with this prior. CML effectively enables balancing the task-level trade-off between learning new tasks and retaining previously acquired knowledge since the performance on previous tasks can be recovered with few additional samples.
Despite being intuitive and technically sound, most related works in CML have focused on improving empirical performance. There is still much room for exploration in the theoretical aspects. In particular, there is a lack of rigorous understanding on (1) which factors are important for stability and plasticity, (2) how to address the dilemma effectively by considering these factors.
To theoretically understand these questions, we need to characterize the sequential task-generating process. In fact, many CML approaches are based on the implicit assumption that the non-i.i.d. tasks are originated from a static task environment  $\tau$ , where  $\mu_t \sim \tau, \forall t$ . In addition, recent works reveal that catastrophic forgetting often occurs when transferring a meta-model to a new environment [11, 12] and start trying to empirically address the forgetting in shifting environments, such as [7, 13-16]. We take this case into consideration and formally define a more general CML setting, where at each time  $t$  the task  $\mu_t$  is generated from a possibly shifting environment  $\tau_t$  with  $\mu_t \sim \tau_t$  (the environment is
*Correspondence to: qi.chen.1@ulaval.ca
static if  $\tau_t = \tau, \forall t$ . We found a meta-level trade-off exists in shifting environments, which further induces difficulty in controlling the task-level learning-forgetting trade-off, as discussed in Sec. 4.2.
In this paper, we formally study the bi-level (task- and meta-) learning-forgetting trade-off in CML to fill the theoretical gap. Compared with previous works, our contribution highlights are as follows:
Unified theoretical framework We first introduce a novel and unified theoretical framework for CML in both static and shifting task environments. For each task, the excess risk reflects the true model performance but is intractable. Therefore, we derive a unified form of the excess risk upper bound for various base learners, which contains optimization error with the generalization gap and could be estimated from observations. Based on these, we propose to control the Average Excess Risk (AER) upper bound of all the tasks as the learning objective in CML (shown in Sec. 3). This upper bound can be further decomposed as the optimal trade-off and the algorithmic regret.
Understanding the bi-level learning-forgetting trade-off We discuss the bi-level trade-off in shifting environments with an illustrative example in Sec. 4.1. We theoretically identify in Theorem 5.1 that: (1) Inside each environment, the task-level trade-off is affected by the task similarities, which reflects on the slot diameters and variances. (2) The path length and changing points representing environment similarities and non-stationarity dominate the meta-level trade-off. (3) Task and sample numbers are important to generalization. The optimal trade-off is achieved with a minimal upper bound of AER, so it's helpful to consider these factors for optimizing the meta-parameter sequence.
Theoretically grounded algorithm We propose a novel algorithm (Algo. 1) for addressing the bi-level trade-off in shifting environments, which dynamically adjusts both the meta-parameter and its learning rate when an environment change is detected. Hence, the proposed algorithm can implicitly control the AER by minimizing the dynamic regret to approach the optimal trade-off. We derive a general bound for the proposed algorithm in Theorem 5.1. Improved complexity factors and rates theoretically demonstrate the validity of the proposed algorithm for balancing the trade-off, as illustrated in Theorem 6.1 and 6.2. Furthermore, empirical results on various datasets show improvements in estimated bounds and superior performance compared to baselines, demonstrating that the proposed algorithm can incrementally learn from new environments while maintaining good performance on the previous ones, i.e., a well-balanced trade-off.
Due to the page limit, we only briefly discuss the most related works in this section. A more detailed discussion and comprehensive investigation of various settings are provided in Appendix H.
Continual learning Traditional CL approaches mainly focus on addressing catastrophic forgetting. Some methods use regularization techniques, such as Elastic Weight Consolidation (EWC) [17] and Synaptic Intelligence (SI) [18], that aim to protect important parameters of a neural network from being modified during training on new tasks. Other methods like replay-based approach [19, 20] store or replay previous task examples to aid in preserving knowledge of old tasks. Another line of work is parameter isolation or dynamic architecture methods, which involve creating new network pathways or modules for each new task [21, 22]. Riemer et al. [7] first conceptualize the learning forgetting trade-off in CL as temporally maximizing transfer and minimizing interference by enforcing gradient alignment across examples, which is implemented using experience replay and Reptile [23] to approximate the gradient alignment objective with a first-order Taylor expansion.
Meta-learning Meta-learning aims for quick generalization, allowing for efficient learning of new tasks with limited data. The foundational framework of statistical meta-learning by Baxter [24] assumes tasks are i.i.d. sampled from a task environment. Amit and Meir [25] proposed PAC Bayes bounds with a joint training algorithm, which is not straightforwardly extendable to sequential learning. Chen et al. [26] derived information-theoretic bounds for MAML-like algorithms suitable for sequential few-shot learning. These works assume i.i.d. conditions for both tasks and in-task data. To break the assumption, online meta-learning introduces two main approaches for learning non-i.i.d. sequential tasks. Batch-Within-Online (BWO)[9, 27] methods, similar to CML, learn each task under a statistical batch setting. However, the theoretical analysis is limited, and none addresses shifting task environments. Online-Within-Online (OWO) [28-31] methods employ online algorithms as both base and meta learners. Khodak et al. [31] were the first to consider shifting environments for OWO, providing average regret bounds for gradient-based learners.
In contrast, we consider various statistical batch algorithms in addition to the gradient-based ones as base learners. We further propose a fine-grained algorithm w.r.t environment change that has an improved rate over the bound in [31] (although not directly comparable) for gradient-based base learners, as discussed in Sec. 6.2. This improvement implies a better learning-forgetting trade-off not considered in previous meta-learning literature.
Continual meta-learning FTML [9] extends MAML [4] to sequential learning using Follow The Leader (FTL) as the meta-learner, which requires storing all the previous tasks. MOCA [32] incorporates Bayesian Online Changing-point Detection (BOCD) to identify unknown task boundaries during the continual meta-learning process. The above methods can address the task-level trade-off under a static task environment. However, a meta-level trade-off exists when facing shifting environments. He et al. [8] combines MAML with Bayes Gradient Descent (BGD), slowly updating meta-parameters of small variance to stabilize the learning from non-i.i.d. sequential tasks. Caccia et al. [13] address the environment shift with a soft modulation on meta-updates with the empirical loss. [14], [15], and [16] incorporate additional memory with different environment shift detection methods to address the meta-level trade-off. [14] and [15] model the meta-parameter distribution with a Dirichlet Process Mixture Model (DPMM) and Dynamic Gaussian Mixture Model (DGMM), respectively, which can detect new environments and store the corresponding  $K$  meta-parameters with  $\mathcal{O}(K)$  memory. [16] grows  $K$  subnets by detecting the environment shift with BOCD and stores  $M$  tasks from previous environments, thus having a memory complexity of  $\mathcal{O}(K + M)$ .
Instead of the memory-based approaches, we focus on the more challenging fully online experimental setting first proposed by Caccia et al. [13]. Our algorithm consumes  $\mathcal{O}(1)$  memory and better balances the meta-level trade-off in the online setting. In addition, we consider different levels of non-stationarity and more environment shifts compared to previous works.
Theory on the stability-plasticity dilemma Raghavan and Balaprakash [33] explicitly study the task-level trade-off in continual learning by formulating the problem as a two-player sequential game and prove the existence of a balance point for each task. However, the cost function is based on empirical loss without considering generalization on unseen data. Based on NTK [34] regime, the generalization and forgetting property of orthogonal gradient descent [35] are separately studied in [36] and [37] for continual learning. To the best of our knowledge, we are the first theoretical work studying the stability-plasticity dilemma in CML. We also first provide a theoretical framework for CML in static and shifting environments, offering a formal understanding of the bi-level trade-off.
Let us consider a sequence of different task distributions  $\{\mu_t\}_{t=1}^T$  defined on the same example space  $\mathcal{Z} = \mathcal{X} \times \mathcal{Y}$  with  $T \to \infty$ . Specifically, at time  $t$ , the task  $\mu_t$  is generated from a possibly shifting environment  $\tau_t$  with  $\mu_t \sim \tau_t$ , which is static if  $\tau_t = \tau, \forall t$ . Then a dataset of  $m_t$  examples  $S_t = \{Z_i\}_{i=1}^{m_t}$  are i.i.d. sampled from  $\mu_t$ , where  $S_t \sim \mu_t^{m_t}$ . In CML, we have two kinds of parameters: model parameter and meta-parameter, which are learned through a base learner and a meta learner, respectively. A visual representation of the CML process is provided in Fig. 1. Let the meta-parameters be defined on the same support  $\mathcal{U}$  for all the tasks. Additionally, we assume that the same parametric hypothesis space  $\mathcal{W}$  is used across all tasks with non-negative bounded loss function  $\ell : \mathcal{W} \times \mathcal{Z} \to [0,1]$ . Given any model parameter  $w \in \mathcal{W}$ , the true risk and empirical risk of  $t$ -th task are defined as  $\mathcal{L}_{\mu_t}(w) \stackrel{\mathrm{def}}{=} \mathbb{E}_{Z \sim \mu_t} \ell(w, Z)$  and  $\mathcal{L}_{S_t}(w) \stackrel{\mathrm{def}}{=} \frac{1}{m_t} \sum_{i=1}^{m_t} \ell(w, Z_i)$ . To motivate the CML setting mentioned above, we also provide an analysis of how to apply CML to real-world examples like online recommendation systems in Appendix G.2.
3.1 Base learner
At time  $t$ , base learner  $\mathcal{A}$  takes the task data  $S_{t}$  and the meta-parameter  $u_{t} \in \mathcal{U}$  that represents the prior knowledge learned from previous tasks as the input then outputs the model parameter  $W_{t} = \mathcal{A}(u_{t},S_{t})$ . Specifically,  $\mathcal{A}$  is characterized by a conditional distribution<sup>2</sup>  $P_{W_t|S_t,u_t}$  as in [38, 39]. We further define the corresponding (expected) excess risk on task  $t$  as the expected gap between the true risk of the learned hypothesis  $W_{t}$  and the optimal true risk:
2It will be a delta function if  $\mathcal{A}(u_t,S_t)$  is deterministic.
Figure 1: Illustration of Continual Meta-Learning (CML) process. At each time  $t$ , the CML algorithm  $\mathcal{A}_{\mathrm{CML}}$  (composed of the meta learner and the base learner) takes the current task data  $S_{t}$  and the meta-parameter (prior knowledge)  $u_{t}$  learned from previous tasks as input. Then, it outputs the learned hypothesis  $W_{t}$  of the current task and the updated meta-parameter  $u_{t+1}$  for the next task.
$$
R _ {\mathrm {e x c e s s}} (\mathcal {A}, u _ {t}) \stackrel {\mathrm {d e f}} {=} \mathbb {E} _ {S _ {t}} \mathbb {E} _ {W _ {t} \sim P _ {W _ {t} | S _ {t}, u _ {t}}} \left[ \mathcal {L} _ {\mu_ {t}} (W _ {t}) - \mathcal {L} _ {\mu_ {t}} (w _ {t} ^ {*}) \right], w _ {t} ^ {*} = \arg \min _ {w \in \mathcal {W}} \mathcal {L} _ {\mu_ {t}} (w).
$$
The excess risk of  $\mu_t$  is thus explicitly determined by the meta-parameter  $u_t$  and the base learner, where  $u_t$  is provided by experts or random guesses in single-task learning. Although reflecting the true model performance, the excess risk is intractable due to the unknown distribution  $\mu_t$ .
The following theorem gives a unified form of excess risk upper bound that applies to common base learners such as Stochastic Gradient Descent (SGD), Stochastic Gradient Langevin Dynamics (SGLD), Regularized Loss Minimization (RLM), and Gibbs algorithm. In contrast to the upper bounds that contain the empirical risk, it avoids the task data access for the meta-learner and can thus be used to design new CML algorithms. See detailed proof in Appendix B.5.
Theorem 3.1. For any  $t \in [T]$ , assume that  $\mathcal{L}_{\mu_t}(\cdot)$  has  $\alpha$ -quadratic growth (see Definition A.2), then whenever the base learner is SGD, SGLD, RLM, or the Gibbs algorithm, there exists  $f_t(\cdot)$  that gives
$$
R _ {\text {e x c e s s}} (\mathcal {A}, u _ {t}) \leq f _ {t} (u _ {t}) = \kappa_ {t} \left(a \beta_ {t} + \frac {b \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {t} + \epsilon_ {0}}{\beta_ {t}} + \Delta_ {t}\right), \kappa_ {t}, \epsilon_ {t}, \beta_ {t}, \Delta_ {t} \in \mathbb {R} ^ {+}, a, b, \epsilon_ {0} > 0. \tag {1}
$$
The meta-parameter  $u_{t} = (\beta_{t},\phi_{t})$  decomposes into an initialization or bias  $\phi_t$  and a learning rate  $\beta_{t}$  (if  $\mathcal{A}\in \{SGD,SGLD\}$ ) or a regularization coefficient (if  $\mathcal{A}\in \{RLM,Gibbs\}$ ). Moreover,  $w_{t}$  denotes the (single) output for the (possibly randomized) base learner, and  $\epsilon_0\stackrel {\mathrm{def}}{=}2\alpha ||w_t - \mathbb{E}W_t||^2$  characterizes the randomness of the output. Finally,  $a$  and  $b$  are constants, and  $\epsilon_t,\kappa_t,\Delta_t$  are functions of the task sample size  $m_t$  that characterize the base learner (see Appendix B and D).
Remark 3.2. (a) The above bound could be estimated from observation, so we can use it to design meta-learning algorithms to control the excess risk. (b) Besides learning the initialization/bias, the corresponding learning rate/regularization coefficient should be further considered to control  $R_{\mathrm{excess}}$ . (c)  $f_{t}$  is convex w.r.t  $u_{t}$  (proved in Appendix B.6). (d) This upper bound gives an explicit form of task-level trade-off that depends on the meta-parameter, where a large  $\beta_{t}$  represents conserving less model prior (larger forgetting) and learning more from data to obtain the model  $w_{t}$ . The similarity  $\| \phi_t - w_t\|$  between the model prior and the new task can affect the choice of  $\beta_{t}$  for a better generalization.
3.2 Meta learner
Since the tasks are sequentially encountered without the i.i.d. assumption and the excess risk upper bound (cost function)  $f_{t}$  is convex, we can naturally consider the meta-learning process as a repeated game, following the Online Convex Optimization (OCO) regime [40]. At each time  $t$ , the meta-learner first selects  $u_{t} \in \mathcal{U}$  with prior knowledge learned from previous tasks, then the base learner outputs  $w_{t}$  given  $u_{t}$  and dataset  $S_{t}$ . After that, the cost function value  $f_{t}$  is revealed. The goal of the meta learner is to select a sequence of meta-parameters  $u_{1:T}$  so as to minimize the regret over rounds.
Static task environment In static environments, the task distributions  $\mu_t \sim \tau, \forall t \in [T]$  are assumed to be sampled from a fixed environment  $\tau$ . The corresponding static regret is defined as the gap between the total cost of  $u_{1:T}$  and that of an optimal meta-parameter in hindsight  $u_T^*$ :
$$
R _ {T} ^ {\text {s t a t i c}} \left(u _ {1: T}\right) \stackrel {{\text {d e f}}} {{=}} \sum_ {t = 1} ^ {T} f _ {t} \left(u _ {t}\right) - \sum_ {t = 1} ^ {T} f _ {t} \left(u _ {T} ^ {*}\right), u _ {T} ^ {*} \stackrel {{\text {d e f}}} {{=}} \arg \min  _ {u \in \mathcal {U}} \sum_ {t = 1} ^ {T} f _ {t} (u).
$$
The hindsight  $u_{T}^{*}$  converges to the true minimum  $u^{*}$  w.r.t  $\tau$  as  $T \to \infty$ . So the task-level trade-off can be well-balanced by designing algorithms with sub-linear regret to approach  $u^{*}$ .
Shifting task environment In a more general CML setting where the task environment  $\tau_{t}$  can change at each time  $t$ , using a meta-learner designed to obtain sub-linear static regret will suffer a shifting comparator. As a result, the optimal  $u_{T}^{*}$  may not converge to a fixed point, and  $\sum_{t=1}^{T} f_{t}(u_{T}^{*})$  can be vacuous when  $T \to +\infty$ , which induces the meta-level forgetting. See a formal justification in Appendix A.2.2. To this end, we assume the horizon  $T$  of the sequential tasks can be divided into  $N$  slots, and the  $n$ -th slot contains  $M_{n}$  tasks. We further assume that the environment does not change with  $t$  within each slot. The  $N$  optimal priors, in hindsight, are noted as  $u_{1:N}^{*}$ . Consequently, we define a more versatile dynamic regret as:
$$
R _ {T} ^ {\mathrm {d y n a m i c}} (u _ {1: N} ^ {*}) \stackrel {\mathrm {d e f}} {=} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {M _ {n}} \left[ f _ {n, k} (u _ {n, k}) - f _ {n, k} (u _ {n} ^ {*}) \right], u _ {n} ^ {*} \stackrel {\mathrm {d e f}} {=} \arg \min _ {u \in \mathcal {U}} \frac {1}{M _ {n}} \sum_ {k = 1} ^ {M _ {n}} f _ {n, k} (u).
$$
Given  $\sum_{n=1}^{N} M_n = T$ , if  $N \to T$ , the definition is equivalent to the conventional dynamic regret. If  $N \to 1$ , it recovers the static regret. To balance the meta-level trade-off, we need to find algorithms with sub-linear dynamic regret to sequentially approach the optimums in  $N$  slots (environments).
3.3 Continual Meta-Learning objective
In practice, we aim to train the task-specific model with few-shot data and hope that the model will generalize on unseen data. Hence, we formulate the CML objective as selecting  $u_{t}$  at each time  $t$  to ensure a small Average Excess Risk (AER) upper bound for the task sequence  $\{\mu_t\}_{t = 1}^T$ , which is defined as follows given the base learner  $\mathcal{A}$ :
$$
\mathrm {A E R} _ {\mathcal {A}} ^ {T} \stackrel {\text {d e f}} {=} \frac {1}{T} \sum_ {t = 1} ^ {T} R _ {\text {e x c e s s}} (\mathcal {A}, u _ {t}) \leq \frac {1}{T} \sum_ {t = 1} ^ {T} f _ {t} (u _ {t}) = \frac {1}{T} R _ {T} ^ {\text {d y n a m i c}} \left(u _ {1: N} ^ {*}\right) + \frac {1}{T} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {M _ {n}} f _ {n, k} \left(u _ {n} ^ {*}\right).
$$
We adopt the dynamic regret defined in shifting environments since it can recover the static setting w.l.o.g. Considering the two terms in the above upper bound of AER, the CML objective is two-fold: (a) design an appropriate online meta-learner to minimize the dynamic regret. (b) choose the optimal split of the  $T$  tasks to  $N$  stationary slots. Based on this objective, we propose a novel continual meta-learning framework and provide a corresponding theoretical analysis in the following sections.
4 Balancing bi-level learning-forgetting trade-off
4.1 An illustrative example
The excess risk upper bound provides the intuition that we can quickly learn a new task similar to our prior knowledge by relying on this prior. In this case, the initialization/bias  $\phi_t$  (model prior) is close to the base learner output  $w_t$  (i.e., a small distance  $\| \phi_t - w_t \|^2$ ), and the excess risk bound will be small. So we will elaborate on the learning-forgetting trade-off on an example of a shifting environment visualized in Fig. 2, based on the following definitions of task similarities characterized by the distances in the model parameter space [30].
Definition 4.1. Let us denote  $\hat{\mathcal{W}}$  as the set of all the learned model parameters:  $\hat{\mathcal{W}} = \{w_t = \mathcal{A}(u_t, S_t)\}_{t=1}^T \cup \{\mathbf{0}\}$ . Let  $\hat{D}$  be the diameter of  $\hat{\mathcal{W}}$  w.r.t norm  $\| \cdot \|$ :  $\forall w, v \in \hat{\mathcal{W}}, \| w - v \| \leq \hat{D}$ . In shifting task environments, we denote  $\hat{D}_n$  the corresponding diameter for  $n$ -th slot. Assume  $\mathcal{W}$  is a convex set containing all the possible model parameters. Then let  $D$  be the diameter of  $\mathcal{W}$ :  $\forall w, v \in \mathcal{W}, \| w - v \| \leq D$ .
Figure 2: Illustration of parameter spaces in a shifting environment, where  $\mathcal{W},\hat{\mathcal{W}}$  and  $D,\hat{D},\hat{D}_n$  are defined in Def. 4.1.  $\phi_n^\star$  is the  $n$ -th slot hindsight.
The task environment in Fig. 2 changes within three slots (i.e., three circles with diameters  $\hat{D}_{1:3}$  in light red). The dark red point is denoted as  $\phi_n^{\star}$  - the optimal initialization in hindsight for each slot.
Purple points  $w_{1:3}$  are the learned model parameters in different slots. In static environments, CML can access additional few-shot data to address the task-level forgetting. Assume the models in  $n$ -th slot can recover performance (without forgetting) of similar tasks within a distance  $\hat{D}_n / 2$ . Then, the optimal  $\phi_n^*$  learned by CML can cover all the tasks in the slot with diameter  $\hat{D}_n$ . For instance, the two tasks in the first slot, shown in Fig. 2, have a large distance  $\| w_1 - w_2 \| > \hat{D}_1 / 2$ . Directly adapting from one to another suffers catastrophic forgetting. However, keeping the optimal prior  $\phi_1^*$  can address the forgetting, where  $\| \phi_1^* - w_1 \| < \hat{D}_1 / 2$  and  $\| \phi_1^* - w_2 \| < \hat{D}_1 / 2$ .