Title: On the Stability-Plasticity Dilemma in Continual Meta-Learning: Theory and Algorithm
Venue: NeurIPS 2023
Accepted: yes
Keywords: continual meta-learning; transfer learning; stability-plasticity dilemma;
Abstract: We focus on Continual Meta-Learning (CML), which targets accumulating and exploiting meta-knowledge on a sequence of non-i.i.d. tasks. The primary challenge is to strike a balance between stability and plasticity, where a model should be stable to avoid catastrophic forgetting in previous tasks and plastic to learn generalizable concepts from new tasks. To address this, we formulate the CML objective as controlling the average excess risk upper bound of the task sequence, which reflects the trade-off between forgetting and generalization. Based on the objective, we introduce a unified theoretical framework for CML in both static and shifting environments, providing guarantees for various task-specific learning algorithms. Moreover, we first present a rigorous analysis of a bi-level trade-off in shifting environments. To approach the optimal trade-off, we propose a novel algorithm that dynamically adjusts the meta-parameter and its learning rate w.r.t environment change. Empirical evaluations on synthetic and real datasets illustrate the effectiveness of the proposed theory and algorithm.

On the Stability-Plasticity Dilemma in Continual Meta-Learning: Theory and Algorithm

Abstract

We focus on Continual Meta-Learning (CML), which targets accumulating and exploiting meta-knowledge on a sequence of non-i.i.d. tasks. The primary challenge is to strike a balance between stability and plasticity, where a model should be stable to avoid catastrophic forgetting in previous tasks and plastic to learn generalizable concepts from new tasks. To address this, we formulate the CML objective as controlling the average excess risk upper bound of the task sequence, which reflects the trade-off between forgetting and generalization. Based on the objective, we introduce a unified theoretical framework for CML in both static and shifting environments, providing guarantees for various task-specific learning algorithms. Moreover, we first present a rigorous analysis of a bi-level trade-off in shifting environments. To approach the optimal trade-off, we propose a novel algorithm that dynamically adjusts the meta-parameter and its learning rate w.r.t environment change. Empirical evaluations on synthetic and real datasets illustrate the effectiveness of the proposed theory and algorithm.

1 Introduction

An essential goal in real-time intelligent systems is to balance stability (preserve past knowledge; minimize catastrophic forgetting [1]) and plasticity (rapidly learn from new experiences; generalize quickly [2]). For addressing this dilemma, a promising direction is to incorporate meta-learning [3, 4] with Continual Learning (CL) [5-7], which constitutes the Continual Meta-Learning (CML) [8-10] problem. Specifically, CML sequentially learns a meta-model from few-shot tasks as the common prior. Then, for new tasks, it quickly adapts task-specific models with this prior. CML effectively enables balancing the task-level trade-off between learning new tasks and retaining previously acquired knowledge since the performance on previous tasks can be recovered with few additional samples.

Despite being intuitive and technically sound, most related works in CML have focused on improving empirical performance. There is still much room for exploration in the theoretical aspects. In particular, there is a lack of rigorous understanding on (1) which factors are important for stability and plasticity, (2) how to address the dilemma effectively by considering these factors.

To theoretically understand these questions, we need to characterize the sequential task-generating process. In fact, many CML approaches are based on the implicit assumption that the non-i.i.d. tasks are originated from a static task environment τ , where µ t ∼ τ, ∀ t . In addition, recent works reveal that catastrophic forgetting often occurs when transferring a meta-model to a new environment [11, 12] and start trying to empirically address the forgetting in shifting environments, such as [7, 13-16]. We take this case into consideration and formally define a more general CML setting, where at each time t the task µ t is generated from a possibly shifting environment τ t with µ t ∼ τ t (the environment is

static if τ t = τ, ∀ t ). We found a meta-level trade-off exists in shifting environments, which further induces difficulty in controlling the task-level learning-forgetting trade-off, as discussed in Sec. 4.2.

In this paper, we formally study the bi-level (task- and meta-) learning-forgetting trade-off in CML to fill the theoretical gap. Compared with previous works, our contribution highlights are as follows:

Unified theoretical framework We first introduce a novel and unified theoretical framework for CML in both static and shifting task environments. For each task, the excess risk reflects the true model performance but is intractable. Therefore, we derive a unified form of the excess risk upper bound for various base learners, which contains optimization error with the generalization gap and could be estimated from observations. Based on these, we propose to control the Average Excess Risk (AER) upper bound of all the tasks as the learning objective in CML (shown in Sec. 3). This upper bound can be further decomposed as the optimal trade-off and the algorithmic regret.

Understanding the bi-level learning-forgetting trade-off We discuss the bi-level trade-off in shifting environments with an illustrative example in Sec. 4.1. We theoretically identify in Theorem 5.1 that: (1) Inside each environment, the task-level trade-off is affected by the task similarities, which reflects on the slot diameters and variances. (2) The path length and changing points representing environment similarities and non-stationarity dominate the meta-level trade-off. (3) Task and sample numbers are important to generalization. The optimal trade-off is achieved with a minimal upper bound of AER, so it's helpful to consider these factors for optimizing the meta-parameter sequence.

Theoretically grounded algorithm We propose a novel algorithm (Algo. 1) for addressing the bi-level trade-off in shifting environments, which dynamically adjusts both the meta-parameter and its learning rate when an environment change is detected. Hence, the proposed algorithm can implicitly control the AER by minimizing the dynamic regret to approach the optimal trade-off. We derive a general bound for the proposed algorithm in Theorem 5.1. Improved complexity factors and rates theoretically demonstrate the validity of the proposed algorithm for balancing the trade-off, as illustrated in Theorem 6.1 and 6.2. Furthermore, empirical results on various datasets show improvements in estimated bounds and superior performance compared to baselines, demonstrating that the proposed algorithm can incrementally learn from new environments while maintaining good performance on the previous ones, i.e., a well-balanced trade-off.

2 Related work

Due to the page limit, we only briefly discuss the most related works in this section. A more detailed discussion and comprehensive investigation of various settings are provided in Appendix H.

Continual learning Traditional CL approaches mainly focus on addressing catastrophic forgetting . Some methods use regularization techniques, such as Elastic Weight Consolidation (EWC) [17] and Synaptic Intelligence (SI) [18], that aim to protect important parameters of a neural network from being modified during training on new tasks. Other methods like replay-based approach [19, 20] store or replay previous task examples to aid in preserving knowledge of old tasks. Another line of work is parameter isolation or dynamic architecture methods, which involve creating new network pathways or modules for each new task [21, 22]. Riemer et al. [7] first conceptualize the learning forgetting trade-off in CL as temporally maximizing transfer and minimizing interference by enforcing gradient alignment across examples, which is implemented using experience replay and Reptile [23] to approximate the gradient alignment objective with a first-order Taylor expansion.

Meta-learning Meta-learning aims for quick generalization , allowing for efficient learning of new tasks with limited data. The foundational framework of statistical meta-learning by Baxter [24] assumes tasks are i.i.d. sampled from a task environment. Amit and Meir [25] proposed PAC Bayes bounds with a joint training algorithm, which is not straightforwardly extendable to sequential learning. Chen et al. [26] derived information-theoretic bounds for MAML-like algorithms suitable for sequential few-shot learning. These works assume i.i.d. conditions for both tasks and in-task data. To break the assumption, online meta-learning introduces two main approaches for learning non-i.i.d. sequential tasks. Batch-Within-Online (BWO)[9, 27] methods, similar to CML, learn each task under a statistical batch setting. However, the theoretical analysis is limited, and none addresses shifting task environments. Online-Within-Online (OWO) [28-31] methods employ online algorithms as both base and meta learners. Khodak et al. [31] were the first to consider shifting environments for OWO, providing average regret bounds for gradient-based learners.

In contrast, we consider various statistical batch algorithms in addition to the gradient-based ones as base learners. We further propose a fine-grained algorithm w.r.t environment change that has an improved rate over the bound in [31] (although not directly comparable) for gradient-based base learners, as discussed in Sec. 6.2. This improvement implies a better learning-forgetting trade-off not considered in previous meta-learning literature.

Continual meta-learning FTML [9] extends MAML [4] to sequential learning using Follow The Leader (FTL) as the meta-learner, which requires storing all the previous tasks. MOCA [32] incorporates Bayesian Online Changing-point Detection (BOCD) to identify unknown task boundaries during the continual meta-learning process. The above methods can address the task-level tradeoff under a static task environment. However, a meta-level trade-off exists when facing shifting environments. He et al. [8] combines MAML with Bayes Gradient Descent (BGD), slowly updating meta-parameters of small variance to stabilize the learning from non-i.i.d. sequential tasks. Caccia et al. [13] address the environment shift with a soft modulation on meta-updates with the empirical loss. [14], [15], and [16] incorporate additional memory with different environment shift detection methods to address the meta-level trade-off. [14] and [15] model the meta-parameter distribution with a Dirichlet Process Mixture Model (DPMM) and Dynamic Gaussian Mixture Model (DGMM), respectively, which can detect new environments and store the corresponding K meta-parameters with O ( K ) memory. [16] grows K subnets by detecting the environment shift with BOCD and stores M tasks from previous environments, thus having a memory complexity of O ( K + M ) .

Instead of the memory-based approaches, we focus on the more challenging fully online experimental setting first proposed by Caccia et al. [13]. Our algorithm consumes O (1) memory and better balances the meta-level trade-off in the online setting. In addition, we consider different levels of non-stationarity and more environment shifts compared to previous works.

Theory on the stability-plasticity dilemma Raghavan and Balaprakash [33] explicitly study the task-level trade-off in continual learning by formulating the problem as a two-player sequential game and prove the existence of a balance point for each task. However, the cost function is based on empirical loss without considering generalization on unseen data. Based on NTK [34] regime, the generalization and forgetting property of orthogonal gradient descent [35] are separately studied in [36] and [37] for continual learning. To the best of our knowledge, we are the first theoretical work studying the stability-plasticity dilemma in CML. We also first provide a theoretical framework for CML in static and shifting environments, offering a formal understanding of the bi-level trade-off.

3 Problem setup

Let us consider a sequence of different task distributions { µ t } T t =1 defined on the same example space Z = X × Y with T → ∞ . Specifically, at time t , the task µ t is generated from a possibly shifting environment τ t with µ t ∼ τ t , which is static if τ t = τ, ∀ t . Then a dataset of m t examples S t = { Z i } m t i =1 are i.i.d. sampled from µ t , where S t ∼ µ m t t . In CML, we have two kinds of parameters: model parameter and meta-parameter, which are learned through a base learner and a meta learner, respectively. A visual representation of the CML process is provided in Fig. 1. Let the meta-parameters be defined on the same support U for all the tasks. Additionally, we assume that the same parametric hypothesis space W is used across all tasks with non-negative bounded loss function glyph[lscript] : W×Z → [0 , 1] . Given any model parameter w ∈ W , the true risk and empirical risk of t -th task are defined as L µ t ( w ) def = E Z ∼ µ t glyph[lscript] ( w,Z ) and L S t ( w ) def = 1 m t ∑ m t i =1 glyph[lscript] ( w,Z i ) . To motivate the CML setting mentioned above, we also provide an analysis of how to apply CML to real-world examples like online recommendation systems in Appendix G.2.

3.1 Base learner

At time t , base learner A takes the task data S t and the meta-parameter u t ∈ U that represents the prior knowledge learned from previous tasks as the input then outputs the model parameter W t = A ( u t , S t ) . Specifically, A is characterized by a conditional distribution 2 P W t | S t ,u t as in [38, 39]. We further define the corresponding (expected) excess risk on task t as the expected gap between the true risk of the learned hypothesis W t and the optimal true risk:

EnvT1

EnvTt

EnvTT

Currenttask

data St

ACML

Taskμ1

Taskμt

TaskμT

ut

ut+1

Abase

Prior knowledge

priorknowledge

Data S1

Data St

Data ST

for current task

fornext task

Ameta

u1

ACML

ACML

ACML

uT+1

Learned hypothesis

of current task

W1

Wt

WT

Wt

The excess risk of µ t is thus explicitly determined by the meta-parameter u t and the base learner, where u t is provided by experts or random guesses in single-task learning. Although reflecting the true model performance, the excess risk is intractable due to the unknown distribution µ t .

The following theorem gives a unified form of excess risk upper bound that applies to common base learners such as Stochastic Gradient Descent (SGD), Stochastic Gradient Langevin Dynamics (SGLD), Regularized Loss Minimization (RLM), and Gibbs algorithm. In contrast to the upper bounds that contain the empirical risk, it avoids the task data access for the meta-learner and can thus be used to design new CML algorithms. See detailed proof in Appendix B.5.

Theorem 3.1. For any t ∈ [ T ] , assume that L µ t ( · ) has α -quadratic growth (see Definition A.2), then whenever the base learner is SGD, SGLD, RLM, or the Gibbs algorithm, there exists f t ( · ) that gives

The meta-parameter u t = ( β t , φ t ) decomposes into an initialization or bias φ t and a learning rate β t (if A ∈ { SGD , SGLD } ) or a regularization coefficient (if A ∈ { RLM , Gibbs } ). Moreover, w t denotes the (single) output for the (possibly randomized) base learner, and glyph[epsilon1] 0 def = 2 α || w t -E W t || 2 characterizes the randomness of the output. Finally, a and b are constants, and glyph[epsilon1] t , κ t , ∆ t are functions of the task sample size m t that characterize the base learner (see Appendix B and D).

Remark 3.2 . (a) The above bound could be estimated from observation , so we can use it to design meta-learning algorithms to control the excess risk. (b) Besides learning the initialization/bias, the corresponding learning rate/regularization coefficient should be further considered to control R excess . (c) f t is convex w.r.t u t (proved in Appendix B.6). (d) This upper bound gives an explicit form of tasklevel trade-off that depends on the meta-parameter, where a large β t represents conserving less model prior (larger forgetting) and learning more from data to obtain the model w t . The similarity ‖ φ t -w t ‖ between the model prior and the new task can affect the choice of β t for a better generalization.

3.2 Meta learner

Since the tasks are sequentially encountered without the i.i.d. assumption and the excess risk upper bound (cost function) f t is convex, we can naturally consider the meta-learning process as a repeated game, following the Online Convex Optimization (OCO) regime [40]. At each time t , the meta-learner first selects u t ∈ U with prior knowledge learned from previous tasks, then the base learner outputs w t given u t and dataset S t . After that, the cost function value f t is revealed. The goal of the meta learner is to select a sequence of meta-parameters u 1: T so as to minimize the regret over rounds.

Static task environment In static environments, the task distributions µ t ∼ τ, ∀ t ∈ [ T ] are assumed to be sampled from a fixed environment τ . The corresponding static regret is defined as the gap between the total cost of u 1: T and that of an optimal meta-parameter in hindsight u ∗ T :

The hindsight u ∗ T converges to the true minimum u ∗ w.r.t τ as T →∞ . So the task-level trade-off can be well-balanced by designing algorithms with sub-linear regret to approach u ∗ .

Shifting task environment In a more general CML setting where the task environment τ t can change at each time t , using a meta-learner designed to obtain sub-linear static regret will suffer a shifting comparator. As a result, the optimal u ∗ T may not converge to a fixed point, and ∑ T t =1 f t ( u ∗ T ) can be vacuous when T → + ∞ , which induces the meta-level forgetting. See a formal justification in Appendix A.2.2. To this end, we assume the horizon T of the sequential tasks can be divided into N slots, and the n -th slot contains M n tasks. We further assume that the environment does not change with t within each slot. The N optimal priors, in hindsight, are noted as u ∗ 1: N . Consequently, we define a more versatile dynamic regret as:

Given ∑ N n =1 M n = T , if N → T , the definition is equivalent to the conventional dynamic regret. If N → 1 , it recovers the static regret. To balance the meta-level trade-off, we need to find algorithms with sub-linear dynamic regret to sequentially approach the optimums in N slots (environments).

3.3 Continual Meta-Learning objective

In practice, we aim to train the task-specific model with few-shot data and hope that the model will generalize on unseen data. Hence, we formulate the CML objective as selecting u t at each time t to ensure a small Average Excess Risk (AER) upper bound for the task sequence { µ t } T t =1 , which is defined as follows given the base learner A :

We adopt the dynamic regret defined in shifting environments since it can recover the static setting w.l.o.g. Considering the two terms in the above upper bound of AER, the CML objective is two-fold: (a) design an appropriate online meta-learner to minimize the dynamic regret. (b) choose the optimal split of the T tasks to N stationary slots. Based on this objective, we propose a novel continual meta-learning framework and provide a corresponding theoretical analysis in the following sections.

4 Balancing bi-level learning-forgetting trade-off

4.1 An illustrative example

The excess risk upper bound provides the intuition that we can quickly learn a new task similar to our prior knowledge by relying on this prior. In this case, the initialization/bias φ t (model prior) is close to the base learner output w t ( i.e. , a small distance ‖ φ t -w t ‖ 2 ), and the excess risk bound will be small. So we will elaborate on the learning-forgetting trade-off on an example of a shifting environment visualized in Fig. 2, based on the following definitions of task similarities characterized by the distances in the model parameter space [30].

Definition 4.1. Let us denote ˆ W as the set of all the learned model parameters: ˆ W = { w t = A ( u t , S t ) } T t =1 ∪{ 0 } . Let ˆ D be the diameter of ˆ W w.r.t norm ‖ · ‖ : ∀ w,v ∈ ˆ W , ‖ w -v ‖ ≤ ˆ D . In shifting task environments, we denote ˆ D n the corresponding diameter for n -th slot. Assume W is a convex set containing all the possible model parameters. Then let D be the diameter of W : ∀ w,v ∈ W , ‖ w -v ‖ ≤ D .

The task environment in Fig. 2 changes within three slots (i.e., three circles with diameters ˆ D 1:3 in light red). The dark red point is denoted as φ glyph[star] n - the optimal initialization in hindsight for each slot.

Purple points w 1:3 are the learned model parameters in different slots. In static environments, CML can access additional few-shot data to address the task-level forgetting . Assume the models in n -th slot can recover performance (without forgetting) of similar tasks within a distance ˆ D n / 2 . Then, the optimal φ ∗ n learned by CML can cover all the tasks in the slot with diameter ˆ D n . For instance, the two tasks in the first slot, shown in Fig. 2, have a large distance ‖ w 1 -w 2 ‖ > ˆ D 1 / 2 . Directly adapting from one to another suffers catastrophic forgetting. However, keeping the optimal prior φ ∗ 1 can address the forgetting, where ‖ φ ∗ 1 -w 1 ‖ < ˆ D 1 / 2 and ‖ φ ∗ 1 -w 2 ‖ < ˆ D 1 / 2 .

However, this does not work in shifting environments. If we view three slots as one static environment (with large variance), the optimal overall model prior φ ∗ will be close to the origin, and the diameter is ˆ D , which is much larger than any one of the slot ˆ D n . Alarger diameter ( e.g. , ‖ φ ∗ -w 3 ‖ glyph[greatermuch] ‖ φ ∗ 3 -w 3 ‖ ) implies the need for more samples for each task to avoid forgetting, which is often not satisfied. A theoretical interpretation is provided in the discussion following Theorem 6.1. To address this, in the next session, we propose an algorithm that dynamically adapts the meta-parameter when environment change occurs so as to rapidly reconstruct meta-knowledge in each static slot. Moreover, it is not as catastrophic as directly adapting model parameters since the forgetting is w.r.t meta-knowledge, which is constrained in a core set of φ ∗ 1: T , which has a diameter much smaller than ˆ D .

4.2 Dynamic algorithm

As discussed in the last section, it would be problematic in a shifting environment if the metalearner adopts the same updating strategy as in the static setting. Specifically, if we aim to optimize the static regret, the learning rate of metaparameter decays with O (1 /t ) for Follow The Leader (FTL) [41] and O (1 / √ t ) for Online Gradient Descent (OGD). Clearly, such strategies hardly learn from new tasks in a shifting environment. On the other side, a constant learning rate of the meta-parameter is often adopted for optimizing the regular dynamic regret [40]. This strategy could perform poorly when the environment only occasionally changes with a huge step, whereas the theoretical optimal learning rate becomes large. A large learning rate causes the meta learner to easily forget the previous tasks, making convergence difficult inside each slot. We offer a theoretical comparison with this in the discussion of Theorem 6.2.

To balance the learning and forgetting tradeoff of meta-knowledge, we proposed the Dynamic Continual Meta-Learning (DCML) algorithm (see Algo. 1) that dynamically adjusts the learning rate of the meta-parameter. When a changing point is detected, the meta-learning

glyph[negationslash]

rate γ t adapts to a large hopping learning rate ρ , then relaunches the decay, in O (1 / √ k ) , from γ 0 in each slot. Besides, the meta-parameter itself is also adaptively updated. According to φ t +1 = (1 -2 bκ t γ t β t ) φ t + 2 bκ t γ t β t w t in Algo. 1, adjusting β t and γ t represent the explicit control on task-level and meta-level trade-off, respectively. A corresponding discussion is provided in Appendix G.1. For a better understanding of DCML, we present a detailed introduction to it using SGD as the base learner in Appendix. G.3

Finally, the changing point can be detected with a broad class of Out-Of-Distribution (OOD) detection methods, e.g. , Bayes Online Changing-point Detection (BOCD) [42], loss threshold [13], or setting a fixed-length sliding window. Specifically, when the window size equals 1 , a constant learning rate is set to the meta learner, which recovers optimizing the typical dynamic regret as in [40].

5 Main theorem

We present a general theorem for the proposed DCML framework, which could be deployed for various base learners with the same excess risk upper bound form as Eq. (1).

Theorem 5.1. Consider both static and shifting environments. If the excess risk's upper bound of the base learner A ( u t , S t ) can be formulated as a unified form in Eq.(1), then, the AER of DCML (in Algo. 1) is upper bounded by:

Here, κ n = ∑ M n k =1 κ n,k , ∆ n = ∑ M n k =1 κ n,k ∆ n,k , where the subscript n, k indicates k -th task in n -th slot. The optimal meta-parameter in hindsight of n -th slot is u ∗ n = ( β ∗ n , φ ∗ n ) and the path length of N slots is P ∗ = ∑ N -1 n =1 ‖ u ∗ n -u ∗ n +1 ‖ +1 . φ ∗ n = ∑ M n k =1 κ n,k κ n w n,k is the weighted average of the base learner outputs within the slot. β ∗ n = √ ( bV 2 n + glyph[epsilon1] n + glyph[epsilon1] 0 ) /a , where V 2 n = ∑ M n k =1 κ n,k κ n ‖ φ ∗ n -w n,k ‖ 2 2 is the variance in n -th slot and glyph[epsilon1] n = ∑ M n k =1 κ n,k κ n glyph[epsilon1] n,k . Moreover, G n is the maximal gradient norm of the cost function, ˜ D n is the diameter of meta-parameters in n -th slot, and ˜ D max = max { ˜ D n } N n =1 .

The above theorem can be applied to static and shifting environments, where the proof is provided in Appendix D.2. For comparison, we also derive a general theorem from static regret in Appendix D.1.

Important factors for bi-level trade-off (a) The first term in Theorem 5.1 indicates the ✿✿✿✿✿✿ optimal ✿✿✿✿✿✿✿ trade-off ✿✿✿ can ✿✿✿ be ✿✿✿✿✿✿✿ achieved with the N optimal meta-parameters, which is affected by the task similarities within each environment (slot variance V 2 n ) and the environment changing points. (b) The second term is the average static regret over slots, which is related to the task similarities via the slot diameter ˜ D n and represents ✿✿✿ how ✿✿✿✿ well ✿✿✿ the ✿✿✿✿✿✿✿✿ algorithm ✿✿✿ can ✿✿✿✿✿✿✿ balance ✿✿✿ the ✿✿✿✿✿✿✿✿ task-level ✿✿✿✿✿✿✿✿ trade-off. (c) The last term is the regret w.r.t the environmental shift, reflecting ✿✿✿✿ how ✿✿✿✿ well ✿✿✿ the ✿✿✿✿✿✿✿✿ algorithm ✿✿✿ can ✿✿✿✿✿✿✿ address ✿✿ the ✿✿✿✿✿✿✿✿ meta-level ✿✿✿✿✿✿✿✿ trade-off. It is affected by the path length P ∗ , determined by the environment similarities and the non-stationarity (how frequently the environment changes).

Remark 5.2 . The above theorem has considered different sample numbers for each task. We can see the best initialization (bias) in hindsight ( φ ∗ n ) is related to the weighted average of the task models w n,k in n -th slot through κ n,k . E.g., for SGD, κ n,k = √ 1 /Kα +2 /m n,k αL implies that the task trained on more samples will have a smaller weight. Since it's easier to recover model performance for the tasks with more samples, less information on these tasks is conserved in the meta-knowledge.

6 Results on specific base learners

Theorem 5.1 provides general theoretical guarantees without specifying the base leaner. In this section, we discuss the theorem in depth with two typical base learners: SGD within the meta-initialization setting and Gibbs algorithm for the meta-regularization.

6.1 Gibbs algorithm

Theorem 6.1. Let W ⊂ R d . Assume that the loss glyph[lscript] ( · , z ) ∈ [0 , 1] is L -Lipschitz ∀ z ∈ Z and that L µ t ( · ) has α -quadratic-growth for all t ∈ [ T ] . Let w t = A Gibbs ( β t , φ t , S t ) be the output of the Gibbs algorithm (Definition B.2) on S t , where φ t is the mean of a prior Gaussian N ( φ t , σ 2 t 1 d ) with σ t = m -1 / 4 t d -1 / 4 L -1 / 2 and β t is the inverse temperature. Consider DCML (Algo. 1) and further assume that each slot has equal length M and each task uses the sample number m . Then we have

The detailed proof is provided in Appendix E.1 and E.3. We also proved in Appendix E.2 that the static AER bound is in O (( ( V +1) + 1 / √ T ) m -1 / 4 ) . As we claimed before, when N = 1 (which means the environment is static), the dynamic AER bound recovers the static AER bound.

Benefits of DCML Arbitrarily setting the prior mean φ t , the excess risk for single-task learning with Gibbs algorithm has an upper bound in O ( ( D +1) m -1 / 4 ) , where D is the diameter of the parameter space. While the static AER becomes O (( V +1) m -1 / 4 ) with rate O (1 / √ T ) . Since V 2 is the variance of all the outputs of the base learner, we have V ≤ ˆ D ≤ D , which illustrates the benefits of the proposed DCML algorithm. Moreover, in shifting environments , the bound is further improved in Theorem 6.1, where the complexity factor is decreased by 1 N ∑ N n =1 V n ≤ V with rate O (1 / √ M ) . This means that by considering environment change, DCML can achieve the same AER with a smaller M (fewer tasks), i.e. , faster-constructing meta-knowledge in new environments.

6.2 Stochastic Gradient Descent (SGD)

Theorem 6.2. Let W ⊂ R d . Consider that glyph[lscript] ( · , z ) is convex, β -smooth, L -Lipschitz ∀ z ∈ Z . Assume that L µ t ( · ) has α -quadratic-growth for all t ∈ [ T ] . Let SGD be the base learner for each task where it outputs w t = A SGD ( η t , φ t , S t ) with learning rate η t and initialization φ t . Consider DCML (Algo. 1) for the SGD cost function. Further, assume each slot has equal length M , and each task uses the sample number m and the same number of updating steps K . Then we have

The detailed proof is provided in Appendix E.4 and E.6. The CML setting uses offline batch-training for each task, so the related rate of the base learner is O ( √ 1 /K +1 /m ) , which is determined by both the step number and the sample size. Let ¯ V = 1 N ∑ N n =1 V n . CML focuses on fast learning new tasks and quickly recovers the model performance on past tasks with few-shot examples ( m is small). Hence, the optimal trade-off is dominated by the average deviation over slots ¯ V and the path length P ∗ , where P ∗ reflects the environment similarities and the non-stationarity.

Static environment If the environment is static, i.e. , N = 1 , we have P ∗ = 1 , M = T . The bound becomes in O ( ( V + 1 √ T ) √ 1 K + 1 m ) , which recovers the static AER bound in Appendix E.5.

Shifting environment When the environment occasionally changes with a large step, as discussed in Sec 4.2, N is small, and M,P ∗ is large. The proposed bound has an improved rate O (1 / √ M ) on P ∗ compared to O ( ¯ V + 1 √ M + √ P ∗ NM ) - an equivalent form of the task average regret bound in [31]. A more detailed comparison w.r.t [31] has been discussed in the proof for both static and dynamic AER. Similar to Theorem 6.1, the dynamic AER is improved w.r.t. the static AER with the improvement on complexity factor by ¯ V ≤ V . In addition, if the environments differ a lot and frequently change , N,P ∗ is large, and M is small. It's impossible to obtain a small AER, as demonstrated in the experiment on the Synbols dataset in Tab. 1.

7 Experiments

We first conduct analytic experiments on a synthetic dataset to offer an in-depth understanding of the theoretical counterpart. Then, we test the proposed algorithm on a recent CML benchmark - OSAKA [13], which empirically considers shifting environments on a large scale. The experimental results validate the proposed theory and illustrate the superiority of DCML in shifting environments.

7.1 Moving 2D Gaussian

We introduce a simple synthetic dataset, where the data is generated by the following strategy. Let the environment be a 2D Gaussian distribution with a moving mean. Initially, τ 0 = N ( ˜ φ ∗ 0 , I 2 ) with ˜ φ ∗ 0 = ( -6 , 6) T . Then at time point t , τ t = N ( ˜ φ ∗ t , I 2 ) is updated by: ˜ φ ∗ t = ˜ φ ∗ t -1 + (1 , 1) with probability p and ˜ φ ∗ t = ˜ φ ∗ t -1 with probability 1 -p . Then we have N changing points { t n } N n =1 . At each time step, we first sample w ∗ t ∼ τ t from the current environment τ t as the mean of the task distribution µ t = N ( w ∗ t , 0 . 1 I 2 ) , which is also a 2D Gaussian. For task t , we sample m examples S tr t ∼ µ t m as train data, m ′ samples S te t ∼ µ m ′ t as test data. Then the corresponding empirical risk is

⌈}˜{∐√˜(√˜√√(⌈}√√}

























√∐√⌋(⌉˜∐{(˜√√]⌉∐√]}{√

[]{̂√]˜[√(˜{√(⌉˜∐{

∐̂√√∐⌈(˜{√(⌉˜∐{

(a)

/

/

/









⌈}˜{〉∫}

̂glyph[arrowbt]{∐⌉]̂(〉∫

√√∐√]̂(〉∫

/

/

/

/

⊕˜√∐(⌈√















√

/

/

/

∐√˜(⌈√

(b)

0.2

0.4

0.6

0.8

1.0

Probability of environment change

0.80

0.81

0.82

0.83

0.84

0.85

0.86

0.87

0.88

0.89

Average test accuracy (

t

=10000

)

ANIL

MAML

CMAML

L S tr t ( w ) = 1 m ∑ m i =1 ‖ w -x i ‖ 2 . SGD uses φ t as initialization and a learning rate η t to optimize this objective, and outputs w t = A SGD ( η t , φ t , S tr t ) . Since w t is obtained from limited data, it can be far away from w ∗ t . If the environment shifts to the n -th slot at t n , then the predicted environment mean ˆ φ ∗ n = φ t n . We denote ˜ φ ∗ n = ˜ φ ∗ t n the mean of the actual environment, and the hindsight environment mean of n -th slot as φ ∗ n = 1 t n +1 -t n ∑ t n +1 -1 t = t n w t .

In Fig. 3 (a), we visualize an example of tracking the environment shifts, where the task mean estimation is w t , the output of SGD. The other three terms are as described above. We further visualize the test loss, the dynamic, and the static AER bounds in Fig. 3 (b). The red dotted lines represent the actual environment-changing points. We observe that the static AER is much larger than the dynamic AER, consistent with the result in Theorem 6.2. In addition, the evolving trend of dynamic AER is the same as the average test loss, which does not hold for static AER. We also find that the learning rate of the initialization φ t (meta lr in Fig.3 (b)) is scheduled by the changing points, and the base learning rate is adjusted automatically by the algorithm. Additional experimental settings and results are provided in Appendix F.

7.2 OSAKA benchmark

Experimental set-up OSAKA considers a task-agnostic setting of unknown task boundaries and simultaneously switches the task and environment with low probabilities. We slightly modify the setting to introduce more non-stationarity by always switching to a new task and changing the environment with probability p at each timestep. We pre-train a meta-model as initialization from one environment. The new environment is selected with probability 0 . 5 as the pre-trained one and 0 . 25 for each of the two unseen environments, consisting of OOD tasks. We further test DCML on two representative datasets in OSAKA. For Synbols [43] dataset, the model is pre-trained to classify characters from an alphabet on randomized backgrounds. During CML time, the model is exposed to the pre-trained environment, a new alphabet for character classification, and a heterogeneous environment that conducts font classification, which induces significant concept shift . For OmniglotMNIST-FashionMNIST (OMF), we pre-train the meta-model on the first 1000 classes of Omniglot [44]. Then, it is exposed to the full Omniglot dataset and two unseen datasets - MNIST [45] and FashionMNIST [46] at CML time. More details and baselines are deferred to Appendix F.1.1.

We tested 5 -level of non-stationarity with p = { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } for OMF and for Synbols we set p = { 0 . 2 , 0 . 4 , 1 . 0 } . The average test accuracy of all encountered tasks is used as the performance metric, which reflects the AER. In addition, we denote DCML (oracle) for the algorithm as offering correct environment-changing points and DCML (window) as using a fixed-length sliding window.

Results The average test accuracies at the last timestep for all environments and every single environment are reported in Tab.1 for OMF with p = 0 . 4 and Synbols with p = 0 . 2 . Additional experimental results can be found in Appendix F.2.2. Clearly, DCML constantly outperforms all the baselines with improvements of 1 ∼ 2% points on OMF data and 0 . 7% points on Synbols data compared to the best baseline method. A comparison w.r.t powerful baselines is presented in Fig. 5 for OMF data, where the average test accuracy over rounds in three environments is visualized separately.

0

2000

4000

6000

8000

10000

t

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

Average Test Accuracy

ANIL

MAML

CMAML

(a)

0

2000

4000

6000

8000

10000

t

0.56

0.58

0.60

0.62

0.64

0.66

Average Test Accuracy

ANIL

MAML

CMAML

(b)

0

2000

4000

6000

8000

10000

t

0.65

0.70

0.75

0.80

0.85

Average Test Accuracy

ANIL

MAML

CMAML

(c)

Analysis of learning and forgetting In Fig. 5 for OMF data, MAML and ANIL do not suffer forgetting since they never update meta-parameters. The proposed algorithm forgets a bit more than CMAML but learns much faster in new environments, illustrating a better trade-off. The fine-grained adjustment for the meta-learning rate makes it possible to quickly reconstruct meta-knowledge w.r.t environment change and constantly learn in new environments. On the other hand, MetaBGD and MetaCOG perform poorly since using BGD hinders acquiring new knowledge. All the methods perform worse on Synbols data in Tab. 1, where each environment is more diverse than OMF, and the environments differ a lot, indicating a large ˆ D n and P ∗ . Moreover, we observe a performance drop of CMAML on Synbols. Since CMAML uses empirical loss for modulation, it is prone to overfitting and can cause difficulty of convergence inside each slot when ˆ D n is large, as discussed in Sec. 4.1.

Analysis of stability w.r.t shifting probability We conduct an ablation study on OMF w.r.t environment change probability in Fig. 4. The results demonstrate the stability of the proposed algorithm and the increase in performance w.r.t p . Since the environments have high similarities in OMF, the meta-model is better learned by seeing more tasks given the fixed time steps with a larger p .

Analysis of environment change detection Empirical results suggest that DCML (window) performs better than DCML (oracle). This is reasonable when slots of small length exist where adapting to new environments causes additional context-switch costs (justified in Appendix F.2.1). Given the changing probability p , the expected slot length is 1 /p , used as the sliding window size. In practice, even though p is unknown, hyper-parameter searching for the one-dimensional window size is much easier. Because the similarity between any two environments can differ a lot, it's hard to find a single optimal hyper-parameter for change detection methods like BOCD or loss threshold.

8 Conclusion

This paper theoretically studies the stability and plasticity dilemma in CML. Based on the novel AER objective, it proposes a continual meta-learning framework in both static and shifting environments. The proposed DCML algorithm can quickly reconstruct meta-knowledge to alleviate forgetting and quickly adapt to new environments when change occurs. The corresponding theory provides tighter bounds and more flexible base learner selections. In addition, adaptively learning the meta-parameter can facilitate the training process in deep learning. Empirical evaluations on both synthetic and real datasets demonstrate the superiority of the proposed method.

Acknowledgments and Disclosure of Funding

We appreciate constructive feedback from anonymous reviewers and meta-reviewers. This work is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, the Collaborative Research and Development Grant from SSQ Assurances and NSERC, and the China Scholarship Council.

References

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

Gail A Carpenter and Stephen Grossberg. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer vision, graphics, and image processing , 37(1): 54-115, 1987.

Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In International Conference on Learning Representations , 2018.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909.08383 , 2(6):2, 2019.

Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123 , 2018.

Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 , 2018.

Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906.05201 , 2019.

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning , pages 1920-1930. PMLR, 2019.

Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos Storkey. Defining benchmarks for continual few-shot learning. arXiv preprint arXiv:2004.11967 , 2020.

Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel. Incremental few-shot learning with attention attractor networks. Advances in neural information processing systems , 32, 2019.

Sung Whan Yoon, Do-Yeon Kim, Jun Seo, and Jaekyun Moon. Xtarnet: Learning to extract task-adaptive representation for incremental few-shot learning. In International Conference on Machine Learning , pages 10852-10860. PMLR, 2020.

Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, et al. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances in Neural Information Processing Systems , 33:16532-16545, 2020.

Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. Advances in Neural Information Processing Systems , 32, 2019.

Qiang Zhang, Jinyuan Fang, Zaiqiao Meng, Shangsong Liang, and Emine Yilmaz. Variational continual bayesian meta-learning. Advances in Neural Information Processing Systems , 34: 24556-24568, 2021.

Zhenyi Wang, Li Shen, Tiehang Duan, Donglin Zhan, Le Fang, and Mingchen Gao. Learning to learn and remember super long multi-domain task sequence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7982-7992, 2022.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning , pages 3987-3995. PMLR, 2017.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems , 30, 2017.

Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems , 30, 2017.

Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 , 2017.

Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning , pages 4528-4537. PMLR, 2018.

Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 , 2(3):4, 2018.

Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research , 12:149-198, 2000.

Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended pac-bayes theory. In International Conference on Machine Learning , pages 205-214, 2018.

Qi Chen, Changjian Shui, and Mario Marchand. Generalization bounds for meta-learning: An information-theoretic analysis. Advances in Neural Information Processing Systems , 34, 2021.

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 , 2018.

Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-learn stochastic gradient descent with biased regularization. In International Conference on Machine Learning , pages 1566-1575. PMLR, 2019.

Pierre Alquier, Massimiliano Pontil, et al. Regret bounds for lifelong learning. In Artificial Intelligence and Statistics , pages 261-269. PMLR, 2017.

Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradientbased meta-learning. In International Conference on Machine Learning , pages 424-433. PMLR, 2019.

Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient-based metalearning methods. arXiv preprint arXiv:1906.02717 , 2019.

James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. Advances in neural information processing systems , 33:17571-17581, 2020.

Krishnan Raghavan and Prasanna Balaprakash. Formalizing the generalization-forgetting tradeoff in continual learning. Advances in Neural Information Processing Systems , 34:17284-17297, 2021.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics , pages 3762-3773. PMLR, 2020.

Mehdi Abbana Bennani, Thang Doan, and Masashi Sugiyama. Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006.11942 , 2020.

Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics , pages 1072-1080. PMLR, 2021.

Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization error bounds for noisy, iterative algorithms. In 2018 IEEE International Symposium on Information Theory (ISIT) , pages 546-550. IEEE, 2018.

Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory , pages 1674-1703. PMLR, 2017.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2):107-194, 2012.

Ryan Prescott Adams and David JC MacKay. Bayesian online changepoint detection. arXiv preprint arXiv:0710.3742 , 2007.

Alexandre Lacoste, Pau Rodríguez López, Frédéric Branchaud-Charron, Parmida Atighehchian, Massimo Caccia, Issam Hadj Laradji, Alexandre Drouin, Matthew Craddock, Laurent Charlin, and David Vázquez. Synbols: Probing learning algorithms with synthetic datasets. Advances in Neural Information Processing Systems , 33:134-146, 2020.

Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society , volume 33, 2011.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157 , 2019.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems , pages 2524-2533, 2017.

Maxim Raginsky, Alexander Rakhlin, Matthew Tsao, Yihong Wu, and Aolin Xu. Informationtheoretic analysis of stability and bias of learning algorithms. In 2016 IEEE Information Theory Workshop (ITW) , pages 26-30. IEEE, 2016.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

gwgundersen. Bayesian online changepoint detection. https://github.com/gwgundersen/ bocd.git , 2020.

Changjian Shui, Gezheng Xu, Qi Chen, Jiaqi Li, Charles X Ling, Tal Arbel, Boyu Wang, and Christian Gagné. On learning fairness and accuracy on multiple subgroups. Advances in Neural Information Processing Systems , 35:34121-34135, 2022.

Changjian Shui, Qi Chen, Jiaqi Li, Boyu Wang, and Christian Gagné. Fair representation learning through implicit path alignment. In International Conference on Machine Learning , pages 20156-20175. PMLR, 2022.

Min Wen, Osbert Bastani, and Ufuk Topcu. Algorithms for fairness in sequential decision making. In International Conference on Artificial Intelligence and Statistics , pages 1144-1152. PMLR, 2021.

Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 , 2019.

Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv:1810.12488 , 2018.

Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(5): 5513-5533, 2022.

Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems , 30, 2017.

Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems , 32, 2019.

Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. Advances in Neural Information Processing Systems , 33:11588-11598, 2020.

Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn , pages 3-17. Springer, 1998.

Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning to learn around a common mean. In ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 31 (NIPS 2018) , volume 31. NIPS Proceedings, 2018.

Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in neural information processing systems , 8, 1995.

Anastasia Pentina and Christoph Lampert. A pac-bayesian bound for lifelong learning. In International Conference on Machine Learning , pages 991-999, 2014.

Anastasia Pentina and Christoph H Lampert. Lifelong learning with non-iid tasks. Advances in Neural Information Processing Systems , 28, 2015.

Giovanni Cavallanti, Nicolo Cesa-Bianchi, and Claudio Gentile. Linear algorithms for online multitask classification. The Journal of Machine Learning Research , 11:2901-2934, 2010.

Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. The Journal of Machine Learning Research , 17(1):2853-2884, 2016.

Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems , pages 9516-9527, 2018.

Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. In International Conference on Learning Representations , 2018.

Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems , pages 7332-7342, 2018.

Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems , 32, 2019.

Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7201-7211, 2022.

Qi Chen and Mario Marchand. Algorithm-dependent bounds for representation learning of multi-source domain adaptation. In International Conference on Artificial Intelligence and Statistics , pages 10368-10394. PMLR, 2023.

Changjian Shui, Qi Chen, Jun Wen, Fan Zhou, Christian Gagné, and Boyu Wang. A novel domain adaptation theory with jensen-shannon divergence. Knowledge-Based Systems , 257: 109808, 2022.

Appendix

Table of Contents

A Preliminaries

In the appendix, we assume that all sets are convex subsets of R d , and we use ‖ · ‖ to denote the Euclidean norm.

A.1 Definitions

Definition A.1 (Lipschitzness) . A function f : W → R is L -Lipschitz w.r.t norm ‖ · ‖ if for all w 1 , w 2 ∈ W , | f ( w 1 ) -f ( w 2 ) | ≤ L ‖ w 1 -w 2 ‖

Definition A.2 (Quadratic Growth) . A function f : W → R has α -quadratic-growth w.r.t ‖ · ‖ for α > 0 if for any w ∈ W , we have:

where w ∗ denotes the global minimum point of f which is closest to w .

Definition A.3 (Convex) . A function f : W → R is convex w.r.t norm ‖ · ‖ if f is everywhere sub-differentiable and if ∀ x, y ∈ W ,

where ∇ f ( x ) is a subgradient of f at x .

Definition A.4 (Strongly Convex) . A function f : W → R is α -strongly convex w.r.t norm ‖ · ‖ if f is everywhere sub-differentiable and if ∀ x, y ∈ W ,

where ∇ f ( x ) is a subgradient of f at x .

Definition A.5 (Smoothness) . Afunction f : W → R is β -smooth w.r.t norm ‖·‖ if f is everywhere sub-differentiable and if ∀ x, y ∈ W ,

where ∇ f ( x ) is a subgradient of f at x .

Definition A.6 (Bregman divergence) . Let R : U → R be an everywhere sub-differentiable strongly convex regularization function. Then the Bregman divergence w.r.t function R for any x, y ∈ U is defined as:

A.2 Online Convex Optimization (OCO)

A.2.1 Definitions of regrets

Definition A.7 (Static regret) . The static regret of an OCO algorithm on the action set U w.r.t a sequence of cost functions { f t : U → R } T t =1 is defined as:

Definition A.8 (Traditional dynamic regret [40]) . The dynamic regret of an OCO algorithm on the action set U w.r.t a sequence of cost functions { f t : U → R } T t =1 and the comparator sequence ψ 1: T , ∀ ψ t ∈ U is defined as:

A.2.2 Static regret may be vacuous in shifting environments

Proof. For a shifting environment that has u ∗ T = arg min u 1 T ∑ T t =1 f t ( u ) changing with T , we can prove that the static regret may be vacuous as T goes to infinity.

Assume that the cost functions are non-negative, which is usually the case. Whenever we have f t ( u ∗ T ) -f t ( u ∗ T -1 ) ≥ a > 0 , we can obtain

The above inequality holds for any T ≥ 1 , so we can rewrite it as F t -F t -1 ≥ a ( t -1) + f t ( u ∗ t ) . By summing each side from t = 1 to t = T , we have F T ≥ F 0 + aT ( T -1) 2 + ∑ T t =1 f t ( u ∗ t ) ≥ aT ( T -1) 2 . Consequently, we have 1 T F T = 1 T ∑ T t =1 f t ( u ∗ T ) ≥ a ( T -1) 2 , which means the static regret will be vacuous when T →∞ .

A.2.3 Online algorithms

Definition A.9 (Follow The Leader (FTL)) . Given a sequence of strongly convex cost functions { f t : U → R } t ≥ 1 , FTL plays for t > 1 :

Definition A.10 (Follow The Regularized Leader (FTRL)) . Given a sequence of convex cost functions { f t : U → R } t ≥ 1 , a strongly convex regularization function R : U → R , and the starting point u 1 = arg min u ∈U R ( u ) , FTRL plays for t > 1 :

Khodak et al. [31] write FTRL as u t = arg min u ∈U ∑ t -1 i =1 f i ( u ) + B R ( u ‖ u 1 ) , while we follow Shalev-Shwartz et al. [41] and Hazan et al. [40].

Definition A.11 (Online Mirror Descent (OMD)) . Given a sequence of convex cost functions { f t : U → R } t ≥ 1 , a strongly convex regularization function R : U → R , and u 1 = ∇ R ∗ (0) , OMD plays for t > 1 :

where R ∗ (˜ u ) = max u 〈 u, ˜ u 〉 -R ( u ) is the Fenchel conjugate of R ( u ) .

There exists an equivalent linearized version of the above FTRL and lazy OMD through the linearization of the convex cost functions.

Definition A.12 (Lazy OMD) . Given a sequence of convex cost functions { f t : U → R } t ≥ 1 , a strongly convex regularization function R : U → R , and ∇ R (˜ u 1 ) = 0 , u 1 = arg min u ∈U B R ( u ‖ ˜ u 1 ) , then lazy OMD and FTRL play for t > 1 :

The proof of the equivalence can be found in Hazan et al. [40].

B Missing Proofs in Section 3

Definition B.1. Consider any base learner A that takes a fixed prior information u with a data set S ∼ µ m as input and outputs W = A ( S, u ) ∼ P W | S,u . Define the expected generalization gap of algorithm A as

and the expected excess risk as

where w ∗ = arg min w ∈W L µ ( w ) is the hypothesis that achieves the minimum true risk.

B.1 Excess risk bound for Gibbs algorithm

Lemma B.2. Let Q be an arbitrary distribution on W and let β > 0 be the inverse temperature that balances fitting and generalization. Then, we can jointly denote u = ( β, Q ) . Let S ∼ µ m be a sample of examples. The solution to the optimization problem

is given by the Gibbs algorithm which satisfies

See Proof of Theorem 5 in [48].

Theorem B.3 (Excess risk bound for Gibbs algorithm of meta-parameter u = ( β, φ ) ) . Suppose W = R d and assume that glyph[lscript] ( · , z ) ∈ [0 , 1] is L -Lipschitz for all z ∈ Z . Let W g ∼ P W g | S,u denote the output of the Gibbs algorithm 3 applied on data set S with meta-parameter u = ( β, Q ) . Let w ∗ be the hypothesis that achieves the minimum true risk among W . The excess risk of W g satisfies

Specifically, if we assume Q is the Gaussian distribution N ( φ, σ 2 1 d ) with σ = m -1 / 4 d -1 / 4 L -1 / 2 and denote u = ( β, φ ) , then we have

Proof. The original proof can be found in Corollary 3 [48], we provide the proof below with more details and some tiny modifications. Consider an arbitrary data-free distribution related to the unknown minimum w ∗ of the true risk, i.e. , N ( w ∗ , σ 2 1 d ) . Then for any σ > 0 , we have:

The first step is obtained with previous work's results that Gibbs algorithm is ( 2 β m , 0) -differentially private [49]. The second inequality makes use of the non-negativity of KL divergence. The last inequality derives naturally from the fact that the output of the Gibbs algorithm follows the optimal distribution for the objective function in Lemma B.2.

Since glyph[lscript] ( ., z ) is L -Lipschitz for all z ∈ Z , we can obtain the following by combining the Jensen's inequality:

Then we have

So we have for any σ > 0 ,

The last three lines are obtained with the assumption that Q is the Gaussian distribution N ( φ, σ 2 1 d ) and setting σ = m -1 / 4 d -1 / 4 L -1 / 2 .

B.2 Excess risk bound for SGD

Theorem B.4 (Excess Risk Bound for SGD of meta-parameter u = ( η, φ ) ) . Suppose W = R d , let w ∗ def = arg min w ∈W L µ ( w ) be the hypothesis that achieves the minimum population risk in W . Consider the SGD algorithm with K updates starting from W 1 = φ : W k +1 = W k -η ∇ glyph[lscript] ( W k , Z k ) , where Z k is a sample randomly selected from data set S . Assume the loss function glyph[lscript] ( · , z ) is convex, β -smooth and L -Lipschitz for all z ∈ Z . Let the output of the algorithm be W = 1 K ∑ K k =1 W k . Then, the excess risk bound is given by:

Proof. The excess risk can be decomposed as

The first term is the generalization gap defined in Definition B.1. Note that the second term is not the optimization error glyph[epsilon1] W opt def = | E W,S [ L S ( W )] -E W,S [ L S ( w ∗ S )] | used in [38], which is the expected gap between the SGD output W and the ERM output w ∗ S . The optimization error indicates how close the SGD output can approximate the possibly intractable exact ERM solution, which is an upper bound of the absolute value of the second term | E W,S L S ( W ) -E S [ L S ( w ∗ )] | ≤ glyph[epsilon1] W opt (can be derived with Lemma 5.1 in [50]).

Here, we directly make use of the convexity and obtain the gap compared to the true minimum. It's easy to bound E W,S L S ( W ) -E S L S ( w ∗ ) using the method provided in Shalev-Shwartz and Ben-David [51] with some tiny modifications.

Since we have W k +1 = W k -η ∇ glyph[lscript] ( W k , Z k ) , W 1: K is related to the randomness introduced by the sampling path Z 1: K from S . Moreover, for the convex loss function, we have:

where V k = ∇ glyph[lscript] ( W k , Z k ) . The inequalities are obtained by using Jensen's inequality and the last two equalities are obtained with the fact that W 1: K is determined by V 1: K -1 .

Moreover, we have E [ V k | W k ] = E [ V k | V 1: k -1 ] = ∇L S ( W k ) . The first equality comes from the update rule, where W k is determined by the previous k -1 gradients and the meta parameter u = ( η, φ ) . Note that u is not a random variable and hence no randomness needs to be considered. The second equality holds because Z k is uniformly sampled from S , so V k is an unbiased estimator of ∇L S ( W k ) .

Hence,

By using Lemma14.1 in Shalev-Shwartz and Ben-David [51], we can obtain that

Since the loss function is L -Lipschitz, we can further obtain:

We have assumed that the loss function is convex and β -smooth. Therefore, following Theorem 4.7 in Hardt et al. [50], we have the following upper bound for the generalization gap of SGD if the step size is smaller than 2 β :

So, we can conclude

B.3 Excess risk bound for SGLD

Theorem B.5 (Excess risk bound for SGLD of meta-parameter u = ( η, φ ) ) . Suppose W = R d , let w ∗ def = arg min w ∈W L µ ( w ) be the hypothesis that achieves the minimum population risk in W . For SGLD algorithm with K adaptation starting from W 1 = φ : W k +1 = W k -η ∇ glyph[lscript] ( W k , Z k ) + ξ k , where Z k is a sample randomly selected from data set S , ξ k is the independently injected isotropic Gaussian noise N (0 , σ 2 1 d ) . Assume the loss function glyph[lscript] ( · , z ) is convex and L -Lipschitz for all z ∈ Z . And let the output of the algorithm be W = 1 K ∑ K k =1 W k . Then we have the following excess risk bound:

Proof. Analogous to the proof of SGD, we can decompose the excess risk as

According to Pensia et al. [38], we have | gen ( µ, SGLD ) | ≤ √ 1 4 m Kη 2 L 2 σ 2 .

In addition, we have the update for SGLD: W k +1 = W k -η ∇ glyph[lscript] ( W k , Z k )+ ξ k , where W 1: K is related to the randomness introduced by the sampling path Z 1: K from S and the injected noise ξ 1: K .

Conduct similar proof as the above SGD algorithm and use Lemma 14.1 in [51], we can obtain that

Since the loss function is L -Lipschitz, we have

Combining the aforementioned generalization gap bound of SGLD, we conclude the proof.

B.4 Excess risk bound for Regularized Loss Minimization (RLM)

Theorem B.6 (Excess risk bound for RLM of meta-parameter u = ( β, φ ) ) . Suppose W = R d , let w ∗ def = arg min w ∈W L µ ( w ) be the hypothesis that achieves the minimum population risk in W . Let us denote the output of RLM with Tikhonov regularization as W = arg min w ∈W L S ( w ) + 1 β ‖ w -φ ‖ 2 . Suppose the loss function glyph[lscript] : W×Z → [0 , 1] is convex and L -Lipschitz, so the excess risk of the above algorithm is bounded by:

Proof. Here, since the algorithm is deterministic, we can consider P W | S,u as a delta distribution centered on w ∗ S = A ( u, S ) . Then, the bound becomes

which is equivalent to Corollary 13.8 in [51] with φ = 0 , β = 1 λ .

To prove the above bound, we use the same decomposition of excess risk:

From Corollary 13.6 in [51], the stability of RLM rule satisfies

which means the RLM rule using Tikhonov regularization described above is on average-replace-onestable with rate 2 L 2 β m . From the definition of this algorithm, we have for any w ∈ W ,

Take expectation w.r.t S , we can obtain E S L S ( w ∗ S ) ≤ L µ ( w ) + 1 β ‖ w -φ ‖ 2 , ∀ w ∈ W . Finally, combining the stability bound and letting w = w ∗ , we conclude the proof.

B.5 Unified form of excess risk upper bound

Lemma B.7. Assume that L µ t ( · ) has α -quadratic growth (defined in Appendix A.2) ∀ t ∈ [ T ] and that for a given base learner A , we have R excess ( A , u t ) ≤ b t + c t ‖ φ t -w ∗ t ‖ 2 β t + d t β t + e t , with b t , e t ≥ 0 , c t , d t > 0 . Then we have R excess ( A , u t ) ≤ f t ( u t ) , where f t ( u t ) is given by eq. (1).

Proof. Let ¯ w t = E W t be the expected output, and ξ 0 def = ‖ w t -¯ w t ‖ 2 characterizes the randomness of the algorithm output, which should be small enough.

Then, using Titu's lemma, Jensen's inequality, and the α -quadratic growth assumption, we get:

Rearrange and put the upper bound of ‖ φ t -w ∗ t ‖ 2 into the excess risk bound, replace √ c t /d t β ′ t = αβ t -4 c t with β t > 4 c t /α and use again the Titu's lemma, we get

It's easy to relate κ t , ∆ t , glyph[epsilon1] t , glyph[epsilon1] 0 to the terms in the above bound. Consequently, we have glyph[epsilon1] 0 = 2 αξ 0 = 2 α ‖ w t -E W t ‖ 2 , κ t = 2 √ c t d t , a = 1 2 α , b = 2 α , glyph[epsilon1] t = αb t 2 c t + 8 d t c t α +2 e t , and ∆ t = 4 √ c t d t α + e t 2 √ c t d t .

We prove Proposition E.1 and Proposition E.4 without using this Lemma and provide more details in the corresponding derivations.

Theorem B.8. For any t ∈ [ T ] , assume that L µ t ( · ) has α -quadratic growth 4 (see Definition A.2), then whenever the base learner is SGD, SGLD, RLM, or the Gibbs algorithm, there exists f t ( · ) that gives

The meta-parameter u t = ( β t , φ t ) decomposes into an initialization or bias φ t and a learning rate β t (if A ∈ { SGD , SGLD } ) or a regularization coefficient (if A ∈ { RLM , Gibbs } ). Moreover, w t denotes the (single) output for the (possibly randomized) base learner, and glyph[epsilon1] 0 def = 2 α || w t -E W t || 2 characterizes the randomness of the output. Finally, a and b are constants, and glyph[epsilon1] t , κ t , ∆ t are functions of the task sample size m t that characterize the base learner (see Appendix B and D).

Proof. According to Theorem B.3, Theorem B.4, Theorem B.5 and Theorem B.6, we have

Combining Lemma B.7, we observe that b t = 0 , c t = d 1 / 2 m 1 / 2 t L 2 , d t = 1 2 m t , e t = m -1 / 4 t L 1 / 2 d 1 / 4 for Gibbs algorithm; b t = 0 , c t = 1 2 K t , d t = L 2 2 + L 2 K t m t , e t = 0 for SGD; b t = dσ 2 2 , c t = 1 2 K t , d t = L 2 2 + √ K t 4 m t L σ , e t = 0 for SGLD; and b t = 0 , c t = 1 , d t = 2 L 2 m t , e t = 0 for RLM. Therefore, simply adding the transformation η t = √ c t /d t β t /α +4 c t /α inside the algorithm 5 , the algorithms mentioned above have a unified form of excess risk upper bound as defined in eq. (1).

B.6 Convexity of the unified upper bound

Lemma B.9. ∀ t ∈ [ T ] , the cost function that has the following form:

is convex w.r.t u t = ( β t , φ t ) , where φ t , w t ∈ W ⊆ R d .

Proof.

The Hessian matrix of f t ( β t , φ t ) :

For any h = ( h 0 , h d ) ∈ R × R d , and β t > 0 , φ ∈ R d we have:

, so f t is convex w.r.t u t .

C Table of Notations

D Missing Proofs in Section 5

D.1 AER bound in static environments

where ˆ D is the diameter of the base learner outputs, φ ∗ = ∑ T t =1 κ t ∑ T t =1 κ t w t is the optimal initialization (or bias) in hindsight, V 2 = ∑ T t =1 κ t ∑ T t =1 κ t ‖ φ ∗ -w t ‖ 2 is the variance of the base learner outputs, and glyph[epsilon1] = ∑ T t =1 κ t glyph[epsilon1] t ∑ T t =1 κ t , κ max = max { κ 1: T } , glyph[epsilon1] max = max { glyph[epsilon1] 1: T } , ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v . glyph[epsilon1] 0 is the across tasks maximum corresponding term in eq.(1), and glyph[epsilon1] v ∈ R + , If glyph[epsilon1] v > bV 2 + glyph[epsilon1] , we replace glyph[epsilon1] 0 in the bound with ˜ glyph[epsilon1] 0 . Let B def = [ √ ˜ glyph[epsilon1] 0 /a, √ ( b ˆ D 2 + glyph[epsilon1] +˜ glyph[epsilon1] 0 ) /a ] , then, the meta-parameter set in Algo. 2 is U def = B × W .

Proof. At first, in a static environment, all the tasks share the same optimal prior information u ∗ = arg min u ∈U ∑ T t =1 f t ( u ) (the comparator in static regret). As proved in Lemma B.6, the cost functions are convex, so the sum of T convex functions ∑ T t =1 f t ( u ) is convex. Then we have the minimum attained at u ∗ = ( β ∗ , φ ∗ ) . By solving min u ∑ T t =1 f t ( u ) , we get φ ∗ = ∑ T t =1 κ t ∑ T t =1 κ t w t and β ∗ = √ ( bV 2 + glyph[epsilon1] + glyph[epsilon1] 0 ) /a , where glyph[epsilon1] = ∑ T t =1 κ t glyph[epsilon1] t ∑ T κ t and V 2 = ∑ T t =1 κ t ∑ T κ t ‖ φ ∗ -w t ‖ 2 .

According to Algo. 2 , φ t is determined by the previous base learner outputs w 1: t -1 and the initialization φ 1 = 0 . As long as 2 bκ t γ t β t ≤ 1 , ∀ t , we have φ t is always constrained within the ball (of diameter ˆ D ) that contains ˆ W . So we have 0 ≤ ‖ φ t -w t ‖ ≤ ˆ D and V 2 = ∑ T t =1 κ t ∑ T t =1 κ t ‖ φ ∗ -w t ‖ 2 ≤ ˆ D 2 . Moreover, we constrain β t ∈ [ √ ˜ glyph[epsilon1] 0 /a, √ ( b ˆ D 2 + glyph[epsilon1] +˜ glyph[epsilon1] 0 ) /a ] , where ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v is related to the randomness of the base learner output glyph[epsilon1] 0 and a factor glyph[epsilon1] v to control the range of β t . We will see later that glyph[epsilon1] v can affect the gradient norm of the meta-parameter.

To derive the static regret bound, we need to use the following result first:

So we have 〈∇ f t , ( u t -u ∗ ) 〉 ≤ ‖ u t -u ∗ ‖ 2 -‖ u t +1 -u ∗ ‖ 2 2 γ t + γ t ‖∇ f t ‖ 2 2 . Moreover, with β t ∈ [ √ ˜ glyph[epsilon1] 0 /a, √ ( b ˆ D 2 + glyph[epsilon1] +˜ glyph[epsilon1] 0 ) /a ] , we have

Hence, ‖∇ f t ‖ 2 ≤ a 2 κ 2 t ( b ˆ D 2 + glyph[epsilon1] max ) 2 ˜ glyph[epsilon1] 2 0 + 4 ab 2 κ 2 t ˆ D 2 ˜ glyph[epsilon1] 0 .

In addition, ‖ u t -u ∗ ‖ 2 = ‖ φ t -φ ∗ ‖ 2 + | β t -β ∗ | 2 ≤ ˆ D 2 + b ˆ D 2 + glyph[epsilon1] max a . So the static regret is bounded by:

We have the learning rate schedule γ t = γ √ t and 1 /γ 0 = 0 . The third inequality holds because of the aforementioned bounds of ‖ u t -u ∗ ‖ 2 and ‖∇ f t ‖ . The last step is obtained by setting γ = ˜ glyph[epsilon1] 0 √ ˆ D 2 + b ˆ D 2 + glyph[epsilon1] max a κ max √ 4 ab 2 ˜ glyph[epsilon1] 0 ˆ D 2 + a 2 ( glyph[epsilon1] max + b ˆ D 2 ) 2 and ∑ T t =1 1 √ t ≤ 2 √ T , where κ max = max { κ 1: T } , glyph[epsilon1] max = max { glyph[epsilon1] 1: T } . According to the setting of ˜ glyph[epsilon1] 0 , we have two conditions.

Then, for the optimal comparator in hindsight u ∗ = ( β ∗ , φ ∗ ) , we have the accumulated cost:

Finally, we obtain:

2. If glyph[epsilon1] v > bV 2 + glyph[epsilon1] , β ∗ = √ ( bV 2 + glyph[epsilon1] + glyph[epsilon1] 0 ) /a < √ ˜ glyph[epsilon1] 0 /a cannot be attained. We can simply upper bound the cost function by replacing glyph[epsilon1] 0 with ˜ glyph[epsilon1] 0 and obtain a newly attainable optimal comparator in hindsight ˜ u ∗ = ( ˜ β ∗ , φ ∗ ) , where ˜ β ∗ = √ ( bV 2 + glyph[epsilon1] +˜ glyph[epsilon1] 0 ) /a ∈ [ √ ˜ glyph[epsilon1] 0 /a, √ ( b ˆ D 2 + glyph[epsilon1] +˜ glyph[epsilon1] 0 ) /a ] . Then, we have the accumulated cost:

Finally, we obtain:

D.2 AER bound in possibly shifting environments (Proof of Theorem 5.1)

Theorem D.2. Consider both static and shifting environments. If the excess risk's upper bound of the base learner A ( u t , S t ) can be formulated as a unified form in Eq.(1), then, the AER of DCML (in Algo. 1) is upper bounded by:

Here, κ n = ∑ M n k =1 κ n,k , ∆ n = ∑ M n k =1 κ n,k ∆ n,k , where the subscript n, k indicates k -th task in n -th slot. The optimal meta-parameter in hindsight of n -th slot is u ∗ n = ( β ∗ n , φ ∗ n ) and the path length of N slots is P ∗ = ∑ N -1 n =1 ‖ u ∗ n -u ∗ n +1 ‖ +1 . φ ∗ n = ∑ M n k =1 κ n,k κ n w n,k is the weighted average of the base learner outputs within the slot. β ∗ n = √ ( bV 2 n + glyph[epsilon1] n + glyph[epsilon1] 0 ) /a , where V 2 n = ∑ M n k =1 κ n,k κ n ‖ φ ∗ n -w n,k ‖ 2 2 is the variance in n -th slot and glyph[epsilon1] n = ∑ M n k =1 κ n,k κ n glyph[epsilon1] n,k . Moreover, G n is the maximal gradient norm of the cost function, ˜ D n is the diameter of meta-parameters in n -th slot, and ˜ D max = max { ˜ D n } N n =1 .

Proof. A possibly shifting environment can be considered slot-wise static or static, so we need to derive the dynamic regret bound. We use the subscript n, k to indicate the k -th task inside the n -th slot and the corresponding timestep is t = ∑ n -1 i =1 M i + k . To obtain such a bound, we first use the following result for the meta-parameter updates inside the n -th slot:

Let G n be the upper bound of the gradient norm related to the n -th slot, i.e. , max k {‖∇ f n,k ‖} ≤ G n . Applying the above result and the update rule in Algo. 1, we have the dynamic regret:

The meta-parameter updates that transition between slots adopt the hopping learning rate ρ , which reflects the environment change.

Denote

and

First, let us tackle T 2 ,

The first inequality is obtained with

where ˜ D max = max { ˜ D n } N n =1 is the maximal slot-wise diameter of the set of meta-parameters. Since we have assumed that 0 ∈ W , it gives ‖ u n +1 , 1 ‖ ≤ ˜ D max , ∀ n . The last inequality of the above proof holds because -‖ u N +1 , 1 ‖ 2 +2 u glyph[latticetop] N +1 , 1 u ∗ N -‖ u ∗ N ‖ 2 2 ρ = -‖ u N +1 , 1 -u ∗ N ‖ 2 2 ρ ≤ 0 .

Let ˆ D n be the diameter of the base learner outputs in n -th slot, and let ˆ D max = max { ˆ D n } N n =1 , κ max n = max { κ n,k } M n k =1 , glyph[epsilon1] max n = max { glyph[epsilon1] n,k } M n k =1 , glyph[epsilon1] max = max { glyph[epsilon1] max n } N n =1 .

We constrain β n,k ∈ [ √ ˜ glyph[epsilon1] 0 /a, √ ( b ˆ D 2 n + glyph[epsilon1] max +˜ glyph[epsilon1] 0 ) /a ] , where ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v is related to the randomness of the base learner glyph[epsilon1] 0 and a factor glyph[epsilon1] v to control the range of β t . We will see later that glyph[epsilon1] v can affect the upper bound of the gradient norm of the meta-parameter. We can further verify that 2 bγ n,k κ n,k β n,k ≤ 1 can always hold with ˜ glyph[epsilon1] 0 < a 2 ( glyph[epsilon1] max + b ˆ D 2 n ) 4 b 2 . Hence, φ n,k is always inside the smallest ball that contains ˆ W n . Then, we can define ˜ D 2 n def = ˆ D 2 n + b ˆ D 2 n + glyph[epsilon1] max n +˜ glyph[epsilon1] 0 a . Therefore, we have

Now we tackle T 1 . Since we scale the learning rate of the meta-parameter for shifting environments, we need to set ρ > γ n,k for better tracking of the change. Then, we note that if we define 1 γ n, 0 = 0 , we have

In addition, we have ‖ u n,k -u ∗ n ‖ 2 = ‖ φ n,k -ˆ φ ∗ n ‖ 2 + | β n,k -β ∗ n | 2 ≤ ˆ D 2 n + b ˆ D 2 n + glyph[epsilon1] max n a ≤ ˜ D 2 n . So, we can bound the dynamic regret as

The last inequality is derived with ‖ u ∗ N ‖ ≤ ˜ D max and ignoring the negative terms.

Denote the path length of the reference sequence as P ∗ = ∑ N -1 n =1 ‖ u ∗ n -u ∗ n +1 ‖ + 1 . By setting γ n,k = ˜ D n G n √ k (satisfies 1 /γ n, 0 = 0 ), ρ = √ 2 P ∗ ˜ D 2 max ∑ N n =1 G 2 n , and using ∑ M n -1 k =1 1 √ k ≤ 2 √ M n -1 , we have

For each slot n , the optimal prior in hindsight is u ∗ n = ( φ ∗ n , β ∗ n ) . By solving min u ∑ M n k =1 f n,k ( u ) , we can get:

where V 2 n = ∑ M n k =1 κ n,k ∑ Mn k =1 κ n,k ‖ φ ∗ n -w n,k ‖ 2 and glyph[epsilon1] n = ∑ Mn k =1 κ n,k glyph[epsilon1] n,k ∑ Mn k =1 κ n,k .

Thus, the accumulated cost for the slot-wise optimal priors in hindsight:

Finally, we obtain the bound in the main theorem:

Moreover, by limiting the range of β n,k ∈ B def = [ √ ˜ glyph[epsilon1] 0 /a, √ ( b ˆ D 2 n + glyph[epsilon1] max +˜ glyph[epsilon1] 0 ) /a ] , we have

Therefore, ‖∇ f n,k ‖ 2 ≤ a 2 ( κ max n ) 2 ( b ˆ D 2 n + glyph[epsilon1] max n ) 2 ˜ glyph[epsilon1] 2 0 + 4 ab 2 ( κ max n ) 2 ˆ D 2 n ˜ glyph[epsilon1] 0 , ∀ k ∈ [ M n ] .

Moreover, we let G 2 n = a 2 ( κ max n ) 2 ( b ˆ D 2 n + glyph[epsilon1] max n ) 2 ˜ glyph[epsilon1] 2 0 + 4 ab 2 ( κ max n ) 2 ˆ D 2 n ˜ glyph[epsilon1] 0 , replace ˜ D n with the tighter √ ˆ D 2 n + b ˆ D 2 n + glyph[epsilon1] max n a and use ˜ D 2 max = (1 + b/a ) ˆ D 2 max +( glyph[epsilon1] max +˜ glyph[epsilon1] 0 ) /a .

Analogous to the proof of static AER, if glyph[epsilon1] v ≤ bV 2 n + glyph[epsilon1] n , β ∗ n = √ ( bV 2 n + glyph[epsilon1] n + glyph[epsilon1] 0 ) /a ∈ B can be attained. Then, the above AER can be alternately bounded as:

Otherwise, if glyph[epsilon1] v > bV 2 n + glyph[epsilon1] n , β ∗ n = √ ( bV 2 n + glyph[epsilon1] n + glyph[epsilon1] 0 ) /a < √ ˜ glyph[epsilon1] 0 /a cannot be attained by optimizing within B . We can simply upper bound the cost function by replacing glyph[epsilon1] 0 with ˜ glyph[epsilon1] 0 and obtain newly attainable optimal comparators in hindsight ˜ u ∗ n = ( ˜ β ∗ n , φ ∗ n ) , where ˜ β ∗ n = √ ( bV 2 n + glyph[epsilon1] n +˜ glyph[epsilon1] 0 ) /a ∈ [ √ ˜ glyph[epsilon1] 0 /a, √ ( b ˆ D 2 + glyph[epsilon1] max +˜ glyph[epsilon1] 0 ) /a ] . Finally, we can obtain a bound for this case by replacing glyph[epsilon1] 0 in the first line of the above dynamic AER bound with ˜ glyph[epsilon1] 0 .

E Missing Proofs in Section 6

E.1 Cost function for Gibbs base learner

Proposition E.1. Suppose W ⊂ R d , let w ∗ t be the hypothesis that achieves the minimum population risk among W . Suppose glyph[lscript] ∈ [0 , 1] , and glyph[lscript] ( · , z ) is L -Lipschitz ∀ z ∈ Z . Assume that L µ t ( · ) has α -quadratic-growth for all t ∈ [ T ] . Let w t = A ( φ t , β t ) denote the output of the Gibbs algorithm applied on dataset S t , and Q t be the prior distribution and β t the inverse temperature of t -th task. Assume Q t is a gaussian distribution N ( φ t , σ 2 t 1 d ) with σ t = m -1 / 4 t d -1 / 4 L -1 / 2 . To meet the general form of the cost function, we replace β t with β ′ t by αβ t -2 √ dm t L = m 3 4 t d 1 4 √ Lβ ′ t . Hence, the upper bound of the excess risk of the Gibbs algorithm is given by

Proof. From Theorem B.3, we have:

Let ¯ w t = E W t be the expected output of the Gibbs algorithm, and let glyph[epsilon1] 0 = 2 α ‖ w t -¯ w t ‖ 2 be the randomness of the algorithm, which should be a small. Then, using Titu's lemma and the α -quadratic growth assumption on the loss function, we get

Consequently, we can obtain:

E.2 Static AER of Gibbs

Theorem E.2. Consider the Gibbs algorithm as described in Proposition E.1 to be the base learner. Apply the Meta-OGD (Algo. 2) on the cost function f t ( β ′ t , φ t ) . Further, assume that each task uses the same sample number m . Then

Proof. From Lemma B.6, we know that f t is convex w.r.t u , so we can directly apply OGD . Following Proposition E.1, we have

To apply Theorem D.1, we first find a = 1 2 α , b = 2 α , glyph[epsilon1] t = 2 α √ d m t L +2 √ Ld 1 4 m 1 4 t , ∆ t = 1 + 2 d 1 4 √ L αm 1 4 t ,

glyph[epsilon1] 0 = 2 α ‖ w t -¯ w t ‖ 2 , and κ t = d 1 4 √ L m 1 4 t . Since we have assumed m t = m, ∀ t ∈ [ T ] , then we have

Taking the above values into Theorem D.1, we have:

Since we have assumed the bounded gradient norm for the cost function, i.e. , L ′ -Lipschitz, then we can obtain:

which implies ˜ glyph[epsilon1] 0 ≥ aκglyph[epsilon1]/L ′ , where aκglyph[epsilon1]/L ′ ∈ O (1 / √ m ) . So it suffices to ensure ˜ glyph[epsilon1] 0 > Ω(1 / √ m ) .

For the Gibbs algorithm, by the law of large numbers, we have glyph[epsilon1] 0 = 2 α ‖ w t -E S t , A W t ‖ 2 → 2 α ‖ w t -E A W t ‖ 2 as m →∞ . In addition, since the variance of Gaussian is σ = m -1 / 4 d -1 / 4 L -1 / 2 , we have glyph[epsilon1] 0 ∈ o m (1) .

So if we set glyph[epsilon1] v = bV 2 < bV 2 + glyph[epsilon1] with constant V , then ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v satisfies the above condition. Consequently, we have

E.3 Proof of Theorem 6.1 (Dynamic AER of Gibbs)

Theorem E.3. Let W ⊂ R d . Assume that glyph[lscript] ( · , z ) ∈ [0 , 1] is L -Lipschitz ∀ z ∈ Z . Assume that L µ t ( · ) has α -quadratic-growth for all t ∈ [ T ] . Let w t = A Gibbs ( β t , φ t , S t ) be the output of the Gibbs algorithm (Definition B.2) on S t , where φ t is the mean of a prior Gaussian N ( φ t , σ 2 t 1 d ) with σ t = m -1 / 4 t d -1 / 4 L -1 / 2 and β t is the inverse temperature. Consider Algo. 1 ( DCML ) and further assume that each slot has equal length M and each task uses the sample number m . Then

For simplicity, let us assume that each environment slot has the same number of tasks, i.e. , M n = M , T = NM , and each task has the same number of samples, i.e. , m n,k = m . So we have κ n,k = κ max n = κ = d 1 / 4 L 1 / 2 m -1 / 4 and glyph[epsilon1] n,k = glyph[epsilon1] max n = glyph[epsilon1] = 2 α √ d/mL +2 √ Ld 1 / 4 m -1 / 4 .

Taking these values into Theorem 5.1, we can obtain:

Since we have assumed the bounded gradient norm for the cost function, i.e. , L ′ -Lipschitz, then we can obtain:

which implies ˜ glyph[epsilon1] 0 ≥ aκglyph[epsilon1]/L ′ , where aκglyph[epsilon1]/L ′ ∈ O (1 / √ m ) . So it suffices to ensure ˜ glyph[epsilon1] 0 > Ω(1 / √ m ) .

For the Gibbs algorithm, by the law of large numbers, we have glyph[epsilon1] 0 = 2 α ‖ w t -E S t , A W t ‖ 2 → 2 α ‖ w t -E A W t ‖ 2 as m →∞ . In addition, since the variance of Gaussian is σ = m -1 / 4 d -1 / 4 L -1 / 2 , we have glyph[epsilon1] 0 ∈ o m (1) . So if we set glyph[epsilon1] v = bV 2 n < bV 2 n + glyph[epsilon1] with constant V n , ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v satisfies the above condition. Consequently, we have

E.4 Cost function for SGD base leaner

Proposition E.4. Consider W ⊂ R d . Let w ∗ t be the hypothesis that achieves the minimum population risk among W . Suppose glyph[lscript] ( · , z ) is convex, β -smooth, and L -Lipschitz for all z ∈ Z . Assume that L µ t ( · ) has α -quadratic-growth ∀ t ∈ [ T ] . Let w t denote the output of the SGD algorithm applied on dataset S t and φ t be the initialization of t -th task, η t is the learning rate, and K t is the gradient updates number. To meet the unified form in Eq. (1), let η ′ t = ( K t αη t -2) κ t . Then, the upper bound of the excess risk is given by

From Theorem .B.4, we have the following excess risk bound for SGD:

Proof.

Let

Let ¯ w t = E W t be the expected output of the SGD algorithm of t -th task, and w t is the actual output. Let glyph[epsilon1] 0 = 2 α ‖ w t -¯ w t ‖ 2 be the randomness of the algorithm, which should be small. With the quadratic growth property, we have

Therefore, we can obtain

E.5 Static AER of SGD

Theorem E.5. Consider the SGD algorithm as described in Proposition E.4 to be the base learner. Apply the Meta-OGD (Algo. 2) algorithm on the cost function f t ( η ′ t , φ t ) . Further, assume that for each task, we have the same sample number m and that SGD performs the same number of gradient steps K . Then

Proof. According to Proposition E.4, we have κ t = √ ( L 2 +2 L 2 Kt mt ) K t α , a = 1 / 2 , b = 2 α and glyph[epsilon1] t = 2( L 2 +2 L 2 Kt mt ) K t α = 2 κ 2 t , ∆ t = 2 κ t , glyph[epsilon1] 0 = 2 α ‖ w t -¯ w t ‖ 2 .

For simplicity, we assume the SGD step and sample number are the same for different tasks. Hence, we have K t = K , m t = m , κ t = κ max = √ 1 Kα + 2 mα L = κ and glyph[epsilon1] t = glyph[epsilon1] = glyph[epsilon1] max = 2 κ 2 .

Put these variable into the bound of Theorem D.1, we have:

Since we have assumed the bounded gradient norm for the cost function, i.e. , L ′ -Lipschitz, then we can obtain:

which implies ˜ glyph[epsilon1] 0 ≥ aκglyph[epsilon1]/L ′ , where aκglyph[epsilon1]/L ′ ∈ O ( m -3 / 2 + K -3 / 2 ) . So it suffices to ensure ˜ glyph[epsilon1] 0 ∈ Ω( m -3 / 2 + K -3 / 2 ) .

For SGD, by the law of large numbers, we have glyph[epsilon1] 0 = 2 α ‖ w t -E S t , A W t ‖ 2 → 2 α ‖ w t -E A W t ‖ 2 as m →∞ . In addition, 2 α ‖ w t -E A W t ‖ 2 → 0 as K →∞ , so we have glyph[epsilon1] 0 ∈ o m (1) + o K (1) .

Then, if we set glyph[epsilon1] v = bV 2 < bV 2 + glyph[epsilon1] , s.t. ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v satisfies the above condition, we have

Considering a constant variance, it gives the following bound that has the same complexity factor and rate as Theorem 3.2 in [31] (Khodak et al. [31] used OGD as the base learner, where each step only takes one sample, i.e. , K = m ).

If we have V → 0 , we can set glyph[epsilon1] v = bV 2 + glyph[epsilon1] +1 /T 1 / 4 > bV 2 + glyph[epsilon1] , s.t. ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v satisfies the above condition. In this case, we have:

which has an additional term o m (1) + o K (1) compared to Theorem 3.2 in [31]. This is because the variance V 2 (task similarity) in our paper is calculated via the average SGD iterates. A similar term will be introduced using the same output for estimating V in [31].

E.6 Proof of Theorem 6.2 (Dynamic AER of SGD)

Theorem E.6. Let W ⊂ R d . Consider that glyph[lscript] ( · , z ) is convex, β -smooth, L -Lipschitz forallz ∈ Z . Assume that L µ t ( · ) has α -quadratic-growth for all t ∈ [ T ] . Let SGD be the base learner for each task where it outputs w t = A SGD ( η t , φ t , S t ) with learning rate η t and initialization φ t . Consider the Algo. 1 ( DCML ) on the SGD cost function. Further, assume each slot has equal length M , each task has the sample number m , and that SGD performs the same number of updating steps K . Then

Proof. For simplicity, we assume the SGD step and sample number are the same for different tasks. Hence, K n,k = K and m n,k = m . Based on Proposition E.4, we have a = 1 / 2 , b = 2 α . Then, we get κ n,k = √ 1 Kα + 2 mα L = κ glyph[epsilon1] n,k = glyph[epsilon1] = 2 κ 2 , and ∆ n,k = 2 κ n,k .

Put these variable into the bound of Theorem 5.1, we can obtain:

Since we have assumed the bounded gradient norm for the cost function, i.e. , L ′ -Lipschitz, then we can obtain:

which implies ˜ glyph[epsilon1] 0 ≥ aκglyph[epsilon1]/L ′ , where aκglyph[epsilon1]/L ′ ∈ O ( m -3 / 2 + K -3 / 2 ) . So it's sufficient to ensure ˜ glyph[epsilon1] 0 ∈ Ω( m -3 / 2 + K -3 / 2 ) .

For SGD, by the law of large numbers, we have glyph[epsilon1] 0 = 2 α ‖ w t -E S t , A W t ‖ 2 → 2 α ‖ w t -E A W t ‖ 2 as m →∞ . In addition, 2 α ‖ w t -E A W t ‖ 2 → 0 as K →∞ , so we have glyph[epsilon1] 0 ∈ o m (1) + o K (1) .

Then, if we set glyph[epsilon1] v = bV 2 n < bV 2 n + glyph[epsilon1] s.t. ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v satisfies the above condition, we have

The above bound has additional o m (1) + o T (1) compared to Theorems in [31]. Since the variance V 2 n (task similarity) in this paper is calculated via the average SGD iterates. A similar term will be introduced using the same algorithm outputs for estimating V in [31]. Moreover, our bound is in terms of average task excess risk, while [31] upper bounds the average task regret for online base learners using a different meta-learning algorithm.

Considering a constant variance, we have

When we have a large P ∗ with a small N (the environment occasionally changes with a large shift), the above bound has an improved rate of O (1 / √ M ) or √ P ∗ on the path length P ∗ (reflecting

environment similarities and shifts) compared to O ( 1 N ∑ N n =1 V n + 1 √ M +min { √ P ∗ MN , P ∗ NM } ) -the equivalent form of Theorem 3.3 [31] in this case.

If we have V n → 0 , we can set glyph[epsilon1] v = bV 2 n + glyph[epsilon1] +1 /M 1 / 4 > bV 2 n + glyph[epsilon1] , s.t. ˜ glyph[epsilon1] 0 = glyph[epsilon1] 0 + glyph[epsilon1] v satisfies the aforementioned condition. In this case, we have:

The above bound has an improvement of rate O (1 / 4 M ) on P ∗ as V n → 0 compared to the equivalent term O ( 1 4 √ M + √ P ∗ NM + o m (1) + o K (1) ) of Theorem 3.3 in [31].

F Experiments

In this section, we provide experimental settings, some additional experimental results, and experimental details. The code is available in https://github.com/livreQ/DynamicCML .

F.1 Experimental settings

lIf static,E1 =E2=...=EN =↑

Environment1

Environment2

Environment N

T1 =...=TK=E1

TK1+1=...=TK2=E2

TKN-1+1=...=TKN=EN

Random Initialization

Task

Task

Task

Task

Task

Task

or Pre-trained EnvTO

μ1

μK1

μK1+1

μK2

μKN-1+1

μKN

~ml

t

p

0.5

Omniglot (A)

0.5

Alphabet(A)

E

→

Ei+1

0.25

0.25

MNIST(B)

→

NewAlphabet(B)

0.25

0.25

FMNIST (C)

Fonts (C)

Moving 2D Gaussian

Omniglot-MNIST-FMNIST(OMF)Dataset

Large Concept Shift Synbols Dataset

Everyenvironmentisdistinct:

E1≠E2≠...≠EN,

P(Ei+1 = A)= 0.5,P(Ei+1 =B)= 0.25,P(Ei+1 = C) = 0.25,

Re-visitableenvironments,numberof environments:N=3

numberofenvironments:N=Tp1

r

At each time t,environment changes with probability p, current env Tt = E, next env P(Tt+1 = Ei+1） = p, P(Tt+1 = E)=1 — p

F.1.1 Detailed setting in OSAKA

Task data generating process Following OSAKA, we use a pre-trained meta-model as initialization. The datasets in OSAKA contain three environments. One is used for pre-training, and the other two are considered unseen, consisting of OOD tasks compared to the pre-trained environment. We run 10 CML episodes for each method, where each episode has a length of T = 10000 time steps.

At each time step, we choose to change to another environment with probability p . We select the new environment with a probability of 0 . 5 for the pre-trained environment and 0 . 25 for one of the other two. Then, a task is sampled from the selected environment. The above generation process is slightly different from the original OSAKA, which mainly considers a task-agnostic setting ( i.e. , unknown task boundary). They used a single probability for switching the task and environment simultaneously. In contrast, we focus on shifting environments under the task-aware setting, where the task switches with probability 1 , and the environment changes with the process described above. A visualization of the data-generating process is presented in Fig. 6.

Datasets We test DCML on two representative datasets in OSAKA.

The Synbols [43] dataset uses characters from different alphabets on randomized backgrounds as the pre-training environment. A new alphabet and font classification tasks are considered unseen environments. This is a large and complex dataset including a heterogeneous environment (font) with extremely large concept shift w.r.t other environments. We conduct a 4 -way 5 -shot classification for this dataset.

For Omniglot-MNIST-FashionMNIST (OMF), we pre-train the meta-model on the first 1000 classes of Omniglot [44]. Then, at CML time, the models are updated on a stream of tasks generated by the process discussed in the previous section exposed to the full Omniglot dataset and two OOD datasets - MNIST [45] and FashionMNIST [46]. In this study, we conduct 10 -way 5 -shot classification as the benchmark does.

Baselines We compared the proposed algorithm with different baselines. For a fair comparison, we further use the same meta-model pre-trained with MAML [4] for all the methods. During the continual learning phase, all the methods adapt with few-shot data and are then evaluated on separate test data for each task.

ANIL is a variation of MAML that only adapts the network's head w.r.t new tasks. Following OSAKA, MAML, and ANIL do not update the meta-model during the CL phase. so they will not suffer from forgetting but lack the plasticity to learn from tasks in new environments. They are compared to show the problem with static representations in shifting environments.

CMAML [13] includes an update modulation phase and a prolonged adaptation phase. The former uses a function g λ : R → (0 , 1) to modulate the learning rate of the meta-model proportionally to the loss value. The latter updates the meta-model with buffered tasks when the environment changes or a task switch is detected. Otherwise, it keeps updating w.r.t the previous model. When λ = 0 , CMAML is equivalent to MAML, which never updates the meta-model. When λ → ∞ , CMAML updates the meta-model with a constant learning rate.

MetaBGD and MetaCOG [8] perform CML based on MAML and Bayesian Gradient Descent (BGD), where Meta-COG introduced a per-parameter mask.

Fine-tuning uses the meta-model as initialization and consistently updates this model w.r.t the tasks encountered, which can be considered a lower bound in the setting to illustrate catastrophic forgetting .

Remark F.1 . Two modifications w.r.t original OSAKA benchmark: (1) We consider a task-aware setting, where the task constantly shifts at each timestep. (2) The cumulative performance in the original OSAKA is evaluated on the model before updating to current task data, which does not use the support query splits. Instead, we follow the CML setting and evaluate all the baseline algorithms on the model after updating to the current task.

Computational Complexity The computational complexities of all the methods are provided in Tab. 3, where the run time is measured in terms of the number of gradient computations. All the methods are assumed to use a K -step SGD as the inner algorithm.

Fine-tuning uses the meta-model as initialization and consistently updates this model w.r.t the tasks encountered. MAML and ANIL do not update the meta-model during the learning phase. Therefore, these methods need to compute T ∗ K gradients for the T encountered tasks.

MetaBGD and MetaCOG perform CML based on MAML and use Bayesian Gradient Descent (BGD) for meta-model adaptation, which requires M Monte Carlo samplings for the meta-gradient computation. Hence, the computation complexity is O ( T ∗ ( K + M )) .

DCML and CMAML need one meta-gradient and K inner gradients computations for each task, so the complexity is O ( T ∗ ( K +1)) .

Average test loss (log scale) at t=200

6

5

4

3

2

1

0

1

0.0

0.2

0.4

0.6

0.8

window static

1.0

p

(a) Average test loss w.r.t changing probability p .

6

4

2

0

2

4

0

25

50

75

100

125

150

175

200

0

50

100

150

200

0

25

50

75

100

125

150

175

200

(b) Bayes online changing-point detection

F.2 Additional results

F.2.1 Moving 2D Gaussian

Comparison of different environment shift detection methods In Fig. 7 (a), we plot the average test loss at t = 200 w.r.t the environment changing probability p in log scale, where we set the window size to 10 for "window". We can see that the "window" performs similarly to the "oracle" when the environment changes occur not too frequently. "Static" shows the worst result, then the "detect ." We simply (without carefully adjusting the hyper-parameters) adapted the Bayes Online Changing-point Detection (BOCD) to the algorithm output, which, in fact, highly depends on the performance of the base learner. Fig. 7 (b) illustrates the detection results of BOCD, where the green lines are the detection results and the red lines are the actual changing points. The upper figure shows one of the dimensions of the algorithm output w t with blue points. The lower figure illustrates the run-length posterior at each time step using a logarithmic color scale, where darker indicates a higher probability. We can see that with the current hyper-parameter setting, detection results are delayed compared to the actual changing points. BOCD is more likely to miss detecting some changing points rather than conducting false detection. The pseudo-code for BOCD is provided in Algo. 3.

Not always a necessity for the exact detection of changing points We observed a better performance of "window" than "oracle" for the OSAKA dataset in the main paper, which implies that the exact changing points detection may not be necessary.

This result is related to the algorithm, which does not necessarily hold for other methods that maintain multiple meta-models. By updating a single meta-model online , the context switch cost exists for reconstructing the meta-knowledge in each slot. Setting a fixed window size can ensure the meta-knowledge quality of each slot and yield good performance on average.

In addition, the phenomenon is related to the overlap between the consecutive environment distributions and the environment shift probability p . We empirically tested these factors, and the result is presented in Fig. 8. In Fig. 8, "oracle" is better than "window" when p < 0 . 1 , and the gap becomes more evident when the distribution overlap is smaller.

Remark F.2 . A larger overlap between distributions indicates that they are more similar, which can provide a better transfer performance (smaller test loss in Fig. 8 (a) than in (b)). In this case, the two distributions are close, so it's hard to determine precisely which distribution the encountered task belongs to. When two distributions are well-separated, it's easy to detect the shift correctly, but the transfer error is larger.

F.2.2 OSAKA benchmark

We present in Tab. 6 additional results for the Synbols dataset with the environment changing probability p = 0 . 4 and p = 1 . 0 . We can see DCML performs much better than baselines on new environments, especially the hardest one - Font.

oracle detect

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

p

0

1

2

3

4

5

Average test loss (log scale) at t=200

oracle

detect

window

static

(a)

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

p

1

0

1

2

3

4

5

6

7

Average test loss (log scale) at t=200

oracle

detect

window

static

(b)

(a)

(b)

(c)

(d)

avg_test_acc_so_far

avg_test_acc_so_far_omniglot

Showing first 10 runs

Showing first 10 runs

一 CMAML

－ DCML_oracle

一DCML_win

MAML

一CMAML

－ DCML_win － DCML_oracle

MAML

ANIL

一 ANIL

0.995

0.88

0.99

0.86

0.985

0.98

0.84

0.975

0.82

Step

0.97

Stepr

2k

4k

6k

8k

2k

4k

6k

8k

avg_test_acc_so_far_fasion_mnist

avg_test_acc_so_far_mnist

Showing first 10 runs

Showing first 10 runs

CMAML

- DCML_oracle

一DCML_win

一 MAML

一

CMAML

－ DCML_win － DCML_oracle

一MAML

ANIL

一 ANIL

0.68

0.9

0.66

0.85

0.64

0.62

0.8

0.6

0.75

0.58

Step

Step

2k

4k

6k

8k

2k

4k

6k

8k

In addition, we provide the experimental records in wandb for the OMF dataset when the environment changes with probability p = 0 . 8 . The average test accuracies w.r.t timesteps in the CL phase are plotted in Fig. 9.

F.3 Experimental details

F.3.1 Moving 2D Gaussian

Model As we described in the main paper, the hypothesis space of the moving 2D Gaussian is R 2 . And we adopt mean square loss ‖ w -x ‖ 2 , which is 2-strongly convex, so it's 2-quadratic growth with α = 2 . Moreover, we limited the data to [ -10 , 10] × [ -10 , 10] . Thus, for the given loss function, the Lipshitz constant is L = 20 √ 2 , and we set L = 30 in the code.

Training Details The hyper parameter settings and training details for Moving 2D Gaussian are presented in Table 5. We can see with the Algorithm 1 provided in the main paper. There is no need to set the hyper-parameters for the base learner. They are automatically adapted to the derived equations. Moreover, the initial learning rate of the meta learner is controlled by D_hat_guess. Then, it's also scheduled automatically with time and environment changing points.

Computing Resource All experiments for Moving 2D Gaussian were tested on a Mac with an Intel Core i5 CPU and 8G memory.

F.3.2 OSAKA benchmark

Model for OMF dataset We used a CNN network architecture as the basic model. The four modules are identical: A 3 × 3 2d convolution layer with 64 filters, stride 1, and padding 1. The following are a batch normalization layer, a Relu layer, and a max-pooling layer of stride 2. The image data passing the aforementioned modules form a 64 × 1 × 1 feature map, which was further taken into a fully connected layer. Then the fully connected layer outputs the logits for a 10 -class classification. At last, the cross-entropy loss is calculated with the logits and the corresponding labels.

Model for Synbols dataset We used a CNN network architecture as the basic model. The four modules are identical: A 3 × 3 2d convolution layer with 64 filters, stride 1, and padding 1. The following are a batch normalization layer, a Relu layer, and a max-pooling layer of stride 2. The image data passing the aforementioned modules form a 64 × 2 × 2 feature map, which was further flattened and taken into a fully connected layer. Then, the fully connected layer outputs the logits for a 4 -class classification. At last, the cross-entropy loss is calculated with the logits and the corresponding labels.

Training Details The hyper-parameter settings and training details for OSAKA data sets are outlined in Table 6.

Computing Resource The experiments for OSAKA were run on a server node with 6 CPUs and 1 GPU of 32GB memory.

Hyper-parameter Search Following OSAKA, the hyper-parameters were tuned by random search, and the same number of trials were allocated for each algorithm. For each trial, we sampled hyperparameter combinations uniformly from the search space presented in Tab. 7 for baselines and in Tab. 8 for DCML.

According to the experimental setting of OSAKA, we do not need a validation dataset for searching the hyper-parameters. New CML episodes (task sequences) generated with different random seeds can be considered held-out data. The results reported in the paper are tested on 10 new CML episodes using the learned hyper-parameters.

F.4 Bayes Online Changing-point Detection

We modify the implementation of BOCD in [52] for our experiments. The detailed method is presented in [42]. Given a sequence of tasks, we use the base leaner outputs w 1: t as observations of BOCD to be divided into non-overlapping product partitions. Each partition can be considered as sampled from the same static environment slot. BOCD model the detection of changing point as estimating the posterior distribution over the current run-length r t ∈ [0 , 1 , ..., t ] , i.e. , P ( r t | w 1: t ) = P ( r t ,w 1: t ) P ( w 1: t ) . Thus, r t = 0 means to start a new partition.

We treat the hypothesis as factorized Gaussian and apply BOCD on each dimension of the algorithm output w t . For Moving 2D Gaussian, we detect the whole w t , since it's just 2D. Here, we provide the pseudo-code in Algorithm 3 with Gaussian Prior for each dimension applied in our experiments.

Algorithm 3 Pseudo code for detecting environment change

Require : P ( r 0 = 0) = 1 , prior mean µ 1 = 0 , prior precision δ 1 = 1 / 0 . 1 , changing probability p , data precision δ w = 1

G Additional Discussion

G.1 Learning-forgetting w.r.t meta-updates

In this section, we offer some intuitive interpretations for the adaptive updates of the meta-parameter of the base learner.

The updates of the meta-parameter ( φ and β ) in Algo. 1 and 2 can be separately expressed as:

Considering the case that all the tasks have the same sample number. β t reflects the confidence of A on the given initialization for learning the task t . For example, β t is the learning rate (step size) for SGD. A large learning rate means that the learner relies more on the data, and a small one means believing the prior. From the update of β , we can see a large β t +1 implies that the current task output is far from its initialization ( ‖ φ t -w t ‖ is large), which indicates the current task t may be an outlier, exceeding the range of meta-model for recovering its performance. Hence, the environment may have changed. So in the next task, the base learner learns more from the task data instead of keeping close to the initialization ( β t +1 is large).

The learning rate of the initialization (or bias) is determined by 2 bκ t γ t β t . As mentioned above, a large distance from the initialization indicates that the current task t may be an outlier. So from t +1 step, the tasks are treated with low confidence to update the common initialization φ t +1 with 2 bκγ t β t +1 being small ( β t +1 is large). Then until the distance w.r.t the common initialization holds a small value for several rounds, β also decays to a smaller value. The tasks start to contribute more to updating the initialization.

The above mechanism works well for static environments and can address forgetting in shifting environments. But it will lose tracking ability in the shifting environment, so we need slot-wise adjusting for γ t to obtain a better trade-off.

G.2 Real-world examples for practicality and necessity of CML

A typical real-world application of CML is the online recommendation system, where we aim to predict the user's preference for various products. Different users' preferences are considered as different task distributions. The distribution of users can be regarded as a task environment. The recommendation system maintains a meta-model that predicts the recent preferences shared across users and adapts task-specific models that predict the user-specific preferences.

The user queries for displaying products arrive sequentially, and the system randomly distributes several (e.g., m t = 200 ) products in response to the query. Denote the products and the corresponding preferences for t -th user as ( X i , Y i ) ∼ µ t , ∀ i ∈ [ m t ] , which are i.i.d.. The labels are initially unknown for this setting, but a few labels can be acquired through the following interaction:

The meta-model predicts the preferences over products and displays them according to the predicted preference order.

Normally, the user will only view a small part ( m ′ t = 50 ) of the distributed products, and the preference can be determined with the clicks or view time over these products, which gives the few-shot labeled samples.

Then, the task-specific model is adapted from the meta-model with these examples. We hope the model can generalize on the rest of the unseen examples with correct preference prediction, which can provide new recommendations.

Finally, the task-specific model of each user is used for updating the meta-model. Since the shared preferences change with the season, fashion trends, and accidental events, the task environment is shifting.

G.3 More details on DCML

Relate parameters in eq. 1 to specific base learner Since the proposed framework is valid for different base learners, the parameters like a , b , and κ in the unified form of excess risk upper bounds

(cost function) are related to the specific algorithm . Since all of the experiments adopt SGD, we give here a detailed description of it.

According to Proposition D.4, the K -step SGD with learning rate η t and initialization φ t for the t -th task has the following cost function:

where a = 1 2 , b = 2 α, κ t = L √ 1 Kα + 2 m t α , glyph[epsilon1] t = 2 κ 2 t , ∆ t = 2 κ t . The learning rate of the base learner η t = ( β t κ t + 2) / ( Kα ) is related to the sample number m t of each task via κ t , which can reflect the generalization.

Impact of the initial meta-parameter on the model's behavior The initial meta-parameter substantially affects the model's behavior. For instance, the aforementioned recommendation system suffers from a well-known cold-start problem. A well-pertained model initialization φ 1 can help address the problem.

Given a good initialization φ 1 , β 1 also affects the performance substantially. According to the meta-parameter adaptation, if β 1 is too large, the algorithm tends to keep the prior knowledge and will not learn from the data. If β 1 is too small, the algorithm will ignore the initialization and learn from scratch.

How does DCML adjust the meta-parameter The updates in Algo. 1 for the meta-parameter u = ( β t , φ t ) are:

We adjust the learning rate of the meta-parameter ( γ t ) with the following strategy:

For k -th task inside the n -th environment (slot), γ t = γ 0 / √ k , where the initialization is determined by the theoretical optimal value γ 0 = glyph[epsilon1] 0 κ t √ (1+ b/a ) ˆ D 2 + glyph[epsilon1] t /α 4 ab 2 ˆ D 2 glyph[epsilon1] 0 + a 2 ( b ∗ ˆ D 2 + glyph[epsilon1] 2 t ) 2 .

When an environment change is detected, γ t is set to a large hopping rate γ t = ρ , which is related to the path length.

How to set hyper-parameters In the experiments, we tune the following hyper-parameters α, K, L, ˆ D,glyph[epsilon1] 0 , ρ, β 1 by random search on the defined space in Tab. 8. Specifically for the simple moving 2D Gaussian: since the loss function is 2-strongly convex, we have α = 2 .

G.4 Limitations

The proposed theory works for non-convex loss when using the Gibbs algorithm. However, We do not offer analysis for SGD with non-convex loss, where it's hard to find an easily optimizable form for the excess risk upper bound. Excess risk analysis for gradient-based learning algorithms with non-convex loss is a promising but challenging direction for the deep learning community. We hope to work on this topic in the future.

In this paper, we do not provide theoretical guarantees for memory-based approaches, which can better address meta-level forgetting with additional memory cost. Future theoretical studies can be conducted to complete the framework.

G.5 Broader Impacts

In general, this work is fundamental. It aims to understand the bi-level learning-forgetting trade-off in CML and improve the model performance in real-time intelligent systems with a better trade-off balance. The potential negative impacts depend on the specific application.

Potential negative impacts Since the main objective is to obtain the optimal average excess risk over rounds, when each task relates to a person in some applications like personalized recommendation systems or medical diagnosis systems, it may cause some fairness issues for individuals, especially those who appear around the changing points.

How to address? The aforementioned potential unfairness can be mitigated by adding calibration modules as in [53-55] or providing more training data for tasks around changing points.

H Additional Related Works

H.1 Summary and comparison of related settings

We present different settings related to the Continual Meta-Learning (CML) problem studied in this paper in Tab. 9. The detailed discussions in each setting can be found in the following sections.

Supervised Learning (SL) methods learn as a single task from the train data and evaluate on independent test data. The data points are assumed, i.i.d. sampled from the same distribution. Continual Learning (CL) aims to learn from a sequence of non-i.i.d. tasks, where the data points inside each task are i.i.d. Meta-Learning (ML) has many settings. Here, we present the statistical version. All the tasks are assumed to be i.i.d. sampled from the same task environment. The evaluation is conducted on test tasks to measure the generalization performance of the algorithm on new tasks.

In Meta-Continual Learning (MCL), a continual learning prediction problem (CLP) is defined as a non-i.i.d. data stream, which is a task sampled from the stationary distribution p ( T ) . Random subsequences of dependent data points of length k are sampled from each CLP task for training. The major difference between CML and MCL settings is that the non-stationarity of CML comes from the sequentially encountered non-i.i.d. task distributions. In contrast, the non-stationarity of MCL comes from the non-i.i.d. data points within a task.

H.2 Continual Learning

Except for the taxonomy w.r.t different approaches mentioned in the main paper, CL can also be divided into the following three scenarios w.r.t the different properties of the task distributions [56, 57]. Class Incremental Learning (CIL) [58] involves learning new classes over time without having access to all classes at once during training. The label space is growing over time, where Y t ⊂ Y t +1 , ∀ t . In the final step, the predictor is aimed to be capable of classifying all the seen classes. Domain Incremental Learning (DIL) [59] defines on the same label space Y that has a fixed associated semantic meaning, and the marginal distributions on input space µ t ( X ) varies over time. Task Incremental Learning (TIL) [5] can define on different label spaces Y t = Y t +1 , ∀ t or the same label space Y associated to different semantic meanings over tasks. In conventional definitions, CIL and DIL are under a weak task-agnostic setting (the task identities or task boundaries are unknown during testing but known during training), while TIL is under the task-aware setting with known task identities. However, with increasing research diversities, TIL has also been studied in task-agnostic settings recently. In this paper, we study CML within a similar TIL scenario of CL.

glyph[negationslash]

Key Differences between CL and CML The two settings share the common objective of addressing the stability-plasticity dilemma, which is the typical objective for learning from non-stationary data. Their key differences are:

Continual Learning:

Standard CL methods sequentially adapt a task model and aim for a final model that performs well on all the tasks encountered. (Some memory-based approaches like dynamic architecture grow a subnet for each task.)

The current task model is adapted from the previous task model.

Standard CL reports the final performance of the agent on all the tasks at the end of its lifetime.

Standard CL approaches mainly focus on minimizing catastrophic forgetting without considering quick generalization for tasks with few-shot data, e.g., the replay-based and regularization-based methods.

Continual Meta-Learning:

CML methods sequentially adapt a meta-model and aim to recover the performance on previous tasks by adapting the meta-model with few additional samples. Therefore, it's more suitable for learning few-shot tasks. (Some memory-based approaches grow a subnet for each environment.)

The current task model is adapted from the meta-model.

CML reports the cumulative performance of the agent throughout its lifetime.

CML focuses on quick generalization for new tasks and fast recovering performance of previous tasks with few-shot data.

H.3 Meta-Continual Learning

Javed and White [60] and Gupta et al. [61] study a different Meta-Continual Learning (MCL) setting. A continual learning prediction problem (CLP) is defined as a non-i.i.d. data stream, which can be considered a CLP task T . The CLP tasks are sampled from a stationary distribution p ( T ) , where for each CLP task, random subsequences of dependent data points are sampled from the task for training. [60] conducts offline meta-representation learning w.r.t two objectives MAML-REP and OML. MAML-REP uses batch data for inner adaptation. OML is calculated with a single data point for each inner adaptation, which can reflect the degree of forgetting in CLP tasks. Gupta et al. [61] proposes La-MAML, which adopts a replay buffer and conducts meta-initialization with optimization on the OML objective in an online manner. In addition, La-MAML adopts a modulation of per-parameter learning rates in the meta-learning updates. The optimization of learning rates in La-MAML is w.r.t empirical loss, while our learning rates optimization is based on the excess risk upper bound. So, our approach suffers less from over-fitting.

H.4 Continual Meta-Learning

Static environments Harrison et al. [32] studies continual meta-learning (CML) under task agnostic setting, where the task boundaries are unknown. It proposed an algorithmic framework MOCA that incorporates different meta-learning methods with Bayesian Online Changing-point Detection (BOCD) to identify unknown task boundaries during the meta-learning process. The task environment is still assumed static. On the contrary, we focus on the bi-level trade-off in shifting task environments with known task boundaries, where we detect the environmental distribution shift, not the task distribution shift.

Shifting environments To address the environment shift, Jerfel et al. [14] proposes a nonparametric Dirichlet process mixture of hierarchical Bayesian models that allows adaptively adding new task clusters and selecting over a set of learned meta-initialization parameters for the newly encountered task. A similar approach is also used to detect the task shift in a task-agnostic setting. For both task-agnostic and task-aware settings, [14] has tested the proposed algorithm on datasets with two times environment shifts. The proposed method requires additional O ( K ) memory for

storing the initialization parameters for the detected K clusters. For each task, it needs to update the task-specific model and the meta-model for all the K initializations. K can grow with the time.

Zhang et al. [15] use a Dynamic Gaussian Mixture Model (DGMM) to model the distribution of the meta-parameters. Different from [14], which applies point estimation during inference, [15] derived a structured variational inference method that can reduce overfitting. The experiment is conducted with three times environment shifts of four datasets. [15] needs O ( K ) extra memory for storing the meta-parameters.

Wang et al. [16] studies a slightly different setting where each environment has a super long task sequence. In this case, the cartographic forgetting for meta-knowledge is more obvious. To address this, they use a memory buffer to store a small number of training tasks from previous environments and a shared representation for different task environments and grow the subnets when a new environment is detected on latent space using BOCD. For deciding which cluster the task belongs to, they store the average embedding for each environment for calculating the distances. Hence, its memory complexity is O ( K + M ) .

H.5 Meta-Learning

Statistical Learning to Learn (LTL) LTL [62] has been intensively studied with various theoretic tools. The early theoretical framework was proposed by Baxter [24], where they first defined the notion of the task environment and derived the uniform convergence bounds. And the tasks are i.i.d sampled from the environment. Denevi et al. [63, 28] provided excess risk for ridge regression and discussed the train-validation split impact. Amit and Meir [25] applied PAC Bayes theory and provided generalization bounds for stochastic neural networks. They also derived a joint training algorithm that simultaneously updates the meta-parameter and task parameters, which cannot be extended to the sequential learning scenario. Recently, Chen et al. [26] derived an informationtheoretic generalization bound for the MAML-like algorithm, which provides non-vacuous bounds for deep few-shot learning and can be applied to sequential task learning.

Statistical Lifelong Learning The difference between Lifelong learning [64] and LTL is that the tasks are observed sequentially, while LTL has all the tasks in hand. Pentina and Lampert [65] first applied PAC Bayes theory to Lifelong Learning, where they also assumed that all the tasks are, i.i.d. sampled from the task environment. In a subsequent work [66], they relaxed the i.i.d. assumption with two scenarios. The first is that the tasks are dependent, but a dependency graph is known so that they can prove a statistic bound. In the second, they also consider a shifting environment change. However, they assume that the base learner output of the current task only depends on the current and the previous task data, and the expected performance of the base learner does not change with time. Under these assumptions, they obtain statistical guarantees for the gradually changing environment.

Online LTL The online LTL methods have all the tasks in hand, but at each round t , they sample a task from the N tasks in hand. Cavallanti et al. [67] works on this setting for online multi-task learning. Multi-task learning is often treated as a simplified version of LTL in previous works [68].

Online Meta-Learning Two main settings to meta-learn sequential tasks in an online manner are referred to as online-within-online (OWO) [28-31] and batch-within-online (BWO) [9, 27]. The former applies online algorithms for both base and meta learners. The latter treats task-level learning differently as a statistical batch setting. The BWO setting is close to the CML studied in this paper, while the theoretical work is rare, and none of the previous BWO methods considers shifting task environments. Khodak et al. [31] first considered shifting environments in the OWO setting, where the base learner is the gradient-based online learner. In contrast, we consider more choices of batch base learners and a fine-grained algorithm w.r.t environment change. Even though their bounds are not w.r.t AER and the results are not directly comparable , we make an intuitive comparison, which suggests the proposed algorithm in this paper has an improved rate for gradient-based base learners in slot-wise stationary environments over their related results. Moreover, we conducted a rigorous analysis of the bi-level trade-off, which is missing in the related works mentioned above. Finally, although not clearly defined in previous meta-learning literature, we note Meta Continual Learning (MCL) [60, 61] can be named in a similar way as Online-Within-Batch (OWB), where the tasks are processed in one batch, but the data within each task are processed online.

Other related works Model-Agnostic Meta-Learning (MAML) [4] that uses the higher-order gradients for meta-updates has gained tremendous success in practice. Therefore, a lot of work emerges to improve MAML. Finn et al. [69], Grant et al. [70], Yoon et al. [71] combine MAML with Bayesian methods. Rajeswaran et al. [72] improves MAML with implicit gradient calculation for the meta-updates. Since meta-learning considers learning and generalization for both seen and unseen tasks, it has not only been applied to CL for addressing catastrophic forgetting but has also been applied to Test-Time Domain Adaptation (DA) for quick generalization. For more related works in Test-Time DA or DA, please refer to [73-75].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the balance between stability (preserving past knowledge) and plasticity (rapid learning from new experiences) in Continual Meta-Learning (CML). The authors propose a novel theoretical framework for CML in both static and shifting task environments. They discuss the bi-level (task- and meta-) trade-off in shifting environments and theoretically identify factors that affect stability and plasticity. They then propose a novel algorithm, Dynamic Continual Meta-Learning, that addresses the bi-level trade-off in shifting environments. DCML minimizes the expected regret by dynamically adjusting the meta-parameter and its learning rate when an environment change is detected.\n\nSTRENGTHS:\n1. The paper is theoretically robust, proposing a unified framework for understanding CML in both static and shifting task environments.\n2. The paper is the first theoretical work studying the stability-plasticity dilemma in CML with bi-level shifts.\n3. Based on the theoretical findings, the authors introduce a novel and practical algorithm which show improved estimated bounds and superior performance compared to baselines.\n\nWEAKNESSES:\n1. From a non-expert perspective, the practicality and necessity of the problem the paper addresses are unclear to me. While meta continual learning is understandable (non-stationarity comes from the non-i.i.d data points within a task), it's not clear why continual meta-learning (non-stationarity comes from the task changes for each time step) is a practical scenario. More real-world examples or explanations of why continual meta-learning is required would be beneficial.\n2. The paper lacks a clear, detailed explanation of the robustness of the proposed algorithm. For instance, it's unclear how to set parameters such as a, b, and kappa for DCML. Additionally, the impact of the initial meta parameters (beta) on the model's behavior is not well-explained.\n\nQUESTIONS:\n1. Could the authors provide more details on how the algorithm dynamically adjusts the meta-parameter and its learning rate when an environment change is detected? Also, it would be helpful to see how beta actually moves during training.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a continual meta-learning setting with both static and dynamically changing environments and seeks to study the trade-off between learning and forgetting. It derives an upper bound on the excess risk, i.e. the expected gap between the true risk of the predictor and the optimal true risk and uses the average excess risk across tasks as a meta-objective to develop an algorithm that can meta-learn online. The resulting algorithm is empirically evaluated on a synthetic task and a subset of the OSAKA continual meta-learning benchmark.\n\nSTRENGTHS:\nThe paper derives novel upper bounds on the average excess risk across a number of methods.\n\nWEAKNESSES:\n- Despite its framing, the general approach of using a bound to derive a meta-objective is not completely novel and related work should be more prominently referenced (e.g. [1]).\n\n- The method introduces many hyperparameters and it is unclear where improvements result from. An empirical verification of the theory in the considered settings is missing and ablations on different components of the algorithm that are not necessarily prescribed by the theory (e.g. annealing $\\gamma_t$) are missing.\n\n- The presentation of the paper needs to be improved by better motivating and contextualizing theorems and ensuring that all symbols are defined in the main text.\n\n[1] Amit, Ron, and Ron Meir. \"Meta-learning by adjusting priors based on extended PAC-Bayes theory.\" International Conference on Machine Learning. PMLR, 2018.\n\nQUESTIONS:\n1. Could you please comment to what extent the assumption of $\\alpha$-quadratic-growth in Theorem 3.1 is justified in the context of the loss of neural networks and the settings considered empirically?\n2. In Figure 2b is the plotted AER computed from the equation in Theorem 5.1 or is it measured independently? I am asking to understand whether this entails an empirical verification of the theory or a visualisation of it.\n3. In Figure 2b BOCD only detects few change points and seems to do so mostly incorrectly. Could you clarify where your algorithm uses these? The plot over the \"meta lr\" suggests that the ground truth change points are used and neither \"oracle\" nor \"window\" rely on empirically detected change points.\n4. Why does \"window\" perform better than \"oracle\" in Figure 3? Would we not expect the opposite?\n5. In line 358-359 you state \"MAML and ANIL do not suffer forgetting since they never update meta-parameters.\" In light of this, why does their performance vary over time in Figure 4? Does this also hold true for Figure 3? If so have they been pre-trained for the Moving 2D Gaussian task?\n6. In Figure 4, why is there already such a clear gap between algorithms at $t=0$?\n7. How are the hyperparameters tuned? Specifically how have they been tuned for baselines for the modified OSAKA tasks? Could you please report these details?\n\nLIMITATIONS:\nLimitations have been addressed appropriately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically studies the stability and plasticity dilemma in Continual Meta-Learning (CML). \n\nThis paper formulates the CML objective as selecting u_t at each time t to ensure a small Average Excess Risk (AER) upper bound for the task sequence. Based on the AER objective, it proposes a continual meta-learning framework in both static and shifting environments.\n\nThe proposed Dynamic Continual Meta-Learning (DCML) algorithm can quickly reconstruct meta-knowledge to alleviate forgetting and fast adapt to new environments when change occurs. \n\nThe corresponding theory provides tighter bounds and more flexible base learner selections. \n\nExperiments demonstrate the superiority of the proposed method.\n\nSTRENGTHS:\nThis paper first introduces a novel and unified theoretical framework for CML in both static and shifting task environments.\nAdaptively learning the meta-parameter can facilitate the training process in deep learning. \n\nThe paper is written and organized well.\n\nThe source code is provided.\n\nThe paper has good and relevant citations.\n\nThe results are quite extensive and convincing.\n\nWEAKNESSES:\nThe paper lacks a clear figure that contextualizes the base model, learner, and how continual meta-learning fits into the setup. This visual representation would enhance understanding for readers.\n\nQUESTIONS:\nCan you discuss what applications this paper will have to memory-based approaches and what theoretical explorations can be done in the future?\n\nLIMITATIONS:\nThe proposed theory works for non-convex loss when using Gibbs algorithm. However, this paper does not offer an analysis for SGD with non-convex loss.\n\nThis paper does not provide theoretical guarantees for memory-based approaches, which can better address the meta-level forgetting with additional memory cost.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors present a framework for continual meta-learning, that is the training of meta-learning models through exposure to a sequence of tasks. The paper presents a formalism for the setting which allows for non-stationary task distributions.
Theoretical analysis is provided both for the framework and for the proposed method, and experimental validation is reported for a reasonable set of continual learning tasks.

STRENGTHS:
- The problem setting is interesting, and the work is thorough.
- The experimental verification is convincing.

WEAKNESSES:
- I am not sure if the comparison with non-continual meta-learning method is fair.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper presents a comprehensive theoretical framework for continuous meta-learning in both static and shifting task environments. It also conducts a theoretical analysis of the bi-level trade-off in shifting environments. Furthermore, the paper proposes a method to adapt meta-parameters and the corresponding learning rate.

STRENGTHS:
1. The main contribution of the paper is the theoretical analysis of continual meta-learning, particularly in the shifting task environment section, which illustrates the concept of meta-level forgetting.
2. The paper provides theoretical evidence of the task-level trade-off and meta-level trade-off in continual meta-learning, offering valuable insights for optimizing the sequence of meta-parameters.
3. The paper introduces a novel AER objective and presents the DCML algorithm, which effectively mitigates forgetting and accelerates adaptation to new environments.

WEAKNESSES:
1. In my opinion, both continual meta-learning and continual learning share the common objective of addressing the stability-plasticity dilemma, albeit in different settings.  What is the key difference between them? And are there any real applications where the settings of continual meta-learning are more suitable?
2. It is unclear about the training efficiency compared to other methods.

QUESTIONS:
1. See Weakness 1 
2. To better evaluate this method, it would be valuable to provide a comparison of the computation costs with other existing methods.

LIMITATIONS:
adequately addressed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07