Task-Agnostic Online Meta-Learning in Non-stationary Environments
Authors: Daouda Sow
Sen Lin
Yingbin Liang
Junshan Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: online meta-learning
domain shift
dynamic regret
out of distribution detection

Abstract: Online meta-learning has recently emerged as a marriage between batch meta-learning and online learning, for achieving the capability of quick adaptation on new tasks in a lifelong manner. However, most existing approaches focus on the restrictive setting where the distribution of the online tasks remains fixed with known task boundaries. In this work we relax these assumptions and propose a novel algorithm for task-agnostic online meta-learning in non-stationary environments. More specifically, we first propose two simple but effective detection mechanisms of task switches and distribution shift based on empirical observations, which serve as a key building block for more elegant online model updates in our algorithm: the task switch detection mechanism allows reusing of the best model available for the current task at hand, and the distribution shift detection mechanism differentiates the meta model update so as to preserve the knowledge for in-distribution tasks and quickly learn the new knowledge for out-of-distribution tasks.
Motivated by the recent advance in online learning, our online meta model updates are based only on the current data, which eliminates the need of storing previous data as required in most existing methods. This crucial choice is also well supported by our theoretical analysis of dynamic regret in online meta-learning, where a sublinear regret can be achieved by updating the meta model at each round using the current data only. Empirical studies on three different benchmarks clearly demonstrate the significant advantage of our algorithm over related baseline approaches. 

Under review as a conference paper at ICLR 2023
TASK-AGNOSTIC ONLINE META-LEARNING IN NON- STATIONARY ENVIRONMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Online meta-learning has recently emerged as marriage between batch meta- learning and online learning, for achieving the capability of quick adaptation on new tasks in a lifelong manner. However, most existing approaches focus on the re- strictive setting where the distribution of the online tasks remains fixed with known task boundaries.
this work we relax these assumptions and propose a novel al algorithm for task-agnostic online meta-learning in non-stationary environments. More specifically, we first propose two simple but effective detection mechanisms of task switches and distribution shift based empirical observations, which serve as a key building block for more elegant online model updates in our algorithm: the task switch detection mechanism allows reusing of the best model available for the current task at hand, and the distribution shift detection mechanism differentiates the meta model update so as preserve the knowledge for in-distribution tasks and quickly learn the new knowledge for out-of-distribution tasks. Motivated by the recent advance in online learning, our online meta model updates are based only on the current data, which climinates the need of storing previous data as required in most existing methods. This crucial choice is also well supported by our theoretical analysis of dynamic regret in online meta-learning, where sublinear regret can be achieved by updating the meta model at each round using the current data only. Empirical studies three different benchmarks clearly demonstrate the significant advantage of our algorithm over related baseline approaches.
INTRODUCTION
Two key aspects of human intelligence are the abilities to quickly learn complex tasks and continually update their knowledge base for faster learning of future tasks. Meta-learning (Koch et al.. 2015) Ravi & Larochelle| 2016)] Finn al.) 2017) and online learning (Hannan
& Singer] 2007; Cesa-Bianchi & Lugosi 2006) are two main research directions that try to equip learning agents with these abilities. In particular, meta-learning aims to facilitate quick learning of new unseen tasks by building a prior over model parameters based the knowledge of related tasks, whereas online learning deals with the problem where the task data is sequentially revealed to learning agent. To achieve the capability of fast adaptation online meta-learning (Finn et al. 2017| Harrison et al. 2020] Yao et al. (020) has attracted much attention recently. Considering the setup where online tasks arrive one at time, the objective of online meta-learning is to continuously update the meta prior based on which the new task can be learnt more quickly after the agent encounters more tasks.
tasks in lifelong manner,
In online meta-learning, the agent typically maintains two separate models, i.e., the meta-model to capture the underlying common knowledge across tasks and the online task model for solving the current task in hand. Most of the existing studies (Finn et al. 2017: Acar et al., (021) in online meta-learning follow "resetting" strategy: quickly adapt the online task model from the meta model using the current data, update the meta model and reset the online task model back to the updated meta model at the beginning of the next task. This strategy generally works well when the task boundaries are known and the task distribution remains stationary. However, in many real-world data streams the task boundaries are not directly visible to the agent (Rajasegaran et al., 2022 Caccia ett al. 2020; Harrison et al.) 2020), and the task distributions can dynamically change during the online learning stage. Therefore, in this work we seek to solve the online meta-learning problem in such more realistic settings.

---

Under review as a conference paper at ICLR 2023
Needless to say, how to efficiently solve the online meta-learning problem without knowing the task boundaries in the non con-stationary environments is nontrivial due to the following key questions: (1) How to update the meta model and the online task model? Clearly, the "resetting" strategy at the moment of new data arriving is not desirable, as adapting from the previous task model is preferred when the new data belongs to the same task with the previous data. On the other hand, the meta model update should be distinct between in-distribution (IND) tasks, where the current knowledge should be preserved, and out-of-distribution tasks (OOD), where the new knowledge should be learnt quickly. (2) How to make the system lightweight for fast online learning? ? The nature of online meta-learning precludes sophisticated learning algorithms, as the agent should be able to quickly adapt to different tasks typically without access to the previous data. And dealing with the environment non-stationarity should not significantly increase the computational cost, considering that the environment could change fast during online learning.
The main contribution of this work is novel online meta-learning algorithm in non-stationary environments without knowing the task boundaries, which appropriately addresses the problems above. More specifically, we first propose two simple but effective mechanisms to detect the task switches using the classification loss and detect the distribution shift using the Helmholtz free energy (Liu et al., 2020), respectively, as motivated by empirical observations. Based on these detection mechanisms, our algorithm provides finer treatment the online model updates, which brings in the following benefits: (1) (task knowledge reuse) The detection of task switches enables our algorithm to reuse the best model available for each task, avoiding the "resetting" to the meta model each step as in most previous studies; (2) (judicious meta model update) The detection of distribution shift allows our algorithm update the meta model in a way that the new knowledge can be quickly learnt for out-of-distribution
tasks whereas the previous knowledge can be preserved for in in-distribution tasks; (3) (efficient memory usage) Motivated by the advance online learning (Mokhtari et al.. 2016) Hazan et al., 2016) where updating the model online with the current data is sufficient to guarantee sublinear regret, our algorithm does not reuse/store any of the previous data and updates the meta model at each online episode based only on the current data, which clearly differs from most existing studies (Finn et al. [2019] Yao et al. (2020; Rajasegaran et al. (2022) in online meta- meta-learning. This design choice is also well supported by our theoretical analysis which shows that updating the meta model at each round with only the current data can lead to desirable sublinear dynamic regret growth. Extensive experiments in three different benchmarks clearly show that our algorithm significantly outperforms existing methods.
Related Work: Meta-learning. Also known as learning to learn, meta-learning (Finn al. 2017 Vinyals et al. 2016: Li et al. 2017) is a powerful tool for leveraging past experience from re related tasks to quickly learn good task-speeific models for new unseen tasks. As a pioneering method that drives recent success in meta-learning, model-agnostic meta-learning (MAML) (Finn et al. 2017) seeks to find good mnta-initialization such that one or few gradient descent steps from the meta-initialization leads to good task- model for a new task. Several variants of MAML have been introduced (Finn & Levine 2018 Finn et al., 2018
Raghu et al.] 2019 Rajeswaran et al. 2019) 2019 Zhou et al., 2019). Other approaches
Nichol & Schulman| 2018 Nichol et al. 201 8 Mi et al.
are essentially model based (Santoro et al. 2016 Bertinetto et al. 2018 Ravi & Larochelle 2016
Munkhdalai & Yu 2017) and metric space based (Koch et al. 2015 Vinyals et al.
2016 Snell et al.
2017 Sung et al.] 2018).
Online Learning. In online learning (Hannan 1957
& Lugosi 2006 Hazan et al.
2007], the cost functions are sequentially revealed to an agent which is required to select an action before seeing each cost. One of the most studied approach is follow the leader (FTL) (Hannan 1957).
which updates the parameters at each step using all previously seen loss functions. Regularized versions of FTL have also been introduced to improve stability (Abernethy et al., 2009 Shalev
Shwartz et al. 2012). Similar in spirit to our work in terms of computational resources, online gradient descent (OGD) [Zinkevich] 2003) takes gradient descent step at each round using only the revealed loss. However, traditional online learning methods do not efficiently leverage past experience and optimize for zero-shot performance without any adaptation. In this work, we study the online meta-learning problem, in which the goal is to optimize for quick adaptation on future tasks as the agent continually sees more tasks.
Continual Learning. Continual learning (CL; a.k.a lifelong learning) focuses on overcoming
"catastrophic forgetting" (MeCloskey & Cohen 1989 Ratelifi, 1990) when learning from a sequence of non- con-stationary data distributions. Existing approaches rehearsal- based (Lopez-Paz & Ranzato
2

---

Under review as a conference paper at ICLR 2023
Chaudhry et al. 2018 Riemer al. 2018), regularization-baset (Kirkpatrick et al. 2017 Aljundi et al.] 2018), and expansion-based (Rusu et al., 2016) Yoon et al., 2018 Sarwar et al. 2019). For instance, rehearsal- for experience "replay" to avoid forgetting. However, traditional CL. methods evaluate the final model on all previously seen tasks online meta-lcarning and evaluate models with the average online performance (c.g., accuracy) after adaptation, which better captures the ability to quickly adapt to new online tasks (Caccia et al. 2020). Even though we do not specifically focus on avoiding forgetting, we update the meta-model in way that preserves knowledge of in-distribution domain while also improving fast adaptation for nut-of-distribution domains, as demonstrated in our various experiments.
2017:
methods store subset of previous tasks data and reuse
to measure forgetring. In this work we are interested in
Online Meta-learning. Online meta learning was first introduced in Finn et al. (019)). Pioneering methods (Finn et al. 2019, Yao et al. [020 Yu et al. 2020) follow a FTL--like design approach, which requires storing previous tasks and leads to linear growth of memory requirement. Follow- the-regularized-leadee (FTRL) Shalev-Shwartz
al. 2012) approach has also been extended to
the online meta learning setting in (Balcan et al. 2019 Khodak et al. 2019), resulting in a better memory requirement. Acarr al. (2021) proposed memory-effictee approach based on summarizing previous task experiences into one state vector. However, these approaches require knowledge of task boundaries and "reset" the task model to the meta model each online episode (Finn al., 2017]] Denevi et al. (019). Similar to |Acar et al. (202T), our algorithm also overcomes the linear memory scaling. But unlike their method, our algorithm does not have access to task boundaries and can operate under dynamic environments. The method in Rajasegaran et al. (2022) tries to alleviate the "resetting" issue by updating the online model always starting from its previous state, which however needs to store previous models and has limited performance especially in dynamic environments where successive tasks can be very different.
None of the methods above considered the online meta-learning problem in a dynamic environment setting where the task distributions change substantially over time without knowing the task bound- aries. [Caciia et al.] (2020) is the first work that empirically evaluated the proposed algorithm in a dynamic environment, but did not propose a method to quickly learn the knowledge for out-of- distribution tasks. In stark contrast, we update the meta representations in a way that preserves the in-distribution knowledge while continually improving fast adaptation for out-of-distribution tasks.
2 BACKGROUND AND PROBLEM FORMULATION
Background. Before introducing the online meta-learning problem in non-stationary environments, we first briefly discuss some related concepts.
Meta-learning via MAML. Meta- Meta-learning (Finn et al., 2017 7 Vinyals et al. 2016 Li et al. 2017 7).
a.k.a., learning to learn, seeks to quickly learn a new task with limited samples by leveraging the knowledge from similar tasks. More specifically, the objective therein is to learn a meta model based on set of tasks (7.).M, drawn from some unknown distribution (T), from which task -secifpei models can be quickly obtained for new tasks from the same stationary distribution P(T). Taking MAML as example, the objective therein is to learn model initialization @ such that one few gradient descent steps from 0 can lead to a good model for new task T PPT),, by solving the following optimization problem with training tasks (T.)
1 M
ΣM f: (T)())
0 arg min 0 M
(1)
where the task model Φ; = (,) = a f()) f. and f, correspond to the training and test losses
for task T. respectively.
Online learning. In the general online learning problem, loss functions sequentially revealed to learning agent: at each step f, the agent first selects an action 6,, and then a cost fe())) is incurred. The goal of the agent is to select a sequence of actions so as to minimize the following static regret
T R(T)= ΣT fe (0.) min Σ 1= (0).
i.., the gap between the agent's 's predictions (f) and the performance of the best static model in hindsight. A successful agent achieves a regret RTT) that grows sublinearly in T. Online learning well studied field and refer the interested readers to Hazan et al. (2016) for more information.
3

---

Under review as a conference paper at ICLR 2023
Online meta- -leannie.nnnn..... As a marriage between online learning and meta-leaming, online meta-learning Finn et al. 2019; Yao et al. 2020; Harrison et al. 2020) aims to achieve the following two features: (1 last adaplation to the current task (the meta- meta-learning aspect); (ii) learn to adapt even faster as it sees more tasks (the online learning aspect). Specifically, the agent observes a stream of tasks S = (Ti, T2, ..., Tr) sampled from P(T), where tasks are revealed one at time. For each task Ti, the agent has access to support set St for task task-specific adaptation and query set Q: for evaluation. say The goal here to select sequence of meta models (W,) for achieving sublinear growth of the following regret
Raneta (T) = E Sa (U) min Σ St (U.(()
(3)
where Ue is the task adaptation function depending the support set S., and the cost function ft is evaluated using the adapted parameters Ug (We) on the query set Qt. Intuitively, the agent seeks to learn better meta model which leads to better task models for future tasks after seeing more tasks.
Online meta-learning in non-stationary environments. Differently from most online meta- mearnng studies (Finn et al.] [019 Yao et al., 2020; Yu et al. 22200] Harrison et al. 2020 Rajasegaran et al. 2022 Acar et al. 2021). in this work we consider the online meta-learning problem in a more realistic
scenario:
Pre-trained meta model. In many real applications, there is plenty of data available for pre-train
and it is unrealistic to deploy an agent in complex dynamic environments without any basic knowledge of the tasks at hand (Caccia et al. 2020). Therefore, following the same line as in Caccia et al. (2020), we assume that there is a set of training tasks (7,0), J==1 drawn from some unknown distribution (T) And as standard in meta-leaming each pre-training task 7, has a support dataset S and query dataset º. In this work, we employ MAMI. over the training tasks to learn a pre-trained meta model.
Unknown task boundaries. During the online meta-learning phase, we assume that the task boundaries are unknown, i.e., the so-called task-agnostic setup (Caccia et al.) 2020), in the sense that the agent does not know if the new coming data at time & belongs to the previous task or new task. To model the uncertainty about task boundaries, we assume that at any time t the new data belongs to the previous task with probability E (0, 1) or to new task with probability 1 1-11 p.
Non-stationary task distributions. During the online meta-learning phase, the agent could encounter new tasks that are sampled from other distributions instead of the pre-training one (T). To capture this non- statiinatin in task distribution, we assume that whenever a new task arrives during online learning, it will be sampled either from PO with probability 1) E (0, 1) from new (w.r.t. (T)) distribution with probability 1 - n/. Note that we do not restrict the number of new distributions that can be encountered during online learning and the task distributions can be revisited.
PROPOSED ALGORITHM UNDER DISTRIBUTION SHIFTS
To address the online meta-learning problem mentioned above for non-stationary environments, next propose a simple but effective algorithm, called onLine mËta lEarning under Distribution Shifts (LEEDS), based on the detection of task switches and distribution shift assist fast online learning. Following most studies (Rajasegaran et al., 22022; Caccia et al. (020)) in online meta-learning, we maintain two separate models during the online learning stage: 0 for the meta model and ó for the online task model.
Detection of task switches and distribution shift: learning, it is clear that the detection mechanisms can not be overly sophisticated, but in the meanwhile have to be efficient with high detection accuracy. Towards this end. we propose two different methods for detecting the task switch and the distribution shift, respectively, which work in concert as key components of LEEDS.
To enable fast learning of a new task in online
Detection of task switches. To understand the learning behaviors under task switches, we evaluate the classification loss of the previous task model using the newly coming data, i.e., L(Ø.-1 S,) for time f, where L is the loss function, d, is the previous online model at time t 1, and S, is the current support set. The left plot in Fig. shows the empirical results on an online few-shot image recognition problem. As depicted, the loss value keeps decreasing as the agent receives more data from the same task but suddenly increases whenever a new task arrives. This is clearly reasonable as the learnt online model for the previous task does not fit the new task anymore. Inspired by this
4

---

Under review as a conference paper at ICLR 2023
empirical observation, task boundaries: there a task switch whenever the loss is above some pre-defined threshold. As demonstrated later in Section [ such simple mechanism is indeed quite effective corroborated by its high detection accuracies on various online meta-learning problems.
simple mechanism based the value of L(0r-1; St) to detect the
Detection of distribution shift. To efficiently determine if task is IND or OOD, i.e., sampled from the pre-trate pre-training task distribution or not, we consider an energy-based OOD detection mechanism with binary classifier Cr (=; defined as follows
E(x; 8) S C+(x;0) 0 --E(x: >
(4)
where E(x; 0) -olog Ek=1 exp (-9k (x: 0)/6) corresponds to the Helmholtz free energy for input (s 0) is the k-th component of the prediction model's output, and ö is the temperature parameter: The hyperparameter t a threshold that can be set using the pre-training tasks. As shown in Liu et al. (2020), the negative energy is linearly aligned with the likelihood of the of input under model 0, making it a useful score for distinguishing IND and OOD tasks.
Loas 'tt task zxdtcc
Method
Memory
Ours
O[I
Rajasegaran et al. (2022)
O(7)
Caccia et al. (2020)
O(1/p)
Acar et al.
2021)
O(1)
Finn et al.
2019
O(T)
Figure 1: Left plot: Variations of the online loss for pre-trained meta model using MAML which deployed for online learning. Red dot at means no task switch at that time, and at means the task switched at that time. Right table: Comparison of the memory requirements among different methods. T is number of online rounds and p E (0, 1) is non-stationarity level.
Update of meta and online parameters: Based on the two detection schemes, the next question how to update the meta and task models accordingly for enabling fast adaption in dynamic environments.
Without such detection mechanisms, previous online meta-learning algorithms (Finn et al. 2019 Acar et al.] 2021) typically adapt the task model from the meta model using support set, evaluate the adapted model on a query set, and reset the task model to the meta model when new data is received from the online data stream process, and then repeat the process again. However, such a "resetting" scheme can be sub-optimal in realistic scenarios. For instance, the newly received data belongs to the same task with the previous data, the agent should update the task model by starting from previously adapted parameters instead of from the meta model.
In contrast, the simple but effective detection mechanisms in this work enable more elegant treatment to the knowledge update during online learning:
(1) If there is task switch at time I, i.e., L(0--1; S.) > l where e is the threshold, adapting from the meta model is generally better than adapting from the task model of the previous task. Therefore, we first obtain the online task model o, from the meta model using the new data:
Oadapt = f,- a1 V C(0: St).
and then update the meta model no matter if there is a distribution shift, so as to incorporate the knowledge of the new task to the meta model:
0. = 0,-1 02 V V (0Ddaaps Qt).
(2) If there is no task switch, i.e. L(Ø,
;;) St) < f, we continue update the task model from the previous task model using the new data, different from the "resetting" scheme the literature:
Φ: 1 an V のr 1(0
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 onLine mEta lEarning under Distribution Shifts (LEEDS)
1: Input: Dynamic stream S. pre-training distribution 1o (T), stepsizes on and c2,, thresholds i and
2: Perform pre-training phase using MAML method on tasks drawn from (T)
3: while stream S is ON do
4: D. S // receive current data from online data stream St, Q+
5:
St 1,,0++= -
D: I/ split data into support and query
6:
(() v (i.c., no switch) then
7:
de ===00 Ph-1 - on o L(o--1; Se) adapt starting from previous online model
8:
Evaluate 6r query set Q:
9:
if C, (Sr; 11 (i.e., covariate shift) then
10:
Badapt 06-1 ai VoL(0r-1;;
11:
0: 0= 01 VoL((nnapp Q.) I/ update meta model
12:
end if
13:
else
14:
Oadaps 01-1 01 Vo,-, ((0,-1; S.) // adapt starting from meta model using support set
15:
Set Φr = Oadapt and Evaluate Φr. on query set Q:
16:
0: 0.-1- 02 V VoL Qc) // update meta model
17:
end if
18:
end while
To accelerate the knowledge learning for the new domains, we further distinguish the meta model update for IND and OOD tasks. In particular, if the current task is IND task, we will only update the meta model once at the beginning of this task. That to say, the meta model will not be further updated within the same task. In stark contrast, if the current task is an OOD task, we continue to update the meta model whenever new data for this task arrives as follows:
adapt = - an Vo 8, = 8 02 Vo (fadapti Qt).
Memory friendly: One important feature of the proposed algorithm LEEDS is that the meta model update is only based the current data, motivated by the theoretical advance in online learning (Mokhtari et al. 2016 Hazan et al.) (0166) where updating the online model with only the current data is sufficient guarantee a sublinear regret. This design is also well supported by our theoretical analysis in the following section, which indicates that updating the meta model with the current data also sufficient for online meta-leantn veta-learning achieve a dynamic regret that grows sublinearly. As comparison, most of the previous studies store the previous data in the memory for the meta model update. A comparison of the memory requirements among different approaches is summarized in the right table in Fig. [
THEORETICAL RESULTS
In the online meta-learning with distribution shifts, it is clear that the static comparator in eq. (3) is not sufficient to capture the non-stationarity, as one cannot expect to have a single meta-model for all task distributions. Hence, we consider the following more sophisticated dynamic regret (Mokhtari et al. 2016 Hazan al. 2016 Zinkevich 2003)
Roneta (Z), -.., ZT) Σ'_, Si (O.) E (U.())))
(5)
where the meta comparator 2, can change over time. If zt arg arg ming fe (U, (8)), then eq. 5 becomes the worst-case dynamic regret formulation. To make the dynamic regret in eq. (5) more suited to the realistic scenario of online meta-learning in which the task distribution changes after some time-steps, we consider the setting where the d-th encountered distribution stays stationary for K tasks before the distribution shift, leading to the following dynamic regret:
D Ka
Reetta (Bi...,
"i........ 85)
f
(Ut
(0%)
min
fh (U)
1=1
K2
ΣΣ k=
t (u: (0:))
fá (U
(6)

---

Under review as a conference paper at ICLR 2023
Acuncy damall (14667)
Online evaluations in each of the encountered domains during online learning phase for the Omiiglo-MNIST--FasiionMM
Figure 2:
benchmark. First row corresponds to statiintton level 0.9. In second row p = 0.75. LEEDS is the only method that is able to preserve pre-training knowledge while substantially increasing performance OOD domains. Legend in first plot only.
where the meta comparator 8 = arg ming Eke få (U). Note that the task distributions can be revisited, i.e., the i-th distribution could be the same as the (i j)-th distribution but distinct to the (i - 1) for 1.
To show that updating the meta model based only the current data (which is consistent with algorithm) is sufficient for achieving sublinear regret, consider the following update rule which updates the meta model at each step based on the current task only without reusing any previous data:
g = N F ( (13()
with 0'41 = 000 0K
And we make the following standard assumptions: Assumption 1.
Each function f has bounded gradient norm, i.e., V (w) SGAN
Assumption 2. Each composition function få o U convex.
We can have the following theorem which shows that updating the meta model based only on the current task without storing any previously data achieve sublinear growth of the regret formulation in
Theorem 1.
Suppose that Assumptions
andd hold. Let PD = Eb= 10 -02 1l1 and at
Then the dynamic regret is bounded as Rmota (07, ".":...,')) S O PDy /
05)
Ed Ka
VED Ka
Further, if of
V
Po Ed K.²
then Rimeta (0) (.......0))))) 0 S PD Ia
Ka
We have the following remarks on Theorem (a) Theorem shows that under Assumptions I and 2
the dynamic regret associated with the algorithm defined in eq. 7 grows sublinearly with respect to
the total number of steps Ed= Kg. This implies that the gap between the average loss (per task) of online models and that optimal models in hindsight is asymptotically decreasing as more tasks observed. (b) The dynamic regret depends on PD (see its definition in the statement of Theorem [], which captures the accumulative model shift (that further reflects the domain distribution shift over time). Clearly, smaller shift implies lower regret over time. (c) The dependency on Pp also shows that the upper bound becomes automatically tighter when the model shift across domains becomes small (i.e., dynamic comparators are close to each other), and thus the upper bound is adaptive. (d) Comparing the two regret bounds obtained in Theorem T we note that the dependency of the dynamic regret Pp can be improved by appropriately selecting a when Pp is known advance.
7

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTS
We conduct extensive experiments to answer the following questions: (i) How our algorithm perform when compared to existing online meta-learning methods for dynamic and static environments? (ii) How certain characteristic of the environment such as the non-stationarity level affects the performance of the algorithms? How our detection schemes perform when deployed realistic dynamic environments. Further, we conduct ablation studies to investigate the advantage of the key domain adaptation module that permits our algorithm process differently IND and OOD tasks. More comprchensive experimental results are provided in Appendix B due space limitation.
Experimental setup. We pre-train the meta model in one domain and then deploy it in dynamic environment where tasks can be drawn from new domains. We evaluate all the algorithms using the average test losses obtained throughout the entire online learning stage. To investigate the impact of the non-stationary level on the learning performance, further consider two different cases of the environment non-stationarity: A moderately stationary case where the probability of not switching p 0.9, and low stationary where p 0.75. We do consider the
task is cases where p is very small, as an algorithm that just assumes task switches at each round should perform well in such cases. We compare algorithms over 10000 episodes unless otherwise stated. Due to space limitation, we defer details about datasets and baselines in Appendix C
For all the experiments, whenever a new task needs be revealed, it will be drawn from either the pre- traininnnng domain with probability 0.5, or from one of the OOD domains with probability 0.5. For the Hhered-Imageha distribution, we increase the sampling probability of ood2 to 0.5 which is consistent to the protocol 50% - 50% for IND and OOD tasks in all our experiments. More details about the experimental setup including the neural network architectures and the hyperparameter search are deferred to Appendix D
dataset, because only ood2 is trully OOD with respect to the pre-training task
domaino
Accuracy on all domains
17
Accuracy
Accuracy on OOD domains
Figure 3: Online evaluations for the Tiered- Sorred-ImageNet (TI) and Synbols (SB) benchmarks under p 0.9. First columns correspond to TI and second column to SB. More results including performance each domain and under different p can found Appendix B Legend shown first plot only.
5.1 MAIN RESULTS
(OMF). The online evaluations of the compared methods are shown in fig. for non- conn-ttttionayyyy levels 0.9 and 0.75. For each setting we report separately the online accuracies on pre-training domain and on the other two OOD domains, to show how our method keeps improving on the OOD domains while also remembering the pre-tarining tasks. As shown the plots, Our method LEEDS achieves superior performance compared to all other baseline algorithms in both settings. More specifically, on the IND domain all methods pre-trained using MAML perform similarly, but are outperformed by LEEDS and CMAML+ which can detect task boundaries. However, the OOD domains baselines, including CMAML++ This is due to the key OOD adaptation module that allows LEEDS to dynamically adapt the meta model based the task distribution. Interestingly, comparing the
Results on Muniglot-MNIST--aahhiii
algorithm significantly outperforms all other
8

---

Under review as a conference paper at ICLR 2023
performance for MAML and ANIL provides some insights on the limitations of re-using pre-traines trained representations in non non-stationary environments. In fact, the ANIL baseline, which does not adapt its inner representations, performs poorly compared to MAML on the OOD domains, but achieves similar results on the pre-training domain. Also, the results highlight some limitations of the recently introduced FOMI, (Rajasegaran al., 2022) method, which achieves lower performance than other competitive baselines. This is because FOML requires the tasks to be not mutually exclusive, which may not hold for the standard few-shot benchmarks considered in our experiments.
Results on Tiered- -mageNee (TT) and Synbols (SB) benchmarks. We report the online accuracies Due to space limitation, results for each domain are deferred to appendix B Because the distribution of the pre-training tasks similar to the OOD ones for the Tiered- Srrred-ImmgeNeet benchmark, methods such as MAMI. can perform reasonably well. In fact, in the lower non-stationary case (p 0.75, can be found in Appendix (B), MAML is able to outperform the more complex CMAML++ baseline. However, our algorithm still achieves the best performance under both non-stationary levels and in both benchmarks. Note that in the larger TI dataset case, the FOML algorithm, which stores all previously seen tasks, runs out of memory after around 6500 online episodes. Again because of similarity between OOD and IND tasks in the TI benchmark, static representations learned by ANIL are useful for all domains.
on all domains and on OOD domains for these two benchmarks in fig. 3
5.2 ABLATION STUDIES
Task boundaries detection. The table in the right of fig. provides the precision and recall scores of the task switch detection schemes for our method and CMAML++ Our detection scheme outperforms that CMAML++ all metrics. This because, the detection scheme in CMAML++ is based on comparing successive losses, which could lead to over detection of task boundaries, especially when the task loss is too high at the first time the task is revealed to the online algorithm.
Method
Prec.
Rec.
LEEDS
96.6
99.27
TT
CMAMI+
90.93
92.98
LEEDS
91.81
99.37
SB
CMAML+
87.09
93.89
Figure 4: Left: LEEDS under different p. Center: LEEDS with and without domain adapta- Right Task boundaries detection on Wiered-ImageNe (TT) and Synbols (SB).
Importance of domain adaptation module. We investigate the importance of the distribution shift detection module that allows our algorithm LEEDS to update the meta model differently for in-distribution and nut-of-distribution tasks. Fig. 4 shows the performance of our algorithm with and without the distribution shift detection module. The performance of the algorithm significantly improves (~~ 4.3% improvement) with this module. This shows that such a simple mechanism can effectively boost the online learning performance by allowing the agent to learn more from OOD data while also remembering pre- pre-training knowledge.
Sensitivity to frequency of task switches. Fig. 4 shows the performance of our algorithm for different values of the probability p of task switches. The performance increases with p. which shows that our algorithm LEEDS can successfully re-use previous task knowledge to increase performance.
6 CONCLUSIONS
In this work, we study the online meta-learning problem in non-stationary environments without knowing the task boundaries. To address the problems therein, we propose LEEDS for efficient meta model and online task model updates. In particular, based two simple but effective detection mechanisms of the task switches and the distribution shift, LEEDS can efficiently reuse the best task model available without resetting to the meta model and distinguish the meta model updates for in-distribution tasks and out-of-distribution tasks so as to quickly learn the new knowledge for new distributions while preserving the old knowledge of the pre-training distribution. In particular, the meta model update in LEEDS is based on the current data only, eliminating the need of storing previous data. The theoretical analysis on the dynamic regret has clearly justified this design and extensive experiments corroborate the superior performance of LEEDS over related baseline methods on multiple benchmarks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jacob D Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. 2009.
Durmus Alp Emre Acar, Ruizhao Zhu, and Venkatesh Saligrama. Memory efficient online meta learning. International Conference on Machine Learning, pp. 32-42. PMLR, 2021.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 139-154, 2018.
Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradient- based meta-learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 424 433. PMLR, 09-15 Jun 2019. URL http::ttt://tmmcmmm htttps://proceeddnggg
m mlr.po wt.press/v9/. labaalea can 9a. htm
Luca Bertinetto, Joao F Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference Learning Representations (ICLR), 2018.
Massimo Caccia, Pau Rodríguez I. lópez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam H. Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. In NeurIPS, 2020.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420, 2018.
Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within-online meta-learning. Advances in Neural Information Processing Systems, 32, 2019.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. ArXiv, abs/1710. 1622, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proc. International Conference on Machine Learning (ICML), pp. 1126 1226-1135 1135, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta- meta-learning. In International Conference on Machine Learning, pp. 1920-1930. PMLR, 2019.
James Hannan. Approximation bayes risk in repeated play. Contributions the Theory of Games, 3(2):97-139, 1957.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta- learning without tasks. Advances in neural information processing systems, 33:17571-1758,, 2020.
Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 169 2007.
Elad Hazan et Introduction to online convex optimization. Foundations and Trendy@ in Optimiza- tion, 2(3-4):157-325,20 2016.
Mikhail Khodak, Aaria-Florina F Balcan, and Ameet S Talwalkar, Adaptive gradient-based meta- learning methods. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Grabska-Barwinsker Barwinska, et al. Overcoming
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, pp. 0. Lille, 2015.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv: 09835, 2017.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based Advances in Neural Information Processing Systems, 13:21464-214777 2020.
detection.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Fei Mi, Minlie Huang, Jiyong Zhang, and Boi Faltings. Meta-learning for low-resouree natural language generation task-oriented dialogue systems. arXiv preprint arXiv: 905.05644, 2019.
Aryan Mokhtari, Shahin Shahrampour Ali Jadbabaie, and Alejandro Ribeiro. Online optimization in
dynamic environments: Improved regret rates for strongly convex problems. In 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 7195-7201. IEEE, 2016.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pp. 2554-2563. PMLR, 2017.
Alex Nichol and John Schulman. Reptile: scalable metalearning algorithm. arXiv: Learning. 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint SXXV:1803300999 2018.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of MAML. International Conference on Learning Representations (ICLR), 2019.
Jathushan Rajasegaran, Chesea Finn, and Sergey Levine. Fully online meta-lcarning without task boundaries. arXiv preprint arXiv:2202.00263 2022.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems (NeurlPS), pp. 113-124, 2019.
Roger Ratcliff. Connectionist models recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Conference on Leurning Representations (ICLR), 2016.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint 1810 11910, 2018
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 044,016606.0661,, 2016.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with -agmened neural networks. In International conference on machine learning, pp. 1842- 1850. PMLR, 2016.

---

Under review as a conference paper at ICLR 2023
Syed Shakib Sarwar, Aayush Ankit, and Kaushik Roy. Incremental learning in deep convolutional neural networks using partial network sharing. IEEE Access, 8:4615-4628, -618,, 2019.
Shai Shalev-Shwartz and Yoram Singer. Online learning: Theory, algorithms, and applications. 2007.
Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends@ in Machine Learning, 4(2):107-19,, 2012.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NIPS), 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 1199-1208, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NIPS), 2016.
Huaxiu Yao, Yingbo Zhou, Mehrdad Mahdavi, Zhenhui Jessie Li, Richard Socher, and Caiming Xiong. Online structured meta-learning, Advances Neural Information Processing Systems, 33: 6779-6790 2020.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Represemtations, 2018.
Tianhe Yu, Xinyang Geng, Chelsea Finn, and Sergey Levine. Variable-shot adaptation for online meta-learning. arXiv preprint arXiv:2012.07769, 2020.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint 1003::18003.1133, 2018.
Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, and Jiashi Feng. Efficient meta learning via minibatch proximal update. Advances in Neural Information Processing Systems, 32, 2019.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), pp. 928-929--99 9889936, 2003.
12

---

Under review as a conference paper at ICLR 2023
Supplementary Material
We provide the details omitted in the main paper. The sections are organized as fellows:
We provide more empirical results including final average aceuracy of each of the methods and for all settings.
Appendix B
Appendix We provide further details about datasets and baseline methods.
Appendix We provide further experimental specifications and discuss the heuristics used to set the thresholds.
Appendix
We provide our proof of Theorem !
A SENSITIVITY TO THRESHOLDS & AND T AND TEMPERATURE 6
Figures 5 and 6 illustrate the sensitivity of our algorithm LEEDS with respect to the thresholds and and the temperature parameter ó in the energy-based detection module. As depicted in fig. [ when the threshold & is too small, the algorithm tends to over detect task switches (as indicated by low Recall for & 0.5 in the table), which results in inferior performance of the algorithm due to ineffective reuse of task knowledge. On the other hand when l is too large, the high misdetection rate (e.g.. indicated by low Precision for = results in the algorithm mostly fine-tuning the online task model Φe with the current task support data. As expected, this results in a failure mode (the algorithm diverges) due to the adversariality of different tasks. We find that values of in the rage [1.5, 2.3] yield the best performance of our algorithm.
Figure 6 (a) shows that larger values r, which collapse to updating the meta-model at each step (even for pretraining task distribution), does not substantially improve the performance. This demonstrates the advantage of the distinct update scheme proposed for in- and out-of diftiiitibuution tasks, which avoids unnecessary frequent meta-updates for the pretraining tasks and thus allows more judicious usage of computational budget. Lower values the pertaining one, and thus corresponds to eliminating the domain adaptation component of our algorithm. We also find that simply setting the temperature 8 = 1 in the energy expression yields the best performance, and large values of ó also eliminates the effectiveness of the energy-based detection module (fig. 6 (b)). This in fact in accordance to the finding in Liu et al. (220), which also suggests setting
(e.g. T = 15) tend to detect all task distributions
1.5 0.3
69
Precision
Recall
8= 0.5
99,9
68.0
0.7
{= 1.5
99.9
98.4
8 1.9
99.9
99.3
68
= 2.3
99,9
99.7
38.13
99.9
24
34
Figure 5: Performance of LEEDS for different values of the threshold f. Left plot: Performance on all encountered domains during online learning. Right table: Task boundaries detection for different values of é. Experiments are conducted on the Miiigloo-MIIST--asshiinnont
benchmark.
13

---

Under review as a conference paper at ICLR 2023
adeptation
29
0.85
0.
Stop
24
(a) Sensitivity energy threshold
(b) Sensitivity to temperature scale
Figure 6: Performance of LEEDS for: (a) different values of the energy threshold 7 and (b) different scales the temperature 8. For both plots we report the performance on all encountered domains during online learning, Experiments are conducted on the Omniglo-MNNISTFaasionnn mark.
bench-
B MORE EXPERIMENTAL RESULTS
In this section, we provide more experimental results for each of the benchmarks. Figures and (and8sshow show the online evaluations of the different methods for the Tiered-Imagee ImageNet and Synbols benchmarks under p = 0.75. These additional results further show that our algorithm LEEDS outperforms other baseline algorithms in the OOD domains and at the same time also retains its performance on the pre-training tasks. The performance of the methods that do not adapt meta-paraan petampaarseterss during online learning phase (such as MAML and ANIL) drops drastically when OOD tasks are far away from the pre-training ones (as shown in Table for Wunnggoot-NIISS.FashiinNNNEEEEE
For settings which OOD tasks close to the pre-training ones (such as in the Tiered- ImageNt dataset), MAML can perform similarly to CMAML++, as depicted in Table
Further, Tables 2, and indicate that the performance of all compared baselines decrease as the non-stationarity level decreases (smaller p indicates a higher task switch frequency). In fact, this is also captured by theoretical upper bound for the dynamic regret in Theorem T as the accumulative model shift Pp is likely to increase when the environment becomes less stationary. Such intuition/result is also confirmed by the plots in Figure 9 As already shown in Figure 4 for all encountered domains, Figure 9 depicts the same dependency on for the pre-training domain. The superior performance of LEEDS even for the pre- pre-training domain particularly shows that the re-use of task knowledge is beneficial for online meta- meta-lcarning, as opposed to the usual practice of "resetting" to meta- pettapprrseters each step.
By comparing the two plots in Figure 10 it can be seen that the advantage of our domain adaptation module is more significant when the OOD domains are far away from the pre-training one, as the case for the FashionMNIST OOD domain compared to the Omniglot pre-training domain.
C FURTHER DESCRIPTIONS ABOUT DATASETS AND BASELINE METHODS
C. DATASETS
We study dynamic online meta-learning on the following benchmarks:
Omniglot-MNIIST- F.AMMIIITT-PShOOMMMNISTTT dataset. For this dataset, we consider 10-ways 5-shots classifi- cation tasks. We pre-train the meta model on subset of the Omniglot dataset and then deploy it in the online learning environment where tasks are sampled from either the full Omniglot dataset, or from one of the OOD datasets, i.e., the MNIST FashionMNIST datasets.
Tiered-Ine ImageNet dataset. We consider consider5-way: 5-ways 5-shots classification tasks for this dataset. Following (cite paper), split the original Tiered- ImageNett dataset into the pre-training domain and the OOD
14

---

Under review as a conference paper at ICLR 2023
Accuracy on
Accuracy on OOD domains
LEEDS
011L11
FOML
0.5
0.4
B2D
HotaBED
Hatactn 03
Step
Step
0.2
5k 12k 15k
Figure 7: Online evaluations for the Tiered- Imtaee-t (TT) benchmark under 0.75. Left: Accuracies on all encountered domains during online learning. Right: Accuracies on encountered OOD domains during online learning. We compare all baselines on 16GB GPU memory budget and FOML runs out of memory for this benchmark due its linear growth memory requirement.
Accuracy on domains
Accuracy on 000 domains
0.6
** 1FE05
0.5
Neta050
NetaBG0
NPL
02
RAZ
Ste
Figure 8: Online evaluations for the Synbols (SB) benchmark under p = 0.75. Left: Accuracies on all encountered domains during online learning. Right: Accuracies all encountered OOD domains during online learning.
domains. More specifically, we pre-train on the first 200 classes of the original training set, use the remaining classes the first OOD domain (oodl), and the original test classes as second OOD domain (ood2). During online learning phase, we set the full training set to be IND domain, and ood and ood2 as OOD domains. We evaluate algorithms over 20000 online learning episodes.
Synbols dataset. We consider 4-ways 4-shots classification tasks in this dataset. The meta model is pre-trained on characters from different alphabets and deployed on characters from a new alphabet (ood1). We also consider font classification tasks additional OOD tasks (ood2).
C.2 BASELINE METHODS
We compare algorithm with the following baseline methods for online meta-learning.
(1) C-MAML Caccia et al. 2020 and C-MAML++: the continual MAML approach (C-MAML) pre-tree pre-trains the meta model using MAML and employs an online learning strategy based task boundaries detection. Since C-MAML does not evaluate the task models on separate query sets, for fair comparison adapt it do and call the resulting algorithm C-MAML++.
15

---

Under review as a conference paper at ICLR 2023
Method
non-stationarity p 0.9
non-stationarity
0.75
pre-train
oodl
ood2
pre-train
ood1
ood2
LEEDS
99.39 +.009
96.44 10.11
82.87
10.19
98.97
10.10
95.68
10.12
81.49
10.22
C
98.78 10.12
92.52
10.19
76.16
#0.28
97.39
10.11
89.07
10.20
73.35
40.35
CMAML
89.79 +0.54
84.06
+0080
69.70
#0.63
75.51
10.11
70.41 #1.22
58.58 11.27
FOML
89.20 #0.61
70.84
#0.76
64.83
#0.74
81.68
+0.59
59.24 10.78
58.07 #0.77
MAML
95.07 #0.10
10.14
54.67 #0.17
95.51 #0.11
62.31 10.13
54.83 #0.13
ANIL
96.54 #0.12
10.16
40.12 10.13
96.88
#0.11
42.08 #0.11
40.00 #0.13
MetaOGD
84.05 +1.66
73.73
60.03
+1.60
85.67
+1.57
+1.43
60.51 +1.65
BGD
63.58 +2.25
+2.10
44.89 +1.41
23.86
+2.36
17.97 +2.17
19.81 +1.63
MetaBGD
77.73 #1.26
59.11 +0.84
54.67 #0.82
43.95 +1.31
27.87 +0.94
30.14 +0.98
Table 1: Average accuracy over 10000 online episodes on Onmiglo-NNNISTPPsssooNNNNYY
benchmark under different non-stationaritt levels. "pre-train" domain: Omniglot: "ood1" domain: MNIST, "ood2" domain: FashionMNNIST. The advantage of our algorithm LEEDS over the other baselines is more significant in the ood domains.
Method
non-stationarity p 0.9
non-stationarity 0.75 ood domains
all domains
ood domains
all domains
LEEDS
66.07 #0.24
67.43 #0.38
64.52 #0.17
65.80 #0.31
CMAML++
63.83 #0.27
63.75 +0.55
61.28 =0.23
61.96 #0.41
FOML
35.90 10.56
35.87 10.83
32.02 10.42
31,61 10.69
MAML
62.37 10.46
61.00
10.72
62.54 10.37
60.88 10.65
ANIL
59.78 +0.21
57.61 +0038
59.57 +0.22
57.38 +0.36
MetaOGD
57.01 +0.28
57.32 +0.66
56.80
+0.25
56.94 +0.42
BGD
40.95 #0.85
41.44 #1.5
35.48 #0.76
35.97 #1.09
MetaBGD
49.21 #1.05
50.01 #125
44.58
#1.12
45,30 1120
Table 2: Average accuracy over 20000 online episodes on Nieged-I benchmark under different non-stationarity levels. The different domains are distinct splits of the original Tiered- ImageNet dataset (please see experimental setup Section for details on how these splits are obtained).
Method
non-stationarity p = 0.9
non-stationarity p 0.75 ood domains
all domains
ood domains
all domains
LEEDS
85.12 10.91
67.48 10.97
82.22 10.32
63.68 10.36
+
81.14 +1.05
62.39 +1.00
79.74 +1.07
60.70 +1.12
FOMI.
46.40 +0061
41.73 +0.73
37.46 +0.27
34.13 +0.31
MAML
76.25 #0.63
42.70
#0.68
74.87
#0.42
43.84 #0.45
ANIL
64.58 #0.32
34.51 #0.54
72.69
+0.30
35.66 +0.49
MetaOGD
72.04 10.67
46.69
10.77
67.93
10.59
42.66 10.62
BGD
25.63 10.07
25.61
10.09
27.53
10.08
27.17 10.11
MetaBGD
53.74 10.41
42.25
10.52
40.79
10.23
34.63 10.33
Table 3: Average accuracy over 10000 online episodes on Synbols benchmark under different non- stationarity levels. The "pre-train" domain corresponds to 3 different alphabets from Synbols dataset, "ood1" corresponds to new (w.r.t. the pre-training one) alphabet from Synbols dataset, and "ood2" contains font classification tasks.
(2) FOML Rajasegaran et al. 2002):: the fully online meta-learning method updates online parame- ters using the latest online data and maintains a concurrent meta meta-training process to guide the online updates regularized by the meta model.
(3) MAML Finn et al. (2017) and ANIL Raghu et al. (2019): the MAML baseline consists of an offline pre-training phase and online deployment phase. During offline pre-training the method
16

---

Under review as a conference paper at ICLR 2023
8.8
3.955
2.39
1585
3.9
0.58
Step
0.88
24
6
6k
Figure 9: Performance of our algorithm LEEDS under different p. Left plot: Evaluations in pre- training domain. Right plot: Evaluations in all domains.
sdactation
aothout accptation
adsptation vithout
0.36
0.58
0.7
0.95
Stap
0.51
4
Figure 10: Performance of LEEDS with and without energy-based domain adaptation module. Left plot: Evaluations in OOD domain FashiOnMINISTT, Right plot: Evaluations in OOD domain MNIST.
learns common meta-initialization that will be used for all tasks, and the meta-initialization will never be updated at the online stage. The task model is adapted from the meta-ing inttaiiitializzttinnn using the support set and evaluated on the query set. ANIL is similar MAML but with partial parameter adaption, i.e., only the last layer is adapted for each task.
(4) MetaBGD Caccia et al. 2020) and BGD Zeno al. (2018) the baseline MetaBGD combines MAML and the Bayesian gradient descent method during online learning.
(5) MetaOGD Zinkevich (2003): the meta online gradient descent method simply updates the meta- model at each step using a MAML-like meta-objective evaluated on the current task data.
D FURTHER EXPERIMENTAL DETAILS AND HYPERAMETER SEARCH
D.1 FURTHER EXPERIMENTAL SPECIFICATIONS
In all our experiments, we consider classification tasks. The cross-entropy loss between predictions and true labels is used to train all models. We use the same convolutional neural network (CNN) architectures widely adopted in few-shot learning literature Finn al. (0017 2019 ; Caccia et al.
(2000), which include four convolutional blocks followed by a linear classification layer. Each convolutional block is stack of one 3 3 convolution layer followed by BatchNormalization. ReLU, and 2 x MaxPooling layers. For the Omniglot-M
benchmark, we use 64 filters each convolutional layer and downsample the gray-scale images to 28 28 spacial resolution
17

---

Under review as a conference paper at ICLR 2023
have 64-dimensional feature vectors before classification. For Tiered-ImageNet and Synbols datasets, the inputs are respectively x 64 x 64 and 3 x 32 x 32 RGB images, resulting respectively in 1024- and 256-dimensional feature vectors for 64 filters in each convolution layer.
D.2 FURTHER IMPLEMENTATION SPECIFICATIONS
We will make all our codes publicly available after the review process. The implementation of FOML Rajasegaran al. (2022) method is not released yet, and hence compare with own implementation of their algorithm. For all other baselines, we used their publicly available implementations. Codes for our algorithm LEEDS and all other baselines are provided in the
supplementary materials of our submission. All codes are tested with Python 3.6 and Pytorch 1.2.
For example to run our algorithm LEEDS with the best hyperparameters that we obtained for the Omniglot-MNIST- WANNIST-FashonMMNISS dataset under 0.9, one can run the following command:
python main. -aago leeds - use uss_bestt
The experiment setting (e.g., the dataset to use) can be changed in configurations.yaan yaml file. We run all methods on single NVIDIA Tesla P100 GPU. All compared algorithms except FOML were able to run 16GB GPU memory. FOML requires at least 32GB to reach 12000 online episodes for the fiered-ImageeNe dataset.
E HEURISTICS FOR SETTING THE THRESHOLDS
For the energy threshold follow the strategy in Liu et al. (2020), i.e., set the threshold using the pre-training tasks. More specifically, we set so that 95% of the pre-training inputs are correctly detected as pre-training data. In standard few-shot learning experimental setups, the true labeling for each individual task is usually randomly chosen. When there is task switch, the task specific model learnt from the previous task generally does fit the new task anymore, where the learning performance could be similar to that of a random model. Thus motivated, we find that good heuristic choose a starting value for the threshold is to use the loss value evaluated on a random model. For example, for 10-ways classification tasks that value would be é,- = (gg(111) 2.3.
F PROOF OF THEOREM I
Consider the following algorithm for minimizing the regret eq. (6). which updates the meta- parameters at each step based on current task only:
g
04 V la (U) (05), with 0441 0K
(8)
For convenience, we next re-state the result in Theorem
Theorem (Re-sta Suppose that Assumptions and 2 hold. Define the path length as Pp Ele 100 0 11 + and set the learning rate a
Then the dynamic regret eq.
VELR Ka
is bounded as
R Rmeta (00 (f....., (......b)) 08)50000 S0 PDy
( Eu Kd
Further, if
Pp Din K.
then
R 00. 5 ( Σd Ka
(10)
Proof of Theorem In the following, let h = fa U. and define M to be such that ||eb < M for all d and &. Such an upper bound can enforced by implementing the project gradient descent so that the model parameter is always updated within compact with radius M.
18

---

Under review as a conference paper at ICLR 2023
First, note that for any given domain index d, the term Σ få (U) in the dynamic regret in eq. (6) corresponds to a static regret defined over Ka steps for loss functions
K2
- fä (U)))))
h få Uz. Hence, applying steps similar to those in the proof of Theorem 3.1 in Hazan et al.
6 for static regret, we have
| - 0all - |103 0all² + (a ViS(0))) (hh(0))) (0) - 02)
Using Assumption 1 land rearranging the terms, we obtain
27)) (0) 02) 10 87
02
2
+ G
ad
Telescoping the above inequality from & = to Ka and using the convexity of ht yield
Ka
Ka Σ
(0))
(0 S
100
02ll
110/at2
02
2a
2a
1=1
Ga K₁
1
110 0211
Σ a
(11)
a
a'
x-1
Hence, tixing the stepsizes to a
we obtain
v
V Ka
Σ 12(0
Σ
Ed=1 Kd
(10)) 03ll -1108 110gat1-020
G²
2
2 Ed Ka
Ed=1 Ka
G2Ka
(110) - 0āll -1102-1 - 021l²)
2
2V Ed Ka
where the last equality applies the transition scheme 0Kat1 02 Now, telescoping from to D, we obtain
Σ
Σ 1
VE
K3
Σ
(10)
0all -110241 031l²)
+
G²
Σ
Ka-
2
d=1
d=1
(12)
Next, we upper-bound the term P. We have
|[0]
0g|l
10
= 2 2 2(4 02) - -1102 112 + (02411
=1102| 2 | 11 2 20 03) 2 02-1) + 02))
-2 2 (02 03)
Therefore, we obtain
D [10] 0/|12 | 0z]l²) < ||0t1| - -110b+1 1|2 2 .8) + Σ
Σ d=1
(0) 0 07 07
d=2
< 2 + 2M2² 2M Σ 00 02-11l
< <3M² 1+ [++00
10 0
0-111
2=2
19

---

Under review as a conference paper at ICLR 2023
=3M2P
(13)
where the last equality follows from the definition of PD = Eds combining eq. (12) and eq. (3). we obtain
100
02-1l + 1. Hence,
PM
D
G
D
meta (0)
Σ
Kd
+
Σ
K2
2
2
d=1
d=1
D Σ
0
PD
Kd
(14)
d=1
which finishes the proof for the first result.
Similarly, setting a
Po Ka
we obtain the following upper bound for the dynamic regret in
eq. 66
Rmeta .... .b) 5o
PD Ks
Ka
(15)
d=1
20


Reviews:
[{"review_id": "v7I0YDV17-G", "reviewer": "Reviewer_R5Rh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new algorithm for task-agnostic online meta-learning when data is non-stationary. The algorithm has two components. First, the algorithm can detect task switches. Second, the algorithm can adapt to distribution shifts. The paper provides a regret analysis of the algorithm.", "summary_of_the_review": "This paper proposes a new algorithm for online meta-learning to simultaneously address the problem of agnostic tasks and the problem of distribution shift. The paper is well-written. However, I think the paper can benefit from more discussion on the theoretical results.", "strength_and_weaknesses": "Strength:\n\nThis paper seeks to solve a challenging problem, and provides a nice solution. The solution makes intuitive sense and is supposed by a regret analysis.\n\nWeakness:\n\n1. How is the threshold $\\ell$ in Algorithm 1 chosen? When the algorithm is used online, it seems unclear to users about how to choose an appropriate $\\ell$. Some practical guidance will be very useful. \n\n2. What doesn't the regret bound depend on $\\ell$? It seems if we choose $\\ell = 0$, then the regret should grow linearly in $\\sum_d K_d$.  In the regret definition (6), does $\\sum_d K_d$ equal $T$?\n\n3. Consider two scenarios with two very different tasks (data from each task are drawn from a stationary distribution). The first scenario is that from time 1 to T/2, it is the first task, and from time T/2+1, ..., T, it is the second task. The second scenario is that two tasks frequently switch between each other. Clearly, the proposed algorithm performs well in the first scenario, but not in the second. I don't seem how the regret varies between these two scenarios. Some clarification will be helpful.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wn7BTytETM5", "reviewer": "Reviewer_ZoyY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of online meta-learning and proposes an algorithm to handle the case where the task boundaries are unknown and the the comparator can change over time. They propose detection mechanisms for task switches and a method that adapts the meta-model when the switches occur and the fine-tuned model when they do not. The proposal is supported with theoretical and experimental analysis.", "summary_of_the_review": "I do not believe should be accepted in its current form due to the issues with the theoretical results outlined above.", "strength_and_weaknesses": "### Strengths:\n1. The problem is well-motivated as being able to adapt quickly in change environments is a core aspect of learning.\n2. The switch detection mechanisms seem interesting, although I do not know if they are novel; are they connected to change-point detection algorithms? A comparison with past work such as CMAML++ would also be useful\n3. Code is provided in the supplementary material.\n\n### Weaknesses:\n1. My main concern with this paper is in the theory, where there are several issues:\n    1. The statement “as shown in Finn et al. (2019), Assumption 2 holds when the function is convex and the adaptation mapping corresponds to the one-step gradient descent” is incorrect; the relevant statement in that paper (Theorem 1) holds only for strongly convex, strongly smooth, and Hessian smooth functions, and it is unclear if the result can be extended to convex functions without making the step-size of the adaptation mapping zero and thus uninteresting in this context.\n    2. It is a bit difficult to understand exactly which algorithm Theorem 1 gives guarantees for, but my understanding is that they are for Algorithm 1 assuming the detection mechanisms are correct every time. What this means is that, while the paper is advertised as not knowing the task boundaries, for the actual guarantee the task boundaries are indeed known. The issue then becomes that the authors do not compare to the trivial baseline, which is restarting gradient descent whenever a new task is detected; in the regime where all tasks have at most K rounds doing this has dynamic regret O(D*sqrt(K)), which is better than the provided guarantee unless the path-length is o(sqrt(D)). If the task-boundaries are known, the setting studied by the authors is in-fact more akin to that of online-within-online meta-learning (Denevi et al., 2019; Khodak et al., 2019), where you have a sequence of tasks with some number of rounds per task; the only difference is here the loss functions are composite (but convex, as in those papers). Notably, Theorem 3.3 of the latter gives a guarantee of O(D^(3/4)*sqrt(K)) as a special case, which is better than the provided guarantee unless the path-length is o(D^(1/4)).\n    3. Even if Theorem 1 were providing guarantee for unknown task-boundaries, work by Jadbabaie et al. (2015) shows a dynamic regret of O(sqrt((P+1)*T) for a similar notion of path-length P (basically without the square on the norm) and to my understanding without needing to know the path-length to set the step-size. Applying their algorithm to the convex composite losses here would yield what I would view as a better result than the proposed approach (and in-fact can adapt to other types of regularity in the sequence).\n2. The discussion of memory-efficiency is a bit overstated; theoretically at least, the O(T) issue in Finn et al. (2019) is the result of the odd choice of using FTL to optimize the sequence of strongly convex functions, when adaptive OGD (Hazan et al., 2007) would have achieved the same regret and with O(1) memory. Methods studying the online-within-online setting (Denevi et al., 2019; Khodak et al., 2019) also use O(1) memory. Notably, memory efficiency is not a “theoretical advance in online learning” that occurred in 2016.\n3. It is strange to motivate an adversarial theoretical analysis and algorithm with a discussion of distribution shifts, which suggests a stochastic problem.\n\n\n### References:\n- Denevi, Stamos, Ciliberto, Pontil. Online-within-online meta-learning. NeurIPS 2019.\n- Finn, Rajeswaran, Kakade, Levine. Online meta-learning. ICML 2019.\n- Hazan, Rakhlin, Bartlett. Adaptive online gradient descent. NeurIPS 2007.\n- Jadbabaie, Rakhlin, Shahrampour, Sridharan. Online optimization: Competing with dynamic comparators. AISTATS 2015.\n- Khodak, Balcan, Talwalkar. Adaptive gradient-based meta-learning methods. NeurIPS 2019.\n- Mokhtari, Shahrampour, Jadbabaie, Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. CDC 2016", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K8Dw95J-Bg", "reviewer": "Reviewer_8xrn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose to investigate meta-continual learning scenario where the task boundaries are not known and whether distribution shift occur or not. To solve this challenging problem, they focus on some empirical findings that the loss values or free energies sharply increases when the boundary meets. Thus they introduce two threshold values for detecting task boundaries and distributional shifts, in order to selectively control between task-specific learning and meta-updates. The empirical results demonstrate the efficacy and effectivenss of their methodologies on various learning scenarios.", "summary_of_the_review": "In summary, although there are some concerns regarding the novelty and quality of the submission. I think the strength outweights its weaknesses which I think can be easily addressed. Thus I recommend (weak) acceptance.", "strength_and_weaknesses": "Strength\n- The target problem is largely important for real-world applications.\n- the paper is well written and method easy to understand.\n- Method performs well on various learning scenarios, outperforming current state-of-the-arts on the given problem.\n- Theoretical results support their claim.\n\nWeaknesses\n- The proposed method seems to largely based on empirical observation that the loss value sharply increases when the task switches. The corresponding thresholding scheme is also very naive, although it seems intuitive and empirically effective. I wonder how they set the threshold value in practice - Have those hyperparameters been found with a holdout meta-validation set? or meta-test set?\n- The quality of the paper is a bit low. \\citep and \\citet are not clearly distinguished. Also, the learning curves are just copy and past of wandb plots, making the quality of the paper looking very low. Some of the baselins have not been fully run (e.g., FOML in Fig 3)\n- I'm not sure whether some of the existing baselines such as FTML by Finn et al. and OSML by Yao et al. are not applicable the setting the authors considered. Could you compare against FTML and OSML as well? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Et7i-oEkPN", "reviewer": "Reviewer_1HZs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers online meta-learning in a non-stationary environment where new tasks might come and task distribution can change. To solve these problems, this work designs two heuristic detectors to detect the task switches and distribution shifts. It also provides a theoretical analysis of the dynamic regret under certain assumptions.", "summary_of_the_review": "This paper has studied an interesting but challenging problem. But I am concerned about the mismatch between the proposed algorithm and the theoretical analysis. Besides, though the shift detector is novel to me, the criterion is mainly established by the previous work [Liu et al., 202], and the theoretical analysis is standard. Overall, I think this paper has some interesting parts, but the flaws outweigh its merit. So, I would like to reject this paper.\n", "strength_and_weaknesses": "### Strength:\n- This paper considers an interesting but hard problem that the new task/distribution could appear in the online meta-learning process.\n- The paper is clearly structured and well-written.\n- Theoretical analysis of the dynamic regret is provided.\n\n### Weakness:\n- My main concern about the paper is the mismatch between the theoretical analysis and the proposed methods. The authors have proposed two heuristics to detect the shift on tasks and task distribution, which are the main algorithm contributions of this paper. But, the theoretical analysis is irrelevant to the proposed mechanism.\n- About the novelty of the dynamic regret bound. The analysis for the dynamic regret seems a direct application of that for the OGD algorithm [Zinkvich,2003] under the convexity assumption (Assumption 2). In this sense, the theoretical technical novelty of this is somewhat limited to me.\n- About the correctness of the dynamic regret bound. It seems to me that $P_T$ in Theorem 1 should be defined as $P_T = \\sum \\Vert \\theta_d^*-\\theta_{d+1}^*\\Vert$ instead of the second-order formulation. It is unclear to me why the second inequality of Eq. (13) holds.\n- About the claim on the sublinear regret: It seems that the proposed regret bound will suffer from a linear regret when $P_T = O(T)$. Therefore, the claim on the sublinear dynamic regret is somewhat an overclaim to me. \n- About the experiments: To validate the effectiveness of the proposed detection scheme, it would be better to provide a more detailed inspection of the shift detection instead accuracy instead of just providing the comparison on the overall performance.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4Os-3s7aKs2", "reviewer": "Reviewer_HHCs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose an algorithm for task-agnostic online meta-learning in nonstationary environments.", "summary_of_the_review": "**Don't panic**: This paper wants to be accepted.  You just need to address my concerns in order for me to raise my score.\n\nMy primary concern is the task switching heuristic.  Both the heuristic and the description of the heuristic are lacking.  Most of the clarity issues I identified were related to this aspect.  Just fix them.\n  * Step 1: better describe what you are actually doing to set the temperature.  \n  * Step 2: provide some experimental commentary on the sensitivity of the technique to the choice of temperature.\n  * Step 3: quantify how homogenous (or not) the task loss distributions are in the setups you are using to give an idea of why the heuristic is working and when it would fail.\n  * Step 4: provide some commentary relative to what dynamic regret algorithms *actually* do to detect environment drift: they do regret bound arithmetic, e.g., [1]\n\t\nThe other concerns are lower in importance but to the extent you address them you'll have a better paper.\n\n**Updated**: 5->6 after author response\n\n\t\n[1] https://arxiv.org/abs/2102.05406", "strength_and_weaknesses": "Strengths:\n  * The insight that the simplicity of the task-adaptation step (one single application of batch gradient) convexifies the problem despite the use of deep architectures, allowing the application of online learning techniques.\n  \nWeaknessess:\n  * Task switch detection heuristic does not correspond to dynamic regret algorithm techniques from the online learning literature.  The paper lacks details on how this heuristic (threshold) is tuned in practice.\n  * The paper emphasizes the setting where task boundaries are not known, but without comment the analysis (e.g., Theorem 1) appears to assume task distributions are switching at a regular cadence.  \n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject