Fully Online Meta Learning
Authors: Jathushan Rajasegaran
Chelsea Finn
Sergey Levine
Venue: Submitted to ICLR 2023 (2023)
Keywords: Meta learning
Online learning

Abstract: While deep networks can learn complex functions such as classifiers, detectors, and trackers, many applications require models that continually adapt to changing input distributions, changing tasks, and changing environmental conditions. Indeed, this ability to continuously accrue knowledge and use past experience to learn new tasks quickly in continual settings is one of the key properties of an intelligent system. For complex and high-dimensional problems, simply updating the model continually with standard learning algorithms such as gradient descent may result in slow adaptation. Meta-learning can provide a powerful tool to accelerate adaptation yet is conventionally studied in batch settings. In this paper, we study how meta-learning can be applied to tackle online problems of this nature, simultaneously adapting to changing tasks and input distributions and meta-training the model in order to adapt more quickly in the future. Extending meta-learning into the online setting presents its own challenges, and although several prior methods have studied related problems, they generally require a discrete notion of tasks, with known ground-truth task boundaries. Such methods typically adapt to each task in sequence, resetting the model between tasks, rather than adapting continuously across tasks. In many real-world settings, such discrete boundaries are unavailable, and may not even exist. To address these settings, we propose a Fully Online Meta-Learning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries and stays fully online without resetting back to pre-trained weights. Our experiments show that FOML was able to learn new tasks faster than the state-of-the-art online learning methods on Rainbow-MNIST, CIFAR100 and CELEBA datasets.

FULLY ONLINE META LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
While deep networks can learn complex functions such as classifiers, detectors, and trackers, many applications require models that continually adapt to changing input distributions, changing tasks, and changing environmental conditions. Indeed, this ability to continuously accrue knowledge and use past experience to learn new tasks quickly in continual settings is one of the key properties of an intelligent system. For complex and high-dimensiooal problems, simply updating the model continually with standard learning algorithms such as gradient descent may result in slow adaptation. Meta- Meta-learning can provide a powerful tool to accelerate adaptation yet is conventionally studied in batch settings. In this paper, we study how meta- learning can be applied to tackle online problems of this nature, simultaneously adapting to changing tasks and input distributions and meta-training the model in order to adapt more quickly in the future. Extending meta-learning into the online setting presents its own challenges, and although several prior methods have studied related problems, they generally require a discrete notion of tasks, with known ground-truth task boundaries. Such methods typically adapt to each task in sequence, resetting the model between tasks, rather than adapting continuously across tasks. In many real-world settings, such discrete boundaries are unavailable, and may not even exist. To address these settings, we propose Fully Online Meta- Learning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries and stays fully online without resetting to pre- trained weights. Our experiments show that FOML was able to learn new tasks faster than the state-of art online learning methods on various datasets.
INTRODUCTION
Flexibility and rapid adaptation are hallmark of intelligence: humans can not only solve complex problems, but they can also figure out how to solve them very rapidly, as compared to our current machine learning algorithms. Such rapid adaptation is crucial for both humans and computers: for humans, it is crucial for survival in changing natural environments, and it is also crucial for agents that classify photographs on the Internet, interpret text, control autonomous vehicles, and generally make accurate predictions with rapidly changing real-world data. While deep neural networks are remarkably effective for learning and representing accurate models He et al. (2015): Krizhevsky et al. (Simonnnm & Zisserman (2014); SSzeedyy et al. (2015), they comparatively unimpressive when comes to adaptability, due to their computational and data requirements. Meta-learning in principle mitigates this problem, by leveraging the generalization power of neural networks to accelerate adaptation to new tasks Finn et al. (2019): Li et al. (2017); Nichol et al. (2018) 8); Nichol & Schulman h018): Park & Olival (2019); Antoniou et al. (018). However, standard meta-learning algorithms operate in batch mode, making them poorly suited for continuously evolving environments. More recently, online meta- learning methods have been proposed with the goal of enabling continual adaptation Finn et al. (2019); Jertel et al. (2018): Yao et al. (2020): Nagubandi et al. (2018); Li & Hospedales (2020). where constant stream of data from distinct tasks is used for both adaptation and meta-training In this scheme, meta-traie trainttiining is used to accelerate how quickly the network can adapt to each new task it sees, and simultaneously use that data from each new task for meta-training. This further accelerates how quickly each subsequent task can be acquired. However, current online meta-learning methods fall short of the goal of creating an effective adaptation system for online data in several ways: (1) they typically require task boundaries in the data stream to be known, making them ill-suited to settings where task boundaries are ill-defined and tasks change or evolve gradually, common tread in real-world; (2) as a result, they typically re-adapt from the meta-trained model

---

Standard Online Meta Learning
Fully Online Meta Learing
Rodt
dee
Aepulcion hone
Taot
Taet 2
Tiet
Tank
Taee
Times
Thik shif gridaall, no known boundanes Time
Figure 1: Comparison of standard online meta-learning and FOML: PTML, al. (2019), shown on the left, adaptation is performed on one task a time, and the algorithm "resets" the adantation process at task houndarics. For example, MAML-bbase method would reset the current parameters back the meta-trained parameters. In our approach (right), knowledge of task boundaries is not required, and the algorithm continually keeps track of online parameters e and meta- parammeaaeetesss 0. The online parameters are simply updated on the latest data, and the mera-parameterss are updated to "pull" the online parameters toward fast-adapting solutions via MAML-style meta-update.
In standard online meta-Icarning (e.g..
very "discrete" mode of operation, where the model adapts to a task, then resets, then adapts to a new one. These limitations restrict the applicability of current online meta-learning methods to real-world settings. We argue that task boundary assumption somewhat artificial in online settings, where the stream of incoming data is cleanly partitioned into discrete and well-separated tasks presented in sequence. In this paper, we instead develop a fully online meta-learning approach, which does not assume knowledge of task boundaries and does not re-adapt for every new task from the meta parameters.
on each task, resulting
Standard meta-learning methods consist of meta-training phase, typically done with standard SGD, and an "inner loop" adaptation phase, which computes task specific parameter o; for the task T, from support set to make accurate predictions on query set. For example, in model-agnostic meta- learning (MAML), adaptation consists of taking a few gradient steps on the support set, starting from the meta-trained parameter vector 0, leading to a set of post-adapiation parameters, and meta-training optimizes the meta-trained parameters 0 so that these gradient steps lead to good results. Previous extensions of such approaches into the online setting typically observe one task at a time, adapt to that task (i.e., compute cest-adaptation parameters on it), and then reset Φ back to the meta-trained parameters the beginning of the next task. Thus, the algorithm repeatedly adapts, resets back to pretrained meta parameters at the task boundary, adapts again, and repeats. This is illustrated in Figure (left). However, in many realistic settings, the task boundaries are not known, and instead the tasks shift gradually over time. The discrete "resetting" procedure is poor fit in such cases, and we would like to simply continue adapting the weights over time without ever resetting back to the meta- meta-trained parameters, still benefit from a concurrent meta- veta-training process. For example, meta- trained image-tagging model on the Internet (e.g., tagging friends in photographs) might gradually adapt to changing patterns and preferences of its users over time, where it would be unnatural to assume discrete shifts in what users want to tag. Similarly, a traffic prediction system might adapt to changing traffic patterns, including periodic changes due to seasons, and unexpected changes due to shifting economic conditions, weather, and accidents. In this spirit, our method does not require any knowledge on the task boundaries as well as stays fully-online through out the learning.
The main contribution of our paper is FOML (fully online meta-learning), an online meta-learning algorithm that continually updates its online parameters with each new datapoint or batch of datapoints, while simultancously performing meta geatdint updates on a separate set of meta-parama parameters using a buffer of previously seen data. FOML does not require ground truth knowledge of task boundaries, and does not reset the online parameters back to the meta-parameters between tasks, instead updating the online parameters continually in a fully online fashion. We compare FOML empirically to strong baselines and a state-of-the thr prior online meta-learning method, showing that FOML learns to adapt more quickly, and achieves lower error rates, both on a simple sequential image classification task from prior work and more complex benchmark that we propose based on the CIFAR 100 dataset, with a sequence of 1200 tasks.
2 RELATED WORK
Online meta- leetn-le brings together ideas from online learning, meta learning, and continual learning, with the aim of adapting quickly to each new task while simultaneously learning how to adapt even more quickly in the future. We discuss these three sets of approaches next.

---

Meta Learning: Meta learning methods try to learn the high-level context of the data, to behave well on new tasks (Learning to learn). These methods involve learning a metric space Koch et al.
(2015; ; Vinyals et al. (2016] Snell et al. (2017) Yang et al. (217)) gradient based updates Finn et al.
(2017)
; Li et al. (201 7): Park & Oliva 2019); Nichol et al. 20;8); Nichol & Schulman (01 8), or
some specific architecture designs Santooo et al. (2016); Munkhdalai & (017) Ravi & Larochelle (0166).
In this work, we are mainly interested in gradient based meta learning methods for online learning. MAML Finn et al. (0017) and its variants Nichol et al. (2018); Nichol & Schulman (2018): Li et al. (2017): Park & Oliva (2019); Antoniou et al. (2018) first meta train the models in such a way that the meta parameters are close to the optimal task specific parameters (good initialization). This way. adaptation becomes faster when fine tuning from the meta parameters. However, directly adapting this approach into an online setting will require more relaxation on online learning assumptions, such as access to task boundaries and resetting back and froth from meta parameters. Our method does not require knowledge of task boundaries.
Online learning methods update their models based on the stream of data sequen- tially. There are various works on online learning using linear models Çesa-Bianchi & Lugosi (2006), non-linear models with kernels|Kiving
Online Learning:
et al. (2044: y Jin et al. (2010), and deep neural networks Zhou et al. (2012). Online learning algorithms often simply update the model on the new data, and do not consider the past knowledge of the previously seen data to do this online update more efficiently.
However, the online meta learning framework, allow to keep track of previously seen data and with
the "meta" knowledge we can update the online weights to the new data more faster and efficiently.
Continual Learning: A number of prior works on continual learning have addressed catastrophic forgetting McCloskey & Cohen (1989); Li & Hoiem (0017); |Ratcliff (1990); Rajasegaran et al. 2019 2020), removing the need to store all prior data during traiinng Our method does not address catastrophic forgetting for the meta- eeta-training phase, because we must still store all data so as to "replay" for mnta--raining. though it may be possible to discard or sub-sample old data (which we leave to future work). However, our adaptation process is fully online. A number of works perform meta- learning for better continual learning, i.e. learning good continual learning strategies Sh-Shvdivat
ct al. (2017); Nagabandi et al. (2018); Javed & White (2019); Harrison et al. (2019); He al. (2019); Beaulieu et al. (2020). However, these prior methods still perform batch-mode meta-training, In batch-mode meta- tett-training, these methods first collect all of the past data and train model with a meta-learning algorithm (e.g. MAML, Reptile) then take this pretrained weights and tne-tune this model with data from new task. Oh the other hand, our method performs the meta-training incrementally online. In other words, we do not stop at a task boundary and train a model will all data, and re-start again, our method continuously update meta-parameters.
The closest work to ours is the follow the meta-leader (FTML) method Finn et al. (201)) and other online meta-learning methods Yao al. (2020). FTML is a variant of MAML that finetunes each new task in turn, resetting to the meta-trained parameters between every task. While this effectively accelerates acquisition of new tasks. requires ground truth knowledge of task boundaries and, as we show in our experiments, our approach outperforms FTML even when FTML has access to task boundaries and our method does not. Note that the memory requirements for such methods increase with the number of adaptation gradient steps, and this limitation is also shared by our approach. seine-wwithin-oeee
Beethod/Finn
Denevi et al. 2019)) also aims to accelerate online updates by leveraging prior tasks, but still requires knowledge of task boundaries. MOCA Harrison et al. (2200) instead aims to infer the task boundaries. In contrast, our method does not even attempt to find the task boundaries, but directly adapts without them. A number of related works also address continual learning via meta-leamnan leaannea but with the aim of minimizing catastrophic forgetting Gupta et al. (202)); Caccia et al. (2020). Our aim is not to address catastrophic forgetting. Our method also meta-trains from small datasets for thousands of tasks, whereas prior continual learning approaches typically focus on settings with fewer larger tasks (e.g., 10-100 tasks).
FOUNDATIONS
Prior to diving into online meta learning, we first briefly summarize meta learning, model agnostic meta-learning, and online learning in this section.
Meta-learning: Meta-learning address the problem of learning to learn. It uses the knowledge learned from previous tasks to quickly learn new tasks. Meta-learning assumes that the tasks

---

drawn from stationary distribution T ~ P(T). During the meta-training phase (outer-loop), N tasks are assumed to be drawn from this distribution to produce the meta-training set, and the model is trained in such way that, when new task with its own training and test data T (D4, Dif is presented to it at meta-test time, the model should be able to adapt to this task quickly (inner-loop). Using / to denote the meta-trained parameters, the meta-learning objective is:
0 = arg max EDy where T~P(T) (,F)
where Fg is the meta-learned adaptation process that reads in the training set Dj" and outputs spsk-fpecific parameters, prototypes, or features (depending on the method) for the new task T,.
Model-agnostic meta-learuing: In MAML Finn et al. 2017), the inner-loop function is (stochas- tic) gradient descent. Hence, during the MAML inner-loop adaptation, (D( becomes 0 a Lo (0, D)")) (or, more generally, multiple gradient steps). Intuitively, what this means is that meta-training with MAML produces a parameter vector I that can quickly adapt to any task from the meta-training distribution via gradient descent on the task loss. The principle benefits of this is that, when faced with new task that differs from those seen during meta-training, the algorithm "at worst" adapts with regular gradient descent, and at best is massively accelerated by the meta-training.
Online learning: In online learning, the model faces a sequence of loss functions {cr)² and a sequence of data {D,- = {zz, })] for every time step t. The function f x - ÿ maps inputs x to predictions ÿ. The goal of an online learning algorithm is to find a set of parameters for each time step { such that the overall loss between the predictions and the ground truth labels y is minimized over the sequence. This is typically quantified in terms of regret:
T
T
Regrety = C. D.) - Cor. D,).
where, de argming Lo( D,). The first term measures the loss from the online model, and the second term measures the loss of the best possible model on that task. Various online algorithms try to minimize the regret as much possible when introducing new tasks.
4 ONLINE META-LEARNING: PROBLEM STATEMENT AND METHODS
In an online meta-lcarning setting Finn et al. (201),, the model fo observes datapoints one at a time from an online data stream S. Each datapoint consists of an input s'n, where is the task index and m is the index of the datapoint within that task, and label yfn The task changes over time and the model should be able to update the parameters o to minimize the loss at each time step. The goal of online meta-learning according to the specified loss function.
quickly learn each new task T, and perform well as soon as possible
Here, we define task as group of samples based on some discrete variable properties in the samples. For example, it can be grouped by classes, a set of classes, semantic categories or time stamp etc. simple baseline solution would be just train the model on the current task T, We denote this baseline as TFS (Train from Scratch). For every new task, the model simply trains a new set of parameters using all of the data from the current task T, that has been seen so far:
M
OTFS = arg min & M
1
Σ L,(W,, (2'm Vim))
TFS has two issues. First, it requires the task boundaries to be known, which can make it difficult to apply to settings where this information is not available. Second, it does not utilize knowledge from other tasks, which greatly limits its performance even when task boundaries are available.
straightforward way utilize knowledge from other tasks in the online data stream is to store all the seen tasks in large buffer B, and simply keep training the model on all of the seen tasks. We will refer to this baseline method as TOE (Train on Everything):
STOF = arg min
Σ
Σ
L, (#'ns V).
Mt
TOE learns function that fits all of the previously seen samples. However, this function may be far from optimal for the task at hand, because the different tasks may be mutually exclusive. Therefore,

---

fitting single model all of the previously seen tasks might not provide good task-specific model for the current task. A more sophisticated baseline, which we refer to as FTL (Follow the Leader), pre-trains a model on all of the previous tasks, and then fine-tunes it only on the data from the current task. Note that this is subtly different from FIL in the classic online learning setting, due to the difference in problem formulation. This can be achieved by initializing & with pretrained weights up to the previous task éTOF:
M
SPTL = arg min
Σ m==
LolTre (iims vmn)).
(5)
M
Here, for the task i, we take a model that is pre-trained on all previously seen tasks (f ) and fine-tune on the current task data. In this way. FTL can use the past knowledge to more quickly adapt to the new task. However, pre- pre-training on past tasks may not necessarily result in initialization that is conducive to fast adaptation Finn et al. Nicho Nichol al. (2018); ; Nichol & Schulman (2018); Li et al. (207)). Finn el al. Finn et al. (2019) proposed a MAML- -based online meta-learning approach, where MAML is used to meta-train a "meta-leader" model on all previously seen tasks, which is then adapted on all data from the current task. This way, the meta- meta-leader parameters will be much closer
to new task optimal parameters, and because of this it is much faster to adapt to new tasks from the
online data.
Ó'TML arg min Eest, whn [C, MAML (I'm )(m))
(6)
AMAME
1-1 where, PMAML
arg argmin Byy PIIT DI
(6, D(,)),
Here, FTML algorithm first train model using MAML algorithm on last seen t-1 tasks T TT,,T...... TT......TTT1 to generate meta-weight & AMAAM (Eq.7). After this pre-training stage, FTML algorithm fine-tunes the meta-weights using the data from next tasks to find optimal parameters 6FTML for task t. FTL and FTML aim to efficiently use knowledge from past tasks quickly adapt to the new task. However, the pre-trained weights from FTL do not guarantee fast adaptation, and both methods require ground-truie truth task boundaries. This assumption may not be realistic in real-world settings, where the tasks may change gradually and no external information is available to indicate task transitions. Although FTML can enable fast adaptation, the model needs to be "reset" at each task, essentially creating a "branch" on each task. This requires maintaining two independent learning processes: a) an adaptation process, whose result is discarded completely at the end of the task, and b) meta-training process, which does not influence the current task at all, and is only used for forward transfer into future tasks. See Figure | for the branching in standard meta learning setting. This branching at task boundaries and "resetting" after the adaptation makes these the parameter trajectory not continuous, hence we argue that FTL and FTML are not fully online. In this work, our aim is to develop a fully online meta-learning method that continually performs both "fast" updates and "slow" meta-updates, does not periodically "reset" the adapted parameters back to the meta-parameters, and does not require any ground truth knowledge of task boundaries.
FULLY ONLINE META- LEARNING WITHOUT TASK BOUNDARIES
We first discuss the intuition behind how our approach handles online meta-lcarning without task boundaries. In many real-world tasks, we might expect the tasks in the online data stream change gradually. This makes it very hard to draw a clear boundary between the tasks. Therefore, it is necessary to relax task boundary assumption if we want a robust online learner that can work on real-world data stream. Additionally, since nearby data points are most likely to belong to the same or similar task, we would expect adaptation to each new data point to be much faster from a model that has already been adapted to other recent data points.
FOML maintains two separate parameter vectors for the online updates (Ø) and the meta updates (0). Both parameterize the same architecture, such that fo and fg represent the same neural network, but with different weights. The online model continuously reads in the latest datapoints from the online data stream, and updates the parameters Φ in online fashion, without any boundaries or resets. However, simply updating the online model on each data point naïvely will not meta-train to adapt more quickly, and may even result in drift, where the model forgets prior data. Therefore, we also incorporate a regularizer into the online update that is determined by a concurrent meta-learning process (see Fig. 2). Note that FOML only incorporates the meta- peeaa-prasetesss into the online updates via the meta-learned regularization term, without ever resetting the online parameters back to the meta-parame meta-parameters (in contrast, e.g., to FTML Finn et al. (2019)).

---

The meta-updates of previous MAML-based online meta-learning approaches involve sampling data from all of the tasks seen so far, and then updating the meta- partametessssss based on the derivatives of the MAML objective. This provides diverse sampling of tasks for the meta update, though it requires storing all of the seen data Finn et al. (2019). We also use a MAML update for the meta parameters, and also require storing the previously seen data. To this end, will use B to denote buffer containing all of the data seen so far. Each new datapoint is added to B once the label observed.
However, since we do not assume knowledge of task boundaries, we cannot sample entire tasks from B, but instead must sample individual datapoints. We therefore adopt different strategy, which we describe in Section [5.2] as shown in Fig instead of aiming to sample in complete tasks from the data buffer, we simply sample random past datapoints, and meta-train the regularizer so that the online updates retain good performance on all past data. We find that this strategy is effective at accelerating acquisition of future tasks in online meta-learning settings where the tasks are not mutual exclusive. We define both types of updates in detail in the next sections.
Figure 2: Overview of FOML learning: FOMLup-
Data Buffer
New Data
dates the online parameters & using only the most recent K datapoints from the buffer B. Meta-learning learns regularizer, parameterized by meta-parameten 0, via corond-order MAML-style updates. The goal of meta- learning is to make é perform well on randomly sampled prior datapoints after performing K steps with the meta- trained regularizer.
Updne
Meta Updates
④
10
Evaluate
Online Updates
Regularization
5.1
FULLY ONLINE ADAPTATION
At each time step, FOML observes a data point Tt, predicts its label Ut. then receives the true label yo and updates the online parameters. In practice, we make updates after observing N new datapoints (N = 10 in our experiments), so as to reduce the variance of the gradient updates. We create a small dataset Dj with these N datapoints for the time step j. The true label for these datapoints can be from class labels, annotations, rewards, or even self-supervision, though we focus on the supervised classification setting in our experiments.
However, the online updates are based only on the most recent samples, and do not make use of any past data. Therefore, we need some mechanism for the (slower) meta-training process to "transfer" the knowledge it is distilling from the prior tasks into this online parameter vector. We can instantiate such a mechanism by introducing a regularizer into the online parameter update that depends on the
meta-parameters 0, which we denote as R(, 0). While a variety of parameterizations could be used for RØØ 0), we opt for simple squared error term of the form R(Ø, 0) = (Ø 0)2, resulting in the following online update at each step j:
G Φi a1 V のi (C( (C D(,) + ( 0)}
(8)
+-- a Va Φ - L(d D/.)) + , 3, +-
(9)
task specific update
meta directional update
In the case of classification, L is the cross-entropy loss. 01, 31 are hyperparameters.
Next, we discuss how these meta-parament are trained so as to maximize the effectiveness of this regularizer at accelerating adaption to new tasks.
5.2 META-LEA WITHOUT TASK BOUNDARIES
As discussed in the previous section, the online updates to di include regularizer R that transfers knowledge from the parameeasssssrss / into each online update. Additionally, our method maintains buffer B containing all data seen so far, which is used for the meta-update.
In contrast to prior methods, which explicitly draw a training and validation set from the buffer (i.e.. query and support set) and then perform separate "inner loop" update on this training set Finn et al. (2019), our meta-updates recycle the inner loop adaptation that is already performed via the online updates, and therefore
only draw validation set Dm from the buffer B. Specifically, sample a set of N datapoints at random from to form Dm val- We then update the meta uueta-paaaeetess using the gradient of the loss on Dm and the regularizer R after the last K updates on şb.

---

In other words, adjust the meta- metameersmeters in such way that, if an online update is regularized with this meta-weights, then the loss on the online update will be minimized. This can be expressed via following meta update:
0 0 0 V [(o) Diaal +
Σ R(O, de k=0
(10)
Here, the of has dependence on 0 due to the previous online update shown in Eq. Subsequently, 0 and ø² are only related via regularization term, unlike FTML Finn et al. 22119 which sets o0 = every task boundary.
The choice of sampling Dia at random from B has several interpretations. We can interpret this as regularizing @ prevent the online parameters from drifting away from solutions that also work well on the entire data buffer. However, this interpretation is incomplete, since the meta-update doesn't t simply keep & close to a single parameter vector that works well on Dew. but rather changes the regularizer that gradient updates with that regularizer maximally improve performance D This has the effect of actually accelerating how quickly é can adapt to new tasks using this regularizer, so long as past tasks are reasonably representative of prior tasks. We experimentally verify this claim in our experiments. Note, however, that this scheme does assume that the tasks are not mutually exclusive. We summarize the complete algorithm in Algorithm
Algorithm I Online Meta Learning with FOML
1: procedure META TRAINING
2: Require: 0,B, Buffer B, Data stream S
@" <
Initiate
with
4: while Data stream S available do
5:
get new data from online data-stream add new data to the buffer D partition the data into train and validation splits make predictions the train set o¹ -'- -a V Lf (0" + ((((0)1),))
D <- S
6:
B B Dj
7:
Di,, Di + Dj
8:
åer + 1
9:
10:
Dont (Daai)
Evaluate the updated model the validation set B sample random batch from buffer
Din rrandon-samleell
12:
0 ++o 0 ag VoCrus (d);; DmL
13:
i
6 EXPERIMENTAL EVALUATION
Our experiments focus online meta-learning of image classification tasks. In these settings, an effective algorithm should adapt to changing tasks as quickly as possible, rapidly identify when the task has changed, and adjust its parameters accordingly. Furthermore, successful algorithm should make use of past task shifts to meta-learn effectively, thus accelerating the speed with which it adapts to future task changes. In all cases, the algorithm receives one data point at a time, and the task changes periodically. In order to allow for a comparison with prior methods, which generally assume known task boundaries, the data stream is partitioned into discrete tasks, but our algorithm is aware of which datapoint belongs to which tasks or where the boundaries occur. The prior methods, in contrast, are provided with this information, thus giving them an advantage. We first describe the specific task formulations, and then the prior methods that we compare to.
Online meta-learning should adapt each task as quickly as possible, and use data from past to accelerate acquisition of future tasks. We report our learning curves, with one axis corresponding to the number of seen tasks, and the other axis corresponding to the cumulative error rate on that task. This error rate is computed using held-out validation data for the task after adaptation.
We evaluate prior online meta-learning methods and baselines on three different datasets (Rainbow- MNIST. CIFAR 100 and CELEBA). TOE (train everything), TFS (train from scratch), FTL (follow the leader), FTML (follow the meta-lcader) Finn et al. (019), LwF Li & Hoiem (017). Rebuff et al. (2066) and MOCA Harrison al. (2020) the baseline methods we compare against our method FOML. See Section 4 for more detailed description of these methods. Please see Appendix A.2 for more details on the baseline methods.
Datasets: We compare TOE, TFS, FTL, FTML, LwF, iCaLR and FOML on three different datasets. Rainhow-MNIST Finn et al. (2019) created by changing the background color, scale and rotation

---

Rainbow-MNIST
Ordine- CIF AR100
Online-CSL88n
Taxt CreP
Teat Error
Figure 3: Comparison between online algorithms: We compare our method with baselines and prior approaches, including TFS (Train from Scratch), TOE (Train on Everything). FTL (Follow the Leader) and FTML (Follow the Meta Leader). a: Performance relative to the number of tasks seen over the course of online training on the Rainhow-MNIST dataset. As the number of task increases, FOML. achieves lower error rates compared to other methods. We also compare our method with continual learning baselines: LwF Li & Hoiem (2017). (Rebuff al. (2016) and MOCA Haarison al. 2020). MOCA Harrison et al. (2020) archive similar performance to ours at the end the learning. but FOML makes significantly faster progress.
Error rates
on the Ouline CAfine-CIFRROOO 100 dataset. Note that FOML not only achieves lower error rates on average, but also reaches the lowest error (of around 17%) more quickly than the other methods. Performance of FOML on the CELEBA dataset. This datasct contains more than 1000 classes, and we follow the same protocol as in Online-CIFARI00 experiments. Our method, FOML, learns more quickly on this task as well.
of the MNIST dataset. It includes 7 different background colors, 2 scales (full and half) and 4 different rotations. This leads to a total of 56 number of tasks. Each individual task is to classify the images into 10 classes. We use the same partition with 900 samples per each task, as in prior work Fiinn et al. (019). However, this task contains relatively simple images, and only 56 tasks. To create a much longer task sequence with significantly more realistic images, we modified the CIFAR-100 and CELEBA datasets to create an online meta-learning benchmark, which we call online-CIFAR 100 and online-CEEEB,, respectively. Every task is a randomly sampled set of classes, and the goal is to classify whether two images in this set belongs to same class or not. Specifically, each task classes, and every datapoint consists of pair of images, each corresponding to one of the 5 classes for that task. The goal is to predict whether the two images belong to the same class or not. Note that different tasks are not mutually exclusive, which in principle should favor TOE-style method, since meta-learning is known to underperform with non-mutually-exclsst tasks Yin et al. (2019). To make sure the data distribution changes smoothly over tasks, we only change subset of the classes between consecutive tasks.
corresponds
Results on Rainbow-MNIST: As shown in Fig FOML attains the lowest error rate on most tasks in Hhinbow-MNIST. except small segment in the beginning. The performance of TFS is similar across all the tasks, and does not improve over time. This is because it resets its weights every time it encounters a new task, and therefore cannot not gain any advantage from past data. TOE has larger error rates at the start, but as we add more data into the buffer, TOE is able to improve. On the other hand, both FTL and FTML start with similar performance, but FTML achieve much lower error rates at the end of the sequence compared to FTL, consistently with prior work Finn el al. (2019). The final error rates of FOML are around 10%, and it reaches this performance significantly faster than FTML, after less than 20 tasks. Note that FTML also has access to task boundaries, while FOML does not.
We use a Siamese network for this experiment, where each image is fed into a 7-layer convolutional network, and each branch outputs 128 dimensional embedding vector. difference of these vectors are fed into fully connected layer for the final classification. Each task contains data from 5 classes, and each new task introduces three new classes, and retains two of the classes from the previous task, providing a degree of temporal coherence while still requiring each algorithm to handle persistent shift in the task. Fig 3 shows the error rates of various online learning methods, where each method is trained over a sequence of 1200 tasks for online- CIReeRCIIIOOOOO All the methods start with initial error rates of 50%. The tasks are not mutually exclusive, so in principle TOE can attain good performance, but it makes the slowest progress among all the methods, suggesting that simple pretraining is not sufficient accelerate learning. FTL uses a similar pre-training strategy as TOE. However it has an adaptation stage where the model is fine-tuned on the new task. This allows it to make slower progress. As expected from prior work Finn et al. (2019). the meta- progress than FTL. However, FOML makes faster progress on average, and achieves the lowest final emror rate (~ 15%) after sequence of 1200 tasks.
Results on Online CIFARIOO and
procedure used by FTML allows it to make faster

---

Online Meta Learning
Online Neta Learning
5.
Number of Thcks
Number tacke
Figure 4: Ablation experiments: a) We vary the number of online updates K used before the meta-update, to see how it affects the performance of our method. The performance of FOML improves the number of online updates is increased. b) This experiment shows how FOML performs with and without meta updates, confirm that the meta-training is indeed an essential component of our method. With meta-updates, FOML learns more quickly, and performance improves with more tasks.
6.1 ABLATION STUDIES
We perform various ablations by varying the number of online parameters used for the meta-update K, importance of meta-model analysis the properties of our method. For additional ablations, please see the Appendix.
Number online parameters used for the meta-update: Our method periodically updates the online weights and meta weights. The meta-updates involves taking K recent online parameters and updating the meta model via MAML gradient. Therefore, meta-updates depend on the trajectory of the online parameters, In this experiment, we investigate how the performance of FOML changes as we vary the number of parameters used for the uedate (K in Algorithm 1. Fig shows the performance of our method with various values of K: K [1, 2, 3, 5, 10). We can that the performance improves when we update the meta parameters over longer trajectory of online parameters (larger K). We speculate that this is due the longer sequences providing a clearer signal for how the mera-parrmeeters influence online updates over many steps.
Importance of meta update: FOML keeps track of separate online parameters and partaparameterss and each is updated via corresponding updates. However, only the online parameters á are used for evaluation. The meta-parameters 0 only influence them via the regularizer and do not directly affect evaluation. This might raise the question: how important is the contribution of the meta- eeta-parameters to the performance of the algorithm during online training? We train model with and without meta-updates, and the performance is shown in Fig None that, the model without meta-updates is identical to our method, except that the meta-updates themselves are not performed. We can clearly see that the model trained with meta-updates outperforms model trained without meta-updates. The model trained without meta-updates generally does not improve significantly as more tasks are seen, while the model trained with meta-updates improves with each task, and reaches significantly lower final error. This shows that, even though 0 and 6 are decoupled and only connected via a regularization, the meta- leta-ne component of our method really is critical for its good performance.
CONCLUSION
We presented FOML,, a MAML-based algorithm for online meta-learning that does not require ground truth knowledge of task boundaries, and does not require resetting the parameter vector back to the meta-learned parameters for every task. FOML is conceptually simple, maintaining just two parameter vectors over the entire online adaptation process: vector of online parameters Φ., which are updated continually on each new batch of datapoints, and a vector of meta-parameters 0. which are updated correspondingly with meta-updates to accelerate the online adaptation process, and influence the online updates via a regularizer. We find that even a relatively simple task sampling scheme that selects datapoints at random from a buffer of all seen data enables effective meta-training that accelerates the speed with which FOML can adapt to each new task, and we find that FOML reaches a final performance that is comparable to or better than baselines and prior methods, while learning to adapt quickly to new tasks significantly faster. While our work focuses on supervised classification problems, a particularly exciting direction for future work is to extend such online meta-le leeanle methods to other types of online supervision that may be more readily available, including self-supervision and prediction, so that models equipped with online meta-learning can continually improve they see more of the world.

---

REFERENCES
Maruan Al- Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. arXiv preprint arXiv: wrXiv:/710.036411 2017.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How train your maml. arXiv preprint arXiv:/8/0.09500, 2018.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. arXiv preprint arXiv:2002.09557, 2020.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, et al. Online fast adaptation and knowledge accumulation: new approach to continual learning. arXiv preprint arXiv:2003.05856 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021.
Nicolo Scsa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within-online meta-learning. In ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), volume 32, pp. 1-11. Neural Information Processing Systems (NeurIPS 2019), 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1126-1135, PMLR, 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning, In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
1920-1930. PMLR, 09-15 Jun 2019.
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. arXiv preprint 1909. 2019.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-maml: Look-ahead meta learning for continual learning, arXiv preprint arXiv:2007.13904 2020.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. arXiv preprint arXiv: 2019.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-lcarning without tasks. Advances in neural information processing systems, 33, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. corr 1bs/1512.03385 (2015), 2015.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint (90.::9900002011 2019,
Khurram Javed and Martha White. Meta-learning representations for continual learning. arXiv preprint arXiv: 12588, 2019.
Ghassen Jerfel, Erin Grant, Thomas Griffiths, and Katherine Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. arXiv preprint arXiv: 122.00000 2018.
Rong Jin, Steven CH Hoi, and Tianbao Yang. Online multiple kernel learning: Algorithms and mistake bounds. In International conference on algorithmic learning theory, pp. 390-404 Springer, 2010.

---

Jyrki Kivinen, Alexander J Smola, and Robert C Williamson. Online learning with kernels. IEEE transactions on signal processing, 52(8))21655223) 2176, 2004.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. ICML deep learning workshop, volume 2. Lille, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Da Li and Timothy Hospedales. Online meta-learning for multi-source and semi-supervised domain adaptation. In European Conference on Computer Vision, pp. 382-403. Springer, 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv: 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(1 40(12):2935-2947, 2017.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pp. 2563-2563. PMLR, 2017.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl. arXiv preprint arXiv:1812.07671 2018.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2(3):4 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02990 2018.
Funbyung Park and Junier B Oliva. Meta-curvature. arXiv preprint arXiv: N:XXv:/902.0335555 2019.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for incremental learning. Advances in Neural Information Processing Systems, 2019.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13588-1355 13597, 2020.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2))285, 1990.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, and Christoph H Lampert. icarl: Incremental classi- fier and representation learning. coir abs/1611.07725 (2016). arXiv preprint arXiv:1611.07722 2016.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with -aamory-augmented neural networks. In International conference machine learning, pp. 1842-1850. PMLR, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for fho-shot learning. arXiv preprint 1703.00700..5511 2017.

---

Xingyou Song, Wenbo Gao, Yuxiang Yang, Krzysztof Choromanski, Aldo Pacchiano, and Yunhao Tang. Es-maml: Simple hessian free meta learning. arXiv preprint arXiv:1910.012/1. 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29:3630-366 --33,, 2016.
Flood Sung Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot (2018) 2017.
Huaxiu Yao, Yingbo Zhou, Mehrdad Mahdavi, Zhenhui Li, Richard Socher, and Caiming Xiong. Online structured meta-learning, arXiv preprint arXiv:2010.11545, 2020.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. arXiv preprint arXiv:1912.03822 2019.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey Hinton. Lookahead optimizer: k steps forward, 1 step back. Advances in neural information processing systems, 32, 2019.
Guanyu Zhou, Kihyuk Sohn, and Honglak Lee. Online incremental feature learning with denoising autoencoders, In Artificial intelligence and statistics, pp. 1453-1461. PMLR, 2012.


Reviews:
[{"review_id": "hfX7LBeKkZJ", "reviewer": "Reviewer_71fS", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to perform meta-learning without task boundaries.", "summary_of_the_review": "I am puzzled with this submission as I cannot find anything to comment on. For instance, the authors introduce the notion of regret (by the way, it seems from it that the goal is to maximize the loss functions ?! See also eq.(1)), but for what reason? It is never used afterwards, neither to state a bound or motivate the heuristics proposed. Clearly, the contribution proposed is not enough to warrant acceptance at ICLR.", "strength_and_weaknesses": "**Strengths**\n- the paper is globally clear and well writen\n- the topic is interesting to the ICLR community\n\n**Weaknesses**\n- absence of any theoretical justification", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UIzAveYfZ7", "reviewer": "Reviewer_a6xH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an extension of model adaptation via meta learning in an online fashion. This paper particularly targets the problem model adaptation in the environment with changing tasks and input distributions. With the ability for fast adaptation, the proposed method called as Fully-Online Meta-Learning (FOML) adapt the model parameters quickly and continuously for any incoming tasks. The experiments show that FOML can continuously adapt to a new task quickly and beat prior methods on meta-learning on several challenging datasets  e.g., Rainbow-MNIST, CIFAR100, and CELEBA.", "summary_of_the_review": "This paper has some potentials to address online meta-learning problems with no task boundaries. However, there are some unclear messages and problem descriptions in text. Also, there are some inconsistencies between the claims and the experiments to prove them. ", "strength_and_weaknesses": "\nStrengths:\n- The problem of learning from constant stream of data with distinct tasks in this work is challenging and relevant to rapid online adaptation of neural networks for many real-world applications e.g., traffic predictions systems.\n-  The formulation for online meta-learning is somewhat novel and uses no task boundary knowledge to learn from newly incoming tasks.\n- The experiments show the effectiveness and superiority of the proposed method to tackle online meta-learning problems compared to other prior arts e.g., FTML, LwF, iCARL, and MOCA.\n\n\nWeaknesses:\n- The definition and the role of task boundaries are not very clear for fully online meta learning and prior methods. Is this the boundary that has a task id indicating a specific task? What are “well-separated tasks”? How do the tasks for FOML have no boundaries? As understood from the FTML paper, Standard Online Meta Learning also does not need this additional knowledge to run the algorithm and adapt the model.  Does “task boundary” here mean that the evaluation task might come from past tasks i.e., not necessarily from the currently training tasks? Also, if there is no specified boundaries between tasks, can we derive the problem as standard online learning?\n- The “gradual changes of tasks” also need to be explained in more detail. If there are no clear boundaries, how are tasks verified changed overtime? In my opinion, the motivation described in introduction does not perfectly align with the setting. It is vague to capture the gradual changes but discarding the concept of continual learning. The strong assumption in meta learning is about the stationarity of the data distribution. The gradual change of its distribution is known as continual learning problems.\n- The meta directional update is conceptually similar to the formulation in Elastic Weight Consolidation (EWC) and Reptile. There is no head-to-head discussion with these two methods and direct comparison in terms of formulation. Especially, This work has no empirical comparison with these methods (that can be applied directly to the problem at hand).\n- It is not straightforward to undertand the empirical results in Figure 3 with a little description on the experimental setup. The competitive advantage of FOML is to learn without task boundaries, but the problem still has task boundaries and the same partition as in the previous work  by Finn et al. (2019) to compare between online algorithms. In my opinion, even though the task information is not explicitly given for learning purposes but FOML is exposed with task information indirectly due to the setup described in Section 6, especially with the task partitions.\n- There is a big question: why does TOE consistently perform poorly on average among all other methods? Why is this method performing really bad given the fact that it can access all past tasks? As mentioned in the paper, this approach trains the model using all tasks in the large buffer. \n- There is minimum information about LwF, iCARL, and MOCA on how they are implemented for the online meta-learning setup. Please elaborate on how to adapt these methods for the online meta-learning setting. The supplementary material also does not contain this information.\n- Also, the use of a Siamese Network needs to be explained for reproducibility. Moreover, there are no citations for this type of network as well.\n- There is an inconsistency in writing the proposed method. In formulation, there is a number $K$ for regularization but it is omitted in Algorithm 1.\n- The paper needs some corrections:\n-- Please fix the manuscript with numbered equations. The number stops at 5.\n-- Typo and grammatical errors: varaint, a online data stream, evaulate.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GF8Slp8lMb", "reviewer": "Reviewer_VVzJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to tackle a new online meta learning setting where the task boundaries are not known (no access or costly to obtain) or do not exist (e.g. gradual change). \n\nMeta learning consists of the meta-training step and the adaptation step. During the adaptation step, instead of re-setting the task parameter to the meta-parameter at the beginning of the task (e.g. requiring to know the task boundary to know when to reset), the proposed method uses regularisation to pull parameter back gradually and softly (see Fig. 1) thus does not require to know the task boundaries. \n\nExperiments show that the proposed method can learn new tasks faster than the state-of-the-art online learning methods on Rainbow-MNIST, CIFAR100 and CELEBA datasets.", "summary_of_the_review": "I am happy to accept the paper at the moment and open to change when I see other reviews and rebuttal.", "strength_and_weaknesses": "Strength: \n1) it tackles a new problem (or at least in a new setting);\n2) the proposed method makes sense;\n3) presentation is clear;\n\nWeakness:\n1) more ablation study will be helpful particularly because this is a new setting. For example, an ablation study to show when task boundaries are accessible, can the proposed method discover/recover task boundaries well? One way to check is to see if the regularised pulled back task parameter intersects with the meta-parameter at or near the ground truth task boundaries. \n2) a couple of concrete real applications to show task boundaries are not accessible. Note task boundaries are different from distribution shifts, and in most cases the users know the task has changed. I struggle to come up with a real example of task boundaries not accessible in real applications. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9cEsTSTsV5", "reviewer": "Reviewer_K4eM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method to continually adapt a deep network to smoothly-changing tasks, by meta-learning a regularizer online. This regularizer consists of a L2 loss that binds the online-adapted parameters and (meta-learned) reference parameters together.\n", "summary_of_the_review": "\nThe paper seems to present a nice method but requires some work to make the explanations more clear and correct notation mistakes. The missing appendix gives the impression that the submission was a bit rushed. It could also be made more complete in terms of related work. As it is, this paper is very borderline, but can be published if the authors can visibly correct all the mentioned mistakes, and improve it generally.\n", "strength_and_weaknesses": "\nStrengths: The motivation for the method is clear and makes sense, given the problem setting of online (continual) learning. The method seems novel, and has good experimental results in the analyzed settings.\n\nWeaknesses:\n- The main weakness is a series of mistakes in the equations, which makes them hard to parse, and the textual description of the method not being very clear.\n- The appendix seems to be missing (!) so important implementation details are missing; this harms the reproducibility.\n- Some discussion of related work is missing, but this is a smaller issue.\n- Comparison with the most related Continual Learning methods would be better, namely those that also train online with a buffer of past samples: Tiny Episodic Memories, and GDumb.\n\nThe paper would benefit from discussing the relation between the proposed method, of a fast-learning network trailed by a (meta-learned) slow-learning \"anchor\" network, and others with a similar set-up. One such example is:\n- Zhang et al., \"Lookahead Optimizer: k steps forward, 1 step back\", NeurIPS 2019\nOther works that focus on learning by \"self-distillation\" (where usually the two models are described as teacher and student) should also be discussed:\n- Caron et al., \"Emerging Properties in Self-Supervised Vision Transformers\", 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject