Fully Online Meta-Learning Without Task Boundaries
Authors: Jathushan Rajasegaran
Chelsea Finn
Sergey Levine
Venue: ICLR 2022 Submitted (2022)

Abstract: While deep networks can learn complex classifiers and models, many applications require models that continually adapt to changing input distributions, changing tasks, and changing environmental conditions. Indeed, this ability to continuously accrue knowledge and use the past experience to learn new tasks quickly in continual settings is one of the key properties of an intelligent system. For complex and high-dimensional problems, simply updating the model continually with standard learning algorithms such as gradient descent may result in slow adaptation. Meta-learning can provide a powerful tool to accelerate adaptation but is conventionally studied in batch settings. In this paper, we study how meta-learning can be applied to tackle online problems of this nature, simultaneously adapting to online to changing tasks and input distributions and meta-training the model in order to adapt more quickly in the future. Extending meta-learning into the online setting presents its own challenges, and although several prior methods have studied related problems, they generally require a discrete notion of tasks, with known ground-truth task boundaries. Such methods typically adapt to each task in sequence, resetting the model between tasks, rather than adapting continuously across tasks. In many real-world settings, such discrete boundaries are unavailable, and may not even exist. To address these settings, we propose a Fully Online Meta-Learning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries and stays fully online without resetting back to pre-trained weights. Our experiments show that FOML was able to learn new tasks faster than the state-of-the-art online learning methods on Rainbow-MNIST, and CIFAR100 datasets.

Under review as a conference paper at ICLR 2022
FULLY ONLINE META-LEARNING WITHOUT TASK BOUNDARIES
Anonymous authors Paper under double-blind review
ABSTRACT
While deep networks can leam complex functions such as classifiers, detectors, and trackers, many applications require models that continually adapt to changing input distributions, changing tasks, and changing environmental conditions. Indeed, this ability to continuously accrue knowledge and use past experience to learn new tasks quickly in continual settings is one of the key properties of an intelligent system. For complex and high-dimensional problems, simply updating the model continually with standard learning algorithms such gradient descent may result in slow adaptation. Meta-learning can provide a powerful tool to accelerate adaptation yet is conventionally studied in batch settings. In this paper, we study how meta- learning can be applied to tackle online problems of this nature, simultaneously adapting changing tasks and input distributions and meta- peea-training the model in order to adapt more quickly in the future. Extending meta-learning into the online setting presents own challenges, and although several prior methods have studied related problems, they generally require a discrete notion of tasks, with known ground-truth task boundaries. Such methods typically adapt to each task in sequence, resetting the model between tasks, rather than adapting continuously across tasks. In many real-world settings, such discrete boundaries are unavailable, and may not even exist. To address these settings, propose Fully Online Meta- Learning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries and stays fully online without resetting back pre- trained weights. Our experiments show that FOMI. was able to learn new tasks faster than the oftte-of-thhaat online learning methods on Rainbow-MNIST, and CIFARIO0 100 datasets.
INTRODUCTION
Flexibility and rapid adaptation are hallmark of intelligence: humans can not only solve complex problems, but they can also figure out how to solve them very rapidly, as compared to our current machine learning algorithms. Such rapid adaptation is crucial for both humans and computers: for humans, it is crucial for survival in changing natural environments, and it is also crucial for agents that classify photographs on the Internet, interpret text, control autonomous vehicles, and generally make accurate predictions with rapidly changing real-world data. While deep neural networks are remarkably effective for learning and representing accurate models (He al.] 2015 Krizhevsky et al.) 2012; Simonyan & Zisserman 2014| SSzeedyy et al., 2015), they are comparatively unimpressive when comes to adaptability, due their computational and data requirements. Meta-learning in principle mitigates this problem, by leveraging the generalization power of neural networks to accelerate adaptation new tasks (Finn et al. [019 [L et al.) 2017 Nichol et al. 2018 Nichol
& Schulman 2018 Park & Oliva, 2019 Antomiou et al. 2018). However, standard meta- meta-learming
algorithms operate in batch mode, making them poorly suited for continuously evolving environments.
More recently, online meta-learning methods have been proposed with the goal of enabling continual
adaptation (Finn et al., 2019; Jerfel et al.] 2018 Yao et al., 2020; Nagabandi et al., 2018 Li & Hospedales] (0200), where a constant stream of data from distinct tasks is used for both adaptation and meta-training In this scheme, meta-training is used to accelerate how quickly the network can adapt each new task it sees, and simultanously This further accelerates how quickly each subsequent task can be acquired. However, current online meta-learning methods fall short of the goal of creating effective adaptation system for online data
that data from each new task for meta-training.

---

Under review as a conference paper at ICLR 2022
Standard Online Meta Learning
Fully Online Mcta Learning
Hey clise and
Tast
Task
Tadk
Tink
Time
Task shift gradrally, no known boundaries Timee
In standard online meta-learning (e.g.. FTML (Finn al. 2 shown on the left, adaptation is performed on one task a time, and the algorithm "resets" the adaptation process at task boundaries. For example, a MAML-based method would reset the current parameters back to the meta-trained parameters. In our approach (right), knowledge of task boundaries is not required, and the algorithm continually keeps track of online parameters e and meta parameters 0. The online parameters are simply updated the latest data, and the mera--aasmeters are updated "pull" the online parameters toward fast-adapting solutions via MAML-style meta-update.
Figure Comparison of standard online meta-learning and FOML:
in several ways: (1) they typically require task boundaries in the data stream to be known, making them ill-suited to settings where task boundaries are ill-defined and tasks change or evolve gradually, common tread in real-world; (2) as a result, they typically re-adapt from the meta-trained model on each task, resulting in very "discrete" mode of operation, where the model adapts to a task, then resets, then adapts to a new one. These limitations restrict the applicability of current online meta-learning methods to real-world settings. We argue that task boundary assumption is somewhat artificial online settings, where the stream of incoming data cleanly partitioned into discrete and well-separated tasks presented in sequence. In this paper. we instead develop fully online meta-lcarning approach, which does not assume knowledge of task boundaries and does not re-adapt for every new task from the meta parameters.
Standard meta-learning methods consist of a meta-training phase, typically done with standard SGD, and an "inner loop" adaptation phase, which computes task specific parameter & for the task T, from a support set to make accurate predictions on a query set. For example, in model-agnostic meta- learning (MAML), adaptation consists of taking few gradient steps the support set, starting from the meta-trained parameter vector 0, leading to a set of post-adaptation parameters, and meta-training optimizes the meta-trained parameters @ so that these gradient steps lead to good results. Previous extensions of such approaches into the online setting typically observe one task at a time, adapt to that task (i.e., compute post-adaptation parameters on it), and then reset Φ; back to the meta-trained parameters A at the beginning of the next task. Thus, the algorithm repeatedly adapts, resets at the task boundary, adapts again, and repeats. This is illustrated in Figure 1 (left). However, in many realistic settings, the task boundaries are not known, and instead the tasks shift gradually over time. The discrete "resetting" procedure is a poor fit such cases, and we would like to simply continue adapting the weights over time without ever resetting back to the meta-traines parameters, but still benefit from a concurrent meta- trainit process. For example, a meta- vreta-trained image-tagging model on the Internet (e.g., tagging friends in photographs) might gradually adapt changing patterns and preferences of its users over time, where it would be unnatural to assume discrete shifts in what users want tag. Similarly, traffic prediction system might adapt to changing traffic patterns, including periodic changes due to seasons, and unexpected changes due to shifting economic conditions, weather, and traffic accidents. In this spirit, our method does not require any knowledge on the task boundaries as well as stays fully-online through out the learning.
The main contribution of our paper is FOML (fully online meta-learning), an online meta-learning algorithm that continually updates its online parameters with each new datapoint or batch of datapoints, while simultaneously performing meta veeaaggradeet updates on a separate set of meta-parameters using a buffer of previously seen data. FOML does not require ground truth knowledge of task boundaries, and does not reset the online parameters back to the meta-parameters between tasks, instead updating the online parameters continually in fully online fashion. We compare FOML empirically to strong baselines and a ofate-of-the-aae art prior online meta-learning method, showing that FOML learns to adapt more quickly, and achieves lower error rates, both on simple sequential image classification task from prior work and a more complex benchmark that we propose based on the CIFAR 100 dataset, with sequence of 1 1200 tasks.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
Online meta- weta-leerning brings together ideas from online learning, meta learning, and continual learning. with the aim of adapting quickly to each new task while simultaneously learning how to adapt even more quickly in the future. We discuss these three sets of approaches next.
Meta Learning: Meta learning methods try to learn the high-level context of the data, to behave well on new tasks (Learning to learn). These methods involve learning a metric space (Koch et al.
2013 Vinyals et al., 2016]] [Snell et al. 2017] Yang et al.] (2077), gradient based updates (Finn et al.
2017 Li al. 2017, & Olival [019 Nichol et al. 20188] Nichol & Schulman 2018). some
specific architecture designs (Santoro et al.
2016 Munkhdalai & Yu 2017: & Larochelle 2016).
In this work, we mainly interested in gradient based meta learning methods for online learning.
MAML (Finn et al.
(017) and its variants (Nichol et al. 2018 Nichol & Schulman 2018]] Li et al.)
2017, Park & Oliva 2019; Antoniou et al. 2018) first meta train the models in such a way that the
meta parameters are close to the optimal task specific parameters (good initialization). This way, adaptation becomes faster when fine tuning from the meta parameters. However, directly adapting this approach into an online setting will require more relaxation on online learning assumptions, such as access to task boundaries and resetting back and froth from meta parameters. Our method does not require knowledge of task boundaries.
Online learning methods update their models based on the stream of data sequen- tially. There are various works on online learning using linear models Cessa-Bianchi & Lugosi, (2006). non-linear models with kernels (Kivinen et al. 2004 Jin et al. 2010), and deep neural networks (Zhou et al. 2012). Online learning algorithms often simply update the model on the new data, and do not consider the past knowledge of the previously seen data to do this online update more efficiently. However, the online meta learning framework, allow us to keep track of previously seen data and with the "meta" knowledge we can update the online weights to the new data more faster and efficiently.
Online Learning:
Continual Learning: A number of prior works on continual learning have addressed catastrophic forgetting (McCloskey & Cohen [989; Li& Hoiem 2017)) Ratcliff 1990, Rajasegaran et al.) 2019)] 2020), removing the need to store all prior data during training. Our method does not address catastrophic forgetting for the meta-training phase, because we must still store all data so as to "replay" it for meta-training, though it may be possible to discard sub-sample old data (which we leave to future work). However, our adaptation process is fully online. A number of works perform meta-learning for better continual learning, i.c. learning good continual learning strategies (AI- Shedivat et al. 2017: Nagabandi et al. 2018 Javed & White 2019: Harrison et al. 2019 He et al. 2019][Beaulie et al. 2020p. However, these prior methods still perform bateh-mode meta- training. while our method also performs the meta-training itself incrementally online, without task boundaries.
The closest work to ours is the follow the meta-leader (FTML) method (Finn et al. 2019) and its variants (Yao et al. 2020). FTML. is a varaint of MAMI. that finetunes to each new task in turn, resetting to the meta- trained parameters between every task. While this method effectively accelerates acquisition of new tasks, it requires ground truth knowledge of task boundaries and, our experiments, our approach outperforms FTML even when FTML has access to task boundaries and our method does not. One of the limitations of these gradinet based meta learning methods is that, the memory requiment linearly increases with the number of inner loop updates. Ours has the same limitations MAML and FTML, and have same order of memory requirement. Further, there are some recent works which try to address Online meta learning Harrison et al. 2020); Denevi et al.. (2019) in a similar setting to ours. Donline-within-Onll adaptation by learning from old tasks. However, OWO Denevi et al. (2019 requires the knowledge about the task boundaries, while our method does not need any extra information about the task boundaries. MOCA Harrison et al. (2020) address very similar problem as ours, yet it tries to find the task boundaries or change points from past data. On the other hand our method can work even without estimating the change points. Additionaly Gupta et al. (2020): [accia et al. (2020) address on solving continual learning with meta learning. Both works tries to minimize catastrophic forgetting, by precondition the meta gradients. While FOML focuses rapidly learning new tasks, it can learn from 1000+ tasks, while continual learning methods usually has 10 to 100 tasks.
show in
Denevi et al. (2019) tries to learn fast
FOUNDATIONS
Prior to diving into online meta learning, we first briefly summarize meta learning, model agnostic meta-learning, and online learning in this section.
3

---

Under review as a conference paper at ICLR 2022
Meta-learning address the problem of learning to learn. It uses the knowledge learned from previous tasks to quickly learn new tasks. Meta-leaming assumes that the tasks are drawn from stationary distribution T ~ P(T). During the meta-training phase (outer-loop), N tasks are assumed to be drawn from this distribution to produce the meta-training set, and the model is trained in such way that, when presented to it at meta-test time, the model should be able to adapt to this task quickly (inner-loop). Using 0 denote the meta-trained parameters, the meta-learning objective is:
Meta-learning:
task with its own training and test data T = (D4, Djj is
8 argmin EDy where T-P(T) (C(F((D)), (FF))
9
where Fg is the meta-learned adaptation process that reads in the training set Dj' and outputs task-specific parameters, prototypes, features (depending the method) for the new task T,-
Model-agnostic meta-learning: In MAML (Finn et al.) 2017). the inner-loop function is (stochastic) gradient decent. Hence, during the MAML inner-loop adaptation, (D( becomes 0 aV (0, D (or, more generally, multiple gradient steps). Intuitively, what this means is that meta-training with MAML produces parameter vector 8 that can quickly adapt to any task from the meta- veta-training distribution via gradient descent on the task loss. One of the principle benefits of this is that, when faced with new task that differs from those seen during teta-training, the algorithm "at worst" adapts with regular gradient descent, and at best is massively accelerated by the meta-training.
Online learning: In online learning. the model faces a sequence of loss functions {Le) and a sequence of data {D, = {z, y)} }/2 for every time step t. The function f - ÿ maps inputs a to predictions ý. The goal of an online learning algorithm is to find a set of parameters for each time step { such that the overall loss between the predictions and the ground truth labels is minimized over the sequence. This is typically quantified in terms of regret:
T
T
Regret, = E Coor..D.) 1=1
(2)
where, de argming Lo( D,). The first term measures the loss from the online model, and the second term measures the loss of the best possible model on that task. Various online algorithms try to minimize the regret as much as possible when introducing new tasks.
4 ONLINE REEAA-LERNNING: PROBLEM STATEMENT AND METHODS
In an online meta-learning setting FFnn et al. 2019), the model S observes datapoints one at time from a online data stream S. Each datapoint consists of an input z'm.. where is the task index and m is the index of the datapoint within that task, and label y'm. The task changes over time and the model should be able to update the parameters to minimize the loss at each time step. The goal of online meta-learning is to quickly learn each new task T, and perform well as soon as possible according the specified loss function.
A simple baseline solution would be to just train the model on the current task Te- We denote this baseline TFS (Train from Scratch). For every new task, the model simply trains a new set of parameters using all of the data from the current task Te that has been seen so far:
1 ΦTFS argmin M
Σ L, (I'ms v-m)). m=1
TFS has two issues. First, it requires the task boundaries to be known, which can make it difficult to apply to settings where this information is not available. Second, it does not utilize knowledge from other tasks, which greatly limits its performance even when task boundaries are available.
straightforward way to utilize knowledge from other tasks of the online data stream is to store all the seen tasks in large buffer B, and simply keep training the model on all of the seen tasks. We will refer to this baseline method as TOE (Train on Everything):
M
TO argmii 1
Li (4 Vim)).
ML
m=1
4

---

Under review as a conference paper at ICLR 2022
TOE learns function that fits all of the previously seen samples. However, this function may be far from optimal for the task at hand, because the different tasks may be mutually exclusive. Therefore, fitting single model all of the previously seen tasks might not provide good task-specific model for the current task. A more sophisticated baseline, which we refer to as FTI. (Follow the Leader), pre-trains a model on all of the previous tasks, and then fine-tunes it only on the data from the current task. Note that this is subtly different from FTL in the classic online learning setting, due to the difference in problem formulation. This can be achieved by initializing & with pretrained weights up to the previous task PTOE
SFTL argmin 1 M Σ LLOTOE (o'me Vnm)).
Here, for the task t, we take model that is pre-trained on all previously seen tasks (fo) and fine-tune on the current task data. In this way, FTL can use the past knowledge to more quickly adapt to the new task. However, vre-training on past tasks may not necessarily result in an initialization that is conducive to fast adaptation (Finn et al. 2017: Nichol et al. 2018 Nichol & Schulman 2018 Li al. 2017). Finn al. (Finn et al. 2019) proposed MAML- based online meta-learning approach, where MAML used to meta-train a "meta-leader" model on all previously seen tasks, which is then adapted on all data from the current task. This way. the meta-leader parameters will be much closer to new task optimal parameters, and because of this it is much faster to adapt to new tasks from the
online data.
@FTML = argmin E (.r J-r
(f Fm)
d-1 where, OMAML = argmin BrgminBr, Bry Ve,(
FTL and FTML try to efficiently use knowledge from past tasks to quickly adapt to the new task. However, pre-trained weights from FTL do not guarantee fast adaptation, and both methods require ground-truth knowledge of task boundaries. This knowledge may not be realistic in many weal-world settings, where the tasks may change gradually and no external information available to indicate task transitions. Although FTML can enable fast adaptation, the model needs to be "reset" at each task, essentially creating a "branch" on each task and maintaining two independent learning processes: an adaptation process, whose result is discarded completely at the end of the task, and a meta-training process, which does not influence the current task at all, and is only used for forward transfer into future tasks. For this reason, we argue that FTL and FTML are not fully online. In this work, our aim is to develop a fully online meta-learning method that continually performs both "fast" updates and "slow" uudates,, does not need to periodically "reset" the adapted parameters back to the meta- meta-paramettes, and does not require any ground truth knowledge of task boundaries.
5 FULLY ONLINE META-LEARNING WITHOUT TASK BOUNDARIES
We first discuss the intuition behind how our approach handles online meta-learning without task boundaries. In many real-world tasks, we might expect the tasks in the online data stream to change gradually. This makes it very hard to draw a clear boundary between the tasks. Therefore, it is necessary relax task boundary assumption if we want a robust online learner that can work real-world data stream. Additionally, since nearby data points are most likely to belong to the same or similar task, we would expect adaptation to each new data point to be much faster from a model that has already been adapted to other recent data points.
FOML maintains two separate parameter vectors for the online updates (Ø) and the meta updates (0). Both parameterize the same architecture, such that fo and fg represent the same neural network, but with different weights. The online model continuously reads in the latest datapoints from the online data stream, and updates the parameters á in online fashion, without any boundaries or resets. However, simply updating the online model on each data point naïvely will not meta-train to adapt more quickly, and may even result in drift, where the model forgets prior data. Therefore, also incorporate a regularizer into the online update that is determined by concurrent meta-learning process (see Fig. Note that FOML only incorporates the meta-pan parameters into the online updates via the meta-learned regularization term, without ever resetting the online parameters back to the meta- pattmaaarreeeess (in contrast, e.g., to FTML (Finn et al. 2019)).
The meta-updates of previous MAML-based online meta-learning approaches involve sampling data from all of the tasks seen far, and then updating the meta- parameters 0 based on the derivatives
5

---

Under review as a conference paper at ICLR 2022
of the MAML objective. This provides diverse sampling of tasks for the meta update, though it requires storing all of the seen data (Finn et al. 2019). We also use a MAML--tyle update for the meta parameters, and also require storing the previously seen data. To this end, will use B to denote a buffer containing all of the data seen so far. Each new datapoint is added to B once the label is observed. However, since entire tasks from B. but instead must sample individual datapoints. We therefore adopt different strategy, which we describe in Section 5.2] as shown in Fig instead of aiming to sample in complete tasks from the data buffer, we simply sample random past datapoints, and meta-train the regularizer so that the online updates retain good performance on all past data. We find that this strategy is effective at accelerating acquisition of future tasks in online meta-learning settings where the tasks are not mutual exclusive. We define both types of updates in detail in the next sections.
do not assume knowledge of task boundaries,
cannot sample
Figure 2: Overview of FOML learning: FOML up- dates the online parameters & using only the most recent K datapoints from the buffer B. Meta-learning learns a regularizer, parameterized by meta- maraa-aaameters 6, via second-order MAML-style updates. The goal of meta- learning is to make perform well on randomly sampled prior datapoints after performing K steps with the meta- trained regularizer.
Data Buffer
New Dara
Updae Daffer
Meta Updates
Evaluate
Online Updates
Regularization
5.1
FULLY ONLINE ADAPTATION
At each time step, FOML observes a data point 24, predicts its label it, then receives the true label ye and updates the online parameters. In practice, we make updates after observing .N new datapoints (N = 10 in our experiments), as reduce the variance of the gradient updates. We create a small dataset D/, with these N datapoints for the time step j. The true label for these datapoints can be from class labels, annotations, rewards, or even self-supervision, though we focus on the supervised classification setting in our experiments.
However, the online updates are based only on the most recent samples, and do not make use of any past data. Therefore, we need some mechanism for the (slower) meta-training process to "transfer" the knowledge it is distilling from the prior tasks into this online parameter vector. We can instantiate such a mechanism by introducing a regularizer into the online parameter update that depends on the meta- parameters 0, which we denote as R(Ø, 0). While a variety of parameterizations could be used for R( 8), we opt for simple squared error term of the form R(o, 0) = (Ø- 0)2, resulting in the following online update at each step j:
,0} V 0e (C( ( + 3, R
V C( D(o)) 2a, 31 Φ
task specific update
meta directional update
In the case of classification, C is the cross-entropy loss. 01 81 are hyperparameters. Next, we discuss how these meta-parameters are trained so as to maximize the effectiveness of this regularizer at accelerating adaption to new tasks.
5.2 META-LEARNING WITHOUT TASK BOUNDARIES
As discussed in the previous section, the online updates to di include a regularizer R that transfers knowledge from the meta-paaameters 0 into each online update. Additionally, our method maintains a buffer B containing all data seen so far, which is used for the meta-update. In contrast to prior methods, which explicitly draw a training and validation set from the buffer (i.e., a query and support set) and then perform a separate "inner loop" update this training set (Finn et al. 2019), our meta-updates recycle the inner loop adaptation that is already performed via the online updates, and therefore we only draw validation set Dm from the buffer B. Specifically, we sample a set of N datapoints at random from B to form DT We then update the meta- meta-parameters 0 using the gradient ó. In other words, adjust the
of the loss on Dm and the regularizer R after the last K updates meta-paras parameeersss in such a way that, if an online update is regularized with this meta-wes meia-ssights,, then the loss the online update will be minimized. This can be expressed via following meta update:
K
0= -0agv Vo E(O) ( + B2
Σ k=0
R(O, 0i -i)

---

(Finn et al. 2019, which sets Under review as a conference paper at ICLR 2022
Here, I and ø² are only related via regularization term, unlike FTML 6 = 0 at every task boundary.
The choice of sampling n at random from B has several interpretations. We interpret this as regularizing ø to prevent the online parameters from drifting away from solutions that also work well the entire data buffer. However, this interpretation is incomplete, since the meta-update doesn't 't simply keep & close to a single parameter vector that works well on D but rather changes the regularizer so that gradient updates with that regularizer maximally improve performance on Dim This has the effect of actually accelerating how quickly á can adapt to new tasks using this regularizer, so long as past tasks are reasonably representative of prior tasks. We experimentally verify this claim in our experiments. Note, however, that this scheme does assume that the tasks are not mutually exclusive.
future work, it would also interesting to explore more sophisticated strategies for sampling Di for example by leveraging temporal coherence and drawing N sequential points, which are more likely belong to the same task. We summarize the complete algorithm in Algorithm
Algorithm Online Meta Learning with FOML.
1: procedure META TRAINING
2: Require: 0, 6, Buffer B, Data stream S
3: 6 +- Ø
4: while Data stream S available do
5:
D get new data from online data-stream add new data to the buffer > partition the data into train and validation splits make predictions on the train set Φ 0-- an Va (frakk D(r)) + (
D <
6:
B x B+D
7:
Dír, D D D
8:
Der + for (
9:
10:
Dual 2 (Dina)
D Evaluate the updated model on the validation set sample random batch from buffer
11:
Dm ~ random srandom-ssmpee(6
12:
0 - 0 0z V (rask ( Din )
B1 EK R(0s-k,0))
13:
j jtj+l j
6 EXPERIMENTAL EVALUATION
Our experiments focus on online meta-learning tasks based on supervised classification. In these settings, an effective algorithm should adapt to changing tasks as quickly as possible, rapidly identifying when the task has changed and adjusting its parameters accordingly. Furthermore, successful algorithm should make use of past task shifts to meta-learn effectively, thus accelerating the speed with which it adapts future task changes. In all cases, the algorithm receives one data point at a time, and the task changes periodically. In order to allow for a comparison with prior methods, which generally assume known task boundaries, the data stream is partitioned into discrete tasks, but our algorithm is not aware of which datapoint belongs to which tasks or where the boundaries occur. The prior methods, in contrast, are provided with this information, thus giving them advantage. We first describe the specific task formulations. and then the prior methods that we compare to.
Online meta-learning algorithms should adapt to each task as quickly as possible, and also use data from past tasks to accelerate acquisition of future tasks. Therefore, we report our results as learning curve, with one axis corresponding the number of seen tasks, and the other axis corresponding to the cumulative eтror rate on that task. This error rate is computed using held-out validation data for each task after the adaptation that task. We evaluate prior online meta-learning methods and baselines on two different datasets (Rainbow-MNIST and CIFAR 100). TOE (train on everything). TFS (train from scratch), FTL (follow the leader) and FTML (follow the meta-leader) (Finn et al., 2019) are the baseline methods compare against method FOML. See Section 4 for more detailed description of these methods.
Datasets: We compare TOE, TFS, FTL, FTML and FOML two different datasets. Rainbow- MNIST (Finn et al. 2019) was created by changing the background color, scale and rotation of the MNIST dataset. It includes 7 different background colors, 2 scales (full and half) and different rotations. This leads to a total of 56 number tasks. Each individual task is to classify the images into 10 classes. We use the same partition with 900 samples per each task, as in prior work (Finn)
7

---

Under review as a conference paper at ICLR 2022
Figure 3: Comparison between online algorithms: We compare our method with baselines and prior approaches, including TFS (Train from Scratch), TOE (Train on Everything), FTL (Follow the Leader) and FTML (Follow the Meta Leader). a: Performance relative to the number tasks seen over the course of online training on the Rainbow-MNIST dataset. As the number of task increases, FOML achieves lower error rates compared to other methods. b: Error rates the Inline-CIFARIDO 100 dataset. Note that FOML not only achieves lower error rates average, but also reaches the lowest error (of around 17%) more quickly than the other methods. e: We also compare our method with continual learning baselines LwF Li Hoiem (2017). iCaRL [Rebulti] MOCA[Hariisn et al. (2020). MOCA Harrison et al. (2200) archive similar performance to ours at the end of the learning, FOML can converge quickly, and faster compared to MOCA. d: Performance of FOMI. on CEI EBA dataset. This dataset contains more than 1000 classes and we follow the same protocol as in Online-CIFAR
(et a (2016) and MOCA Harrison
experiments. Our method converges faster than other baseline methods on CELEBA with 150 number of tasks.
et al. 2019). However, this task contains relatively simple images, and only 56 tasks. To create a much longer task sequence with significantly more realistic images, we modified the CIFAR-100 dataset to create an online meta-learning benchmark, which we call online-CIFAR 100. Every task is a randomly sampled set of classes, and the goal is to classify whether two images in this set belongs to same class or not. Specifically, each task corresponds to 5 classes from CIFAR- 100, and every datapoint consists of a pair of images, each corresponding to one of the classes for that task. The goal is to predict whether the two images belong the same class not. Note that different tasks not mutually exclusive, which in principle should favor a TOE-style method, since meta-learning is known to underperform with non-mutually-exclusiv tasks (Yin et al. 2019). To make sure the data distribution changes smoothly over tasks, we only change a subset of the classes between consecutive tasks. This allow us to create very large number of tasks (1200), and evaluate our method and the baselines on much longer and more realistic task sequences.
Implementation Details: We use a simple 4 layer convolutional neural network with 8,16,32,64 filters at each layer for both experiments. However, for the CIFAR- 100 experiments, a Siamese version of the same network is used. All the methods were trained via cross-entropy loss, with their best performing hyper-paramette parameters, on single NVIDA-2080 GPU machine. Please see Appendix A. for more details on hyper-parameters and network architecture.
Results on Rainbow-MNIST: As shown in Fig 3 FOML attains the lowest error rate on most tasks in MNNIS,, except a small segment in the very beginning. The performance of TFS is similar performance across all the tasks, and does not improve over time. This is because it resets its weights every time it encounters a new task, and therefore cannot not gain any advantage from past data. TOE has larger error rates the start, but as add more data into the buffer, TOE is able to improve. On the other hand, both FTL and FTML start with similar performance, but FTML achieve much lower error rates at the end of the sequence compared FTL, consistently with prior work (Finn al.) 2019). The final error rates of FOML are around 10%, and it reaches this performance significantly Taster than FTML, after less than 20 tasks. Note that FTML also has access to task boundaries, while FOML does not.
Results on Online CIFARI-- We use a Siamese network for this experiment, where each image is fed into a 7-layer convolutional network, and each branch outputs a 128 dimensional embedding vector. A difference of these vectors are fed into a fully connected layer for the final classification. Each task contains data from 5 classes of CIFAR- 100, and each new task introduces three new classes, and retains two of the classes from the previous task, providing a degree of temporal coherence while still requiring each algorithm to handle persistent shift in the task. Fig 3 shows the error rates of various online learning methods, where each method is trained over a sequence of 1200 tasks. All the methods start with initial error rates of 50%. The tasks are not mutually exclusive, so in principle TOE can attain good performance, but it makes the slowest progress among all the methods, suggesting that simple pretraining is not sufficient to accelerate learning. FTL. uses a similar pre-training strategy as TOE. However it has an adaptation stage where the model is fine-tuned on the new task. This allows to make slower progress. As expected from prior work (Finn al. 2019), the meta-learning procedure used by FTML allows it to make faster progress than FTL. However, FOML makes faster progress on average, and achieves the lowest final error rate (~ 15%) after sequence of 1200 tasks.
8

---

Under review as a conference paper at ICLR 2022
6.1 ABLATION STUDIES
We perform various ablations by varying the number of online parameters used for the meta- update K, importance of meta-modee to analysis the properties of our method.
Number of online parameters used for the meta-update: Our method periodically updates the online weights and meta weights. The meta-updates involves taking K recent online parameters and updating the meta model via MAML gradient. Therefore, meta-updates depend on the trajectory of the online parameters. In this experiment, we investigate how the performance of FOML changes as we vary the number of parameters used for the meta-update (K in Algorithm D. Fig ?? shows the performance of our method with various values of K: K = [1,2, 3, 5, 10]. We can see that the performance improves when update the meta parameters over longer trajectory of online parameters (larger K). We speculate that this is due to the longer sequences providing a clearer signal for how the meta-parameters influence online updates over many steps.
Importance of meta update: FOML keeps track of separate online parameters and mata-parameters, and each of them is updated via corresponding updates. However, only the online parameters are used for evaluation, while the meta- pettaparsssters 0 only influence them via the regularizer, and have no direct effect on evaluation performance. This might raise the question: how important the contribution of the meta-parameter to the performance of the algorithm during online training? We train a model with and without meta updates and the performance shown in Fig ??. None that, the model without meta-updates is identical to our method, except that the meta-updates themselves are not performed. We can clearly see that the model trained with meta-updates preforms much better than model trained without meta-updates. The model trained without meta-updates generally does not improve significantly as more tasks are seen, while the model trained with meta-updates improves with each task, and reaches significantly lower final error. This shows that, even though 0 and ø are decoupled and only connected via a regularization, the meta-learning component of our method really is critical for its good performance.
Figure 4: Ablation experiments: a) We vary the num- ber of online updates K used before the meta-update, see how it affects the performance of method. The performance of FOML improves as the number online updates is increased. b) This experiment shows how FOML performs with and without meta updates, to confirm that the meta-training is indeed essential component of method. With meta-updates, FOML learns more quickly, and performance improves with more tasks.
Oeter
7 CONCLUSION
We presented FOML, a MAMLL-ased algorithm for online meta-learning that does not require ground truth knowledge of task boundaries, and does not require resetting the parameter vector back to the meta-learned parameters for every task. FOML is conceptually simple, maintaining just two parameter vectors over the entire online adaptation process: a vector of online parameters ạ), which are updated continually on each new batch of datapoints, and a vector of meta-parameters 0, which are updated correspondingly with meta-updates to accelerate the online adaptation process, and influence the online updates via regularizer. We find that even relatively simple task sampling scheme that selects datapoints at random from a buffer of all seen data enables effective meta-training that accelerates the speed with which FOML can adapt to each new task, and we find that FOML reaches a final performance that is comparable to or better than baselines and prior methods, while learning adapt quickly new tasks significantly faster. While our work focuses on supervised classification problems, particularly exciting direction for future work is to extend such online meta-learning methods to other types of online supervision that may be more readily available, including self-supervision and prediction, so that models equipped with online meta-learning can continually improve as they see more of the world.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Maruan Al-Shedivan Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. arXiv preprint ww..v:1170..3600 2017.
Antreas Antoniou, Harrison Edwards, and Amos Storkey, How to train your maml. arXiv preprint arXiv:1810.00500. 2018.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney, Learning to continually learn. arXiv preprint arXiv:2002.09571, 2020.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, al. Online fast adaptation and knowledge accumulation: a new approach to continual learning. arXiv preprint arXiv:2003.05856 2020.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within- meta-learning. In ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 20/9), volume 32, pp. 1-11, Neural Information Processing Systems (NeurIPS 2019), 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- 1920-1930. PMLR, 09-15 Jun 2019.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-maml: Look-ahead meta learning for continual learning. arXiv preprint arXiv:2007.13904, 2020.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. arXiv preprint arXiv:1912.0886. 2019.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. Advances in neural information processing systems, 33, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. corr abs/1512.03385 (2015). 2015.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint 1rXiv:1906.05201, 2019.
Khurram Javed and Martha White. Meta-learning representations for continual learning. arXiv
preprint arXiv: 12588, 2019.
Ghassen Jerfel, Erin Grant, Thomas L Griffiths, and Katherine Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. arXiv preprint arXiv:1812.06088 2018.
Rong Jin, Steven CH Hoi, and Tianbao Yang. Online multiple kernel learning: Algorithms and mistake bounds. In International conference on algorithmic learning theory, pp. 390-404. Springer, 2010.
Jyrki Kivinen, Alexander J Smola, and Robert C Williamson. Online learning with kernels. IEEE transactions signal processing, 52(8):2165 22662 2004.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
10

---

Under review as a conference paper at ICLR 2022
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Da Li and Timothy Hospedales. Online meta-learning for multi-source and semi-supervised domain adaptation. In European Conference on Computer Vision, pp. 382-403. Springer, 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09833 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pp. 2554-2563, PMLR, 2017.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl. arXiv preprint arXiv:1812.0767,, 2018.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2(3):4, 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.0220 02999, 2018.
Eunbyung Park and Junier Oliva. Meta-curvature. arXiv preprint arXiv: MBXiv:9902.0335555 2019.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz. Khan, and Ling Shao. Random path selection for incremental learning. Advances in Neural Information Processing Systems, 2019.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 13588-13597, 2020.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2))285, 1990.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, and Christoph H Lampert. iearl: Incremental classi- fier and representation learning. corr Mtss/1611.0772. (2016). arXiv preprint 1rXiv:1611.07722 2016.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augment neural networks. In International conference machine learning, pp. 1842-1850. PMLR, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409 1556, 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint
00155, 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638, 2016.

---

Under review as a conference paper at ICLR 2022
Flood Sung Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot lear))ng. 2017.
Huaxiu Yao, Yingbo Zhou, Mehrdad Mahdavi, Zhenhui Li, Richard Socher, and Caiming Xiong. Online structured meta-learning. arXiv preprint arXiv:2010.11545, 2020.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. arXiv preprint arXiv:1912.03880, 2019.
Guanyu Zhou, Kihyuk Sohn, and Honglak Lee. Online incremental feature learning with denoising autoencoders, In Artificial intelligence and statistics, pp. 1453-1461, PMLR, 2012.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPLEMENTATION DETAILS-RAINROWMNED
All the images from Rainbow-MNIST are in 28 28 pixel resolution and with 3 channels. We used 4 layer convolutional neural network for these experiments, with each layer having [32,32,64, 32, number of filters. After every convolution ReLU activation and max pooling applied the features. At the last layer we take an average pooling. Finally, 64 dimensional features are passed through fully connected layer with a sigmoid activation. Since the Rainbow-MNISS uses MNIST data to classify, the final layer has 10 neurons, and the objective is to correctly classify the digits of different scales and colors into 10 classes (actual numbers irrespective of the rotation and background color). All the models were trained with 50 gradient updates.
A.2 IMPLEMENTATION DETAILS- WWW.OONNNNNN-IRRRIIIIII
All the images from CIFARI00 are in 32 32 pixel resolution and with 3 channels. We used similar 7 layer convolutional neural network in siamese network architecture for these experiments. with each layer having [32,32,32,64, 24,,,444,,64 128] number of filters. After every convolution ReLU activation and batchnorm applied to the features. Every other layers have max-pooling. At the last layer we take average pooling, The L2 distance between two images are calculated and fully connected layer is used to classify the two images as same class or not.
A.3 BASELINE METHODS
TFS:
Every time this model sees new task (This needs know the task boundary), the model resets to new sets of random weights. After that the model is trained only using the data from the new task. At the same time, we also rest the optimizer state remove any effect of momentum from previous updates. The model is trained with Adam optimizer with a learning rate of 0.001.
TOE: Although TOE optimize the parameters on the all available task, training the model using all the available data is practically impossible. Therefore, we sample datapoints from the buffer and update the model parameters. However, since the number of data is increasing over time, the model also needs be updated on a good representative data of the online stream. Therefore, we increase the number of gradient updates over time. For ONLINE- CINEE-IIIARII 100 experiment, the number of gradients updates are increased by 10 for every 100 tasks. The model is trained with Adam optimizer with learning rate of 0.001.
This method, first pretrains model on the past data, and then fine-tune it the very recent samples. Similar to TOE, we first train a set of weights using the datapoints in the data buffer. After that, the model is fine-tuned on the correct task data. However, after this adaptation and evaluation the fine-tuned weights are simply discarded and the pretraining starts from the initial weights of the adaptation process. Adam optimizer with same configuration is used to train this model.
FTL:
FTML: The FTML updates the meta peta-paarmeters using MAML style update. For that, we trained the FTML with 5 inner-loop updates (each inner-loop have task-specific data for the inner-loop adaptation). After the meta-update, the peta-parameters are used as the initialization for the inner- loop adaptation, and similar FTL the adapted weights are simply discarded after evaluation. The inner-loop is trained with SGD with a learning rate of 0.001 and the outer-loop is trained with learning rate of 0.0005.
Our method also have two sets of parameter vectors, thus two different optimizers. We use Adam for both online and meta updates with a learning rate of 0.001 for both cases. Since, our online updates share the meta-knowledge via the regularization term, the 31 controls how much pull applied to the meta parameters by the online parameters. We use 0.01 for 3,- Similarly the pull on meta- parameprsseters by the online parameters is controlled by 8g and we use 0.001 in our experiments.
FOML:
13


Reviews:
[{"review_id": "sT_6Z-qOViy", "reviewer": "Reviewer_XzKp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n1. Overall speaking, the writing is clear for people to understand the motivation, the background, the proposed method, and experiments. \n2. The algorithmic changes to previous FTML are incremental (to suit the new assumption) but the justification behinds these design choices are reasonable. \n3. Experiments on synthetic datasets validated the design choices to some degree, as the proposed approach achieves better fitting performances.\n\nWeakness/Question to Authors (order does not matter):\n- I am less familiar with the literature and would like to understand the evaluation metric on both datasets. How was the error rate / test error being calculated? Are they essentially the regret that represents your fitting performance? Do you also measure backward transfer performances and forward transfer performances? \n- In meta-update, with a large K, you essentially need to store K copies of all network parameters? Then how is the storage cost comparing your method to previous approaches?\n- Why don't you also compare to other non-meta-learning based online continual learning? This would help people to understand the status quo of meta-learning based online continual learning. It is okay even though you can not outperform them.\n- What would be the optimal fitting performance? Say, for all  the1200 tasks, you first perform offline batch training on all of them sufficiently, and then fine-tune on each of those 1200 tasks? How close would your model's performance to this optimal performance (of the same model achitecture)?\n- Did you compare different strategy for the buffer, such as reservoir buffer vs. fIFO buffer vs other buffer?\n- Why don't you consider realistic online continual learning datasets, such as [1] and [2]?\n\n  - [1] Drinking from a Firehose: Continual Learning with Web-scale Natural Language. \n  - [2] Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data\n\nMinor Comments: \n1. Section 1 Paragraph 1: \"… simultaneously that data for each new task is also used\" missing verb after simultanously?\n2. Section 1 Paragraph 1: \"fall short of the goal of creating an effective and adaptation system\" -> \"effective adaptation system\"?\n3. Section 3 Paragraph of \"Online Learning\": I am not sure why loss function is also a sequence of time step. Then, what would the loss function be if there is an concrete example. Isn't the loss function determined by the data (and label)? Maybe I misunderstand?\n4. Equation of FTL, maybe it would be more clear to reader if you rewrite phi as phi_{TOE}^{t-1}.\n5. Figure 3. Does not your approach called FOML? The method name in the legend is OML\n6. Implementation Details. Broken appendix hyperlink.", "summary_of_the_paper": "This paper extends meta-learning algorithm for continual online learning. Different from previous meta-learning based online continual learning, this algorithm removes assumption of knowing task boundaries of data beforehand. The proposed approach extends the previous online MAML (FTML), by removing the discrete 'resets' of task-specific parameters. One key difference besides removing reset is to change the replay buffer (memory) strategy used by meta-updates, where prior approach always sample data of all previous tasks and the proposed approach samples all data points seen so far (so there is no need to know which task we are currently solving).\n\nAs a result, this adapted algorithm cope with data of continuous task boundaries. To validate the proposed approach, authors conducted experiments on a synthetic continual image recognition benchmark (of 1,200 tasks) based on CIFAR100, where they outperforms previous continual meta-learning based approach.", "summary_of_the_review": "I am not an expert in this area so I believe that my opinion should be discounted. Based on my understanding, I think this is a solid and well-written paper, with extensive results that demonstrates the improvement of presented approach to its comparing prior work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X7lWyu7t0C", "reviewer": "Reviewer_dcpm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. The idea uses online learning to remove the task boundary is interesting\n2. The paper is reasonably written with clear background and diagrams for the overall architecture. It’s well written and easy to follow. \n3. The experiments demonstrate that FOML outperformed FTML\n\nWeakness\n1. The idea is not new, authors are missing some closely relevant and major literature in online meta-learning [1] and meta-learning without task boundaries.[2]\n2. I might have misunderstood the definition of ‘reset’ in the last paragraph of Section 4; as stated FTML needs to be reset at each task and the meta-training process does not influence the current task at all; But in algorithm 1 of the paper FTML, the model uses the result of meta-update to evaluate performance at data points within round t, and meanwhile the result of the adaptation process is recorded. In algorithm 1, why calculate predictions on the train set and validation set? Are they for loss? I do not see the utilization of them. And in lines 9 and 12, what’s the meaning of \\L_{task}? Is this loss related to specific tasks?  The FTML uses \\phi to meta-update \\theta and then uses the \\theta to update \\phi. The proposed method utilizes the \\theta to regularize and update \\phi, and then update \\theta by \\phi. It is not clear to me how the task boundaries are removed. More explanations are expected.  \n3. The experiments are weak. For example, there is no performance evaluation of the proposed FOML in Figure 3 which makes your claim unsupportive. I assume the OML represents the FOML? I’m not sure about that.  \n4. According to my previous point, important baselines are missing, the work [2] also targets meta-learning without the task boundaries.\n5. Is there any mathematical guarantee to show that FOML can learn without task boundaries? If not, more experiments on extra datasets are expected to support this claim.  \n6. Some minor presentation issues, e.g., \n- In Implementation Details, there is an error of “Please see Appendix ??”\n- For figure 3, the method name should be FOML rather than OML in the diagram.\n- In A.2 Implementation Details-Online CIFAR100, all the images should from ‘CIFAR100’ instead of Rainbow-MNIST.\n- For the caption of figure 1, online parameters should be \\phi and meta-parameters should be \\theta.\n\n\nReference\n\n[1] Denevi, G., Stamos, D., Ciliberto, C. and Pontil, M., 2019. Online-within-online meta-learning. In ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019) (Vol. 32, pp. 1-11). Neural Information Processing Systems (NeurIPS 2019).\n[2] Harrison, J., Sharma, A., Finn, C. and Pavone, M., 2020. Continuous Meta-Learning without Tasks. Advances in Neural Information Processing Systems, 33.\n", "summary_of_the_paper": "This paper proposes an online meta learning method which can remove the discrete task boundaries. Different from existing online meta-learning method, FOML utilizes online learning to remove the task boundaries. \n\nThe FOML is able to continually update online parameters with new datapoint, and meanwhile perform meta-gradient updates on a set of meta-parameters using a buffer of previous data. The authors compare the model to state-of-the-art like FTML on benchmarks such as Rainbow-MNIST and CIFAR100 datasets. Experimental results show that FOML learns to adapt faster and can achieve lower error rates on classification tasks. However, it lacks sufficient experiments and comparison with baseline methods to support the claim that FOML can learn without the task boundaries.\n", "summary_of_the_review": "This paper proposes a method named FOML which is a further step of the existing method FTML that tries to remove the task boundary. However, such an idea is not new and there is already a similar work that exists. Experiments are not strong enough to support the claim. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6yPWyT28Er", "reviewer": "Reviewer_dEVK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n* The proposed method is simple and it performs better than FTML and other previous approaches.\n* In general, the text is clear and easy to read.\n* The authors provide an Algorithm.\n* Implementation details are provided in the Appendix.\n\n# Weaknesses\n* Since FOML continuously updates the online weights, it is not clear what would happen if an OOD task is encountered. It would be interesting to see some ablation experiment or some discussion about this scenario, as well as possible ways to deal with it.\n* Some examples of non-cited works that tackle this problem are [A] or [B], which try to address this problem and also introduce new versions of MAML. In fact, it seems like [A] could be highly related to your approach. In their case, rather than optimizing a regularizer, they optimize the inner loop learning rate, which in the end, has the effect of preventing the online weights from drifting too much when the new encountered task has a high interference. \n\n[A] Gupta, Gunshi, Karmesh Yadav, and Liam Paull. \"La-maml: Look-ahead meta learning for continual learning.\" arXiv preprint arXiv:2007.13904 (2020).\n[B] Caccia, Massimo, et al. \"Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning.\" Advances in Neural Information Processing Systems 33 (2020).\n\n\n## Typos\n* Pag 4. algorithms tries -> algorithms try\n* Pag 4. with a pretrained weights -> with pretrained weights\n* Pag 5. necessarily results -> necessarily result\n* Pag 5. we will B to denote -> we will use B to denote\n* Pag 7. changings -> changing\n* Pag 8. see Appendix ??\n* Pag 9. depends on -> depend on", "summary_of_the_paper": "The authors propose FOML, a new online learning algorithm based on MAML. It introduces two new features. 1. Instead of resetting the inner loop weights back to the meta-weights after each task, the online weights are always kept and updated. 2. It introduces a regularizer that aims to keep the online weights always close to the meta-weights and the other way around. Rainbow MNIST and online CIFAR100  experiments show that FOML outperforms TFS, TOE, FTL, and FTML. In ablation experiments, the authors show that FOML woks better for higher numbers of inner update iterations as well as the importance of the meta-update.", "summary_of_the_review": "The authors present a simple approach that is sound, effective in a specific online continual learning scenario, where a replay buffer is kept, and new tasks are not completely unrelated to previous tasks. On the other hand, the authors only provide experiments on MNIST and CIFAR, and do not provide any additional information on how their algorithm would handle more challenging situations. \n\nOverall, the proposed method is interesting for the research community but the authors should provide more information on how it would behave under more challenging scenarios, discuss its limitations, and include in the text a more complete comparison with other setups that justifies for a non-expert reader why they did not compare with Harrison et al. 2019, He et al. 2019, [A] ...", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5j0Meropsu8", "reviewer": "Reviewer_me6U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe approach is quite simple and intuitive. \n\nThe paper is written clearly and is easy to understand.\n\nConcerns\n\nLack of clarity on the problem statement: The authors cast the problem as meta-learning that must be done online. I am not convinced with this problem statement. As I understand, the goal in meta-learning (particularly the MAML framework adopted by the authors) is to learn a good initialization for quickly adapting to a new task. However, the current problem seems to be more on the lines of online learning and avoiding catastrophic forgetting, as there is no adaptation step after meta-training. The authors explicitly mention that their goal is not to solve catastrophic forgetting. But then, what is the goal? If I am mistaken, and there is an adaptation step(akin to meta-test), it would be interesting to see the number of adaptation steps required by the various models to adapt to the new task. \n\nMissing literature: Online learning/continual learning is a well-studied research area. There is a large body of literature on regularization-based methods (e.g., Kirkpatrick et al. 2017, Ritter et al., 2018), memory-based methods, online continual learning (Caccia et al. 2020), etc. I find the discussion on the related work to be quite narrow, missing out on approaches trying to achieve similar goals. Furthermore, evaluating these related works over and above the few methods listed in the experiments is also important.\n\nWeak experimental setup: The specially curated datasets (and the tasks) appear quite simple (even the online-CIFAR100 task). Existing literature has used more complex sequences of tasks (from miniimagenet, tieredimagenet). Experiments on these datasets are warranted to conclude FOML’s efficacy.\n\nFinally, the lack of theoretical justification on the effectiveness of the FOML is also a cause of concern.  \n\nMinor comments\n\nPage 5 last paragraph - To this end, we well \\script{B} to denote - we use \\script{B} to denote.\n\nImplementation Details Section on Page 8 - Appendix information is missing.\n", "summary_of_the_paper": "The paper proposes an online learning approach that utilizes the meta-learning framework. The proposed method - FOML has two main components: a regularizer that does not allow the updated parameters of the model to deviate too far from the previously learned set and a meta update on the parameters using the online updates, and a validate set drawn from the buffer of examples seen in the past. FOML is evaluated on two datasets and compared against a few other similar approaches and baselines. The results suggest faster adaptation of FOML compared to previous methods.\n", "summary_of_the_review": "Overall, while the paper is well-written, I have doubts about the motivation behind the problem statement. I am concerned about the missing relevant literature and inadequate experiments.\n\n--- Post rebuttal update\n\nI thank the authors for the detailed response to clarify some of my concerns. I also read the other reviews and the response from the authors. The authors have clarified the existence of the problem statement citing prior work. I agree with the existence of the problem statement. However, I am not entirely convinced with the motivation. Experiments on real-world applications requiring online meta-learning would have significantly strengthened the contribution. I acknowledge that the experiments comparing other online-learning algorithms (such as LwF, iCaRL) and algorithms such as MOCA is a step in the right direction, but requires more analysis (using more datasets). \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper propose a Fully Online Meta-Learning (FOML) method which extend MAML for continual learning in a fully online learning  without requiring the knowledge of the task boundaries. Experiments show that FOML was able to learn new tasks faster than several existing online learning methods on Rainbow-MNIST, and CIFAR100 datasets. 

There are a few major concerns from reviewers. One concern is about the lack of clarity on the problem statement: The authors cast the problem as meta-learning that must be done in a fully online setting, but it requires to store all the training data in a buffer storing all the training data seen so far, which contradicts to the principle of “online learning”. Another major weakness is the poorly written literature survey, which missed to cite a large body of related work in continual learning and online-meta-learning (such as Online Continual Learning, task-free continual learning, continual learning without task boundaries, etc). These should at least be discussed carefully if not fully compared in the empirical studies. Also experiments are quite weak in both settings, datasets and rather out-of-date baselines. Finally, there also lacks of theoretical justification or analysis. 

Therefore, the paper is not recommended for acceptance in its current form. I hope authors found the review comments informative and can improve their paper by addressing these review comments carefully in future submissions.
Decision: Reject