user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
A central capability of intelligent systems is the ability to continuously build upon previous experiences to speed up and enhance learning of new tasks.
Two distinct research paradigms have studied this question.
Meta-learning views this problem as learning a prior over model parameters that is amenable for fast adaptation on a new task, but typically assumes the set of tasks are available together as a batch.
In contrast, online (regret based) learning considers a sequential setting in which problems are revealed one after the other, but conventionally train only a single model without any task-specific adaptation.
This work introduces an online meta-learning setting, which merges ideas from both the aforementioned paradigms to better capture the spirit and practice of continual lifelong learning.
We propose the follow the meta leader (FTML) algorithm which extends the MAML algorithm to this setting.
Theoretically, this work provides an O(logT) regret guarantee for the FTML algorithm.
Our experimental evaluation on three different large-scale tasks suggest that the proposed algorithm significantly outperforms alternatives based on traditional online learning approaches.
Two distinct research paradigms have studied how prior tasks or experiences can be used by an agent to inform future learning.
Meta-learning (Schmidhuber, 1987) casts this as the problem of learning to learn, where past experience is used to acquire a prior over model parameters or a learning procedure.
Such an approach, where we draw upon related past tasks and form associated priors, is particularly crucial to effectively learn when data is scarce or expensive for each task.
However, meta-learning typically studies a setting where a set of meta-training tasks are made available together upfront as a batch.
In contrast, online learning (Hannan, 1957 ) considers a sequential setting where tasks are revealed one after another, but aims to attain zero-shot generalization without any task-specific adaptation.
We argue that neither setting is ideal for studying continual lifelong learning.
Metalearning deals with learning to learn, but neglects the sequential and non-stationary nature of the world.
Online learning offers an appealing theoretical framework, but does not generally consider how past experience can accelerate adaptation to a new task.
In this work, we motivate and present the online meta-learning problem setting, where the agent simultaneously uses past experiences in a sequential setting to learn good priors, and also adapt quickly to the current task at hand.
Our contributions: In this work, we first formulate the online meta-learning problem setting.
Subsequently, we present the follow the meta-leader (FTML) algorithm which extends MAML (Finn et al., 2017) to this setting.
FTML is analogous to follow the leader in online learning.
We analyze FTML and show that it enjoys a O(log T ) regret guarantee when competing with the best metalearner in hindsight.
In this endeavor, we also provide the first set of results (under any assumptions) where MAML-like objective functions can be provably and efficiently optimized.
We also develop a practical form of FTML that can be used effectively with deep neural networks on large scale tasks, and show that it significantly outperforms prior methods in terms of learning efficiency on vision-based sequential learning problems with the MNIST, CIFAR, and PASCAL 3D+ datasets.
Due to space constraints, we review the foundations of our work (meta-learning and online learning) in Appendix A. We consider a general sequential setting where an agent faces tasks one after another.
Each task corresponds to a round, denoted by t. In each round, the goal of the learner is to determine model parameters w t that perform well for the corresponding task at that round.
This is monitored by f t : w ∈ W → R, which we would like to be minimized.
Crucially, we consider a setting where the agent can perform some local task-specific updates to the model before it is deployed and evaluated T t=1 be the sequence of models generated by the algorithm.
Then, the regret we consider is: DISPLAYFORM0 Notice that we allow the comparator to adapt locally to each task at hand; thus the comparator has strictly more capabilities than the learning agent, since it is presented with all the task functions in batch mode.
Achieving sublinear regret suggests that the agent is improving over time and is competitive with the best meta-learner in hindsight (since Regret T /T → 0 as T → ∞).
In the batch setting, meta-learning has been observed to perform better than jointly training a single model to work on all the tasks (Santoro et al., 2016; Finn et al., 2017; Vinyals et al., 2016; Li & Malik, 2017; Ravi & Larochelle, 2017) .
Thus, we may hope that learning sequentially, but still being competitive with the best meta-learner in hindsight, provides a significant leap in continual learning.
Our algorithmic approach, follow the meta leader (FTML), takes inspiration from follow the leader (FTL) (Hannan, 1957; Kalai & Vempala, 2005) and adapts it to the online meta learning setting.
FTML chooses the parameters according to: DISPLAYFORM0 This can be interpreted as the agent playing the best meta-learner in hindsight if the learning process were to stop at round t. In practice, we may not have full access to f k (·), such as when it is the population risk and we only have a finite size dataset.
In such cases, we will draw upon stochastic approximation algorithms to solve the optimization problem in Eq. (2).We concentrate on the case where the update procedure is 1 step of stochastic gradient descent, as in the case of MAML, i.e. U t (w) = w − α∇f i (w).
We assume that each loss function, {f t ,f t } ∀t, is C 2 −smooth (i.e. G−Lipschitz, β−smooth, and ρ−Lipschitz Hessian) and µ−strongly convex.
See Appendix B for more details and implications of these assumptions, and connections to standard online learning setting.
Importantly, these assumptions do not trivialize the meta-learning setting.
There is a clear separation between meta-learning and joint training even for linear regression (simplest strongly convex problem).
See Appendix E for an example illustration.
Theorem 1.
Suppose f andf : R d → R satisfy the stated assumptions.
Letf be the function evaluated after the gradient update procedure, i.e.f (w) := f w − α∇f (w) .
If the step size is selected as α ≤ min{ 1 2β , µ 8ρG }, thenf isβ = 9β/8 smooth andμ = µ/8 strongly convex.
Since the objective function is convex, we may expect first-order optimization methods to be effective, since gradients can be efficiently computed with standard automatic differentiation libraries (as discussed in Finn et al. (2017) ).
In fact, this work provides the first set of results (under any assumptions) under which MAML-like objective function can be provably and efficiently optimized.
An immediate corollary of our main theorem is that FTML now enjoys the same regret guarantees (up to constant factors) as FTL does in the comparable setting (with strongly convex losses).
Corollary 1. (inherited regret bound for FTML) Suppose that for all t, f t andf t satisfy assumptions 1 and 2.
Suppose that the update procedure in FTML (Eq. 2) is chosen as U t (w) = w − α∇f t (w) with α ≤ min{ 1 2β , µ 8ρG }.
Then, FTML enjoys the following regret guarantee DISPLAYFORM1 More generally, our main theorem implies that there exists a large family of online meta-learning algorithms that enjoy sub-linear regret, based on the inherited smoothness and strong convexity of f (· DISPLAYFORM2 Here, ν t (·) denotes a sampling distribution for the previously seen tasks (we use a uniform distribution in our experiments).
L(D, w) is the loss function (e.g. cross-entropy) averaged over the datapoints (x, y) ∈ D for the model with parameters w. While U t in Eq. (3) includes only one gradient step, we observed that it is beneficial to take multiple gradient steps in the inner loop (i.e., in U t ), which is consistent with prior works (Finn et al., 2017; Grant et al., 2018; BID3 .The overall algorithmic procedure proceeds as follows.
We first initialize a task buffer B = [ ].
When presented with a new task at round t, we add task T t to B and initialize a task-specific dataset D t = [ ], which is appended to as data incrementally arrives for task T t .
As new data arrives for task T t , we iteratively compute and apply the gradient in Eq. (3), which uses data from all tasks seen so far.
Once all of the data (finite-size) has arrived for T t , we move on to task T t+1 .
Our experimental evaluation studies the practical FTML algorithm (Section 2.1) in the context of vision-based online learning problems.
These problems include synthetic modifications of the MNIST dataset, pose detection with synthetic images based on PASCAL3D+ models (Xiang et al., 2014) , and realistic online image classification experiments with the CIFAR-100 dataset.
The aim of our experimental evaluation is to study the following questions: (1) can online meta-learning (and specifically FTML) be successfully applied to multiple non-stationary learning problems?
and (2) does online meta-learning (FTML) provide empirical benefits over prior methods?To this end, we compare to the following algorithms: (a) Train on everything (TOE) trains on all available data so far (including D t at round t) and trains a single predictive model.
This model is directly tested without any specific adaptation since it has already been trained on D t .
(b) Train from scratch, which initializes w t randomly, and finetunes it using D t .
(c) Joint training with fine-tuning, which at round t, trains on all the data jointly till round t − 1, and then finetunes it specifically to round t using only D t .
This corresponds to the standard online learning approach where FTL is used (without any meta-learning objective), followed by task-specific fine-tuning.
We compare the algorithms on two metrics: (1) task performance (e.g. classification accuracy) for each task in the sequence; and (2) learning efficiency or amount of data needed to reach a proficiency threshold (e.g. 90% classification accuracy).
We note that TOE is a very strong point of comparison, capable of reusing representations across tasks, as has been proposed in a number of prior continual learning works (Rusu et al., 2016; BID1 Wang et al., 2017) .
However, unlike FTML, TOE does not explicitly learn the structure across tasks.
Thus, it may not be able to fully utilize the information present in the data, and will likely not be able to learn new tasks with only a few examples.
Further, the model might incur negative transfer if the new task differs substantially from previously seen ones, as has been observed in prior work (Parisotto et al., 2016) .
FTL with fine-tuning represents a natural online learning comparison, which in principle should combine the best parts of learning from scratch and TOE, since this approach adapts specifically to each task and benefits from prior data.
However, in contrast to FTML, this method does not explicitly meta-learn and hence may not fully utilize any structure in the tasks, and may also overfit in the fine-tuning stage.
In the rainbow MNIST experiment, we transform the digits in a number of ways to create different tasks, such as 7 different colored backgrounds, 2 scales (half size and original size), and 4 rotations of 90 degree intervals.
A task involves correctly classifying digits with a randomly sampled background, scale, and rotation.
As seen in the results curves in FIG0 , FTML learns tasks more and more quickly with each new task.
We also observe that FTML substantially outperforms the baselines in both efficiency and end performance.
FTL is better than TOE since it performs task-specific adaptation, but still worse than FTML.
We hypothesize that, while TOE and FTL improve in efficiency over the course of learning as they see more tasks, they struggle to prevent negative transfer on each new task.
Our last observation is that training independent models does not learn efficiently, compared to models that incorporate data from other tasks; but, their asymptotic performance with a large data size is similar.
Our next experiment studies a 3D pose prediction problem.
Each task involves learning to predict the global position and orientation of an object in an image.
We construct a dataset of synthetic images using 50 object models from 9 different object classes in the PASCAL3D+ dataset (Xiang et al., 2014) , rendering the objects on a table using the renderer accompanying the MuJoCo physics engine (Todorov et al., 2012) .
To place an object on the table, we select a random 2D location, as well as a random azimuthal angle.
Each task corresponds to a different object with a randomly sampled camera angle.
For the loss functions, we use mean-squared error, and set the proficiency threshold to an error of 0.05.
We show the results of this experiment in FIG0 .
The results demonstrate that meta-learning can significantly improve both efficiency and performance of new tasks over the course of learning, solving many of the tasks with only 10 datapoints.
Unlike the previous settings, TOE substantially outperforms the independent task models, indicating that it can effectively make use of the previous data from other tasks (likely due to greater structural similarity in this task).
However, the efficiency and performance of online meta-learning demonstrates that even better transfer can be accomplished by explicitly optimizing for the ability to quickly and effectively learn new tasks.
Appendix C presents a more detailed evaluation and results on the CIFAR task.
In this paper, we introduced the online meta-learning problem statement, with the aim of connecting the fields of meta-learning and online learning.
Online meta-learning provides, in some sense, a more natural perspective on the ideal real-world learning procedure.
An intelligent agent interacting with a constantly changing environment should utilize streaming experience to both master the task at hand, and become more proficient at learning new tasks in the future.
We summarize prior work related to our setting in Appendix D. For the online meta-learning setting, we proposed the FTML algorithm and showed that it enjoys logarithmic regret.
We then illustrated how FTML can be adapted to a practical algorithm.
Our experimental evaluations demonstrated that the proposed practical variant outperforms prior methods.
Here, we summarize the foundations of our online meta-learning formulation.
In particular, we concentrate on model agnostic meta-learning and online (i.e. regret based) learning.
To illustrate the differences in setting and algorithms, we will use the running example of few-shot learning, which we describe below first.
We emphasize that online learning, MAML, and the online meta-learning formulations have a broader scope than few-shot supervised learning.
We use the few-shot supervised learning example primarily for illustration.
In the few-shot supervised learning setting (Santoro et al., 2016), we are interested in a family of tasks, where each task T is associated with a notional and infinite-size population of input-output pairs.
In the few-shot learning, the goal is to learn a task while accessing only a small, finite-size labeled dataset D i := {x i , y i } corresponding to task T i .
If we have a predictive model, h(·; w), with parameters w, the population risk of the model is f i (w) := E (x,y)∼Ti [ (x, y, w)], where the expectation is over the task population and (·) is a loss function, such as the square loss or crossentropy between the model prediction and the correct label.
For example, in th e case of square loss we have, (x, y, w) = ||y − h(x; w)|| 2 .
Let L(D i , w) represent the average loss on the dataset D i .Being able to effectively minimize f i (w) is likely hard if we rely only on D i due to the small size of the dataset.
However, we are exposed to many such tasks from the family -either in sequence or as a batch, depending on the setting.
By being able to draw upon the multiplicity of tasks, we may hope to perform better, as for example demonstrated in the meta-learning literature.
A.2 META-LEARNING AND MAML Meta-learning, or learning to learn (Schmidhuber, 1987) , aims to effectively bootstrap from a set of tasks to learn faster on a new task.
It is assumed that tasks are drawn from a fixed distribution, DISPLAYFORM0 are drawn from this distribution and datasets corresponding to them are made available to the agent.
At deployment time, we are faced with a new test task T j ∼ P(T ), for which we are again presented with a small labeled dataset D j := {x j , y j }.
Meta-learning algorithms attempt to find a model using the M training tasks, such that when D j is revealed from the test task, the model can be quickly updated to minimize f j (w).
MAML (Finn et al., 2017) does this by learning an initial set of parameters w MAML , such that at meta-test time, performing a few steps of gradient descent from w MAML using D j minimizes f j (·).
To get such an initialization, at meta-training time, MAML solves the optimization problem: DISPLAYFORM1 The inner gradient, ∇f i (w), is based on a mini-batch of data from D i .
Hence, MAML optimizes for few-shot generalization.
Note that the optimization problem is subtle: we have a gradient descent step embedded in the actual objective function.
Regardless, Finn et al. (2017) show that gradient-based methods can be used on this optimization objective with existing automatic differentiation libraries.
Stochastic optimization techniques are used to solve the optimization problem in Eq. (4) since the population risk is not known directly.
At meta-test time, the solution to Eq. 4 is fine-tuned as: w j ← w MAML − α∇f j (w MAML ) with the gradient obtained using D j .MAML and other meta-learning algorithms are not directly applicable to sequential settings for two reasons.
First, they have two distinct phases: meta-training and meta-testing or deployment.
We would like the algorithms to work in a continuous learning fashion.
Second, meta-learning methods generally assume that the tasks come from some fixed distribution, whereas we would like methods that work for non-stationary task distributions.
In the online learning setting, an agent faces a sequence of loss functions {f t } ∞ t=1 , one in each round t. These functions need not be drawn from a fixed distribution, and could even be chosen adversarially over time.
The goal for the learner is to sequentially decide on model parameters {w t } ∞ t=1 that perform well on the loss sequence.
In particular, the standard objective is to minimize some notion of regret defined as the difference between our learner's loss, T t=1 f t (w t ), and the best performance achievable by some family of methods (comparator class).
The most standard notion of regret is to compare to the cumulative loss of the best fixed model in hindsight: DISPLAYFORM0 The goal in online learning is to design algorithms such that this regret grows with T as slowly as possible.
In particular, an agent (algorithm) whose regret grows sub-linearly in T is non-trivially learning and adapting.
One of the simplest algorithms in this setting is follow the leader (FTL) Hannan (1957), which updates the parameters as: DISPLAYFORM1 FTL enjoys strong performance guarantees depending on the properties of the loss function, and some variants use additional regularization to improve stability Shalev-Shwartz (2012) .
For the few-shot supervised learning example, FTL would consolidate all the data from the prior stream of tasks into a single large dataset and fit a single model to this dataset.
As observed in the meta-learning literature, such a "joint training" approach may not learn effective models.
To overcome this issue, we may desire a more adaptive notion of a comparator class, and algorithms that have low regret against such a comparator, as done in the online meta-learning formulation.
In this section, outline the assumptions and proofs.
We make the following assumptions about each loss function {f t ,f t } ∀t in the learning problem.
Let θ and φ represent two arbitrary choices of model parameters.
Assumption 1. (C 2 -smoothness) 1. (Lipschitz in function value) f has gradients bounded by G, i.e. ||∇f (θ)|| ≤ G ∀ θ.
Assumption 2. (Strong convexity) Suppose that f is convex.
Furthermore, suppose f is µ−strongly convex, i.e. ||∇f (θ) − ∇f (φ)|| ≥ µ||θ − φ||.These assumptions are largely standard in online learning, in various settings Cesa-Bianchi & Lugosi FORMULA1 , except 1.3.
Examples where these assumptions hold include logistic regression and L2 regression over a bounded domain.
Assumption 1.3 is a statement about the higher order smoothness of functions which is common in non-convex analysis Nesterov & Polyak FORMULA1 ; Jin et al. (2017) .
In our setting, it allows us to characterize the landscape of the MAML-like function which has a gradient update step embedded within it.
Importantly, these assumptions do not trivialize the meta-learning setting.
A clear difference in performance between meta-learning and joint training can be observed even in the case where f (·) are quadratic functions, which correspond to the simplest strongly convex setting.
See Appendix E for an example illustration.
We analyze the FTML algorithm when the update procedure is a single step of gradient descent, as in the formulation of MAML.
Concretely, the update procedure we consider is U t (w) = w − α∇f t (w).
We restate our main theorem below for completeness.
Theorem.
Suppose f andf : R d → R satisfy assumptions 1 and 2.
Letf be the function evaluated after a one step gradient update procedure, i.e. DISPLAYFORM0 If the step size is selected as α ≤ min{ 1 2β , µ 8ρG }, thenf is convex.
Furthermore, it is alsoβ = 9β/8 smooth andμ = µ/8 strongly convex.
Proof.
First, the smoothness and strong convexity of f andf implies µ ≤ ||∇ 2f (θ)|| ≤ β ∀θ.
Thus, DISPLAYFORM1 Also recall the earlier notationθ = U (θ) = θ − α∇f (θ).
For α < 1/β, we have the following bounds: DISPLAYFORM2 since we have U (θ) − U (φ) = I − α∇ 2f (ψ) (θ − φ) for some ψ that connects θ and φ due to the mean value theorem on ∇f .
Using the chain rule and our definitions, DISPLAYFORM3 Taking the norm on both sides, for the specified α, we have: DISPLAYFORM4 Similarly, we obtain the following lower bound DISPLAYFORM5 which completes the proof.
The following corollary is now immediate.
satisfy assumptions 1 and 2, then the MAML optimization problem, DISPLAYFORM0 } is convex.
Furthermore, it is 9β/8-smooth and µ/8-strongly convex.
Since the objective function is convex, we may expect first-order optimization methods to be effective, since gradients can be efficiently computed with standard automatic differentiation libraries (as discussed in Finn et al. (2017) ).
In fact, this work provides the first set of results (under any assumptions) under which MAML-like objective function can be provably and efficiently optimized.
Another immediate corollary of our main theorem is that FTML now enjoys the same regret guarantees (up to constant factors) as FTL does in the comparable setting (with strongly convex losses).Corollary. (inherited regret bound for FTML) Suppose that for all t, f t andf t satisfy assumptions 1 and 2.
Suppose that the update procedure in FTML (Eq. 2) is chosen as U t (w) = w − α∇f t (w) with α ≤ min{ 1 2β , µ 8ρG }.
Then, FTML enjoys the following regret guarantee DISPLAYFORM1 Proof.
From Theorem 1, we have that each functionf t (w) = f t (U t (w)) isμ = µ/8 strongly convex.
The FTML algorithm is identical to FTL on the sequence of loss functions {f t } T t=1 , which has a O( performance after 100 datapoints on the current task.
Right: The task performance after all 900 datapoints for the current task have been received.
Lower is better for all plots.
FTML can learn new tasks more and more efficiently as each new task is received, demonstrating effective forward transfer.
In this appendix, we present the extended experimental evaluation that is introduced in Section 3.
In this experiment, we create a sequence of tasks based on the MNIST character recognition dataset.
We transform the digits in a number of ways to create different tasks, such as 7 different colored backgrounds, 2 scales (half size and original size), and 4 rotations of 90 degree intervals.
As illustrated in FIG1 , a task involves correctly classifying digits with a randomly sampled background, scale, and rotation.
This leads to 56 total tasks.
We partitioned the MNIST training dataset into 56 batches of examples, each with 900 images and applied the corresponding task transformation to each batch of images.
The ordering of tasks was selected at random and we set 90% classification accuracy as the proficiency threshold.
The learning curves in FIG3 show that FTML learns tasks more and more quickly, with each new task added.
We also observe that FTML substantially outperforms the alternative approaches in both efficiency and final performance.
FTL performance better than TOE since it performs task-specific adaptation, but its performance is still inferior to FTML.
We hypothesize that, while the prior methods improve in efficiency over the course of learning as they see more tasks, they struggle to prevent negative transfer on each new task.
Our last observation is that training independent models does not learn efficiently, compared to models that incorporate data from other tasks; but, their final performance with 900 data points is similar.
In this experiment, we create a sequence of 5-way classification tasks based on the CIFAR-100 dataset, which contains more challenging and realistic RGB images than MNIST.
Each classification problem involves a newly-introduced class from the 100 classes in CIFAR-100.
Thus, different tasks correspond to different labels spaces.
The ordering of tasks is selected at random, and we measure performance using classification accuracy.
Since it is less clear what the proficiency threshold should Figure 4 : Online CIFAR-100 results, evaluating task performance after 50, 250, and 2000 datapoints have been received for a given task.
We see that FTML learns each task much more efficiently than models trained from scratch, while both achieve similar asymptotic performance after 2000 datapoints.
We also observe that FTML benefits from adapting all layers rather than learning a shared feature space across tasks while adapting only the last layer.
On the left, we observe that online meta-learning generally leads to faster learning as more and more tasks are introduced, learning with only 10 datapoints for many of the tasks.
In the center and right, we see that meta-learning enables transfer not just for faster learning but also for more effective performance when 60 and 400 datapoints of each task are available.
Note that the order of tasks is randomized, hence leading to spikes when more difficult tasks are introduced.be for this task, we evaluate the accuracy on each task after varying numbers of datapoints have been seen.
Since these tasks are mutually exclusive (as label space is changing), it makes sense to train the TOE model with a different final layer for each task.
An extremely similar approach to this is to use our meta-learning approach but to only allow the final layer parameters to be adapted to each task.
Further, such a meta-learning approach is a more direct comparison to our full FTML method, and the comparison can provide insight into whether online meta-learning is simply learning features and performing training on the last layer, or if it is adapting the features to each task.
Thus, we compare to this last layer online meta-learning approach instead of TOE with multiple heads.
The results (see Figure 4 ) indicate that FTML learns more efficiently than independent models and a model with a shared feature space.
The results on the right indicate that training from scratch achieves good performance with 2000 datapoints, reaching similar performance to FTML.
However, the last layer variant of FTML seems to not have the capacity to reach good performance on all tasks.
In our final experiment, we study a 3D pose prediction problem.
Each task involves learning to predict the global position and orientation of an object in an image.
We construct a dataset of synthetic images using 50 object models from 9 different object classes in the PASCAL3D+ dataset (Xiang et al., 2014) , rendering the objects on a table using the renderer accompanying the MuJoCo physics engine (Todorov et al., 2012 ) (see FIG1 .
To place an object on the table, we select a random 2D location, as well as a random azimuthal angle.
Each task corresponds to a different object with a randomly sampled camera angle.
We place a red dot on one corner of the table to provide a global reference point for the position.
Using this setup, we construct 90 tasks (with an average of about 2 camera viewpoints per object), with 1000 datapoints per task.
All models are trained to regress to the global 2D position and the sine and cosine of the azimuthal angle (the angle of rotation along the z-axis).
For the loss functions, we use mean-squared error, and set the proficiency threshold to an error of 0.05.
We show the results of this experiment in FIG4 .
The results demonstrate that meta-learning can improve both efficiency and performance of new tasks over the course of learning, solving many of the tasks with only 10 datapoints.
Unlike the previous settings, TOE substantially outperforms training from scratch, indicating that it can effectively make use of the previous data from other tasks, likely due to the greater structural similarity between the pose detection tasks.
However, the performance of FTML suggests that even better transfer can be accomplished by explicitly optimizing for the ability to quickly and effectively learn new tasks.
Finally, we find that FTL performs comparably or worse than TOE, indicating that task-specific fine-tuning can lead to overfitting when the model is not explicitly trained for the ability to fine-tune effectively.
Our work proposes to use meta-learning or learning to learn Thrun & Pratt (1998); Schmidhuber (1987); Naik & Mammone (1992) , in the context of online (regret-based) learning.
We reviewed the foundations of these approaches in Section A, and we summarize additional related work along different axis.
FORMULA0 , nearly all prior meta-learning algorithms assume that the meta-training tasks come from a stationary distribution.
Furthermore, most prior work has not evaluated versions of meta-learning algorithms when presented with a continuous stream of tasks.
Recent work has considered handling non-stationary task distributions in meta-learning using Dirichlet process mixture models over meta-learned parameters Grant et al. (2019) .
Unlike this prior work, we introduce a simple extension onto the MAML algorithm without mixtures over parameters, and provide theoretical guarantees.
Continual learning: Our problem setting is related to (but distinct from) continual, or lifelong learning Thrun (1998); Zhao & Schmidhuber (1996) In this paper, we sidestep the problem of catastrophic forgetting by maintaining a buffer of all the observed data Isele & Cosgun (2018) .
In future work, we hope to understand the interplay between limited memory and catastrophic forgetting for variants of the FTML algorithm.
Here, we instead focuses on the problem of forward transfer -maximizing the efficiency of learning new tasks within a non-stationary learning setting.
Prior works have also considered settings that combine joint training across tasks with task-specific adaptation Barto et al. (1995) FORMULA0 , we also focus on the setting where there are several tens or hundreds of tasks.
This setting is interesting since there is significantly more information that can be transferred from previous tasks and we can employ more sophisticated techniques such as meta-learning for transfer, enabling the agent to move towards few-shot learning after experiencing a large number of tasks.
Online learning: Similar to continual learning, online learning deals with a sequential setting with streaming tasks.
It is well known in online learning that FTL has good regret guarantees, but is often computationally expensive.
Thus, there is a large body of work on developing computationally cheaper algorithms Cesa-Bianchi & Lugosi (2006); Hazan et al. (2006); Zinkevich (2003) ; ShalevShwartz (2012) .
Again, in this work, we sidestep the computational considerations to first study if the meta-learning analog of FTL can provide performance gains.
For this, we derived the FTML algorithm which has low regret when compared to a powerful adaptive comparator class that performs task-specific adaptation.
We leave the design of more computationally efficient versions of FTML to future work.
To avoid the pitfalls associated with a single best model in hindsight, online learning literature has also studied alternate notions of regret, with the closest settings being dynamic regret and adaptive or tracking regret.
In the dynamic regret setting (Herbster & Warmuth, 1995; Yang et al., 2016; Besbes et al., 2015) , the performance of the online learner's model sequence is compared against the sequence of optimal solutions corresponding to each loss function in the sequence.
Unfortunately, lower-bounds (Yang et al., 2016) suggest that the comparator class is too powerful and may not provide for any non-trivial learning in the general case.
To overcome these limitations, prior work has placed restrictions on how quickly the loss functions or the comparator model can change (Hazan & Comandur, 2009; Hall & Willett, 2015; Herbster & Warmuth, 1995) .
In contrast, we consider a different notion of adaptive regret, where the learner and comparator both have access to an update procedure.
The update procedures allow the comparator to produce different models for different loss functions, thereby serving as a powerful comparator class (in comparison to a fixed model in hindsight).
For this setting, we derived sublinear regret algorithms without placing any restrictions on the sequence of loss functions.
We believe that this setting captures the spirit and practice of continual lifelong learning, and also leads to promising empirical results.

assistant: text: We introduce the online meta learning problem setting to better capture the spirit and practice of continual lifelong learning.