Variable-Shot Adaptation for Online Meta-Learning
Authors: Tianhe Yu
Xinyang Geng
Chelsea Finn
Sergey Levine
Venue:  (2021)
Keywords: meta-learning
deep learning

Abstract: Few-shot meta-learning methods consider the problem of learning new tasks from a small, fixed number of examples, by meta-learning across static data from a set of previous tasks. However, in many real world settings, it is more natural to view the problem as one of minimizing the total amount of supervision --- both the number of examples needed to learn a new task and the amount of data needed for meta-learning. Such a formulation can be studied in a sequential learning setting, where tasks are presented in sequence. When studying meta-learning in this online setting, a critical question arises: can meta-learning improve over the sample complexity and regret of standard empirical risk minimization methods, when considering both meta-training and adaptation together? The answer is particularly non-obvious for meta-learning algorithms with complex bi-level optimizations that may demand large amounts of meta-training data. To answer this question, we extend previous meta-learning algorithms to handle the variable-shot settings that naturally arise in sequential learning: from many-shot learning at the start, to zero-shot learning towards the end. On sequential learning problems, we find that meta-learning solves the full task set with fewer overall labels and achieves greater cumulative performance, compared to standard supervised methods. These results suggest that meta-learning is an important ingredient for building learning systems that continuously learn and improve over a sequence of problems.

Under review as a conference paper at ICLR 2021
VARIABLE-SHOT ADAPTATION FOR ONLINE META-
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot meta-learning methods consider the problem of learning new tasks from small, fixed number of examples, by meta-learning across static data from set of previous tasks. However, in many real world settings, it is more natural to view the problem as one of minimizing the total amount of supervision of examples needed to learn a new task and the amount of data needed for meta- learning. Such a formulation can be studied in sequential learning setting, where tasks are presented in sequence. When studying meta-learning in this online setting, a critical question arises: can meta-learning improve over the sample complexity and regret of standard empirical risk minimization methods, when considering both meta-training and adaptation together? The answer is particularly non-obvious for meta-learning algorithms with complex bi-level optimizations that may demand large amounts of meta- training data. To answer this question, we extend previous meta-learning algorithms to handle the variable-shot settings that naturally arise in sequential learning: from many-shot learning at the start, to zero-shot learning towards the end. On sequential learning problems, solves the full task set with fewer overall labels and achieves greater cumulative performance, compared to standard supervised methods. These results suggest that meta-learning is an important ingredient for building learning systems that continuously learn and improve over a sequence of problems.
both the number
find that meta-learning
INTRODUCTION
Standard machine learning methods typically consider a static training set, with a discrete training phase and test phase. However, in the real world, this process almost always eyclical: machine learning systems might be improved with the acquisition of new data, repurposed for new tasks via finetuning, or might simply need to be adjusted to suit the needs of changing, non-stationary world. Indeed, the real world is arguably so complex that, for all practical purposes, learning is never truly finished, and any real system in open-world settings will need to improve and finetune perpetually (Chen & Asch 2017 Zhao et al.) 2019). In this continual learning process, meta- learning provides providess the appealing prospect of accelerating how quickly new tasks can be acquired using past experience, which in principle should make the learning system more and more efficient the course of its lifetime. However, current meta-learning methods are typically concerned with asymptotic few-shot performance (Finn et al., 2017 [Snell al., 2017). For a continual learning system of this sort, we instead need method that can minimize both the number of examples per task, and the number of tasks needed to accelerate the learning process.
Few-shot meta-learning algorithms aim to learn the structure that underlies data coming from a set of related tasks. and use this structure to learn new tasks with only few datapoints. While these algorithms enable efficient learning for new tasks at test time, it is not clear if these efficiency gains persist in online learning settings, where the efficiency of both meta-training and few-shot adaptation is critical. Indeed, simply training model on all received data, i.e. standard supervised learning with empirical risk minimization, a strong competitor since supervised learning methods are known to generalize well to in in-distribution tasks in zero-shot manner. Moreover, it's not clear that meta- lett-le algorithms can improve over such methods by leveraging shared task structure in online learning settings. Provided that it is possible for single model to fully master all of the tasks with enough data, both meta-learning and standard empirical risk minimization approaches should produce model of equal competence. However, the key hypothesis of this work is that learele

---

Under review as a conference paper at ICLR 2021
Round t: Sofa
Round t+1: Airplane
Variable Shot Training
Varialle Shar Adaptation Evaluation Pene the oo oltuch
Varable Shor Training
Fisst paformunce 2C
Gunent dataset
Ø
liftest performance-c
Add dataeet
Figure I: Online incremental meta-learning, We visualize our online incremental meta-lcarning problem setting in the above figure using the Incremental Pose Prediction dataset discussed in Scction At round the current task predict the pose of sofa with small training set containing several datapoints of the sofa with different orientations. After one epoch of training, we evaluate the few-shot generalization performance on a test datapoint where the number of shots equals the size the task's 's training set. If the test performance exceeds some proficiency threshold C, we advance to the next task, i.c. predicting the pose of the airplane. Otherwise, we add another training example of the sofa to the training set and repeat the above process.
models will become more accurate more quickly in the middle of the online learning process, while data is still being collected, resulting in lower overall regret in realistic problem settings.
To test this hypothesis, consider practical online learning problem setting, which we refer as online incremental learning, where the algorithm must learn a sequence of tasks, and datapoints from each task are received sequentially. Once a model reaches a certain level of proficiency, the algorithm may move on to training the model on the next task. This problem is crucial to solve in the real world, especially in settings where online data collection and supervision signals are costly to obtain. One example of such a problem definition is a setting where company receives requests for object classifiers, sequentially at different points in time. Data collection and labels are expensive, and the company wants to spend the least amount of money on acquiring good classifier for each request. A major challenge for meta-learning that arises in this problem setting is to design a meta-learning algorithm that can generalize with variable shots: As data from new tasks is incrementally introduced, at any given point in time, the model may have access to zero, a few, or many datapoints for a provided task. The goal of this work is to achieve variable-shot adaptation while minimizing the total amount of supervision in terms of number of shots required for each added task. The desired online incremental meta-learning algorithm is expected generalize new task with decreasing number of shots over the course of training. We visualize our problem setting in Figure
The key contributions of this work are (a) a new meta-learning algorithm that can adapt to variable amounts of data, and (b) an online version of this algorithm that addresses the above problem setting of online incremental learning. We theoretically derive our variable-shot meta-learning algorithm and combine it with deep neural networks for effective online learning on challenging sequential problem settings. Perhaps surprisingly, we find that our approach can outperform empirical risk minimization and a previous online meta- meta-learming method (Finn et al. (2019) on two online image classification problems consisting of sequences of classification tasks and one online regression problem. Further, we find that, in the offline setting, our approach performs comparably to previous state-of-the-art algorithms in sew-shot learning, and provides considerable gains in the variable-shot setting.
2 PRELIMINARIES
Meta-learning algorithms optimize for efficient adaptation to new tasks. To define the meta-learning problem, let p(T) denote task distribution, where each task T; p(T) consists of dataset D. = {x,y) } with i.i.d. input and output pairs. If we have a predictive model h(x; 0) with some parameter 0 and a loss function f, such as the cross entropy between the predicted label distribution and the true label distribution in a classification problem, the risk of T fe. can be computed as fi(0) (x,))>> [[(h(x; y)] At meta-training time, N tasks {T:). are sampled from p P(T and meta-learning algorithms aim to learn how to quickly learn these tasks such that, at meta-test time, the population risk f, (0) of the unseen tasks T, ~ p(T) is minimized quickly.
In this work, we build on the aggostic meta- meta-learning (MAML) algorithm (Finn et al. 2017). MAML achieves fast adaptation to new tasks by optimizing
of initial parameters OMAMI. that can be quickly adapted to the meta-training tasks [ Thus, at meta-test time, after a small number of gradient steps on 0MAM) with K datapoints from D,. the model can minimize f for the new task Ty. Note that K is a small and fixed number across all tasks. Formally, MAML achieves such an N ( a, K;)) where
initialization by optimizing the following objective: mine
2

---

Under review as a conference paper at ICLR 2021
fK is the inner gradient step size, U, a, K,) 8 a Vof. (0) denotes the gradient update in the K Σ (),,) y,) is the empirical risk of T, based on a minibatch with K datapoints sampled from D.. Hence, MAML can be viewed as optimizing for K-shot generalization.
inner loop for task i, and f.K (0) = 1
(FTML) In the online setting, prior work (Finn et al. 2019 has proposed the whwpow--tee-e-----eeee
algorithm, which adapts MAML to an online meta-lcarning setting where the meta-learner learns to adapt to tasks that arrive in a sequence. In this online meta-learning setting, the meta-learner faces sequence of loss functions (4, ( in each round f, where each of the tasks share some common structure that could be captured by the meta-learner. The goal of the meta-learner is to find a sequence of models parametrized by {0.)2= that performs well on the sequence of loss functions after making some task task-specific update procedures (f( a, K) : 0g E O, K E N- + 8' E @ where K is a fixed number of datapoints used to perform the update. In FTML, which adapts MAML to the online setting, (0,,, a, K). as defined in the paragraph above, is the inner gradient update w.r.t. the empirical risk with Khshot data fo at round f. The performance of FTML is typically measured using the regret that compares the sum of population risks to the cumulative population risks with some fixed model parameter computed in hindsight: Regrety = Σx= (FTL) algorithm (Hannan 1957 Kalai & Vempala
Analogously to the standard follow-the-leane 1= fe( (u, a, K)) - ET 2005), FTML optimizes 0₂ as 0.+1 arg ming f number of datapoints used for computing updates, K, is always fixed in both offline and online meta-learning settings. However, in the sequential learning setting, where datapoints for each task arrive incrementally, the model is required to handle varying numbers of datapoints from different tasks, and hence needs to generalize to new tasks quickly using any amount of data. Our method is designed to tackle this problem setting, which we will discuss in the next section.
tt fo a, K)),
a, K)). As mentioned above, the
3 ONLINE INCREMENTAL META -EARNING PROBLEM STATEMENT
The online incremental meta-learning problem statement extends the online meta-learning set- ting (Finn et al., 2019), where the tasks arrive one at time. In our proposed incremental setting, the data within each task also arrives one data point at a time, and the goal of the model is to minimize the cumulative regret summed over all of the tasks. A model that can adapt a new task more quickly in fewer shots should attain lower regret. A model that can do so while having seen as few previous tasks as possible will be even better. Formally, the learning process is divided into "tasks" and "shots". Each time step, the learner either receives one more "shot" (data point) for the current task, or transitions to a new task and receives the first shot for that task. In practice, instead of generating I datapoint every time step, the learner receives a small batch of datapoints every few time steps. Let t denote the current task counter, and the current shot counter. At each step s for task f, the model has access to (s), the dataset for the current task, which presently contains & shots. The model can adapt to the current task using (s)), and must then attain the lowest possible risk on that task.
To realize this setting, we extend the update procedure defined in Section 2 to the variable-shot setting, formally (,,O,, a, s). The risk in round / with shots in this incremental setting is defined as f.(U,,Ot, at, min (s, M)) where M is some large scalar representing the maximum number of shots per task. The risk defined above suggests that the performance of the model is only evaluated using the s-shot risk fe (U,, a, s)), when there are s <= M datapoints in the current datasets, and always use M-ooot risk when there are more than M datapoints.
Now that we have defined our measure of incremental risk, we will discuss when the model can transition from one task to the next. Intuitively, we want the model to be able to "complete" a task and transition to a new task as soon as it has achieved requisite accuracy on that task. For instance, in the example in Section of the company that receives requests for object classifiers, the company might choose to stop collecting training data for a given task once accuracy on that task is good enough, and move on to the next customer. We formalize this by introducing a threshold value C, such that when (0,,, a, s)) < C at step s, the learner switches to the next task immediately. In this way, very efficient model that can adapt and master task in just few data points will attain lower overall regret and solve the full sequence of tasks with fewer overall datapoints, since it will need fewer shots for each task.
To summarize, the overall online incremental meta-lcarning process looks like this: (1) At the start of round I, the meta-learner selects model parameterized by parameter 0;; (2) The world chooses
3

---

Under review as a conference paper at ICLR 2021
task To- ~ p(T); (3) At each step O..... ...., the dataset (s)s) receives one datapoint; (4) At step s = O...., the model updates 8g as 0/ = U, a, s); (5) At step s = o..... the model is evaluated with (U,, a, s)) and if fi (U, a, s)) I C, the agent will advance to the next round. When the advancement happens, denote St = s. The objective of incremental meta-learning is to minimize the regret over all the rounds, resembling the objective of online learning (Hannan) 1957
Kalai &
Vempala 2005), which we state as follows:
S. f((U,(0r, a, (in M))) min ΣΣ f((U,(0, m, Mf)).
T S.
Regrety
where (0=) is sequence of models round = 1...., T. Similar FTMI., aim to obtain meta-lcarner that learns each task in the data-incremental setting, while attaining minimal regret.
4 VARIABLE-SHIE SHOT
META-LEARNING
The first step toward designing an online incremental meta-learning algorithm that can address this problem statement is to design method that can handle variable numbers of shots, since in the incremental setting, the model will need to accommodate settings ranging from zero-shot at the beginning of each task to many-shot at the end. Attaining the lowest regret therefore depends crucially on maximally utilizing whatever task data may be available. We will therefore first design a variable-shot generalization of MAML in the standard offline setting, and then apply it the online setting to provide a viable approach for online incremental meta-learning.
4.1 NAÏVE SOLUTION
A naive solution would simply involve applying MAML with varying numbers of datapoints used to compute the inner graduate updates. During meta-training, the number of datapoints could be drawn uniformly from {0, (O,.... M) for some M Z+. The corresponding meta-training objective is then
given ming N Σ ()
f:(0- a aV a
Unfortunately, as we will see in Section 7 this approach does not perform well when the number of shots varies, since the adaptation process is not aware of the number of shots that are available. Intuitively, when more data is available, the model should be able to deviate further from the prior parameter vector 0, since the larger dataset provides with more information about the task. Mathematically, this can be captured by the variance of the inner gradient. The variance of
the inner gradient depends on 8 as follows: Var(Vo]((0)) = Var Ef+ ((((x;)))))))
(Ve(((x))( f),y)) y)) where the last equality follows from the property of the variance of the sum of i.i.d. random variables. The above equation indicates that the inner gradient variance inversely scales with the number of shots, but the model is unaware of this fact, since it is only updated via the average gradient. Therefore, the model has way know that it should "trust" the gradient more when more shots are available, leading to poor performance in the variable shot setting. This is also shown empirically in Section 7.1
4.2 THE LEARNING RATE SCALING METHOD
Intuitively, if we have a lower variance gradient, we can take larger steps along that gradient. Therefore, natural way vary the model update based on the number of datapoints is to change the inner learning rate. It is well known in standard supervised learning that the optimal learning rate scales with batch size (Smith & Le 2018). One way to incorporate this observation into variable-shor MAML algorithm would be to meta-learn different learning rates for each number of shots. As we will see in Section 7.T this approach, which we call MAML-VL. (variable-shot and learning), can indeed improve over standard MAMI. However, this requires meta-learning an additional parameter for every possible number of shots, which can become impractical as the maximum number of shots increases.
Instead, we can derive a simple rule for the optimal learning, which we will show can work even better than meta-learning separate rates for each number of shots. First, let us define U.(0, a) 9 - Ve rule for the limiting case of infinite shots, since U,(0, a) lim,-+oo (, s) in distribution by the law of large numbers. Let 8 be the optimal learning rate such that U,(9, (+)) attains optimal performance, i.e. 8 = arg ming Ey, (O, 8))]. This can be interpreted the optimal inner
to be the ideal update rule given the average gradient. We can view this as the update
4

---

Under review as a conference paper at ICLR 2021
learning rate when oc. We want to find an optimal s-shot learning rate a that imitates this behavior for a finite s. Specifically, we solve for the value of a; that minimizes the mean square error between the adapted parameters: of arg min, BT. [1] a, s) (, 11
Theorem 1. The optimal learning rate for the s-shot case can be calculated
as O (1 -1--+ 8 where C1 ET, Var and C2
of TT, [| Val(h((x;))) y)
We provide the proof in Appendix B and thus require re-estimation each time we update during meta-training. practice, we denote Ca as the learned scaling factor, and treat them as == 3 as the learned learning rate and 1] C1
The quantities C2, C2 and 8d are all functions of the parameters
parameters that are meta-learned jointly with O. Denote (3, = (1 1+78s 3 as the learnable scaled learning rate with parameters 3 and 7). In contrast to MAMI--VI,, which must meta-learn a number of learning rates that scales with the maximum number of shots, here we only meta-learn two additional scalar-valued parameters. This leads to our proposed model-agnostic meta-learning with variable-shot and scaling (MAML-VS) algorithm, whose meta-training objective is given below:
N
min 0,8.7 N
E Un--Ui((O [f((0 ((((-aa(3.)Veef100))
(1)
ONLINE INCREMENTAL META-LEARNING WITH VARIABLE-SHOT MAML
With MAML-VS defined in the section above, we are ready to present the algorithm for the online incremental meta-learning setting. following two subsections, we will discuss an online incremental meta-learning algorithm capable of variable-shot generalizations (Section 5.1) and its practical instantiation (Section 5.2)
5.1 FOLLOW THE META LEADER WITH VARIABLE- SHOT AND SCALING
We extend FTML FFinn et al. 2019) to the online incremental meta-learning setting by enabling the meta-learner to handle variable-shot data with the scaling rule MAML-VS. Specifically, in the online incremental meta-learning setting, the model parameter 0 and the scaled learning rate a, (3,3,) defined in Section 4.2 are updated in the following way:
Oct1, (.11,8,11 = arg min
ΣΣ ((U, 0,8,0 J=1
(
We term our online incremental meta-learning algorithm FTML-VS. Intuitively, for the current round + 1, FTML-VS plays the best variable-shot meta- meta-learner in hindsight after each round based all the meta-learned models attained in previous rounds j = 1...t. t. In practice, we of course cannot calculate the exact loss fj. and therefore must approximate it using stochastic gradient descent methods, which will be presented in the next subsection.
5.2 FTML-VS IN PRACTICE
Optimizing Equation 2 is not feasible in practice as it has no closed form solution. We only have access to a portion of T, in round t, and thus the population risk ft is impossible to compute exactly, as discussed by Finn et al. (2019). We adopt online stochastic gradient descent algorithms to resolve these issues. At step in round I, we compute gradients of our model parameters 0, and scale learning rate o (8, 7) where K is the number of shots sampled in minibatch using the following procedure:
1. Draw a task Tj : ~ s((t) (or a minibatch of tasks) uniformly from the set of tasks the agent has seen far {T.) where si() is the uniform distribution on {1,...,4). Let
D; (s) if = / and otherwise Dj = (S,)), which are the data accumulated for task T; in the previous round.
2. Sample K v(j), where v(j) is uniformly distributed on ....... (()., with M())
Mi,
3. Sample minibatch DY Dr(s)) of size K to compute the inner gradient updates using the derived scaled learning rate ox n)-
5

---

Under review as a conference paper at ICLR 2021
4.
Sample another minibatch Djai C (ss) to compute the gradients g". g.h... gg for updating the model parameter 0g along with parameters of scaled learning rate, 8 and 1j respectively.
where we denote = Kwwr()), [A(DTT)
3 U,(O,, ax(3, n). U. a, K) = 0r o'k (8, V 0, (C(D), 0c)) and gº Va,J, J. go = VaJ and g" = V,J. Note that in the case of zero-shot adaptation, i.e. K = O, U,(O,, (3,(), ;) K) corresponds to the parameter at the previous round, i.c. 01-1.
We outline the complete algorithm in Algorithm I in Appendix A with some implementation details in Appendix C The algorithm learns 0, 3, and 17. We also include the training procedure and the evaluation protocol in Appendix A
6 RELATED WORK
Prior work in meta-learning has studied how to acquire learning rules (Schmidhuber) 1987 Bengio
et al. 1992 Hochreiter al.) (00T), accelerate optimization (Andrychowicz al.) 20T6). and acquire priors suitable for few-shot learning (Finn et al. (017)). Modern meta-learning algorithms can broadly
be classified into three high-level approaches: black-box meta-learners that parameterize learning procedure using neural network (Santoro et al. [016 Ravi & Larochelle| 2017 Munkhdalai & Yu
2017
Duan et al.] 2016 Wang et al. [2.6 Mishra et al. (017)). mon--pramettre meta-learners (Koch)
2015 Vinyals et al. 2016) (Seell et al. 2017), and optimization-based
meta-learnen that embed an (Finn et al. 2017: Li et al. 2017b Rusu et al., 2018)) 2018 Rajeswaran et al. 2019 Lee al. 2019] Yoon et al. F018,Finn et al. 2018). We choose to build upon the latter class of optimization based meta-lcarners, since they produce a well-formed optimization process that tends to be robust to out-of-distribution
optimization procedure into the
Zintgraf et al. 2018 Bertinetto et al.
tasks, a useful characteristic in online meta-learning settings.
While some works have evaluated neta-learning algorithms for K-shot N-way classification for breadth of K and N (Snell et al., 2017; Triantafillou et al. 2019| Allen et al.) 2019b at Hsu et al.. 2018 Cao et al. 2020). these works typically train different networks for each value of N and K. with some specifically noting poor generalization to values of K not used in training (Snell et al.
2017 Hsu et al. 2018). Unlike these works, we consider the setting where a single meta-learned model must handle multiple different values of K. This setting has been considered in black- box methods that receive datapoints and make predictions incrementally (Santoro et al. 2016 Woodward & Finn 2017), but this approach is known to underperform the setting where the model is trained for a specific value of K (Mishra et al. 2017). Recent works have proposed
parammertiic meta- learners that work effectively with variable numbers of shots (Allen et al. 2019b; Cao et al. 2020)). Non-parametric methods such as these prior works require at least one shot per class, whereas we consider setting where there is even fewer datapoints available. This is possible in settings with non-muture non-mutually-exxlhue tasks, such that the meta-lcarned model can provide a sufficiently strong prior about the class labels.
Our variable-shot meta-learning setting is specifically motivated by a problem setting that resembles online learning (Shalev-Shward 2012) and continual learning (Thrun 1998). It is well known that algorithms such as follow the leader (FTL) are computationally--Na exppnsiieeeeeeeeee.eeeeeee and many works in online
learning and continual learning focus on developing more computationally--ffcieen methods (Kirk-
patrick et al. 2017 Rebufli et al. 2017: Lopez-Paz et al., 2017, French 1999 Chaudhry et al. 2018)
Zenke et 2017). Following multiple prior methods (F (Finn al.) [2019] Tessler et al., 2016 Rolnick
et al.] 2019). we instead focus on settings where it is practical to maintain a replay buffer of data, and
focus on the problem of effective forward transfer when presented data from a sequence of tasks. We
do so by combining employing meta- vetaaleeannng in the online setting. Unlike prior online meta-learning
works (Finn et al. 2019 Grant et al., 2019 Khodak et al.) 2019 Zhuang et al., 2019), we develop an algorithm that is specifically tailored to leverage variable amounts of data, allowing it to outperform prior tfate-oftthe-ee art online meta-learning approach (Finn et al. 2019). Finally, our work considers a problem setting that is distinct from works that study online continual learning in the inner loop of meta- leeanle (Al-Shedivat et al. 2017 Nagabandi et al., 2018 Javed & White, 2019
Denevi
et al. 2019 He et al. 2019 Harrison et al. 2019).

---

Under review as a conference paper at ICLR 2021
7 EXPERIMENTS
Our experiments aim to address the following questions, in both offline and online problem settings. In the more standard offine meta-learning setting, we aim to answer: (1) How does our variable- meta-learning method compare to prior methods, when evaluated with variable shots in the offline setting? (2) Does our theoretically motivated learning rate rule match the performance of learning per-shot learning rates? We then integrate our variable-shot method into the online setting, to study: (3) Does variable-shot learning improve the regret in the online incremental meta-learning setting? (4) Does our online incremental meta-learning algorithm attain better cumulative regret than standard empirical risk minimization? Details regarding the architecture of the models and the training setup are presented in Appendix D
While method can accommodate any standard meta-learning problem, we intentionally focus our evaluation on meta-learning problems with non-exclusive tasks, since this setting is particularly relevant in the online setting, which we will discuss in the subsection below and since this setting known to be especially difficult (Yin et al. 2019). With non-exclusive tasks, it is technically possible for the model to learn solve all tasks in zero shot, which provides for a fair comparison to non-meta- learning online methods, such as the "follow the learder" method that trains on all data seen so far (indicated as "TOE" in Section 7.2). However, in order to attain the best possible regret, an effective meta-trained model must still learn to adapt, so as to solve new tasks quickly before it has learned to solve all tasks in zero shot. Standard meta-learning methods often perform poorly in this regime, due to memorization problem (Yin et al. 2019), where they learn to only solve the meta-training tasks in zero shot, failing to either solve or adapt to the new task. We use three non-mutually exclusive datasets for our offline and online meta-learning experiments respectively and one mutually-exclusic dataset for the offline setting only, which are listed as follows:
(1) Monnmuttallly exclusive) Rainbow MNIST (Finn et al., 2019). Based the MNIST digit recognition dataset, this dataset consists of digits transformed in different ways: 7 backgrounds with different colors, 2 scales (half and original size), and 4 rotations of 90 degree intervals, which leads to 56 total tasks in combination. Each task contains 900 images applied with one of the transformation listed above (See Appendix D for visualization); (2) (Non-mutually exclusive) Contextual Minilmagenet. This dataset, adapted from Minilmagenet (Ravi & Larochelle, 2016)). www.uaallllalllsssiiiiii
variant of the Minilmagenet few-shot recognition task, such that non-meta-learning baselines can feasibly solve this task. We use the same images and classes as Minilmagenet, but formulate each class as separate binary classification problem. The model receives two images as input: reference image that determines the current task, and query image, and the goal is to output a binary label indicating whether the query image belongs to the same class as the reference image, or not. In this case, each class in the Minilmagenet constitutes task, which makes the task space large and the problem challenging. We include a visualization of the architecture for solving this dataset in Figure iin Appendix D.3 (3) (Non-mutually exclusive) Pose Prediction (Finn et al., 2019; Yin et al. 2019). In this setting, we construct a multi-task dataset with 65 tasks based the Pascal 3DD data (Xiang al. 2014) where each task corresponds to a different object with a random camera angle. We render the image each object using MuJoCo (Todoroy) al. 2012) with a random orientation. We only use this dataset in the online setting; (4) (Mutually exclusive) Omniglot (Lake et al. 2011]. This dataset has 20 instances of 1623 characters from 50 different alphabets where each instance is drawn by a different person. We only use this dataset in the offline setting and include the results in Appendix F
meant to provided
7.1 OFFLINE META-LEARNING EXPERIMENTS
We first evaluate our variable-shot methods, MAML-VS and MAML-VL, in the offline setting, in comparison to MAML (Finn et al., 2017) and MANN (Santoro et al., 2016), which uses a non- gradient based recurrent meta-learrer. The offline experiments mainly serve as samity check of our online incremental experiments, where variable-shot learning is practically important. Experiment setup details, including descriptions of baselines, the dataset setup, and model architectures can be found in Appendix D Note that as discussed in Section 6 nonpparaittric meta-learning methods require at least one shot for each class, making them inapplicable to non-mutually exclusive tasks where the total number of support datapoints is often much less than the total number of classes. Hence, we choose not to compare to those approaches. As shown in Table our method MAML-VL achieves the best performance in 5 out of total settings in both datasets while MAML-VE VS achieves the best performance in the 20-shot classification accuracy on the Contextual Minilmegenet, which
7

---

Under review as a conference paper at ICLR 2021
Datasct
Mothod
0-Shot
I-Sbot
10-Sbot
20-Shot
MANN Santero
2016
66.41 + 1.35
66.49
1.68
67.09 + 1.38
66.95 + 1.12
Rainhow MNIST
MAME (inn et al 2017
71.30 1 0.53
71.13
0.91
74.78 1 1.12
78.67
0.68
MAMI--VL (our))
73.14 0.65
73.45
74.87
0.38
76.59 4: 0..36
MAML--V. VS (ours)
72.72 1 0.45
0.15 72.91
73.89
1 0.60
77.17 1. 0.10
MANN (Santoro 2016
64.85 + 0.01
01.90
61.78 0.63
65.30 4 0.50
Contextual Minilmagenet
MAML (Frmm et al. 2017
61.36 1. 0.48
56,80
0.65
63.82 1. 0.09
63.84 1 0.31
MAMI-VIL (cuns)
61.17
1.95
59.11
3.06
66.09
2.93
66.15 4: 4.74
MAML-VS (ours)
63.37 1 0.71
63.19
1.14
65.89 1 0.10
66.94 1. 0.14
Table I:
Complete Results for Offline Rainbow MNIST and Offline Contextual Minilmagenet. Comparison of MAMI, MANN MAML-VL. (ours) and MAML-VS (ours). The results are classifi- cation accuracies averaged over 3 random seeds, + one standard deviation. On Rainbow MNIST, MAML-VL (ours), that learns learning rate per shot, does better in O, 1, and 10-shot setting, which is the most effective in this relatively simple digit classification task. MAML-VS (ours) achieves comparable performance in all four settings in this experiment. On Contextual Minilmagenet, which is a challenging image classification problem with large task space, MAML- VS (ours) achieves better performance in 20-shot classification accuracies while maintaining competitive performances in O, I and 10-shot settings. Overall, MAML-VS achieves the best or comparable performances in all 8 settings, which indicates that our theoretically-motivee motivated inner learning rate scaling rule would excel in the online setting, which is empirically shown in Table
Method
Incremental Rainhow MNIST
Incremental Contextual Minilmagenet
Incremental Pose Prediction
TOE (Finn et al. 2019
16516.6 1: 172.5
2314.5 1. 363
125.2 d 0.6
FTMI (IMI.(Pimo 2019
4M04.2 2 302.8
10337.7 1 116.1
1335.6 EE 18.8
FIMLAMstO-SG) Lt al. 2017a1
1699.00 0 + 212.0
1027.3 = 35.8
125.0 = 9.4
FTML-VL (ours)
1502.7 177.0
1033.9 21.5
141.3 30.3
FTML (ouns)
4484.7 =: 133.8
1020.0 1 40.8
119.1 1 3.2
Table 2: Final cumulative regret on Incremental Rainbow MNIST, Incremental Contextual Minilmagenet and Incremental Pose Prediction. Results are cumulative regrets averaged over 3 random seeds, + one standard deviation. FTML-VS outperforms other methods in all three domains.
is a challenging problem with large task space and attains comparable performance in all other 7 settings. In both experiments, training for variable shot with a single learning rate (MAML) causes the algorithm to sacrifice O-shot and I-shot performance in favor of better performance with more shots (10 and 20). In the Contextual Minilmagenet setting especially, 1-shot performance significantly lower than even the O-shot performance. For MANN, variable shot training instead biases the model toward good Ohshot and 1-shot performance, and sacrifices performance with more shots. We expect both issues to be problematic in the online setting: good few-shot performance is important in the later stages of training, where each task can be mastered with just a few samples, and good many-shot performance is critical for difficult tasks or early on in training. Our method, MAML-VS and MAML- VL. performs well with all numbers of shots, and performance improves as more examples observed.
7.2 ONLINE INCREMENTAL META-LEARNING EXPERIMENTS
To answer questions (3) and (4), we evaluate FTML-VS in the online incremental setting. To evaluate the performance of our approach, we compare our FTML-VS to: (1) Train on everything (TOE) (Finn et al. 2019), which is the standard empirical risk minimization approach that trains
single predictive model all the data available so far, i.e. { 12 1_ The trained model is directly tested on Diest without any adaptation; (2) FTML (Finn et al., 2019). which trains an online meta- learner with fixed inner learning rate and evaluated on the adaptation performance on Diss. (3) FTML-VL (Ours), which trains an online meta-learner with a learned inner learning rate for each number of shots ranging from 0 to M: (4)FTML+Meta-
(Li et al., 7a), which learns per- parameter learning rates for FTML; (5) Incremental ProtoNet (Snell et 2017), which adapts Snell et al. (2017), one of the most widely used non-parametric meta-learning method to the online incremental meta-learning setting; (6)A-GEM (Chaudhry et al., 2018). which is a popular continual learning algorithm that prevents catastropic forgetting.
For Incremental Rainbow MNIST, we set the proficiency threshold C' at 85% classification accuracy computed minibatch of test data Diest c DD where Djeat n (,(s) @ for each task t, and move on to the next task when this threshold is crossed or after 2000 steps. For Incremental Contextual Mini set the proficiency threshold C' at 75% classification accuracy computed on Dieat
8

---

Under review as a conference paper at ICLR 2021
Datasct
A-GEM (Chaudhyy et al. 2018
FS (ours)
Incremental Rainhow MNIST
14292.19 201.72
4484.70 113.83
Table 3: Comparison between A-GEM Chaudhry et al. 2018 and FTML-VS on Incremental Rainbow MNIST. Results are cumulative regrets averaged over 3 random seeds, + one standard deviation. FTML-VS achieves superior results compared to A-GEM.
Datasel
Incremental ProtoNett (nell
2017
FTML VS (ours)
Incremental Rainbow MNIST
34812.7 6197.7
19207.7 1: 282.4
Table 4: Comparison between Incremental ProtoNet (Snell et al. MNIST with adapted data-generating scheme discussed in Section 7.2 Results are cumulative regrets averaged
2017
and FTML-VS on Incremental Rainbow
over 3 random seeds, + one standard deviation. FTML-VS significantly outperforms Incremental ProtoNet.
For Incremental Pose Prediction, we consider mean-square emor as the metric and do not advance the task automatically. We discuss the details of the online procedure of each dataset in Appendix E We evaluate the performance based on the final cumulative regret Regrety, which computed by accumulating the test losses on Diest during evaluation time for each task t, as shown in Table 2 and include the regret curves over the online incremental meta-learning process of all methods in all domains in Appendix E We also include additional ablation studies in Appendix G
According to Table in all three domains, FTML-VVS attains smallest regret after training all the tasks sequentially compared to other methods, suggesting the importance of using our learning rate rule in the online incremental meta-learning setting with non- mutually exclusive datasets. Moreover, FTML-VS outperforms TOE by large margin, which indicates that our online meta-learning method is able to solve tasks more efficiently than empirical risk minimization by leveraging the shared task structure to quickly adapt to each new task. Finally, FTML-VS also achieves lower final cumulative regret than FTML Meta-SGD, suggesting that our theoretically sound rule of selecting learning rates for different number shots is more effective than simply learning learning rates for each parameter. is also worth noting that learning per- cerpparameter learning rates is complementrry to our method and can be combined to further improve performances. We will leave this for future work.
Moreover, we conduct the comparison between FTML-VS and A-GEM on the Incremental Rainbow MNIST dataset with the same online procedure described above. As shown in Table 3 FTML-VS achieves much better cumulative regret compared to A-GEM. This result is unsurprising, as prior continual learning works focus primarily on minimizing negative backward transfer and compute considerations, as opposed to our goal of accelerating forward transfer through neta-learning.
Finally, for the comparison to Incremental ProtoNet, we consider the Incremental Rainbow Mnist dataset. As described in Section 7.1 paraamamettiic meta-learning methods is applicable to settings where the number of shots is smaller than the number of classes. Hence, in order to adapt [Snell al. (2017), when we do not have the support datapoints for particular class, i.e. digit, of the current task, we compute the prototype of this class by sampling data of this class from previous tasks. We compare FTML-VS VS to Incremental ProtoNet following such data-generaiie and the same online procedure as in the Incremental Rainbow MNIST experiment described in the paragraph above. As shown in Table FTML-VS outperforms Incremental ProtoNet by significant margin, suggesting that our theoretically motivated learning rate selection rule is pivotal in online incremental meta-learning setting compared to non-pan non-parametric meta-learnin methods, which require the minimum number of shots to be greater or equal to the number of classes.
protocol
8 CONCLUSION
In this work, we studied meta-learning in the context of a sequence of tasks. While most meta-learning works study how to achieve few-shot adaptation, succeeding in sequential learning settings requires both meta-training and adaptation to be efficient and performant. Unlike prior works, we focused on an online meta-learning setting where data for each new task arrives incrementally. Motivated by the challenges of this setting, we introduced a variable-shot meta-learning algorithm that optimizes for good performance after adapting with varying amounts of data. Our approach introduces scaling rule for the learning rate that scales with the number of shots. This approach strongly outperformed strictly more expressive approach of learning individual learning rates for each number of shots, validating the correctness of our derivation. On both offline and online meta-learning settings, we observe significant benefits from approach compared to prior methods.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Picter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. CoRR, abs/1710.03641, 2017.
Kelsey R Allen, Evan Shelhamer, Hanul Shin, and Joshua B Tenenbaum. Infinite mixture prototypes for few-shot learning. arXiv preprint arXiv::902.044 2019a.
Kelsey R Allen, Hanul Shin, Evan Shelhamer, and Josh B. Tenenbaum. Variadic learning by bayesian nonparametric deep embedding, 2019b. URL https: /ttp/////peeeeieeeeeeeeeeeeeeeee net forum?? ? ?id=
SJf6BB 6B6AgBg
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 3981-3989 2016.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Optimality in Artificial and Biological Neural Networks, 1992.
Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed- form solvers. arXiv preprint
2018.
Tianshi Cao, Marc T Law, and Sanja Fidler. A theoretical analysis the number of shots in few-
shot learning. In International Conference Learning Representations, 2020. URL https //openreview net/ argeeneeveeee.eee//oooaetateeeee Eormm22d--HccB22210P
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient 2018.
lifelong learning with a-gem. arXiv preprint
Jonathan H Chen and Steven M Asch. Machine learning and prediction in medicine beyond the peak of inflated expectations. The New England journal of medicine, 376(26):500,, 2017.
Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-within-online meta-learning. In Advances Neural Information Processing Systems, pp. 13089-13099, 2019.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning, 1.X17:1611.02279, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning (ICML), 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta- lett-learning. In Advances in Neural Information Processing Systems. pp. 9516-9527, 2018.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. International Conference Machine Learning (ICML), 2019.
Robert French. Catastrophic forgetting in connectionist networks. Trends cognitive sciences, 1999.
Erin Grant, Ghassen Jerfel, Katherine Heller, and Thomas Griffiths. Modulating transfer between tasks in gradient-baat based meta-lcarning, 2019. URL https://opeeeerewwweeee////////e Id=HyxpNnRcFX
James Hannan. Approximation to bayes risk in repeated play. Contributions to the Theory of Games, 1957.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. arXiv preprint arXiv:1912.08866 2019.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu. Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint / 2019.
10

---

Under review as a conference paper at ICLR 2021
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In Interaational Conference on Artificial Neural Networks, 2001.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning, arXiv preprint 1810.02310 2018.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, pp. 18 1-1888, 2019.
Adam Tauman Kalai and Santosh Vempala. Efficient algorithms for online decision problems. J. Comput. Syst. Sci., 71:291-307, 2005.
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Provable guarantees for gradient- meta-learning. arXiv preprini arXiv: 1902: 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinskar et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy Sciences, 2017.
Gregory Koch. Siamese neural networks for one-shot image recognition. ICML Deep Learning Workshop, 2015.
Brenden M Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Conference of the Cognitive Science Society (CogSci), 2011.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. arXiv preprint arXiv: 2019.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835 2017a.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint 1701:00007 2017b.
David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv:1707.03141, 2017.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. International Conference on Machine
Learning (ICML), 2017.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl. arXiv preprint arXiv:1812.0766 2018.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, pp. 113-124, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proc. CVPR, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, pp. 348-358, 2019.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint 1oXivv187705540 2018.
11

---

Under review as a conference paper at ICLR 2021
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with cemmore-ugmmentdd neural networks. In International Conference on Machine Learning (ICML), 2016.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. Diploma thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
Shai Shalev-Shwartz. Online learning and online convex optimization. "Foundations and Trends in Machine Learning", 2012.
Samuel L. Smith and Quoe Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference Learning Representations, 2018. URL https://
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. 1604 2016.
Sebastian Thrun. Lifelong learning algorithms. In Learning learn. Springer, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems (IROS), 2012.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: dataset of datasets for learning to learn from few examples. arXiv preprint arXiv: 1903. 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Neural Information Processing Systems (NIPS), 2016.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning reinforcement learn. arXiv:1611.05763. 2016.
Mark Woodward and Chelsea Finn. Active one-shot learning. arXiv preprint arXiv: 1702.0655,, 2017.
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: benchmark for 3d object detection in the wild. In Conference Applications of Computer Vision (WACV), 2014.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. arXiv preprint arXiv:1912.03822 2019.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian aodel-agnostic meta-learning. In Advances in Neural Information Processing Systems, pp. 7332-7342, 2018
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. Recommending what video watch next: multitask ranking system. In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 43-51, 2019.
Zhenxun Zhuang, Yunlong Wang, Kezi Yu, and Songtao Lu. Online meta-learning on non-convex setting, arXiv preprint arXiv: 191.1099900 2019.
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. arXiv preprint Xrviv: 11100080000343 2018.
12

---

Under review as a conference paper at ICLR 2021
A ALGORITHM OUTLINE
Algorithm Online Incremental Meta- Leannigg with TSS
1: Input: Performance threshold of proficiency, C
2: randomly initialize 8,, .1.8.
3: initialize the task buffer as empty, B +-
4: initialize the regret Regret, <-
5: for l 1.... do
6: initialize (0) =
7: Add B + B+ [T
8: initialize
9: while L (D! 0f) > do
Add datapoints {(x y)} to (s) until [Dr(s) = g(s)
11: Ge. 0,7 VS-Meta -at---ett--udaateeee 8, n, t,,,)
[D.(s) > then
13:
0, VS
( B,v, D.(s))
14:
else
8, - 0e
end if
17:
f ++1
18:
Regret, Regrety + C. (D) ; 0.)
19:
end while
Record regret up until task Ti using Regret,
21:
Set St 6
22:
0.+1 + 0c
end for
Algorithm 2 FTML-VS Subroutines
Input: Hyperparameters parameters 7,
2: function VS - -at-aeett-apaeeeeee (0, (3, B, (,
3: for Tim 1, Numss steps do
4: Sample task T₁: x(t) // (or minibatch of tasks)
5:
Sample the random number of shot ~ v(j).
6:
Sample minibatches Dj with K datapoints and Dj uniformly from D:(s)
7:
Compute gradient gi gi using D Dyat, and Procedure
8:
Update parameters @ +- 0 y ge (or use Adam)
9: Update 3 +- a gg "x VB0K and v7 +- 17 78. V
end for
Return a.
12:
end function
13: function vs-Dpoate-Prccedure(.
D)
14: if [D) MM then
15:
Sample minibatch D ~ D with D M and set D < D
16: end if
Tt 1...., Ngaa steps do - - B ا VL(D,O)
18:
19:
end for
20:
Return 0
21:
end function
For each task t, we discuss the training procedure and evaluation protocol below.
At training time, we maintain a dataset (i(s) that receives one datapoint incrementally at each step of task Tr, i.c. [D]( rate 3, and the learned scaling factor 7 are updated as per Procedure following the procedure discussed above and in VS Mes-MMeaaae -Update in Algorithm in Appendix A
After appending data to Ds)s), the model parameters 0, learned learning
During evaluation, sample minibatch of test data Diest C D, where Di n Di(s) 0. We compute the adapted parameter 6/ using VS-Met a-Met--Upaae a-update, using either all the data for task so far, maximum of M datapoints if more than M datapoints have been collected. Then,
13

---

Under review as a conference paper at ICLR 2021
compute the loss on the test set Die using 9'. Hence, the online incremental meta-learner is evaluated based on the performance from 0 to MM. If the loss the test reaches the proficiency threshold C' (e.g., 90% accuracy), advance to the next task and record the data efficiency of Ti-
adaptation to unseen data with an incremental number of shots ranging
The optimal scaled learning rate for each number of shots is computed as a (3,
I+tR We keep buffer B, initialized as empty, to store the task data seen far. The cumulative regret Regret, is computed by accumulating the test losses during evaluation time for each task I. the model learns to advance to the next task with improving data efficiency, the cumulative regret would tend to level off over number of tasks, which suggests that we achieve adaptation from many-shot to few-shot over the course of the online incremental meta-learning.
B PROOF OF THEOREM
Proof. We apply variance and bias decomposition for the mean squared error to obtain:
Ey, |1, a, 3,(0,33)11
-E,, ED, [1),,(0,a,5) - ED, (0 a. 11 + Bn [| a,,0, U,(0,8")11 0), + (a- BT, [[ (e)(hh(x))))))l
=a
K BT.
Minimizing this quadratic function with respect to a, we get
1
0 =
1
C2 1+
Ci
with the C and C2 as defined in Theorem
C IMPLEMENTATION DETAILS FOR FTML-VS
In practice, we find that using multiple inner gradient updates U, ag (3, n), K) improves the performance of the method.
Moreover, it is also helpful to add meta-reguulrrization YYn et al. 2019) to improve the performance on few-shot adaptation in variable-shot learning and online incremental meta-learning settings. Specifically, we model the parameters with stochasticity and at each round f, 02 ~ q(;; 0, 0z) drawn from distribution q with mean 0, and variance 9,- We typically use q(0: 02z 0z) as a normal distribution. Following Yin et al. (2019). we regularize the meta-objective by adding Dxl(g(0s 0n,, Og (0)) term Procedure where r(0) prior distribution on the model parameters (e.g., standard normal distribution).
D DETAILED INFORMATION FOR OFFLINE EXPERIMENT SETUP
D.1 VISUALIZATION OF DATASETS
We provide visualizations of images in Rainbow MNIST and Pose Prediction in Figure 2 For visualizations of Contextual Minilmagenet, see Figure
D.2 RAINBOW MNIST
D.2.1
DATASET
We use the same dataset as the Incremental Rainbow MNIST dataset in (Finn et al.) 2019). Due to the relatively low zero-shot difficulty of the dataset, we only take the first 20 tasks as the training set.
D.2.2 MODEL ARCHITECTURES
We use the same neural network architecture for MAML and MAML-VS. The network consists of 4 3x3 kernel 32 channel convolution layers with leaky ReLU activation, and a final fully connected layer to output logits for 10 classes. For MANN, first have an embedding layer that converts the 10-class adaptation labels to 256 dimensional vectors. We pass the adaptation image through a feature network consists of 4 3x3 kernel 32 channel convolution layers, followed by fully connected
14

---

Under review as a conference paper at ICLR 2021
+
Figure 2: Visualizations of Rainbow MNIST (Top) and Pose Prediction (Bottom).
Inpui
Corv
Concal
RaLU
Reference
We visualize the Stamese network architecture for Contextual Minilmagenet, which takes in pair of input and reference images and classify they belong same class.
Figure 3:
layer that output 256 dimensional image features. We then concatenate these image features with the label embeddings, and feed it into a GRU layer. The GRU layer outputs a 256 dimensional summary for the adaptation data. We then pass the test images into the same image feature network to obtain 256 dimensional test image features. We concatentate the GRU layer's output with the test image features, and feed them into a final fully connected layer to obtain the logits.
D.3 CONTEXTUAL MINIIMAGENET
We present visualization of the setup in Figure
D.3.1 DATASET
We use the same dataset as the MinilmageNet dataset in (Finn et al. 2017: Snell et al. 2017). ). In order make the tasks non-mutually-exxluse the model receives an ordered pair of two images and classifies whether they belong to the same class. Different tasks in this dataset then correspond the category of the first image in the pair. Each time we sample batch, we sample half batch of pairs with images from the same class and half batch of pairs with images from different tasks. We use the first 90 classes as our training set.
cast into binary classification problem where
D.3.2 MODEL ARCHITECTURES
We use the same neural network architecture for MAML and MAML-VS. The network consists of 3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by 2x2 max pooling layer, followed again by 2 3x3 kernel 64 channel convolution layers with leaky ReL.U activation, followed by a 2x2 max pooling layer and a final fully connected layer that ouputs 2 logits. For MANN, the model is the same feature network consists of 2 3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by 2x2 max pooling layer, followed again by 2 3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by a 2x2 max pooling layer and a final fully connected layer that ouputs 256 dimensional image embeddings.
used in Rainbow MNIST experiments, except
different image
15

---

Under review as a conference paper at ICLR 2021
Cumuletive Regret
Regret Diff w.r.t. FTML
Cumulative Regret
Regret Diff w.r.t. TOE
200
-009
ITM.
-000
100
PL (burs)
PIRELAL S0rre
1000
-20
tasic insex
task index
Tast indre
task index
(a)
(b)
(c)
(d)
Cumulative Regret
Regret Diff w.r.t. FIML
Cumulative Regret
Regret Diff w.r.t. TOE
12/00
100
-100
PTN.
3000
FINL (0urs)
(TN--11 ifumt
-50e
ETHL-A jounti
FinM, vi 10urst
FINL vs (Durs)
TCC
-1000
FIME VS 10uesh
index
(e)
(g)
Figure 4: Curves regret versus number of tasks on Incremental Rainbow MNIST with 3 random seeds. Top row: with automatic advancement to the next task based on proficiency on the current task. Bottom row: training each task with fixed number of steps.
Cumulative Regret
Regret Diff w.r.t. FTML
Cumulative Regret
Regret Diff w.r.t. TOE
nogret
ITH.
-T itver)
FTME wtt Dourt
TOr
PMM (Our))
PINL vs (0us)
FIML V5 (Duss)
Lask indexx"
(a)
(b)
(c)
(d)
Cumulative Regret
Regret Diff FTML
Cumuletive Regret
Regret TOE
regret diffreence
ITNLVS (Ours)
FTHL-V (Curs)
FTNL-V (Durs)
FTHL-S (Ours)
task indax
teak index
lusk Indac.
task index."
(c)
())
(h)
Figure 5: Curves of regret versus number of tasks on Incremental Contextual Minilmagenet with 3 random sceds. Top row: with automatic advancement to the next task based proficiency on the current task. Bottom row: training cach task with fixed number of steps.
D.4 TRAINING FOR VARIABLE NUMBER OF SHOTS
During training time, we train the model with variable number of shots by averaging the loss for different shots. For zero-shot, we directly use the prior parameter without adaptation.
E DETAILED INFORMATION FOR ONLINE INCREMENTAL EXPERIMENT SETUP
E. INCREMENTAL RAINBOW MNIST
During training, we randomly sample a permutation of the 56 classes in Rainbow MNIST, where each class constitutes task. For each task 1,...,56 we add 4 datapoints to (.(s) every 10 steps. We set the maximum number of shots to be evaluated at M = 20. When [D)) E {0, 4, 40).. we evaluate [b)( shot performance, using one half of the task dataset for adaptation (i.e., Diriin and the other half for evaluation (i.c.. ) Note that TOE does not perform few-shot adaptation, but does use the available data for the new task for standard supervised training.
2
For meta- veta-training, we sample 25 tasks from the buffer B at each step, and randomly sample the number of shots K Uniff0, 2......, 20). We use the same architecture of the model in the offline setting. To train FTML, FTML-VEL FTML-VS, we take inner gradient steps with inner gradient initialized 0.1.. Inner gradient magnitudes are clipped within 10. We train all methods
.....
16

---

Under review as a conference paper at ICLR 2021
Curulaative Regret
Regret Diff w.r.t FTML
Curulative Regret
Regret Diff w.r.t TOE
ferencc
MTL
MUWLIO (bors)
PINUAL 20rr
tast indixx
task index
Tack Videe
task index
(a)
(b)
(c)
(d)
Figure 6: Curves of regret versus number of tasks on Incremental Pose Prediction with 3 random sceds.
using Adam with learning rate 0.0001 with gradient clipping 10. As discussed in Section 5.2 we use meta-regularization (Yin et al.
2019) with coefficient 0.1. For FTML-VS, we initialize 7j 1.0.
We run all methods with 3 random seeds with and without the automatic advancement to the next task based on proficiency on the current task discussed in Section 5.2 and include the results in Figure We add datapoints to (s) every and 2 steps for with and without automatic advancement experiments respectively. We train maximum 2000 steps for each task in the setting where automatic advancement happens. In this setting, we advance the next task once the model achieves over 85% success rate on the current task. In the case where we don't advance the task based on the current task performance, we train each task for fixed 800 steps. As shown in Figure 4 FTML-VS outperforms the baselines in both experimental settings, where the advantage of FTML- VS over other baselines is more apparent in the setting where automatic advancement doesn't 't occur.
E.2 INCREMENTAL CONTEXTUAL MINIIMAGENET
As before, we generate a randomly shuffled sequence of 100 classes from Minilmagenet. We pretrain all methods on the first 40 tasks and then feed the next 60 tasks incrementally. We add datapoints to the dataset of the current task every 20 steps, for maximum of 1400 steps per task. We set the maximum number of shots to be M = 20 as before. We set the proficiency threshold C' at 75%
classification accuracy. When [s)] E {0, 8, 16,.... ..., 80), we evaluate [D(()]l shot performance of all the methods except TOE, for the same reason as before.
During training, we sample 16 tasks from B and randomly sample K 2ni 20}. We also use the same architecture of the model as in the offline setting. Similar to Incremental Rainbow MNIST, we take inner gradient steps with inner gradient initialized at 0.03. We train all methods using Adam with learning rate 0.0001 with gradient clipping 10. We also use meta-regularization (Yin ct al. 2019) with coefficient 0.1. For FTML-VS, we initialize = 1.0.
Similar to evalutions of Incremental Rainbow MNIST, We also include the results of 3 random seeds on settings with and without automatic advancement in Figure 5 We add datapoints to Dr(s) every 20 and 10 steps for with and without automatic advancement experiments respectively. For automatic advancement, we train each method with maximum 1400 steps and pick the proficiency threshold at 75%. In the setting without automatic advancement, we train each task for fixed 700 steps. As shown in Figure 5e|5h 5h| FTML-VS outperforms the FTML but gets slightly higher overall regret compared to FTML-VL. where automatic advancement doesn't happen whereas in Figure 5a 5d FTML-VVS achieves the minimal regret among all methods over the course of training.
E.3 INCREMENTAL POSE PREDICTION
Similarly. construct random sequence of the 65 tasks in the 3D pose dataset. We add 4 datapoints to the dataset of the current task every 10 steps, for a maximum of 240 steps per task. We set the
maximum number of shots to be M 15. When [D{(s) E {0, 8,........ 96), we evaluate [é)())). shot performance as in Incremental Rainbow MNIST. In this setting, we consider mean-square ervor and does not advance the task automatically. We set the proficiency threshold C at 85% classification accuracy, and move on to the next task when this threshold is crossed or after 2000 steps.
As before, we present the cumulative regret curves of random seeds on the Incremental Pose Prediction dataset. As shown in Figure 6b 6d FTML-VS attains the best regret among all approaches. Note that TOE outperforms vanilla FTML in this dataset despite that FTML-VS achieves better cumulative regret over TOE, suggesting that empirical risk minimization works better than vanilla online meta-lcarning in this case while our method can further improve the performance of vanilla online meta-learning approaches and surpass the level of empirical risk minimization.
17

---

Under review as a conference paper at ICLR 2021
F OFFLINE EXPERIMENTS ON MUTUALLY EXCLUSIVE TASKS
We compare out method against MAML in the variable-shot settings the Omniglot dataset, which has mutually exclusive tasks. We test 50-way classification on I, 2 and shots in order to make the task more difficult. The results shown in Table We same as our baseline method MAML. This is because in the mutually exclusive settings, the number of datapoints of K-shot classification for computing inner gradient adaptation is K times the number of classes, compared to merely K in the mutually non-exclusive setting. For example, in this case, 5-shot corresponds 250 images for computing inner gradient updates. This is fairly large number of images so that the gradient might have little variance, and therefore large gradient step could be taken without overfitting.
that our method performs about the
Method
1-Shot
2-Shot
5-Shot
MAML
90.94 0.44
95.29 + 0.19
96.81 0.07
MAMI-VI, (ours)
91.08 018 + 0.28
95.35 + 0.26
96.84 + 0.10
MAML- (ours)
90.94 + 0.12
95.05 + 0.23
96.66 + 0.20
Table 5: Offline 50-way Omniglot results.
G ABLATION STUDIES OF ONLINE INCREMENTAL EXPERIMENTS
Method
M 20
=
FTML.
7710.0 769.8
4804.2 + 302.8
4250.7 253.6
FTML-VS (ours)
5643.3 149.0
4484.7 + 133.8
3969.0 E 203.7
Table 6: Ablation study of M, the maximum number of shots, Ineremental Rainbow MNIST.
We conduct an ablation study on the sensitivity of M, the maximum number of shots, on the incremental Rainbow MNIST dataset by comparing FTML and FTML-VV. As shown in Table as M increases, the cumulative regret of both FTML, and FTML-VS decreases since learning becomes easier with larger numbers of shots. Mcanwhile, FTML-VS attains better performance compared to FTML with different values M and the performance gap becomes larger as M decreases, suggesting that our theoretically motivated learning rate scaling rule based on the number of shots is important for different values of the maximum number of shots and particularly effective when the maximum number of shots is small.
18


Reviews:
[{"review_id": "hCF9SQ_Eh9i", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors propose a meta-learning algorithm for online incremental settings where not only the tasks but also the points belonging to each task arrive in a sequential order. In order to effectively minimise the total amount of supervision required by the method (both for meta-training and learning each new tasks), the author extend a previous K-shots learning method to be able to deal with the variable-shots learning scenario that naturally arise when considering the sequential order in which the task datapoints are observed. In the experimental section they compare with standard meta-learning methods in a variable-shots task both offline and online. They also show that in the online case that their method is more efficient than empirical risk minimization.   \n\n## Comments\n\nThe paper is well motivated and the writing is clear although the mathematical notation could be improved.\n\nThe key observation of the paper (which is quite interesting) is that in many real world scenario tasks and datapoints arrive in sequential order, and in that case, it make sense to treat the problem as a variable shot learning problem where the aim is to minimize the total amount of supervision. Based on this observation, the authors proposal is very simple: to modify standard meta-learning algorithms (e.g. MAML offline and FTML online) by rescaling the learning rate in the inner loop according to the variable mini-batch size. The relation between the learning rate and the mini-batch size is given by a function parameterized by two additional parameters, where the functional form is derived in theorem 1. Despite its simplicity, it seems to improve the performance especially in the online setting.\n\nAs for the experimental section, the offline experiments seems a bit artificial and it is not very clear what the application would be. Also, it is unclear why the results for MAML-VL do not appear in the main manuscript, specially since it seems to perform better. On the other hand, the online experiments are more compelling and demonstrate improvement with respect to the baseline. Here I am missing a discussion about the sensitivity of the algorithm to the hyper-parameter M. Finally, while the problem tackled in the paper is different from the online meta-learning with online algorithm in the inner loop (as the authors point out in the related work) it would be interesting to add to the experimental section some baselines from these family of methods since they can be used to solve the problem setting described in the paper. \n\n## Minors\n\n* In the preliminaries a task is defined as a dataset, while I think it should be defined as a distribution from which a dataset is sampled by iid sampling a set of examples. \n* A task dataset is defined as \\mathcal{D}=\\{x_i, y_i\\} implying that x_i represent the covariates of all the examples in the dataset. However, later on at the beginning of page 3 x_j is used as a single datapoint. I would recommend using x_{ij} to avoid ambiguities.\n* At the end of section 2, in the equation for \\theta_{t+1}, I think the minimization should be over \\theta (not over \\theta_j). Same in equation (2).\n* At beginning of section 4.3 it should be added what VL stands for\n* Appendix A: VS-Meta-Update receives \"s\" as input. However, \"s\" does not appear inside the function\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "crhZz5T2gSg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nFollowing Finn et al. 2019, this paper aims at solving incremental meta-learning problems. A new setup is proposed as illustrated in Fig 1, which is motivated by learning a model that is capable of generalizing to a new task with decreasing number of shots. For the offline and online settings of incremental meta-learning, this paper proposes the MAML-VS algorithm and the FTML-VS algorithm, which are based on Finn et al. 2017 and Finn et al. 2019. Offline and online experiments are conducted on 4 benchmarks showing good performance comparing to baselines, where the Contextual MiniImageNet is a new a dataset proposed by this paper. \n\n## Contributions\n1. Proposed a practical setup for few-shot learning when the tasks are non-mutually exclusive.\n2. Proposed the learning rate scaling method for variable shots.\n\n## Issues\n1. It is not clear when and why zero-shot should work if no information about that task is revealed. More details should be elaborated on this point. \n2. In online meta-learning, each task as a different loss function, while in Theorem 1, the result is based on the assumption that all loss functions are the same. Can your result as well as the learning rate scaling method extend to the general setting? \n3. The performance of MAML-VS on Omniglot is worse than MAML. Can you try MAML-VS on other mutually exclusive datasets such as MiniImageNet? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "UA6jwJaVggg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper defined a new problem called “variable-shot adaptation for incremental meta-learning”. In the proposed problem, the data within each task arrives one data point at a time, and the goal of the model is to minimize the cumulative regret summed over all of the tasks. It also proposed an algorithm that aimed to address the problem by using a scaling rule for the learning rate that scales with the number of shots. This algorithm is evaluated on four datasets.\n\nPros:\n\n1- The idea of using a scaling rule for the learning rate that scales with the number of shots is interesting. The authors also provided the proof in the appendix.\n\n2- The method is technically sound, and the experiment results can show its efficiency in some of the settings.\n\nCons:\n\n1- About the problem formulation. This paper defined a new problem named “variable-shot adaptation for incremental meta-learning”. For its formulation, I have the following questions. (a) The system receives one data point at each step (Section 3, Page 3). However, for most incremental learning systems, a batch of training samples arrives at each time, e.g., in [A] and [B]. I think the latter one is more realistic as the training samples in vision systems aren’t often collected one by one. So why do you use a one-by-one setting instead of the common setting used in [A] and [B]? (b) In Section 3, for the Regret_T, you use theta_t instead of theta_T. It means the system is only evaluated on the current task, and you don’t care whether the model forgets the previous knowledge or not. Instead, most existing class-incremental learning and continual learning methods [A, B, I] evaluated the last model theta_T on all previous tasks, i.e., f_t(U_t(theta_T, alpha, min{s, M})) for t=1,...,T. It is commonly agreed that an incremental learning system should have the ability to retain the knowledge for all previous tasks instead of only the current one. If the proposed system is used for learning 100 different tasks, it will need to store 100 different models, which is not realistic given the memory budget in incremental learning. Again, my question is why did this paper choose this different setting (instead of the commonly agreed/used one in related works)?\n\n2- About the baselines. This paper used MAML and FTML as the baselines. It claimed that these baselines don’t work well on variable-shot cases, e.g., can not meta-train different initialization weights for different shot numbers using MAML. While, if so, why not use metric-based methods like [C] and [D]? These methods have proved to be very effective in recent few-shot learning papers [E] and incremental learning papers [B]. Besides, it can be definitely applied to variable-shot learning cases (and in a direct manner).\n\n3- About the novelty. The idea of meta-learning the base learning rates is not novel. It has been widely applied in many related papers, e.g., [F], [G], and [H]. [F] meta-learns base-learning rates for all base-learner parameters. [G] meta-learns layer-wise learning rates. [H] meta-learns a deep model to generate task-specific base learning rates. As a summary, the contribution of this submission is incremental, as it only simplifies the original MAML-VL to MAML-VS for the variable-shot settings. \n\n[A] Tao, Xiaoyu, et al. \"Few-Shot Class-Incremental Learning.\" CVPR 2020.\n\n[B] Rebuffi, Sylvestre-Alvise, et al. \"icarl: Incremental classifier and representation learning.\" CVPR 2017.\n\n[C] Snell, Jake, et al. \"Prototypical networks for few-shot learning.\" NeurIPS 2017.\n\n[D] Vinyals, Oriol, et al. \"Matching networks for one shot learning.\" NeruIPS 2016.\n\n[E] Chen, Yinbo, et al. \"A new meta-baseline for few-shot learning.\" arXiv preprint arXiv:2003.04390 (2020).\n\n[F] Li, Zhenguo, et al. \"Meta-sgd: Learning to learn quickly for few-shot learning.\" arXiv preprint arXiv:1707.09835 (2017).\n\n[G] Antoniou, Antreas, et al. \"How to train your MAML.\" ICLR 2019.\n\n[H] Liu, Yaoyao, et al. \"An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning.\" ECCV 2020.\n\n[I] Li, Zhizhong and Derek Hoiem. \"Learning without forgetting.\" IEEE TPAMI 2017.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-oqhwOZUhm4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors aim to tackle the meta learning problem in online settings, with both training and testing samples received in a streaming fashion. The authors proposed both offline and online methods, MAML-VS and FTML-VS, built upon MAML and FTML, respectively. The key contribution is to learn meta-learning rates besides the initial network parameters. Overall, the proposed approach sounds.\n\nThe contributions of this paper are as follows:\n\n1.\tThe proposed online and offline meta learning algorithms aim to tackle the few-shot meta-learning problems with variable amounts of data received in a stream.\n\n2.\tThe paper shows the limitation of MAML and FTML in the online setting. It also provides theoretical results on the meta learning rates in Theorem 1.\n\n3.\tThe authors have done extensive experiments to evaluate the effectiveness of the proposed approaches in various datasets, e.g., RAINBOW MNIST, Contextual Mini-Imagenet, etc.\n\nTo the current status of the paper, I have a few concerns below.\n\nFirst, the proposed solutions build upon MAML and FTML, which seems incremental. The differences include (i) a modification on the objective function in eq. (1) for the streaming/online setting, and (ii) a meta-learned learning rate for better model convergence property. \n\nSecond, in table 1 (offline setting), it shows that MAML-VS outperforms baselines in fewer-shot cases with Rainbow MNIST, and in 10/20-shot cases with Contextual imageNet. It is unclear how to explain this inconsistency. Overall, the proposed MAML-VS does not perform better than baselines in offline settings.\n\nMoreover, it would be interesting to also show how the proposed approach works in reinforcement learning settings.\n\nMinors:\n\nIn the pseudo-code in section5.2 (line 3), it should be $j=t$.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: For meta-learning with variable shot, this paper proposes a method for adapting the learning rate by a function of the number of training examples. The functional form is theoretically derived, and the method is simple and effective. However, meta-learning methods that adapt learning rates have been proposed, and the novelty is not high enough.
Decision: Reject