Bayesian Online Meta-Learning
Authors: Pauching Yap
Hippolyt Ritter
David Barber
Venue:  (2021)
Keywords: Bayesian online learning
few-shot learning
meta-learning

Abstract: Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem for large-scale supervised classification, little has been done to overcome catastrophic forgetting for few-shot classification problems. Few-shot meta-learning algorithms often require all few-shot tasks to be readily available in a batch for training. The popular gradient-based model-agnostic meta-learning algorithm (MAML) is a typical algorithm that suffers from these limitations. This work introduces a Bayesian online meta-learning framework to tackle the catastrophic forgetting and the sequential few-shot tasks problems. Our framework incorporates MAML into a Bayesian online learning algorithm with Laplace approximation or variational inference.  This framework enables few-shot classification on a range of sequentially arriving datasets with a single meta-learned model and training on sequentially arriving few-shot tasks. The experimental evaluations demonstrate that our framework can effectively prevent catastrophic forgetting and is capable of online meta-learning in various few-shot classification settings.

Under review as a conference paper at ICLR 2021
BAYESIAN ONLINE META-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem for large-scale supervised classification, little has been done over- come catastrophic forgetting for few-shot classification problems. Few-shot meta- learning algorithms often require all few-shot tasks be readily available in batch for training. The popular gradient- model-agnostic meta-learning al- gorithm (MAML) is typical algorithm that suffers from these limitations. This work introduces a Bayesian online meta-learning framework to tackle the catas- trophic forgetting and the sequential few-shot tasks problems. Our framework incorporates MAML into Bayesian online learning algorithm with Laplace ap- proximation or variational inference. This framework enables few-shot classifica- tion range of sequentially arriving datasets with a single meta-learned model and training on sequentially arriving few-shot tasks. The experimental evaluations demonstrate that our framework can effectively prevent catastrophic forgetting and is capable of online meta- meta-learning in various few-shot classification settings.
INTRODUCTION
Image classification models and algorithms often require an enormous amount of labelled exam- ples for training to achieve state-of-thh-an performance. Labelled examples can be expensive and time-consuming to acquire. Human visual systems, on the other hand, are able to recognise new classes after being shown few labelled examples. Few-shot classification (Miller et al. 2000 Li jet al.) 2004 2006 Lake et al. [201T) tackles this issue by learning to adapt to unseen classes (known as novel classes) with very few labelled examples from each class. Recent works show that meta- learning provides promising approaches to few-shot classification problems (Santoro et al. 2016 Finn et al. 2017 Li et al.) 20177; Ravi & Larochelle, 2017). Meta-learning
-earning-to-learn (Schmidhuber 1987 Thrun & Pratt 1998) takes the learning process a level deeper instead of learning from the labelled examples the training classes (known base classes), meta-learning learns the example-learning process. The training process in meta-learning that utilises the base classes called the meta-training stage, and the evaluation process that reports the few-shot per- formance on the novel classes is known as the meta-evaluation stage.
Despite being a promising solution few-shot classification problems, meta-learning methods suf- fer from several limitations:
1. Unable to continually learn from sequential few-shot tasks: It is mandatory to have all base classes readily available for meta-training. Such meta-learning algorithms often require sampling number of few-shot tasks in every iteration for optimisation.
2. Unable to retain few-shot classification ability on sequential datasets that have evident distributional shift: A meta-learned model restricted perform few-shot classification on a specific dataset, in the sense that the base and novel classes have to originate from the same dataset distribution. A meta-learned model loses its few-shot classification ability on previous datasets as new ones arrive subsequently for meta-trant meta-training.
We emphasise that the task mentioned in this paper refers the few-shot task for meta-learning. This paper considers meta-learning a single model for few-shot classification in the sequential datasets and sequential seo- tasks settings respectively.

---

Under review as a conference paper at ICLR 2021
We introduce Bayesian online meta-learning framework that train a few-shot learning model under the sequential few-shot tasks setting and train a model that is applicable to a broader scope of few-shot classification datasets by overcoming catastrophic forgetting. We extend the Bayesian online learning (BQL.) framework (Opper) 1998) to a Bayesian online meta-learning framework us- ing the model-agnostic meta-learning (MAML) algorithm (Finn et al., 2017). MAML. finds good model parameter initialisation (called meta pettaparameters) that can quickly adapt to novel classes using very few labelled examples, while BOL provides a principled framework for finding the pos- terior of the model parameters. Our framework aims to combine both BOL and MAML to find the posterior of the meta- peraa-aarmssters. Our work builds on on/Rtt et al. (2018a) which combines the BOL framework and Laplace approximation with block-diagonal croonccer-factooee Fisher approxima- tion, and Nguyen et al. (2018) which uses variational inference with BOL to overcome catastrophic forgetting in large-scale supervised classification.
An important reason to implement Bayesian inference over non-Bayesian methods for an online setting is that BOL provides a grounded framework that suggests using the previous posterior as the prior recursively. Bayesian inference inherits an advantage for robust meta-learning (Yonn et al.) 2018) to overcome training instability problems addressed by Antoniou et al. (2019). BOL implicilly keeps a memory on previous knowledge via the posterior, in contrast to recent online meta-learning methods that explicitly accumulate previous data in task buffer (Finn et al. 2019 Zhuang al., 2019). Explicitly keeping a memory on previous data often triggers an important question: how should the surried-frwarre data be processed in future task rounds, in order to accumulate knowl- edge? Finn et al. (2019) update the meta-paresn peeaa-aarsssters at each iteration using previous few-shot tasks in the task buifer. This defeats the purpose of online learning, which by definition means to update the parameters each round using only the new data encountered. Having to re-train on previous data to avoid forgetting also increases the training time as the data accumulate (Finn et al. 2019; (He) eet al., 2019). Certainly one can clamp the amount of data at some maximal limit and sample from the buffer, but the final performance of such an algorithm would be dependent on the samples being informative and of good quality which may vary across different seed runs. In contrast to memoris- ing the datasets, having an implicit memory via the posterior automatically deals with the question on how to process farried-forward data and allows a better carry forward in previous experiences.
New
Below are the contributions we make in this paper:
We develop the Bayesian online meta-learning (BOMI.) framework for sequential few-shot classification problems. Under this framework we introduce the algorithms Bayesian online meta-learning with Laplace approximation (BOMLA) and Bayesian online meta-learning with variational inference (BOMVI).
We propose simple approximation the Fisher corresponding to the BOMLA algorithm that carries over the desirable block-diagonal Kronecker-factored structure from the Fisher approximation in the non-meta-learning setting.
We demonstrate that BOML can overcome catastrophic forgetting in the sequential few-shot datasets setting with apparent distributional shift in the datasets.
We demonstrate that BOML can continually learn to few-shot classify the novel classes in the sequential meta-training few-shot tasks setting.
META-LEARNING
Most meta-learning algorithms comprise an inner loop for example-lcaring and an outer loop that learns the example-lcarning process. Such algorithms often require sampling a meta-batch of tasks each iteration, where task formed by sampling subset of classes from the pool of base classes or novel classes during meta- meta-training or neta-evaluation respectively. The N-way K-shou task, for instance, refers to sampling N classes and using K examples per class for few-shot quick adaptation.
An offline meta-learning algorithm learns few-shot classification model only for a specific dataset Di+1 where all base classes of D:+1 have to be readily available for meta-training. For notational convenience, we drop the t + subscript in this section, as there is only one dataset involved in offline meta-learning. The dataset 1 is divided into the set of base classes D and novel classes D for meta-training and meta-evaluation respectively. Upon completing meta-training on the base class
2

---

Under review as a conference paper at ICLR 2021
set D. the goal of few-shot classification is to perform well on unseen task D' sampled from the novel class set D after quick adaptation small subset D (known as the support set) of D The performance of this unseen task evaluated on the query set D'. where D' D Dr... Since 'D is not accessible during meta-training, this support-query split is mimicked on the base class set D for meta-training.
Model-agnostic meta-learning We are interested in the well-known meta-learning algorithm MAML (Finn et al. 2017). Each updating step of MAML aims to improve the ability of the meta- parameters to act as a good model initialisation for a quick adaptation on unseen tasks. Each iteration of the MAMI algorithm samples M tasks from the base class set DD and runs a few steps of stochas- tic gradient descent (SGD) for an inner loop task-specific learning, The number of tasks sampled per iteration is known as the meta-batch size. For task m, the inner loop outputs the task-specific parameters 0Th from step SGD quick adaptation on the objective L(0, Dim.S) with the support set Dmn.S and initialised 0:
8Th = SGDx(C(O, (C
(1)
where m ,,...,M. M. The outer loop gathers all task-specific adaptations to update the meta- parameters 0 using the loss L(0Th, (Dm.Q) on the query set D2m.?
The overall MAMI, optimisation objective is
M Σ CISGD)(C(0, B)..0). m==1
1 M
arg min
(2)
Like most offline meta-learning algorithms, MAML requires all base classes to be readily available for tasks-sampling at each iteration. We aim to overcome this limitation by meta-learning model that can few-shot classify unseen tasks from the novel classes, while the tasks from the base classes arrive sequentially for meta-training. MAML also assumes stationary task distribution during meta-training and meta-evaluation. Under this assumption, a meta-learned model is only applicable to specific dataset distribution. When the model encounters sequence of datasets with appar- ent distributional shift, it loses the few-shot classification ability on previous datasets as new ones arrive for meta-training. Our work also aims meta-learn single model for few-shot classifica- tion on multiple datasets that arrive sequentially for meta-training. We achieve these two goals by incorporating MAML into the BOL framework to give the Bayesian online meta-learning (BOML) framework that finds the posterior of the meta-parana parameters.
3 OVERVIEW OF OUR BAYESIAN ONLINE META-LEARNING APPROACH
Our central contribution is to extend the benefits of meta-learning to the Bayesian online scenario, thereby training models that can generalise across tasks whilst dealing with parameter uncertainty in the setting of sequential tasks or sequential datasets.
Sequential datasets setting In this setting, online meta-training occurs sequentially on the @@@ datasets D,,..., .., Dr. Each dataset D; be derlying task distribution pTT.). newly-arrived D++1 1 is separated into the base class set D and novel class set D:41 for meta- metattraining and meta-evaluattin respectively, where the tasks in these two stages are drawn from the task distribution ((T++1).
knowledge domain with associated un-
New
Sequential tasks setting The sequential tasks setting only involves one dataset D with an associ- ated underlying task distribution p(T), where D separated into the base and novel class sets. In this setting, D,,... Dr+1 denote the non-overlapping tasks formed from the base class set and they arrive sequentially for meta- seeta-raiiing. These tasks D,...., De+1 and the meta-evaluation tasks are drawn from the task distribution p(T).
@ @@@ @
New
Notationally, for both sequential tasks and sequential datasets settings, let D. and D denote the collection of support sets and query sets respectively from D that D Di D41-
1+1
3

---

Under review as a conference paper at ICLR 2021
We are interested in MAP estimate 0" arg max, p(8(D1a01). p gives the recursive formula
Using Bayes' rule on the posterior
p p(0)Drat1) p(D 104
(3)
= =0(D D (( p 11a)
(4)
- - p( p(0 Di d0 p(Dii110) ) p(())))))((00).)))
(5)
where Eq. (3 follows from the assumption that each dataset is independent given 0.
From the meta-learning perspective, the parameters 9 introduced in Eq. (5) can be viewed as the task-specific parameters in MAML. There are various choices for the distribution p(0 D in
Eq. (5) In particular if we choose to set it as the deterministic function of taking several steps of SGD on loss L with the support set collection D and initialised at 0, we have
0010 1 1 (0 = SGD, 1)}
(6)
and this recovers the MAML inner loop with SGD quick adaptation in Eq. (I) The recursion given by Eq. (5) forms the basis of our approach and the remainder of this paper explains how we implement this. order do so we give mini tutorial Appendix A on Bayesian online learning.
Laplace approximation and variational continual learning.
BAYESIAN ONLINE META-LEARNING IMPLEMENTATION
This section demonstrates how we arrive at the algorithms Bayesian online meta-learning with Laplace approximation (BomLA) and Bayesian online meta-learning with variational inference (BOMVI) by implementing Laplace approximation and variational continual learning respectively to the posterior of the BOML framework in Eq. ()) These algorithms from the grounded BOML framework are useful for online training the sequential few-shot classification datasets tasks.
4.1 BAYESIAN ONLINE META-LEARNIN WITH LAPLACE APPROXIMATION
We discover that the Laplace approximation method provides a well-fitted meta-training framework for Bayesian online learnig Eq. (5). Each updating step the approximation procedure can be modified to correspond to the meta- meta- parameters for few-shot classification, instead of the model parameters for large-scale supervised classification.
Laplace approximation rationalises the use of Gaussian approximate posterior by Taylor expanding the log-posterior around mode up to the second order, described in Appendix A. The second order term corresponds to the log- probability of Gaussian distribution. The BOML framework in Section 3 with a Gaussian approximate posterior of mean and precision d, = {1r, Ar3 from the Laplace approximation gives MAP estimate:
{ log
db Log ; (. -ud) (7)
0" = arg max
0
For an efficient optimisation, we the deterministic Ô in Eq. (6) The objective in Eq. 7 can be batched (for sequential tasks) or meta- meta-batched (for sequential datasets). This leads to minimising the objective
M Σ
BOMLA (0, pc, Ar) = M Σ f
log DDi p0m)
m.
logg( D + (1)= Ae(O pt),
(8)
where 6m = SGD, S) for 1,..,, M. In the sequential datasets setting M denotes the number of tasks sampled per iteration, whereas in the sequential tasks setting M denotes the number of batches per epoch. The first term of the objective in Eq. (8) corresponds to the MAML objective in Eq. 2 with a cross-entropy loss, the second term can be viewed as the pre-adaptation loss the support set and the last term can be seen regulariser.
4

---

Under review as a conference paper at ICLR 2021
4.2 HESSIAN APPROXIMATION
We calculate a block-diagonal Kronecker-factored Hessian approximation in order to update the precision A,, as explained in Appendix A.3 The Hessian approximations in both sequential datasets and sequential tasks settings are very similar, except that the sequential datasets setting averages over the meta-batch size and the sequential tasks setting averages over the number of batches.
The Hessian matrix corresponding to the first term of the BOMLA objective in Eq. (8) is
M
22
Σ
(B000G) log (Dm 2+1
((m))
8=p++1
is worth noting that the BOMLA Hessian deviates from the original BOL Hessian Ap- pendix A.2, This requires deriving an adjusted approximation to the Hessian with some further assumptions.
The BOL Hessian for single data point be approximated using the Fisher information matrix F to ensure its positive semi-definiteness (Martens & Grosse 2013):
F
Ex,y
log log (x, 0) log logp(ulæ, 0)7
(10)
d8
db
Each (z. y) pair for the Fisher in BoMLA is associated task (or batch) m. The Fisher infor- mation matrix corresponding to the BOMLA Hessian in Eq. (9) for single data point is
1
Σ E Bx.y
(06m
d
logp(v/æ, 8m) d dhm
log @m))
@06m 27 )²
(11)
M
08
dhm
00
The additional Jacobian matrix a6m
29 breaks the cenneckrr-factorde structure described by Martens & Grosse 2015) for the original Fisher in Eq. (IO).
The results in Finn al. (2017) show that the first step of the quick adaptation in 6thh contributes the largest change to the meta-evas mneta-vvaluation objective, and the remaining adaptation steps give a relatively small change to the objective. It is reasonable to assume that the quick adaptation is one-step SGD for Fisher approximation:
fTh = 0 - (oL D
(12)
By imposing this assumption, the (i, entry of the Jacobian term can be interpreted as
3gm ij
00 = 1 0 (-
(13)
a0(0
where is the corresponding identity matrix and the objective L involved is the negative log- likelihood. The Hessian for single data point in the second term of Eq. (13) can be approximated by F in Eq. (10) via the usual block-diagonal Kronecker-factored approximation. Putting the Jaco- bian back into Eq. (TT) and expanding the factors give terms that multiply two or more Kronecker products together. The detailed derivation F is explained in Appendix A.331 We introduce the posterior regulariser À when updating the precision: Ar+1 AH,+1 + A, and the rationale for introducing À is explained in Appendix A.3.2 The pseudo-code of the BOMLA algorithm can be found in Appendix B.]
4.3 BAYESIAN ONLINE META-LEARNIE WITH VARIATIONAL INFERENCE
The variational continual learning (VCL) framework (guuyen et al. 2018 is directly applicable to BOML. This section demonstrates how we arrive at the BOMVI algorithm by implementing VCL to the posterior of the BOMI. framework in Eq. (5).
As described in Appendix A.4 VCL approximates the posterior by minimising the KL-ddieegencc over some pre-determined approximate posterior family Q. Fitting the BOML posterior in Eq. (5)
into the VCL framework gives the approximate posterior:
((f))411 arg min DKL q
00000 DE.)) dē
) 1002)110) (14)
5

---

Under review as a conference paper at ICLR 2021
Similar to BOMLA, we the deterministic @ in Eq. (6. and the objective in Eq. (4) can be batched (for sequential tasks) or meta- batthbbtcched (for sequential datasets). This leads to minimising the objective
M Σ
BBOMVI (0, (0)
1 =
M Σ
E [0oe) (Do 10T)
Eecolo) [ogg(("(+11)) (15)
M
+ DKL (1000100))))((())))))
where = SGDx D'41")) }) for m I,..., M. In the sequential datasets setting M denotes
the number of tasks sampled per iteration, whereas in the sequential tasks setting M denotes the number of batches per epoch. We use Gaussian mean-field approximate posterior ((0)
of,a). where &e = {ue,a: or,a) minimised over Φ. The pseudo-code of the BOMVI algorithm can be found Appendix B.
1 D = dim(@) and the objective in Eq. 15 is
The first term in Eq. 15 is rather cumbersome to estimate in optimisation. To compute its Monte Carlo estimator, we have to generate samples 0,- ~ q for = I..... R. and run quick adaptation on each sampled meeaa-arameters 0, before evaluating its ligeliill.oood. This is computationally intensive and it gives an estimator with large variance. We propose a workaround by modifying the inner loop SGD quick adaptation, and the details can be found in Appendix B.2
RELATED WORK
Online Meta-Learning There are two common problem settings in the current online meta- learning works:
@ New
Sequential tasks assumed to originate from the same underlying task distribution ((siiiioon()))))))) in this setting. Our work in the sequential tasks setting be- longs to this category. (Denevi et al. (2019) introduce the online-within-bon batch (OWB) settings, where OWO encounters tasks and examples within tasks sequentially while OWB encounters tasks sequentially but examples within tasks are in batch. The BOML framework in the sequential tasks setting corresponds to the OWB set- ting. On the other hand, our work in the sequential datasets setting is novel in overcoming few-shot catastrophic forgetting, where the goal is to few-shot classify tasks drawn from a sequence of distributions (T........ p......pp(pp (T as explained in Section He et al. (0119). Harrison et al. (2019) and Perfel et al. (2019) look into continual meta-learning for non- stationary task distributions where the task boundaries are unknown to the model. Jerfel et al. (0119) consider latent task structure to adapt to the non-stationary task distributions.
Underlying task distribution:
(OWO) and
Regret minimisation: In this setting, the goal is to minimise the regret function, and the assumptions are made on the loss function rather than the task distribution. Recent works Finn et al. (219)); Zhuang et al. (2019) belong this category, where the aim is to compete with the best meta- leaener and supersede it. These methods accumulate data as they arrive and meta-learn using all data acquired so far. Data accumulation is not desirable as the algorithmic complexity of training grows with the amount of data accumulated, and training time increases as new data arrive (Finn et al., 2019 He et al.] 2019). The agent will eventually run out of memory for a long sequence of data. The BOML framework on the other hand is advantageous, consideration during optimisation. This gives a framework with an algorithmic complexity independent of the length of the dataset sequence.
it only takes the posterior of the meta- peta-parrmeters into
Offline Meta-Learning Previous meta-learning works attempt solve few-shot classification problems in an offline setting, under the assumption of having a stationary task distribution dur- ing meta-training and meta-evaluation.
@ @@C
New
single meta-learned model is aimed to few-shot classify one specific dataset with all base classes of the dataset readily available in batch for meta-training There are two general frameworks for the offline meta-learning setting:
Probabilistic framework: The MAML algorithm can be cast into a probabilistic infer- ence problem (Finn al. 2018) with hierarchical Bayesian structure (Grant et al.)

---

Under review as a conference paper at ICLR 2021
2018). Grant et al. (2018) discuss the use of Laplace approximation in the task spec-ice inner loop to improve MAML using the curvature information, whilst
2018
Yoon al.
Yoon et al. (2018) use Stein Variational Gradient Descent (SVGD) for task-specific learn- ing. Gordon et al. (2019) implement probabilistic inference by considering the posterior predictive distribution with amortised networks.
Non-probabilistic framework: Gradient-basce meta-learning (Finn et al. 2017 Nichol et al. 2018; Rusu et al. 2019) updates the meta-parameters by accumulating the gradients of a meta- bsta-batch of task-specific inner loop updates. The meta- meta-paramann parameters will be used model initialisation for a quick adaptation on the novel classes. Metric-based meta- learning (Koch et al.) 2013:: VVnyall et al.] 2016; [Snell et al.] 2017) utilises the metric distance between labelled examples. Such methods assume that base and novel classes from the same dataset distribution, and the metric distance estimations can be generalised the novel classes upon meta-learning the base classes.
Continual Learning Modern continual learning works (Goodfellow et al. 2013 Lee et al. 2017
Zenke et al.) (017) focus primarily on large-scale supervised learning, in contrast to our work that looks into continual few-shot classification across scquential tasks and datasets. [Wen et al. (2018) utilise few-shot learning to improve overcoming catastrophic forgetting via logit matching
small sample from the previous tasks. The online learning element in this paper is closely related to (Kirkpatrick al.] 2017 Zenke et (017 Ritter et al. [2018|] Nguyen all] 2018) that overcome catastrophic forgetting for large-scale supervised classification. In particular, our work builds on the online Laplace approximation method in (Ritter al. 20T8a). We extend this the meta-learning scenario to avoid forgetting in few-shot classification problems. Nguyen et al. (2018) provide the alternative of using variational inference instead of Laplace approximation for approximating the posterior. It is a reasonable approach to adapt variational approximation methods to approximate the posterior of the meta- parameersss by adjusting the KL.-divergence objective.
6 EXPERIMENTS
6.1 OMNIGLOT: SEQUENTIAL. TASKS
We run the sequential tasks experiment on the Omniglot dataset. To increase the difficulty level, we split the datasets based on the alphabets (super-classes) instead of the characters (classes). The goal of this experiment is to classify the 5-way 5-shot novel tasks sampled from the meta-evaluation alphabets. The experimental details and the alphabet splits can be found in Appendix C.I
We compare algorithms to the following baselines:
1. Train-On-Hin On-Everything (TOE): When new task (or dataset) arrives for meta- traiinnai we randomly re- re-initialise the datasets) encountered so far. Once meta-training is completed in this stage, we do not update the posterior of the meta-parameters like we would in BOMLA and BOMVI.
and perform meta--raining on all tasks (or
2. Train-From From-Scratch (TFS): Upon the arrival of new task (or dataset), we randomly initialise the meea--parameters and meta-train only on the newly-arrived task (or dataset). Similar to TOE, the posterior of the meta-parameters is not updated in TFS.
3. Follow The Meta-Leader (FTML): We introduce slight modification FTML Finn (et al.] 2019) on its evaluation method, as FTML is not designed for few-shot learning on unseen tasks. In our experiment, we apply Update-Proced re in FTML. to the data from unseen tasks, rather than the data from the same training task as in the original FTML.
New2:
FTML.
& error-
band
As the tasks arrive sequentially for meta-training, Figure 1 shows that BOMLA and BOMVI can accumulate the few-shot classification ability on the novel tasks over time. The knowledge acquired from previous meta-training tasks are carried forward in the form of a posterior, which is then used as the prior when a new task arrives for meta- trainnta. The baselines TOE and TFS have similar performances. Despite having access all previous tasks, TOE shows no positive forward transfer in the meta-evaluation accuracy each time it encounters a new task. BOMLA with À = 0.1 gives the best performance in this experiment.
7

---

Under review as a conference paper at ICLR 2021
Weta-evaluationn accuracy (%)
96
92
92
94
90
93
92
86
91
BOMIA
BOMM
11MML
BOMIA, - 0.1
BOMLA,
82
TF5
82
TFS
89
BONVI
50
200
150
200
250
50
150
200
250
50
100
150
200
250
Meta-training task index
Figure 1: Meta-evaluation accuracy across seed runs on the novel tasks along meta-training. Left: compares BOMLA to the baselines, centre: compares BOMVI to the baselines, right: compares BOMLA with different À values to BOMVI.
6.2 PENTATHLON: SEQUENTIAL DATASETS
We implement BomLA and BOMVI to the pentathlon 5-way 1-shot classification sequence:
Omniglot CIFAR-FS minilmageNet -> VGG-Flowers Aircraft
The details of this experiment and the datasets can be found in Appendix C.2 We compare BOMLA
and BOMVI to the baseline TOE, and running MAML continuously on the sequential datasets for meta-training.
Figure "eeureeshow shows that BOMLA and BOMVI are able to prevent few-shot catastrophic forgetting. TOE is also able to retain the few-shot performance as it has access to all datasets encountered so far. However, since it learns all datasets from random re-initialisation each time encounters new dataset, the meta-training time required to achieve similarly good meta-evaluation performance is longer compared other runs. The sequential MAML, on the other hand, catastrophically forgets the previously learned datasets but has the best performance runs. TOE can be memory-intensive as the dataset sequence becomes longer. It takes the brute- force approach to prevent forgetting by memorising all datasets. Unlike TOF, our BOMI approach only takes the posterior of the meta-parameter into consideration during optimisation. This gives framework with algorithmic complexity independent of the length of the dataset sequence.
@ @ @
New2:
band
new datasets compared to other
Tuning the posterior regulariser mentioned Section 4.2 corresponds to balancing between a smaller performance trade-off new dataset and less forgetting previous datasets. As shown in Appendix C.2 Figure a larger = 1000 results in a more concentrated Gaussian posterior and therefore unable learn datasets well, but better retain the performances previous datasets. smaller value
New2:
comparing in
on the other hand gives a widespread Gaussian posterior and learns better on new datasets by sacrificing the performance on the previous datasets. In this experiment, the value À = 100 gives the best balance between old and new datasets. Ideally we seek for good performance
App. C.2
both old and new datasets, but in reality there is trade-off between retaining performance on old datasets and learning well on new datasets due to posterior approximation errors.
As shown in Figures I and 2 BOMLA with appropriate À is superior BOMVL This is due to BoMLA having better posterior approximation than BOMVI. Whilst BOMLA has a Gaussian approximate posterior with block-diagonal precision, BOMVI uses a Gaussian mean-field approx- imate posterior. TTripe & Turner (2017) compared the performances of variational inference with different covariance structures, and discovered that variational inference with block-diagonal covari- ance performs worse than mean-field approximation. This is because the block-diagonal covariance in variational inference prohibits variance reduction methods such as local reparameterisation trick for Monte Carlo estimation. The variance of the Monte Carlo estimate has been proven problematic (Kingma et al., 2015) Trippe & Turner 2017). We address this issue in Section Appendix specifically to the meta-learning setting by modifying the inner loop quick adaptation.
@
New
8

---

Under review as a conference paper at ICLR 2021
accuracy (%)
100
Ommiglot
8b
60
CIFMI-FS
50
45
minhmagen
40
VGG-Flowers
6o
BOMLA, A 100
50
Baseline
Aircraft
Sequential MAML TOE (BOMLA)
TOE (BOMVI)
Omniglot meta-train
CIFAR-FS meta-train
minlimagenet meta-train Meta-training time
VGG-Flowers meta-train
Aircraft meta-train
Figure 2: Meta-evaluation accuracy across 3 seed runs each dataset along meta-training (refer to Figure 3 for the enlarged version). Higher accuracy values indicate better results with less forgetting proceed to datasets. BOMLA with À 100 gives good performance in the off- diagonal plots (retains performances on previously learned datasets). and has a minor performance trade-off in the diagonal plots (lears less well new datasets). Sequential MAML gives better performance in the diagonal plots (learns well on new datasets) but worse performance in the off- diagonal plots (forgets previously learned datasets). BOMVI also able retain performance on previous datasets, although it may be unable to perform as good as BoMLA due to sampling and estimator variance.
CONCLUSION
We introduced the Bayesian online meta-learning (BOML) framework with two algorithms: BoMLA and BOMVI. Our framework overcome catastrophic forgetting in few-shot classification prob- lems and can handle sequentially arriving few-shot tasks for online meta-learning. BOML merged the BOL framework and the MAMI. algorithm via Laplace approximation variational contin- ual learning. We proposed the necessary adjustments in the Hessian and Fisher approximation for BOMLA, as we are optimising the meta-parsmeters for few-shot classification instead of the usual model parameters in large-scale supervised classification. The experiments show that BoMLA and BOMVI are able retain the few-shot classification ability when trained on sequential datasets with evident distributional shift, resulting in the ability to perform few-shot classification on multiple datasets with a single meta-learned model. BOMLA and BOMVI are also able continually learn to few-shot classify novel tasks as the meta-training tasks arrive sequentially for learning.
REFERENCES
A. Antoniou, H. Edwards, and A. Storkey. How to Train Your MAML. In International Conference on Learning Representations, 2019.
L. Bertinetto, F. Henriques, P. Torr, and A. Vedaldi. Meta-Learning with Differentiable Closed- Form Solvers. In International Conference Learning Representations, 2019.

---

Under review as a conference paper at ICLR 2021
A. Botev, H. Ritter, and D. Barber. Practical Gauss-Newter Optimisation for Deep Learning, In Proceedings of the 34th International Conference on Machine Learning, 2017.
G. Denevi, D. Stamos, C. Ciliberto, and M. Pontil. Online vances in Neural Information Processing Systems 32, 2019.
Online-Within-nnle Within-Online Meta-Learning. In Ad-
J. S. Denker and Y. LeCun. Transforming Neural-Net Output Levels to Probability Distributions. In Advances in Neural Information Processing Systems 3, 1991.
C. Finn, P. Abbeel, and S. Levine. Model-Agnostie Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning. 2017.
C. Finn, K. Xu, and S. Levine. Probabilistic Model-Agnostic Meta-Learning. In Advances Neural Information Processing Systems 31, 2018.
C. Finn, A. Rajeswaran, S. Kakade, and S. Levine. Online Meta-Learning. In Proceedings of the 36th International Conference on Machine Learning, 2019.
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. Empirical Investigation of Catastrophic Forgetting in Gradient-Baut Neural Networks. arXiv preprint, arXiv:1312.60 2013.
Gordon, Bronskill, M. Bauer, S. Nowozin, and R. Tumne. Meta-Leaming Probabilistic Inference for Prediction. In International Conference on Learning Representations, 2019. URL https:
//openreview ew. net. umm d=a.mmdddkkssxocC55
E. Grant, C. Finn, Levine, T. Darrell, and Griffiths. Recasting Gradient-Based Meta-Learning as Hierarchical Bayes. In International Conference on Learning Representations, 2018.
R. Grosse and J. Martens. A Kronecker-Factoree Approximate Fisher Matrix for Convolution Lay- ers. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
Harrison, A. Sharma, C. Finn, and M. Pavone. Continuous Meta-Lcarning without Tasks. arXiv 2019.
preprint, 19Xiv:1912.080.
X. He, J. Sygnowski, A. Galashov, A. A. Rusu, Y. Teh, and R. Pascanu. Task Agnostic Continual Learning via Meta Learning. arXiv preprint, arXiv: 190xv::1900.02001, 2019.
G. Jerfel, E. Grant, T. Griffiths, and K. A. Heller. Reconciling Meta-nn and Continual Learn- ing with Online Mixtures of Tasks, In Advances Neural Information Processing Systems 32, 2019.
D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
D. Kingma, Salimans, and M. Welling. Variational Dropout and the Local Reparameterizztioo Trick. In Advances in Neural Information Processing Systems 28, 2015.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, Barbinsk,, D. Hassabis, C. Clopath, Kumaran, and R. Hadsell, Over- coming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences, 2017.
G. Koch, R. Zemel, and R. Salakhutdinov. Siamese Neural Networks for One-Shot Image Recogni- tion. In 32th International Conference Machine Learning Deep Learning Workshop, 2015.
B. Lake, R. Salakhutdinov, J. Gross, and J.B. Tenenbaum. One Shot Learning of Simple Visual Concepts. In Proceedings of the 33rd Annual Conference of the Cognitive Science Society, 2011.
Lee, J. Kim, J. Jun, J. Ha, and B. Zhang. Overcoming Catastrophic Forgetting by Incremental Moment Matching. In Advances in Neural Information Processing Systems 30, 2017.
F. Li, R. Fergus, and P. Perona. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. In Proceedings the Computer Society Conference Computer Vision and Pattern Recognition Workshops, 2004.
the IEEE
10

---

Under review as a conference paper at ICLR 2021
F. Li, R. Fergus, and P. Perona. One-Shot Learning of Object Categories. IEEE Transactions Pattern Analysis and Machine Intelligence, 2006.
Z. Li, F. Zhou, F. Chen, and H. Li. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning. arXiv preprini, 1XXX:1107,009835 2017.
D. J. MacKay, A Practical Bayesian Framework for Backpropagation Networks. Neural Compu- tation, 1992.
S. Maji, E. Rahtu, J. Kannala. M. Blaschko, and A. Vedaldi. Fine-Grained Visual Classification of Aireraft. arXiv preprint, arXiv:1306.5151 2013.
Martens and R. Grosse. Optimizing Neural Networks with Kronecker-Pactored Approximate Curvature. In Proceedings of the 32nd International Conference on Machine Leurning, 2015.
E. G. Miller, N. E. Matsakis, and P. A. Viola. Learning from One Example Through Shared Den- sities on Transforms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2000.
C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Tumne. Variational Continual Learning. In International Conference on Learning Representations, 2018.
A. Nichol, J. Achiam, and J. Schulman. On First-Order Meta-Learning Algorithms, arXiv preprint, arXiv:1803.000 02099,, 2018.
M. Nilsback and Zisserman. Automated Flower Classification over a Large Number of Classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
M. Opper. A Bayesian Approach to Online Learning. Cambridge University Press, 1998.
S. Ravi and H. Larochelle. Optimization a Model for Few-Shot Learning. In International Conference on Learning Representations, 2017.
H. Ritter, A. Botev, and D. Barber. Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting. In Advances in Neural Information Processing Systems 31, 2018a.
H. Ritter, A. Botev, and D. Barber. A Scalable Laplace Approximation for Neural Networks. In International Conference on Learning Representations, 2018b.
H. Robbins and S. Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 1951.
A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta- Learning with Latent Embedding Optimization. In International Conference on Learning Repre- sentations, 2019.
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and Lillicrap. Mcta-Learning with Memory- Augmented Neural Networks. In Proceedings of the 33rd International Conference Machine Learning, 2016.
J. Schmidhuber. Evolutionary Principles in Self-Referential Learning, On Learning How to Learn: The Petta-MetaMMeta.--HHoo Diploma thesis, Institut für Informatik, Technische Universität München, 1987.
J. Snell, K. Swersky, and R. Zemel. Prototypical Networks for Few-Shot Learning. In Advances in Neural Information Processing Systems 30, 2017.
Thrun and L. Pratt. Learning to Learn: Introduction and Overview. Springer, Boston, MA, 1998.
B. Trippe and R. E. Turner. Overpruning in Variational Bayesian Neural Networks. In Advances in Neural Information Processing Systems 30- Advances in Approximate Bayesian Inference Workshop, 2017.
O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching Networks for One Shot Learning. In Advances in Neural Information Processing Systems 29, 2016.

---

Under review as a conference paper at ICLR 2021
J. Wen, Y. Cao, and R. Huang. Few-Shot Self Reminder to Overcome Catastrophic Forgetting, arXiv preprint, 1.XXv:1812200544: 2018.
J. Yoon, T. Kim, O. Dia, S. Kim, Y. Bengio, and S. Ahn. Bayesian Model-Agnostic Meta-Learning. In Advances Neural Information Processing Systems 31, 2018.
Zenke, Poole, and Ganguli, Continual Learning through Synaptic Intelligence. Proceed- ings of the 34th International Conference on Machine Learning, 2017.
Z. Zhuang, Y. Wang, K. Yu, and S. Lu. No-Regret Non-Convex Online Meta-Leaming. arXiv preprint, 19X0v:1110.11110 2019.
12

---

Under review as a conference paper at ICLR 2021
A BACKGROUND
This section provides a background explanation of using BOL to find the posterior of model pa- rameters and overcome catastrophic forgetting, commonly for large-scale supervised classification. We will then apply this approach to our recursion in Eq. (5).
The posterior is typically intractable due to the enormous size of the modern neural network ar- chitectures. This leads to the requirement for a good upproximation parameters. A particularly suitable candidate for this purpose in meta-learning is the Laplace ap- proximation (MacKay 1992] Ritter et al.] 2018b). as it simply adds a quadratic regulariser to the training objective. Variational inference is another possible method to obtain approximation for the posterior of the meta-parametert
the posterior of the meta-
A.1 BAYESIAN ONLINE LEARNING
Upon the arrival of the new D we interested in MAP estimate 0 arg max p( (1:++1)) for the parameters of a neural network. Using Bayes' rule on the posterior gives the recursive formula
(16) where Eq. 16 follows from the assumption that each dataset is independent given 0. As the nor- malised posterior ((x) is usually intractable, it may be approximated by parametric distribu- tion q with parameter d/. The BOL framework consists of the update step and the projection step
p(0 & p(Pc+110)p(01P))
(Opper 1998). The update step uses the approximate posterior ( (e)) obtained from the previous step for update in the form of Eq. (I6):
61) x p(D++1 (
(17)
The new posterior p(0 1 )t) might not belong to the same parametric family ((0)))). In
this case, the new posterior has to be projected into the same parametric family to obtain {(f (0++)). Opper (1998) performs this projection by minimising the KL-divergence between the posterior and the parametric q, while Ritter et al. (2018)) use the Laplace approximation and Nguyen et al. (018) use variational inference.
A.2 LAPLACE APPROXIMATION
We consider finding a MAP estimate following from Eq. (16) 07+1
arg max p(0) D1a+1) arg max + log p(0) Di:)).
(18)
Since the posterior D(0) of a neural network is intractable except for small architectures, the un- normalised posterior 7 (()) is considered instead. Performing Taylor expansion on the logarithm of the unnormalised posterior around mode 0F gives
log log log 1o0 o-00 -0-
(19)
where A, denotes the Hessian matrix of the negative log-posterior evaluated at 07. The expansion in Eq. (19) suggests using Gaussian approximate posterior. Given the parameter di = {Mr Ar), a mean M4++1 for step + be obtained by finding mode of the approximate posterior follows via standard gradient-based optimisation:
14+1 = arg max log (Drr(DD+1))) (0 1r - mr).
(20)
0
The precision matrix is updated as Ac+1 = H1+1 + A,, where Ho+1 is the Hessian matrix of the negative log-likelihood for D evaluated at Pe+1 with entries
Hy = 000 log logg(D.-11))
(21)
8=pe11
For neural network model, gradient-based optimisation methods such as SGD (Robbins & Monro 1951) and Adam (Kingma & Ba 2015) are the standard gradient-base methods in finding a mode for the Laplace approximation in Eq. (0)). We show in Section 4.1 that this provides well-suited
skeleton to implement Bayesian online meta-learning in Eq. (5) with the mode-secking optimisation procedure.
13

---

Under review as a conference paper at ICLR 2021
A.3 BLOCK- -OCG-DIAA HESSIAN APPROXIMATION
Since the full Hessian matrix in Eq. 21 is intractable for large neural networks, we seek for an ef- ficient and relatively close approximation to the Hessian matrix. Diagonal approximations (Denker) & LeCCnn 1991; Kirkpatrick et al. 2017) are memory and computationally efficient, but sacrifice approximation accuracy as they ignore the interaction between parameters. Consider instead sepa- rating the Hessian matrix into blocks where different blocks are associated to different layers of neural network. A particular diagonal block corresponds to the Hessian for a particular layer of the neural network. The block-diagonal
connecker-factoree approximation (Martens & Grosse 2015 Grosse & Martens] [2016] [ootv et al., (017)) utilises the fact that each diagonal block of the Hessian Kronecker- for single data point. This provides better Hessian approximation as takes the parameter interactions within a layer into consideration.
A.3.1 SONNECKER-FACTOREE APPROXIMATION
Consider neural network with L layers and parameter 0 [vec(Wi)) ...., Le where W; is the weight of layer & for é = {1....,A, } and vec denotes stacking the columns of a matrix into vector. We denote the input of the neural network as do and the output of the neural network as h2, As the input passes through each layer of the neural network, we have the pre- are-activation for layer as
W 1 and the activation as a₂ fe(he) where fe the activation function of layer l. If a bias vector is applicable in calculating the pre-activation of layer, we append the bias vector the last column the weight matrix and append scalar one to the last element of the activation. The gradient 9r of loss LL(x, y) log ply x 0) with respect to he for input-target pair (ær, y) the pre-activation gradient for layer E.
Martens & Grosse (2015) show that the ê-th diagonal block F3 of the Fisher information matrix F can be approximated by the Kronecker product between the expectation of the outer product of the (E - 1)-th layer activation and the č-th layer pre-activation gradient:
Fe = Ez,y (ar 1 ? gege)
(22)
= (at- 10 s0 Begllgeg
(23)
= Ac-1 8 Gé.
(24)
where Ar-1 = Ex(ar-1a7]] and Ge = En [. Grosse & Martens (016) extend the block-
diagonal Kronecker-factored Fisher approximation for fully-connected layers to that for convolution
layers. The Gaussian log-probability term can be calculated efficiently without expanding the Kro- necker product using the identity
(Ae-1 @ Ge) veo - W2) = vec(G. W2)A
(25)
As we mentioned in Section assumption results in having terms that multiply two or more Kronecker products together. The 2-th diagonal block of F in Eq. (11 is
approximating the Hessian with the one-step SGD inner loop
F M
-0-
GT (A @ GT)
AT & GT
(26)
where A 3 G the Kronecker product corresponding to the Hessian in Eq.
13 for task or
batch m. We expand Fr using the Kronecker product property:
(A 2 GTT) (A & GTD) A Ar @ GT
(27)
This gives
M
F
m
& 5m
A |
m 8 G
G. mm
A
8 G (GT))
M
(28)
m==
+ Am
mm
(Am) 1" 2 G G (GTYT)
14

---

Under review as a conference paper at ICLR 2021
Finally, moving the meta-batch (or batch) averaging into the Kronecker factors gives the approxi- mation:
FA @ G. A A G,G, - A -A & (.,))
(29)
A (A+-1)* &.G.
where A 1 Am G = 1 M Em Õp, A AC-1 M Em A Am and so on.
A.3.2 POSTERIOR REGULARISING HYPERPARAMETER FOR PRECISION UPDATE
Ritter et al. (2018a) use hyperparameter À as multiplier to the Hessian when updating the preci- sion:
A++1 AH4+1 As.
(30)
In the large-scale supervised classification setting, this hyperparameter has a regularising effect on the Gaussian posterior approximation for balance between having good performance dataset and maintaining the performance on previous datasets (Ritter et al. 2018a). A large À results datasets well, but can
new
in sharply peaked Gaussian posterior and is therefore unable learn prevent forgetting previously learned datasets. A small À on the other hand gives a dispersed Gaus- sian posterior and allows better performance on new datasets by sacrificing the performance on the previous datasets.
A.4 VARIATIONAL CONTINUAL LEARNING
The variational continual learning method (Nguyen al. 2018) also provides suitable meta- training framework for Bayesian online meta-learning in Eq. ()). Consider approximating the pos- terior q by minimising the KL--ivergence between the parametric g and the posterior in the projection step in Eq. (17 where g belongs to some pre-determined approximate posterior family Q with parameters de:
((( arg min DEL (ell qE
(31)
arg min { Bg(n)) (e0
(32)
9e
The optimisation in Eq. 32 leads the objective
0.41 = arg min { (log + ++)))))))))
(33)
One Gaussian mean- mean-field approximate posterior (f( Th ( o..)) where Φ = (Htr.d; d= and D dim(f). The first term Eq. 33) can be estimated via Monte Carlo with local reparameterisation trick (Kingma et al. 2015), and the second KL-divergence term has closed form for Gaussian distributions.
B ALGORITHMS
B. BOMLA AND BOMVI
Algorithm gives the pseudo-code of the BOMLA algorithm for the sequential datasets setting, with the corresponding variation for the sequential tasks setting in brackets. The algorithm formed of three main elements: meta-training mean (line [2) and updating the Gaussian precision (line 13 16.. For the precision update, we approximate the Hessian using block-diagonal
specific dataset task (line 4 TT, updating the Gaussian
connecker-factoree approximation (BD-KFA).
A 2 the pseudo-code the BOMVI algorithm for the sequential datasets setting, with the corresponding variation for the sequential tasks setting in brackets. The algorithm is formed of two main elements: meta-training on a specific dataset or task (line [T] and updating the parameters of the Gaussian mean-field approximate posterior (line 12).
15

---

Under review as a conference paper at ICLR 2021
Algorithm Bayesian online meta-learning with Laplace approximation (BOMLA)
1: Require: sequential datasets (or tasks) Di,.... Dr, learning a, posterior regulariser X,
number of meta-training iterations (or epochs) J, meta-batch size (or number of batches) M
2: Initialise: μ4. Ao. 0
3: for t = 1 to T do
4:
for 1,....... J do
meta-training on dataset or task Dr New: added
5:
for m to M do Sample task (or split the batch) D Dimr. U D.D. Inner update fTT = SGDx (C(0, D, 5.m..)) end for
6:
7:
8:
9:
Evaluate loss fBOMLA (0, 11 A, in Eq. Outer update 0 + Voos ABOOMA A(-)
11:
end for
Update mean 14 t
update posterior mean
13:
For sequential datasets, sample M tasks for Hessian approximation
Run inner update in line I each task (or for each batch)
15:
Approximate H, with BD-KFA to F in Eq. (1
16:
Update precision A, + AH, Ar-1
update posterior precision
end for
Algorithm Bayesian online meta-learning with variational inference (BOMVI)
1: Require: sequential datasets (or tasks) DI,..., Dr. learning rate a, number of meta-training iterations (or epochs) J, meta-batch size (or number of batches)
2: Initialise: Φ0 {μo, J
3: for t- to T do
4: for i = 1,2.
do
meta-training dataset task D, @@@ New: added J
5:
for m to M do Sample task (or split the batch) Dr =D. U D.m. Inner update 0Th SGD (C(0, end for
6:
(C, D.m..))
7:
8:
9:
Evaluate loss f. fBOMVT (Ø, Φ in Eq.
15
10: Outer update μ μ NuJt (0, @1-1), and σ a a BRMMVT (0, Φ1-1)
-BOMVI
11: end for
12: Update fe 1 and
update posterior parameters
13:
end for
B.2 BOMVI MONTE CARLO ESTIMATOR
Recall that the VI objective is:
M Σ
M
BBOMVI (g. = JA
Eycerss [logp(D (0m)]
Σ Ey log logg(("""" (1))))))
D
where 0m SGDx(L(O, D: for 1....,M M. The Monte Carlo estimator for the first term of the BOMVI objective difficult to compute, as every sampled meta-parameters 0, for 1,..., R. has undergo few-shot quick adaptation prior to the log-likelihood evaluation. As consequence the estimator is prone to large variance. Moreover, every quickly-adapue sample 8, contributes to the meta-learning gradients of the posterior mean and covariance, resulting in a high computational cost when taking the vreaa-gradeents
To solve these impediments, we introduce slight modification to the SGD quick adaptation Thh In- stead of taking the gradients with respect the sampled meta- petaa-prameters, we consider the gradients with respect to the posterior mean. A one-step SGD quick adaptation, for instance, becomes:
nm 0 aV uu L(u4, D
(34)
16

---

Under review as a conference paper at ICLR 2021
This gives 8t diag((?)) where
jAe = f4 - aV ja C(p,, Dr
(35)
since 0 ~ NJu, di) A quick adaptation with more steps works in similar fashion. With this modification, we can calculate the Monte Carlo estimator for the first term using the local reparameterisation trick as usual.
C EXPERIMENTS
C.I OMNIGLOT: SEQUENTIAL TASKS
In this experiment, use the model architecture proposed by Vinyals et al. (2016) that takes modules with 64 filters of size 3 x 3, followed by batch normalisation, ReLU activation and 2 max-moooioee moollin. fully-connected layer is appended to the final module before getting the class probabilities with softmax. Table shows the hyperparameters used in this experiment.
The Omniglot dataset comprises 50 alphabets (super-classes). Each alphabet has numerous charac- ters (classes) and each character has 20 instances. As the meta-training alphabets arrive sequentially, we form non-overlapping sequential tasks from each arriving alphabet, and the tasks also do overlap in the characters. We use 35 alphabets for meta-training, 7 alphabets for validation and 8 alphabets for meta-evaluation. The alphabet splits are as follows:
35 alphabets for meta-training:
Kannada, Burmese_ (Myanmar),
Grantha, Balinese,
Atlantean, Ojibwe_ Ojibwe Japanese__Kaatkkana), Hebrew, Asomtavruli_
Keble,
'old "old__Churcc__slaoniie_
Tengwar, Aurek-Besh, Sanskrit, Manipuri, Farly_Aramaic, Oriya,
Mongolian, Avesta, Malayalam, Tifinagh, Angelic, Latin, Braille, (Canadian_Aboriggna____ Armenian, Korean, Gurmukhi, ULOG, Bengali, Gujarati, Sylheti
Alghabet_of_th___ee
7 alphabets for validation:
Ge_ez, Cyrillic, Clagolitic, N_Ko, Arcadian, Anglo-Saxon Blackfoot_ (
8 alphabets for meta-evaluation:
Eeriac_(Serrtan Atemayar_ _earysyr_eelaaer,,,eeeaa_a Tibetan, Futurama, Mkhedruli_ (Georgian), Syriac (Estrangelo), Tagalog, Greek
Table 1: Hyperparameters for the Omniglot sequential tasks experiment
Hyperparameter
BoMLA
BOMVI
Posterior regulariser À
0.1
Precision initialisation values Covariance initialisation values Number Monte Carlo samples
10
1
10
exp(
10)
Number of batch M
Number of query samples per class (meta-evaluation)
15
15
Number of epochs per task
50
50
Number of inner SGD steps in meta-training (k)
5
5
Inner SGD learning rate (a)
0.1
0.1
Outer loop optimiser
Adam
Adam
Outer loop learning rate Number of tasks sampled for meta-evaluation
0.001
0.001
100
100
Number inner SGD steps in meta-evaluation (k)
10
10
17

---

Under review as a conference paper at ICLR 2021
C.2 PENTATHLON: SEQUENTIAL DATASETS
We use the model architecture proposed by Vinyals et al. (2016) in this experiment, as we did for the sequential tasks experiment. Tables and are the hyperparameters used in this experiment.
The Omniglot dataset (Lake et al. 201 comprises 1623 characters from 50 alphabets and each character has 20 instances. New classes with rotations in the multiples of 90 are formed after splitting the classes for meta-training, validation and meta-evaluation. We use 1 1100 characters meta-training, 100 characters for validation and the remaining for meta-evaluation.
Omniglot:
The CIFAR-FS dataset (Bertinetto al. 2019) has 100 classes of objects and each class comprises 600 images. We use the same split as Bertinetto et al. (2019): 64 classes for meta- training, 16 classes for validation and 20 classes for meta-evaluation.
CIFAR-FS:
The minilmageNet dataset (Vinyals et al., 2016) takes 100 classes and 600 in- stances each class from the ImageNet dataset. We use the same split Ravi Larochelle (2017): 64 classes for meta-training, 16 classes for validation and 20 classes for meta-evaluation.
minilmageNNet:
VGG-Flowers:
The VGG-Flowers dataset Nilssack & Zisserman 2008 comprises 102 different types of flowers as the classes. This dataset has 8,189 instances in total. We randomly split 66 classes for meta-training, 16 classes for validation and 20 classes for meta-evaluation.
Aircraft:
The Aireraft dataset (Maji al. 2013) a fine-grained dataset consisting of 100 differ- ent aircraft models as the classes and each class has 100 instances. We randomly split 64 classes for meta-training, 16 classes for validation and 20 classes for meta-evaluation.
Table 2: Hyperparameters for the pentathlon experiment (same value for all datasets)
Hyperparameter
BOMLA
BOMVI
Posterior regulariser À
(various values)
Precision initialisation values
10-4
10-2
Number of tasks sampled for Hessian approx.
5000
Covariance initialisation values
exp(-3)
Number of Monte Carlo samples Meta-batch size M
20
32
32
Number of query samples per class
15
15
Number of iterations per dataset Outer loop optimiser
5000
5000
Adam
Adam
Outer loop learning rate
0.001
0.001
Number of tasks sampled for meta-evaluation
100
100
Table 3: Hyperparameters for the pentathlon sequential datasets experiment (individual datasets)
Hyperparameter
Omniglot
CIFAR-FS
minilmageNet
VGG-Flowers
Aircraft
Number of inner SGD 1
5
steps in meta-training (k)
Inner SGD learning rate 0.4
0.1
0.1
0.1
0.1
(a)
Outer learning rate decay
x0.1
x0.1 halfway
x0.1 per 1000
x0.1
schedule
halfway 10
iterations
halfway
Number of inner SGD
3
10
steps in meta-evaluation
18

---

Under review as a conference paper at ICLR 2021
Aircraft meta-train
VGG-Flowers meta-train
minilmagenet meta-train traainta time
BOML BOMLA, = 100 BOMV
Baseline Sequential MAML TOE (BOMLA)
TOE (BOMVI)
CIFAR-FS meta-train
Meta-evaluation accuracy (%)
Omniglat meta-train
Soowees 8
100
Ommillee
 5
B aiiiieeeent
Arrce Seratt
Airce
Figure 3: Meta-evaluation accuracy across 3 seed runs on each dataset along meta-training. Higher accuracy values indicate better results with less forgetting with A = 100 gives better performance in the off-diagonal plots (retains performances on previously learned datasets), and has minor performance trade-off in the diagonal plots (leams less well new datasets). Sequential MAML gives better performance the diagonal plots (learns well on new datasets) but worse performance in the off-diagonal plots (forgets previously learned datasets). BOMVI is also able to retain performance on previous datasets, although it may be unable to perform good BOMLA due sampling and estimator variance.
proceed new datasets. BOMLA
19

---

Under review as a conference paper at ICLR 2021
Aircraft meta-train
VGG-Flowers meta-train
minilmagenet meta-train eata-ttraniig time
BOMLA, Á = 1000
Sequential MAML
BOMLA, A 100
BOMLA. ==1
Baseline
BOML
BOMV
.....
CIFAR-FS meta-train
Meta-evaluation accuracy (%)
Omniglat meta-train
100
CEARFFS CPRA G
Miiiiimmee mmniieegeet cereenee
VGG VGGGHfoWaI we
Arccaatt
omnigllot
Figure 4: Meta-evaluation accuracy across 3 seed runs on each dataset along meta-training. Higher accuracy values indicate better results with less forgetting with a large À = 1000 gives better performance in the off-diagonal plots (retains performances on previously learned datasets) but worse performance in the diagonal plots (does not learn well datasets). A small À = 1 gives better performance in the diagonal plots (lears well on new datasets) but worse performance the off-diagonal plots (forgets previously learned datasets). BoM VI is also able to retain performance on previous datasets, although it may be unable to learn new datasets as good as BOMLA due to sampling and estimator variance.
proceed new datasets. BOMLA
20


Reviews:
[{"review_id": "k12Rk1IC6qe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a Bayesian approach to meta-learning from sequential data. Two algorithms are proposed. The first is based on the Laplace approximation to the model posterior which is made tractable by using K-FAC approximation of the Hessian. The second approaches uses a variational approximation for the posterior, where meta-learning corresponds to learning the variational prior. The experiments present results of the proposed method in sequential Omniglot and a pentathlon task involving different datasets.\n\nThe major drawback of the paper is a lack of description of the precise problem setting. Furthermore, certain motivations and comments appear to be unsubstantiated, and it would be great for the authors to provide rationale for them. Overall, in the current form, I find it hard to recommend the paper for acceptance. However, I am willing to reconsider based on author response to the questions.\n\n**What is the setting of the paper?**\nThe paper over-emphasizes the sequential nature of the setting, but does not provide clear problem setup beyond this. There are multiple well-defined problem setups that all fall under the purview of sequential data. Some examples are:\n\n1. *Regret minimization* : Here, no (distributional) assumptions about the tasks/datasets are made whatsoever. The goal is to compete with the best (meta) learner in hindsight. Cesa-Bianchi and Lugosi textbook would be a classic reference and Finn et al. 2019 (FTML) is a modern take in the context of meta-learning.\n\n2. *i.i.d. datasets* : Here, tasks come from the same underlying distribution, just sequentially one after the other. In contrast to above, this places a stronger assumption about the world, but as a result may be able to develop more specialized algorithms. Note that regret minimization algorithms can be used for i.i.d. datasets (typically called online to batch conversion), while the reverse is generally not true.\n\nIt is unclear which setting the paper studies. My understanding is that setting 2 is more relevant to the paper, but the authors should clarify and explain the setting of the paper explicitly. I also suggest the authors make Section 2 to be related works and explain settings that are already known, in order to better position the contribution relative to literature. This also ensures better dissemination of scientific information.\n\n**Why should we learn the MAP?**\n\nThe paper in Section 2 and 3 sets up an objective that aims to find the MAP estimate, i.e. $\\theta^* = \\arg \\max_\\theta P(\\theta | \\tilde{D}_{1:t+1})$. However it is not clear why this should be the objective. In a streaming setting, doing well on the past need not imply doing well on future streams of data unless some assumptions are made about the data distribution.\n\n**Why cant we store the data in a \"replay buffer\"?**\n\nThe paper in multiple places states that storing historical data increases the algorithmic complexity (e.g. for FTML). Is this really a bottleneck? Can you present data or experiments to back this up -- my guess is that for the tasks/experiments considered in this work, storing historical data in memory is not that expensive. Furthermore, disk/memory costs are cheap and it is routine in ML these days to work with very large datasets. For example, deep RL stores all historical data in the form of a replay buffer for learning. Thus, it is unclear to me why FTML has not been considered for comparison?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dh8pp-rWFgf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a Bayesian meta-learning framework for sequential data. Two approximate Bayesian inference techniques, Laplace approximation and variational inference, are proposed for estimating parameters. The proposed approach helps to improve catastrophic forgetting in online settings. \n\nThe method is sound though the majority of the paper is based on other algorithms (Opper et al, Fin et al., Ngyuen et al., etc.). However, I can consider this work as the right combination of various work to solve an interesting problem. \n\nThis is framed as a typical Bayesian model selection problem. Although meta-learning is nothing more than model selection, this is apparent in section 2.2. I encourage explaining \"why\" performing Bayesian inference helps to mitigate the catastrophic forgetting problem well ahead in the paper. \n\nPerforming Bayesian inference over parameters, implicitly keep a memory of the history. How does the proposed approach would be different from explicitly keeping some representative samples or sufficient statistics in a memory buffer [1]? Alternatively, we can maintain an ensemble. It would perhaps be better than Laplace or mean-field approximations for this kind of complex distributions. What is the relationship of the proposed method to [2]?  Is it possible to find some meaningful online datasets such as those used in robotics or signal processing? [2] seems to have such experiments. How reasonable is the block-diagonal assumption? \n\n[1] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap,“Meta-learning with memory-augmented neural networks, ICML 2016.\n[2] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, Sergey Levine, Online Meta-Learning, ICML 2019. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "D1JVbT25D5e", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an Bayesian approach to online meta-learning. This is done by lifting (approximate) sequential Bayesian inference from the model parameters to the meta-parameters. Two approaches are proposed to do this: (i) Laplace Approximation (LA), thereby extending Ritter et al’s method from online learning to online meta-learning; and (ii) VI, thereby extending Nguyen et al’s Variational Continual Learning (VCL) method in a similar way. Experiments are performed by converting existing meta-learning benchmarks into online settings in two ways, which they call “sequential tasks” and “sequential datasets” respectively. The experimental results show resistance to catastrophic forgetting in both of these experimental settings. \n\nIn general, the novelty is somewhat limited, as this extension of LA/VCL to meta learning is similar in principle to the Grant et al formulation as a hierarchical Bayesian model. The main difference  is that the inner loop is done using the standard fast adaptation method from MAML rather than using LA/VI.\n\nIf you’re only using a point estimate (“We are interested in a MAP estimate $\\theta^*$”) then this is not truly Bayesian online learning. I assume that the choice of doing it this way is to sidestep the difficulty of integrating over the parameters to form the posterior predictive, but loses the advantages of having a Bayesian posterior. \n\nMissing references:\n- ML-PIP/VERSA (Gordon et al 2019, Meta Learning Probabilistic Inference for Prediction, ICLR). This does Bayesian meta learning by directly optimising the posterior predictive, rather than the usual posterior in VI. It also makes use of amortisation networks so that at test time only forward passes through NNs are required. They don’t provide an online formulation. However, The posterior predictives can be used as prior predictives for future rounds of inference.\n- BMAML (Kim et al 2018, Bayesian model-agnostic meta-learning, NeurIPS). This uses Stein Variational Gradient Descent (SVGD) to obtain the task posterior for a novel task, casting MAML as a hierarchical probabilistic model. \n- Jerfel et al 2019, Reconciling meta-learning and continual learning with online mixtures of tasks, NeurIPS. This paper also uses a hierarchical Bayesian formulation, but takes a non parametric mixture approach using Dirichlet processes. They handle distribution shift in the online meta-learning setting by modelling latent task structure. \n\nBaselines:\n- The baselines in this paper are a bit weak. The two methods ML-PIP and BMAML mentioned above, along with the Grant et al hierarchical Bayes method, could all be applied here, but since they weren’t originally formulated for the setting it would require extra work to do so. However, several methods were mentioned in the related work setting which should have been included: the He et al Task agnostic continual learning and Harrison et al continuous meta learning methods.\n\nEdit post reviewer responses:\n- I had some misunderstandings in the review above, which have mostly been cleared up. I’ve raised my score accordingly", "extra_scores": {}, "extra_text": {}}, {"review_id": "3WMQdw373gJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper develops a semi-online Bayesian approach to meta-learning, where tasks arrive sequentially and learning within any task is performed in batch mode (hence my terminology semi-online). It suggests a sequential between-task Bayesian update, eq. 5, and proposed three approximations to aid computation. The basic setup is motivated within the recently introduced MAML framework where the learning takes place by adapting a within-task parameter to effectively set up learning within each individual task, allowing the learner to transfer information between tasks, while remaining adaptive to a specific novel task. The authors phrase this idea in the Bayesian language of posterior distributions, that are updated both within and between tasks. The posterior formed after learning t tasks, serves as a prior for learning a new task. The authors suggest 3 approximation schemes, a Laplace approximation, a Hessian approximation, and a variational approximation. Finally, a set of experiments are presented comparing performance to 2 baselines, namely TOE (train of everything) and TFS (train from scratch). A particularly interesting application is to 5 standard sets of images, testing for catastrophic forgetting of previous tasks and the transferability of  information across tasks in the face of distributional shift.\n\nStrong points: The framework is of great practical and conceptual importance. The relation to the MAML framework is clear and its incorporation within a Bayesian framework holds significant promise. The simulation results on the 5 sequential tasks provide good evidence for the effectiveness of the method.\n\nWeak points:\nThe precise contribution of the work is a little difficult to discern, as it revolves very closely around previous work (MAML), Bayesian learning and Laplace approximation (e.g., Ritter et al 2018a), Variational online learning (e.g., Nguyen et al., 2018) and Hessian approximation schemes (e.g., Grosse and Martens). The approach mostly combines these methods into a single framework. It would be good to be very clear here about the novelty of the contribution.\nThe authors state that MAML requires all base classes to be available for sampling at each iteration. However, this does not seem to be the case for the online extension in Finn et al., 2019.\nThe empirical results to not compare to some of the SOTA methods in the increasingly populated domain of continual meta-learning (e.g., Finn et al., 2019), and indeed to previous methods to which the present paper is an extension (e.g., Ritter et al. 2018a).\nError bars are missing from the figures, or should, at least be mentioned.\n\nSpecific issues:\nThe authors use the terminology Laplace approximation for the Taylor expansion of the posterior around its maximum. As far as I am aware, the term Laplace approximation is used when this is done within an intractable Bayesian integral, in order to yield  a tractable integration. What they term a Laplace approximation is simply a Taylor expansion.\nIt is surprising that the BOMVI is often inferior to the simpler BOMLA (with an appropriate parameter \\lambda). It would be good to discuss this issue in more detail. Can the authors suggest an approach to adapting \\lambda to the data?\nThe title online learning is a little misleading, as the learning within tasks is done in batch. This should be compared to recently proposed fully online methods (e.g., Finn et al 2019, Khodak et al, Adaptive Gradient-Based Meta-Learning Methods, NeurIPS 2019)\nThe loop over i on line 4 of Algorithm 1 of the appendix has no termination point. Please clarify.\n\nConcerns: The key concerns are incrementality WRT previous work in the field (and lack of precise elucidation of the novel aspects of the paper), lack of theory, and a lack of comparison of the empirical results to SOTA methods. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an online meta-learning algorithm. 3 out of 4 reviews were borderline. The main concern during the discussion was that it is unclear what kind of online learning this paper does. For instance, in theory, the online learner competes with the best solution in hindsight. This is a regret-minimizing point of view. The other online learning is streaming. In this case, there is no regret. The goal is a sublinear representation that is competitive with some baseline that uses all space.

After the discussion, I read the paper to understand the points raised by the reviewers. I agree that this paper is not ready to be accepted. My quick review is below:

The authors combine MAML and BOL to have online updates (not all tasks are required beforehand) and handle distribution shift. But the way of combining these is not well justified. In particular,

1) The distribution-shift story is not convincing. The reason is that the proposed algorithm is posterior-based. By definition, when you use posteriors as in (3)-(5), you assume that the datasets are sampled i.i.d. given \theta. This means independently and identically. So no distribution shift. I am familiar with Kalman filtering. For that, you need p(\theta_t | \theta_{t - 1}) in (3)-(5), which would be sufficient for tracking stochastic distribution shifts.

2) I find the use of BOL unnatural. Since MAML is gradient-based, it would be more natural to have a gradient-based online learner. Gradient descent has online guarantees and does not require i.i.d. assumptions.

3) The authors should clearly state what the objective of their online algorithm is. In particular, the informal justification of (3)-(5) as doing something similar to MAML (the paragraph around (6)) is highly confusing. I could not understand what the authors mean.
Decision: Reject