Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a Bayesian meta-learning framework for sequential data. Two approximate Bayesian inference techniques, Laplace approximation and variational inference, are proposed for estimating parameters. The proposed approach helps to improve catastrophic forgetting in online settings. 

The method is sound though the majority of the paper is based on other algorithms (Opper et al, Fin et al., Ngyuen et al., etc.). However, I can consider this work as the right combination of various work to solve an interesting problem. 

This is framed as a typical Bayesian model selection problem. Although meta-learning is nothing more than model selection, this is apparent in section 2.2. I encourage explaining "why" performing Bayesian inference helps to mitigate the catastrophic forgetting problem well ahead in the paper. 

Performing Bayesian inference over parameters, implicitly keep a memory of the history. How does the proposed approach would be different from explicitly keeping some representative samples or sufficient statistics in a memory buffer [1]? Alternatively, we can maintain an ensemble. It would perhaps be better than Laplace or mean-field approximations for this kind of complex distributions. What is the relationship of the proposed method to [2]?  Is it possible to find some meaningful online datasets such as those used in robotics or signal processing? [2] seems to have such experiments. How reasonable is the block-diagonal assumption? 

[1] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap,“Meta-learning with memory-augmented neural networks, ICML 2016.
[2] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, Sergey Levine, Online Meta-Learning, ICML 2019. 
review_title: Interesting problem, sound method

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper develops a semi-online Bayesian approach to meta-learning, where tasks arrive sequentially and learning within any task is performed in batch mode (hence my terminology semi-online). It suggests a sequential between-task Bayesian update, eq. 5, and proposed three approximations to aid computation. The basic setup is motivated within the recently introduced MAML framework where the learning takes place by adapting a within-task parameter to effectively set up learning within each individual task, allowing the learner to transfer information between tasks, while remaining adaptive to a specific novel task. The authors phrase this idea in the Bayesian language of posterior distributions, that are updated both within and between tasks. The posterior formed after learning t tasks, serves as a prior for learning a new task. The authors suggest 3 approximation schemes, a Laplace approximation, a Hessian approximation, and a variational approximation. Finally, a set of experiments are presented comparing performance to 2 baselines, namely TOE (train of everything) and TFS (train from scratch). A particularly interesting application is to 5 standard sets of images, testing for catastrophic forgetting of previous tasks and the transferability of  information across tasks in the face of distributional shift.

Strong points: The framework is of great practical and conceptual importance. The relation to the MAML framework is clear and its incorporation within a Bayesian framework holds significant promise. The simulation results on the 5 sequential tasks provide good evidence for the effectiveness of the method.

Weak points:
The precise contribution of the work is a little difficult to discern, as it revolves very closely around previous work (MAML), Bayesian learning and Laplace approximation (e.g., Ritter et al 2018a), Variational online learning (e.g., Nguyen et al., 2018) and Hessian approximation schemes (e.g., Grosse and Martens). The approach mostly combines these methods into a single framework. It would be good to be very clear here about the novelty of the contribution.
The authors state that MAML requires all base classes to be available for sampling at each iteration. However, this does not seem to be the case for the online extension in Finn et al., 2019.
The empirical results to not compare to some of the SOTA methods in the increasingly populated domain of continual meta-learning (e.g., Finn et al., 2019), and indeed to previous methods to which the present paper is an extension (e.g., Ritter et al. 2018a).
Error bars are missing from the figures, or should, at least be mentioned.

Specific issues:
The authors use the terminology Laplace approximation for the Taylor expansion of the posterior around its maximum. As far as I am aware, the term Laplace approximation is used when this is done within an intractable Bayesian integral, in order to yield  a tractable integration. What they term a Laplace approximation is simply a Taylor expansion.
It is surprising that the BOMVI is often inferior to the simpler BOMLA (with an appropriate parameter \lambda). It would be good to discuss this issue in more detail. Can the authors suggest an approach to adapting \lambda to the data?
The title online learning is a little misleading, as the learning within tasks is done in batch. This should be compared to recently proposed fully online methods (e.g., Finn et al 2019, Khodak et al, Adaptive Gradient-Based Meta-Learning Methods, NeurIPS 2019)
The loop over i on line 4 of Algorithm 1 of the appendix has no termination point. Please clarify.

Concerns: The key concerns are incrementality WRT previous work in the field (and lack of precise elucidation of the novel aspects of the paper), lack of theory, and a lack of comparison of the empirical results to SOTA methods. 

review_title: The paper develops a semi-online Bayesian approach to meta-learning. The framework is interesting and tackles an important problem. My key concern is incermentality WRT previous work. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes a Bayesian approach to meta-learning from sequential data. Two algorithms are proposed. The first is based on the Laplace approximation to the model posterior which is made tractable by using K-FAC approximation of the Hessian. The second approaches uses a variational approximation for the posterior, where meta-learning corresponds to learning the variational prior. The experiments present results of the proposed method in sequential Omniglot and a pentathlon task involving different datasets.

The major drawback of the paper is a lack of description of the precise problem setting. Furthermore, certain motivations and comments appear to be unsubstantiated, and it would be great for the authors to provide rationale for them. Overall, in the current form, I find it hard to recommend the paper for acceptance. However, I am willing to reconsider based on author response to the questions.

**What is the setting of the paper?**
The paper over-emphasizes the sequential nature of the setting, but does not provide clear problem setup beyond this. There are multiple well-defined problem setups that all fall under the purview of sequential data. Some examples are:

1. *Regret minimization* : Here, no (distributional) assumptions about the tasks/datasets are made whatsoever. The goal is to compete with the best (meta) learner in hindsight. Cesa-Bianchi and Lugosi textbook would be a classic reference and Finn et al. 2019 (FTML) is a modern take in the context of meta-learning.

2. *i.i.d. datasets* : Here, tasks come from the same underlying distribution, just sequentially one after the other. In contrast to above, this places a stronger assumption about the world, but as a result may be able to develop more specialized algorithms. Note that regret minimization algorithms can be used for i.i.d. datasets (typically called online to batch conversion), while the reverse is generally not true.

It is unclear which setting the paper studies. My understanding is that setting 2 is more relevant to the paper, but the authors should clarify and explain the setting of the paper explicitly. I also suggest the authors make Section 2 to be related works and explain settings that are already known, in order to better position the contribution relative to literature. This also ensures better dissemination of scientific information.

**Why should we learn the MAP?**

The paper in Section 2 and 3 sets up an objective that aims to find the MAP estimate, i.e. $\theta^* = \arg \max_\theta P(\theta | \tilde{D}_{1:t+1})$. However it is not clear why this should be the objective. In a streaming setting, doing well on the past need not imply doing well on future streams of data unless some assumptions are made about the data distribution.

**Why cant we store the data in a "replay buffer"?**

The paper in multiple places states that storing historical data increases the algorithmic complexity (e.g. for FTML). Is this really a bottleneck? Can you present data or experiments to back this up -- my guess is that for the tasks/experiments considered in this work, storing historical data in memory is not that expensive. Furthermore, disk/memory costs are cheap and it is routine in ML these days to work with very large datasets. For example, deep RL stores all historical data in the form of a replay buffer for learning. Thus, it is unclear to me why FTML has not been considered for comparison?
review_title: Review of Bayesian Online Meta-Learning

final_score: rating: 7: Good paper, accept
review_content: The paper proposes an Bayesian approach to online meta-learning. This is done by lifting (approximate) sequential Bayesian inference from the model parameters to the meta-parameters. Two approaches are proposed to do this: (i) Laplace Approximation (LA), thereby extending Ritter et al’s method from online learning to online meta-learning; and (ii) VI, thereby extending Nguyen et al’s Variational Continual Learning (VCL) method in a similar way. Experiments are performed by converting existing meta-learning benchmarks into online settings in two ways, which they call “sequential tasks” and “sequential datasets” respectively. The experimental results show resistance to catastrophic forgetting in both of these experimental settings. 

In general, the novelty is somewhat limited, as this extension of LA/VCL to meta learning is similar in principle to the Grant et al formulation as a hierarchical Bayesian model. The main difference  is that the inner loop is done using the standard fast adaptation method from MAML rather than using LA/VI.

If you’re only using a point estimate (“We are interested in a MAP estimate $\theta^*$”) then this is not truly Bayesian online learning. I assume that the choice of doing it this way is to sidestep the difficulty of integrating over the parameters to form the posterior predictive, but loses the advantages of having a Bayesian posterior. 

Missing references:
- ML-PIP/VERSA (Gordon et al 2019, Meta Learning Probabilistic Inference for Prediction, ICLR). This does Bayesian meta learning by directly optimising the posterior predictive, rather than the usual posterior in VI. It also makes use of amortisation networks so that at test time only forward passes through NNs are required. They don’t provide an online formulation. However, The posterior predictives can be used as prior predictives for future rounds of inference.
- BMAML (Kim et al 2018, Bayesian model-agnostic meta-learning, NeurIPS). This uses Stein Variational Gradient Descent (SVGD) to obtain the task posterior for a novel task, casting MAML as a hierarchical probabilistic model. 
- Jerfel et al 2019, Reconciling meta-learning and continual learning with online mixtures of tasks, NeurIPS. This paper also uses a hierarchical Bayesian formulation, but takes a non parametric mixture approach using Dirichlet processes. They handle distribution shift in the online meta-learning setting by modelling latent task structure. 

Baselines:
- The baselines in this paper are a bit weak. The two methods ML-PIP and BMAML mentioned above, along with the Grant et al hierarchical Bayes method, could all be applied here, but since they weren’t originally formulated for the setting it would require extra work to do so. However, several methods were mentioned in the related work setting which should have been included: the He et al Task agnostic continual learning and Harrison et al continuous meta learning methods.

Edit post reviewer responses:
- I had some misunderstandings in the review above, which have mostly been cleared up. I’ve raised my score accordingly
review_title: Combination of Bayesian online learning and meta-learning - neat formulation but missing key references and baselines

Meta-review: This paper proposes an online meta-learning algorithm. 3 out of 4 reviews were borderline. The main concern during the discussion was that it is unclear what kind of online learning this paper does. For instance, in theory, the online learner competes with the best solution in hindsight. This is a regret-minimizing point of view. The other online learning is streaming. In this case, there is no regret. The goal is a sublinear representation that is competitive with some baseline that uses all space.

After the discussion, I read the paper to understand the points raised by the reviewers. I agree that this paper is not ready to be accepted. My quick review is below:

The authors combine MAML and BOL to have online updates (not all tasks are required beforehand) and handle distribution shift. But the way of combining these is not well justified. In particular,

1) The distribution-shift story is not convincing. The reason is that the proposed algorithm is posterior-based. By definition, when you use posteriors as in (3)-(5), you assume that the datasets are sampled i.i.d. given \theta. This means independently and identically. So no distribution shift. I am familiar with Kalman filtering. For that, you need p(\theta_t | \theta_{t - 1}) in (3)-(5), which would be sufficient for tracking stochastic distribution shifts.

2) I find the use of BOL unnatural. Since MAML is gradient-based, it would be more natural to have a gradient-based online learner. Gradient descent has online guarantees and does not require i.i.d. assumptions.

3) The authors should clearly state what the objective of their online algorithm is. In particular, the informal justification of (3)-(5) as doing something similar to MAML (the paragraph around (6)) is highly confusing. I could not understand what the authors mean.
Decision: Reject