Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors tackle the problem of continual meta-learning, where the tasks are streamed. This settings comprises the additional challenge of catastrophic forgetting.
They introduce a training method VC-BML, which updates a GMM over the meta-parameters dynamically. The idea is to cluster the posterior meta parameters over the modes of the GMM. Additionally, they leverage structured variational inference in order to approximate the posteriors.
The experiments show an improvement in terms of mean meta test accuracy over the stream of tasks.
main_review: # Originality
1- **Are the tasks or methods new?**

The method is new overall.

2- **Is the work a novel combination of well-known techniques?** 

The work builds up on techniques from variational inference and meta learning, overall it's not a simple combination of well-known techniques.

3- **Is it clear how this work differs from previous contributions?**

The difference from previous contributions is discussed thoroughly.
 
4- **Is related work adequately cited ?**

The related works are adequately cited, to my knowledge.

# Quality
1- **Is the submission technically sound ?** 

The submission is technically correct.

2- **Are claims well supported (e.g., by theoretical analysis or experimental results) ?**  

- The theoretical analysis is correct and supports the paper's motivation.
- The experiments are clearly split and support the claims of the paper.

3- **Are the methods used appropriate ?** 

- The methods used are tailored to the problem (DGMM, variational inference ....)

4- **Is this a complete piece of work or work in progress ?**  

It is a complete piece of work.

5-  **Are the authors careful and honest about evaluating both the strengths and weaknesses of their work ?** 

The authors also evaluate the limits of their work in terms of :
- The importance of the KL down-weighting parameter
- How to select the maximum number of mixtures

# Clarity: 

1- **Is the submission clearly written?** 

The submission is easy to follow and clearly written. Also, the theoretical analysis is clearly outlined.

2- **Is it well organized? (If not, please make constructive suggestions for improving its clarity.)** 

The paper is well organized and easy to follow.

3- **Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)**

- The source code is provided.
- The hyperparameters are also presented in the appendix.

# Significance: 
1- **Are the results important?** 

Beyond the improvement of the state of the art, the results are important in the way they validate the DGMM for meta learning as a reliable proof of concept.

2- **Are others (researchers or practitioners) likely to use the ideas or build on them?** 

It is likely that practitioners may apply the method to real world systems, given the improvement in comparison with previous methods.

3- **Does the submission address a difficult task in a better way than previous work?** 

The results in Table 2 show that the proposed method outperforms previous works on most settings presented. 

4- **Does it advance the state of the art in a demonstrable way?** 

It advances the state of the art, though the improvement is around 1 to 2 percents in terms of meta-test accuracy. 

# Minor comments
- There is a typo in the page 2 of the Appendix, line 29 "is infer" -> "is to infer" :)  
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose Variational Continual Bayesian Meta-Learning (VC-BML) to handle various tasks which arrives sequentially. In particular, the authors consider the tasks follow a non-stationary distribution and assume that the previous dataset cannot be stored in memory. To deal with the non-stationarity and low-resource, the authors propose to perform sequential Bayesian inference where the meta-parameters are estimated by Dynamic Gaussian Mixture Model (DGMM) whose number of components are determined by Chinese Restaurant Process.
In the experiments, the authors demonstrate VC-BML successfully learns the new task without suffering the negative knowledge transfer issue while avoiding the catastrophic forgetting and maintaining the ability to solve the previous tasks.
main_review: The paper is well organized and the idea is easy to follow.
The method is technically sound; a meta-learning framework is combined with DGMM and Chinese Restaurant Process which are well established method to deal with the unknown number of components.
The drawback of the method is that the method has to keep the meta-parameter for each task, which becomes difficult when the number of tasks increase.
The experimental results are convincing as the proposed method is compared with the baselines and existing methods including DPMM and BOMVI. It shows the benefit to combine the Chinese Restaurant Process.

I have a few minor comments.
1. D_t^S appears in Figure 1 and Equation (15) without any explicit explanation. Please add explanation on D_t^S.
2. EBLO should be ELBO at Lines 205, 261, and 311.
limitations_and_societal_impact: I agree with the limitations raised by authors, and feel they are adequate.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper attempts to solve the problem of meta-learning when data arrive sequentially. Unlike previous approaches, the authors propose a method to maintain a dynamic Gaussian mixture model over the meta-parameters. The number of clusters is picked using a Chinese restaurant process. Various experiments have been conducted to show the accuracy and superiority of the model.
main_review: This paper attempts to solve the problem of meta-learning when data arrive sequentially. Unlike previous approaches, the authors propose a method to maintain a dynamic Gaussian mixture model over the meta-parameters. The number of clusters is picked using a Chinese restaurant process. Various experiments have been conducted to show the accuracy and superiority of the model.

It is great to see a truly Bayesian treatment for meta-learning. Maintaining a multimodal latent distribution for meta-parameters is a neat idea. I did not check all the derivations but the paper follows the standard structured variational inference derivations and looks legit. Though using Chinese restaurant process for selecting the clusters is ideal, I am wondering if it is overkill. Possibly, we could have made the number of clusters a parameter of the mixture and learn it through variational inference itself. In that case, the mixing coefficients could follow a Dirichlet distribution the parameters of each Gaussian would be Gaussian-Wishart. 

I would like the authors to discuss more about the practical aspects of the algorithm. What is the dimensionality of latent spaces?, how do we avoid mode collapse?, what is the computational time?, etc. If our initial distribution is not broad and diverse enough, I am skeptical if the posterior distribution can indeed pick the necessary modes of the new distribution of incoming data. As another limitation, if the parameters of the new underlying dataset are truly multimodal [1], the variational approximation would not be able to capture the variational distribution as it would pick only one mode by construction. 

Minor: 
* Line 58 - “It’s” - avoid contractions
* Line 101 - Use parentheses as {(x,y)} to show that what repeats is the input-output pair.

[1] Itkina et al., "Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders," NeurIPS'20

limitations_and_societal_impact: Adequate
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **Update after author response:** I have raised my score from 4 to 6, even leaning towards 7 (but I would need to see the revised manuscript to confidently give a 7, which is unfortunately not possible, because a fair amount of my criticism is aimed at various passages throughout the paper and the overall claims and style of presenting them).

The paper addresses task agnostic continual (meta-) learning from a Bayesian perspective. The setting in continual learning is to learn a sequence of (potentially quite different) tasks, image classification tasks in the paper - the main difficulty is to learn novel tasks while avoiding catastrophic forgetting of previously seen tasks and negative interference (i.e. decreased learning or suboptimal solutions) from previously learned solutions. As discussed in the existing literature, a principled approach to do this is to perform Bayesian inference over the parameter-posterior given the current experience. This posterior can act as a prior for new experience - the standard sequential Bayesian posterior update. In practice, this update is intractable and the paper resorts to approximate inference: in essence a pior distribution is learned per cluster of related tasks. These task-group priors allow to rapidly adapt to each task within the cluster (which is performed in the paper in a MAML like fashion, i.e. via gradient update steps starting from an initial parameter-draw from the task-group pior). The main idea is thus to use a mixture distribution (mixture over Gaussians in the paper) to represent a multi-modal approximate posterior over task-parameters, which acts as a “good” prior for new observations, and thus allows for knowledge transfer between tasks of the same cluster. Since the number of clusters is not known in advance, and, more importantly, needs to grow with an increasing number of tasks observed the model employs a non-parametric posterior approximation (a Dirichlet process that models the number of Gaussian mixture components). The method is compared, and performs well, against a number of previously proposed methods.

**Contributions:**

1) Extension of the model used by a previous method [4] by using a mixture of Gaussians for the approximate posterior - each mixture component is supposed to model one cluster of tasks (in the previous paper each mixture component is a delta-distribution, leading to a Dirichlet Process Mixture Model DPMM). The number of mixture components is inferred from the data via a Dirichlet process (leading to a Dirichlet Process Gaussian Mixture Model DPGMM) - in contrast to the paper’s claims this has been done in precisely the same fashion in [4] (see Section 5 in that paper, which also uses the CRP construction). Significance: the improvement in the model is sensible but fairly incremental - it would have been interesting to allow for multi-modality on this level as well. Results are in favor of the improvement, but do not convincingly demonstrate that the observed improvements are solely caused by this modification (see comments below).

2) In contrast to [4] (which derives a MAP inference scheme), a variational inference scheme is derived. The innermost step of that scheme (MAML-type optimization using a few gradient-descent steps) relies on meta-learned amortized variational inference, variational inference of \theta relies on ELBO-type optimization via SGD and the reparametrization trick. Significance: the VI scheme is interesting and shown to scale to neural networks. Given the model complexity this is not trivial - however this complexity introduces many hyper-parameters and application of the model to new data might not be straightforward (the VI can easily break in subtle ways that are hard to diagnose and debug). Since the paper does not report in detail how most hyperparameters were found, and how sensitive the results are w.r.t. their choice becomes even more difficult.

3) Experiments on a sequence of four image classification tasks with comparison against two baselines and several related models from the literature. Results suggest that the model outperforms all other models in most cases. Significance: results look indeed promising, but at the current stage it is hard to attribute the observed gains in performance to the various improvements made. Important control experiments and ablations are missing to make sure the observed gains are actually due to the hypothetical claims that went into the model and inference improvements (more on this, see Improvements below).
main_review: **Originality, Quality, Clarity, Correctness:**

While the use of a Dirichlet-process-Gaussian-mixture-model is certainly not novel, and fairly straightforward from a Bayesian viewpoint on continual learning, I have not seen this particular instantiation of the model in the continual learning literature before (though I might have some gaps, hence my lower confidence score). Getting VI to work reliably for such a model (with high-dimensional neural network weight-matrices as the parameters at the bottom of the hierarchical model) is no easy feat, and results in the paper seem to suggest that this works successfully. Unfortunately some of the claims in the paper are overstated and/or not sufficiently backed up by experimental evaluation - perhaps most importantly the differences to the model in [4] are more incremental than the paper claims (see criticism below for details). The clarity of the writing could be improved (see comments), and while there is no part that is clearly wrong in the paper the correctness of the interpretation of the results is unclear since important ablations and control experiments are missing (to show that the observed performance gains are due to the reasons claimed).

**Verdict:**

I think the main ingredients for a good paper are there - the central idea is a small, but sensible improvement in the model and the VI scheme looks interesting. Results shown in the paper are promising, but now extra work needs to be put in to make sure that the observed results come from the main innovation and to disentangle which innovation contributes to the performance gains to which degree. If done thoroughly, I think the paper has the potential for a fairly significant impact in the continual learning community. At its current stage I personally think that the paper does not have sufficient rigor and experimental evidence to back up all the claims made. I therefore suggest a major revision, and want to strongly encourage the authors to “go the extra mile” and turn this into the strongest paper possible, rather than rushing with publishing preliminary findings. I am happy to reconsider my verdict of course in light of the other reviews and the authors’ response.

After re-reading my review I want to point out that I’m not an author of [4] (or collaborator of the authors) - my aim is not to “defend” [4], but the differences need to made clear (both formally and experimentally).

**Pros:**

 * Clearly articulated, focussed problem setting. Main model improvement is sensible.
 * Derivation of a fairly involved VI scheme.
 * Promising results

**Cons:**

 * Overstated (or wrong) claims w.r.t. previous work - most importantly [4] in the paper uses a Dirichlet process to automatically infer (and increase) the number of mixture components.
 * Unclear whether the improved results (w.r.t. [4]) come from:
   * the Gaussian mixture model
   * the VI scheme as opposed to MAP inference in 4
   * the annealing of the regularizer
   * potentially more/better hyper-parameter tuning for the main method but not for the competitors
 * Some parts of the writing could be more rigorous (being precise about what exactly is more or less ‘Bayesian’, clearly describing what meta-parameters refers to, the params of the whole DPGMM?, the params of the Gaussian mixture?, the distr. over params induced by a single mixture component?, …)

**Criticism w.r.t. specific claims**

*Claimed advantages of the method*

 * L 221:  “the meta-parameters in VC-BML are represented by a dynamically updated mixture model compared to a single static distribution in [4].” - this claim is wrong, Section 5 in [4] uses the same CRP construction to dynamically adjust the number of mixture components.

 * L 229: “ Third, the KL-divergence in our objective function acts as a regularization term, the weight of which can be tuned to reduce over-fitting” - this is also true in principle for MAP estimates (which implies computing a posterior) as in [4], the posterior can be regularized via the prior.

*Other claims*

 * L41: “they use point estimation to infer parameters”  - be more precise, which parameters?

 * Line 87 to 91: “propose to cluster task-specific parameter distributions and allow the meta-learner to select over a mixture of these distributions. Nonetheless, the meta-parameters still follow a single stationary distribution and the latent variables are inferred by point estimation, which is found prone to suffer from the catastrophic forgetting issue” - what do you mean with “follow a single stationary distribution”? As far as I can tell this refers to the mixture governed by the Dirichlet process, which is not stationary. If used like this, the word “single” also applies to the current paper (a single DPGMM distribution).

 * L94-95: “ We overcome these weaknesses by representing meta-parameters with a dynamically updated mixture model and inferring latent variables via structural variational inference.” Again, the first half or this sentence is what [4] also does, and it is unclear (from the arguments in the text) why “inferring latent variables via structural VI” (as opposed to MAP estimation?) helps overcome the weaknesses. Be more precise here.

 * Sec. 4 title - not sure what’s particularly Bayesian or non-Bayesian about the proposed model. Both, equation 1 and 2 approximate the Bayesian posterior over parameters \theta. 2 uses a hierarchical model to do so (but the parameters \phi could in principle be absorbed into a more complex parameter \theta). There isn’t really anything more or less Bayesian about it - it’s just in one notation (Eq. 2) the hierarchical model is made explicit, whereas in Eq. 1 it could either be a hierarchical model or a flat model (the latter is of course more commonly seen in the literature). I agree that the model proposed is able to capture some uncertainty that cannot be captured by e.g. [4]. I personally would find it more fitting to call the model DPGMM, Dirichlet prior **Gaussian** mixture model.

 * Line 139-140: “Consequently it will be inefficient if the task-specific parameters \phi_t are adapted from the same distribution of meta-parameters \theta_t.” This is strictly speaking not true (unless the notation is interpreted in a particular way) - the distribution of meta-parameters does not necessarily mean that there is only a single mode for \theta_t - the notation parameterizing the meta-distribution with \theta_t includes the case where \theta_t is a Gaussian mixture model or some other potentially multimodal distribution.

 * Line 144: “As meta-parameters follow Gaussian distributions” - why is this the case? Is this the modeling assumption? In general the meta-distribution should be multi-modal (which is exactly what is achieved with the mixture model), a Gaussian is clearly a wrong / overly simplifying modeling choice.

 * Line 228: “For most dissimilar tasks, point estimates can be dramatically changed, which is prone to forgetting previous knowledge”. This is to some degree also true for unimodal (Gaussian) distributions used in the paper (the only way they can cope with dissimilar tasks is by increasing their variance dramatically). The main mechanism to overcome this here (and in [4]) is via the mixture model.


**Improvements:**

1) Be more precise and very clear about differences and claimed novel advantages (see criticism above).

2) The question is where exactly the observed performance differences, particularly w.r.t. DPMM [4] come from. The following ablations are needed to determine this (the emphasis on comparing against [4] is because it is a very similar model with a different inference procedure)

2a) The proposed model reduces to DPMM if the variance of the Gaussians is manually fixed to very low values. Keeping everything else fixed, does the model perform better/worse or the same compared to DPMM? (this highlights the performance difference due to the different VI scheme).

2b) Another source of the improvements could be the annealing of one of the regularizer terms. Show a control experiment without annealing, and ideally attempt a similar annealing for the competitor methods (I am aware the latter can be tricky). It’s important to know how much of the performance gain is due to annealing.

2c) It is currently unclear how hyper-parameters for the method were found (including the parameter for the priors which act as regularizers). The performance gains might come from the fact more time went (perhaps accidentally while developing the method) into hyper-parameter tuning. Ideally show a clear process for hyper-parameter tuning and apply the same process to the competitor methods to make results more comparable.

3) Ablation: show how sensitive the results are to the various hyper-parameters. Do they need to be selected very carefully, or is the VI scheme quite robust

4) What was the (effective) number of clusters used by DPMM - is it comparable to DPGMM, and was there any attempt to tune the concentration hyper-parameter of DPMM?

5) Just to be sure (and since the writing might suggest otherwise) - the DPMM implementation that is being compared against is using the Dirichlet process (as in Section 5 of [4]) and not the static model (as in the first part of [4])? How was the DPMM implementation obtained?

7) Meta-comment: the benefits of the method (compared to point-wise meta-parameters as in [4]) should really become apparent when multiple different, but related tasks belong to the same cluster. This might or might not happen implicitly in the experiments shown (implicit subtasks within each dataset might exist). It would be interesting to see this clearly in a controlled experiment with synthetic data.






**Comments:**

 * Line 206: “Variational inference of \phi” - a point-wise global initialization is used in conjunction with (a few steps of) gradient descent. This sounds very similar to what [4] does via MAML. So what’s the difference between this and [4]? The appendix points out this similarity to MAML, what’s important to distinguish from [4] is whether the sample is drawn from the Gaussian or simply its mode/mean is selected for the global initialization (which would correspond exactly to [4]). The pseudocode in the appendix confirms that the value is sampled. Since this is one of the main differences this needs to be pointed out more clearly in the main text.
 * Compare (discuss, not necessarily via experiments) against “Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning” (Kessler, UAI 2021). What are advantages and disadvantages of the construction proposed in this paper? (I am not an author btw.)
 * Define in the intro: what is a ‘low resource’ task?
 * Error-bars in Table 2 often overlap between the first and second best solution. Please point this out in the text.
 * Another pass to check grammar and spelling
 * Line 185: “ Specifically, we use handy Dt to recursively infer latent variables” I don’t understand this sentence.
limitations_and_societal_impact: Some limitations of the method are discussed. What is missing is how to set the hyperparameters (annealing schedule, prior-parameters, etc.) in practice and how brittle the VI scheme is. Additionally current experiments show results for a small number of tasks, I’d expect difficulties with inference for larger numbers of tasks, but it’s unclear whether that problem becomes apparent for slightly larger numbers of tasks or whether the method is reliable for dozens of tasks.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This article presents a fully variational method to address the continual learning problem: training a neural network sequentially on different tasks, without forgetting preceding tasks.

The theoretical framework, namely variational inference with hierarchical random variables, is classical, which makes the general idea easy to understand.

The experiments are sound, concurrent methods have been replicated and tested (including naive baselines), and the main potential flaws or limitations have been discussed and addressed.
main_review: ## Detailed summary

The method is based on a hierarchical structure of random variables managing the final weights of the neural network to train. In order to deal with an unknown number of tasks, the weights $\phi$ of the NN depend on a categorical random variable $z$, whose range of values increases during training. In a nutshell, the range of $z$ is supposed to increase when the number and the variety of tasks increases.

The main theoretical difficulty is deriving the update rules for all variational parameters.

## Experiments

Setup: train sequentially the model on 4 given tasks (image classification), in a given order.

The authors compare their method to 7 other ones, including 2 naive baselines, which is sufficient to evaluate the method. They compare the results on each (seen) dataset during the sequential training on each task. The difficulty consists in keeping a good performance on a dataset when the model is being trained on another dataset. The proposed methods dominates all concurrent methods, except when learning the first task (first phase in Fig. 3).

Appendix F.2 is very important to check the fairness of the choice of the order of tasks in Section 5: the authors rerun the same experiments with the same tasks, *in inverse order*. The proposed method dominates all concurrent methods for each task, at any moment of the training. So we can reasonably be confident in the results presented in Fig. 3.

There is no comparison of training time between methods.

Section 5.2 confirms the intuition behind the range of the variable $z$: $z$ takes as many values as the number of tasks to perform. However, to be rigorous, Figure 4 should be replicated with 3 and 2 tasks, in order to check that the number of values $z$ can take are respectively 3 and 2.

## Questions

Why are results with OSAKA on Omniglot better in this paper than in the original one?

Is data augmentation used? It may be the cause of the impact of the KL mentioned in Section 5.3.

What about RNNs? For instance, text prediction (which is a simple task).

## Clarity

There is no issue of clarity if the reader is already familiar with hierarchical variational inference. Otherwise, the article may be hard to understand. For instance, the name of Section 4.2 could be changed to mention the update of variational parameters, which is the object of the section.

## Significance

The proposed method is an expected improvement in a field of medium importance (continual learning).

## Rebuttal

I am satisfied with the authors' answers. I maintain my rating, with a higher confidence.
limitations_and_societal_impact: As mentioned in the main review, potential limitations have been addressed.
review_title: null

Meta-review: The paper proposes a new, fully-Bayesian method for online meta-learning: VC-BML. A mixture models over the meta-parameters is updated dynamically. This is used as an informative prior for the upcoming tasks.

The Reviewers highlighted the novelty of the method with respect to previous works. The paper is theoretically sound, and VC-BML improves over known meta-learning algorithms in practice. Therefore, I will recommend to accept the paper.

Please take into account all the comments of the Reviewers for the camera-ready version of the paper. Especially, Reviewer h2Cd pointed out a few problems in the writing of the paper and in the comparison with Jerfel et al. 2019. The Reviewer updated his/her evaluation of the paper following your detailed reply and your promise to revise the paper accordingly.
Decision: Accept (Poster)