Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a new method for (offline) continual learning using bayesian hypernetworks to obtain a distribution over a task-based parameter generating function. This meta perspective enables replay at the level of level of the hypernet parameters (rather than at the input level), while the bayesian approach provides reliable uncertainty estimates which the authors show can be used to perform task inference at test time. 

The hypernet setup, (which is really a 3-level hierarchy) comprises a 1) Task-Conditioned network, which maps a task embedding to hypernet parameters $\theta_t$, 2) a hypernet with params $\theta_t$ remaps a noise sample $z$ to a set of weights $w$, 3) which are used as parameters for the downstream classifier. 

The authors provide two toy setups to illustrate the validity of their method, before benchmarking their method on Split-Cifar and Split Mnist. 
main_review: __Originality__:  The proposed method is novel: the bayesian hypernet perspective is a fresh take in the CL field. The related works section is complete and cited adequately. 

__Quality__: The submission is technically sound and the claims are well supported via motivation and empirical results. The toy experiments are beautifully designed, especially the second one. The work is well rounded and the authors have made a good use of the place available. 

__Clarity__: The method is well explained. It is a very involved method (a 3-stage pipeline) but the authors do a good job to convey the important information across. The only caveat I will add to this is regarding the spectral Stein gradient estimator. Ideally the authors should provide some intuition on how it works, or refer to the appendix for a clearer explanation. To me this part seems quite important, as it key to optimizing implicit hypernets. 

__Significance__: The results are significant, especially due to the task inference module. This area is largely unexplored and the authors take a solid step to perform principled inference. 

__Other__: 
One thing I would like the authors to shed light on is the difficulty of the optimization process. The proposed method is quite complex, and from the looks of it seems nontrivial to optimize. How stable / sensitive to hparams is the method ?
Moreover, it would be best if the authors discuss how the hparam selection was made for the baselines considered. Were all methods given a similar budget ?
Lastly, a good addition to the paper would be an ER baseline, with a buffer size equivalent to the size of the core-set used for `-CS` methods. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a Bayesian approach for continual learning that models the weights distributions in a task-conditioned fashion, then relying on task inference.
As expected, the main main issue is the task inference phase.
The experimental validation is complete, although it is limited to only a small number of tasks in the same sequence.
main_review: Main merits:
- The paper is interesting and the proposed method is sound.

Main limitations:
- split-MNIST and split-CIFAR are very simple baselines, and all continual learning tests presented only have a small number of tasks.
- i would definitely not call cifar10 a 'natural image dataset'

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to tackle task-agnostic continual learning from a Bayesian deep learning perspective. In contrast to previous work along these lines the paper proposes to parameterize the posterior (over network weights) using a mixture distribution with one mixture component per task (seen so far). Three main options are explored - a point estimate (i.e. a single set of network weights) per task, a parametric distribution (Gaussian w. diagonal covariance) per task, and an “implicit” distribution parameterized by a neural network (using the reparametrization trick essentially). All three options can be unified/generalized by learning a weight-generator which, when conditioned on the task, produces a distribution of weights for the network that solves the task (e.g. the classifier). Importantly, the parameters of the weight-generator are produced by another neural network (the hyper-network) whose input is conditioned on the task. The question is how this task-conditioning is obtained: the paper proposes to automatically infer the task via the (epistemic) uncertainty modeled by the distribution of functions (classifiers) defined by the task-conditioned posterior over parameters (produced by the hyper-network). Experiments on synthetic data and continual learning versions of MNIST and CIFAR-10 are shown - with lots of ablations and control experiments. Results look promising, though they hint at some scalability issues for implicit distributions (an issue commonly observed in the Bayesian Deep Learning literature), as well as issues with reliable task inference via epistemic uncertainty on OOD data (another important open problem whose solution is beyond the scope of this paper). Limitations and conclusions are well discussed and related literature if well summarized.


**Contributions:**

1) Extension of a previously published approach to Bayesian CL via hyper-networks. The main novelty is to allow for the hyper-networks to parameterize distributions over networks via the weight-generator (including the earlier work as a special case - the MAP case with point-estimates for the posterior). The necessity of this extension is well motivated and the benefits are demonstrated nicely on synthetic data and result in significant performance gains on the image classification tasks. Significance: a main drawback of a SOTA method was identified and solved in a well-motivated and satisfactory manner (particularly from a Bayesian perspective). While similar approaches can be found in the literature (as discussed in the paper), the combination with automatic task inference is novel, and crucial relies on this innovation. I think the method will receive a fair amount of attention in the Bayesian CL community.

2) Automatic task inference via the epistemic uncertainty on OOD data given by the Bayesian ensemble (parameterized by the hyper-network). While previous CL methods have proposed to use task inference - the combination with contribution 1) allows for a potentially powerful and principled solution to the problem. Benefits (and some remaining issues) are shown in the results. Significance: the approach is very reasonable from a theoretical/principled point of view. The combination of contributions 1) and 2) is perhaps one of the most promising advantages of Bayesian CL. The particular way this is executed in the paper is novel and original (to the best of my knowledge), and it is well motivated and critically analysed in the paper. The results obtained here will be important for the Bayesian CL community, and allow for a clear separation of the task identification problem such that improvements in reliable modeling of epistemic uncertainty (which is an important topic outside CL) will directly contribute to improvements of Bayesian CL.

3) Solving the problem of catastrophic forgetting and task interference with a regularizer (posterior meta-replay) that acts on the level of the hyper-network (parametrization of the posterior over classifier weights). Significance: solving the problem is crucial to successful CL - the approach taken here is sound and empirical results underline the success of the regularizer. This is an interesting result for the Bayesian CL community.

4) Well-written and comprehensive discussion of the two main innovations, including a range of specific implementations for each. These variants are discussed, related with the current literature and evaluated in a large number of experiments. Results are critically evaluated and limitations are nicely discussed. Significance: given how well the paper is written and executed, and the wealth of experimental results, ablations and control experiments, I think the paper has the potential to become a seminal next step in Bayesian CL. My only slight worry is that the paper certainly pushes against the constraints of a conference-format paper, with many interesting parts of the paper pushed to the appendix, and a quite high information density in the main text.

main_review: **Originality, Quality, Clarity, Correctness:**
To the best of my knowledge, the precise method proposed in the paper is novel and original. There are some highly related methods in the literature, which are discussed in the paper and the innovations/distinctions are made very clear. The method is well motivated and aligns well with a principled Bayesian CL approach. The paper is very well written, though the density of material in the main text is very high and the supplementary material is extensive. The main innovations are well motivated and these motivations are supported with illustrations and experiments on synthetic data. Rather than simply aiming to “beat some benchmark numbers” the paper conducts a very differentiated analysis through a set of control experiments and ablations. The main results of this are promising and in favor of the method. Limitations and caveats are well discussed in the paper.

**Verdict:**
To me, this is a well written paper, with clearly articulated main ideas, which are critically analyzed and discussed. The main problem (continual learning) is timely and important, and a Bayesian solution is sound and promising - the focus in the paper is on sequences of image classification tasks on the same data-set, where i.i.d. sampling is possible within each sub-task. This is an important limitation, but is clearly pointed out in the paper, and helps to make the focus of the work clear. Perhaps the main value of the paper is to provide a framework with various choices for parameterizing the weight generator, implementing the task inference, etc. Future improvements for these individual parts can easily be incorporated, and I would expect to see a fair number of follow-up work in this direction (e.g. normalizing-flow based implicit distributions). Given my current understanding of the literature I am clearly in favor of accepting the paper. I have some small comments and one concrete suggestion for improving estimation of epistemic uncertainty. I am, of course, happy to reconsider my verdict based on the other reviews and the authors’ response. I am not a top expert on the Bayesian CL literature (but very familiar with Bayesian DL and uncertainty estimation), hence my confidence rating.

**Pros:**

 * Principled approach, well embedded in the current literature and SOTA methods for Bayesian CL and Bayesian neural networks.
 * Well explained and illustrated motivations. Good explanation of the individual parts of the method, what role each part plays, and what reasoning went into the choices made for each part (and potential alternatives). This allows future work to easily connect/improve by focusing on individual parts.
 * Promising results, backed up and critically analyzed through a range of control experiments, comparisons and ablations.
 * Great discussion of current limitations (and potential societal impacts), relations to current literature made very clear.

**Cons:**

 * Paper at the limit of the conference format - main paper quite dense, loads of interesting discussion, experiments, etc. pushed to appendix (which is itself quite extensive)
 * No results beyond CIFAR-10, which is a common limitation in Bayesian DL
 * Some issues with reliable estimation of epistemic uncertainty and scalability of variational inference for the implicit parametrization remain (though this is clearly shown and discussed, and these are two important unsolved problems in the wider literature) 

**Improvements:**

1) Improvement to estimate epistemic uncertainty via the entropy (line 233-236). Ent (entropy over predictive uncertainty) captures epistemic and aleatoric uncertainty (which is undesirable). The two factors can be teased apart fairly easily by a scheme similar to Eq. (3) in [1]. The main idea is to take the difference between the entropy of the average prediction and the average entropy of the individual predictions, where the average is taken w.r.t. the MC samples from q_\theta(W)). For high aleatoric uncertainty but no epistemic uncertainty both quantities should be equal and cancel out by subtracting. High epistemic uncertainty is quantified with a strictly positive quantity (a mutual info term, similar to an entropic term).

[1] Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning, Depeweg et al. 2018.

2) Experiments beyond CIFAR-10 would be nice (though I personally don’t regard them as crucial for publication).

3) The main text is fairly dense (and given this density it is very well written). The method consists of a number of components (with various choices per component), which further increases the complexity of the text (again, given this complexity I think the text is very well written). Additionally, many results and some discussion does not fit into the main paper at all, leading to a fairly lengthy appendix. Overall I think the paper would have benefited from a publication format that allows for longer main text. Given the constraints of a conference format I think the authors did a great job, and I certainly think that the work is an excellent fit for NeurIPS. Just wanted to mention this here for consideration for future publications.


**Comments:**

1) Potential drawback that should be discussed: for some tasks a lot of the task-specific posteriors might look very similar - having a single joint posterior in this case could lead to efficiency gains in terms of learning and transfer, compared to having a separate posterior per task. These gains might be partially harnessable through the hyper-network (which, after all, is shared across tasks) but it is unclear to which degree this would happen.

2) Which inference metric was used for identifying the task in 4.3 - Ent or Agreement? Perhaps using BALD (as suggested in the minor improvement above) leads to improvements (particularly over Ent).

3) I found Fig. 2 very helpful :)

limitations_and_societal_impact: Limitations are nicely discussed in a separate section, including societal impact.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This submission proposes a continual learning method based on a shared hypernetwork that parameterises a distribution over the network parameters for each task given a learned task embedding. It presents this as a Bayesian method, which is **not correct** as the objective function does not minimize a divergence from the posterior. Practically, the method is mostly **incremental** over (von Oswald et al., 2020), replacing the weight outputs of the hypernetwork with the parameters of a distribution over these. Finally, the paper omits large parts of the literature on continual Bayesian deep learning, displaying a disappointing **lack of scholarship**. Overall, this is a **clear rejection**.

Reference:
J. von Oswald et al. Continual learning with hypernetworks. In ICLR 2020.
main_review: **Technical correctness**

The paper argues that it approximates the posterior for each task. Considering that the objective introduces an ad-hoc term to prevent forgetting, this assertion is not correct. Optimizing the parameters of a distribution w.r.t. the variational lower bound is a principled approximation as this is equivalent to minimizing the KL divergence between the approximate and true posterior, hence we get the exact result if the variational family is sufficiently expressive. Optimizing a distribution over the parameters does not automatically constitute a Bayesian method, even if the variational lower bound appears in the objective function, as the addition of ad-hoc terms to the objective breaks the property of minimizing the deviation from the posterior. This is exacerbated here by changing the approximate posterior on a task when further training on subsequent tasks. I'd be curious if a similar objective as the one used in this work could be derived from a hierarchical probabilistic model, however there is no discussion pointing towards this unfortunately.

The paper further proposes to infer the task identity based on the predictive uncertainty of each task-specific weight distribution. This is referred to as "principled" without further explanation. To me, a principled approach would involve modelling the task identity as part of the probabilistic model, approximating the posterior given a data point and then accordingly marginalizing(!) over all task-specific posteriors for the final predictions. The paper does not justify its approach theoretically. Semantics aside, the major drawback of Bayesian neural networks is the computational cost of having to calculate multiple forward passes. The proposed approach further aggravates this by requiring multiple forward passes  _for each task_ observed so far, which is entirely impractical considering that ensembling of a single model is already prohibitive in many settings. The computational cost should at least be discussed and not swept under the rug (there are some runtime results in the appendix, but those appear to be for the setting with given task identities at test time).

**Originality**

The core of the method is a modification of [1] (referred to as "PR-Dirac" in this paper), which parameterizes the network parameters via a meta-network. PR-Exp instead outputs a per-parameter mean and a variance, samples the parameters from the corresponding factorized Gaussian and regularizes with a divergence measure towards previous distributions instead of a quadratic penalty, both of which are a rather incremental change over [1]. More interestingly, PR-Imp outputs the parameters of yet another hypernetwork, which in turn samples the parameters of the network of interest based on a noise input. However, this seems to introduce little additional difficulty and judging from the paper can be achieved with similar techniques as the original hypernetwork in [1]. Further, hierarchical models for neural network weights have been considered in various works on Bayesian neural networks, see also the scholarship section.

**Clarity**

Overall the paper is reasonably clear in its description of the method. Key variables are not defined unambiguously in some places, e.g. $\mathcal{L}_{task}$ is only specified in the appendix. Also, which divergence was used for PR-Exp in eq. (1)? The text insinuates the KL divergence, but how did you choose between forward and reverse KL? The equation implies that you're requiring the old distribution to be covered by the new one, but that seems like it would allow for parameter settings that perform poorly on the old data?
Finally, the pervasive use of custom acronyms makes the paper rather cumbersome to read.

**Significance**

The methodological advances over [1] are incremental. The proposed method for test-time task inference is prohibitively expensive and appears to only work on MNIST, placing the significance of this work at the low end of the spectrum.

**Scholarship**

There are numerous prior works on Bayesian methods for continual deep learning, which the paper mostly ignores [2-7]. It is  also  inaccurate to state that EWC (in constrast to online EWC) finds a shared posterior for all data due to the per-task terms (see the note by Huszar referenced as [25] in the paper). Besides works on Bayesian continual learning, a discussion of federated learning methods, e.g. [8,9], would seem relevant, as these typically work with per-task posteriors as well. [10] similarly consolidates per-task Gaussians into a shared Gaussian that it carries forward, so PR-Exp could be related as an amortized version of that procedure (which maintains the ability of producing per-task Gaussians). Finally, works on hierarchical weight models for neural networks are again not discussed [11-13]

**Further comments**

- The references are formatted inconsistently (venue names, title capitalization, inclusion/omission of page/issue/etc numbers) and look like they have been copy-pasted from Google scholar.
- Figure 2c suggests that the implicit posterior is potentially multi-modal, however the paper presents no evidence that this is the case in the experiments beyond toy datasets, making the figure potentially misleading. Given the similar performance of PR-Imp to PR-Exp I would find it highly surprising if the distribution was indeed multimodal.
- Are you sure that Figure 3a shows PR-Exp and not PR-Imp? I have not seen a mean-field Gaussian variational posterior with a remotely comparable increase in predictive uncertainty away from the data anywhere in the literature or my own experiments before. If this is indeed the result for PR-Exp, this would point towards this not really being a variational method as discussed above.

References

[1] von Oswald et al. Continual learning with hypernetworks. In ICLR 2020.

[2] Titsias et al. Functional regularisation for continual Learning with Gaussian processes. In ICLR 2020.

[3] Loo et al. Generalized variational continual learning. In ICLR 2021. (on arxiv since Nov 2020)

[4] Pan et al. Continual deep learning by functional regularisation of memorable past. In NeurIPS 2020.

[5] Adel et al. Continual learning with adaptive weights. In ICLR 2020.

[6] Ritter et al. Online structured Laplace approximations for overcoming catastrophic forgetting. In NeurIPS 2018.

[7] Zeno et al. Task agnostic continual learning using online variational Bayes with fixed-point updates. arXiv preprint arXiv:2010.00373, 2020.

[8] Bui et al. Partitioned variational inference: A unified framework encompassing federated and continual learning. arXiv preprint arXiv:1811.11206, 2018.

[9] Yurochkin et al. Bayesian nonparametric federated learning of neural networks. In ICML 2019.

[10] Lee et al. Overcoming catastrophic forgetting by incremental moment matching. In NeurIPS 2017.

[11] Dwaracherla et al. Hypermodels for exploration. In ICLR 2020.

[12] Karaletsos et al. Probabilistic meta-representations of neural networks. arXiv preprint arXiv:1810.00555, 2018.

[13] Karaletsos & Bui. Hierarchical Gaussian process priors for Bayesian neural network weights. In NeurIPS 2020.
limitations_and_societal_impact: n/a
review_title: null

Meta-review: We thank the authors for their detailed clarifications. Most reviewers agreed that this paper makes interesting contributions to the area of continual learning, which is relevant to the NeurIPS community. There were extended discussions about the Bayesian positioning of the paper and gaps in the related work. The former was settled by the authors' detailed response how online EWC can be obtained from a series of approximations and how posterior meta-replay can be derived by approximations of the posterior. Regarding the latter, I would encourage the authors to not merely cite, but to clearly explain how their work builds on prior work.
Decision: Accept (Poster)