Posterior Meta-Replay for Continual Learning
Authors: Christian Henning
Maria Cervera
Francesco D'Angelo
Johannes von Oswald
Regina Traber
Benjamin Ehret
Seijin Kobayashi
Benjamin F Grewe
Joao Sacramento
Venue: NeurIPS 2021 Poster (2021)
Keywords: Continual Learning
Uncertainty Estimation
Bayesian Neural Networks
Task Inference
Hypernetworks

Abstract: Learning a sequence of tasks without access to i.i.d. observations is a widely studied form of continual learning (CL) that remains challenging. In principle, Bayesian learning directly applies to this setting, since recursive and one-off Bayesian updates yield the same result. In practice, however, recursive updating often leads to poor trade-off solutions across tasks because approximate inference is necessary for most models of interest. Here, we describe an alternative Bayesian approach where task-conditioned parameter distributions are continually inferred from data. We offer a practical deep learning implementation of our framework based on probabilistic task-conditioned hypernetworks, an approach we term posterior meta-replay. Experiments on standard benchmarks show that our probabilistic hypernetworks compress sequences of posterior parameter distributions with virtually no forgetting. We obtain considerable performance gains compared to existing Bayesian CL methods, and identify task inference as our major limiting factor. This limitation has several causes that are independent of the considered sequential setting, opening up new avenues for progress in CL.

Posterior Meta-Replay for Continual Learning
Christian Henning", Maria R. Cervera", Francesco D' Angelo, Johannes von Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, Benjamin F. Grewe, João Sacramento
*Equal contribution
Institute of Neuroinformatics University of Zürich and ETH Zürich
Zürich, Switzerland (henninge, mww.igccmmiiacrreeeeee Cetaaaceeeeeeeeeeeeeeeeea ch
Abstract
Learning a sequence of tasks without access to i.i.d. observations is a widely studied form of continual learning (CL.) that remains challenging. In principle, Bayesian learning directly applies to this setting, since recursive and one-off Bayesian up- dates yield the same result. In practice, however, recursive updating often leads to poor trade-off solutions across tasks because approximate inference is nec- essary for most models of interest. Here, we describe an alternative Bayesian approach where task-conding cendicondt parameter distributions are continually inferred from data. We offer a practical deep learning implementation of our framework based on probabilistic task cessk-ondtiioned hypernttwokss, an approach we term poste- rior meta-replay. Experiments on standard benchmarks show that our probabilistic hyperetworks compress sequences of posterior parameter distributions with virtu- ally no forgetting. We obtain considerable performance gains compared to existing Bayesian CL methods, and identify task inference as our major limiting factor. This limitation has several causes that are independent of the considered sequential setting, opening up new avenues for progress in CL.
Introduction
In recent years, variety of continual learning (CL) algorithms have been developed to overcome the need to train neural networks with an independent and identically distributed (i.i.d.) sample. Most CL literature focuses the particular scenario of continually learning sequence of T tasks with datasets ((1), ..... D(T), Because only access to the current task is granted, successful training of discriminative model that captures p( Y X) has occur without i.i.d. training sample from the overall joint (1:T)) i.idd p(X))((Y X).
The advantages of a Bayesian approach for solving this problem are numerous and include the ability to drop all i.i.d. assumptions across and within tasks in mathematically sound way, the ability to revisit tasks whenever new data becomes available, and access to principled uncertainty estimates capturing both data and parameter uncertainty. Up until now, Bayesian approaches to CL essentially focused on finding a combined posterior distribution via a recursive Bayesian update p(W D(::)) x p(W D(::T-1))(DD" W). Because the posterior of the previous task is used as prior for the next task, these approaches are also known as prior-focused [17]. In theory, the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

above recursive update can always recover the posterior pl ( data is presented. However, because proper Bayesian inference intractable, approximations are needed in practice, which lead to errors that are recursively amplified. As result, whether solutions that are easily found in the i.i.d. setting can be obtained via the approximate recursive update strongly depends on factors such as task ordering, task similarity and the considered family of distributions. These factors limit the effectiveness of the recursive update and have a detrimental effect on the performance of prior-focused methods, especially in task-agnostic CL settings.
[D(IT)), independently of how the
To overcome these limitations, we propose an alternative Bayesian approach to CL that does not rely on the recursive update learn distinct tasks and instead aims to learn task-specific pos- teriors (Fig. [] refer to SM F.T for a detailed discussion of the graphical model). In this view, finding trade-off solutions across tasks not re- quired, and knowledge transfer can be explicitly controlled for each task via the prior, which is no longer prescribed by the recursive update and can thus be set freely. By introducing probabilis- tic extensions of land-condit hypemetworks [9.] we show how task-specific posteriors can be learned with single shared meta-modes an approach we term posterior meta-replay.
MAP solutions
P(W (())
P(W D
p(W D)4)
W2
The proposed posterior meta-replay framework learns task-specific posteriors p (W D(t)) via a single shared meta-model, with task- specific point estimates (e.g. MAP) being a limit case. In this view, the modelled solution space is not limited to admissible solutions that lie in the overlap of all task-specific posteriors. By contrast, prior-focused methods learn a single posterior p WW D(::T) recursively and thus require the existence of trade-off solutions between learned and future tasks in the currently modelled solution space. Shaded areas indicate high density regions.
Figure 1:
This approach introduces two challenges: forget- ting at the level of the hypernetwrk, and the need to know task identity to correctly condition the hypernetwork. We empirically show that forget- ting at the meta-level can be prevented by using a simple regularizer that replays parameters of previous posteriors. In task-agnostic inference settings, often referred to as tlass-incrementae learning in the context of classification benchmarks [8]] the main hurdle therefore becomes task inference at test time. Here we focus on this task-agnostic setting, arguably the most challenging but also the most natural CL scenario, since the obtained models can be deployed just like those obtained via i.i.d. training (e.g., irrespective of the sequential training, the final model will be a classifier across all classes). In order to explicitly infer task identity from unseen inputs without resorting to generative models, we thoroughly study the use of principled uncertainty that naturally arises in Bayesian models. We show that results obtained in this task agg-ssic setting with our approach constitute a leap in performance compared to prior-focused methods. Furthermore we show that limitations in task inference via predictive uncertainty are not related our CL solution, but depend instead on the combination of approximate inference method, architecture, uncertainty measure and prior. Finally, we investigate how task inference can be further improved through several extensions.
We summarize our main contributions below:
We describe a Bayesian CL framework where task-conditioned posterior parameter distributions are continually learned and compressed in a hypernetwork.
series of synthetic and real-world CL. benchmarks we show that our cask-conditioned hyper- networks exhibit essentially no forgetting, both for explicitly parameterized and implicit posterior distributions, despite using the parameter budget of a single model.
Compared to frior-focused methods, our approach leads to a leap in performance in task-agnostic inference while maintaining the theoretical benefits of Bayesian approach.
Our approach scales to modern architectures such as ResNets, and remaining performance limita- tions are linked to uncertainty-basee -utt---distriuution detection but not to our CL solution.
Finally, we show how prominent existing Bayesian CL methods such elastic weight consolidation can be dramatically improved in task-agnostic settings by introducing a small set of task-specific parameters and explicitly inferring the task.
2

---

2 Related Work
Continual learning. CL algorithms attempt to mitigate catastrophic interference while facilitating transfer of skills whenever possible. They can be coarsely categorized as (1) invllarizationnmettooe that put constraints on weight updates, (2) replay-methods that mimic pseudo-i.i.d. training by rehearsing stored or generated data and (3) dynamic architectures which can grow to allocate capacity for new knowledge 7 Most related to our work is the study from von Oswald et al. [91] that introduces cask-conditioned hypernetworks for CL, and already considers task inference via predictive uncertainty in the deterministic case. Our framework can be seen as a probabilistic extension of their work, which provides task task-specific point estimates via shared meta-model (cf. Sec. 3). Follow-up work also achieves task inference via predictive uncertainty, e.g., Wortsman et al. [94] use it to select learned binary mask per task that modulates random base network. Here we complement these studies by thoroughly exploring task inference via several uncertainty measures, disclosing the factors that limit task inference and highlighting the importance of parameter uncertainty.
variety of methods tackling CL have been derived from a Bayesian perspective. A prominent example are prior-focused methods [17], which incorporate knowledge from past data via the prior and, in contrast to our work, aim to find a shared posterior for all data. Examples include (Online) EWC [38 78 and VCL 54]. Other methods like CN-DPM [46 use Bayes' rule for task inference on the joint p( [X, C). where is a discrete condition such as task identity. An evident downside of CN-DPM is the need for a separate generative and discriminative model per condition. More generally, such an approach requires meaningful density estimation in the input space, a requirement that is challenging for modern ML problems [64].
Other Bayesian CL approaches consider instead task specific posterior parameter distributions. Lee et al. 47 learn separate task-specific Gaussian posterior approximations which are merged into single posterior after all tasks have been seen. CBIN [49] also learns a separate Gaussian posterior approximation per task but later tries to merge similar posteriors in the induced Gaussian mixture model. Task inference is thus required and achieved via predictive uncertainty, although for a more reliable estimation all experiments consider batches of 200 samples that are assumed to belong to the same task. Tuor et al. [85] also learn a separate approximate posterior per task and use predictive uncertainty for task-boundary detection and task inference. In contrast to these approaches, we learn all task-specific posteriors via a single shared meta-model and remain agnostic to the approximate inference method being used. A conceptually related approach is MERLIN 33), which learns task-specific weight distributions by training an ensemble of models per task that is used as training set for a task-conditioned variational autoencoder. Importantly, MERLIN requires fine-tuning stage at inference, such that every drawn model is fine-tuned on stored coresets, i.e., a small set of samples withheld throughout training. By contrast, our approach learns the parameters of approximate Bayesian posterior p( [W| D(t)) per task t, and no fine-tuning of drawn models is required.
Bayesian neural networks. Because neural networks are expressive enough to fit almost any data (8) and are often deployed in an overparametrized regime, it is implausible to expect that any single solution obtained from limited data generalizes to the ground truth p( Y w, X) R p(Y X) almost everywhere on p(z)). By contrast, Bayesian statistics considers a distribution over models, explicitly handling uncertainty to acknowledge data insufficiencies. This distribution is called the posterior parameter distribution p(W D) x p(D W) p( W), which weights models based on their ability to fit the data (via the likelihood p(D W)), while considering only plausible models according to the prior p( (W) Predictions are made by marginalizing over models (for introduction see MacKay (7). Bayesian neural networks (BNN) apply this formalism to network parameters w, whereas for practical reasons hyperparameters like architecture chosen deterministically [56].
While a deterministic discriminative model can only capture aleatoric uncertainty (i.e., uncertainty intrinsic to the data p( Y X)), a Bayesian treatment allows to also capture epistemic uncertuinty by being uncertain about the model's parameters (parameter uncertainty). This proper treatment of uncertainty is of utmost importance for safitt-critical applications, where intelligent systems are expected to know what they don't know. However, due to the complexity of modelling high- dimensional distributions at the scale of modern deep learning, BNNs still face severe scalability issues 822 Here, we employ several approximations to the posterior based on variational inference [5] from prior work, ranging from simple and scalable methods with mean-field variational family like goyes-by-BBckprug (BbB, (6) to methods with complex but rich variational families like the spectral Stein gradient estimator 79. For more details see Sec. 3 and C
3

---

(A)
Weight Generntor
(b) Weight Generator
Posterior Distribution
p(W] DIo)
Main Network
p(Z) - 1)
W (w ((0)
p(2)
W- Soie (W)
(xx w)
Implicit C Expplicit
w 9 + ox
|/9 (1111 x
Task-
Conditioned
fro 19)
apprex. posterior parameters
Hypernetwork
learned parameters
Ogion ()
random variable
ML
Figure 2: Posterior meta-replay for CL. (a) The architecture consists of main network M that processes inputs x and generates predictions ý according to a set of weights w generated by a weight generator (WG). The WG is a deterministic function [wa(2, (())) that transforms a hase distribution p(Z) into a distribution over main network weights, where (()) are the parameters of the approximate posterior Qot (W). Crucially, ((r) are task-specific, and generated by a task-conditioned (TC) hypereetwor, which receives task embeddings e(*) as input. The embeddings and the parameters yb of the TC are learned continually via simple meta-replay regularizer (Eq. D (b) We refer the approximate posteriors as explicit if fwo is predefined. In Berrss-y-Backpgggg (BbB), for example, the reparametrization trick transforms Gaussian noise into weight samples. (c) More complex, implicil posterior approximations are parametrized by an auxiliary hypernetwork, which receives its task-conditioned parameters from the TC, which now plays the role of hyper-hypernetwork. The obtained posterior approximations are more flexible and can, for example, capture multi-modality.
3 Methods
We start by introducing task-conditioned hypernetworks as tool to continually learn parameters of task-specific posteriors, each which is learned using variational inference (SMC..] We then explain how the framework can be instantiated for both simple, explicit posterior approximations, and complex ones parametrized by an auxiliary network, and describe how forgetting can be mitigated through the use of a meta- regularizer. We next explain how predictive uncertainty, naturally arising from a probabilistic view of learning, can be used to infer task identity for both Posterior PosteriorReplay methods, and PriorFocused methods that use multihead output. Finally, we outline ways boost task inference.
In this section we describe our posterior meta- meta-replay framework (Fig. 2)
condicond hypernetworks. Traditionally, hypernetworks are seen as neural networks that generate the weights w of main network M processing inputs ý fM(x, w) [22] 77). Here, we consider instead hypernetwork that learn to generate O, the parameters of a distribution qo (W) over main network weights. By taking low-dimensional task embeddings e as inputs and computing @ = (*)(e()) )b), task-conditional (TC) computation is possible. Sampling is realized by trans- forming a base distribution p(Z) via weight generator (WG) fwo (z, o())), whose choice determines the family of distributions considered for the approximation (i.e., the variational family). In our framework, weights w yoco (W) are directly used for inference without requiring any fine-tuning.
Importantly, all learnable parameters are comprised in the TC system, which can be designed to have less parameters than the main network, i.c., dim( (b) + E, di)( < diim Such constraint is vital to ensure fairness when comparing different CL methods, and is enforced in all our computer vision experiments. Additional details can be found in SM C.2
Posterior-replay with explicit distributions. Different families of distributions can be realized within our framework. In the special case of a point estimate qo (W) = 6(W - -(()), the WG system can be omitted altogether as it corresponds to the identity (()) = fwa( (z, 0())). This reduces our solution to the deterministic CL. method introduced by von Oswald et al. [9]], which we refer to as TosteriorReelay-Diie
However, capturing parameter uncertainty is a key ingredient of Bayesian statistics that is necessary for more robust task inference (cf. Sec. 4.2) We thus turn first step to explicit distributions (gtz) (W), for which the WG system samples according to a predefined to finding a mean-field Gaussian approximation via the BbB algorithm (SM (SM/C.3.) C.3.1 66) In this case, ((() corresponds to the set of means and variances that define a Gaussian for each weight, which is directly generated by the TC. In the SM, we also report results for another instance of explicit distribution (cf. SM C.3.2
function. We refer as ResteyiorReplly-eee

---

Posterior-replay with implicit distributions. Since the expressivity of explicit distributions is limited, we also explore the more diverse variational family of implicit distributions [9] are parametrized by a WG that now takes the form of an auxiliary neural network, making the parameters ((t) of the approximate posterior dependent on the chosen WG architecture. This setting, referred PosteriorReplay-nue
31] These
results in a hierarchy of three networks: a TC network generates task-specific parameters O for the approximate posterior, which is defined through an arbitrary base distribution pl (Z) and the WG hypernetwork, which in turn generates weights w for a main network M that processes the actual inputs of the dataset D(O), Interestingly, the TC now plays the role of a hyper-hyyernetwork as it generates the weights of another hypernetwrrk (Fig. Dh and Fig. D
Variational inference commonly resorts to optimizing an objective consisting of a data-dependent term and prior-matching term (L(go( (W) || p( (W)) Estimating the prior-matching term when using implicit distributions is not straightforward since we do not have analytic access to the density nor the entropy of 9ocs)
resort to the spectral Stein gradient estimator (SSGE, SM C.4.2 (9).. This method is based on the insight that direct access to the log-density is not required, but only to its gradient with respect to W. Noticing that this quantity appears in Stein's ss identity, the authors consider a spectral decomposition of the term and use the Nyström method to approximate the eigenfunctions. We test alternative method for dealing with implicit distributions in the SM that is based on estimating the log-density ratio C.MII.
(W) To overcome this challenge.
As an additional challenge introduced by the use of implicit distributions, the support of goco) (W) is limited to a low-dimensional manifold when using an inflating architecture for WG, causing the prior- matching term to be ill-defined. To overcome this,
investigate the use of small noise perturbations in WG outputs (SM C.4.3). Normalizing flows (70) can also be utilized as WG architectures to gain analytic access to Late (W), albeit at the cost of architectural constraints such invertibility.
Overcoming forgetting via meta-replay. Since all learnable parameters are part of the TC system, forgetting only needs to be addressed at this meta-level. With Lusk the task-specific loss (SM Eq. 3 and (-Il--) a divergence measure between distributions, the loss for task becomes:
(1b,,, E, ((t)) = Luss ( + +B
> Digur.. (Wlll (W))
(1)
where E = the regularizer and (((',+) fre(ell.>) :*), x/²) are the parameters of the posterior approximation obtained from a checkpoint of the TC parameters before learning task t: {w²}} U (elt' ( The checkpointed meta-model allows replaying posterior distributions from the past and retaining them via divergence minimization as detailed below, hence the name posterior meta-replay. Importantly, the loss on task only depends on the corresponding dataset D(t), but knowledge transfer across tasks is possible because task-specific models are learned through a shared meta-model Since the computation required to compute this regularizer linearly scales with the number of tasks, we also explore stochastically regularizing on subset of randomly selected tasks in each update (cf. SM [.3)). and show that this does not impair performance. Notably, our PosteriorReplay method does not incur in a significant increase of runtime or memory usage F2.
|- (e(())) is the set of task embeddings up to the current task, B is the strength of
The evaluation of Eq. I requires estimates posterior approximations of past tasks. Because these are required at every loss evaluation and need to be cheap to compute, we do not consider sample-base based estimates but only estimates that directly utilize posterior parameters. This goal is easy to achieve for families of posterior approximations that possess an analytic expression for a divergence mcasure (c.g., Gaussian distributions). More specifically, for PosteriorReplay Exp, consider the forward KI., backward KL. and the 2- 2-Wasserstein distance but did not observe that the specific choice of divergence measure is crucial in practice (cf. SMC...] and Table (S14). In all other cases, approximations are required. Specifically, we resort in our experiments to the use of an L2 regularizer at the output of the TC network:
divergence measure to prevent changes learned
Σ (llftc(e(("))) ,b*))= - fte(e + P<L
Perhaps surprisingly, we observe that this crude regularization, reminiscent to the one used in von Oswald al. 91 for point estimates, does not harm performance and leads to models that exhibit
'hhiee this type of transfer is rather implicit, explicit knowledge transfer can be realized via task-specific priors (cf. SM E7
5

---

virtually no forgetting. However, we discuss in SM F.3 how this isotropic regularization could be improved given that the KL is locally approximated by a norm ||-|| induced by the Fisher information matrix F on 20 (W).
Task inference. A system with task-specific solutions requires access task identity when processing unseen samples. In our framework, this amounts to selecting the correct task embedding to condition
the TC. Although auxiliary systems can be used to infer task identity [24 91. here we exploit predictive uncertainty, assuming task identity can be inferred from the input alone. For task inference approach based on predictive uncertainty to work, the properties of the input data distribution p² (a:) need to be reflected in the uncertainty measure, i.e., uncertainty needs to be low for in-distribution data and high for out-of-distribure discriminative models only captures aleatoric uncertainty, making its overall validity debatable. Indeed, alcatoric uncertainty is only calibrated in-distribution and its OOD behavior is hard to foresee. Instead, we argue that epistemic uncertainty arising naturally in a Bayesian setting crucial for robust uncertainty-basel task inference. In our case, epistemic uncertainty stems from the fact that the posterior parameter distribution p( (W))())) in conjunction with the network architecture induces a distribution over functions. If this distribution captures a rich set of functions, diversity of predictions on OOD data can be expected even if those functions agree in-distribution (cf. Sec. 4.2 Note, however, that inducing such diverse distribution over functions is not straightforward with neural networks, and that more research is required to justify uncertainty-basee based OOD detection (II.
(OOD) data [T] Uncertainty-based task inference with deterministic
We explore two different ways to quantify uncertainty for task inference. In Ent the task leading to the lowest entropy the predictive posterior p(y [D(0; x) is selected, where p(y (D(); x) fw ply ) (WID())) d' is approximated via Monte- with samples from goco (W). This approach captures both aleatoric and epistemic uncertainty when used a probabilistic setting. In Agree the task leading to the highest agreement in predictions across models drawn from qoto (W) is selected. This approach exclusively measures epistemic uncertainty and can therefore only be estimated in a probabilistic setting. Although we generally consider task inference for individual samples, we also explore bateh-wise (BW) task inference for batches of 100 samples that are assumed to belong the same task. Such approach drastically boosts task inference simply due to a statistical accumulation effect when having above chance level task inference for single inputs. Intuitively, BW corresponds to accumulating evidence to decrease uncertainty (c.g., an agent looking at an object from multiple perspectives). Further details can be found in SM C.6. and using uncertainty for lask-boundary detection when training without explicit access task identity is explored in SM
Facilitating task inference through coresets. A key advantage of Bayesian statistics is the ability to update models as new evidence arrives. When continually learning sequence of tasks, posteriors may for example undergo a post-hoc fine-tuning on stored coresets to mitigate catastrophic forgetting of earlier tasks. Specifically, given a dataset split D(t) \ c( UCt*), one can perform final update pl (WID x p(W D(t) p(C() W) using stored coreset ((t) in conjunction with an already learned posterior approximation for pWW D(t) \ c(t) that now acts as a prior. Interestingly. access to coresets after training on all tasks can also be exploited to facilitate task inference via predictive uncertainty. Here, we explore this idea by encouraging task-specific models to produce uncertain predictions for OOD samples (i.e., coresets from other tasks), an approach that we denote CS (SM C.7). and which we store 100 inputs per task. Intriguingly, training on OOD inputs makes these become in-distribution, and therefore renders model agreement (Agree) inapplicable for task inference, which we empirically observe.
Improving prior-focused CL. We investigate ways to improve PriorFocused methods within our framework. First, we endow them with implicir posterior approximations go ()) parametrized by WG hypernetwork, an approach we refer to as PriorFocused-Imp Because the posterior is shared across tasks, no TC system is required and the parameters @ can be directly optimized via SSGE. Second, we enrich PriorFocused methods with small set of task-specific parameters that enable uncertainty-
task inference for prior-focused methods too. Specifically, the learned parameters w consist of a set of shared weights & and set of task-specific output heads with weights (5 )Z This approach is in contrast with how PriorFocused methods like Online EWC are commonly deployed in agnk-agn inference settings, where the softmax output grows as new tasks arrive (e.g., (88). The use of growing softmax causes the model class parametrized by w to change over time, and therefore violates the Bayesian assumption that the approximate posterior obtained from model class containing the ground-truth model. We show that this leads to limitations that

---

can be overcome by a multihead approach. For Online EWC, we refer to the growing softmax and multihead scenarios as E-Crowwing and EWC-multihead, respectively (cf. SM C.5.2. We also explore the prior-focused instantiation of BbB, known as VCL (cf. SM C.5.T
4 Experiments
In this section, we start by illustrating the conceptual advantage of the PosteriorReplay approach compared PriorFocused methods, as well as the importance of parameter uncertainty for robust task inference. We then explore scalability to more challenging computer vision CL benchmarks.
To assess forgetting, we provide During scores, measured directly after training each task, and Final scores, evaluated after training on all tasks. We consider two different testing scenarios: (1) either task-identity is explicitly given (TGiven) or (2) task-identity has to be inferred (TInfer), e.g. via predictive uncertainty. Unless explicitly mentioned, task inference is performed for each sample in isolation and is obtained using the Enf criterion (Tinfer-Final), Whenever the wrong task is inferred, the sample is directly considered as incorrectly classified. Supplementary results and controls are provided in DM and all experimental details can be found in SM
4.1 Simple 1D regression illustrates the pitfalls of prior-focused learning
To illustrate conceptual differences between PosteriorReplay and PriorFocused methods we study ID regression, for which the predictive posterior can be visualized. Each task-specific posterior obtained with PosteriorReplay-Exp fits the training data well (Fig. (34) and ex- hibits increasing uncertainty when leaving the in-distribution domain, as desired for success- ful task inference. Interestingly, when studying low-dimensional problems, we generally found it easier to find viable hyperparameter configura- tions for PosteriorReplay with implicil methods than with explicil ones. As we do not consider multihead for this problem, PriorFocused meth- ods have fit single posterior to all polynomi- als in a sequential manner and struggle to find a good (Fig. Bb), independent of the type of posterior approximation used. Results for other methods and analysis of the correlations and multi-modality that can be captured by implicit methods in weight space can be found in SM
(a)
(h)
-3
0
3
Figure 3: PriorFocused methods struggle to learn three ID polynomial regression tasks. (a) Differ- ent colors represent the task-specific posterior ap- proximations within the final PosteriorReplay-Ei model. For unseen inputs x the posterior with the lowest predictive uncertainty is chosen to make predictions. (b) Predictive posterior using the nal approximation of p( W ((1:3)), obtained via PriorFocused-Imgy Imp. Shaded areas represent stan- dard deviation, and black dots training samples.
Because we use mean squared error (MSE) loss, the likelihood is a Gaussian with fixed variance (SMC...J and all z-dependent uncertainty originates from parameter uncertainty. We next consider classification problems where both epistemic and aleatoric uncertainty can be explicitly modelled.
4.2 Maintaining parameter uncertainty crucial for robust task inference
To investigate the importance of parame- ter uncertainty, we consider a 2D classifi- cation problem for which uncertainty be visualized in- and
Table 1:
2D mode classification accuracies (Mean + standard error of the mean (SEM) in %, n 10). Task identity is inferred via predictive uncertainty using an entropy (Ent) or model agreement (Agree) criterion. PR denotes PosteriorReplay.
Classification tasks are of special interest as it is possible to model arbitrary input- dependent discrete distributions, e.g. via softmax at the outputs. This surprisingly often results in meaningful OOD perfor- mance in nigh-dimensional benchmarks without any treatment of parameter uncer-
Final Acc
TGiven
Tinfer (Ent)
Tinfer (Agree)
PR-Dirac
99.78 + 0.21
44.90 +: 5.74
N/A
PR-Exp
100.0 0.00
81.07 + 6.78
90.02 + 3.57
PR-Imp
100.0 + 0.00
100.0 + 0.00
100.0 + 0.00
tainty 82
Source code for all experiments (including all baselines) is available at: (ttps://githin chrhenning/ chrremming/posteriien pttp:///gpppptrrrroorrppae.//
7

---

Here, we consider a Gaussian mixture of two modes per task, each mode being a different class (Fig. 4b. Tinfer-Final (Agree), which is indicative of the im- portance of epistemic uncertainty for OOD detection, is the most robust measure for task infer- ence in this experiment (Table T}. PosteriorReplay-Dizee Dirac, which does not incorporate epistemic certainty, performs poorly. Fi- nally, implicit methods maintain an advantage over explicit ones, presumably due to the increased flexibility in modelling the posterior.
4
0.6 (b)
Mikk
back
#20
0
4 0
4
Figure 4: Parameter uncertainty is crucial for robust task infer- ence. (a) Density of input distribution p(x) across tasks. Dots represent training points, colors task-affiliation and lines deci- sion boundaries for each of the three consecutively learned 2D binary classification tasks. (b) Entropy of predictive distribu- tion induced by the approximate posterior of task 2 learned via SeteriooRRplay-Dirrcee (c) Same as (b) for PosteriorReplay-
To better understand why differences between methods arise provide uncertainty maps over the input space (Fig. 4 Consistent with the observed low task inference accuracy, PosteriorReplay Rirra-Draaa displays arbitrary uncertainty away from the training data (Fig. 4b). By contrast, PosteriorReplay-mme (Fig. [ approaches the desired behavior of displaying high uncertainty away from the training data of the corresponding task. We provide detailed analysis in SM
4.3 Multiple factors affect uncertainty-basse task inference accuracy
To investigate the factors that affect uncertainty- task inference, we next consider SplitMNIST 196. a popular variant of the MNIST dataset, adapted to CL by splitting the ten digit classes into five binary classification tasks. The results can be found in Table
While all methods successfully prevent forget- ting (i.e., Final scores are maxed-out and close to the During accuracies, D.3) and achieve acceptable Final accuracies when task identity is provided, large differences can be observed when the task needs to be inferred. Methods with task specific solutions outperform by large margin PriorFocused approaches such as Online EWC, whose performance substantially improves when using uncertainty-bar task in- ference through multihead. Despite superior performance of PosteriorReplay approaches, gap in performance between task-inferred and task-given scenarios remains. However, train- ing separate posteriors that are not embedded in a hypernetwork "eeparatePosterions similar results, showing that task inference limi- tations are not linked to our solution. These lim- itations can be surmounted by inferring the task on batches rather than single samples (BW) or by using coresets encourage high uncertainty for OOD data (CS), which leads to performances comparable to generative-replay methods which explicitly capture p(X) (i.e., HNET+R and DGR).
Table 2: Accuracies of SplitMNIST experiments 10) after Icarning all
(Mean + SEM in %, tasks when task identity is provided "GGiven-Fina and when it needs to be inferred (I'nfer-Final, based on the Eni criterion if explicit task-inference is required). Results are shown for MLP with two hidden layers of 400 neurons (MLP-400,400), an MLP- -.P,000,1000 or a Lenet PR denotes Poste- riorReplay and SP SeparatePosterion
TGiven- Final
Tinfer-Final
EWC-growing [87
N/A
19.96 + 0.07
EWC-multihead
96.40 0.62
47.67 : 1.52
VCL-multihead
96.45 - 0.13
58.84 1 0.64
leads to
PR-Dirac PR-Exp
99.65 0.01
70.88 1 0.61
99.72 0.02
71.73 1 0.87
PR-Imp
99.77 1 0.01
71.91 1 0.79
SP-Dirac
99.77 0.01
70.39 1 0.27
SP-Exp
99.81 +: 0.00
68.40 1 0.23
PR- PR-Exp-BW BW PR-Exp-CS
99.72 + 0.02
99.72 + 0.02
98.50 + 0.09
90.83 + 0.24
DGR
87
N/A
91.79 + 0.32 95.30 + 0.13
HNET+R 91
N/A
To better understand the factors that influence task inference, we consider variety of approxi- mate inference methods and architectures. Since epistemic uncertainty seems to play a vital role for task inference, the approximate inference method will likely affect Tinfer performance (e.g., Exp vs. Imp: in supplementary results). In addition, because diversity in function space enables uncertainty- OOD detection and because different architectures induce different priors in func- tion space [93], one can expect that prior and architecture play a key role as well, which we observe
PR-Dirac PR-Exp-
99.72 1 0.01
63.41 1 1.54
99.75 0.01
70.07 + 0.56
PR-Dirac" PR-Exp"
99.87
0.04
72.33 + 2.75
99.20 = 0.67
74.09 + 1.38
8

---

by comparing the Thfer performance for different architectures. Additional SplitMNIST results can be found SM3] and results showing scalability to sequences of up to 100 PermutedMNIST tasks can be found in SM D.4 and D.5
4.4 PosteriorReplay scales to natural image datasets
While BNNs are advocated because of their theoretical promises, practitioners are often put off by scalability issues. Here we show that our approach scales to natural images by considering [39], dataset consisting of five tasks with two classes each. Results obtained with a Resnet-32 [23] (Table
SplitCIFAR-10 10
3) show performance gains in the task-agnostic setting compared to recent methods like EBM [50] and to the PriorFocused method VCL.
Furthermore, our results reveal considerable improvements through the incorporation of epistemic uncertainty, as shown by differences between PosteriorReplay-E. Exp and DosteriorReplayDDine
Notably, PosteriorReplay-EB Exp-BBW solves CIFAR-10 10 with a perfor- mance comparable to classifier trained on all data at once, with the caveat that successive unseen samples are assumed to belong to the same task. In contrast to low- dimensional problems, the implicit method PosteriorReellay -mp does not exhibit competitive advan- tage, as it appears to suffer from scalability issues. Other baselines and results for a WRN-28- 10 can be found in SM D.6 and results showing that our framework scales to the SplicCIFAR- 100 benchmark can be found in D.7.
Table 3: Accuracies of SplitCIFAR- 10 experiments (Mean +
SEM in %, = 10). Tinfe-Final is based on the Ent criterion if explicit task-inference required and PR denotes Posterior- Replay.
TGiven-During
V'Given-Final
Tinfer-Final
VCL-multihead
95.78 i 0.09
61.09 0.54
15.97 + 1.91
PR-Dirac
94.59 0.10
93.77 + 0.31
54.83 + 0.79
PR-Exp
95.59 + 0.08
95.43 + 0.11
61.90 + 0.66
PR-Imp
94.25 + 0.07
92.83 + 0.16
51.95 + 0.53
PR- TR--Ex-BWW PR-Ex RR-Ex--CS
95.59 + 0.08
95.43 + 0.11
92.94 + 1.04
95.15 + 0.11
92.48 + 0.13
64.76 + 0.34
EBM 50
N/A
N/A
38.84 + 1.08
5 Discussion
In this study we propose posterior meta-replay, a framework for continually learning task-specific posterior approximations within a single shared meta-model. In contrast to prior-focused methods based on recursive Bayesian update, our approach does not directly seek trade-off solutions across tasks. This results more flexibility for learning new tasks but introduces the need to know task identity when processing unseen inputs.
Task Inferenee. Probabilistic inference on task identity can be achieved by additionally considering inputs and task embeddings as random variables, strategy that would require task condiiiiio gen- erative models with tractable density [46]. However, learning generative models on high-dimensional data is challenging problem and, even if tractable densities are accessible, these do not currently reflect the underlying data-generative process 64 To sidestep these limitations, we study the use of predictive uncertainty for task inference 191] and show that an entropy-based criterion works best for both deterministic and Bayesian models. Nevertheless, we highlight that proper task in- ference requires epistemic uncertainty (e.g., measured in terms of model disagreement). Indeed, in-distribution samples with high aleatoric uncertainty can lead to high predictive entropy, causing them to be misclassified as OOD. This does not pose a problem in highly-curated ML datasets where samples with high aleatoric uncertainty are excluded [62], but drastically limits the applicability of entropy- basedd uncertainty estimation in more practical scenarios. For these reasons, we advocate for the use of Bayesian models whose epistemic uncertainty can induce diversity in function space for OOD inputs and enable more robust task inference
"Interestingly, concurrent study by de Ven al. 89 successfully trains conditindi generative models, indicating that this approach could nevertheless feasible tackle task inference.
'Note, while also models with deterministic parameters may be well suited for OOD detection (e.g.. Lakshmi- narayanan et al. (43) that utilizes deterministic distance preserving input- souut-t--iil -idden mapping), these solutions

---

Limitations. Compared to methods performing deterministic inference, the Bayesian model av- erage incurs in significant computational overhead. This overhead is reinforced when performing uncertainty-based task inference, since each predictive posterior needs to be evaluated in parallel. Moreover, despite strong performance gains compared to prior-focused approaches, we observe general limitations of such task inference procedure. These could be overcome once better un- derstanding of how epistemic uncertainty influences OOD behavior in neural networks is available. In addition, our work builds on algorithms to perform variational inference, and is therefore only applicable to problems where these can be successfully deployed. Finally, all our experiments consist of set of clearly defined tasks within which i.i.d samples available. Although this scenario is in line with most existing CL literature, it might be of limited relevance for practical CL problems, and focus established benchmarks adhering to these constraints could therefore misguide research on this area. Indeed, a more natural CL. problem might arise from the need to online learn from a stream of autocorrelated samples. In this context, it is important to note that unlike non-Bayesian CL methods, our approach can utilize any type of online prior-focused method (such as FOO-VP VB (97) to also learn within tasks in non-i.i.d. manner. Therefore, long some coarse split into lasks is meaningful, such hierarchical approach holds great promise. However, it should be noted that any progress towards learning from non-i.idd. data opens the door to training algorithms from raw, uncurated datasets, and could therefore counter some of the efforts that are currently done to mitigate algorithmic bias.
Conclusion. Taken together, our work shows that it is possible to continually learn an approximate posterior per task without increased parameter budget, and that task-agnostic inference can be achieved via predictive uncertainty to obtain a Bayesian CL approach that is scalable to real-world data. Since forgetting not a prevalent issue in our experiments and task inference limitations not linked to our CL solution, progress in the field of incertainty-base OOD detection will automatically translate into further improvements of our method.
Acknowledgements
This work was supported by the Swiss National Science Foundation (B.FG. TRSS15-173722 and 315230_ 189251), Science Center (B.F.G, C17-18, J. v. O. P18-03). J.S. was supported by an Ambizione grant (PZ00P3 (PZ00P3_186027) from the Swiss National Science Foundation. We grateful for discussions with Harald Dermutz, Simone Carlo Surace and Jean-Pascal Pfister. We also would like to thank Sebastian Farquhar for discussions on Radial posteriors and for proofreading our implementation of this method.
315230_189251). ETH project funding (B.F.G. ETH-20 19-01) and funding from the Swiss Data
References
[1] T. Adel, H. Zhao, and R. E. Turner. Continual learning with adaptive weights (claw), In International Conference Learning Representations, 2020.
[2] R. Aljundi, E. Belilovsky, T. Tuytelaars, L. Charlin, M. Caccia, M. Lin, and L. Page-Caccia. Online Continual Learning with Maximal Interfered Retrieval. Advances in Neural Information
Processing Systems, 32, 2019.
[3] M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. In International Conference Learning Representations, 2017.
[4] O. Bachem, M. Lucic, and A. Krause. Coresets for nonparametric estimation the case of dp-means. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 209-217, Lille. France, 07-09 Jul 2015. PMLR.
[5] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-88) 877, 2017. doi: 10.1080/ 01621459, 2017.1285773.
[6] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on
are limited by the fact that the Bayesian recursive update is not applicable and therefore parameters cannot be updated in sound way when learning continually.
10

---

Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613-1622 Lille. France, 07-09 Jul 2015. PMLR.
[7] Z. Borsos, M. Mutny, and A. Krause. Coresets via bilevel optimization for continual learning and streaming. In H. Larochelle, M. Ranzato, R. Hadsell, M. F, Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14879-14899 Curran Associates, Inc., 2020.
[8] O. Chang, I.. Flokas, and H. Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2020.
[9] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ran- zato. Continual learning with tiny episodic memories. arXiv, 2019.
[10] Y. Chen, T. Diethe, and Flach. Discriminative representation loss (drl): Connecting deep metric learning to continual learning. arXiv, 2020.
[11] F. D' Angelo and C. Henning. Uncertainty-basee out-of-distribun detection requires suitable function space priors. arXiv, 2021.
[12] M. De Lange and T. Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. arXiv, 2020.
[13] S. Depeweg, J.- Hernandez-LLoato F. Doshi- Velez, and S. Udluft. uncertainty in bayesian deep learning for efficient and risk-sensitive learning, In International Conference on Machine Learning, pages 1184-1193, PMLR, 2018.
Decomposition of
[14] V. Dwaracherla, X. Lu, M. Ibrahimi, Osband, Z. Wen, and B. Roy. Hypermodels for exploration. In International Conference on Learning Representations, 2020.
[15] B. Ehret, C. Henning, M. R. Cervera, A. Meulemans, J. von Oswald, and B. F. Grewe. Continual learning in recurrent neural networks. In International Conference on Learning Representations, 2021.
[16] S. Farquhar and Y. Gal. Towards Robust Evaluations of Continual Learning. Lifelong Learning:
A Reinforcement Learning Approach Workshop at ICML, 2018.
[17] S. Farquhar and Y. Gal. A unifying bayesian view of continual learning. Bayesian Deep Learning Workshop at NeurIPS, 2018.
[18] Farquhar, M. Osborne, and Y. Gal. Radial bayesian neural networks: Beyond discrete sup- port in large-scale bayesian deep learning. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Staristics, volume 108 of Proceedings of Machine Learning Research, pages 1352-1362. PMLR, 26-28 Aug 2020.
[19] I. Goodfellow, J. Pouget-Abadie. M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems
27, pages 1672-26800 Curran Associates, Inc., 2014.
[20] I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv, 2013.
[21] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012.
[22] D. Ha, A. Dai, and Q Le. Hypernetorrss. In International Conference on Learning Represen- lations, 2017.
[23] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778. 2016.
[24] X. He, J. Sygnowski, A. Galashov, A. A. Rusu, Y. W. Teh, and R. Pascanu. Task agnostic continual learning via meta learning. arXiv, 2019.
[25] M. Hein, M. Andriushchenko, and J. Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how mitigate the problem. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 41-50, 2019.
[26) D. Hendrycks and K. Gimpel. A Baseline for Detecting Misclassified and Examples in Neural Networks, arXiv, 2016.

---

[27] C. Henning, J. von Oswald, J. Sacramento, S. C. Surace, J.-P. Pfister, and B. F. Grewe. Ap- proximating the predictive distribution via adversarially hypernetworks. In NewrIPS Bayesian Deep Learning Workshop, 2018.
[28] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv, 2015.
[29] S. Hochreiter and J. Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, Jan. 1997.
[30] T. Hu, Z. Chen, H. Sun, J. Bai, M. Ye, and G. Cheng. Stein neural sampler. arXiv, 2018.
[31] Huszár. Variational inference using implicit distributions. arXiv, 2017.
[32] F. Huszár. Note the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, 1 115(1 (15(111):244662222
Mar. 2018.
[33] J. J and V. N Balasubramanian. Meta-consolidation for continual learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14374-14386 Curran Associates, Inc., 2020.
[34] T. Karaletsos and T. D. Bui. Hierarchical gaussian process priors for bayesian neural network weights. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17141-17152. Curran Associates, Inc., 2020.
[35] T. Karaletsos. P. Dayan, and Z. Ghahramani. Probabilistic meta-reppresentations of neural networks. arXiv, 2018.
[36] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[37] D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 2575-2583, Curran Associates, Inc., 2015.
[38] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 1 114((3)3351-3352, Mar. 2017.
J. Quan, T. Ramalho, A.
[39] A. Krizhevsky and Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[40] D. Krueger, C.-W. Huang, R. Islam, R. Turner, A. Lacoste, and A. Courville. Bayesian Hypernetworks, arXiv, 2017.
[41] R. Kurle, B. Cseke, A. Klushyn, P. van der Smagt, and S. Günnemann. Continual learning with bayesian neural networks for non-stationary data. In International Conference on Learning Representations, 2020.
[42] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 6402-6413. Curran Associates, Inc., 2017.
[43] B. Lakshminarayanan, D. Tran, J. Liu, S. Padhy, T. Bedrax- Weiss, and Z. Lin. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In Advances in Neurul Information Processing Systems 33, 2020.
[44] Y. LeCun, L. Bottou. Y. Bengio, and P. Haffner. Gradient- based learning applied document recognition. Proceedings of the IEEE,
1998.
[45] K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations, 2018.
[46] S. Lee, J. Ha, D. Zhang, and G. Kim. A neural dirichlet process mixture model for task-free continual learning. In International Conference on Learning Representations, 2020.
[47] S.-W. Lee, J.-H. Kim, Jun, J.-W. Ha, and B.-T. Zhang. Overcoming catastrophic forgetting by incremental moment matching. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 4652-4662. Curran Associates, Inc., 2017.
12

---

[48] A. Li, A. J. Boyd, P. Smyth, and S. Mandt. Variational beam search for novelty detection. In Third Symposium on Advances in Approximate Bayesian Inference. 2021.
[49] H. Li, P. Barnaghi, S. Enshaeifar, and F. Ganz. Continual learning using bayesian neural networks. JEEE Transactions on Neural Networks and Learning Systems, 2020.
[50] S. Li, Y. Du, G. M. van de Ven, A. Torralba, and I. Mordatch. Energy-based models for continual learning. arXiv, 2020.
[51] Y. Li and R. E. Turner. Gradient estimators for implicit models. In International Conference on Learning Representationss 2018.
[52] L.-J. Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning. 8(3-4):293-322 1992.
[53] Q Liu, J. Lee, and M. Jordan. A kernelized stein discrepancy for goodness-of-fr tests. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 276-284, New York, New York, USA, 20-2 Jun 2016. PMLR.
[54] N. Loo, S. Swaroop, and R. E. Turner. Generalized variational continual learning, In Interna- tional Conference on Learning Representations, 2021.
[55] C. Louizos and M. Welling. Multiplicative Normalizing Flows for Variational Bayesian Neural Networks, In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pages 2218-2227. JMLR.org, 2017.
[56] J. J. MacKay, A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3):448-472, 1992.
[57] D. J. MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
[58] A. Malinin, B. Mlodozeniec, and M. Gales. Ensemble distribution distillation. In International Conference on Learning Representations, 2020.
[59] J. Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1-76, 2020.
[60] M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer. Class- incremental learning: survey and performance evaluation on image classification. arXiv, 2020.
[61] L. Mescheder, S. Nowozin, and A. Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2391-2400, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
[62] Mukhoti, A. Kirsch, J. van Amersfoort, P. H. Torr, and Y. Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv, 2021.
[63] M. Mundt, Y. W. Hong, Pliushch, and V. Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. arXiv, 2020.
[64] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminaraaaaann Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019.
[65] C. V. Nguyen, Li, T. D. Bui, and R. E. Turner. Variational continual learning, In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
[66] F. Nielsen. Cramér-rao lower bound and information geometry. In Connected at Infinity II, pages 18- 37. Springer, 2013.
[67] S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 271-279. Curran Associates, Inc., 2016.
[68] J. V. Oswald, S. Kobayashi, J. Sacramento, A. Meulemans, C. Henning, and B. F. Grewe. Neural networks with late-phase weights. In International Conference on Learning Representations, 2021.
13

---

[69] P. Pan, S. Swaroop, A. Immer, R. Eschenhagen, R. Turner, and M. E. E. Khan. Continual deep learning by functional regularisation of memorable past. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4453 15534466. Curran Associates, Inc., 2020.
[70] G. Papamakarios, Nalisnick, D. J. Rezende. S. Mohamed, and B. Lakshminarayanan. Nor- malizing flows for probabilistic modeling and inference. arXiv, 2019.
[71] G. I. Parisi, R. Kemker, J. I.. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11:54-77, May 2019. ISSN 0893-6080. doi: 10. 10016/..cunet.2019.00
[72] R. Pascanu and Y. Bengio. Revisiting natural gradient for deep networks. arXiv, 2013.
[73] N. Pawlowski, A. Brock, M. C. Lee, M. Rajchl, and B. Glocker. Implicit weight uncertainty in neural networks. arXiv, 2017.
[74] A. Prabhu, P. H. Torr, and P. K. Dokania. Gdumb: A simple approach that questions our progress in continual learning. European Conference on Computer Vision, pages 524-540. Springer, 2020.
[75] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010 2017.
[76] H. Ritter, A. Botev, and D. Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 3738-3748. Curran Associates, Inc., 2018
[77] J. Schmidhuber. Learning to Control Besth Memories: An Alternative to Dynamic Recurrent Networks. Neural Comput., 4(1):131-139, Jan 1992. ISSN 0899-76677 doi: 10.1162/neco. 1993.4.1.131.
[78] J. Schwarz, W. Czarnecki, J. Luketina, A. Trabska-Barwinske Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In J. Dy and A. Krause, editors. Proceedings of the 35th International Conference Machine Learning, vol- ume 80 of Proceedings of Machine Learning Research, pages 3328-4537 Stockholmssmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
[79] J. Shi, S. Sun, and J. Zhu. A spectral approach to gradient estimation for implicit distributions. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Ma- chine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4644 4653, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
[80] H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual Learning with Deep Generative Replay. In
I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 2990-2999. Curran Associates, Inc., 2017.
[81] L. Smith and Y. Gal. Understanding Measures of Uncertainty for Adversarial Example Detection. arXiv, 2018
[82] Snoek, Y. Ovadia, E. Fertig, B. Lakshminarayanan, S. Nowozin, D. Sculley, Dillon, J. Ren, and Z. Nado. Can you trust your model's 's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pages 13969-13988 2019.
[83] S. Swaroop, C. V. Nguyen, T. D.D.Bu.. Bui, and R. E. Turner. Improving and understanding variational continual learning. Continual Learning Workshop at NeurIPS, 2018.
[84] M. K. Titsias, J. Schwarz, A. G. de G. Matthews, R. Pascanu, and Y. W. Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations, 2020.
[85] T. Tuor, S. Wang, and K. Leung. Continual learning without knowing task identities: Do simple models work? OpenReview, 2021.
[86] G. M. van de Ven and A. S. Tolias. Generative replay with feedback connections as a general strategy for continual learning. arXiv, 2018
14

---

[87] G. M. van de Ven and A. S. Tolias. Three scenarios for continual learning. arXiv, Apr 2019.
[88] G. M. van de Ven, H. T. Siegelmann, and A. S. Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):1-14, 2020.
[89] G. M. van de Ven, Z. Li, and A. S. Tolias. Class-incremental learning with generative classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3611-3620, June 2021.
[90] V. K. Verma, K. J. Liang, N. Mehta, P. Rai, and L. Carin. Efficient feature transformations for discriminative and generative continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13865-13875, 2021.
[91] J. von Oswald, C. Henning, J. Sacramento, and B. F. Grewe. Continual learning with hypernet- works. In International Conference Learning Representations, 2020.
[92] F. Wenzel, K. Roth, B. Veeling, J. Swiatkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin. How good is the Bayes posterior in deep neural networks really? H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10248-10259. PMLR, 13-18 Jul 2020.
[93] A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In Advances Neural Information Processing Systems, 2020.
[94] M. Wortsman, V. Ramanujan, R. Liu, A. Kembhavi, M. Rastegari, Yosinski, and A. Farhadi. Supermasks in superposition. In Advances in Neural Information Processing Systems, 2020.
[95] S. Zagoruyko and N. Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, 2016.
[96] F. Zenke, B. Poole, and S. Ganguli. Continual Learning Through Synaptic Intelligence. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pages 3987-3995. JMLR.org, 2017.
[97] C. Zeno, I. Golan, E. Hoffer, and D. Soudry. Task-agnostic continual learning using online variational bayes with fixed-point updates. Neural Computation, (5(11):313-3311 2021.
[98] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
15


Reviews:
[{"review_id": "tq2Bw6y7z06", "reviewer": "Reviewer_KFBY", "summary": "The paper proposes to tackle task-agnostic continual learning from a Bayesian deep learning perspective. In contrast to previous work along these lines the paper proposes to parameterize the posterior (over network weights) using a mixture distribution with one mixture component per task (seen so far). Three main options are explored - a point estimate (i.e. a single set of network weights) per task, a parametric distribution (Gaussian w. diagonal covariance) per task, and an “implicit” distribution parameterized by a neural network (using the reparametrization trick essentially). All three options can be unified/generalized by learning a weight-generator which, when conditioned on the task, produces a distribution of weights for the network that solves the task (e.g. the classifier). Importantly, the parameters of the weight-generator are produced by another neural network (the hyper-network) whose input is conditioned on the task. The question is how this task-conditioning is obtained: the paper proposes to automatically infer the task via the (epistemic) uncertainty modeled by the distribution of functions (classifiers) defined by the task-conditioned posterior over parameters (produced by the hyper-network). Experiments on synthetic data and continual learning versions of MNIST and CIFAR-10 are shown - with lots of ablations and control experiments. Results look promising, though they hint at some scalability issues for implicit distributions (an issue commonly observed in the Bayesian Deep Learning literature), as well as issues with reliable task inference via epistemic uncertainty on OOD data (another important open problem whose solution is beyond the scope of this paper). Limitations and conclusions are well discussed and related literature if well summarized.\n\n\n**Contributions:**\n\n1) Extension of a previously published approach to Bayesian CL via hyper-networks. The main novelty is to allow for the hyper-networks to parameterize distributions over networks via the weight-generator (including the earlier work as a special case - the MAP case with point-estimates for the posterior). The necessity of this extension is well motivated and the benefits are demonstrated nicely on synthetic data and result in significant performance gains on the image classification tasks. Significance: a main drawback of a SOTA method was identified and solved in a well-motivated and satisfactory manner (particularly from a Bayesian perspective). While similar approaches can be found in the literature (as discussed in the paper), the combination with automatic task inference is novel, and crucial relies on this innovation. I think the method will receive a fair amount of attention in the Bayesian CL community.\n\n2) Automatic task inference via the epistemic uncertainty on OOD data given by the Bayesian ensemble (parameterized by the hyper-network). While previous CL methods have proposed to use task inference - the combination with contribution 1) allows for a potentially powerful and principled solution to the problem. Benefits (and some remaining issues) are shown in the results. Significance: the approach is very reasonable from a theoretical/principled point of view. The combination of contributions 1) and 2) is perhaps one of the most promising advantages of Bayesian CL. The particular way this is executed in the paper is novel and original (to the best of my knowledge), and it is well motivated and critically analysed in the paper. The results obtained here will be important for the Bayesian CL community, and allow for a clear separation of the task identification problem such that improvements in reliable modeling of epistemic uncertainty (which is an important topic outside CL) will directly contribute to improvements of Bayesian CL.\n\n3) Solving the problem of catastrophic forgetting and task interference with a regularizer (posterior meta-replay) that acts on the level of the hyper-network (parametrization of the posterior over classifier weights). Significance: solving the problem is crucial to successful CL - the approach taken here is sound and empirical results underline the success of the regularizer. This is an interesting result for the Bayesian CL community.\n\n4) Well-written and comprehensive discussion of the two main innovations, including a range of specific implementations for each. These variants are discussed, related with the current literature and evaluated in a large number of experiments. Results are critically evaluated and limitations are nicely discussed. Significance: given how well the paper is written and executed, and the wealth of experimental results, ablations and control experiments, I think the paper has the potential to become a seminal next step in Bayesian CL. My only slight worry is that the paper certainly pushes against the constraints of a conference-format paper, with many interesting parts of the paper pushed to the appendix, and a quite high information density in the main text.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality, Quality, Clarity, Correctness:**\nTo the best of my knowledge, the precise method proposed in the paper is novel and original. There are some highly related methods in the literature, which are discussed in the paper and the innovations/distinctions are made very clear. The method is well motivated and aligns well with a principled Bayesian CL approach. The paper is very well written, though the density of material in the main text is very high and the supplementary material is extensive. The main innovations are well motivated and these motivations are supported with illustrations and experiments on synthetic data. Rather than simply aiming to “beat some benchmark numbers” the paper conducts a very differentiated analysis through a set of control experiments and ablations. The main results of this are promising and in favor of the method. Limitations and caveats are well discussed in the paper.\n\n**Verdict:**\nTo me, this is a well written paper, with clearly articulated main ideas, which are critically analyzed and discussed. The main problem (continual learning) is timely and important, and a Bayesian solution is sound and promising - the focus in the paper is on sequences of image classification tasks on the same data-set, where i.i.d. sampling is possible within each sub-task. This is an important limitation, but is clearly pointed out in the paper, and helps to make the focus of the work clear. Perhaps the main value of the paper is to provide a framework with various choices for parameterizing the weight generator, implementing the task inference, etc. Future improvements for these individual parts can easily be incorporated, and I would expect to see a fair number of follow-up work in this direction (e.g. normalizing-flow based implicit distributions). Given my current understanding of the literature I am clearly in favor of accepting the paper. I have some small comments and one concrete suggestion for improving estimation of epistemic uncertainty. I am, of course, happy to reconsider my verdict based on the other reviews and the authors’ response. I am not a top expert on the Bayesian CL literature (but very familiar with Bayesian DL and uncertainty estimation), hence my confidence rating.\n\n**Pros:**\n\n * Principled approach, well embedded in the current literature and SOTA methods for Bayesian CL and Bayesian neural networks.\n * Well explained and illustrated motivations. Good explanation of the individual parts of the method, what role each part plays, and what reasoning went into the choices made for each part (and potential alternatives). This allows future work to easily connect/improve by focusing on individual parts.\n * Promising results, backed up and critically analyzed through a range of control experiments, comparisons and ablations.\n * Great discussion of current limitations (and potential societal impacts), relations to current literature made very clear.\n\n**Cons:**\n\n * Paper at the limit of the conference format - main paper quite dense, loads of interesting discussion, experiments, etc. pushed to appendix (which is itself quite extensive)\n * No results beyond CIFAR-10, which is a common limitation in Bayesian DL\n * Some issues with reliable estimation of epistemic uncertainty and scalability of variational inference for the implicit parametrization remain (though this is clearly shown and discussed, and these are two important unsolved problems in the wider literature) \n\n**Improvements:**\n\n1) Improvement to estimate epistemic uncertainty via the entropy (line 233-236). Ent (entropy over predictive uncertainty) captures epistemic and aleatoric uncertainty (which is undesirable). The two factors can be teased apart fairly easily by a scheme similar to Eq. (3) in [1]. The main idea is to take the difference between the entropy of the average prediction and the average entropy of the individual predictions, where the average is taken w.r.t. the MC samples from q_\\theta(W)). For high aleatoric uncertainty but no epistemic uncertainty both quantities should be equal and cancel out by subtracting. High epistemic uncertainty is quantified with a strictly positive quantity (a mutual info term, similar to an entropic term).\n\n[1] Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning, Depeweg et al. 2018.\n\n2) Experiments beyond CIFAR-10 would be nice (though I personally don’t regard them as crucial for publication).\n\n3) The main text is fairly dense (and given this density it is very well written). The method consists of a number of components (with various choices per component), which further increases the complexity of the text (again, given this complexity I think the text is very well written). Additionally, many results and some discussion does not fit into the main paper at all, leading to a fairly lengthy appendix. Overall I think the paper would have benefited from a publication format that allows for longer main text. Given the constraints of a conference format I think the authors did a great job, and I certainly think that the work is an excellent fit for NeurIPS. Just wanted to mention this here for consideration for future publications.\n\n\n**Comments:**\n\n1) Potential drawback that should be discussed: for some tasks a lot of the task-specific posteriors might look very similar - having a single joint posterior in this case could lead to efficiency gains in terms of learning and transfer, compared to having a separate posterior per task. These gains might be partially harnessable through the hyper-network (which, after all, is shared across tasks) but it is unclear to which degree this would happen.\n\n2) Which inference metric was used for identifying the task in 4.3 - Ent or Agreement? Perhaps using BALD (as suggested in the minor improvement above) leads to improvements (particularly over Ent).\n\n3) I found Fig. 2 very helpful :)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are nicely discussed in a separate section, including societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "tMWJ2FIBOG0", "reviewer": "Reviewer_mFdX", "summary": "This submission proposes a continual learning method based on a shared hypernetwork that parameterises a distribution over the network parameters for each task given a learned task embedding. It presents this as a Bayesian method, which is **not correct** as the objective function does not minimize a divergence from the posterior. Practically, the method is mostly **incremental** over (von Oswald et al., 2020), replacing the weight outputs of the hypernetwork with the parameters of a distribution over these. Finally, the paper omits large parts of the literature on continual Bayesian deep learning, displaying a disappointing **lack of scholarship**. Overall, this is a **clear rejection**.\n\nReference:\nJ. von Oswald et al. Continual learning with hypernetworks. In ICLR 2020.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Technical correctness**\n\nThe paper argues that it approximates the posterior for each task. Considering that the objective introduces an ad-hoc term to prevent forgetting, this assertion is not correct. Optimizing the parameters of a distribution w.r.t. the variational lower bound is a principled approximation as this is equivalent to minimizing the KL divergence between the approximate and true posterior, hence we get the exact result if the variational family is sufficiently expressive. Optimizing a distribution over the parameters does not automatically constitute a Bayesian method, even if the variational lower bound appears in the objective function, as the addition of ad-hoc terms to the objective breaks the property of minimizing the deviation from the posterior. This is exacerbated here by changing the approximate posterior on a task when further training on subsequent tasks. I'd be curious if a similar objective as the one used in this work could be derived from a hierarchical probabilistic model, however there is no discussion pointing towards this unfortunately.\n\nThe paper further proposes to infer the task identity based on the predictive uncertainty of each task-specific weight distribution. This is referred to as \"principled\" without further explanation. To me, a principled approach would involve modelling the task identity as part of the probabilistic model, approximating the posterior given a data point and then accordingly marginalizing(!) over all task-specific posteriors for the final predictions. The paper does not justify its approach theoretically. Semantics aside, the major drawback of Bayesian neural networks is the computational cost of having to calculate multiple forward passes. The proposed approach further aggravates this by requiring multiple forward passes  _for each task_ observed so far, which is entirely impractical considering that ensembling of a single model is already prohibitive in many settings. The computational cost should at least be discussed and not swept under the rug (there are some runtime results in the appendix, but those appear to be for the setting with given task identities at test time).\n\n**Originality**\n\nThe core of the method is a modification of [1] (referred to as \"PR-Dirac\" in this paper), which parameterizes the network parameters via a meta-network. PR-Exp instead outputs a per-parameter mean and a variance, samples the parameters from the corresponding factorized Gaussian and regularizes with a divergence measure towards previous distributions instead of a quadratic penalty, both of which are a rather incremental change over [1]. More interestingly, PR-Imp outputs the parameters of yet another hypernetwork, which in turn samples the parameters of the network of interest based on a noise input. However, this seems to introduce little additional difficulty and judging from the paper can be achieved with similar techniques as the original hypernetwork in [1]. Further, hierarchical models for neural network weights have been considered in various works on Bayesian neural networks, see also the scholarship section.\n\n**Clarity**\n\nOverall the paper is reasonably clear in its description of the method. Key variables are not defined unambiguously in some places, e.g. $\\mathcal{L}_{task}$ is only specified in the appendix. Also, which divergence was used for PR-Exp in eq. (1)? The text insinuates the KL divergence, but how did you choose between forward and reverse KL? The equation implies that you're requiring the old distribution to be covered by the new one, but that seems like it would allow for parameter settings that perform poorly on the old data?\nFinally, the pervasive use of custom acronyms makes the paper rather cumbersome to read.\n\n**Significance**\n\nThe methodological advances over [1] are incremental. The proposed method for test-time task inference is prohibitively expensive and appears to only work on MNIST, placing the significance of this work at the low end of the spectrum.\n\n**Scholarship**\n\nThere are numerous prior works on Bayesian methods for continual deep learning, which the paper mostly ignores [2-7]. It is  also  inaccurate to state that EWC (in constrast to online EWC) finds a shared posterior for all data due to the per-task terms (see the note by Huszar referenced as [25] in the paper). Besides works on Bayesian continual learning, a discussion of federated learning methods, e.g. [8,9], would seem relevant, as these typically work with per-task posteriors as well. [10] similarly consolidates per-task Gaussians into a shared Gaussian that it carries forward, so PR-Exp could be related as an amortized version of that procedure (which maintains the ability of producing per-task Gaussians). Finally, works on hierarchical weight models for neural networks are again not discussed [11-13]\n\n**Further comments**\n\n- The references are formatted inconsistently (venue names, title capitalization, inclusion/omission of page/issue/etc numbers) and look like they have been copy-pasted from Google scholar.\n- Figure 2c suggests that the implicit posterior is potentially multi-modal, however the paper presents no evidence that this is the case in the experiments beyond toy datasets, making the figure potentially misleading. Given the similar performance of PR-Imp to PR-Exp I would find it highly surprising if the distribution was indeed multimodal.\n- Are you sure that Figure 3a shows PR-Exp and not PR-Imp? I have not seen a mean-field Gaussian variational posterior with a remotely comparable increase in predictive uncertainty away from the data anywhere in the literature or my own experiments before. If this is indeed the result for PR-Exp, this would point towards this not really being a variational method as discussed above.\n\nReferences\n\n[1] von Oswald et al. Continual learning with hypernetworks. In ICLR 2020.\n\n[2] Titsias et al. Functional regularisation for continual Learning with Gaussian processes. In ICLR 2020.\n\n[3] Loo et al. Generalized variational continual learning. In ICLR 2021. (on arxiv since Nov 2020)\n\n[4] Pan et al. Continual deep learning by functional regularisation of memorable past. In NeurIPS 2020.\n\n[5] Adel et al. Continual learning with adaptive weights. In ICLR 2020.\n\n[6] Ritter et al. Online structured Laplace approximations for overcoming catastrophic forgetting. In NeurIPS 2018.\n\n[7] Zeno et al. Task agnostic continual learning using online variational Bayes with fixed-point updates. arXiv preprint arXiv:2010.00373, 2020.\n\n[8] Bui et al. Partitioned variational inference: A unified framework encompassing federated and continual learning. arXiv preprint arXiv:1811.11206, 2018.\n\n[9] Yurochkin et al. Bayesian nonparametric federated learning of neural networks. In ICML 2019.\n\n[10] Lee et al. Overcoming catastrophic forgetting by incremental moment matching. In NeurIPS 2017.\n\n[11] Dwaracherla et al. Hypermodels for exploration. In ICLR 2020.\n\n[12] Karaletsos et al. Probabilistic meta-representations of neural networks. arXiv preprint arXiv:1810.00555, 2018.\n\n[13] Karaletsos & Bui. Hierarchical Gaussian process priors for Bayesian neural network weights. In NeurIPS 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "pSzVeAXDCOs", "reviewer": "Reviewer_jqRM", "summary": "The authors propose a Bayesian approach for continual learning that models the weights distributions in a task-conditioned fashion, then relying on task inference.\nAs expected, the main main issue is the task inference phase.\nThe experimental validation is complete, although it is limited to only a small number of tasks in the same sequence.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main merits:\n- The paper is interesting and the proposed method is sound.\n\nMain limitations:\n- split-MNIST and split-CIFAR are very simple baselines, and all continual learning tests presented only have a small number of tasks.\n- i would definitely not call cifar10 a 'natural image dataset'\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5"}, "extra_text": {}}, {"review_id": "dwAXr4r_NlE", "reviewer": "Reviewer_noon", "summary": "The authors propose a new method for (offline) continual learning using bayesian hypernetworks to obtain a distribution over a task-based parameter generating function. This meta perspective enables replay at the level of level of the hypernet parameters (rather than at the input level), while the bayesian approach provides reliable uncertainty estimates which the authors show can be used to perform task inference at test time. \n\nThe hypernet setup, (which is really a 3-level hierarchy) comprises a 1) Task-Conditioned network, which maps a task embedding to hypernet parameters $\\theta_t$, 2) a hypernet with params $\\theta_t$ remaps a noise sample $z$ to a set of weights $w$, 3) which are used as parameters for the downstream classifier. \n\nThe authors provide two toy setups to illustrate the validity of their method, before benchmarking their method on Split-Cifar and Split Mnist. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Originality__:  The proposed method is novel: the bayesian hypernet perspective is a fresh take in the CL field. The related works section is complete and cited adequately. \n\n__Quality__: The submission is technically sound and the claims are well supported via motivation and empirical results. The toy experiments are beautifully designed, especially the second one. The work is well rounded and the authors have made a good use of the place available. \n\n__Clarity__: The method is well explained. It is a very involved method (a 3-stage pipeline) but the authors do a good job to convey the important information across. The only caveat I will add to this is regarding the spectral Stein gradient estimator. Ideally the authors should provide some intuition on how it works, or refer to the appendix for a clearer explanation. To me this part seems quite important, as it key to optimizing implicit hypernets. \n\n__Significance__: The results are significant, especially due to the task inference module. This area is largely unexplored and the authors take a solid step to perform principled inference. \n\n__Other__: \nOne thing I would like the authors to shed light on is the difficulty of the optimization process. The proposed method is quite complex, and from the looks of it seems nontrivial to optimize. How stable / sensitive to hparams is the method ?\nMoreover, it would be best if the authors discuss how the hparam selection was made for the baselines considered. Were all methods given a similar budget ?\nLastly, a good addition to the paper would be an ER baseline, with a buffer size equivalent to the size of the core-set used for `-CS` methods. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: We thank the authors for their detailed clarifications. Most reviewers agreed that this paper makes interesting contributions to the area of continual learning, which is relevant to the NeurIPS community. There were extended discussions about the Bayesian positioning of the paper and gaps in the related work. The former was settled by the authors' detailed response how online EWC can be obtained from a series of approximations and how posterior meta-replay can be derived by approximations of the posterior. Regarding the latter, I would encourage the authors to not merely cite, but to clearly explain how their work builds on prior work.
Decision: Accept (Poster)