Continual Learning with Bayesian Neural Networks for Non-Stationary Data
Authors: Richard Kurle
Botond Cseke
Alexej Klushyn
Patrick van der Smagt
Stephan Günnemann
Venue:  (2020)
Keywords: Continual Learning
Online Variational Bayes
Non-Stationary Data
Bayesian Neural Networks
Variational Inference
Lifelong Learning
Concept Drift
Episodic Memory

Abstract: This work addresses continual learning for non-stationary data, using Bayesian neural networks and memory-based online variational Bayes. We represent the posterior approximation of the network weights by a diagonal Gaussian distribution and a complementary memory of raw data. This raw data corresponds to likelihood terms that cannot be well approximated by the Gaussian. We introduce a novel method for sequentially updating both components of the posterior approximation. Furthermore, we propose Bayesian forgetting and a Gaussian diffusion process for adapting to non-stationary data. The experimental results show that our update method improves on existing approaches for streaming data. Additionally, the adaptation methods lead to better predictive performance for non-stationary data. 

Published as a conference paper at ICLR 2020
CONTINUAL LEARNING WITH BAYESIAN NEURAL NETWORKS FOR NON-STATIONARY DATA
Richard Kurle* 2
Botond Cseke'
Alexej Klushyn
Patrick van der Smagt!
Stephan Günnemann
Volkswagen Group
Technical University Munich
ABSTRACT
This work addresses continual learning for non-stationary data, using Bayesian neural networks and memory-based online variational Bayes. We represent the posterior approximation of the network weights by a diagonal Gaussian distribution and a complementary memory of raw data. This raw data corresponds to likelihood terms that cannot be well approximated by the Gaussian. We introduce a novel method for sequentially updating both components of the posterior approximation. Furthermore, we propose Bayesian forgetting and Gaussian diffusion process for adapting to non-stationary data. The experimental results show that our update method improves on existing approaches for streaming data. Additionally, the adaptation methods lead to better predictive performance for non-stationary data.
INTRODUCTION
Continual learning (CL), also referred to as lifelong learning, is typically described informally by the following set of desiderata for computational systems: the system should (i) learn incrementally from a data stream, (ii) exhibit information transfer forward and backward in time, (iii) avoid catustrophic forgetting of previous data, and (iv) adapí to changes in the data distribution (Ring. 1997; Silver et al., 2013; Chen & Liu, 2016; Ruvolo & Eaton, 2013; Parisi et al., 2018). The necessity to adapt to non-stationary data is often not reconcilable with the goal of preventing forgetting. This problem is dilemma (Grossberg, 1987).
also known as the
The majority of current CL research is conducted in the context of online multi-task learning (Nguyen et al., 2018; Kirkpatrick et al., 2017; Schwarz et al., 2018; Rusu et al., 2016; Fernando et al., 2017), where the main objective is to prevent catastrophic forgetting of previously learned tasks. This focus is reasonable since changes in the statistics of the data distribution are usually an artefact of learning different tasks sequentially. However, changes in the statisties of the data also be real properties of the Data-generating process. Examples include models of energy demand, climate analysis, financial market, or -shrvbeh analytics (Ditzler al., 2015). In such applications, the statisties of the current data distribution are of particular interest. Old data may be outdated and can even deteriorate learning if the drift the data distribution neglected. Consequently, CL systems for non- stassiottonry data require adaptation methods, which deliberately forget outdated information.
In this work, we develop an approximate Bayesian approach for training Bayesian neural networks (BNN) (Hinton & van Camp, 1993; Graves, 2011; Blundell et al., 2015) incrementally with non- stationary streaming data. Similar to variational continual learning (VCL.) (Nguyen et al., 2018) and the Virtual Vector Machine (VVM) (Minka et al., 2009), we approximate the posterior using Gaussian distribution and a complementary memory of previous data. Both components are updated sequentially, while adapting to changes in the data distribution. Our main contributions
follows:
We propose an online approximation consisting of a diagonal Gaussian distribution and running memory, and we provide a novel sequential update method for both components.
We extend the online approximation by two alternative adaptation methods, thereby general- ising online variational Bayes with Bayesian neural networks to non-stationary data.
We compare our sequential update method to VCL in the online-inference setting on several popular datasets, demonstrating that our approach is favorable. Furthermore, we validate our adaptation methods on several datasets with concept drift (Widmer & Kubat, 1996), showing performance improvements compared to online variational Bayes without adaptation.
"Correspondence ri richard raaardd. kurle@tum

---

Published as a conference paper at ICLR 2020
2 BACKGROUND: ONLINE INFERENCE
Consider a stream of datasets (Dr, [P=,)==> observed. For the moment, we assume that these datasets and the samples within are generated independently and identically distributed (i.i.d... Methods for non-i.id. data are considered in Sec. 4.
=== where ls are the time points at which datasets Dr, are
In the Bayesian approach to online learning, we want to infer the posterior distribution p( ( of our model parameters, with the restriction that the data is processed sequentially." Using Bayes rule. recursive posterior inference equation emerges naturally:
p(w(Drrste) p( w/ (Dtiit--)) p(Pr, w, Di = (WwpDradt--. 1 ((Dr,lw).
where the last step follows from the i.idd assumption of the data.
In this paper, we consider Gaussian and multinomial likelihoods, parametrised by neural network with weights w and prior p( (/]]0) = (w) = -N(w; (o, do). Furthermore, we consider supervised (m)
learning, where Dis [d]
(m) In {(x+/)) Nia (/r)) and p(d!") (w) p(yl (n) NNN()))) (n), w)). (w))
2.1 ONLINE VARIATIONAL BAYES
Since exact Bayesian inference is intractable for non-trivial models, various approximations have been developed. Prominent examples include sequential Monte Carlo (Liu & Chen, 1998), assumed density filtering (Maybeck, 1982), and online variational Bayes (Opper, 1998; Ghahramani, 2000; Sato, 2001; Broderick et al., 2013). Online variational Bayes (VB) approximates the posterior of Eq. (1) by a parametrised distribution q0., (w) R p( [[W] (Dtyat) through a sequence of projections:
40.g ww argmin KT. go (w) ZT, 90,__, (w) p p(t,, /w]]
de
where zr is the normalisation constant. The above minimisation equivalent to maximising the evidence lower bound (ELBO) Cr, (0; Pr₂) (go(w) logg((P,, /w] [L[[g(W)) ge (w) In this work, we consider diagonal Gaussian posterior approximations go., (w) for the neural network weights, similar to Nguyen et al. (2018).
2.2 ONLINE VARIATIONAL BAYES WITH MEMORY
Online approximate Bayesian inference methods inevitably suffer from an information loss due to the posterior approximation at each time-step. An alternative approach to online learning is to store and update representative dataset/generatiiee model-and to use it memory-in order to improve inference (Robins, 1995; Lopez-Paz. & Ranzato, 2017; Shin et al., 2017; Kamra et al., 2017). Memory-based online learning has also been combined with online Bayesian inference methods (Minka et al., 2009; Nguyen et al., 2018). A common property of these approaches is to represent the (current) posterior approximation by a product of two factors
p(w(Denstu) = qo., (w) p(Me,
and update them sequentially as new data Dr, is observed. The factor p(Mr, (w) =I M p(miz (m) (w)
is the likelihood of set of M |.M] data points, which refer to running memory; and qe., (w) a Gaussian distribution, which summarises the rest of the data Di = -Di,a M,..
In case of VCL, the factors in Eq (3) are updated in two steps, which we refer to as (i) memory update and (ii) Gaussian update: (i) new memory Mr. C Dr, U M is selected using heuristics such as random selection or the k-center method (a greedy algorithm that selects K data points based on geometric properties of Die U M; (ii) the Gaussian distribution updated with the remaining w).
data D'2- = D,,, U Mi MM (using Eq. (2)) to obtain q0. (w) R3 q0 (0.2-((w)
Note that we cannot sample directly from the posterior approximation in Eq. (3) and thus we cannot easily evaluate quantities such as the posterior predictive distribution. VCL therefore performs second projection
goe (w)= argmin KL go (w) (Z,,) qo., (w) p(M{.
(4)
This distribution should not be confused with the recursively updated variational distribution (Eq. (2)).
strict definition of online learning requires single data samples each time step instead of hatches D.y-
2

---

Published as a conference paper at ICLR 2020
3 IMPROVING MEMORY-BASED ONLINE VARIATIONAL BAYES
In this section, we focus on two problems of existing approaches using online VB with running memory: (i) the memory update does not take into account the approximation error or approximation capabilities of the variational distribution; (ii) the Gaussian update-performe performed by optimising the ELBO (Eq. (2)) only with data Di,- -ca fail for streaming data. This is because VB yields poor posterior approximations if the dataset is too small or the neural network architecture has too much capacity (ef. Ghosh et al. (2018), Fig. 1). In Secs. 3.2 and 3.3, we propose improvements to these two update methods. The mathematical background for our approach is provided in S 3.1.
3.1 PROPERTIES OF THE GAUSSIAN VARIATIONAL APPROXIMATION
There are two important properties of the Gaussian variational approximation that will exploit later: (i) Gaussian approximate posterior distributions factorise into a product of Gaussian terms corresponding to the prior and each likelihood term; (ii) the ELBO can be written the of the approximation's 's normalisation constant and a sum of residuals corresponding to these factors.
Let Po (w) = (N::: /g. Eg) be a Gaussian prior and p(D|w) w) = II, p(d (x)) |w) be the likelihood of the observed data D. Furthermore, let (o(w) NWw: μu, Σ) denote the corresponding Gaussian vari- ational approximation with 0 = {u. }}. Assume that f and Σ are the optimal parameters correspond- ing to (local) maximum of the ELBO L(u,, E; D) The optimality conditions O, Σ; D) 0 and On, Σ; D) = 0 can be rewritten as follows (Knowles & Minka, 2011; Opper & Archambeau, 2008; Cseke al., 2013) (cf. App. C):
Σーt Ep no + >> (o,E [log (x) [w] -20g go (
(5a)
Σ Eg¹ - - yo logp(d w)].
(5b)
Since the sum of natural parameters corresponds to product in distribution space, the above equations show that-at a local optimmm the approximation qo (w) factorises the same way as the posterior p(w/ (D). can be written in the form (w) = Zg PP(w)) II, r(m)) (w). where the factors ((m) (w) are Gaussian functions with natural parameters given by Eqs. (5a) and (5b), and where 2, = po (w) II., r(n) (w) dw is the normalisation constant. These Gaussian functions r(()(w) each correspond to the contribution of the likelihood ((d(")) (w) to the posterior approximation go (w).
The resulting factorisation implies that the ELBO C(u, D) can be written in the form (Opper Winther, 2005) (c.f. App. D)
Clu, E;D) = -log& Zr + Σ (go(()
log (m) (w)]
(6)
the terms pdd /w) were (diagonal) Gaussian w. they would each cancel with the corresponding (diagonal) Gaussian term, leaving only log Zg. Intuitively, the residual terms in Eq. (6) can be used to quantify the quality of the Gaussian approximation.
3.2 MEMORY UPDATE
The authors of VCL propose to use a memory to compensate the information loss resulting from the Gaussian approximation of the posterior distribution. However, their memory update is independent of the approximation error that is due to the chosen distributional family (diagonal Gaussian). An alternative memory update, which specifically targets the above mentioned information loss, has been introduced previously for VVM. Although the latter method was developed for expectation propagation in (linear) logistic regression model-and is thus not directly applicable to online VB we show that some its properties can be transferred to the variational inference setting. The central idea replace the likelihood terms that can be well approximated by a Gaussian distribution by their Gaussian proxies p(de, (w) ss rt, (w; dr,) resulting in go., (w); and retain the data corresponding to the rest of the likelihood terms in the memory. To score candidate memory, Minka et al. (2009) proposed maximise the KI. divergence between the model given in the form of Eq. (3)
3

---

Published as a conference paper at ICLR 2020
and Gaussian posterior approximation, that is, maximise KL Z 90., (w) p(M[w) [@o., (w)] However, this score function is intractable, because the expectation in the KI. includes the likelihood p(M]w). In the following, we develop a tractable score function applicable to VB. Intuitively, we can use Eq. (6) to test how much L(u, 3; D) changes if we replace the exact likelihood terms (of all data which is not contained in the candidate memory) by their Gaussian approximations.
To achieve this, we need to find Gaussian approximations for every data point in the candidate memory. We first approximate the posterior distribution using both Deg and M
go (w) = argmin KL go [w) || ZT:² "o._,, (w) (D htpp://(ppp// p(M [w]
(7)
Next, we use Eqs. (5a) and (5b) to calculate the natural parameters of all Gaussian terms. In practice, we estimate the natural parameters using (unbiased) Monte- Carlo estimators for the expectations. We have now available the likelihood terms and their Gaussian approximations. This allows us to write C(O..: Dos U U in the form of Eq. (6):
[(0re; Dre U M log Zare +
Σ
Egery (w) pog /w) log Fre (w; ]]
d1g E M
where d, are the samples in Dr, M and where It, (w; d are the Gaussian approximation of the corresponding likelihood terms. Note that rt, does not only depend on de,, however, we omit the dependence on the remaining data for notational convenience.
If the likelihood p(d., (w is close to the Gaussian rt, (w; ) in expectation w.r.t. the approximate posterior g (w), then its contribution to C(Or. ; Dr, U Mi, is small. Similarly, likelihood terms that cannot be well approximated by the respective Gaussian have a large contribution, and, hence, the corresponding data should be kept in the memory. For this reason, we propose the score function
St, (M; Drn, U M
Σ
Ege, (w) pog,, ww log rr, (w; dr,d),)
diy EM
and the corresponding memory update Mr. argmaxM S., (M; D., Mr. ). Note that since all residual terms are computed independently, the update results in selecting the top M terms.
3.3 GAUSSIAN UPDATE
The Gaussian update follows from the memory update presented in the previous section: once the memory Mr, has been selected, we update the Gaussian distribution with the approximations corre- sponding to the rest of the data Dr,, U- \Ma. We can update qo,, (w) in two equivalent ways:
g9:, (w) = -0,,,, (w) II Fee (w;
(9a)
qe (w) = qui, (w) / II (w; d..).
(9b)
deg gMy
deg EMi
Note again that the natural parameters of rt, (w: d..) are estimated using Monte Carlo and the products in the above equations imply a summation variance of this sum of estimators, we use Eq. (9a) if [Dr,l S [M and Eq. (9b) if (Dyy > [Ma,] Furthermore, we can compute the average bias from all natural parameter estimates (see App. C). We reduce the bias of our estimates by subtracting the average bias from all estimates. Note that further MMr to compute the update
the natural parameters. In order to reduce the
option to update qz (w) would be to use VB on the data Dr2 U M \ 40 (w) R 40.g-. (w) ((Dr, U M /w) The latter approach is numerically more stable but computationally more expensive. It also turned out that it is less favorable to the update using Eq. (9a) or Eq. (9b) in case of small datasets Di2- because VB applied to BNNs with small datasets often leads poor fit.
Previous work hypothesised that this problem is an artifact of the ELBO and not an optimisation problem (Trippe Turner, 2018; Turner & M. Sahani, 2011). We provide further evidence in Fig. 1, where we infer the posterior of a Bayesian neural network with VB, using 70 and 100 data samples respectively and compare it to posterior inference with MCMC. case VB with 70 samples, the posterior approximation yields a model that is almost linear. These difficulties of posterior inference with variational Bayes are especially problematic in case of the streaming data setting, where the number of observations at each time-step is typically very small. The Gaussian update proposed above can alleaviate the problem of having to train BNNs with small datasets. Specifically, we have Nr. M instead of No, data points to find better optimum of the ELBO.
4

---

Published as a conference paper at ICLR 2020
100
3008
4X00
(a) MCMC, 70 samples
(b) VB, 70 samples
(c) VB, 100 samples
(d) ELBO terms during training
Figure 1: Posterior predictive distribution in the xy-plane (grey) of Bayesian neural network with 2 layers of 16 units, tanh activations, prior (ww -Nww:; 1), and Bernuulii likelihood. In case of variational Bayes (Figs. lb, Ic), the KL divergence the ELBO annealed from 3 = to 3 = 1 over many iterations (450k anncaling, 50k ELBO). Fig. ld shows that the approximation trades off the expected log-likelihood for better KL divergence as 9 is increased. With 70 data points, the annealed KL jumps to significantly lower value, resulting in almost linear decision boundary. By contrast, MCMC yields much better predictive distribution for same number samples. Data visualised red and blue.
4 VARIATIONAL BAYES WITH MODEL ADAPTATION
The incremental learning methods discussed so far assume i.i.d. data (cf. Sec. 2, Eq. (1)). This assumption can be reasonable even in scenarios with changing data distributions, c.g. when the data drift is algorithmic artifact rather than real phenomenon. For example, in online multi-task or curriculum learning we want to learn model of all tasks, but we may choose to learn the tasks incrementally for various reasons (e.g. Nguyen et al., 2018; Kirkpatrick et al., 2017; Schwarz et al., 2018; Rusu et al., 2016; Fernando et al., 2017). However, such approaches are not applicable for modeling non-stationary data: one of the properties of online VB is that the variance of the Gaussian posterior approximation shrinks at a rate of O(N), where N is the total amount of data (e.g. Opper, 1998). Consequently. learning comes to halt to be extended by method that enables it to adapt to changes in the data distribution, e.g., by deliberately forgetting the belief inferred from previous data.
+ oo. To overcome this issue, the model needs
In the following, we describe two alternative methods for adapting to changing data. In Sec. 4.1, we impose Bayesian exponential forgetting, which forgets previous data exponentially by weighting the likelihood terms (or their approximations). In Sec. 4.2, we implement the adaptation through a diffusion process applied to the neural network parameters. Compared to the online learning scenario, we make the following assumptions: (i) we observe datasets Dr, at potentially non-equidistant time steps ty: (ii) data within D;, is assumed i.i.d., however, not between different datasets Di, and Dizt1.
In both approaches, we realise adaptation by additional forgetting step before observing the new data D We denote the distribution, which results from applying the forgetting step to the posterior approximation qe (w) p(Mez (w) by Piz+1 (w).
4.1 ADAPTATION WITH BAYESIAN FORGETTING
Model adaptation through forgetting can be achieved by decaying the likelihood based on the temporal recency the data (Graepel al., 2010; Honkela & Valpola, 2003). has been explored previously as an alternative to filtering and referred to as Bayesian exponential forgetting (Kulhavý & Zarrop, 1993). This approach defines forgetting operator that yields p( w (WWeez (Dertt) directly. Here, a continuous-timg version of this forgetting operation that can be formulated as
58-tk
p(w Di
Po(w) Il P(Dr w)
(10)
where 7 is time-constant corresponding to the average of the time-lags At&+1 = t distribution defined in Eq. (10) can be formulated recursively (cf. App.
tx. The
p( (Dyste-))) x po (w) (-(1-)) A6--1/ p( ((W(Drrst) (1-.)
A12+1/ (
p(D
(w).
(11)
This equation can be viewed as Bayes rule (I). applied after the forgetting step. The first two terms of Eq. (11) can be identified as the forgetting operation, applied to the current posterior. In
5

---

Published as a conference paper at ICLR 2020
18 5.0 do 0.3
20 5.0. 0.25
6.223
10-
8.5
3.5
66-
6:35
3325
8333
3375
0233
02-
1225
303
216
266
266
16
116
158
214
280
382
24
(a) fA
(b) C
(c) pr - 0t. C
0.05
I6:
5,0 d. 0.3
Jo. cy G.L, 2015
00035
1.0
E4225
600
0.5
0.075
5535
E3075
0.3335
0/2
50
380
350
200
26
104
124
240
250
50
319
150
200
25c
(d) fAg
(c)
(f) j4g dr, 8 = 0.025
Figure 2: Time-evolution of distribution parameters of Bayesian Forgetting (top) and the Ornstein-
OrnsteinUUhlenbeck Uhlenbeck
process (bottom) for different adaptation parameter values. The initial distribution (at t = 0) can be seen as the approximate posterior some time-step fg--
order to apply this operation our posterior approximation qo., (w) p(Mr, ww. we modify it by an additional weighting factor for each likelihood term in the memory. Denoting the age of memory item m by At& (m), the forgetting operation for this new posterior approximation then results in
p(m|w) (1 (0 (1-)) (1-e)* (1-e) 0441/7
Ptyli (w) x po())* 1-(1 -()-))
(1--)) A
x g014 (w)
II
mEMrre
- 1-11-))) Mass/r
g0ep (w) (1-c 00
x p(m w) (1-c) a044
(12)
m
where At&+1 (m) Atg((m) + At++1- As can be seen from Eq. (12), BF acts on both factors of the posterior approximation independently: in case of the memory, it re-weights the respective likelihood terms by updating 1 (m). For the Gaussian term q0., (w), BF leads to a weighted product with the prior distribution (i.e. the first two terms of Eq. (12)), resulting in Gaussian with parameters
6 1x41 (1 (1- 2//t////) 0o 2 + (1 -1-66 -c) n0ut2/ Oi
- -1- 2 no + (1 -) a/ Ote Pt..
The 1 =
For Afp-1 + 00, the likelihood term in Eq. (12) converges to the uniform distribution and the Gaussian term reverts the prior. We note, however, that while Eq. (11) is an exact recursive form of Eq. (10), the online VB approximation of Eq. (11) is not generally identical to the (offline) VB approximation of Eq. (10) due its successive approximations. For tuning the hyperparameter e, we note that the weighting of likelihood terms corresponds to an effective dataset size of 1/c N (if all datasets are of equal size N). In Fig. 2, we also visualise the forgetting operation applied the Gaussian part of the posterior approximation for multiple values of
4.2 ADAPTATION WITH DIFFUSION PROCESSES
Model adaptation can also be realised by using dynamic model parameters that evolve according to a stochastic process. In this case, adaptation is achieved by the stochastic transition Plzts. (w/[w) resulting in a prediction distribution
Pieta (w') 1 Pietsite [w) p(w|[(,stt Deyste) ddw
(13)
where we consider Gaussian transitions Ptu (w' w). However, this operation is generally not tractable for our posterior approximation qo., (w) p(Me,//) Moreover, the forgetting operation implied by the transition does not retain the product form in the of BF. For this reason, we consider only a Gaussian posterior approximation (without memory) for this approach, that is Pt (w') = ʃ Pt (w') (w) qo.,, (w) dw.

---

Published as a conference paper at ICLR 2020
As mentioned in Sec. 4.1, BF yields the prior distribution for At&+1 - co. This is desirable property, since it corresponds to forgetting all information conveyed by the data. In case of a Gaussian prior, the only Gaussian process that fulfillss this requirement is the Orsstein- Ommstein-Ullenbeet
(OU) process given by the stochastic differential equation dw, = 0 (No we) dt + do 220 dW,, where @ is the stiffness parameter which controls the drift rate towards /4o- To decouple the adaptation parameter from the rate at which data is observed, we rescale the stiffness parameter 0 a/r. The resulting is defined by the parameters
prediction distribution Pie+ (w) = - (Ptz411 1
Mi++1 (1 - Atet1
A4441
Mo +
1+t1 = (1 2a A1e+1
Jo +
2a
A1e+1
t
An interesting observation is that both parameters evolve independently of each other. In contrast to BF, the mean and auriance-insteal of the natural parameters follow exponential decay. The hyperparameter a can be determined e.g. through the half-time of the exponential decay of the mean parameter, given as 71/2" l/o. We visualise the time evolution of the above parameters in Fig. 2.
5 RELATED WORK
There are many Bayesian approaches to online learning. which differ mostly in the approximation of the posterior distribution at each time-step. Sequential Monte Carlo (Liu & Chen, 1998) approximates the posterior by a set of particles. Assumed Density Filtering (ADF) (Maybeck, 1982) and Bayesian online learning (Opper, 1998) are deterministic posterior approximations based on moment matching. Other deterministic approaches are based on Laplace's approximation (MacKay. 1992): Kirkpatrick et al. (2017) use multiple diagonal Gaussian posterior approximations of previous time-steps to regularise future tasks; Ritter et al. (2018) use a single (block-diagonal summarising all previous time-steps. The latter method is closer to Bayesian online inference, as it is an approximation of Eq. (1). Our work is based on online VB (Opper, 1998; Ghahramani, 2000; Sato, 2001; Broderick et al., 2013), which approximates the posterior at every time-step by minimising the KL-divergence between a parametric (here Gaussian) and the true posterior distribution. In contrast to online VB, we approximate the posterior by a Gaussian distribution and a running memory.
posterior approximation,
Other approaches are based on various types of episodic memory, motivated by their empirical success in preventing catastrophic forgetting. The basic idea of rehearsal (Ratcliff, 1990) is to train on both the new data and subset of previous data or pscudo samples (Robins, 1995; Shin al., 2017; Kemker& & Kanan, 2017) sampled from a generative model. The memory-based online inference methods most similar to our approach are VCL (Nguyen et al., 2018) and VVM (Minka et al., 2009). Both methods use a Gaussian distribution and running memory to approximate the posterior. VCL uses heuristics such as random selection or the k-center method to update the memory. However, both heuristics select the memory independently of the Gaussian approximation. By contrast, VVM updates the memory with data that cannot be well approximated by the Gaussian distribution. VVM uses expectation propagation for the posterior approximation in a logistic regression model and, therefore, it is not directly applicable to our work. We transferred the main idea of VVM to online VB and developed the corresponding memory update method. In our case, the memory is updated with data for which the ELBO changes most if the corresponding likelihood functions are approximated by a Gaussian. In contrast these two approaches, we extend our model by an adaptation method that allows to cope with non-stationary data.
Many adaptation methods were developed in the context of concept drift, however, few of these approaches operate in the Bayesian framework. For example McInerney et al. (2015) treat the learning dynamics of their model as a non-stationary process that allows for adaptation. In contrast, our approach uses an evolving prior and well defined forgetting mechanism that gives better control over the learning process. more closely related approach uses the extended Kalman filter to estimate the optimal parameters of logistic regression classifier Su et al. (2008). However, they consider a transition model which is equivalent to Wiener process (in unit-time) and therefore does not revert to the prior. By contrast, our approach (Sec. 4.2) models the dynamics as a prior-reverting OU process. BF (Kulhavý & Zarrop, 1993) has been applied as an alternative to adaptation with an explicit transition model (e.g. Honkela & Valpola, 2003; Graepel et al., 2010). Compared previous work, we used a continuous-ting time version of BF and extended it to our posterior appoximation consisting of Gaussian and running memory.
7

---

Published as a conference paper at ICLR 2020
M
(a) to
(h) 125
(c) tso
(d) LMI.
Figure 3: Two- mo-msoons dataset. Predictive distribution (Figs. 3a 3c) of BNN (gray) and running memory (rectangular shapes, size proportional to the score), chosen by the memory update proposed in See. 3.2. Data from tu and fcck-1 is visualised as large circles and small dots, respectively. Fig. shows the one-step-ahead (predictive) LML (divided by the number of samples) for data that will be selected for the memory and data that will be evicted. Data that will be selected in the memory tends to have significantly lower predictive likelihood.
GRS (ours)
k-center
random
Table I:
average test LMI,, av- eraged over the last 10% time- steps. Mean and std. deviations computed over 16 indepen- dent runs. The memory size is 15 for Concrete, Boston, En- ergy and Yacht, 25 for Spam and Wine, and 150 for MNIST. Bold indicates best (average) results.
Concrete
0..79 1 0.039
-0.798 + 0.039
-0.800 + 0.039
Boston
0.619 0.111
-0.638 0.093
-0.664 1 0.156
Energy
0.365 = 0.440
-0.119 0.128
-0.078 = 0.087
Yacht
1.925 = 0.229
1.658 + 0.291
1.648 + 0.254
Spam
-0.216 + 0.016
-0.219
-0.217 0.016
Wine
-1.165 + 0.056
1
-1.191 + 0.070
MNIST
-0.148 0.005
-0.158 0.005
-0.153 + 0.005
6 EXPERIMENTS
We validate our proposed inference methods in two stages. In Sec. 6.1, we compare our memory update and Gaussian update (Sec. 3) existing memory-based online inference methods on several standard machine learning datasets. In Sec. 6.2. we evaluate our adaptation methods (Sec. 4) on commonly used datasets with concept drift (Widmer & Kubat, 1996). where the conditional distribution of labels given the features changes over time (i.e. non-stationary data in the context of predictive models).
We found that training (variational) Bayesian neural networks on streaming data is challenging, specif- ically, our approach requires model parameters very close to a local optimum since Eqs. (5a) and (5b) hold only at local extrema of the ELBO. To overcome these difficulties, we use several methods to reduce the variance of the gradient estimates for learning: (i) we apply the local reparametrisation trick (Kingma al., 2015); (ii) use the Adam optimiser (Kingma & Ba. 2014); and (iii) use multiple Monte Carlo samples to estimate the gradients (cf. Tab. 2 for details). Furthermore, we developed methods for determining hyperparameters of the Gaussian prior and the initialisation distribution of Bayesian neural networks. The idea is similar to the initialisation method proposed by Glorot & Yoshua Bengio (2010) and He et al. (2015): we choose the prior and the posterior initialisation such that the mean and standard deviation of the activations in every layer are approximately zero and one, respectively. We refer App. H and App. I for a derivation and further details.
We use the following metrics for evaluation: (i) the avg. test log-marginal likelihood (LML)
No1 Σn log Eñe, (w) [p(d(cclw)) where (m) is sample from heldout test dataset; (ii) the avg. one-
step-ahead LMI. No11 Σ. log Eãe,, (w) (p(a(()) (n) [w)], where l24s data observed at time-step 4++1-
(m)
Both metrics measure the predictive performance, however (i) can be used in the online setting, where
the data i.i.d.; and (ii) is typically used to evaluate models with non-station streaming data.
6.1 ONLINE LEARNING
In this section, we evaluate our running memory (Sec. 3) in an online learning setting. To illustrate how our memory update works, start evaluation with qualitative assessment: train model on 2-dimensional toy data (two- (wwo-moons), where we can visualise the selected memory. The BNN has 2 layers with 16 units and tanh activations, and has prior (o(w) (ww;; 1) all
8

---

Published as a conference paper at ICLR 2020
Concrete
UCI Energy
1..0
-3217
kooree rendem)
-121
10
26
30
40
24
54
110
150
204
259
JC Concrete
UCI Energy
UO
ST
122 11 bodd
-0.100
21
313 13
Gcerra12 28
10 222
-0.27
22
26
20
25
90
15
389
100
Figure 4: Average test LML, evaluated for several memory sizes (top), and evaluated over time (bottom) for a specific memory size (cf. corresponding legend). Cf. Sec. 6.1 for details and App. A for further results.
weights and biases. The memory-size is M 30. The model observes 150 data samples at time-step tu and 15 samples at all consecutive time-steps. In Fig. 3, we visualise the selected memory and the corresponding scores for time-steps to. 125, and tso, respectively. We can make the empirical observation that our method favors data close to the decision boundary. Furthermore, in Fig. 3d, we visualise the one-step-ahead LMI. for data that will be selected and evicted (in the next time-step), respectively. This shows that our memory update tends to select data for which the model has a low predictive LML. These observations support our intuition that the memory is indeed complementary to the Gaussian approximation, selecting data for which the likelihood cannot be well approximated by a Gaussian function. In Fig. 9 of the supplementary material, we visualised the running memory for model trained on MNIST, showing that the memory also accumulates diverse samples over time.
We evaluate our memory- based online inference method (Sec. 3) quantitatively on several standard machine learning datasets, including regression (UCI Boston, UCI Concrete, UCI Energy, UCI Yacht) and classification (MNIST, UCI Spam, UCI Wine) tasks. Here, we refer to our approach as Gaussian Residual Scoring (GRS). We compare GRS to the respective memory update and Gaussian update methods proposed in VCL (Nguyen et al., 2018) (cf. Sec. 2.2). Refer to App. B for an explanatory list of compared update methods. Online learning is performed by observing Nr, samples per time-step (cf. Tab. 2 for the experiment setup and byperparameters.) For evaluation, we use a random held-out test dataset (20% of the data). We perform each experiment with 16 different random data splits and random seeds for the model parameter initialisation. In Fig. 4, we plot the test LML, averaged over the 16 runs, against the memory size, and the LML over all time-steps. In most cases, random selection and the k-center method start with worse initial fit at lo- This is because these methods perform the initial Gaussian update by optimising the ELBO with N.,, - M samples at to; by contrast, GRS uses Gaussian update that first optimises the ELBO with No, samples and subsequently discounts the contribution of the memory. In Tab. 1, we report the mean and std. deviation of the LML., where the mean and std. deviation are taken over the 16 independent runs, each averaged over the last 10% time-steps. The results demonstrate the superior predictive performance of our update methods. We also note that the experiments on the smaller datasets (cf. Tab. 2 in App. B) result in high variance among the random data splits and random seeds. This is the case for all compared methods and it could not be remedied e.g. by using annealing or a different prior.
6.2 ADAPTATION
In this section, we evaluate our adaptation methods (Sec. 4) in settings with concept drift. We begin with a simple logistic regression problem, where the data D,, = Y Vta) X1, E R2 yt, e R is sampled from Xt, ~ Uniform(-3, 3), Yu, Bernoulli(e @ W1, The true model has two time-dependent parameters wh. = 10 sin (a tg). w! = 10 cos(a tt), where a = deg /sec and where we observe data at t, E [0, 1, standard online learning (without adaptation), OU process transitions, and Bayesian forgetting. If the -mee-dependence of the data is ignored (in case of online VB), the class labels are distributed with equal probability in the whole input space. Consequently, as t + co, the weights of the model without adaptation shrink to O. By contrast, the posterior means of BF and the OU process follow a sinusoidal curve the parameters of the true model.
720]. Fig. shows the learned model parameters for

---

Published as a conference paper at ICLR 2020
(a) Online (no adaptation)
(b) Ornstein-
Ornstein-Uhlenheee Uhlenheck process
(c) Bayesian forgetting
Figure 5: Mean and std deviation of the approximate posterior distributions of logistic regression model over 720 time-steps. The model is trained on a toy classification problem with rotating class boundaries (cf. Sec. 6.2). Online VB (left) quickly converges to zero mean, whereas Bayesian forgetting and OU-proces transitions lead sinusoidal curve in the true model.
forgetiing momoty
-243
-6.0
LEAE
-2.40
-153
-055
-0.50
0.5
3.0
3.5
200
0.0
3.5
1-3
300
150
Segeeting
Figure 6: One-step ahead LML on Covertype dataset. Subplots show 3 different adaptation methods (3 left plots), evaluated for several values of the respective adaptation parameter, and Bayesian forgetting with = 0.11. evaluated for multiple memory sizes (right).
We also evaluate our adaptation methods quantitatively on 3 datasets with concept drift (Weather, Gas Sensor Array Drift, Covertype). We compare online VB (without adaptation), the Wiener process (a special case of the OU process), the OU process, and Bayesian Forgetting (with and without memory). All compared variants use the same model architecture and hyperparameters (cf. Tab. 2 in the supplementary material). We report the one-step- with 500 Monte Carlo samples. Results are averaged over the last 50% time-steps, because we are interested in the continual learning performance, and the first few time-steps will be similar for most methods. We report the mean and std. deviation over 8 independent runs with different random seeds. In Fig. (and Fig. in the appendix), we plot the LML against 10 adaptation parameter values (of the respective adaptation method), where the value zero corresponds to online The LML for BF with different memory sizes and a fixed forgetting rate 0.11 is shown in Fig. 6. As can be seen from the results, all adaptation methods significantly improve the performance compared to online VB. Interestingly, the Ormstenn-Uhlenbed process performs better than Bayesian Forgetting, however, using running memory with Bayesian Forgetting closes the gap.
LML, where the expectation is approximated
7 CONCLUSION
In this work, we have addressed online inference for non-stationary streaming data using Bayesian neural networks. We have focused on posterior approximations consisting of a Gaussian distribution and complementary running memory, and we have used variational Bayes to sequentially update the posteriors at each time-step. Existing methods update these two components without having an interaction between them, and they lack methods to adapt to non-stationary data. We have proposed novel update method, which treats both components as complementary, and two novel adaptation methods (in the context of Bayesian neural networks with statiootto data), which gradually revert to the prior distribution if no new data is observed.
Future research could extend our work by drift detection methods and use them to infer the adaptation parameters. This work could also be extended by developing adaptation methods for gradual, abrupt, or recurring changes in the data distribution. Finally, we observed that variational Bayesian neural networks with a uni-modal approximate posterior often find poor local minima if the dataset is small and models are complex. This is especially challenging in scenarios with streaming data. While our Gaussian update alleviates this problem approximation family beyond Gaussians could be beneficial. Progress in this direction would improve our proposed methods and allow scale them more complex models.
certain degree, further research in extending the
10

---

Published as a conference paper at ICLR 2020
REFERENCES
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning- Volume 37, pp. 1613-1622, 2015.
T. Broderick, N. Boyd, A. Wibisono, A. C. Wilson, and M. I. Jordan. Streaming variational bayes. In Proceedings of the 26th International Conference on Neural Information Processing Systems- Volume 2, NIPS' 13, pp. 1727-1735, USA, 2013. Curran Associates Inc.
Z. Chen and B. Liu. Lifelong Machine Learning. Morgan & Claypool Publishers, 2016.
B. Cseke, M. Opper, and G. Sanguinetti. Approximate inference in latent moussian-markon models from continuous time observations. In J. C. Burges, L.. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 971-979. Curran Associates, Inc., 2013.
G. Ditzler, M. Roveri, C. Alippi, and R. Polikar. Learning in nonstationary environments: A survey. Computational Intelligence Magazine, IEEE, 10:12-25, 11 2015.
C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha. A. A. Rusu, A. Pritzel, and D. Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734 2017.
Z. Ghahramani. Online variational Bayesian learning. NIPS Workshop on Online Learning. 2000.
S. Ghosh, J. Yao, and F. Doshi Velez. Structured variational learning of Bayesian neural networks with horseshoe priors. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1744-1753, 2018.
X. Glorot and Y. Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Y. W Teh and M. Titterington (eds.). Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249 256, 2010.
T. Graepel, J. Q Candela, T. Borchert, and R. Herbrich. Web-scale Bayesian click-through rate prediction for sponsored search advertising in Microsoft's Bing search engine. In Proceedings of the 27th International Conference on Machine Learning, ICML, pp. 13-20, USA, 2010. Omnipress.
A. Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24, pp. 2348-2356. Curran Associates, Inc., 2011.
S. Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11(1):23 63, 1987.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance imagenet classification. In Proceedings of the 2015 IEEE International Conference Computer Vision (ICCV), pp. 1026-1034, Washington, DC, USA, 2015. IEEE Computer Society.
G. E. Hinton and D. van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT '93, pp. 5-13, New York, NY, USA, 1993. ACM.
A. Honkela and H. Valpola. On-line variational bayesian learning. In In Proc. of the 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation, pp. 803-808, 2003.
N. Kamra, Gupta, and Y. Liu. Deep Generative Dual Memory Network for Continual Learning. arXiv e-prints, art. 1rXiv:1710.10366. Oct 2017.
R. Kemker and C. Kanan. Fearnet: Brain-inspired model for incremental learning. CoRR, abs/1711.10563,, 2017.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412,6980, 2014.
11

---

Published as a conference paper at ICLR 2020
D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. In Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gareett (eds.), Advances in Neural Information Processing Systems 28, pp.
Curran Associates, Inc., 2015.
Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting neural networks. Proceedings of the National Academy of Sciences, (1((()))322133222 2017.
D. A. Knowles and T. Minka. Non-conjugate variational message passing for multinomial and binary regression. In J. Shawe-Taylor R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances Neural Information Processing Systems 24, pp. 1701-1709, Curran Associates, Inc., 2011. 1.
T. Kulhavý and M. B. Zarrop. On general concept of forgetting. International Journal of Control, 58(4):905-9244 1993.
J. S. Liu and R. Chen. Sequential monte carlo methods for dynamic systems. Journal of the American Statistical Association, 93(44)):1032-15
1998.
D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 1470-6479, USA, 2017. Curran Associates Inc. ISBN
D. J. C. MacKay, A practical Bayesian framework for backpropagation networks. Neural Computu- tation, 4(3):488 472, May 1992.
P. S. Maybeck. Stochastic Models, Estimation and Control. Mathematics in science and engineering. Academic Press, 1982.
J. McInerney R. Ranganath, and D. Blei. The population posterior and bayesian modeling on streams. In Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Curran Associates, Inc., 2015.
Information Processing Systems 28, pp.
T. P. Minka, Xiang, and Y. A. Qi. Virtual Vector Machine for Bayesian online classification. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI '09, pp. 411-4 111-418, Arlington, Virginia, USA, 2009. AUAI Press.
C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner. Variational continual learning. In International Conference on Learning Representations, 2018.
M. Opper. A Bayesian approach to on-line learning. In D. Saad (ed.), On-line Learning in Neural Networks, pp. 363-378. Cambridge University Press, New York, NY, USA, 1998. ISBN 0-521- 65263-4.
M. Opper and C. Archambeau. The variational gaussian approximation revisited. Neural Computation, 21: 21:786-92, 10 2008
M. Opper and O. Winther. Expectation consistent free energies for approximate inference. In L. K. Saul, Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems 17, pp. 1001-1008. MIT Press, 2005.
G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. ArXiv, abs/1802.00556 2018.
R. Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97 2:285-308, 1990.
M. B. Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77-104, 1997.
H. Ritter, A. Botev, and D. Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 3752-3752, USA, 2018. Curran Associates Inc.
12

---

Published as a conference paper at ICLR 2020
A. Robins. Catastrophic forgetting, rehearsal and pseudorchearsal. Connection Science, 7(2):123-1444 1995.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and Hadsell. Progressive neural networks. CoRR, 1467,, 2016.
Ruvolo and E. Eaton. ELLA: An efficient lifelong learning algorithm. In In Proc. of the 30th International Conference Machine Learning, pp. p 507-515, 2013.
M. Sato. Online model selection based on the variational bayes. Neural Computation, 13(7): 1649-1681, 2001.
J. Schwarz, Luketina, W. Czarnecki, A. Crabska-Barrinskar Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In Proceedings of the 35th International Conference on Machine Learning. 2018.
H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. In I. Guyon, U. Luxburg, S. Bengio, Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2990-2999. Curran Associates, Inc., 2017.
D. Silver, Q. Yang, and L. Li. Lifelong machine learning systems: Beyond learning algorithms. In AAAI Spring Symposium Series, 2013.
B. Su, Shen, Y-D., and Xu, W. Modeling concept drift from the perspective of classifiers. In 2008 IEEE Conference Cybernetics and Intelligent Systems, pp. 1055-1060, Sep. 2008
B. Trippe and Turner. Overpruning Variational Bayesian Neural Networks. arXiv e-prints, pp. arXiv: 801 .0200662,,.. Jan 2018.
R. E. Turner and M. M. Sahani. Two problems with variational expectation maximisation for time- series models. In D. Barber, Cemgil, and Chiappa (eds.). Bayesian Time series models, chapter 5, pp. 109-130. Cambridge University Press, 2011.
G. Widmer and M. Kubat. Learning in the presence of concept drift and hidden contexts. Machine Learning, 23:69-100 101,, 1996.
13

---

Published as a conference paper at ICLR 2020
8 APPENDIX
A FURTHER EXPERIMENTAL RESULTS
A.1 MEMORY
Here we provide additional experimental results for the memory update and Gaussian update from Sec. 3. We conducted experiments on additional datasets (UCI Boston, UCI Yacht, UCI Red Wine).
The influence of the memory size and the performance over time (for a specific memory size) are shown in Fig. (corresponding to Fig. in the main text).
UCI Yaclt
UC
-3:45
GBS
Ecariar
Icarar
20
25
30
20
10
20
35
30
miroory 900
UCI Yaclt
kcater 25
-155
10
20
25
00
2.5
30
75
25.
125
25
59
100
350
Figure 7: Average test LML on further datasets not included in the main text. Evaluated for several memory sizes (top), and evaluated over time (bottom) for specific memory size. Cf. Sec. 6.1, App. A. for details.
Furthermore, test the memory update and Gaussian update of GRS separately on UCI Energy and UCI Concrete. For this purpose, i) we combine the k-center method with our Gaussian update from See. 3.3; and ii) we use our memory update from Sec. 3.2 and update the Gaussian distribution by optimizing Eq. (2) with Dr, U UM MMr, (re-fitting). The results are shown in Fig. 8. As can be seen, GRS performs better than one of the components used in combination with a baseline method. GRS with refit performs especially bad, similar to the baselines k-center and random. This is because refitting requires optimising the ELBO with small dataset. As mentioned in Sec. 3.3 (cf. Fig. 1), Bayesian neural networks with VB perform bad on small datasets due to over-regularisation. Consequently, in case of refitting, a good memory update can lead to a worse overall performance due to much worse Gaussian update. While this general issue with Bayesian neural networks (learned with VB) is beyond the scope of this work, it is an important future research direction.
UCI Energy
UCI
UCI Energy
31
0.3
23
0.90
GBG
3.8
with relts
Giis with part
GKG reli
CBS
-04
y
15
30
Concrate
Concrate
Concrate
Concrete
-33
-20
GRS 25
CEC wibh ve11
G15 with mill
Figure 8: Average test LML on UCI Energy (lop) and UCI Concrete (bottom). GRS denotes our approach (Sec. 3), GRS with k-center replaces our memory update by the k-center method; GRS with refit replaces our Gaussian update by the optimization of Eq. (2) with Dr, UM \Mag. Evaluated for several memory sizes (left), and evaluated over time (right) for 3 different memory sizes. Hyperparameters are chosen as See. 6.1.
14

---

Published as a conference paper at ICLR 2020
To better understand our memory update using the score function from Eq. (8), we visualise the running memory for a model trained on MNIST in Fig. 9.
7
2
3
7
9
7
7
7
4
S
9
7 2
3
7
S J 7
7
7
R
Y
F
7
Y
7 7
Z
7 C Z S 7 /
7
X
3
6
R
Q 7 Z 2 7
4 Y
Z
C
9
9
5 Y
7
Y
S 4 3 6
C
?
7
5
5 2 3
S
7 6
9
4
7
(a) f25
(b) tso
(c) f75
(d) 1100
Figure 9: Running memory at different time-steps on MNIST (cf. See. 6.1,, with a memory size = 100. The memory update tends to select non-typical data while showing diversity.
A.2 ADAPTATION
Weete-prooee procee
Oriote -- J.iubbbk process
Sayesian forgetting
Bayas an lorgeting with memory
4.3
-2.0
-26
-0.9
0,333
0.7
0.5
3.0
27
2.5
1.0
05
2.5
3.5
2.0
0.5
10
20
3tPhoo
Wiennr
Drratsin Unlenbeck process
Bayeslar forpetting
Sayedan forgeting with
-0.42
-0403
-100
-0-42
-3.05
4TTeraa 36-2
24-2
Dogating BEAGG
1e-2
Figure 10: One-step ahead LMI. Gas Sensor Array Drift dataset (top) and Weather dataset (bottom). Subplots show 3 different adaptation methods (left), evaluated for several values of the respective adaptation parameter, and Bayesian forgetting with e = 0.0095 (Gas Sensory Array Drift) and = 0.031 (Weather), evaluated for multiple memory sizes (right).
We also evaluated our adaptation methods on 2 additional datasets (Gas Sensor Arra Drift, Weather). In Fig. 10, we visualise the influence of the adaptation parameter for these datasets. Note that the range of the adaptation parameters is on a much smaller range compared to the experiments on Covertype (Sec. 6.2). For larger values, the performance starts to degrade. Surprisingly, the memory degrades the performance in case of the Gas Sensor Array Drift dataset.
A.3 CATASTROPHIC FORGETTING WITH ONLINE VB AND BAYESIAN NEURAL NETWORKS
Here we provide further experimental results for the behavior of online VB (Secs. 2.1, 3) in case of non-stationary data. For this purpose. we train Bayesian neural networks with different architectures on the toy classification problem with a rotating decision boundary from Sec. 6.2. however, with 150 data samples per time-step. In Fig. 11, we visualise the training LML for different architectures. including linear model. It can be seen that Bayesian neural networks with higher complexity (i.e. more layers or more units) drop slower in performance compared to the linear model. However, this is not desired property for online VB, since exact online Bayesian inference would yield the same posterior distribution as offline Bayesian inference. In case of our toy classification data (where the time dependence is ignored), online inference should not be able to classify the data as t + co. Instead, this learning behavior shows that online VB with Gaussian approximate posterior distributions is prone catastrophic forgetting.
15

---

Published as a conference paper at ICLR 2020
-0.5
-0.-
-00
140.4
700
(a) no adaptation (1 layer)
(b) no adaptation (2 layers)
(c) adaptation (3 layers)
-64
-4A
-25
-0.5
-0.
(d) BF and OU (2 layers)
(e) BF with memory
(I) no adaptation with memory
Figure 11: LML for toy classification problem with rotating class boundaries (ef.Sec. A)
EXPERIMENT SETUP
The following is an explanatory list of the update methods used in Sec. 6.1:
k-center (VCL): Uses the k-center method (Sec.2.2) for the memory update and Eq. (2) with (Dra U Mix-1) \ Me for the Gaussian update.
random (VCL): Uses random selection (Sec.2.2) for the memory update and Eq. (2) with (Di,, U MM Mr, for the Gaussian update.
GRS (Gaussian Residual Scoring, ours): Uses Eq. (8) for the memory update (Sec. 3.2) and performs the Gaussian update by first using Eqs. (2) with (Dr, U MM,., and subsequently using Eqs. (5a), (5b) for removing the local contributions of Mr, (cf. Sec. 3.3).
Similarly, the following list summarises the adaptation methods used in Sec. 6.2:
Wiener process: Posterior approximation consists of ge (w) only. Transition pl Waze (w,.) is given by a random walk. We used diffusion that is proportional to the prior standard deviation in every neural network layer (cf. Sec. 4.2). No memory used.
Omsteen-Uhlenbecc process: Posterior approximation consists of qa,, (w) only. Transition p(Wre4a (w,.) is given by the
process (cf. Sec. 4.2). No memory used.
Bayesian forgetting: Posterior approximation consists of qe (w) only, No state-space model assumption, instead uses Bayesian exponential forgetting (ef. Sec. 4.1).
Bayesian forgetting with memory: Posterior approximation consists of qe (w) and M₁. No state-space model assumption, instead uses Bayesian exponential forgetting (ef. Sec. 4.1).
In Tab. 2, we summarise experimental setup (hyperparameters) used for Secs. 6.1 and 6.2.
C FACTORISATION PROPERTY OF THE GAUSSIAN VARIATIONAL APPROXIMATION
Here derive the factorisation property of the Gaussian variational approximation distribution by expressing the natural parameters of the Gaussian approximation as a sum. This can be shown for the Gaussian approximation a local optimum of the BLBOO. For Gaussian prior and posterior the
16

---

Published as a conference paper at ICLR 2020
Table 2: Experiment setup for online experiments. Nt, is the number of observed samples at the first time-step and Nt., is the dataset size of all other time-steps. M refers to the number of samples in the memory. We evaluated different memory sizes for each experiment for experiments in Sec. 4, where the sizes are equally spaced in the given range. Kminn and Kiem the number MC samples used for training and for estimating the Gaussian terms respectively. It and I refer to the number of iterations for the first time-step and all subsequent time-steps, respectively. The architecture denotes the number of units for each hidden layer (e.g. [16, 16) denotes 2 hidden layers with 16 units each).
Sec. and 10 sizes
No
N
M
Kunn
K
Ito
12
Architecture
Concrete
100
10
55.. 25
1k
50k
50k
10k
[16, 16)
Boston
100
10
5... 25
1k
50k
50k
10k
16. 16
Energy
100
10
5... 25
Ik
50k
50k
10k
16, 16
Yacht
100
10
5... 25
IK
50k
50k
10k
16,
Spam
250
25
[10... 50]
400
50k
50k
10k
32, 32
Wine
250
25
[10... 50]
400
50k
50k
10k
32,32
MNIST
2.5k
250
[50 250]
40
50k
50k
10k
64,64
Arrr
500
50
5 50
200
100k
50k
10k
8,
Weather
1k
100
50
100
100k
50k
10k
16, 16
Covertype
1k
1k
200
400
100k
50k
10k
82,32
ELBO is given as
L(n"E"') = - log Eol - loog |Σ7|
d (--- -mo)² 2n (4"
" 100 + (r()"
E ww (w) logg(d(")/w))
+
At a local optimum, we have
ac(=",E)
which yields
Bn
N
Ew (w)
logp(d(")/w) = Eo'(2² 1o)-
Op
n=1
Hence, we obtain
N
wa (w) logp(d(>)ww)
4 = 14 + Eg
E
(14)
0pm
n=1
Similarly, have 0c("" 2')
O. which yields
N
Σ
E wa (w)
logr(d(")/w) = (E") nー1
(15)
2
Hence, we obtain
Σ"
E
2
E
wa (w)
log (
(16)
Next, we calculate the natural parameters from Eqs. (14), (16):
N
A = Ao +
2 +.
E Ew (w) log p( (n))
==1
Ag +
A (n).
17

---

Published as a conference paper at ICLR 2020
17¹ A²p
Ao +
Σ 1
(n)
1
=Ao Au((H (u Mo + Σn
N
0
log ( + +E A 4
E
Ew qo- (w)
0y
N
=Affo =
+
Ew-go- (w)
logp(d("/ww) + A (2)
0p
==1
N
(m)
1oo +
Monte Carlo estimation: The natural parameters A r7(*)) can be estimated with Monte Carlo, by replacing the expectation with an empirical mean. Since the parameters A and Ag (and 17", 7o respectively) are known, the total bias of the parameter estimates can be computed:
N
Ab = (A* - Ao) - AA (
Tb = (17 - -170)
(m)
n=1
n=1
We use this to reduce the bias for the individual terms:
A (m) A())
Ab
1)(m) = (x)
1
mb.
N
N
D ELBO RESIDUALS FORM
Here we show how the ELBO can be written in the form of Eq. (6). Let us define the variational
distribution in the factorised form go(w) L p(w) II r(a)(w) (cf. Sec.3.1). We can then write the ELBO as
N
C(,, D) = (wp(
log (() w) + log p(w) log go (w) Σ
(log w) + logr(")(w)) log r(n) (w)) + log p(w) log (w)
(w)
p(w) I (())(w)
(w))
(log (d) /w) - log (or((((())) log
40 (w())
Zy qo (w)
(w)
Σ
(log (() w) log (()) + log
go (w)
log Zg
(w)) (og
()) w) log logr(()) (w]]
E MEMORY UPDATE SCORE FUNCTION
In Eq. 8, the expectation involving Gaussian terms can be calculated analytically:
Egg, (w) logr (m) (w) = t
f To., (n) 1 (1) wnnww?)ww
2
(n())) 1 (m)) (110)2 2 + E²)
"m (1 1 A (m) ((A7)-1,))² - A
1
2
2
The expectation involving non- estimate Eig,, (w) [log ((d ww] using Monte-Carlo.
non-Gaussie Gaussian terms (in Eq. 8) has no closed-form solution. We therefore
18

---

Published as a conference paper at ICLR 2020
F BAYESIAN FORGETTING RECURSIVE FORMULATION
Here we show how Bayesian forgetting can be rearranged into recursive formulation. We first bring this formula into similar form as Eq. (I), extracting the most recent likelihood term:
K+1
p w/
[(wpDr ( Po(w)-
II p(D,, (w) k=1
II pD (w) 1K---
(w)
(Dexx 1 I/w).
The first two terms can be rewritten as
K
[+1 U/
II p Di. /w) lots lkiik in
K
Pw(w)- II P(Drr
Po(w)
km
Ri
(w) II p(Dr, /w) 3n (1--) 1x01--00
k=1
(D /w) (1-r) - 20-4 (1-e) 1x+1
K II k=1 (p(w]] (erste))
=po(w)-
(1-e) 1K+1 1K41-KK
oc (w)
(w)
lK+1
Po (w))=-11-)
DoW/Ds,,oo st ) (1-c)
Hence, we have shown that the posterior can be expressed recursively as
p( p(w|Drrs (D (1-e)- 11/-
p(w (e )(- - p (Diesslo Iw).
The parameters the Gaussian part q0,, (w) of the posterior approximation (after applying the forgetting operation) can be caleulated easily from the above equation.
Natural parameters:
1 Ao ((1 (11-0) (c) --:4t
14+1-16
Aeti Ao. (1--1--6
Ma241 =10 -1 (1- 1 + mu ((1 c1 2
Covariance parameter:
Etsts = An (1 (1 -1-68 () 1
+ Are ((1 )
-1
Eo (1- (1 - c) 1:1l-
Σny'' ((1 مليي
Location parameter:
Px- Dtzts -7441
(70 (1--) 1044 4k
+ Men ((1 وليييعا
Stuts
44-1-1 + + Ene Men ((1 - c) 1t---
Etess Dit' Mo (1- -)
G PSEUDO-AL.GORITHM
We provide the pseudo algorithm of GRS (Sec. 3) with Bayesian forgetting in Alg. 1. The compu- tational complexity (at each of the K time-steps) is dominated by i) the minimisation of the KI.
19

---

Published as a conference paper at ICLR 2020
divergence, ii) estimating the Gaussian factors, and iii) scoring the memory. The KL minimisation requires Lt sequential iterations with N22 + M data samples and K Monte Carlo samples. The latter can both be processed in parallel parallel hardware. The estimation of the Gaussian factors requires Nt, + .M sequential iterations with Kum parallel Monte Carlo samples. The dominating computation for calculating the scores is the evaluation of the likelihood for No, M data samples and Kainn Monte Carlo samples, both of which can be processed in parallel. The highest scoring can- didate memory is given by the top-M highest scoring data points, thus, the computational complexity of Eq. (8) is only linear in the number of samples.
Algorithm 1 Gaussian Residual Scoring with Bayesian forgetting. The function EstimateGaussian Factors corresponds to Eqs. (5a), (5b) (cf. also App. C). The function ApplyForgetring corresponds to Eq. (12). Note that Pr. (w) includes the adapted likelihood of the memory and all subsequent functions involving the memory use this adapted likelihood.
Inputs: (w)).. 26. (w), K for I in []...K] do t = Get GetTimnStamp(At
Ale La 1--1
Dr, = (elData((4)
if 0 then
Pt. (w) po(w)
else
Pt. (w) = Apply ApplyForgetting(No(() 90., , Me Atx) (Sec. 4.1)
end if
go., (w) = argmin,, KL (L() Z Pcs (w) p(Dr, w)] (Sec. 3.2, Sec. 4.1]
(re, (w; du Idd, EDr, U. = EstimatteCauussinnaaacooo (āo,, (w), Des Me .) (Sec. 3.1)
Mr. argmax,, M St, (M; Dr,, U M (Sec. 3.2)
if Dr. = MMail then
ra (w) = Pie (w) Ild & MiN rea (w; dry)) (Sec. 3.3}
else
g (w) do., (w) /Ma E eMa re (w, (Sec. 3.3)
end if
end for
H PRIOR PARAMETERS
Here we develop a heuristic to choose the initial prior (w). As this will not be specific to the online continual setting, we drop the time index in this section, denoting the prior as p(w). Furthermore, we consider only Gaussian distributions with diagonal covariance matrix. Assume that the data standardised, that is, the first two moments are zero and one. A reasonable choice for the prior parameters is such that the first two moments of the prior-predicts moments of the data distribution. We go one step further and constrain the pre-activations of every neural network layer to have moments zero and one. Denote all weight matrices and weight biases by w = { [Wih U (bi)t, and let Xo denote the input data. Let us further denote the pre-activation (before non-linearity) of layer I and unit as follows.
distribution equals the first two
xi=
Σ (
The constraints are then given as follows.
Ew-p(w) Ex)=p(()) [x]] = O,
Ew-
E Bxo [(x)]]]
=
The first constraint can be easily fulfilled by setting the prior mean to zero for all parameters.
14 ij
0.
20

---

Published as a conference paper at ICLR 2020
This follows immediately from W IL fi-1 (x-1( and the expectation of products of independent random variables. The second moment can then calculated as follows.
E Ewwp(w)) (x.ppwwwwwxppppppp. x
(xx))]
E Bwwp(w)
E Ex)~ [xx)²]]
Ew-)(w)
Varxo~m(D)) [x]] +
Ni-1
-Ew-p(w)
Varxomp(D)
Σ
w fi (x bi]
Σ 1
(WiJ)² Varx)>p(D)) (26_1ح)
-ww)
Mi
(wrJ)²²
Σ Ewr-p(w)
Var Varxo~p(D) [ [(
M-1
Ewr=p(w) (w)
(w?:)²
Bwwx((w)
Varxo~p(D) [t- [x{_1)]
N-1
(w>ss)²
E Bw-p(w)
C/--1
Ni-1
=fh-1"
Σ
Varwwp(w)
W
= Cfi-1" Varw=p(w)
Here we introduced Cft-, to denote a correction factor for the non-lincarity fi-1- case of the linear function, will have Cf-1 1. For arbitrary non-linearisies, we can estimate this factor numerically, assuming that the pre-activations are distributed according to (o.N0.1)
Var Varx x [
c
This can be done beforehand and the factors for common activation functions can be stored in lookup table. Finally, plugging in the constraint for the second moment in the above equation, we obtain the following prior variance.
(o)²)²² Var Var (aw W
(17)
N1-10
I POSTERIOR INITIALISATION
It is known that proper initialisation of standard neural networks is crucial for the optimisation process Glorot Yoshua Bengio (2010); He et al. (2015). In Bayesian neural networks, the matter becomes
complicated, since have deal additionally with the variance of the Monte Carlo estimator due to c--paaamerisationn Analogous to the choice of prior parameters, we seek a posterior initialisation that yields the first two moments of zero and one. A naive attempt would be to initialise the posterior with the prior parameters. However, the significant noise in the Monte Carlo estimation typically leads to bad optimisation results and even numerical instabilities. We propose an initialisation method which fulfills our constraints but allows us determine the variance of the initial posterior with two hyperparameters a and B.
Let us denote the mean and log- log-scale parameters of the approximate posterior as 0 = {00m, P1og }.
We choose the following initialisation distributions.
<(w) -N(w; 20mme=)
where
p(0,)) (0 100, o2)):
21

---

Published as a conference paper at ICLR 2020
and
p(Olog o) (fhogg (0rogg Tion.
Here and in the following, we dropped the time index for the approximate posterior, as well as the indices 1, i, and j for the model parameters 0.
We follow a similar derivation as in Sec. H. As for the prior, the mean of the initialisation distribution will be zero for all parameters.
n = O.
For the second moment, the derivation is as follows.
Eg--((0) (w xo (D)) (xx))] = Ee) B Buwntwwo) EXOrD(D) (xx))
=E
Bwwo((w1))
Var Varxo>p(D)
Σ
Wio S1-1(2 +0
1
=E
Ewrng(w10)
(W/))32
Var (aUxx)>p(D) xf
Ni
= =E
Ewrog(w0))
(W/1)1)) Σ
Eamp(()
Bw-olw/())
Varxo~m(D) 1
(xc
Mi-
= Σ Eapp(() Euwa(wio)) (w])))²]
E
Ewwg(w 0)
Varxo~n(D) [11=1((([11]
3
NL-1
=: Σ Eemple) [02 + 200000)) cfi-1
Ni-1 Cfi EE-ppe) 03 2.20000
EE-pio)) [02] + -Enppio { (22
Ni-1 1
Ni-1 Ch-1"
(Ea-p(0) [0u]²+ Varomp(o) [0p] + 20/0rox o 2Var (0oox c)
pg. og. + 2pmoo +20
NN-1 Cfi
Tb te 2nonax +20
=Ni- Cf-i"
Hence, the second constraint is as follows.
2 00 + 2pein +20
N Cf
In contrast to Sec. H, we are now under-constrained by parameters. We therefore introduce two hyperparameters and 3. We first determine a := Obxxe' for which we generally choose small values = ( (a = 0 corresponds to initialising all posterior variances in the given layer with the same value). The second hyperparameter E [0, 1] determines how much the total variance is due to the variance of the location parameter and how much variance is due to the variance of the log-scale parameter. Inserting a and introducing we obtain the following equations.
00 =
N "
and
é 2/eiog
2
1-B B
Ni-1 Cft-1
22

---

Published as a conference paper at ICLR 2020
Solving the last equation for M the result is follows.
log (1 B3 e 2a²
P0og
N111 Cf--1
We choose a 0.001 and 3 0.999 in all experiments.
note on the relation to initialisation methods for deterministic neural networks. Our result is similar the initialisation methods from Glorot & Yoshua Bengio (2010) and He et al. (2015). The difference is in the correction factor Cfi-1' Whereas Glorot & Yoshua Bengio (2010) considers linear functions (or tanh in the linear regime), both methods base their derivation on the assumption that every data sample Xy is processed by different, random neural network with independent weights, drawn from the initialisation distribution. The assumption is made explicit in (He et al., 2015) by the use of the variance of products of independent variables rule. We note that this assumption is false for both the initialisation of deterministic neural networks, as well as the graphical model assumption in Bayesian neural networks. Consequently, (He et al., 2015) obtains different correction factors (in their case for relu and leaky relu), taking into account the mean after the forward- pass through the non-linearity.
23


Reviews:
[{"review_id": "rygdmkifqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for training Bayesian neural nets on a stream of non-stationary data. The authors propose a new way to approximate the variational distribution in the case where a memory (of past data) is used. The authors also use two forgetting mechanisms that allow the model to adapt to the changes in the data distribution.\n\nNote that contrary to many recent works on continual learning that focus on not forgetting previous distributions, the goal here is to predict the current data.\n\nI found this paper to be well written, the contribution is clear and the background material is well explained. The task is also important. I am not sure about the significance of the proposed approach. The motivation for providing a new memory update that takes into account the approximation error (from the variational approximation) is clear but I was not sure of the efficacy of the approach. Experimental results (Table 1) also show modest gains.  Experimental results using the adaptive method (Figure 5 & 6) are somewhat similar (e.g., why are other benchmarks not studied? gains are also modest).\n\n- In Table 1, what does \"bolding\" indicate?\n- In terms of other benchmarks, I was curious as to why you did not compare to \"The Population Posterior and Bayesian Modeling on Streams\" NIPS'15. As far as I understand this is also a Bayesian method that can accommodate non-stationary streams.\n- VLC is proposed for the case of continual learning without forgetting.  It seems like you could study your method in such a setting (e.g., using online multi-task learning).\n- In Section 4 (p.5 2nd paragraph), you mention that you assume that task boundaries are given. This seems to be a strong assumption and a significant change wrt to previous work. Could you perhaps add a practical example of this setup?\n- I think it would be worthwhile to provide the full proposed algorithm somewhere in the paper and also discuss its computational complexity explicitly.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyeVlFw1qr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\nThe paper extends variational continual learning with memory. In this setting the posterior distribution of the model parameters is approximated using a mean-field Gaussian approximation and a small set of datapoints is kept in a memory to combat catastrophic forgetting.\n\n- The paper proposes a new rule for updating the memory and the Gaussian approximation after examining each batch of points. For the memory update, instead of the previously used k-means or random sampling, it examines the factor 'r(d|w)' that each datapoint contributes and selects the most significant ones.\n\n- A second contribution is the use of two adaptation methods in the online learning setting in case there is a distribution shift in the data: Bayesian forgetting and the Ornstein-Uhlenbeck process.\n\n(1) The first contribution, the new memory update rule and the Gaussian update, is novel to my knowledge. The idea is to calculate the Gaussian factor that each datapoint contributes to the posterior (the update corresponding to each datapoint in the message-passing interpretation) and the select the points that contribute the most. Effectively, it calculates the change in the ELBO if each candidate datapoint was moved to the memory and selects the set that minimizes the ELBO of the remaining points. This  is a sensible heuristic because we want to keep datapoints in the memory that contribute the most to the posterior approximation. If a datapoint contributes nothing to the posterior then it shouldn't be kept in memory.\n\nThe idea clever and it seems to work well in practice. Questions:\n- The Gaussian that each datapoint contributes is an approximation in the sense that if we take that datapoint and its contribution away, the remaining approximate posterior might be suboptimal. This has the consequence that S_tk (eq. 8) is only an approximation to what the optimal ELBO would be without the datapoints selected for the memory. Did the authors conduct experiments or have thoughts on how well S_tk approximates the ELBO of the variational approximation of the remaining points?\n\n-Regarding the experiments, there is a comparison to k-centre and random selection, both of which are proposed in VCL. I am very surprised to see in the experiments that both of these are outperformed by 'no-memory'. For example on 'energy', the baselines at t50 are outperformed by the non memory model at t0. Can this all be contributed to the baselines training their variational approximation without the memory at t0?\n\n- It is unclear whether the gains can be contributed to the new Gaussian update rule or the new memory update rule. To see the contribution of each, there should be an experiment where the proposed Gaussian update rule is used along with k-centre for the memory update. The fact the the no-memory model performed to close to GRS suggests that k-centre with the new Gaussian update rule would be even closer to it.\n\n(2) The second contribution of the paper is the use of Bayesian forgetting and OU to use to deal with the data distribution shift. It shows how these two approaches can be used along with the proposed Gaussian and memory updates. The paper is already very long, so there likely isn't any space for fleshing out this section, but it would be nice to have the experimental results included in the main paper, because most of the experiments are left to the appendix. Perhaps it would be a good idea to focus only on Bayesian forgetting (or the OU process) and try to shorten the section a bit.\n\nIn terms of writing and clarity, I have no complaints. The paper is well written and easy to understand.\n\nMinor:\n- 3.2 'If the likelihood term p(dtk|w) is well approximated by r(w;dtk)' -  I find this sentence a bit confusing. What does it mean for p(dtk|w) to be well approximated by  r(w;dtk)?\n- 3.3 'In order to reduce the variance of the Monte-Carlo approximation'. What is the MC approximation made here?\n- The method never actually uses the assumption that the datapoints are sampled i.i.d.. The algorithm should still work if the datapoints are not examined in a random order, e.g. consider seeing all the images with label '0' and then all the images with label '1' etc.. This would likely degrade the performance significantly but the method should still work.\n\nOverall assessment:\nPros: I like the ideas in the paper and they are presented well.\nCons: The paper is a bit too long. The experiments could more thoroughly investigate the source of the gains.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xtbmP2FB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-rebuttal: My questions below have been addressed and the submission has been modified accordingly. My concern regarding novelty remains unchanged but I still suggest acceptance since the contributions are of practical interests and the paper is well written.\n\n1. Summary:\nThis proposes considers neural networks training with non-stationary streaming data. To address online inference, the paper uses variational online updates and a running memory (coreset) summarising the data points seen so far, as recently used in the variational continual learning approach. First, the paper identifies a gap in this recent approach that coreset point selection process does not take into account the approximation quality. The paper develops a sound procedure to select the memory points, essentially to identify “difficult” data points under the current posterior and put them into the running memory. Second, to handle “concept drift”, the paper considers imposing some decaying on the likelihood of past data and derives post-hoc variational posterior updates for this case. Two contributions are validated on some streaming regression and classification tasks.\n\n2. Opinion and rationales\n\nI’m leaning towards “accept” for this paper since it presents two interesting contributions (albeit of incremental novelty)  to the approximate inference area, it has clear execution and super clean presentation, and the experiments clearly demonstrate the values of the proposed approaches.\n\nI would like the paper to clarify the following:\n\na. I find the objective for selecting memory points interesting, but am wondering if using individual predictability terms by themselves are good enough to select these points. Perhaps, after some thoughts, memory is perhaps not the right word to characterise these points. \n\nIt seems to me (i) these terms indicate how well the current posterior predicts the data points and thus this objective will tend to favour points with low predictive likelihood to be selected. That is, these points are important when *all other points* are presented, (ii) there will be *no diversity* in the memory as illustrated in the 2d classification example.\n\nThis means the points selected here have quite different characteristics compared to coreset points or the full dataset in general. Coreset points ideally can compactly represent the full dataset and can be used for inference in place of the full dataset.\n\nThat said, the contribution presented here is very useful. Just that I’m not sure how well this will work in more challenging continual learning set-up where diversity is important for a long sequence of diverse tasks.\n\nb. The paper presents a post-hoc modification to approximate posterior (‘s mean and variance) to account for the decay in the likelihood contribution from past data. I wonder if this post-hoc adjustment is identical to running approximate inference with the adjusted likelihood?\n\nc. the adaptation parameters need to be tuned/known in advance, which seems to be a disadvantage of the current approach. Is the update presented here somewhat robust to mis-specification of these parameters? How would these be tuned in a more practical setting in which data arrive sequentially and we might not know the underlying “concept”.\n\n3. Minor details:\n\nProxys -> proxies\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an algorithm for online Bayesian learning of both streaming and non-stationary data.  The algorithmic choices are heuristic but motivated by sensible principles.  The reviewers' main concerns were with novelty, but because the paper was well-written and addressing an important problem they all agreed it should be accepted.
Decision: Accept (Poster)