Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method for training Bayesian neural nets on a stream of non-stationary data. The authors propose a new way to approximate the variational distribution in the case where a memory (of past data) is used. The authors also use two forgetting mechanisms that allow the model to adapt to the changes in the data distribution.

Note that contrary to many recent works on continual learning that focus on not forgetting previous distributions, the goal here is to predict the current data.

I found this paper to be well written, the contribution is clear and the background material is well explained. The task is also important. I am not sure about the significance of the proposed approach. The motivation for providing a new memory update that takes into account the approximation error (from the variational approximation) is clear but I was not sure of the efficacy of the approach. Experimental results (Table 1) also show modest gains.  Experimental results using the adaptive method (Figure 5 & 6) are somewhat similar (e.g., why are other benchmarks not studied? gains are also modest).

- In Table 1, what does "bolding" indicate?
- In terms of other benchmarks, I was curious as to why you did not compare to "The Population Posterior and Bayesian Modeling on Streams" NIPS'15. As far as I understand this is also a Bayesian method that can accommodate non-stationary streams.
- VLC is proposed for the case of continual learning without forgetting.  It seems like you could study your method in such a setting (e.g., using online multi-task learning).
- In Section 4 (p.5 2nd paragraph), you mention that you assume that task boundaries are given. This seems to be a strong assumption and a significant change wrt to previous work. Could you perhaps add a practical example of this setup?
- I think it would be worthwhile to provide the full proposed algorithm somewhere in the paper and also discuss its computational complexity explicitly.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Contributions:
The paper extends variational continual learning with memory. In this setting the posterior distribution of the model parameters is approximated using a mean-field Gaussian approximation and a small set of datapoints is kept in a memory to combat catastrophic forgetting.

- The paper proposes a new rule for updating the memory and the Gaussian approximation after examining each batch of points. For the memory update, instead of the previously used k-means or random sampling, it examines the factor 'r(d|w)' that each datapoint contributes and selects the most significant ones.

- A second contribution is the use of two adaptation methods in the online learning setting in case there is a distribution shift in the data: Bayesian forgetting and the Ornstein-Uhlenbeck process.

(1) The first contribution, the new memory update rule and the Gaussian update, is novel to my knowledge. The idea is to calculate the Gaussian factor that each datapoint contributes to the posterior (the update corresponding to each datapoint in the message-passing interpretation) and the select the points that contribute the most. Effectively, it calculates the change in the ELBO if each candidate datapoint was moved to the memory and selects the set that minimizes the ELBO of the remaining points. This  is a sensible heuristic because we want to keep datapoints in the memory that contribute the most to the posterior approximation. If a datapoint contributes nothing to the posterior then it shouldn't be kept in memory.

The idea clever and it seems to work well in practice. Questions:
- The Gaussian that each datapoint contributes is an approximation in the sense that if we take that datapoint and its contribution away, the remaining approximate posterior might be suboptimal. This has the consequence that S_tk (eq. 8) is only an approximation to what the optimal ELBO would be without the datapoints selected for the memory. Did the authors conduct experiments or have thoughts on how well S_tk approximates the ELBO of the variational approximation of the remaining points?

-Regarding the experiments, there is a comparison to k-centre and random selection, both of which are proposed in VCL. I am very surprised to see in the experiments that both of these are outperformed by 'no-memory'. For example on 'energy', the baselines at t50 are outperformed by the non memory model at t0. Can this all be contributed to the baselines training their variational approximation without the memory at t0?

- It is unclear whether the gains can be contributed to the new Gaussian update rule or the new memory update rule. To see the contribution of each, there should be an experiment where the proposed Gaussian update rule is used along with k-centre for the memory update. The fact the the no-memory model performed to close to GRS suggests that k-centre with the new Gaussian update rule would be even closer to it.

(2) The second contribution of the paper is the use of Bayesian forgetting and OU to use to deal with the data distribution shift. It shows how these two approaches can be used along with the proposed Gaussian and memory updates. The paper is already very long, so there likely isn't any space for fleshing out this section, but it would be nice to have the experimental results included in the main paper, because most of the experiments are left to the appendix. Perhaps it would be a good idea to focus only on Bayesian forgetting (or the OU process) and try to shorten the section a bit.

In terms of writing and clarity, I have no complaints. The paper is well written and easy to understand.

Minor:
- 3.2 'If the likelihood term p(dtk|w) is well approximated by r(w;dtk)' -  I find this sentence a bit confusing. What does it mean for p(dtk|w) to be well approximated by  r(w;dtk)?
- 3.3 'In order to reduce the variance of the Monte-Carlo approximation'. What is the MC approximation made here?
- The method never actually uses the assumption that the datapoints are sampled i.i.d.. The algorithm should still work if the datapoints are not examined in a random order, e.g. consider seeing all the images with label '0' and then all the images with label '1' etc.. This would likely degrade the performance significantly but the method should still work.

Overall assessment:
Pros: I like the ideas in the paper and they are presented well.
Cons: The paper is a bit too long. The experiments could more thoroughly investigate the source of the gains.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Post-rebuttal: My questions below have been addressed and the submission has been modified accordingly. My concern regarding novelty remains unchanged but I still suggest acceptance since the contributions are of practical interests and the paper is well written.

1. Summary:
This proposes considers neural networks training with non-stationary streaming data. To address online inference, the paper uses variational online updates and a running memory (coreset) summarising the data points seen so far, as recently used in the variational continual learning approach. First, the paper identifies a gap in this recent approach that coreset point selection process does not take into account the approximation quality. The paper develops a sound procedure to select the memory points, essentially to identify “difficult” data points under the current posterior and put them into the running memory. Second, to handle “concept drift”, the paper considers imposing some decaying on the likelihood of past data and derives post-hoc variational posterior updates for this case. Two contributions are validated on some streaming regression and classification tasks.

2. Opinion and rationales

I’m leaning towards “accept” for this paper since it presents two interesting contributions (albeit of incremental novelty)  to the approximate inference area, it has clear execution and super clean presentation, and the experiments clearly demonstrate the values of the proposed approaches.

I would like the paper to clarify the following:

a. I find the objective for selecting memory points interesting, but am wondering if using individual predictability terms by themselves are good enough to select these points. Perhaps, after some thoughts, memory is perhaps not the right word to characterise these points. 

It seems to me (i) these terms indicate how well the current posterior predicts the data points and thus this objective will tend to favour points with low predictive likelihood to be selected. That is, these points are important when *all other points* are presented, (ii) there will be *no diversity* in the memory as illustrated in the 2d classification example.

This means the points selected here have quite different characteristics compared to coreset points or the full dataset in general. Coreset points ideally can compactly represent the full dataset and can be used for inference in place of the full dataset.

That said, the contribution presented here is very useful. Just that I’m not sure how well this will work in more challenging continual learning set-up where diversity is important for a long sequence of diverse tasks.

b. The paper presents a post-hoc modification to approximate posterior (‘s mean and variance) to account for the decay in the likelihood contribution from past data. I wonder if this post-hoc adjustment is identical to running approximate inference with the adjusted likelihood?

c. the adaptation parameters need to be tuned/known in advance, which seems to be a disadvantage of the current approach. Is the update presented here somewhat robust to mis-specification of these parameters? How would these be tuned in a more practical setting in which data arrive sequentially and we might not know the underlying “concept”.

3. Minor details:

Proxys -> proxies

review_title: Official Blind Review #2

Meta-review: This paper introduces an algorithm for online Bayesian learning of both streaming and non-stationary data.  The algorithmic choices are heuristic but motivated by sensible principles.  The reviewers' main concerns were with novelty, but because the paper was well-written and addressing an important problem they all agreed it should be accepted.
Decision: Accept (Poster)