Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
The authors propose a method to perform continual learning with neural networks by incorporating variational Gaussian Processes as a top layer (also called Deep Kernel Learning) and constructing an objective utilizing the inducing inputs and outputs to memorize across tasks.
They further study ways to approximate this behavior with weight space models and use their model for task boundary detection by utilizing statistical tests and Bayesian model selection.
Experiments show good performance of their method.

Comments:
1. The mathematical formulation of the basic model is very elegant. However, it is not immediately clear to me that the joint ELBO across successive tasks is still lower-bounding the actual objective.
2. The paper is well written overall.
3. To the best of my knowledge using such a model for task boundary detection is novel and quite interesting. There are obvious links to Bayesian changepoint detection in the timeseries setting. Possibly these links would be made more clear by a citation to a recent paper such as Spatio-temporal Bayesian On-line Changepoint Detection with Model Selection by Knoblauch and Damoulas, or any other paper of similar content. The link is quite fascinating.
4. Sections 2.3 and 2.4 of the paper are the weakest points and quite unsatisfactory as they forgo the elegance of the proposed approach to do "something else" that Sec. D explains how to salvage with "tricks". Especially with regards to Sec. 2.4, why can't we just do inference on Z_i and have to pick datapoints via discrete optimization? That comparison would be useful in the experiments. Furthermore, recent papers utilizing GPytorch by Gardner et al have dramatically sped up GP inference. Could we aim to make the original idea fast enough to be used instead of resorting to an approximate model with weight spaces and corrections to extract Z and u per task?
5. The experiments are good, but very focused on MNIST tasks. I would appreciate tasks of different structure given how well the method appears to work.


Decision:
I find the basic idea of the paper quite appealing as it leverages the elegance of the deep kernel learning formulation to yield an attempt at a principled Bayesian version of continual learning and demonstrates empirical value. 
Some discussion on the objective might be warranted to demonstrate that it actually lower bounds the true LLK.
I am quite happy with the task boundary detection section and would encourage the authors to strengthen the link to changepoint detection.
My biggest qualms with the paper are that it departs from that strategy and performs weight space inference for training per task and then "corrects" to move back to the GP representation. A more convincing discussion would be welcome here.
The experiments are functional and show good results, but I would appreciate more diversity in the tasks.
As the paper stands I learn towards recommending acceptance and would strongly encourage the authors to iron out the weaknesses of the paper.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors propose a function-space based approach to continual learning problems (CL), wherein a learned embedding

    $\hat{\mathbf{x}} = \text{NN}(\mathbf{x}; \theta)$

is shared between task-specific GPs s.t.

    $f_{i}(\mathbf{X}) \sim \mathcal{N}(\mathbf{0}, k_{i}(\hat{\mathbf{X}},\hat{\mathbf{X}}))$, 

where the $i$-th task's covariance $k_{i}$ is a defined via standard variational inducing points methods. CL manifests as KL divergences between tasks' variational posteriors $q_{i}$ and their respective priors $p_{i}$. Since the embedding helps define $p_{i}$, its parameters $\theta$ are regularized to promote sharing.

The work investigates both practical and theoretical implications of this setup. On the practical side, the authors discuss enhanced 'on-task' inference via hybridization of function- and weight-space based approaches and, subsequently, strategies for optimizing inducing points. Additionally, a novel approach for automatically detect task switching is introduced that exploits the Bayesian aspects of the proposed framework.

On the theoretical side, points of (personal) interest revolved around differences between weight- and function-space approaches to CL. Here, I think that streamlining the presented argument would go a long ways. Paraphrasing, one of the authors' key insights is that:

  1) CL in weight-spaces is hard, since weights' semantics are moving target that change along with shared parameters.
  2) CL in function-space is easy, since the functions (i.e. tasks) themselves remain the same.

This information is provided in the introduction, but (as a relative newcomer to CL) I failed to connect regularization and rehearsal/replay based methods with the aforementioned spaces. It was only upon reading Sect 2.5 that this intuition 'clicked' for me. Hence, I suggest making this observation as obvious and intuitive as possible.

The provided experiments seem reasonable and do a good job highlighting different facets of the paper. Two additional results would be appreciated:

  a) How well calibrated are FRCL-based classifiers?
  b) How impactful is the hybrid representation (Sect 2.3) for test-time performance?

GP approximations formulated solely in terms of weighted sums of (finitely many) basis functions typically suffer from degradation of predictive uncertainties. Since one often motivates use of GPs via a desire for well-calibrated uncertainty, (a) seem quite pertinent.


Nitpicks, Spelling, & Grammar:
  - Lots of run-on sentences; consider breaking these up.
  - Introductory modifying phrases are missing commas.
  - Consider citing other recent works that use NN basis functions in conjunction with Bayesian Linear Regression.
  - Various missing or superfluous words resulting in some garbled sentences, e.g.:
      - "... our approach looks constraining."
      - "The ability to detect changes based on the above procedure comes from that in"
      - "While the task boundary detection results for Omniglot are less strong, which may due to the smaller batch size (32 for Omniglot, ≥ 100 for the MNIST-versions), resulting a noisier test result."

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper develops a continual learning method based on Gaussian Processes (GPs) applied in the way introduced by prior work as Deep Kernel Learning (DKL). The proposed method summarizes tasks as sparse GPs and use them as regularizers for the subsequent tasks in order to avoid catastrophic forgetting. Salleviating the instability resulting from the representation drift.

Employing inducing point training for task memorization is a novel and interesting idea, which could be useful for the continual learning community. The fact that this approach also captures the uncertainty of the replays contributes fairly to robustness. Lastly, performing knowledge transfer by inheriting the KL term of the ELBO is also interesting, however, its theoretical implications deserve a close look. It would be enlightening to analyze which true posterior the learned model then corresponds to. Would not it be a slightly more principled Bayesian approach (i.e. one that has stronger grounds at first principles) to perform the knowledge transfer to assign the posterior of one task as the prior of the other, alternatively to keeping the entire KL term intact which employs the q(u_i) as the surrogate for q(u_j), i.e. the way introduced by Nguyen et al., 2017?

The presentation clarity of the paper is open for improvement. For instance, the abstract is written in a sort of convoluted way. I do not get how the KL divergence suddenly kicks in and for what exact purpose. Is it variational inference or a hand-designed regularization term?

I find the argumentation from Eq. 1 downwards until the end of Sec 2.1 on BNNs with stochastic weights and their relation to GPs a bit unnecessary complication. These are very well known facts. It would suffice to state briefly that the task learner is a vanilla DKL used within a vanilla sparse GP.

Figure 1 is also not so descriptive. I do not get what the GP here is exactly doing. What is input to and for which output modelity does it find a mapping? What is the calligraphic L in the figure? Is it a neural net loss or an ELBO? 

In general I could not grasp why it makes sense to treat the the output layer params of a neural net treated for continual learning? They will not be sufficient to encode a task anyway, as an expressive enough neural net will leave only a linear mapping to the final layer. What happens if the intermediate representations of the input observations require a domain shift as the tasks evolve?

Overall, the presented ideas are fairly interesting and the experimental results are good enough for proof-of-concept, though not groundbreaking (behind the close relative VCL on MNIST and no comparison against VCL on Omniglot). Hence, this is a decent piece of work that lies somewhere around the borderline. My major concern is that the proposed method is conceptually not novel enough compared to Nguyen et al., 2017. My secondary concern is that the presentation is very much open to improvement in points hinted above.

--
Post-rebuttal: Thanks to authors for their tremendous effort to alleviate my concerns. The fact is that the conceptual novelty of the paper is too slim compared to VCL. As mentioned above, I even find the VCL approach more principled. I could have viewed the outcome of the paper as a slightly bigger news for the community if there was something unexpectedly positive on the reported results. However, as it appears from the comment below, the authors propose a super close variant of VCL that combines a few well-known techniques in a rather straightforward way and achieves in the end a model that performs on par with it. Under these conditions, I have hard time to find a reason to champion this paper for acceptance. That being said, I view this paper a tight borderline case due to its technical depth, hence I will not object to a reject decision either.
review_title: Official Blind Review #2

Meta-review: The authors introduce a framework for continual learning in neural networks based on sparse Gaussian process methods. The reviewers had a number of questions and concerns, that were adequately addressed during the discussion phase. This is an interesting addition to the continual learning literature. Please be sure to update the paper based on the discussion.
Decision: Accept (Poster)