Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose online variational conditioning (OVC), a procedure for efficiently conditioning Stochastic Variational Gaussian Processes (SVGPs) in an online setting that does not require re-training through the evidence lower bound with the addition of new data. The authors propose to efficiently update a posterior distribution after receiving new data; they rely on the idea of a Sparse Gaussian Process Regression (SGPR) model trained on pseudo-data under a block-diagonal Gaussian likelihood, they also provide an extension to non-Gaussian likelihoods by means of the Laplace approximation. The authors show the application of their method in look-ahead acquisitions for Bayesian Optimisation. 
main_review: $\textbf{Originality:}$ 
The work is novel in the way the authors associate the work of “Streaming Sparse GPs” (O-SGPR) with the well-known idea of a Sparse Gaussian Process Regression (SGPR) model. They  highlight that an O-SGPR can be seen as the SGPR  being trained on pseudo-data under a block-diagonal Gaussian likelihood. Additionally, the authors provide an extension to non-Gaussian likelihoods by means of the Laplace approximation.

$\textbf{Quality:} $
Section 3.1 presents the main theoretical analysis and quality contribution of the paper. The section of experiments explores various important scenarios to show the advantages of the method. Nonetheless, in sections 2 and 3,  the work presents some mathematical typos and a few variables are not properly introduced. Also, the authors are probably missing a broader discussion about the results obtained. For instance: 

-Figure 3(a) shows that Exact GP+NEI presents the better performance than Exact GP+KG. We could intuitively expect that SVGP+NEI also presented a better performance than SVGP+KG. Although, when using SVGP+NEI we obtain the poorest performance; in contrast, the SVGP+KG achieved the best performance. It might be useful to discuss about such effect. What changes when using Exact GP+NEI and SVGP+NEI? Why does it degrade the performance?

On the other hand, Figure 1 presents an scenario that makes me wonder about the quality of the approach:

-The authors claim the following regarding Figure 1c: “the SVGP is still able to update its posterior over the latent volatility in response to new data without “forgetting” old observations.”. Nonetheless, the figure 1c shows that the mean predictions and error bars differ much between the top and bottom figures for the old batch of data. Shouldn’t we expect the same mean and error bars behaviour (in top and bottom figures 1c) for the old batch of data, since the model does not “forget” old observations? This figure is lacking of a proper discussion about it. 

$\textbf{Clarity:} $
The paper is not well organised, the writing style seems not to be homogeneous along the sections. Particularly, the way the sections 1, 2 and 3 are written differ much of the writing style of the section of experiments. There are some confusing phrases along the document, some acronyms are multiple time introduced or not even introduced. There are some inadequate mathematical notations in the paper and a possible error in equation (3) (Please see the detailed comments and recommendations  at the end after “Significance”). The work lacks of an appropriate connection between the experiments section and the previous methodology sections. For instance, section 2.3 “Bayesian Optimization and Monte Carlo Acquisitions” seems to be isolated. Equations (3), (4), (7) and (8) are never cited in the text. In the experiments sections would be important to mention the use of the Algorithm 1 presented in the paper. Likewise, it would be worthy to link such experiments sections with sections 2 and 3.  

-Possible error in Eq. (3). The authors refer to the work of Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” for Sparse GP regression and express equations (2) and (3).  Although, the covariance in Eq. (3): 
$\Sigma_{SGPR} = K_{\mathbf{ww}} – K_{\mathbf{wu}}( K_{\mathbf{uu}}^{-1} – S_{\mathbf{u}} )K_{\mathbf{uw}}$, for the predictive distribution seems not to be correct in comparison to Titsias’ work. Such a covariance appears in Eq. (6) of Titsias’ work as:
$\Sigma_{SGPR} = K_{\mathbf{ww}} – K_{\mathbf{wu}}( K_{\mathbf{uu}}^{-1} – B )K_{\mathbf{uw}}$, where $B = K_{\mathbf{uu}}^{-1}(S_{\mathbf{u}})K_{\mathbf{uu}}^{-1}$, and $S_{\mathbf{u}}$ is the covariance of the Variational posterior (it is called $A$ in Titsias’ paper)

$\textbf{Significance:} $
The experiments carried out in the paper might be useful for other practitioners and researchers in the field, particularly the paper might contain unique experimental contribution like the results presented in Figure 3 associated with non-Gaussian likelihoods.    

$\textbf{Some detailed comments and recommendations for the authors:} $

-The acronyms GP, OVC, SVGP, ELBO, qKG and BO were introduced many times. 

-The acronym SGPR appears in: “An improved conceptual understanding of O-SGPR [9] as SGPR trained...” without being introduced.

-The following phrase might be better written to improve the intelligibility: “First, the compute and memory consumption of exact GPs grows at least quadratically with the amount of data [24, 60], generally limiting them to BO problems with fewer than 1,000 function evaluations.”

-It is not clear the sentence: “Second, they are limited to applications that are modeled well by a Gaussian likelihood”. What does it mean “applications that are modeled well”?

-There is a use of the word “compute” that is not clear, for instance: “First, the compute and memory consumption of exact GPs grows…”, also in “Stochastic variational Gaussian processes (SVGPs) [32] have constant compute and memory footprints and are applicable to non-Gaussian likelihoods”. Is it “computation”? Or “computational”? 

-The following phrase is not clear: “…, but they sacrifice closed form expressions for conditional posteriors.”. Is it missing additional explanation? 

-The caption for “Figure 1” is not very informative and a bit confusing. For instance, at the end of the caption appears: “Efficient, closed form condtioning for SVGPs was previously impractical.”, there is a typo in “conditioning” and what is the author referring to when saying “...was previously impractical”? What is the intention of such a phrase in the caption?

-In section 2.1, typo in “Given $X_{train} = [x_i , . . . , x_n ]$”, change “i” for number “1”? 

-The following expression in section 2.1 is not clear: “Computations with $K_{\mathbf{vv}}$ cost $O(n^2 )$ space and $O(n^3 )$ computation.”.

-The matrices $K_{\mathbf{ww}}$ and $K_{\mathbf{vv}}$ appear in section 2.1, but they are not clearly introduced.

-The matrices $K_{\mathbf{uu}}$ and $K_{\mathbf{uv}}$ appear in section 2.2, but they are not clearly introduced.

-Typo in section 2.2, it says: “The optimal $m_{\mathbf{u}}$, $S_{\mathbf{u}}$, and resulting $q(\mathbf{w})$ are”, the variable “$m_{\mathbf{u}}$” should be bold, $\mathbf{m_u}$.

-Possible error in Eq. (3). The authors refer to the work of Titsias “Variational Learning of Inducing Variables in Sparse Gaussian Processes” for Sparse GP regression and express equations (2) and (3).  Although, the covariance in Eq. (3): 
$\Sigma_{SGPR} = K_{\mathbf{ww}} – K_{\mathbf{wu}}( K_{\mathbf{uu}}^{-1} – S_{\mathbf{u}} )K_{\mathbf{uw}}$, for the predictive distribution seems not to be correct in comparison to Titsias’ work. Such a covariance appears in Eq. (6) of Titsias’ work as:
$\Sigma_{SGPR} = K_{\mathbf{ww}} – K_{\mathbf{wu}}( K_{\mathbf{uu}}^{-1} – B )K_{\mathbf{uw}}$, where $B = K_{\mathbf{uu}}^{-1}(S_{\mathbf{u}})K_{\mathbf{uu}}^{-1}$, and $S_{\mathbf{u}}$ is the covariance of the Variational posterior (it is called $A$ in Titsias’ paper). 

-As a recommendation for section 3.1, it would be useful to apply some intuition to the notation. For instance, it is confusing for the reader to associate the definitions: 
“inducing function values $\mathbf{u}^{\prime} := [f(\mathbf{z}_1), . . . , f(\mathbf{z}_p )]$” and 
“$\mathbf{u} := [f (\mathbf{z}^{\prime}_1), . . . ,f(\mathbf{z}^{\prime}_p )]$”. 
It might be easier to associate the “$\mathbf{u}^{\prime}$” with the evaluations of the “$f(\mathbf{z}^{\prime})$” and also the “$\mathbf{u}$” (without prime) with the evaluations of the “$f(\mathbf{z})$” (without prime).

-In section 3.1, the following notations should be corrected to make the dimensionality coincide: 
“… Eq. (6) to define a new SGPR model with pseudo-data ($\hat{X} := [X_t \quad Z_{t-1}]^{\top} , \hat{\mathbf{y}} := [\mathbf{y}_t \quad \tilde{\mathbf{y}}_\mathbf{u^{\prime}}]^{\top}$)”. 
A possible way is to write: 

$\hat{X}:= [{X_t}^{\top}$ $Z^{\top}_{t-1}]^{\top}$ and $\hat{\mathbf{y}} := [\mathbf{y}^{\top}_t \quad \tilde{\mathbf{y}}^{\top}_\mathbf{u^{\prime}}]^{\top}$.
 

-In section 3.1, the distributions “$q_t(\cdot)$”, “$\hat{p}(\cdot|\mathbf{y}_t)$” and $p_t(\cdot)$ are not explained.

-In section 3.1, there is not explanation of what is the variable “$\mathbf{\hat{v}}$” in the phrase: “where the distribution of $\mathbf{y}$ is the pseudo-likelihood $N(\mathbf{\hat{v}}, Σ_{\hat{\mathbf{y}}} )$” 

-In section 3.1, typo word, Should it be “through”? In phrase: “projecting that distribution back into pseudo-data to be combined with a fresh batch of ground-truth data though a joint Gaussian pseudo-likelihood.”.

-Equations (3), (4), (7) and (8) are never cited in the text.

-There is not a clear explanation for the Figure 2. Therefore, it is difficult to understand what are the main comparisons to analyse between the Figure 2 (b), Figure 2 (c) and Figure 2 (d). Additionally, the colour bars of the figures have different ranges making difficult to distinguish their characteristics. For instance, Figure 2 (b) has a colour bar range (1.6,2.5), Figure 2 (c) has a colour bar range (1.7,2.5) and Figure 2 (d) a colour bar range (1.52,2.16). 

-In the caption for Figure 3, the authors refer to NEI (an acronym not introduced previously). For instance in the caption: “Preference learning; SVGPs with qKG slightly outperform Laplace approximations with NEI, and outperform qNEI with SVGPs.”. is the acronym NEI the same as qNEI?
  
-The authors write in the caption of Figure 3: “Only SVGPs can be used here,”. Nonetheless, there is not information of why only SVGP can be used. It might be useful to give some insight to the reader; probably the caption is not the appropriate place to give such an insight.

-How did the authors select the batch sizes q? It might be useful to have an analysis of the influence of the batch size. What could it be the suggestion for a practitioner when selecting the batch size q?


-In experiments for Hotspot Modelling it appears: "We model the responses $y$ (incidence) at locations $x$ with population $n(x)$ with a Binomial likelihood $p(\hat{y}|f, x)$". is it $y$ different to $\hat{y}$?


Check References:

-In [3], [5], [6], [13], [14], [15], [16], [28], [30], [34], [35], [36], [37], [39], [41], [42], [45], [48], [49], [50], [53], [56], [61], [63], [68] and [79] change “gaussian” by “Gaussian”.

-In [8] change “bayes” by “Bayes”.

-In [20], [21], [23], [43], [48], [51], [73] and [74] change “bayesian” by “Bayesian”.

-In [22] change “svm” by “SVM”.

-In [44] change “lipschitz” by “Lipschitz”.

-In [58] change “hessian” by “Hessian”.

-In [72] change “monte carlo” by “Monte Carlo”.
limitations_and_societal_impact: The authors could improve their work by extending a discussion regarding the reliability of the methods, particularly in the context of Disease Incidence where the authors showed different applications. Such a discussion might enhance the usefulness of their work and the potential positive or negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors are interested in the problem of online learning with applications to online Decision making. The focus is on (sparse variational) Gaussian Processes and Bayesian optimization.

The authors 1) propose an alternative formulation of the online learning algorithm of Bui et al, which allow to ‘preserve’ what has been learned from the previous data 2) propose a way to update the posterior when adding observations in the conditioning set for non Gaussian likelihood 3) Thoroughly evaluate the resulting algorithm on various BO tasks.

main_review: 
Originality and Quality:

The paper builds up on the work of Bui et al, where the aim is to update posterior processes when adding observations to the conditioning set.
The authors acknowledge mostly rederiving the method in an alternative fashion with the aim to connect it to Sparse Gaussian Process Regression (SVGP in the conjugate case by Titsias).

The approach consists in essence in explicitly representing the posterior via sites that are updated in an incremental fashion. The ‘unifying’ paper of Bui would be a welcome reference here. As a rederivation rather than introduction, I m not sure it is worth a new name. Also the terminology is not very enlightening. Computing the posterior is always conditioning on observations. online conditioning is not very precise not specific to this method (also see comment below).

The proposed incremental formulation is equivalent to Bui et al, and brings practical advantages when implementing. The particular applications of SVGP to BO is original (recent work on Thompson sampling in this particular combination would be worth mentioning :Vakili 2020).

I have a concern regarding the Laplace approximation. Each likelihood term is optimized independently of the ongoing posterior distribution. For a Bernoulli likelihood, the maximizer is at infinity and the hessian is ‘flat’ (being exponential family doesn't solve that). This problem is why the community has derived ways to compute sites in the context of the ongoing approximation  (khan et at 2017 for VI, or Bui et al 2017 for EP). This and other limitations of this heuristic should be discussed. The lack thereof does not make me confident or provide intuition that errors don’t accumulate. 
An experiment focusing just on the quality of this approximation for online learning would help justify this particular choice.


Clarity:

The manuscript is fairly well written (see comments below for improvement suggestions)

Significance

Look ahead objectives are strong candidates for BO and deriving practical algorithms to update posterior processes approximately and cheaply is needed.
Though tackling an relevant problem, the contribution in the paper is a rewriting of an existing algorithm and I have concerns about the extension.

My rating of this paper reflects these mixed feelings, but I am ready to increase my score.


Remarks:

Eq 3: covariance looks wrong to me. you can’t subtract a covariance from a precision, the following derivations look fine though

‘Conditioning a GP into another GP’ is very informal and doesn’t help the reading. “conditioning into” is not a thing. I’d recommend using more standard formulations.

The S in acronym SVGP is used to denote both Sparse and Stochastic throughout the paper, which is confusing. 
 
A couple of typos (the prime) in section 3.1, makes it hard to parse.


I have not thoroughly reviewed the experimental section and leave this to other reviewers.

limitations_and_societal_impact: Limitations: the authors could be more careful in discussing and assessing the limitations of the approximation introduced for the online learning problem before applying it to decision making.

negative societal impact : None
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The streaming sparse GP (named O-SGPR in this paper) is a variational GP approximation that supports updating the variational posteriors and hyper-parameters online. This paper reinterprets O-SGPR as compressing the past observations into a small set of "pseudo-data". Then updating the posterior based on new observations can be simply treated as a standard GP inference based on the new observation and the pseudo-data. This paper adopts the proposed Online Variational Conditioning (OVC) over Bayesian optimization and active learning, where inferring the updated belief from new observations is important to quantify the contained information. This paper is well-written and conducts extensive experiments to verify the proposed approach.
main_review: ## Method
### Novelty and Significance
Enabling variational GPs for online posterior inference is an important problem. The streaming sparse GP (named O-SGPR in this paper) is a landmark approach that enables updating both the posterior belief and the hyper-parameters for variational GPs. This paper reinterprets O-SGPR in terms of performing GP inference over the new observations and "pseudo data", which I think is a novel contribution. 

### Relations to O-SGPR
As discussed in the paper, OVC is based on a new interpretation of the O-SGPR. However, I think it deserves more discussions on their differences. Based on my understanding, O-SGPR updates the variational posterior by maximizing the ELBO, which prevents it from being directly plugged into the Bayesian optimization problems. In contrast, the OVC directly obtains the posterior by the "exact GP formulation", enabling the explicit reasoning in Bayesian optimizations. However, OVC sacrifices the general applicability towards non-Gaussian likelihoods and turns to Laplacian approximations. 

I think this difference is the major contribution in this paper, given the existence of O-SGPR. And it is the key for being applied in Bayesian optimizations. But it is not obvious from reading the paper.

###  Relations to standard SVGP
Imagine that we have the inducing points $Z_{t-1}$ and the variational posterior $ \phi_{t-1}(u') = \mathcal{N} (\mu, S) $, then the current posterior is a Gaussian process 
$$
GP_{t-1} = \mathcal{GP}( k(\cdot, Z) k(Z, Z)^{-1} \mu, k(\cdot, \cdot) - k(\cdot, Z) k(Z, Z)^{-1} k(Z, \cdot)  ),
$$
Given any new observations, we can also infer the new posterior belief under this $GP_{t-1}$.

Is OVC equivalent to this approach ? If not, what is the advantage of OVC against it ? I noticed that Line147-150 might be relevant, but the two shortcomings listed there apply to OVC as well in my perspective.


## Clarity
This paper is well written in general. Here are some minor comments,
1) Line 158: define $p_t, q_t$ before using them.
2)  Inducing Point Selection: do you select over "[new_obs, pseudo_data]", or  keep all "pseudo_data" and only select from the "new_obs" ? 
3) Figure 2: The captions for b,c,d  are not clear. It is not clear to me what is the difference between (c) and (d).
4) Line 277: the notation $x$ is used in both $a(x; D)$ and $dx$.

## Experiments
1) Do you keep a fixed budget for the number of inducing points, or add a fixed number of inducing points per iteration ? 
2) What is the impact of the number of inducing points ? For example, in Figure 3(a) and Figure 5, it is not clear to me how many inducing points you used compared to the Exact GP. 
3) Do you update the kernel hyperparameters along training ? 
limitations_and_societal_impact: To further improve the paper, I think these are potential directions,
1) A clear discussion on the contribution compared to O-SGPR and a standard SVGP.
2) A clear model description on 

 (a) how the variational distributions are obtained,

 (b) how the hyperparameters are chosen,

 (c) what is the training workflow,

This paper does not lead to negative societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Authors propose a novel method, named OVC for online variational conditioning of GPs under sparse approximations. Particularly, they are interested in the recursive update of the posterior GP distribution after receiving new data per time-step. The model can be used for both regression and classification tasks, bayesian optimisation and active learning as is shown in the experiments.
main_review: **Originality and significance:** 

Good paper. I enjoyed quite a lot while reading it. Particularly, I want to remark the following points that are positive to me:

- The idea of transforming the inducing points and their corresponding function instances into pseudo-data is interesting. It can be considered as some sort of data augmentation. Authors correctly identified the limitations of the approach presented in Bui et al. (2017).
- Figure 1 is great. It is not that easy to obtain such performance, even if the “smoothness” of the signal changes across time. How do you treat the kernel hyperparameters? In that particular case seems like between [-2,2] the length scale is around 1.0, and later between [2,6] it is smaller (0.5 for example). Is the model well fitted to the past data because you are using the cache thing right? Otherwise would be problematic if re-training.
- Experiments are well designed and prove the performance of the tool. I see clearly that the method outperforms Bui et al. (2017).
- Perhaps Bui et al. (2017) could work better with non-Gaussian likelihoods, but the limitations are overcame by OVC here. I like the Laplace approximation, but maybe if considering multivariate likelihood models, things could be more complicated.. 
- I found the “banana dataset” experiment in Figure 8 of the Appendix. What a pity not having included it in the main manuscript, because the results are pretty good. 
- An extra positive point of this work is that you are able to update the model with one-single observation (in a filtering-like model) or with more than one data-point per time step (similarly as Bui et al. 2017). Great!

**Quality and clarity:** 

 In particular, I want to remark the quality of Section 2, which demonstrates the wide knowledge of authors on the online GP topic and how the decisions taken are well-motivated later. Experiments are also thorough and very well presented. Results are convincing. To sign a flaw of the paper, I would say that section 3.1 could be clearer, but this is a little detail. I’m sure the paper will have an impact on the community.

**Questions and other comments:**

- L106: I think that this whitened parametrisation is not that well-known, I suggest a super-short explanation, mainly because later is re-used.
- L156: are the definitions of u’ and u in the correct order? It seems that u’ should refer to f(z’), instead of the written ones.
- The approximation of y_u in the point 1 of Algorithm 1) makes sense to me, but maybe is not that well explained. Do you do it to use the “fixed” variational distribution S_u, right? Otherwise, would be useless at the next time-step t.
- The cache terms a and RR look like they are gonna be critical in the paper at the beginning, but later they are not commented anymore. The task of adding new rows and columns to Kvv (L82) in an online fashion seems like a contribution of other works, is that the reason why authors do not claim it?
- Algorithm 1 would be even better having a recursive nature, like do this every time step t. At point (4) one may get lost and do not know how to use the model again at t=t+1.
- L192: How is the number of inducing points treated? Do authors augment them as long as new data arrives? Which is the criteria?
- Could you add some extra details about the treatment of hyperparameters + kernel models? Are you assuming stationary functions or it may change?

**References:**

- Bui et al. (2017). “Streaming sparse Gaussian process approximations”. NIPS 2017.



limitations_and_societal_impact: Authors adequately addresses both limitations and societal impact of the work in the section A of the Appendix. Particularly, they explained very well the limitations of the method and the cases where it might not work very well.
review_title: null

Meta-review: The paper contributes to the literature of sparse variational Gaussian processes applied to an online setting. The authors introduce the novel idea of treating inducing variables and inducing inputs at time t-1 as pseudo-data to be used at time t, leading to recursive expressions for the posterior distribution. The paper appears technically solid. The most critical reviewer does not oppose acceptance. Comments regarding clarity and typos highlighted by the reviewers can be addressed in the camera-ready version.
Decision: Accept (Poster)