Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper revisits a known, but seemingly forgotten dual parametrization for multivariate Gaussian distributions that naturally lends itself to variational inference in Gaussian process models. This dual parametrization was introduced in previous works for variational Gaussian inference using a Lagrangian relaxation of the ELBO loss. Similarly to commonly used parametrizations, the dual parametrization can be combined with natural gradients based optimization that allows for faster training of the variational distribution by exploiting the information geometry of distribution space. However, the required natural gradients are considerably cheaper to compute in this setting due to only requiring first-order gradients. This parametrization is then incorporated into a sparse variational inference scheme for GPs. One of the computational difficulties when training the the SVGP lies with the coupled optimization of the variational distribution and the prior hyperparameters, for which several heuristics have been introduced in the community. The experiments compare the proposed parametrization for SVGP with the commonly used whitened one. They demonstrate that for small and mid-sized datasets, gradient descent along the trajectory of optimal variational distributions results in more stable learning of the hyperparameters. For larger datasets, the coupled stochastic optimization of the two (with only partial optimization of the variational distribution) leads the dual formulation to equivalent or slightly better optima measured in terms of predictive density, but it is considerably faster than the baseline due to cheaper natural gradients. 
main_review: This is a well-written paper that recycles a previously known parametrization for variational Gaussian inference that has not been employed in the SVGP context before. The main idea is therefore relatively simple, which in concept follows from simply putting these together. However, in my view, there is considerable value in this due to the nontrivial insight that is required to carry out the required calculations, put them into an implementation and evaluate them experimentally. Especially so, because the main problem that the paper is trying to address (optimization in SVGP) is of considerable significance, and the results are of interest. Judging purely by the results, the approach does seem promising and I think this paper can be a valuable contribution to the GP community.

One obstacle I see to these ideas becoming adopted and their use more wide-spread is that the description of the proposed approach is not very clear at the moment. In particular, Section 3 is hard to follow because no intuition is provided about the dual parametrization, and only a brief survey is given (although with several references), which is not an adequate initiation to the approach for those not already familiar with it. Given that the main tenet of the paper is that this parametrization seems to be unexplored in the SVGP context, I would have appreciated if a more detailed description was given. In contrast, Section 2 spends about two pages on recapping well-known ideas. Some of this space could probably be reallocated to the description in Section 3. This also makes Section 4 somewhat hard to follow, where the idea is combined with the SVGP. Section 5 on the other hand is very clear, the experimental evaluation and the results are interesting.

Overall, although neither of the core ideas in the paper are novel, I believe that a novel combination of known ideas can also be a valuable contribution. However, my main criticism of the paper is regarding the presentation of the idea that they are trying to introduce, which in its current form might limit their use due to lack of clarity. If the authors can improve on the presentation in these sections, I would be inclined to slightly raise my rating.
limitations_and_societal_impact: At the moment there is no discussion of limitations, which may be useful to include with the experimental evaluation. In particular, if there are any drawbacks of the proposed parametrization compared to the standard one would be useful to know. It is also not clear to me if the discussed dual parametrization limits the space of approximating distributions, or if any multivariate Gaussian can be represented in this form as well. Also, did the authors face any challenges during the implementation of the proposed model, or during its optimization?
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes to speed-up approximate inference in the sparse variational GP model using a dual parameterisation (similar to an EP approach) where each data point is assigned an additional dual variable that may be updated efficiently using natural gradient descent with reduced complexity on the natural gradient updates that also lead to a tighter bound on the evidence. The paper illustrates the efficacy of the approach over standard SVGP variational inference on a number of standard benchmark datasets and demonstrates good empirical performance whilst reducing training time.
main_review: I found the paper to be clear to read with a good coverage of the background material and justification for the proposed methodology. The authors highlight that the proposal brings together a number of approaches that are already established (e.g. SVGP, natural gradient descent and EP) but, in my opinion, this is not a limitation as the paper treats the material well, does not overclaim the novelty but instead makes the point that bringing these things together in this way yields a very satisfactory approach that is neither well-known nor well used (to the best of my knowledge). That it performs competitively in established code-bases (such as GPflow) is a good indication of importance of the result and its general applicability. In particular, the recognition that improving the calculations of the natural gradient updates is probably the most satisfying part of the paper and the case for this is well made.

Overall, I believe that the paper provides sufficient empirical evidence to support the theoretical ideas behind the proposed method. I can see that there might be objections to datasets such as MNIST but I agree that it is the scaling we are interested in here and the separation of the objective wrt hyperparameters and variational params provides a strong motivation and good result for the paper. 

To the best of my knowledge the approach described has not been explicitly presented before and I am in favour of accepting the paper.

Other comments:

- Thinking about fair comparison for the best q-SVGP - a number of papers have questioned whether the alternation approach of EM does always lead to improved convergence (i.e. is coordinate ascent always better than something like a Gauss-Newton approach that approximates the off-diagonal blocks of the Hessian somehow). I wonder if the authors could comment upon whether or not they have tried these sorts of approaches? I can well believe that the natural gradients lead to better conditioning (effectively the appropriate blocks of the Hessian for approx second-order optimisation) but, if I understand correctly, they will assume “zero blocks” in the Hessian in the cross terms between, say, the natural gradient parameters and the hyper parameters. My understanding is that these are usually dealt with first order approaches with momentum, etc.. but I don’t know if there has been any more in-depth investigation for this approach? E.g. something like L-BFGS or similar?

- Could the authors avoid using “e” as a parameter (e.g. (11)) since its also used as the exponential (e.g. (12))?

-  Typos: lines 81 (-them by-), 91 (_integrals_), 110 (Gradient Descent?)
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce an efficient natural gradient based approximate inference scheme for sparse variational Gaussian Process models. The approach leverages a "dual parameterization" that exploits Lagrange multipliers along the lines of "Decoupled variational Gaussian inference" (Khan 2014). These dual parameters are conceptually similar to site parameters in expectation propagation. In experiments it is shown that this dual parameterization scheme results in faster optimization and tighter bounds (although solid theory is lacking for the latter claim, at least for the non-conjugate case).
main_review: Gaussian process (GP) models are popular due to their good uncertainty quantification, interpretability, and ability to include prior information through kernel design. Thus improvements to GP inference are likely to be of interest to an active if somewhat moderately sized NeurIPS subcommunity. This submission has pretty clear exposition and does a reasonably good job of positioning itself w.r.t. prior work. The experiments are moderately comprehensive and give some amount of confidence in the general claim that this method offers non-negligible inference speed-ups. The work is somewhat incremental, as it directly builds on other references, including e.g. various Khan et. al. and Hensman et. al. papers cited in the submission. Nevertheless, I think this work could be of interest to the NeurIPS community, since it offers concrete speed-ups over natural gradient approaches to SVGP as well as an overview/application of approaches like "Decoupled variational Gaussian inference" that, as the authors say, can be described as a "known, yet rarely used parameterization for Gaussian distributions."

Given the somewhat incremental nature of this work, I think the work could be considerably improved if the empirical evidence for inference speed-ups could be expanded. In particular as far as I can tell all the experiments use a pretty modest number of inducing points (100). Does the quoted 5x speed-up (line 322) persist for 500 or 1000 inducing points? Does the performance improvement disappear or largely disappear when implemented on GPU? As far as I can tell the experiments are all performed on CPU.

Additional comments/questions:
- “The formulation is fast since it avoids the use of sluggish automatic differentiation to compute the 
natural gradients. “ (line 42) Can you clarify why this is exactly? As I understand it we are talking about a multiplicative constant and not an asymptotic speed-up. Has this more to do with the limitations/flexibility of current AD systems or is this something more fundamental?
- Typos: mirrordescent (x2), optimial

## After author response

I thank the authors for their response. Given the (somewhat) incremental nature of the work I maintain my original score. Nevertheless I would be in favor of seeing this submission accepted for publication, as I believe it could be of interest to the community. I suggest the authors include experiments with a larger number of inducing points. In most practical applications (unless one is particularly compute constrained) M ~250-500+ inducing points often deliver tangible performance improvements over much smaller numbers of inducing points. As such this represents a common and important regime.
limitations_and_societal_impact: The authors do not explicitly discuss potential negative societal impact of their work.

As suggested above, I think it would be valuable if the authors further delineated regimes (and verified empirically) in which they do and do not expect substantial speed-ups. In particular CPU versus GPU and few inducing points versus many inducing points.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors argue for a different (existing) parametrization for training (non-conjugate) Gaussian Processes.
They show that this parametrization can be interpreted as the optimal result of a dual problem, and that one can compute easily natural gradient updates.
Additionally this parametrization "may" lead to a tighter ELBO because of a stronger dependency on the hyperparameters (kernel parameters).
Experiments are run to show the improvement over the standard "q-SVGP".
main_review: The paper is well written and exposes clearly all the different elements to understand the theory and method introduced.
The main contributions are to bring a different view to the parametrization introduced in Csato 02' as an optimization problem that is mathematically interesting and derive natural gradient updates for this new parametrization.
Since the parametrization is already well known, the contribution is only limited and some of the mentioned advantages have flaws.

- There is a large flaw in the reasoning and the comparison with Salimbeni 18'. It is argued that  "[t-SVGP] is much cheaper than previous calculations of the natural gradient", because one supposedly does not have so many reparametrizations to work with. However $\nabla_\mu E_{q(f)}[\log p(y|f)]$ requires the same parametrization as Salimbeni 18' since gradients given the second expected parameter $(mm^\top + S)$ cannot be found directly (and therefore require a reparametrization as well)
-  The mention "It is a first order update" (L175)  sounds wrong since gradient derivatives given $S$ imply second order derivatives.
- Using natural gradients in Salimbeni 18' requires extra-care with the learning rate, especially in the first iterations. Some additional comments in this regard for t-SVGP would be good.
- It is not enough in the experiments to compare only to the standard q-SVGP, there should be at least a direct comparison with Opper 09', so one can see what the natural gradient approach actually brings.
- There are missing references for relevant work on this area, notably via the use of latent variable augmentation, see e.g. Wenzel 19' or more generally Galy-Fajou 20'
- Minor comments:
	- L100: "leads to a convex objective" Since the objective is defined to be the ELBO, it should be concave.
	- L192: "after one E-step" Repetition with the beginning of the sentence
- References
	- Csato 02' : Sparse on-line Gaussian processes, 2002, Csato and Opper
	- Opper 09' : The Variational Gaussian Approximation Revisited, 2009, Opper and Archambeau
	- Salimbeni 18' : Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models, 2018, 		Salimbeni, Eleftheriom and Hensman
	- Wenzel 19' : Efficient Gaussian Process Classification Using Polya-Gamma Data Augmentation, 2019, Florian Wenzel, Theo Galy-Fajou, Christan Donner, Marius Kloft and Manfred Opper
	- Galy-Fajou 20' : Automated Augmented Conjugate Inference for Non-conjugate Gaussian Process Models, 2020, Théo Galy-Fajou, Florian Wenzel, Manfred Opper
limitations_and_societal_impact: Authors mention limitations in Section 6, but there is no mention on how the method could fail.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new dual parametrization method to improve the optimization efficiency for variational sparse Gaussian processes. The method speeds up inference than natural gradient descent and achieves a tighter lower bound for hyperparameter learning. 
main_review: Originality: This paper introduces a new method for optimizing the variational sparse Gaussian processes.  

Quality: The authors should focus more on the technical writing to make them error-free:
Line 68: makes it a drawback .....
Line 130: can be approximated....
Line 519 (appendix)
......
Clarity: As mentioned above, the technical writing is not very sound and fluent which is misleading sometimes.

Significance: The experimental results generally demonstrated the technical part in the paper. 
limitations_and_societal_impact: Focus more on the writing
review_title: null

Meta-review: This paper revisits a known alternative, so-called dual, parameterization for models with Gaussian priors and iid likelihoods and applies it to stochastic variational inference in Gaussian process models. It shows _empirically_ that this parameterization results in faster optimization and tighter bounds that improve hyper-parameter learning. Results are shown on small UCI datasets and on MINST (n=70,000 datapoints).

Overall, the reviewers believe that the proposed approach is incremental as it takes a known idea (that of the dual parameterization) and applies to the SVGP context. However, all the reviewers agree that the paper provides a significant contribution to the NeurIPS community and that NeurIPS will benefit from knowing the details and findings in this paper. 

Besides its incremental nature, one of the major criticisms of this paper was with respect to its clarity (Reviewer aCDy), which the authors seemed to have addressed satisfactorily. Another important point (raised by reviewer XG5T) concerns whether the benefits of the approach would extend to larger problems for which one would need a higher number of inducing variables ($m>100$). The authors have clarified that the bigger the $m$ the smaller the marginal gain obtained by the proposed method. However, the proposed approach still provides benefits for settings of practical interest. 

Decision: Accept (Poster)