Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method for approximating the posterior of deep gaussian processes, to allow for non-Gaussian posteriors.  The main idea is to train a generator network that approximates the posterior, while training a discriminator that learns a regularized version of the Stein discrepancy.  They also optimize hyperparameters and inducing point locations via SGD.  The authors also prove some theorems in the (unrealistic, but still interesting) regime of having infinitely expressive neural networks than can be perfectly optimized.
strength_and_weaknesses: Strengths:
* The approach is interesting, and the ability to learn complex posteriors in the DGP framework is promising.

Weaknesses:
* (minor) there are a number of typos throughout the manuscript. This did not interfere with my understanding, but the paper could use some thorough copy eding.   For example there are numerous citation typos (particularly missing spaces before citations in the text).  Also, e.g., "In particular, according to Bayesian formula" --> "In particular, according to Bayes' Rule";  "a quick introduction to these concepts that forms the foundation of our method" --> "a quick introduction to these concepts that form the foundation of our method"; and so on.
* The theorems assume an infinitely expressive neural network architecture, but the role of network architecture on the empirical results is relatively unexplored (e.g., only number of layers, but not number of hidden units, convolutional architectures, etc...)
* In table 1 in the appendix, why does the runtime increase only slightly with the number of inducing points?  If the runtime increases so slightly and accuracy increases substantially, then why not use substantially more inducing points.  E.g., spending an additional 0.011s (presumably per iteration?) on Concrete to go from 50 to 400 inducing points, but going from an RMSE of 0.28 to 0.19 seems like a very good tradeoff.  Why not go on to more inducing points?  When does accuracy plateau or when does the runtime become infeasible?
* In equation (16), a non-random quantity on the left hand side is claimed to be equal to a random quantity on the right hand side.  This equality does not even hold in expectation (i.e., the gradient will be biased) as can be seen by applying Jensen's inequality to Equation (6) in the appendix.  And the authors are certainly aware that this is not a strict equality given the $\approx$ in Equation (6) in the supplement.  As such, I think that calling this a "theorem" with a strict equality is somewhat disingenous. 
clarity,_quality,_novelty_and_reproducibility: The paper presents an interesting, novel approach.  The writing and exposition could certainly be made clearer.  Theorem 1 is presented a theorem when in reality it is more like a proposed scheme for approximating gradients.
summary_of_the_review: The paper presents an interesting approach, but the key assumptions of its theorems are not met in practice (infinitely expressive neural networks, perfect optimization).  The impact of neural network architecture in particular, is not sufficiently explored.  I also found some parts of the paper to be presented in a way to appear rigorous, but are actually sort of pseudo-rigorous, at the expense of both clarity and accuracy (e.g., Theorem 1; repeated claims that neural networks with a finite number of hidden units are expressive enough to model any distribution).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new method for inference of the intractable posterior for deep Gaussian processes (DGPs). The idea is to minimize the Stein discrepancy between the approximated posterior distribution which is demonstrated by a generator and the true posterior distribution. The algorithm utilizes a discriminator to estimate the discrepancy between two distributions. It is claimed that a better approximation to the true posterior is recovered. To support the claim, extensive regression and classification experiments are conducted. 
strength_and_weaknesses: In fact, this paper demonstrates a high resemblance to Implicit Posterior Variational Inference for Deep Gaussian Processes [1]. The difference is that this work decides to minimize the Stein discrepancy while the work of [1] minimizes KL divergence. The design of the generator and discriminator, the iterative optimization procedure, and even the design to represent the inducing variables. 

These two methods are not even compared in this paper, given all these resemblances. From my point of view, this comparison is compulsory. I am happy to raise my score if the authors can provide convincing reasons. 

In the context of the NOVI algorithm, this paper is still far from satisfactory. The convergence guarantee is provided in Theorem 2 and Theorem 3 and yet no analysis or illustrations to prove this theorem. How would this method perform given a known true posterior distribution? This kind of synthetic experiment will make the claims more convincing than RMSE or classification accuracy. 


[1] https://arxiv.org/abs/1910.11998
clarity,_quality,_novelty_and_reproducibility: 
Clarity: The paper is well-written and relatively easy to follow.
Quality & originality: The contribution of this paper is relatively incremental
Reproducibility: The code is provided with the paper, I believe the reproducibility of this work.
summary_of_the_review: Given that the novelty and contribution of this paper are relatively incremental, I am inclined to reject this paper. Moreover, I would like to suggest the authors polish this paper in the direction of comparison of different inference methods for DGPs, the pros, and cons for each.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a novel framework to conduct approximate variational inference in deep Gaussian processes. It is based on minimizing the regularized Stein discrepancy between the approximate distribution and the true posterior through the newly proposed Neural Operator Variational Inference algorithm for DGPs, which iteratively updates a generator and the associated discriminator to estimate more accurately the target discrepancy. This leads to a better final approximation than the ones obtained through the regular mean-field Gaussian assumption usually employed in DGPs. Authors support this claimg by extensive regression experiments, in which they show also that the method presents important properties such as robustness to overfitting. 
strength_and_weaknesses: # Strengths: 

* The method is an interesting addition to the literature in Deep Gaussian Processes, a promising line of research that has layed out very interesting results for the past few years. 

* The approach is fairly easy to understand. Even though there are important details that must be covered in detail, for the most part it is pretty straight-forward. 

* The authors provide extensive experiments, accounting also for the scalability of the method and its applicability in different problems and datasets. This strenghtens the submission, as well as the convergence guarantees and the extra information provided in the supplementary material.

* The paper is mostly well written and easily understandable, covering the most important points on previous works and providing a good description of the method's development.  


# Weaknesses:

* My main concern here is the novelty of the method, which I deem to be a bit lacking. Overall, I see the contribution as interesting to the community, and also potentially very useful. However, as far as I understand it, this method results from the combination of previously well-known previous contributions in a very direct manner. I may be mistaken, and in that case I am entirely open to change my opinion on this matter, but as it is I think the contribution may lack sufficient novelty.

* I also suggest the authors to emphasize in a stronger note the differences between this method and previously existing contributions. This may help with the previous point as well.

* The UCI experiments could be much more robust if other metrics were reported asides from the RMSE. Common choices are the negative log-likelihood or the continously ranked probability score, as a proper scoring rule. I suggest the authors to include some other metrics that help better assess the quality of the predictive distribution, since RMSE does not necessarily provide that information.

* I would appreciate a more extensive experiment with bigger datasets than Energy (e.g. Taxi, Airlines or others similar to these ones). Also, it would help if the authors shed some light on the expected performance of the method in cases with high dimensionality.

* (_minor_) The presentation of the tabulated results in the supplementary material should be revised.

* (_minor_) The "Related Work" section seems limited, since there are several other approximate inference methods strongly related to this one. As an example, the NOVI setup proposed in section 4.2  strongly ressembles the construct made in articles like [5,6] that deal with implicit distributions for approximate inference. Moreover, formulation in the equation of section 4.1 makes me wish there was a discussion between the relation between these methods and function-space inference with implicit stochastic processes, since the two approaches actually have a lot in common. As suggestions, I see strong relationship with works such as [1,2,3,4]. 

* (_minor_) Figure 1 should be re-thought, its quality leaves some improvement to be desired to help understanding the content of the graphical models. 


## Other comments: 

* The language employed in section 4.2 makes it seem sometimes that the authors contribute here to the development of previous techniques, e.g. OVI (Ranganath et al. 2016). I suggest the authors to be clarify the language here to state what contributions are genuinely theirs.  

* Please, clarify further the statement before Eq. 14 regarding the fact that the two distributions must be "equivalent"



## References: 


[1] Ma, C., Li, Y., and Hernández-Lobato, J. M. (2019). “Variational implicit processes”.
In: International Conference on Machine Learning, pp. 4222–4233.

[2] Sun, S., Zhang, G., Shi, J., and Grosse, R. (2019). “Functional variational Bayesian neural networks”. In: International Conference on Learning Representations.

[3] Rodrı́guez Santana, S., Zaldivar, B., & Hernandez-Lobato, D. (2022). Function-space Inference with Sparse Implicit Processes. In International Conference on Machine Learning (pp. 18723-18740). PMLR.
 
[4] Ma, C., & Hernández-Lobato, J. M. (2021). Functional variational inference based on stochastic process generators. Advances in Neural Information Processing Systems, 34, 21795-21807.

[5] Mescheder, Lars, Sebastian Nowozin, and Andreas Geiger. "Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks." International Conference on Machine Learning. PMLR, 2017.

[6] Santana, S. R., & Hernández-Lobato, D. (2022). Adversarial α-divergence minimization for Bayesian approximate inference. Neurocomputing, 471, 260-274.

clarity,_quality,_novelty_and_reproducibility: 
# Clarity

The paper is well-written, covering previous contributions and the derivations needed to understand the proposed method. 

Please, check for typos. Some examples are: 
  * "framwork" (first contribution bullet point, page 2)
  * "objictive" (three lines below Eq. 12)
  * "gemoetry" (second line of Section 4.2)


# Quality

The paper is of good quality, although some details should be polished to make justice to the amount of work behind this contribution. More specifically, providing better plots and graphical models would help, as well as taking care of the mishaps of the tabulated data in the supplementary material.

# Novelty

The ingredients for the method already exist before the proposal, so the contribution is mainly centered on a particular combination of these well-known concepts in a new manner that results beneficial. 


# Reproducibility

Code is already publicly available and the text is clear. Without attempting to reproduce it myself, I deem the contribution reproducible.

summary_of_the_review: 
The contribution seems correct and interesting to the research community both due to its implications in DGPs and VI. However, it may lack sufficient novelty to merit a higher score. The authors should make an effort to highlight in a stronger fashion how does this contribution differenciate from previous works. Some extensions to the current experimental setup could help the case here as well (it is fine already, but some simple additions would make it even stronger).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes a learning and inference algorithm for deep Gaussian processes based on operator variational inference (OVI). 

Particularly, OVI (Ranganath et al., 2016, Hu et al. 2018, Grathwohl et al. 2020) generalizes Kernelized Stein Discrepancy (KSD) to situations where the RKHS space the smooth function (of the Langevin-Stein Operator) is replaced by the L2 space. Key to both OVI and KSD is the computation of the Stein Discrepancy (SD) which is parameterized by both the smooth operator and a surrogate of the true posterior that we wish to sample from. 

In KSD, we can solve for SD in closed form which eases the optimization of the surrogate. In OVI, we parameterize the smooth function by a deep neural net and the SD can be estimated via sampling with re-parameterization & solving a minimax optimization task. Both OVI & SGD can be considered generic inference strategy for deep generative learning (including DGP). This paper extends OVI by also parameterizing the posterior with a neural net generator, which appears to be using the re-parameterization trick (Kingma & Welling, 2013). 

This is demonstrated on deep Gaussian processes applied to 4 UCI regression datasets + 3 classification benchmark datasets (MNIST, Fashion-MNIST and CIFAR-10) showing marginal improvement over two baselines DSVI (Salimbeni & Deisenroth, 2017) and SGHMC (Havasi et al., 2018).
strength_and_weaknesses: Strengths:

+ This paper is well-organized and has a good review section on important background of deep GP, Stein Discrepancy and SD-based inference techniques for generic deep Bayesian models such as KSD and OVI

+ There are theoretical results characterizing the behavior of the optimal loss when the Stein Identity is satisfied; and an upper bound on the bias of the DGP predictor using the model learned via OVI.

Weaknesses:

- The contribution stance of this paper is somewhat problematic. It is in fact not clear to me what are the new algorithmic insights that the proposed algorithm (NOVI) provides in addition to what were found in OVI and KSD -- I will elaborate more on this in the Novelty section.

- There is a lack of comparison with other relevant inference work for deep GP.

- Despite the well-organized structure, this paper has quite a no. of typos -- some of which is quite problematic as it changes the semantic of the sentence into a contribution claim of an existing work -- more on this later.
     
clarity,_quality,_novelty_and_reproducibility: Novelty & Quality:

As stated in the paper summary, my take of this paper is that its main contribution is a re-parameterization of the surrogate posterior q(.) in the formulation of OVI -- this is highlighted in Section 4.1 but this is exactly the same as the re-parameterization trick. 

Moreover, such re-parameterization has been indeed incorporated in the original KSD work of (Liu & Wang, 2016) -- see the text after Eq. (4) in https://arxiv.org/pdf/1608.04471.pdf

As such, NOVI is not a new variant of OVI but it is instead an application to DGP with the choice of re-parameterized surrogate put on the distribution of inducing inputs. 

I view this as an empirical report advocating for the use of OVI in DGP but even in this regard, this is still lacking in comparison with other related works. The field of DGP contains more than the two baselines the authors compare with. 

Experiments:

For example, deep GP learning & inference can also be made via expectation propagation -- https://arxiv.org/abs/1602.04133; the Bayesian treatment of the inducing inputs has also been explored -- see http://web.mit.edu/~jaillet/www/general/neurips2019_main.pdf

For the latter work, isn't the part below Eq. (3) the same as what the authors proposed in 4.1 (which is the only new content added to OVI) & shouldn't it be compared with given that it is more recent and also compared with the two baselines used in this paper on almost the same set of datasets?

There is also another work in the line of Stein Discrepancy: https://arxiv.org/pdf/2009.12141.pdf which is also much more recent.

Note that I wouldn't request comparison with algorithms of other families such as the above if the core of this paper is about a new insight on expanding OVI (even if it is incremental) but in this case, it is a direct application of OVI to DGP and this entire work reads to me as an advocation of using OVI for learning DGP. This is also a fine angle but it has to be compared with other families of algorithms.

Suggestion for Future Improvement:

In fact, given the core components of this paper, I'd suggest the authors to at least revise the stance of this paper towards an empirical comparison of inference algorithms for DGPs and analysis of their pros and cons from both theoretical & experimental perspectives. Then, even if there is no particularly new algorithmic component, the insights from such experiments would still be useful to the community which is meritable. Otherwise, presenting it as an extension of OVI is not convincing.

--

There are also interesting theoretical facts established by this paper but they do not contribute to the algorithm designs. I believe these  are good-to-know facts but do not impact the algorithm design or shed important insights on its numerical behavior.   

Clarity:

The paper is clear on the high-level flow but there are a lot of typos, e.g. the 3rd last sentence of the abstract is only half a sentence; "distrbution" -- before Eq. (7); "objictive" -- 2nd sentence after Eq. (12)

There is also particularly problematic typo -- in 4.2 the authors stated that "In Section 3, we have developed OVI" which is not true given the cited reference in Section 3. An appropriate statement would be "we have reviewed OVI".

Fig 1. can be further improved in quality and clarity, e.g. annotations and labels could be added to make it clearer what the notations stand for (even though those are mentioned in the text narrative) -- the caption can also be more informative.

Citations for the mean-field assumption should be mentioned as soon as the assumption was mentioned in the Introduction. Also, in the first paragraph of the Introduction, there is a statement about the constructed probability distribution (in GP) is far from posterior contraction -- please do consider citing references that explore & demonstrate this.

Below Eq. (11): it is not immediately clear why the authors mention the non-integrability of neural network. It seems to be a condition that needs to be enforced by including a regularizer on the space of neural network generators; so please make it clear the connection between this statement & the construction in Eq. (12) -- why such regularizer helps encourage the neural net generator to meet the L2 constraint.

Reproducibility:

The code is released and the presentation of the algorithm is clear so I believe the reproducibility of this work.
summary_of_the_review: Currently, my take of this paper is that it is an application of OVI to learning DGP. Despite the claim that it develops NOVI which is an extension of OVI via the neural net generator of the posterior in Section 4.1, this is not new. First, it is on high-level the same as re-parameterization which has been incorporated into KSD (a preliminary version of SD-based VI which OVI was built on). Second, even if we want to discuss it specifically in the context of stochastic inducing input, the same perspective has also been explored in a recent NeurIPS-19 paper -- http://web.mit.edu/~jaillet/www/general/neurips2019_main.pdf.

Given that no new algorithmic insight was introduced, I do not believe the current position of this paper merits an acceptance. But I do suggest another angle for this work (see my specific suggestions above) which is more meritable but that requires more extensive experiments, involving comparison with other families of inference algorithm for DGPs as well as detailed analysis of their modeling pros & cons (demonstrating explicitly deficiencies of existing approaches in the specific context of DGP; and showing how those could be remedied by using OVI). 

---

These are my preliminary assessment of this paper. I am of course happy to discuss if the authors disagree.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes an approximate posterior inference algorithm for deep Gaussian processes. A sampleable approximate posterior parameterized by a neural network minimizes a regularized Stein discrepancy objective between itself and the true posterior distribution. Experiments are conducted on regression and classification problems.

The main concern, as shared by Reviewer vt6c and GwnC, is its lack of novelty. The main contribution over, for example, operator variational inference and kernel stein discrepancy, is its use of a neural parameterization and toward the specific model of deep Gaussian processes. This feels too incremental (and in fact OVI already included experiments with a neural generator). A thorough experimental evaluation could also be one of the paper's main contributions, but unfortunately there is also quite a lack of experimental comparison to other DGP inference methods.

All reviewers leaned toward reject, and I agree with their opinion.
justification_for_why_not_higher_score: * Lack of novelty
* Lack of experimental comparison to competing approaches
Decision: Reject