Title: Efficient Sampling for Doubly Stochastic Variational Inference in Deep Gaussian Processes Regression
Venue: NeurIPS 2025
Accepted: no
Keywords: gaussian processes

regression

random fourier features

machine learning

variational inference

neural network
Abstract: Deep Gaussian Processes (DGPs) enhance Gaussian Processes (GPs) in function approximation through multi-layer stacking. However, the inference of DGPs presents challenges as it has no closed-form solution. Existing methods approximate the posterior of DGPs through independent sampling and variational inference. These approaches overlook the samples' correlations and face substantial computational overhead as layers increase, hindering performance improvements. We present Efficient Deep Gaussian Processes (EDGPs) that enable efficient sampling between inner layers while maintaining full covariance characteristics. Unlike existing methods that compromise accuracy for speed, EDGP achieves high efficiency without sacrificing precision. Experiments show that EDGP has comparable, or even better performance than state-of-the-art Doubly Stochastic Deep Gaussian Processes (DSDGPs) while training is almost as efficient as basic neural networks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes efficient deep Gaussian processes (EDGP), a new method for scalable and accurate inference in deep Gaussian processes (DGPs). Standard DGP inference methods like Doubly Stochastic DGPs (DSDGP) rely on variational inference over inducing points and sample inner-layer function values (per datapoint) using the reparameterisation trick where only the  diagonal of the marginal posterior is needed. The crux of this approach is that using a weight space view of GPs and RFF basis functions.\nThe key idea is to sample from the GP prior using RFFs and then apply a closed-form correction to approximate posterior samples. Two theoretical propositions underpin this method:\n\n- Posterior samples can be obtained from prior samples plus a correction term involving the difference between prior and variational samples over inducing points.\n- Prior samples can be efficiently computed as weighted sums of RFF basis functions.\n\nThis approach avoids repeated Cholesky decompositions and maintains the full posterior covariance structure across layers, in contrast to DSDGP’s diagonal sampling. Empirical evaluations show improved runtime performance with stronger MSE / MAE metrics across 4 high-dimensional regression datasets. Important to note that all other aspects of derivation of the ELBO, SVGP minibatch framework are the same...the inducing points, means and covariances of the inducing variables per layer are learnt using the ELBO and so are the weight space hypers.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n- Improved sampling of the marginal posterior while retaining O(NM^2) complexity. What is interesting is that the point about DSDGP implicitly resorting to diagonal approximation of the marginal posterior for sampling in each iteration is a very subtle point and the original Salimbeni, 2017 paper doesn't draw attention to it at all. One has to really pin down the fact that the recursive forward pass is done sample by sample in the minibatch so only variances end up being needed (hence diagonal) for the reparam trick.\n\n- Reparameterization enables avoiding re-sampling, EDGP decouples randomness from input locations, once random weights are sampled, function evaluations at new locations are deterministic evaluations of basis functions.\n\n- Empirically demonstrates better scaling with depth than DSDGP, both in terms of runtime and predictive performance.\n\nWeaknesses\n\n- EDGP relies on weight-space sampling using RFFs, which are only valid for stationary kernels (e.g. RBF) and the authors do mention this. It cannot currently handle non-stationary kernels, however, a more pernicious issue maybe that the accuracy of the RFF-based approximation depends on the number and quality of basis functions, and tuning these introduces additional complexity not present in standard function space GPs.\n\n- Less natural interpretation of posterior samples. The recursive sampling in the weight space + correction framework is harder to understand than function-space sampling.\n\n- Despite the sampling trick, the ELBO still needs to be estimated using Monte carlo samples, and the variance of this estimate could still be an issue in low data regimes.\n\n- The regression evaluation mainly focuses on MSE/MAE ..prediction error based metrics, a glaring omission is the uncertainty quantification, NLPD performance under test data of this sampling scheme.\n\nQUESTIONS:\n1. The empirical evaluation focuses solely on point-prediction metrics like MSE and MAE, which assess accuracy but not calibration or uncertainty quality. Given that the main contribution of EDGP is to improve posterior sampling particularly preserving full covariance and better representing uncertainty it is a glaring omission that no results are reported on negative log predictive density (NLPD) \nWithout them, it's unclear whether the retained posterior correlations actually lead to better-calibrated predictions.\n\n2. How sensitive is the method to the number of basis functions used in the RFF expansion? Is there a trade-off curve between basis count, runtime, and approximation quality?\n\n3. Is the prior correction step stable numerically across layers and time? Could subtracting two sampled quantities lead to variance amplification?\n\n4. Do improved posterior samples translate into better performance in downstream tasks like Bayesian optimization or active learning, where uncertainty matters?\n\n5. What about comparison to Havasi et al. (2018)\n\nLIMITATIONS:\nWhile the authors have talked about the main limitation of this approach only allowing for RBF kernels, in my mind, it is very important to address if the uncertainty calibration is improved or not in terms of covarage rates, NLPD, calibration curves.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of inefficient training in doubly stochastic deep Gaussian processes. The authors appeal to random Fourier features as Gaussian kernel approximations, blending their construction with a weight-space view to use them as more traditional kernels, and randomly sampling them in intermediate layers. Basic numerics are provided on some benchmark datasets.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper uses a neat idea, but in my opinion is not ready for publication. There are a number of places where the paper is unclear, errors, and extensions to the numerics which I believe would greatly improve the work. \n\nThe bibliography is incorrect, citing the authors by last name. This made it confusing to navigate. There are a number of typos and grammatical errors which should be addressed. \n\nThe construction of GPs should be revisited. There are a number of subtleties here and the presentation doesn't appear to be any of the standard formulations. For example, what is meant by $f(y;x) \\sim N(m(X),k(X,X))$? What are coordinate vs non-coordinate random variables?\n\nLines 92-95 are unclear. How exactly is the ELBO obtained in your construction?\n\nRandom Fourier features are usually not constructed in the weight-space view. This is evident in your proof of proposition 2 (which isn't actually proven, but rather some calculations outlined in [40] are given), where it is easy to see that they form a Monte Carlo estimate of the Fourier transform of a Gaussian distribution, which happens to have a complex inner-product structure. The link between these two constructions is never made explicit, which I believe would be beneficial if weights are explicitly being used in the RFF maps. \n\nSome clarity around complexity would be beneficial. For example, by line170 subsampling has already been implicitly introduced via the use of VI and inducing points -- how is O(N^3) still the bottleneck?\n\nI would suggest looking at the proofs again and making sure that it is clear that the statement in the propositions is explicitly being proved.\n\nIt is unclear exactly what is being sampled and what's being trained. It appears that weights are sampled in line 5 in algorithm 1. Is it just the inducing points and linear means that are being optimized in training?  In line 186-187 what is meant by 'adjust the samples based on observations'? Please be clear which parameters are being sampled, and which are being trained.\n\nOne epoch training time is reported. However, different methods can converge at different rates. What is the total training time for each method, and what convergence criteria were used?\nAre the reported MSE/MAE values for training data or a test set?\nStandard deviations would be useful in Table 1 to help distinguish performance. All of the authors methods have been bolded in this table, however it is usual to only bold the best performing method (or methods, if the methods are not statistically distinguishable).\n\nWould be typical to use some baseline datasets that are shared with prior work describing DSDGPs.\n\nQUESTIONS:\nCan you please address the questions posed in the \"Strengths and Weaknesses\" section?\n\nLIMITATIONS:\nSome limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 1, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Efficient Deep Gaussian Processes (EDGP), a novel variational inference scheme for deep Gaussian processes (DGPs) that avoids the standard computational bottlenecks of doubly stochastic methods. EDGP leverages a weight-space formulation using random Fourier features (RFFs) combined with a posterior correction mechanism to retain full covariance structure during inference, while remaining computationally efficient. The method is benchmarked against DSDGP, VFE, and neural baselines on several regression datasets, showing improved runtime and competitive or superior predictive performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n- Proposes a new alternative to DSDGP, avoiding diagonal approximations and Cholesky decompositions.  \n- Preserves full posterior covariance across layers while maintaining linear-time sampling via basis function reuse.  \n- Theoretical contributions (Propositions 1 and 2) are clearly formulated and rigorously proven.  \n- Strong empirical results, particularly in deeper models, with consistent gains in runtime and accuracy on complex datasets.  \n- Reproducibility is prioritized, with thorough experimental setup, multiple runs, and access to code.\n- It is strongly based on prior work, offering meaningful improvements to a known method. At some points, specially due to the content split in the article, the work may feel more incremental than what it really is. However, it is a solid contribution, technically valuable and well-executed.\n\n**Weaknesses**  \n- The method is tied to stationary kernels (e.g., RBF) due to its reliance on RFFs; generalization to non-stationary settings remains an open issue.  \n- Sensitivity to hyperparameters (e.g., number of basis functions, sample size) is briefly touched upon but not fully explored.  \n- Some results (e.g., on SRU) suggest dataset-dependent behavior that could benefit from deeper analysis or adaptive model components.\n\n  **Minor**\n  - The paper spents a lot of effort into describing previous approaches s.a. VFE and DSDGP. It could benefit from a more concise introduction on these topics, focusing on the contribution and key differences with EDGP.\n\nQUESTIONS:\n1. Can the EDGP framework be extended to non-stationary kernels or adaptive basis functions?  \n2. How does the method perform in classification or heteroscedastic settings?  \n3. Are there practical guidelines for selecting the number of basis functions and samples?  \n4. Could the method support joint learning of kernel hyperparameters via gradient-based training?\n5. Could this approach be adapted for Deep Variational Implicit Processes (DVIPs) or other generalized DGP frameworks with function-space formulation? Moreover, how would this compare to those methods or even to baseline Implicit Processes approaches s.a. variational IPs or Sparse IPs?\n\nLIMITATIONS:\n- The method is currently applicable only to stationary kernels and assumes RFF compatibility.  \n- Performance depends on the accuracy of covariance approximation via finite samples and basis functions.  \n- Experiments are limited to regression tasks; broader applicability (e.g., classification, structured outputs) is not demonstrated.  \n- The model assumes fixed kernel forms across layers; compositional or learned kernel structures are not explored.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Efficient Deep Gaussian Processes (EDGP), which aims to accelerate inference in deep Gaussian process models by using a weight-space sampling approach based on random Fourier features, combined with a posterior correction mechanism. The paper claims both theoretical equivalence in computational complexity to existing doubly stochastic DGP (DSDGP) methods and empirical advantages in runtime and accuracy.\nWhile the paper is well-structured, it contains significant issues in terms of prior work attribution, technical accuracy, and empirical validation, which collectively warrant rejection.\n\nSTRENGTHS AND WEAKNESSES:\nMajor Concerns:\n1. Unacknowledged Overlap with Wilson et al. (2020):\nThe core sampling strategy adopted in EDGP — expressing samples from a GP posterior as a prior sample plus a correction term — was previously introduced in Wilson et al. (2020), Efficiently Sampling Functions from Gaussian Process Posteriors (ICML). This is not appropriately acknowledged in the main text.\nIn Section 3 (Lines 177–178), the authors state:\n\"Proposition 1 offers a novel perspective on the inference propagation: rather than sampling directly from the distribution, one can sample from the prior and apply a correction based on observations.\"\nHowever, this is not novel; it mirrors the central idea of Wilson et al. (2020)( Corollary 2 and Equations (9)–(13)), though no citation is provided here\nWilson et al. (2020) [26] is cited only in passing in the introduction (Line 50), in the sentence: \"Spectral methods are limited to stationary conditions [26,27,28],\"\nand grouped with two other unrelated references, without any clear explanation of how the method proposed here differs substantively from that prior work.\nIn its current form, the manuscript gives the impression that this sampling mechanism is an original contribution of the present work, which is misleading and significantly diminishes the claimed novelty.\nMore concretely, the key results presented in Proposition 1 and Proposition 2 of this paper are essentially equivalent to Corollary 2 and Equations (9)–(13) of Wilson et al. (2020). The core idea — using Matheron’s rule to decompose the GP posterior sampling — is the same. The primary difference is that this paper applies the method in the context of DGP path sampling; however, this point is not clearly discussed or acknowledged in the current manuscript, which severely impacts the clarity of the paper’s positioning and contribution.\nFurthermore, Matheron’s rule-based GP sampling, as presented in Wilson et al. (2020), has since been widely adopted and is now a well-known approach (see, e.g., [R1-R4]). It is therefore unclear whether simply applying this established sampling scheme to DGPs constitutes a sufficient novelty contribution for NeurIPS. At a minimum, proper attribution and a clearer delineation of the contribution delta are required.\n[R1] Maddox, W. J., et al. (2021). Bayesian optimization with high-dimensional outputs. NeurIPS.\n[R2] McDonald, T. M., et al. (2023). Nonparametric Gaussian process covariances via multidimensional convolutions. AISTATS.\n[R3] Wilson, J. T., et al. (2021). Pathwise conditioning of Gaussian processes. JMLR.\n[R4] Wenger, J., et al. (2022). Posterior and computational uncertainty in Gaussian processes. NeurIPS.\n2. Incorrect Complexity Claims:\nThe paper contains multiple inaccurate statements about the computational complexity of DGP methods with inducing points.\nFor instance, in Section 3 (Line 169-170), the authors write:\n\"First, even when the distributional form is clear, sampling incurs a time cost of O(N³).\"\nThis is factually incorrect. Sparse GPs with inducing variables typically achieve O(Nm²) training complexity and O(m²) sampling complexity, where m is the number of inducing points.\nSimilar claims are repeated in Lines 202, exaggerating the computational cost of standard variational DGPs. These technical inaccuracies could mislead readers and should be corrected.\n3. Weak and Limited Empirical Evaluation:\nThe experiments are conducted on only four small-to-moderate-scale regression datasets (Section 4). There is no validation on large-scale datasets or more diverse tasks (e.g., classification, image data), significantly restricting the generalizability of the conclusions.\nThe comparisons are limited to the standard DSDGP (proposed in 2017) and overly simplistic baselines (e.g., FCN, single-layer GPR). Recent advances in DGP inference—such as thin and deep GPs [R5], importance-weighted variational inference [R6], global inducing point methods [R7], implicit posterior variational inference [R8], and sparse inducing points with diffusion variational inference [R9]—are not included, leaving the competitiveness of EDGP unclear.\nAlthough the paper claims that EDGP achieves acceleration while maintaining the same theoretical complexity, this assertion is only supported by raw training time measurements. Neither ablation studies nor breakdowns of computational processes (e.g., optimization, inference, or sampling time) are provided, making it impossible to identify the specific source of performance improvement. I recommend that the authors conduct theoretical attribution or error analysis for EDGP's acceleration, which could potentially enhance the paper's contribution.\nAs shown in Tables 1 and 2, EDGP's improvements over DSDGP are limited to MSE and MAE metrics, with marginal gains and even inferior performance on the SRU dataset. However, the authors fail to provide systematic analysis of these inconsistent results (e.g., hyperparameter sensitivity, scalability limits, or dataset characteristics). I suggest incorporating broader evaluation metrics such as NLL (Negative Log-Likelihood) for more comprehensive performance assessment.\nKey References for State-of-the-Art DGP Methods:\n[R5] Thin and Deep Gaussian Processes (NeurIPS 2023)\n[R6] Importance-Weighted Variational Inference for DGPs (NeurIPS 2019)\n[R7] Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes (ICML 2021)\n[R8] Implicit Posterior Variational Inference for DGPs (NeurIPS 2019)\n[R9]Sparse inducing points in deep gaussian processes: enhancing modeling with denoising diffusion variational inference (ICML 2024)\n\n\n4. Lack of Theoretical Justification for Efficiency Claims:\nWhile the paper claims that EDGP is as expressive as DSDGP and offers better scalability, this is not supported by theoretical analysis.\nIn Section 3 (Line 194), the authors claim:\n\"Although EDGP and DSDGP share the the same theoretical computational complexity..., EDGP demonstrates significantly faster empirical performance.\"\nHowever, the paper does not provide any concrete derivation or complexity analysis to justify where and how the empirical efficiency arises.\nThe lack of layer-wise runtime profiling or variance reduction analysis makes it difficult to assess whether the improvements come from algorithmic design or implementation factors.\nTherefore, in its current form, I do not believe this paper meets the standard of novelty and rigor required for NeurIPS.\n\nQUESTIONS:\nPlease the part of Strengths And Weaknesses, where I have listed many issues.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}