Title: Robust and Computation-Aware Gaussian Processes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Gaussian Processes

Bayesian optimization

Probabilist Numerics
Abstract: Gaussian processes (GPs) are widely used for regression and optimization tasks such as Bayesian optimization (BO) due to their expressiveness and principled uncertainty estimates. However, in settings with large datasets corrupted by outliers, standard GPs and their sparse approximations struggle with computational tractability and robustness.
We introduce Robust Computation-aware Gaussian Process (RCaGP), a novel GP model that jointly addresses these challenges by combining a principled treatment of approximation-induced uncertainty with robust generalized Bayesian updating. The key insight is that robustness and approximation-awareness are not orthogonal but intertwined: approximations can exacerbate the impact of outliers, and mitigating one without the other is insufficient.
Unlike previous work that focuses narrowly on either robustness or approximation quality, RCaGP combines both in a principled and scalable framework, thus effectively managing both outliers and computational uncertainties introduced by approximations such as low-rank matrix multiplications.
Our model ensures more conservative and reliable uncertainty estimates, a property we rigorously demonstrate. Additionally, we establish a robustness property and show that the mean function is key to preserving it, motivating a tailored model selection scheme for robust mean functions. Empirical results confirm that solving these challenges jointly leads to superior performance across both clean and outlier-contaminated settings, both on regression and high-throughput Bayesian optimization benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes to combine the idea of computation-aware gaussian processes (CaGP) with robust conjugate GP (RCGP). This way the proposed method allows for conservative uncertainty estimates \nand robustness to outliers. \nExperiments on standard regression tasks and Bayesian optimization tasks suggest that the proposed method leads to better uncertainty estimate with and without outliers when compared to CaGP and RCGP.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Extends theoretical results from CaGP and RCGP to their proposed method.\n- Empirical evaluation showing that proposed method improves uncertainty quantification compared to CaGP and RCGP\n\nWeakness:\n- The proposed method inherits from CaGP and RCGP various hyper-parameters like the projection dimension, choice of mean function, choice of weighting function (= choice of $\\beta$ and $c$).\nFor choosing the mean function, the work assumes that (noisy) expert labels are available that flag potential outliers. This seems to be a rather strong assumption.\nIt is also unclear how the other hyper-parameters are chosen (except $c$ which is explained in the experiments section)\n- RCGP is the only other robust method that is compared.\nHowever, like their proposed method RCGP is known for its sensitivity to mean prior.\nI would find it more convincing, if the work compared also to the method of [Ament, 2024].\n- I guess the projection dimensionality directly affects the computational costs.\nSince one of the main claim is computational awareness, I would like to see how the results of the proposed method change, if less computational resources are used.\nIn particular, in Table 1, currently the proposed method is almost always more computationally expensive than RCSVGP.\n- Proposition 4.2 and Equation (22) are hard to understand. (I guess there is some type in (22)?)\n\nReferences:\n[Ament, 2024]\nAment, Sebastian, et al. \"Robust Gaussian processes via relevance pursuit.\" Advances in Neural Information Processing Systems 37 (2024): 61700-61734.\n\nQUESTIONS:\n- It seems the weight function requires that an expert labels the outliers.\nHow are these labels exactly chosen in the experiments of Table 1 and Figure 3?\n\n- How is $\\beta$ of Equation (12) chosen?\n\n- How is the projection dimensionality chosen.\n\n- Clarification of the above questions and additional results with the method from [Ament, 2024] (see Weakness section) would affect my scoring.\n\nLIMITATIONS:\nno, the choice of all the hyper-parameters is not discussed as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the intertwined challenges of computational scalability and model robustness in Gaussian Processes when applied to large datasets corrupted by outliers. The authors introduce the RCaGP, whose core contribution is the principled integration of two key ideas: (1) leverages the mechanism of RCGPs to down-weight the influence of outliers, (2) simultaneously incorporating the approach of CaGPs to quantify and compensate for uncertainty arising from computational approximations, thereby preventing overconfident predictions. The paper also provides theoretical guarantees for RCaGP's robustness and conservative uncertainty estimates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) The paper is generally well written and well presented. \n(2) The proposed RCaGP model is not merely a simple superposition of existing methods (RCGP and CaGP), but rather a principled integration within a unified probabilistic framework. This approach of jointly solving the problem surpasses previous work that has tackled these challenges in isolation. \n(3) The paper provides theoretical support for the reliability of RCaGP. The model's robustness is proven through a bounded Posterior Influence Function (PIF), which stands in sharp contrast to the unbounded PIF of CaGP.\n\nWeaknesses:\n(1) Parameters such as the soft threshold c, the low-rank approximation dimension, and the size of the approximate action matrix require manual setting or cross-validation by the user, which could otherwise affect the balance between robustness and approximation quality. \n(2) The paper highlights the model's critical dependence on the prior mean m(x). The success of this mechanism hinges entirely on the quality of m(x). The use of an expert-guided prior is intended to directly address this weakness in model's robustness mechanism by providing a high-quality \"initial guess\" to enhance its core ability to identify and handle outliers. While the proposed expert-guided method is a good solution, it also intro a potential weakness of the model: in the complete absence of domain knowledge, how to set a reasonable prior mean remains an open question. If the domain expert's annotations are biased or insufficient, the constructed mean prior could potentially introduce misleading information. \n(3) Although the paper mentions that only posterior expectations are used for computational efficiency, requiring experts to review and correct a certain proportion of potential outliers in large-scale datasets is a process that is inherently difficult to scale, which seems to conflict with the paper's highlight on High-Throughput applications. \n(4) In the \"Related Work\" section, the paper mentions the first category of robust GP methods, which replace the Gaussian noise assumption with heavy-tailed distributions. Although the authors note that these methods often break conjugacy and require approximate inference, a direct comparison of RCaGP with a representative model from this category (e.g., an SVGP based on a Student-t likelihood) would more forcefully demonstrate whether RCaGP, while maintaining its advantage of conjugacy, still leads in terms of performance.\n\nQUESTIONS:\n(1) The effectiveness of the algorithm does not seem to be limited to the Matérn kernel function. It would be beneficial to include some discussion and experiments regarding other common GP kernel functions?\n(2) Outliers are caused by noise. Is there any discussion regarding the outliers generated by different magnitudes of noise? It seems that GPs perform differently under varying noise levels.\n(3) In the BO experiments, while using 250 points as the initial sample size ensures a fair comparison, this number is relatively large. For instance, 20 points are often sufficient for a 6-dimensional problem. This large initial set might mask the differences in sample efficiency among the models during the early exploration phase?\n\nLIMITATIONS:\n(1) The paper's core mechanism for achieving robustness through the weight function w(x,y) is critically dependent on a high-quality prior mean m(x). Algorithm identifies outliers by measuring the deviation of an observation y from m(x). This means that if m(x) is poorly specified, the model may erroneously down-weight normal data points or fail to identify true outliers, leading to a degradation in the performance of the entire robustness framework.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generalized Gaussian process (GP) model that jointly addresses two practical limitations of GPs in real-world settings: robustness to outliers and computational tractability. The approach leverages generalized Bayesian inference with a robust loss to downweight corrupted observations, and incorporates uncertainty stemming from low-rank approximations to the posterior covariance (termed computational awareness). A prior over potentially corrupted data points is introduced. Hyperparameters are optimised using a generalised evidence lower bound. The method is supported by theoretically results on robustness and conservative uncertainty estimates. Experiments are conducted on four UCI regression tasks and four Bayesian optimization (BO) tasks under various data corruption scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n### **Quality**\n\nThe paper is of high technical quality. It builds upon and synthesizes recent ideas from the GP literature on robust inference and computational uncertainty aware approximations. The methodology is sound, and the theoretical contributions are well presented. I expect this work to be of interest to the GP community. \n\n+ **Strengths:** Sound methodology and interesting integration of existing techniques.\n+ **Weaknesses:** Experimental evaluation is limited in both scope (only 4 UCI datatsets and 4 BO tasks) and realism (artifical corruption of real-world data). The proposed model is evaluated only under synthetic corruption regimes, raising concerns about the practical relevance of the results. In the BO experiments in particular, the small data regime and low input dimensionality seem insufficient to highlight the method’s strengths in realistic computational or model mismatch scenarios.\n\n### **Clarity**\n\nThe paper is clearly written, well organized, and enjoyable to read. Background and related work are well covered, and the key contributions are clearly articulated. Figures and tables are generally helpful, although there are a few missing items that would improve clarity (e.g., what \"std\" refers to in Table 1, explicit definition of the robust loss in the main text).\n\n### **Significance**\n\nThe paper tackles an important challenge by aiming to make Gaussian processes more robust to outliers and more reliable under computational constraints. These are meaningful goals, especially for real-world applications where data quality and compute resources are often limiting factors. However, I have concerns regarding the practical applicability of the proposed approach in its current form.\n\nMethodologically, the reliance on a prior over corrupted observations assumes some knowledge about which datapoints are likely to be outliers. In many practical scenarios (especially in Bayesian optimization) such prior knowledge may be unavailable or difficult to specify. A clearer discussion of real-world use cases where this modeling assumption is justified would help assess the method’s relevance. Similarly, the need to set additional hyperparameters in the weight function (e.g., $\\beta$, $c$) raises concerns: how sensitive is the method to these choices in practice and how would they be selected?\n\nExperimentally, the current evaluation setup appears somewhat artificial. The benefits of the method are demonstrated mainly through controlled corruptions of existing datasets. While the method indeed performs well under these scenarios, such setups are closely aligned with the assumptions it was designed around, making the improvements less surprising. In contrast, the paper does not convincingly demonstrate performance gains in settings that more closely reflect real-world noise or model mismatch. The BO tasks used are relatively high-dimensional and involve small datasets ($\\leq 10250$ samples), where misspecification and computational approximations of the posterior are unlikely to be a limiting factor. This undermines the motivation for computational awareness in this context.\n\nUltimately, the method may indeed offer substantial benefits in the presence of real-world corruptions or large-scale settings where exact inference is infeasible. However, the current experimental design does not convincingly establish such relevance. A stronger empirical case would require identifying or constructing realistic scenarios where the proposed robustness and approximation-aware inference offer clear advantages over standard GPs. \n\nAdditionally, reporting statistical significance (e.g., by bolding methods that are not significantly worse than the top performer in Table 1) would help provide a more nuanced view of the method's effectiveness.\n\n### **Originality**\n\nThis paper offers a combination of existing ideas that have not yet been jointly explored. The integration of outlier-aware inference with posterior approximation-aware uncertainty modeling is elegant and fills a gap in the GP literature. The contribution is more incremental than novel, but still worthwhile.\n\nQUESTIONS:\n+ **Prior over corrupted observations:** The method relies on a prior over which points are corrupted. Can the authors clarify how this would be constructed in practical scenarios? What are real-world applications where such a prior is meaningful or available?\n+ **Choice of weight function parameters:** The method depends on the parameters $\\beta$ and $c$ in the weighting function $w(x,y)$. How sensitive is the performance to these values, and how should practitioners choose them?\n+ **Robustness definition:** The robustness guarantee in line 192 appears weak, as it allows for any corruption inducing a non-infinite KL divergence. Can the authors clarify the intuition and implications of this result, and compare it to bounded worst-case divergence?\n+ **Experimental realism:** The experimental setup, especially for BO, appears somewhat artificial. Can the authors include additional BO experiments without synthetic corruption and on higher-dimensional or larger-scale tasks where computational limitations are more pressing?\n+ **Statistical significance:** It would improve interpretability if Table 1 clearly indicated which methods are not significantly worse than the best. A paired sample test could help here.\n+ **Figure 1:** The predictive mean of RCaGP appears to poorly fit data at high input values. Can the authors explain this behavior? Is it due to initialization, optimization, or hyperparameters?\n+ **GP hyperparameters for BO:** How were the GP lengthscales initialized for BO? This is particularly important in high-dimensional tasks like Rover and DNA [1]\n+ **Additional BO results:** Could the authors report results for BO without corruption across all tasks, not just Hartmann and Lunar?\n\n[1] Vanilla Bayesian Optimization Performs Great in High Dimensions, Hvarfner et al. 2024\n\nLIMITATIONS:\nThe authors acknowledge the need for a broader evaluation campaign in the conclusion. I see no obvious societal impact of this work that would have been missed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors extend the computationally aware Gaussian process (CaGP) model by integrating concepts from robust and conjugate Gaussian processes (RCGP) [Altamirano, Briol & Knoblauch, 2024]. They begin by establishing the model with an RCGP prior and then apply CaGP's variational inference approach to the representer weights of the conjugate posterior.\n\nIn addition to this core contribution, the authors:\n\n* Propose a prior mean function based on outlier detection from an expert (Section 5).\n* Demonstrate that the impact of adding contaminated data into the posterior computation, specifically the posterior influence function (PIF), is bounded in terms of the contamination (Proposition 4.1, Appendix D), an approach analogous to Altamirano, Briol & Knoblauch (2024).\n* Extend results from Wenger et al. (2024), based on Kanagawa et al. (2018), to directly link RCaGP's posterior variance with the supremum of the squared difference between the predictive centered mean and all $h(x)$ with an RKHS unit norm or less. This assumes the dataset is generated by a latent function $g:\\\\mathcal{X}\\\\to\\\\mathbb{R}$ and its centered version $h(x) = (g(x)-m\\_w(x))$ belongs to the RKHS of the data generating process (Proposition 4.2, Appendix E).\n* Empirically evaluate their model on four well-known small-scale ($n<1000$) UCI regression datasets and Bayesian optimization tasks, artificially introducing outliers.\n\nSTRENGTHS AND WEAKNESSES:\nI find this paper to be well-rounded in its contributions, translating expected results from its main source papers into this new setting. While some theoretical results might appear to be straightforward extensions of previous work, the proposal for the expert-guided prior mean and the comprehensive experimental evaluation provide interesting and valuable additions to the use of Gaussian Processes for robust regression, which should satisfy most readers.\n\nNonetheless, while the authors compare their method to the original CaGP and RCGP approaches, I believe readers might struggle to contextualize this proposal in terms of uncertainty quality and runtime when compared against non-conjugate GP regression methods, such as those employing MCMC or Sparse Variational Gaussian Processes (SVGP).\n\nQUESTIONS:\nCould the authors clarify the definition of $m\\_w(x)$ in Proposition 4.2? While the meaning of $m\\_w(\\\\mathbf{X})$ is not explicitly clear, we can infer that it also depends on the vector $\\\\mathbf{y}=g(\\\\mathbf{X})$. However, for the assumption that $(g(x)-m\\_w(x))$ belongs to the RKHS of the data-generating kernel, are we meant to assume that $m\\_w(x)$ in this context signifies $m\\_w(x,g(x))=m(x)-\\\\sigma^2\\_{\\\\text{noise}}2\\\\frac{\\\\mathrm{d}}{\\\\mathrm{d}g(x)}\\\\log w(x,g(x))$?\n\nThe assumption that $(g(x)-m\\_w(x))$ belongs to an RKHS is similar to results from Wenger et al. (2024) based on Kanagawa et al. (2018). This assumption appears straightforward to accept in the standard Gaussian Process Regression (GPR) case, as the GP posterior mean is typically a member of the RKHS when the mean function is considered. In the current context, the shrinkage term alters this interpretation, as it effectively influences what would otherwise be considered the prior mean of the GP. I believe the authors should clarify this assumption and discuss the conditions under which it might or might not hold, thereby providing better context for Proposition 4.2.\n\n**Possible Typo:** In the inline equation on line 95 defining $\\\\mathbf{C}\\_i$, the inverse should be placed outside the parenthesis, rather than being attached to $\\\\hat{\\\\mathbf{K}}$.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}