Title: Robust Gaussian Processes via Relevance Pursuit
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian process

robust regression

Bayesian optimization

submodular
Abstract: Gaussian processes (GPs) are non-parametric probabilistic regression models that are popular due to their flexibility, data efficiency, and well-calibrated uncertainty estimates. However, standard GP models assume homoskedastic Gaussian noise, while many real-world applications are subject to non-Gaussian corruptions. Variants of GPs that are more robust to alternative noise models have been proposed, and entail significant trade-offs between accuracy and robustness, and between computational requirements and theoretical guarantees. In this work, we propose and study a GP model that achieves robustness against sparse outliers by inferring data-point-specific noise levels with a sequential selection procedure maximizing the log marginal likelihood that we refer to as relevance pursuit. We show, surprisingly, that the model can be parameterized such that the associated log marginal likelihood is strongly concave in the data-point-specific noise variances, a property rarely found in either robust regression objectives or GP marginal likelihoods. This in turn implies the weak submodularity of the corresponding subset selection problem, and thereby proves approximation guarantees for the proposed algorithm. We compare the model’s performance relative to other approaches on diverse regression and Bayesian optimization tasks, including the challenging but common setting of sparse corruptions of the labels within or close to the function range.

Robust Gaussian Processes via Relevance Pursuit

Abstract

Gaussian processes (GPs) are non-parametric probabilistic regression models that are popular due to their flexibility, data efficiency, and well-calibrated uncertainty estimates. However, standard GP models assume homoskedastic Gaussian noise, while many real-world applications are subject to non-Gaussian corruptions. Variants of GPs that are more robust to alternative noise models have been proposed, and entail significant trade-offs between accuracy and robustness, and between computational requirements and theoretical guarantees. In this work, we propose and study a GP model that achieves robustness against sparse outliers by inferring data-point-specific noise levels with a sequential selection procedure maximizing the log marginal likelihood that we refer to as relevance pursuit . We show, surprisingly, that the model can be parameterized such that the associated log marginal likelihood is strongly concave in the data-point-specific noise variances, a property rarely found in either robust regression objectives or GP marginal likelihoods. This in turn implies the weak submodularity of the corresponding subset selection problem, and thereby proves approximation guarantees for the proposed algorithm. We compare the model's performance relative to other approaches on diverse regression and Bayesian optimization tasks, including the challenging but common setting of sparse corruptions of the labels within or close to the function range.

1 Introduction

Probabilistic models have long been a central part of machine learning, and Gaussian process (GP) models are a key workhorse for many important tasks [54], especially in the small-data regime. GPs are flexible, non-parametric predictive models known for their high data efficiency and well-calibrated uncertainty estimates, making them a popular choice for regression, uncertainty quantification, and downstream applications such as Bayesian optimization (BO) [8, 24, 26] and active learning [6, 55].

GPs flexibly model a distribution over functions, but assume a particular observation model. The standard formulation assumes i.i.d Gaussian observation noise, i.e., y ( x ) = f ( x ) + ϵ , where f ( x ) is the true (latent) function value at a point x and ϵ ∼ N (0 , σ 2 ) , implying a homoskedastic Gaussian likelihood. While mathematically convenient, this assumption can be a limitation in practice, since noise distributions are often heavy-tailed or observations may be corrupted due to issues such as sensor failures, data processing errors, or software bugs. Using a standard GP model in such settings can result in poor predictive performance.

A number of robust GP modeling approaches have been proposed to remedy this shortcoming, most of which fall into the following broad categories: data pre-processing (e.g., Winsorizing), modified likelihood functions (e.g., Studentt ), and model-based data selection and down-weighting procedures.

These approaches offer different trade-offs between model accuracy, degree of robustness, broad applicability, computational requirements, and theoretical guarantees.

In this paper, we propose a simple yet effective implicit data-weighting approach that endows GPs with a high degree of robustness to challenging label corruptions. Our approach is flexible and can be used with arbitrary kernels, is efficient to compute, and yields provable approximation guarantees. Our main contributions are as follows:

We propose a modification to the standard GP model that introduces learnable data-point-specific noise variances.

We introduce a novel greedy sequential selection procedure for maximizing the model's marginal log-likelihood (MLL) that we refer to as relevance pursuit .

We prove that, under a particular parameterization, the MLL is strongly concave in the datapoint-specific noise variances, and derive approximation guarantees for our algorithm.

We demonstrate that our approach, Robust Gaussian Processes via Relevance Pursuit (RRP), performs favorably compared to alternative methods across various benchmarks, including challenging settings of sparse label corruptions within the function's range, see e.g. Figure 1.

2 Preliminaries

We aim to model a function f : X → R over some domain X ⊂ R d . With a standard Gaussian noise model, for x i ∈ X we obtain observations y i = f ( x i ) + ϵ i , where ϵ i ∼ N (0 , σ 2 ) are i.i.d. draws from a Gaussian random variable. ∥·∥ denotes the Euclidean norm unless indicated otherwise.

2.1 Gaussian Processes

A GP f ∼ G P ( µ ( · ) , k θ ( · , · )) is fully defined by its mean function µ : X → R and covariance or kernel function k θ : X × X → R , which is parameterized by θ . Without loss of generality, we will assume that µ ≡ 0 . Suppose we have collected data D = { ( x i , y i ) } n i =1 where X := { x i } n i =1 , y := { y i } n i =1 . Let Σ θ ∈ S n ++ denote the covariance matrix of the data set, i.e., [ Σ θ ] ij = k θ ( x i , x j ) + δ ij σ 2 , where δ ij is the Kronecker delta. The negative marginal loglikelihood (NMLL) L is given by

0.0

0.2

0.4

0.6

0.8

1.0

x

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

y

Posterior Predictive Distributions

Student-t

Relevance Pursuit

Truth

Corruptions

Observations

In the following, we will suppress the explicit dependence of the kernel matrix on θ for brevity of notation. For a comprehensive background on GPs, we refer to Rasmussen et al. [54].

2.2 Noise Models

Additive, heavy-tailed noise Instead of assuming the noise term ϵ i in the observation model to be Gaussian, other noise models consider zero-mean perturbations drawn from distributions with heavier tails, such as the Studentt [32], Laplace [39], or α -Stable [5] distributions. These types of errors are common in applications such as finance, geophysics, and epidemiology [18]. Robust regression models utilizing Studentt errors are commonly used to combat heavy-tailed noise and outliers.

Sparse corruptions In practice, often a small number of labels are corrupted. We will refer to these as 'outliers," though emphasize that the corrupted values may fall within the range of normal outputs. Sparse corruptions are captured by a model of the form y i = Z i f ( x i ) + (1 -Z i ) W i , where

Z i ∈ { 0 , 1 } and W i ∈ R is a random variable. Note that W i need not have (and rarely has) f ( x i ) as its mean. For instance, consider a faulty sensor that with some probability p reports a random value within the sensor range [ y l , y h ] . In this case Z i ∼ Ber ( p ) and W i ∼ U [ y l , y h ] . Software bugs, such as those found in ML training procedures, or errors in logging data can result in sparse corruptions.

3 Related Work

Data pre-processing Data pre-processing can be an effective technique for handling simple forms of data corruption, such as values that fall outside a valid range of outputs. With such pre-processing, outliers are handled upstream of the regression model. Common techniques include the power transformations [15], trimming, and winsorization. These methods can add substantial bias if not used carefully, and generally do not handle data corruptions that occur within the normal range of the process to be modeled. See [17] for a review on data cleaning.

Heavy-tailed likelihoods One class of robust methods uses additive heavy-tailed noise likelihoods for GPs, particularly Studentt [32], Laplace [53], and Huber [1], and could be extended with α -Stable distributions, which follow a generalized central limit theorem [5]. These models are less sensitive to outliers, but they lose efficiency when the outliers are a sparse subset of the observations, as opposed to global heavy-tailed noise. Furthermore, model inference is no longer analytic, necessitating the use of approximate inference approaches such as MCMC [49], Laplace approximation [64], expectation propagation (EP) [32], Expectation Maximization [53], or variational inference [61]. Shah et al. [56] take a related approach using a Studentt process prior in the place of the GP prior. Unfortunately, the Studentt process is not closed under addition and lacks the tractability that makes GPs so versatile. Alternative noise specifications include a hierarchical mixture of Gaussians [19] and a 'twinned' GP model [48] that uses a two-component noise model to allow outlier behavior to depend on the inputs. This method is suited for settings where outliers are not totally stochastic, but generally is not able to differentiate 'inliers' from outliers when they can occur with similar inputs.

Outlier classification Awasthi et al. [11] introduces the Trimmed MLE approach, which identifies the subset of data points (of pre-specified size) under which the marginal likelihood is maximized. Andrade and Takeda [10] fit GPs using the trimmed MLE by applying a projected gradient method to an approximation of the marginal likelihood. The associated theory only guarantees convergence to a stationary point, with no guarantee on quality. When no outliers are present, this method can be worse than a standard GP. Li et al. [40] propose a heuristic iterative procedure of removing those data points with the largest residuals after fitting a standard GP, with subsequent reweighting. The method shows favorable empirical performance but has no theoretical guarantees, and fails if the largest residual is not associated with an outlier. Park et al. [52] consider a model of the form y i = δ i + f ( x i ) + ϵ i , where outliers are regarded as data with a large bias δ i . Their random bias model is related to our model in that it also introduces learnable, data-point-specific variances. However, inference is done in one step by optimizing the NMLL with an inverse-gamma prior on the δ i 's, which - in contrast to the method proposed herein - generally does not lead to exactly sparse δ i 's .

Sample re-weighting Altamirano et al. [2] propose robust and conjugate GPs (RCGP) based on a modification to the Gaussian likelihood function that is equivalent to standard GP inference, where the covariance of the noise σ 2 I is replaced by σ 2 diag( w -2 ) and the prior mean m is replaced by m w = m + σ 2 ∇ y log( w 2 ) . The authors advocate for the use of the inverse multi-quadratic weight function w ( x , y ) = β (1 + ( y -m ( x )) 2 /c 2 ) -1 / 2 , which introduces two additional hyper-parameters: the soft threshold c , and the 'learning rate' β . Importantly, the weights w are defined a-priori as a function of the prior mean m ( x ) and the targets y , thereby necessitating the weights to identify the correct outliers without access to a model. This is generally only realistic if the outlier data points are clearly separated in the input or output spaces rather than randomly interspersed.

4 Robust Gaussian Process Regression via Relevance Pursuit

Our method adaptively identifies a sparse set of outlying data points that are corrupted by a mechanism that is not captured by the other components of the model. This is in contrast to many other approaches to robust regression that non-adaptively apply a heavy-tailed likelihood to all observations, which can be suboptimal if many observations are of high quality.

4.1 The Extended Likelihood Model

We extend the standard GP observation noise variance σ 2 with data-point-specific noise variances ρ = { ρ i } n i =1 , so that the i -th data point is distributed as

This is similar to Sparse Bayesian Learning [60] in which weight-specific prior variances control a feature's degree of influence on a model's predictions. The marginal likelihood optimization of ρ i in (2) gives rise to an automatic mechanism for the detection and weighting of outliers. The effect of y i on the estimate of f vanishes as ρ i → ∞ , similar to the effect of the latent varibales h in Bodin et al. [14]'s extended GP model f ( x , h ) , though h requires MCMC for inference. While many heteroskedastic GP likelihoods model noise as an input-dependent process [28, 35], our formulation does not require such assumptions, and is thus suitable for corruptions that are not spatially correlated.

̸

An elegant consequence of our modeling assumption is that we can compute individual marginallikelihood maximizing ρ i 's in closed form when keeping all ρ j for j = i fixed. In particular,

̸

Lemma 1. [Optimal Robust Variances] Let D \ i = { ( x j , y j ) : j = i } , ρ = ρ \ i + ρ i e i , where ρ , ρ \ i ∈ R n + , [ ρ \ i ] i = 0 , and e i is the i th canonical basis vector. Then keeping ρ \ i fixed,

where y ( x i ) = f ( x i )+ ϵ i . These quantities can be expressed as functions of Σ -1 = ( K + D σ 2 + ρ ) -1 :

where D σ 2 + ρ is a diagonal matrix whose entries are σ 2 + ρ .

The first component E [ f ( x i ) + ϵ i |D \ i ] 2 of (3) is the empirical error to y i of the model trained without the i -th data point, i.e., the leave-one-out (LOO) cross-validation error [54]. The second component V [ f ( x i ) + ϵ i |D \ i ] is the LOO predictive variance. The optimal solution to ρ i is only non-zero for those observations whose squared LOO error is larger than the LOO predictive variance at that point.

4.2 Optimization with a Maximum Number of Outliers

Without additional structure, inference of the noise variances ρ i does not yield desirable models, as the marginal likelihood can be improved by increasing the prior variance ρ i of any data point where Eq. (3) is greater than zero, even if that is due to regular (non-outlier) measurement noise. To avoid this, we constrain the number of non-zero ρ i , that is, ∥ ρ ∥ ∞ = |{ 0 < ρ i }| ≤ k < n . While this sparsity constraint mitigates over-flexibility, it gives rise to a formidably challenging optimization problem, as there are a combinatorial number of sparse outlier sets to consider. Even if the number of outliers n o were known, exhaustive search would still require considering n -choosen o possibilities.

For tractability, we iteratively add data points to a set of potential 'outliers' by setting their associated ρ i to be nonzero, using the closed-form expression for the optimal individual ρ i variances in Lemma 1. As the algorithm seeks to identify the most 'relevant' data points (as measured by L ) upon completion, we refer to it as Relevance Pursuit . This is Algorithm 1 with useBayesianModelSelection as false . Specifically, this is the 'forward' variant; Algorithm 2 in the Appendix presents an alternative 'backward' variant that we found to work well if the number of corrupted data points is large.

Crucial to the performance of the optimizer, it never removes data from consideration completely; a data point is only down-weighted if it is apparently an outlier. This allows the down-weighting to be reversed if a data point appears 'inlying' after having down-weighted other data points, improving the method's robustness and performance. This is in contrast to Andrade and Takeda [10]'s greedy algorithm, in which the exclusion of a data point can both increase or decrease the associated marginal likelihood. This means that their objective is not monotonic, a necessary condition to provide constant-factor submodular approximation guarantees for greedy algorithms, see Section 5.

4.3 Automatic Outlier Detection via Bayesian Model Selection

In practice, it is often impossible to set a hard threshold on the number of outliers for a particular data set. For example, a sensor might have a known failure rate, but how many outliers it produces will depend on the specific application of the sensor. Thus, is often more natural to specify a prior distribution p ( S ) over the number of outliers, rather than fix the number a priori . We leverage the Bayesian model selection framework [66, 45] to determine the most probable number of outliers in a data- and model-dependent way, aiming to maximize p ( S|D ) . This gives rise to Algorithm 1, with useBayesianModelSelction as true .

Computationally, we start by iteratively adding outliers up to the maximal support of the prior, similar to the procedure described in Section 4.2. We store a trace of models generated at each iteration, then approximate the model posterior p ( S i |D ) ∝ p ( D| S i ) p ( S i ) at each point in the trace. As the exact posterior is intractable, we approximate it with p ( D|S i ) = ∫ p ( D|S i , ρ S i ) d ρ S i ≈ p ( D|S i , ρ ∗ S i ) . Finally, we select the model from the model trace {S i } i that attains the highest model posterior likelihood. Imposing a prior on the number of outliers differs notably from most sparsity-inducing priors, which are instead defined on the parameter values, like l 1 -norm regularization. In practice, p ( S ) can be informed by empirical distributions of outliers. For our experiments, we use an exponential prior on |S| to encourage the selection of models that fit as much of the data as tightly as possible.

Regarding the schedule K in Algorithm 1, the most natural choice is simply to add one data point at a time, i.e. K = (1 , 1 , ... ) , but this can be slow for large n . In practice, we recommend schedules that test a fixed set of outlier fractions, e.g. K = (0 . 05 n, 0 . 05 n, . . . ) .

5 Theoretical Analysis

We now provide a theoretical analysis of our approach. We first propose a re-parameterization of the ρ i that maps the optimization problem to a compact domain. Surprisingly, the re-parameterized

problem exhibits strong convexity and smoothness when the base covariance matrix (excluding the ρ i ) is well-conditioned. We connect the convexity and smoothness with existing results that yield approximation guarantees for sequential greedy algorithms, implying a constant-factor approximation guarantee to the optimal achievable NMLL value for generalized orthogonal matching pursuit (OMP), a greedy algorithm that is closely related to Algorithm 1.

5.1 Preliminaries for Sparse Optimization

The optimization of linear models with respect to least-squares objectives in the presence of sparsity constraints has been richly studied in statistics [59], compressed sensing [3, 62], and machine learning [9, 67]. Of central importance to the theoretical study of this problem class are the eigenvalues of sub-matrices of the feature matrix, corresponding to sparse feature selections and so often referred to as sparse eigenvalues . The restricted isometry property (RIP) formalizes this.

Definition 2 (Restricted Isometry Property) . An ( n × m ) -matrix A satisfies the r -restricted isometry property (RIP) with constant δ r ∈ (0 , 1) if for every submatrix A S with |S| = r ≤ m columns,

where x S ∈ R r . This is equivalent to (1 -δ r ) I ⪯ ( A ∗ S A S ) ⪯ (1 -δ r ) I .

The RIP has been proven to lead to exact recovery guarantees [16], as well as approximation guarantees [20]. Elenberg et al. [23] generalized the RIP to non-linear models and other data likelihoods, using the notion of restricted strong convexity (RSC) and restricted smoothness.

Definition 3 (Restricted Strong Convexity and Smoothness) . A function f : R d → R is m r -restricted strong convex and M r -restricted smooth if for all ( x , x ′ ) in the domain D r ⊂ ( R d × R d ) ,

In the context of sparse optimization, we let D r be the set of tuples of r -sparse vectors whose difference is also at most r -sparse. In particular, D r = { ( x , x ′ ) s.t. ∥ x ∥ 0 , ∥ x ′ ∥ 0 , ∥ x ′ -x ∥ 0 ≤ r } .

Generalized orthogonal matching pursuit (OMP) [4, 43, 44] is a greedy algorithm that keeps track of a support set S of non-zero coefficients, and expands the support based on the largest gradient magnitudes, applied to the marginal liklihood optimization problem, S i +1 = S i ∪ arg max j ̸∈S |∇ ρ L ( ρ ) | j . Algorithm 1 generalizes OMP [62] by allowing more general support expansion schedules K , and specializes the support expansion criterion using the special problem structure exposed by Lemma 1.

5.2 The Convex Parameterization

The NMLL L of a GP (1) is the sum of a convex function ( · ) -1 and a concave function log det( · ) of K , and is therefore not generally convex as a function of the hyper-parameters θ , including the robust variances ρ . Here, we propose a re-parameterization that allows us to prove strong convexity guarantees of the associated NMLL. In particular, we let ρ ( s ) = diag( K 0 ) ⊙ ((1 -s ) -1 -1) , where K 0 := k ( X , X ) + σ 2 I and the inverse is element-wise. Note that ρ ( s ) is a diffeomorphism that maps s from the compact domain s ∈ [0 , 1] n to the entire range of ρ ∈ [0 , ∞ ] n .

Henceforth, we refer to the original ρ as the canonical or ρ -parameterization and the newly proposed ρ ( s ) as the convex or s -parameterization. Lemma 4 shows the Hessian of the s -parameterization.

Lemma 4. [Reparameterized Hessian] Let K s = k ( X , X ) + σ 2 I + D ρ ( s ) , ˆ K s = diag( K s ) -1 / 2 K s diag( K s ) -1 / 2 , and ˆ α = ˆ K -1 s diag( K s ) -1 / 2 y . Then

Based on this representation, we now derive conditions on the eigenvalues of ˆ K that imply the m -strong convexity and M -smoothness of the NMLL.

Lemma 5. [Strong Convexity via Eigenvalue Condition] Let ˆ K s as in Lemma 4. Then H s ≻ m if

where λ min , max (resp. ˆ λ min , max ) are the smallest and largest eigenvalues of K s , respectively ˆ K s .

Hessian ConditionNumber

3.0

2.5

101

2.0

1.5

1.0

0.5

0.0

100

0

1

2

3

Hessian ConditionNumber

3.0

2.5

10

2.0

1.5

1.0

0.5

0.0

10

0

一

2

3

The behavior Lemma 5 predicts is surprising and validated in Fig. 3. Notably, the denominator 'blows up' as K becomes close to unitary, making the inequality more likely to be satisfied, an indication that the convexity property of the NMLL is intimately linked to the RIP (Def. 2). Note that Lemma 5 is a condition for non-support-restricted convexity, which is stronger than is necessary for the approximation guarantees that rely on restricted convexity (Def. 3). However, sparse eigenvalues are generally difficult to compute exactly. Fortunately, covariance matrices of GPs naturally tend to exhibit a property that facilitates a different sufficient condition for convexity for all s ∈ [0 , 1] n .

̸

Definition 6 (Diagonal Dominance) . A matrix A is said to be δ -diagonally dominant if the elements a ij satisfy ∑ i = j | a ij | < δ | a ii | for all i .

Intuitively, the ρ i ( s ) that are selected to be non-zero by the greedy algorithm take on large values, further encouraging the diagonal dominance of the sub-matrix of K associated with the support of ρ . For this reason, the following condition on K 0 is sufficient to guarantee convexity for all s ∈ [0 , 1] n .

Lemma7. [Strong Convexity via Diagonal Dominance] Let m> 0 and K 0 be δ -diagonally dominant with δ < ( (5 -m ) - √ 25 -9 m +17 ) / 4 ≤ (5 -√ 17) / 4 ≈ 0 . 44 and

Then the NMLL is m -strongly convex for all s ∈ [0 , 1] n , i.e. ρ ( s ) ∈ [0 , ∞ ] n .

We attain similar results for M -smoothness, see Lemma 13 and Lemma 14 in the Appendix. Having proven m -convexity and M -smoothness conditions, we appeal to the results of Elenberg et al. [23].

Theorem 8. [Approximation Guarantee] Let K 0 = k ( X , X ) + σ 2 I be δ -diagonally dominant, s max > 0 be an upper bound on ∥ s ∥ ∞ , and suppose ∥ y ∥ , δ satisfy the bounds of Lemmas 7 and 14, guaranteeing m -convexity and M -smoothness of the NMLL for some m> 0 , M > 1 / (1 -s max ) 2 . Let s OMP ( r ) be the r -sparse vector attained by OMP on the NMLL objective for r steps, and let s OPT ( r ) = arg max ∥ s ∥ 0 = r, ∥ s ∥ ∞ ≤ s max L ( ρ ( s )) be the optimal r -sparse vector. Then for any 2 r ≤ n ,

where ˜ L ( · ) = L ( · ) -L ( 0 ) is normalized so that max s S ˜ L ( s S ) ≥ 0 for any support S .

Alimitation of the theory is that it assumes the other hyper-parameters of the GP model to be constant, as doing otherwise would introduce the non-convexity that is common to most marginal likelihood optimization problems. In practice, we typically optimize ρ jointly with the other hyper-parameters of the model in each iteration of RRP, as this yields improved performance, see App. D.5 for details.

6 Empirical Results

We evaluate the empirical performance of RRP against various baselines on a number of regression and Bayesian Optimization problems. Specifically, we compare against a standard GP with a Matern5/2 kernel ('Standard GP'), data pre-processing through Ax's adaptive winsorization procedure ('Adapt. Wins.') [12], and a power transformation ('Power Transf.') [15]. Further, we also consider a Studentt likelihood model from Jylänki et al. [32] ('Studentt '), the trimmed marginal likelihood model from Andrade and Takeda [10] ('Trimmed MLL'), and the RCGP model from Altamirano et al. [2]. Unless stated otherwise, all models are implemented in GPyTorch [25] and all experiments in this section use 32 replications. See Appendix D for additional details.

6.1 Regression Problems

Synthetic We first consider the popular Friedman10 and Hartmann6 [21] test functions from the literature. We use two data generating processes: uniform noise, extreme outliers at some fixed value, and heavy-tailed (Studentt ) noise at true function values. In these experiments, we compare the performance predictive log-likelihood. The results are shown in Fig. 4.

4

2

0

Hartmann6, 5% Constant Corruptions

Hartmann6, 20% Constant Corruptions

Hartmann6, Student-t Corruptions

RRP

Student-t

Power Transf.

Standard GP

Adapt. Wins.

4

2

0

2

Friedman10, 10% Uniform Corruptions

RRP

Student-t

Power Transf.

Standard GP

Adapt. Wins.

Friedman10, 25% Uniform Corruptions

0.2

0.4

0.6

0.8

1.0

Friedman10, Student-t Corruptions

RRP

Student-t

Predictive Log Likelihood

Twitter Flash Crash In Fig. 5, we report a comparison to Altamirano et al. [2]'s RCGP on data from the Dow Jones Industrial Average (DJIA) index on April 22-23 2013, which includes a sharp drop at 13:10 on the 23rd. The top panels shows that RCGP exhibits higher robustness than the standard GP, but is still affected by the outliers, when trained on data from the 23rd. RRP is virtually unaffected. Notably, RCGP relies on an a-priori weighting of data points based on the target values' proximity to their median, which can be counter-productive when the outliers are not a-priori separated in the range. To highlight this, we included the previous trading day into the training data for the bottom panels, leading RCGP to assign the highest weight to the outlying data points due to their proximity to the target values' median, thereby leading RCGP to 'trust' the outliers more than any inlier,

resulting in it being less robust than a standard GP in this scenario. See Appendix D.6 for additional comparisons to RCGP, on data sets from the UCI machine learning repository [34].

(a)

(b)

(d)

(c)

6.2 Robust Bayesian Optimization

GPs are commonly used for Bayesian optimization (BO), which is a popular approach to sampleefficient black-box optimization [24]. However, many of the GP models used for BO are sensitive to outliers and may not perform well in settings where such outliers occur. While Martinez-Cantin et al. [46] consider the use of a Studentt likelihood for BO with outliers, the use of other robust GP models has not been thoroughly studied in the literature.

Experimental setup We use Ament et al. [7]'s qLogNoisyExpectedImprovement ( qLogNEI ), a variant of the LogEI family of acquisition functions, 32 replications, and initialize all methods with the same quasi-random Sobol batch for each replication. We follow Hvarfner et al. [31] and plot the true value of the best in-sample point according to the GP model posterior at each iteration. We also include Sobol and an 'Oracle', which is a Standard GP that always observes the uncorrupted value, and consider the backward canonical version of relevance pursuit, denoted by RRP, for these experiments. The plots show the mean performance with a bootstrapped 90% confidence interval.

Synthetic problems We consider the popular 6-dimensional Hartmann test function with three different corruption settings: (1) a constant value of 100 , (2) a U [ -3 , 3] distributed value, (3) the objective value for a randomly chosen point in the domain. The results for a 10% corruption probability are shown in Fig. 6. We also include results for a 20% corruption probability in Appendix D.3.

5

10

15

20

25

30

35

40

Number of Evaluations

3.0

2.5

2.0

1.5

1.0

0.5

Value of Best Inferred Point

Constant outliers value 100 (10%)

5

10

15

20

25

30

35

40

Number of Evaluations

Uniform[-3, 3] outliers (10%)

5

10

15

20

25

30

35

40

Number of Evaluations

Uniform input outliers (10%)

Method

Sobol

Standard GP

Student-t

RRP

Oracle

Real-world problems We include three real-world problems: A 3D SVM problem, a 5D CNN problem, and a 20D rover trajectory planning problem, see the App. D.2 for details. For SVM and CNN, we simulate random corruptions corresponding to an I/O error, which causes the corresponding MLmodel to be trained using only a small subset of the training data. For the rover planning problem we follow the setup in [47] with the main difference that we consider a 20 D trajectory, and the corruptions are generated randomly, causing the rover to break down at an arbitrary point along its trajectory. In most cases, this results in a smaller reward than the reward of the full trajectory.

5

10

15

20

25

30

Number of Evaluations

0.13

0.14

0.15

0.16

0.17

0.18

Value of Best Inferred Point

3D SVM, I/O error with prob. 10%

5

10

15

20

25

30

35

40

Number of Evaluations

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

5D CNN, I/O error with prob. 10%

0

25

50

75

100

125

150

Number of Evaluations

4

2

0

2

4

20D Rover, crashes with prob. 20%

Method

Sobol

Standard GP

Student-t

RRP

Oracle

7 Conclusion and Future Work

Contributions Robust Gaussian Processes via Relevance Pursuit (RRP) provides a novel and principled way to perform robust GP regression. It permits efficient and robust inference, performs well across a variety of label corruption settings, retains good performance in the absence of corruptions, and is flexible, e.g., can be used with any mean or kernel function. Our method can be readily applied to both robust regression problems as well as applications such as Bayesian optimization and is available through BoTorch [13]. Importantly, it also provides theoretical approximation guarantees.

Limitations As our approach does not explicitly consider the locations of the data points in the outlier identification, it may be outperformed by other methods if the underlying noise is heteroskedastic and location-dependent. On the other hand, those methods generally do not perform well in the presence of sparse, location-independent data corruptions.

Extensions Promising extensions of this work include performing Bayesian model averaging, i.e., average the predictions of the different possible sparsity models according to their likelihoods instead of using a MAP estimate, applying RRP to specialized models such as Lin et al. [41]'s scalable learning-curve model for AutoML applications, and Ament et al. [8]'s model for sustainable concrete. On a higher level, the approach of combining greedy optimization algorithms with Bayesian model selection and leveraging a convex parameterization to achieve approximation guarantees might apply to other parameters that are optimized using the MLL objective: length-scales of stationary kernels, coefficients of additive kernels, inducing inputs, and even related model classes like Tipping [60]'s Sparse Bayesian Learning (SBL), which seeks to identify sparse linear models and is intimately linked to greedy matching pursuits [9]. Overall, the approach has the potential to lead to theoretical guarantees, new insights, and performance improvements to widely-adopted Bayesian models.

References

P. Algikar and L. Mili. Robust gaussian process regression with huber likelihood, 2023. URL https://arxiv.org/abs/2301.07858 .

M. Altamirano, F.-X. Briol, and J. Knoblauch. Robust and conjugate gaussian process regression. arXiv preprint arXiv:2311.00463 , 2023.

S. Ament and C. Gomes. On the optimality of backward regression: Sparse recovery and subset selection. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5599-5603, 2021. doi: 10.1109/ICASSP39728.2021. 9415082.

S. Ament and C. Gomes. Generalized matching pursuits for the sparse optimization of separable objectives. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5553-5557, 2022. doi: 10.1109/ICASSP43922.2022. 9747510.

S. Ament and M. O'Neil. Accurate and efficient numerical calculation of stable densities via optimized quadrature and asymptotics. Statistics and Computing , 28:171-185, 2018.

S. Ament, M. Amsler, D. R. Sutherland, M.-C. Chang, D. Guevarra, A. B. Connolly, J. M. Gregoire, M. O. Thompson, C. P. Gomes, and R. B. van Dover. Autonomous materials synthesis via hierarchical active learning of nonequilibrium phase diagrams. Science Advances , 7(51): eabg4930, 2021. doi: 10.1126/sciadv.abg4930. URL https://www.science.org/doi/abs/ 10.1126/sciadv.abg4930 .

S. Ament, S. Daulton, D. Eriksson, M. Balandat, and E. Bakshy. Unexpected improvements to expected improvement for bayesian optimization. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 20577-20612. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 419f72cbd568ad62183f8132a3605a2a-Paper-Conference.pdf .

S. Ament, A. Witte, N. Garg, and J. Kusuma. Sustainable concrete via bayesian optimization, 2023. URL https://arxiv.org/abs/2310.18288 .

S. E. Ament and C. P. Gomes. Sparse bayesian learning via stepwise regression. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 264-274. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ament21a.html .

D. Andrade and A. Takeda. Robust Gaussian process regression with the trimmed marginal likelihood. In R. J. Evans and I. Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , volume 216 of Proceedings of Machine Learning Research , pages 67-76. PMLR, 31 Jul-04 Aug 2023. URL https://proceedings.mlr. press/v216/andrade23a.html .

P. Awasthi, A. Das, W. Kong, and R. Sen. Trimmed maximum likelihood estimation for robust generalized linear model. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum? id=VHmdFPy4U_u .

E. Bakshy, L. Dworkin, B. Karrer, K. Kashin, B. Letham, A. Murthy, and S. Singh. Ae: A domain-agnostic platform for adaptive experimentation. In NeurIPS Workshop on Systems for Machine Learning , 2018.

M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33 , 2020.

E. Bodin, M. Kaiser, I. Kazlauskaite, Z. Dai, N. Campbell, and C. H. Ek. Modulating surrogates for Bayesian optimization. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 970-979. PMLR, 13-18 Jul 2020. URL https: //proceedings.mlr.press/v119/bodin20a.html .

G. E. Box and D. R. Cox. An analysis of transformations. Journal of the Royal Statistical Society Series B: Statistical Methodology , 26(2):211-243, 1964.

E. J. Candes. The restricted isometry property and its implications for compressed sensing. Comptes rendus. Mathematique , 346(9-10):589-592, 2008.

X. Chu, I. F. Ilyas, S. Krishnan, and J. Wang. Data cleaning: Overview and emerging challenges. In Proceedings of the 2016 international conference on management of data , pages 2201-2206, 2016.

P. Congdon. Representing spatial dependence and spatial discontinuity in ecological epidemiology: a scale mixture approach. Stochastic Environmental Research and Risk Assessment , 31(2): 291-304, 2017.

A. Daemi, H. Kodamana, and B. Huang. Gaussian process modelling with gaussian mixture likelihood. Journal of Process Control , 81:209-220, 2019.

A. Das and D. Kempe. Approximate submodularity and its applications: Subset selection, sparse approximation and dictionary selection. Journal of Machine Learning Research , 19(3): 1-34, 2018.

L. C. W. Dixon. The global optimization problem: an introduction. Towards Global Optimiation 2 , pages 1-15, 1978.

K. Dong, D. Eriksson, H. Nickisch, D. Bindel, and A. G. Wilson. Scalable log determinants for gaussian process kernel learning. Advances in Neural Information Processing Systems , 30, 2017.

E. R. Elenberg, R. Khanna, A. G. Dimakis, and S. Negahban. Restricted strong convexity implies weak submodularity. The Annals of Statistics , 46(6B):3539-3568, 2018.

P. I. Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

J. R. Gardner, G. Pleiss, D. Bindel, K. Q. Weinberger, and A. G. Wilson. GPyTorch: Blackbox Matrix-matrix Gaussian Process Inference with GPU Acceleration. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems , NIPS'18, pages 7587-7597, USA, 2018. Curran Associates Inc.

R. Garnett. Bayesian Optimization . Cambridge University Press, 2023. to appear.

J. Gerritsma, R. Onnink, and A. Versluis. Yacht hydrodynamics. UCI Machine Learning Repository, 1981. DOI: https://doi.org/10.24432/C5XG7R.

P. W. Goldberg, C. K. I. Williams, and C. Bishop. Regression with input-dependent noise: A gaussian process treatment. In Advances in Neural Information Processing Systems , volume 10, January 1998.

J. Harrison, David and D. L. Rubinfeld. Hedonic prices and the demand for clean air. Journal of Environmental Economics and Management , 5:81-102, 1978.

R. A. Horn and C. R. Johnson. Matrix analysis . Cambridge university press, 2012.

C. Hvarfner, E. Hellsten, F. Hutter, and L. Nardi. Self-correcting bayesian optimization through bayesian active learning. Advances in Neural Information Processing Systems , 36, 2024.

P. Jylänki, J. Vanhatalo, and A. Vehtari. Robust gaussian process regression with a student-t likelihood. Journal of Machine Learning Research , 12(11), 2011.

O. Karaca, D. Tihanyi, and M. Kamgarpour. Performance guarantees of forward and reverse greedy algorithms for minimizing nonsupermodular nonsubmodular functions on a matroid. Operations Research Letters , 49(6):855-861, 2021.

M. Kelly, R. Longjohn, and K. Nottingham. The uci machine learning repository. URL https://archive. ics. uci. edu , 2023.

K. Kersting, C. Plagemann, P. Pfaff, and W. Burgard. Most likely heteroscedastic gaussian process regression. In Proceedings of the 24th International Conference on Machine Learning , ICML '07, pages 393-400, 2007.

A. Krause and D. Golovin. Submodular function maximization. In L. Bordeaux, Y. Hamadi, and P. Kohli, editors, Tractability , pages 71-104. Cambridge University Press, 2014. ISBN 9781139177801. URL http://dblp.uni-trier.de/db/books/collections/BHK2014. html#0001G14 .

A. Krause and C. Guestrin. Near-optimal nonmyopic value of information in graphical models. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence , UAI'05, page 324-331, Arlington, Virginia, USA, 2005. AUAI Press. ISBN 0974903914.

A. Krause, A. Singh, and C. Guestrin. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. Journal of Machine Learning Research , 9(2), 2008.

M. Kuss. Gaussian process models for robust regressiÏon, classification, and reinforcement learning . PhD thesis, Technische Universität Darmstadt, 2006.

Z.-Z. Li, L. Li, and Z. Shao. Robust gaussian process regression based on iterative trimming. Astronomy and Computing , 36:100483, 2021. ISSN 2213-1337. doi: https://doi.org/10.1016/ j.ascom.2021.100483. URL https://www.sciencedirect.com/science/article/pii/ S2213133721000378 .

J. A. Lin, S. Ament, M. Balandat, and E. Bakshy. Scaling gaussian processes for learning curve prediction via latent kronecker structure, 2024. URL https://arxiv.org/abs/2410.09239 .

D. C. Liu and J. Nocedal. On the limited memory bfgs method for large scale optimization. Math. Program. , 45(1-3):503-528, Aug. 1989. ISSN 0025-5610.

F. Locatello, R. Khanna, M. Tschannen, and M. Jaggi. A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe. In A. Singh and J. Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 860-868. PMLR, 20-22 Apr 2017. URL https://proceedings.mlr.press/v54/locatello17a.html .

F. Locatello, A. Raj, S. P. Karimireddy, G. Raetsch, B. Schölkopf, S. Stich, and M. Jaggi. On matching pursuit and coordinate descent. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3198-3207. PMLR, 10-15 Jul 2018. URL https://proceedings. mlr.press/v80/locatello18a.html .

S. Lotfi, P. Izmailov, G. Benton, M. Goldblum, and A. G. Wilson. Bayesian model selection, the marginal likelihood, and generalization, 2023. URL https://arxiv.org/abs/2202.11678 .

R. Martinez-Cantin, M. McCourt, and K. Tee. Robust bayesian optimization with student-t likelihood, 2017.

N. Maus, K. Wu, D. Eriksson, and J. Gardner. Discovering many diverse solutions with Bayesian optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 206, pages 1779-1798. PMLR, Apr. 2023.

A. Naish-Guzman and S. Holden. Robust regression with twinned gaussian processes. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/ paper_files/paper/2007/file/2ab56412b1163ee131e1246da0955bd1-Paper.pdf .

R. M. Neal. Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification. Technical Report 9702, Department of Statistics, University of Toronto, 1997.

G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming , 14:265-294, 1978.

R. K. Pace and R. Barry. Sparse spatial autoregressions. Statistics & Probability Letters , 33(3): 291-297, 1997.

C. Park, D. J. Borth, N. S. Wilson, C. N. Hunter, and F. J. Friedersdorf. Robust gaussian process regression with a bias model. Pattern Recognition , 124:108444, 2022.

R. Ranjan, B. Huang, and A. Fatehi. Robust gaussian process modeling using em algorithm. Journal of Process Control , 42:125-136, 2016.

C. E. Rasmussen, C. K. Williams, et al. Gaussian processes for machine learning , volume 1. Springer, 2006.

C. Riis, F. Antunes, F. Hüttel, C. Lima Azevedo, and F. Pereira. Bayesian active learning with fully bayesian gaussian processes. Advances in Neural Information Processing Systems , 35: 12141-12153, 2022.

A. Shah, A. Wilson, and Z. Ghahramani. Student-t processes as alternatives to gaussian processes. In Artificial intelligence and statistics , pages 877-885. PMLR, 2014.

N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning , ICML'10, page 1015-1022, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.

S. Sundararajan and S. Keerthi. Predictive approaches for choosing hyperparameters in gaussian processes. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems , volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/ paper_files/paper/1999/file/e8fd4a8a5bab2b3785d794ab51fef55c-Paper.pdf .

R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology , 58(1):267-288, 1996.

M. E. Tipping. Sparse bayesian learning and the relevance vector machine. Journal of machine learning research , 1(Jun):211-244, 2001.

M. E. Tipping and N. D. Lawrence. Variational inference for student-t models: Robust bayesian interpolation and generalised component analysis. Neurocomputing , 69(1):123-141, 2005.

J. A. Tropp. Greed is good: Algorithmic results for sparse approximation. IEEE Transactions on Information theory , 50(10):2231-2242, 2004.

A. Tsanas and A. Xifara. Energy Efficiency. UCI Machine Learning Repository, 2012. DOI: https://doi.org/10.24432/C51307.

J. Vanhatalo, P. Jylänki, and A. Vehtari. Gaussian process regression with student-t likelihood. In Advances in Neural Information Processing Systems , volume 22, 2009.

Z. Wang, C. Gehring, P. Kohli, and S. Jegelka. Batched large-scale bayesian optimization in high-dimensional spaces. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 84 of PMLR , pages 745-754. JMLR, Mar. 2018.

L. Wasserman. Bayesian model selection and model averaging. Journal of mathematical psychology , 44(1):92-107, 2000.

D. Wipf and S. Nagarajan. A new view of automatic relevance determination. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/ paper_files/paper/2007/file/9c01802ddb981e6bcfbec0f0516b8e35-Paper.pdf .

I.-C. Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 1998. DOI: https://doi.org/10.24432/C5PK67.

A Additional Details on the Model

Algorithm 2 below is the 'backward' variant of Algorithm 1 from Sec 4. As its name suggests, the main difference compared to the 'forward' variant is that rather than building up a set of 'outliers', it starts from a (typically large) set of 'outliers' and iteratively removes those data points from the set that have the smallest inferred data-point-dependent noise variance ρ i .

While we have not derived theoretical guarantees for this 'backward' version, we have found it to generally behave similarly to the 'forward' version in terms of performance and robustness. One empirical observation from our studies is that while the 'forward' version tends to perform slightly better than the 'backward' version if there are only few outliers, the opposite is true if the outlier frequency is very high. This behavior is rather intuitive and illustrates that relevance pursuit is particularly well-suited to identify sparse, low-cardinality subsets (note that in the 'backward' variant under large corruptions, the uncorrupted data points can be viewed as the sparse subset that needs to be identified).

B Additional Background on the Theory

B.1 Submodular Functions

Krause and Golovin [36] provides a survey on the maximization of general submodular functions. Here, we focus on applications of submodularity to sparse regression and Gaussian process models.

Sparse Regression Das and Kempe [20] showed that the subset selection problem of regression features with an R 2 objective satisfies a weak submodularity property, which can be invoked to prove approximation guarantees for the greedy maximization of the objective. Elenberg et al. [23] generalized this work by proving that any log likelihood function exhibiting restricted strong concavity gives rise to the weak submodularity of the associated subset selection problem, which can be invoked to prove approximation guarantees for the greedy algorithm. Karaca et al. [33] contains a guarantee for the backward algorithm applied to the maximization of submodular set functions.

Gaussian Processes Submodularity has also found application to Gaussian process models. For a sensor placement problem, Krause et al. [38] proved that the mutual information (MI) criterion, capturing the reduction in uncertainty in the entire search space, can be a submodular function. In this case, MI is not monotonic everywhere, but monotonic for small sets ( 2 k ) of sensors, which is sufficient to apply Nemhauser's guarantee for sparse sets of sensors up to size k [50]. Relatedly, the 'myopic' joint entropy of a set of observables is unconditionally submodular as a consequence of the 'information never hurts' principle [37], but generally leads to lower-quality sensor placements than the MI criterion. Srinivas et al. [57] used the submodularity of the joint entropy in order to prove regret bounds for the convergence of a GP-based BO algorithm using the upper-confidence bound acquisition function.

Elenberg et al. [23] proved that any log likelihood function exhibiting restricted strong concavity and smoothness implies the weak submodularity of the associated subset selection problem.

Definition 9 (Submodularity Ratios [23]) . Let A , B ⊂ [ n ] be two disjoint sets, and f : 2 [ n ] → R . The submodularity ratio of B with respect to A is defined by

The submodularity ratio of a set C with respect to an integer k is defined by

Then given γ > 0 , a function is γ -weakly submodular at a set C with respect to k if γ C ,k ≥ γ .

Theorem 10 (Weak Submodularity via RSC [23]) . The submodularity ratio γ S ,k can be bound below using the restricted convexity and smoothness parameters m |S| + k and M |S| + k ,

Theorem 11 (OMP Approximation Guarantee [23]) . Let x OMP ( r ) be the r -sparse vector selected by OMP, and x OPT ( r ) = arg min ∥ x ∥ 0 = r f ( x ) be the optimal r -sparse vector. Then

where m 2 r , M 2 r are the restricted strong convexity and smoothness parameters of f , respectively.

C Theoretical Results and Proofs

Lemma 1. [Optimal Robust Variances] Let D \ i = { ( x j , y j ) : j = i } , ρ = ρ \ i + ρ i e i , where ρ , ρ \ i ∈ R n + , [ ρ \ i ] i = 0 , and e i is the i th canonical basis vector. Then keeping ρ \ i fixed,

̸

where y ( x i ) = f ( x i )+ ϵ i . These quantities can be expressed as functions of Σ -1 = ( K + D σ 2 + ρ ) -1 :

where D σ 2 + ρ is a diagonal matrix whose entries are σ 2 + ρ .

Proof. First, we partition the covariance matrix K + σ 2 I + D ρ to separate the effect of ρ i and use the block matrix inverse

where

Quadratic Term With the expression for the inverse of Σ above, we can write the quadratic term of the log likelihood as

Determinant Term The determinant of a block matrix is given by

Applying this identity to Σ = ( K + D ρ + σ 2 ) , we get

Log Marginal Likelihood

Noting that ∂ ρ i β i = -β 2 i , the derivative of the difference in log marginal likelihood w.r.t. ρ i , is

While β i = 0 is a root of the derivative, we ignore this solution since β i is never zero when σ 2 > 0 . Therefore, the remaining stationary point is β -1 i = ( y ⊤ \ i u -y i ) 2 . Since we constrain ρ ≥ 0 , this point might not always be attainable. However, because there is only a single stationary point with respect to β i when σ 2 > 0 , and β i is a strictly decreasing function of ρ i , it follows that the marginal likelihood is monotonic as a function of ρ i to both the left and the right of the stationary point. Therefore, the optimal constraint ρ i is simply the optimal unconstrained value, projected into the feasible space. In particular, solving β -1 i = ( y ⊤ \ i u -y i ) 2 for ρ i and projecting to the non-negative half-line, we get

Lastly, note

As stated by Rasmussen et al. [54] (P. 117, Eq. 5.12), originally shown by Sundararajan and Keerthi [58], these quantities can be expressed as simple functions of Σ -1 :

Therefore, all LOO predictive values can be computed in O ( n 3 ) or faster, if an inducing point method is used for K .

The following is a preliminary result for our analysis of the log marginal likelihood w.r.t. ρ . Lemma 12. The gradient and Hessian of the log marginal likelihood L with respect to ρ are given by

Proof. Let α = K -1 y . Regarding the gradient, note that ∂ ρ i K = e i e ⊤ i , where e i is the canonical basis vector with a one as the i th element, and based on Equation 5.9 of Rasmussen et al. [54],

Regarding the second derivatives, according to Dong et al. [22],

Therefore,

since [ ∂ ρ i ∂ ρ j K ] = 0 . The third equality is due to the invariance of the trace to circular shifts of its argument. The forth equality is due to the symmetry of the matrix in brackets.

C.1 Strong Convexity and Smoothness of the Reparameterized Robust Marginal Likelihood

Here, we re-parameterize ρ ( s ) = diag( K 0 ) ⊙ ((1 -s ) -1 -1) , and attain strong convexity for all inputs s , if conditions on the eigenvalues of the covariance matrix and the norm of the data vector ∥ y ∥ are met. The convexity result is surprising in two ways: the negative log marginal likelihood of GPs is generally a non-convex function, and in addition, the negative log likelihoods of many alternative robust regression methods like the Studentt likelihood or α -stable likelihoods are non-convex, and even Huber's proposal is nonstrongly -convex.

Lemma 4. [Reparameterized Hessian] Let K s = k ( X , X ) + σ 2 I + D ρ ( s ) , ˆ K s = diag( K s ) -1 / 2 K s diag( K s ) -1 / 2 , and ˆ α = ˆ K -1 s diag( K s ) -1 / 2 y . Then

Proof. Using the chain-rule, the Hessian H s [ L ] can be expressed as a function of the Jacobian J s [ ρ ( s )] = D ∂ ρ ( s ) , which is diagonal since ρ ( s ) is an element-wise function, and the second derivatives ∂ 2 s ρ ( s i ) . Then

where we substituted the relevant expressions from Lemma 12. Further substituting ρ ( s ) i = [ K 0 ] ii [(1 -s i ) -1 -1] , ρ ′ ( s ) i = [ K 0 ] ii (1 -s i ) -2 , and ρ ′′ ( s ) i = 2[ K 0 ] ii (1 -s i ) -3 , noting that K = K 0 +diag( K 0 )[(1 -s ) -1 -1] = ( K 0 -diag( K 0 )) + diag( K 0 )(1 -s ) -1 , and algebraic manipulation finish the proof.

Lemma 5. [Strong Convexity via Eigenvalue Condition] Let ˆ K s as in Lemma 4. Then H s ≻ m if

where λ min , max (resp. ˆ λ min , max ) are the smallest and largest eigenvalues of K s , respectively ˆ K s .

Proof. We seek to lower-bound the smallest eigenvalue of the Hessian matrix, which-for twicecontinuously-differentiable problems-is equivalent to lower and upper bounds of the problem's Hessian matrix. Starting with the result of Lemma 4,

Now, we bound each of the three additive terms independently from below.

Term 1:

The first inequality comes from K being positive definite, and the absolute value of the diagonal of a matrix, which is already positive for positive definite matrices, being lower bounded by the minimum eigenvalue of the matrix. The last steps is a basic consequence of the eigenvalues of inverses matrices. Note that the eigenvalues of ˆ K can be further bound by the eigenvalues of the original matrix K :

In a similar way, we can show that λ max ( ˆ K ) ≤ λ max ( K ) /λ min ( K ) , which implies λ max ( ˆ K ) -1 ≥ λ min ( K ) /λ max ( K ) .

Term 2: Next, we have

which is due to the Hadamard product being a sub-matrix of the Kronecker product of the same matrices, the largest eigenvalue of the former are bounded by the largest eigenvalue of the latter, which is the product of the largest eigenvalues of the constituent matrices.

Term 3: Lastly, note that

where the last inequality comes from the second to last lower bound of Term 3 being non-positive, and therefore, being able to lower bound it with the largest magnitude entry of ˆ α .

Term 1 + 2 + 3: Putting together the inequalities for all terms, we get

where in slight abuse of notation, we let H s = H s [ -2 L ] be the Hessian of the negative log likelihood.

Using the bound ∥ ˆ α ∥ ∞ ≤ ∥ ˆ α ∥ 2 ≤ λ min ( ˆ K ) -1 ∥ ˆ y ∥ , where ˆ y = diag( K ) -1 / 2 y . Therefore, ∥ ˆ α ∥ ∞ ≤ λ min ( ˆ K ) -1 λ min ( K ) -1 / 2 ∥ y ∥ 2 ≤ λ max ( K ) λ min ( K ) -3 / 2 ∥ y ∥ 2 .

Finally, lower bounding the current lower bound by m > 0 yields a sufficient condition for the convexity at s .

Re-arranging, we attain

which is a non-trivial guarantee when 2 ˆ λ -1 max -ˆ λ -2 min -m> 0 .

Lemma 13. [Smoothness via Eigenvalue Condition] Let ˆ K as in Lemma 4. Suppose that ∥ s ∥ ∞ ≤ s max and

where λ min , max (resp. ˆ λ min , max ) are the smallest and largest eigenvalues of K (resp. ˆ K ) respectively. Then H s ≺ M . This is a non-trivial guarantee when M (1 -s max ) 2 + ˆ λ -2 min -2 ˆ λ -1 max > 0 .

Proof. We now derive an equivalent upper bound for the largest eigenvalue of the Hessian. Starting with the result of Lemma 4,

Therefore, it becomes immediately apparent that we will need to introduce an upper bound on ∥ (1 -s ) -1 ∥ 2 ∞ , which is a restriction on the domain that ρ can take. Proceeding in a similar way as above, we bound the three terms in square brackets, now from above.

Term 1:

Term 2:

Term 3:

where now the last inequality follows because the second to last expression is always non-negative. Term 1 + 2 + 3: Putting together the inequalities for all terms, we get

Finally, upper bounding the current upper bound by M/ ∥ (1 -s ) -1 ∥ 2 ∞ > 0 yields a sufficient condition for the M -smoothness at s . Using the same bound for ∥ ˆ α ∥ ∞ derived for the convexity result,

Re-arranging, we attain

which is a non-trivial guarantee when M (1 -s max ) 2 + ˆ λ -2 min -2 ˆ λ -1 max > 0 .

Lemma 7. [Strong Convexity via Diagonal Dominance] Let m > 0 and K 0 be δ -diagonally dominant with δ < ( (5 -m ) - √ 25 -9 m +17 ) / 4 ≤ (5 -√ 17) / 4 ≈ 0 . 44 and

Then the NMLL is m -strongly convex for all s ∈ [0 , 1] n , i.e. ρ ( s ) ∈ [0 , ∞ ] n .

Proof. Fist, Gershgorin's Disk Theorem implies that the eigenvalues of diag( A ) -1 / 2 A diag( A ) -1 / 2 lie in (1 -δ, 1 + δ ) for a δ -diagonally dominant matrix A . Further, the condition number of A is bounded above by κ ( A ) = λ max ( A ) /λ min ( A ) ≤ (1 + δ ) / (1 -δ ) . See Horn and Johnson [30] for more background on matrix analysis. Plugging these bounds into the results of Lemma 5 yields

Lower bounding the last expression by ∥ y ∥ 2 2 implies m -strong convexity. This gives rise to a non-trivial guarantee whenever the numerator is larger than zero. In particular,

Expanding, noting that 0 < δ < 1 implies δ 3 < δ 2 in order to reduce a power in the resulting expression, and collecting terms with like powers, we attain the following sufficient condition

Note that for this condition to hold if δ = 0 , we need to have m< 1 . Fortunately, this is a quadratic in δ whose smallest positive root is

In particular, for m = 0 , this reduces to

Lastly, note that if K 0 is δ -diagonally dominant, then so is K s = K 0 + D ρ ( s ) , since the robust variances add to the diagonal, making it more dominant. Therefore, the convexity guarantee holds for all ρ ( s ) , if it holds for the base covariance matrix K 0 . Note that λ min ( K ) ≥ σ 2 .

Similarly, we can prove a similar statement relating diagonal dominance with M -smoothness.

Lemma 14. [Smoothness via Diagonal Dominance] Suppose K 0 is a δ -diagonally dominant covariance matrix and suppose we constrain ∥ s ∥ ∞ ≤ s max ≤ 1 -√ 1 /M . Then

implies that the NLML is m -strongly convex for all s ∈ [0 , 1] n , i.e. ρ ( s ) ∈ [0 , ∞ ] n .

Proof. We proceed in a similar way as for Lemma 7, but with Lemma 13 as the starting point.

where we used ( ˆ λ -2 min -2 ˆ λ -1 max ) ≥ -1 , which is tight when ˆ K is unitary. Lower bounding the last expression by ∥ y ∥ 2 2 implies M -smoothness. This gives rise to a non-trivial guarantee whenever the numerator is larger than zero. In particular, M (1 -s max ) 2 -1 > 0 , which implies s max ≤ 1 -√ 1 /M or equivalently, M > 1 / (1 -s max ) 2 . Lastly, note that if K 0 is δ -diagonally dominant, then so is K s = K 0 + D ρ ( s ) , since the robust variances only add to the diagonal. Therefore, if the inequality holds for the base covariance matrix K 0 , the smoothness guarantee holds for all ρ ( s ) such that s ≤ s max ≤ 1 -√ 1 /M . Note also that λ min ( K ) ≥ σ 2 .

Theorem 8. [Approximation Guarantee] Let K 0 = k ( X , X ) + σ 2 I be δ -diagonally dominant, s max > 0 be an upper bound on ∥ s ∥ ∞ , and suppose ∥ y ∥ , δ satisfy the bounds of Lemmas 7 and 14, guaranteeing m -convexity and M -smoothness of the NMLL for some m> 0 , M > 1 / (1 -s max ) 2 . Let s OMP ( r ) be the r -sparse vector attained by OMP on the NMLL objective for r steps, and let s OPT ( r ) = arg max ∥ s ∥ 0 = r, ∥ s ∥ ∞ ≤ s max L ( ρ ( s )) be the optimal r -sparse vector. Then for any 2 r ≤ n ,

where ˜ L ( · ) = L ( · ) -L ( 0 ) is normalized so that max s S ˜ L ( s S ) ≥ 0 for any support S .

Proof. The result is a direct consequence of meeting the m -convexity and M -smoothness conditions of Lemmas 7, 14 above, and the OMP approximation guarantee of Theorem 11 due to Elenberg et al. [23]. Note that the condition on 2 r ≤ n comes from the RSC condition in Theorem 11 being required for subsets of size 2 r . As we proved bounds for the m -convexity of the full Hessian of size n , r has to be smaller than n/ 2 for the assumptions of the theorem to hold. Regarding the upper bound s max on s , we note that the constraint is convex and therefore doesn't change the convexity property of the optimization problem.

Further, note that max ρ S L ( ρ S ) ≤ max ρ S∪ i L ( ρ S∪ i ) , since the additional non-zero element could stay at 0 , if the marginal likelihood does not improve with ρ i increasing. That is, the subset selection problem is monotonic. As a consequence, we can normalize the MLL by ˜ L ( · ) = L ( · ) -L ( 0 ) , which then only takes positive values for any s ∗ S = arg max s \S =0 L ( ρ ( s S )) , i.e. max s S ˜ L ( s S ) ≥ 0 . This normalization is required for the constant factor approximation guarantee to apply, similar to the original work of Nemhauser.

This theoretical approach could lead to approximation guarantees for Tipping [60]'s Sparse Bayesian Learning (SBL) model, for which Ament and Gomes [9] show that greedily optimizing the associated NMLL is equivalent to stepwise regression in the limit of σ → 0 , proving exact recovery guarantees.

D Additional Detail on the Experiments

Our benchmark uses a modified version of the code from Andrade and Takeda [10], available at https://github.com/andrade-stats/TrimmedMarginalLikelihoodGP under the GNU GPLv2 license.

D.1 Synthetic Regression Problems

Model fitting runtimes Fig. 8 summarizes the fitting times for the different models on the different scenarios from Section 6.1. We observe that the outlier type and the fraction of outliers both have relatively limited effect on the fitting times for all of the models. Fig. 9 therefore provides a more compact view of the same data (aggregated across outlier types and outlier fractions), giving a better sense of the distribution of the fitting times. Unsurprisingly, the baselines that simply fit a single GP model ('vanilla', 'winsorize', 'power_transform') are substantially faster than any of the robust approaches. While all of the robust models show similar fitting times on the Hartmann problem, fitting our RRP is significantly faster (note the logarithmic scale of the y-axis) than fitting the Studentt and trimmed MLE models on the 5-dim and 10-dim Friedman problems. The trimmed MLE model in particular ends up being quite slow, especially on the 5d Friedman function.

D.2 BO experiments, additional details

For Hartmann6, we consider the standard domain of [0 , 1] 6 .

SVM The SVM problem, the goal is to optimize the test RMSE of an SVM regression model trained on 100 features from the CT slice UCI dataset. We tune the following three parameters: C ∈ [1e-2, 1e2], ϵ ∈ [1e-2, 1], and γ ∈ [1e-3, 1e-1]. All parameters are tuned in log-scale. Corruptions simulate I/O failures in which case we only train on U [100 , 1000] training points out of the available 50 , 000 training observations.

CNN For the 5D CNN problem the goal is to optimize the test accuracy of a CNN classifier trained on the MNIST dataset. We tune the following 5 parameters: learning rate in the interval [1e-4, 1e-1], momentum in the interval [0, 1], weight decay in the interval [0, 1], step size in the interval [1, 100], and γ ∈ [0, 1]. Similarly to the SVM problem, we only train on U [100 , 1000] of the available training batches when an I/O failure occurs.

Rover The rover trajectory planning problem was originally proposed in Wang et al. [65]. The goal is to tune the trajectory of a rover in order to maximize the reward of its final trajectory. We use the same obstacle locations and trajectory parameterization as in Maus et al. [47], with the main difference being that we parameterize the trajectory using 10 points, resulting in 20 tunable parameters. When a corruption occurs, the rover will stop at a uniformly random point along its trajectory, generally resulting in lower reward than the original trajectory.

D.3 Additional Synthetic BO Results

In addition to the results in Fig. 6, we also include results when the corruption probability is 20% in Fig. 10. We observe that the results are similar as with 10% corruption probability, but that the performance of several baselines regresses significantly.

D.4 Computational Setup and Requirements

Robust GP regression is very data-efficient, focuses on the small-data regime, and runs fast (faster than competing baselines studied in this paper). Therefore, each individual run required very limited compute resources (this includes the baseline methods). To produce statistically meaningful results, however, we ran a large number of replications for both our regression and Bayesian optimization benchmarks on a proprietary cluster. We estimate the amount of compute spent on these experiments to be around 2 CPU years in total, using standard (Intel Xeon) CPU hardware. The amount of compute spent on exploratory investigations as part of this work was negligible (this was ad-hoc exploratory and development work on a single CPU machine).

10 0

10 1

10 2

10 3

fit time (s)

friedman10, constant

friedman10, student-t

friedman10, uniform

10 0

10 1

10 2

10 3

fit time (s)

friedman5, constant

friedman5, student-t

friedman5, uniform

0.05

0.10

0.15

0.20

0.25

10 0

10 1

10 2

10 3

fit time (s)

hartmann6, constant

0.05

0.10

0.15

0.20

0.25

hartmann6, student-t

0.05

0.10

0.15

0.20

0.25

hartmann6, uniform

RRP

RRP (bkwd)

RRP (can)

RRP (bkwd, can)

Student-t

Power Transf.

Standard GP

Adapt. Wins.

D.5 Impact of Convex Parameterization on Joint Optimization of Hyper-Parameters

A limitation of Lemma 4 is that it only guarantees convexity for the sub-problem of optimizing the ρ i 's. In practice, we jointly optimize all GP hyper-parameters, including length scales. In this case, the theory guarantees the positive-definiteness of the submatrix of the Hessian corresponding to ρ ( s ) , and we expect this to improve the quality of the results of the numerical optimization routines.

Indeed, positive-definiteness is beneficial for quasi-Newton optimization algorithms like L-BFGS [42], which restarts its approximation to the Hessian whenever it encounters non-convex regions, because the associated updates to the Hessian approximation are not positive-definite. This leads the algorithm to momentarily revert back to gradient descent, with an associated slower convergence rate.

To quantify the impact of this, we ran convergence analyses using the data from Fig. 1, allowing all ρ to be optimized jointly with other hyper-parameters (length scale, kernel variance and noise

10 0

10 1

10 2

10 3

fit time (s)

friedman5

10 0

10 1

10 2

10 3

fit time (s)

friedman10

10 0

10 1

10 2

10 3

fit time (s)

hartmann6

RRP

RRP (bkwd)

RRP (can)

RRP (bkwd, can)

Student-t

Power Transf.

Standard GP

Adapt. Wins.

5

10

15

20

25

30

35

40

Number of Evaluations

3.0

2.5

2.0

1.5

1.0

0.5

Value of Best Inferred Point

Constant outliers value 100 (20%)

5

10

15

20

25

30

35

40

Number of Evaluations

Uniform[-3, 3] outliers (20%)

5

10

15

20

25

30

35

40

Number of Evaluations

Uniform input outliers (20%)

Method

Sobol

Standard GP

Student-t

RRP

Oracle

variance), recording the achieved negative log marginal likelihood (NLML) as a function of the tolerance parameter ftol of the L-BFGS optimizer. The results are reported in Table D.5, and indicate that the optimizer terminates with a much better NLML using the convex parameterization with the same convergence tolerance.

There are also settings in which we do not actually jointly optimize the hyper-parameters, particularly when we have access to data from the same data generating process that has been manually labeled

by domain experts as outlier-free. Then we can estimate the model hyper-parameters on that data, and fix them for the RRP on new data sets that we do not know to be outlier-free.

D.6 Comparison to Robust and Conjugate Gaussian Processes (RCGP)

We report extended empirical comparisons with Altamirano et al. [2]'s RCGP method, using their experimental setup and method implementation in GPFLow. Including GPFlow in our own benchmarking setup and compute resources proved difficult. To circumvent this, we wrote wrappers for both BoTorch's standard GP and RRP, which also accounts for any orthogonal implementation differences between the two frameworks, and ran the benchmarks locally on an M-series MacBook.

See Tables 2 and 3 for the mean absolute error and negative log predictive density, respectively. The tables include the empirical mean and standard deviation over 20 replications on corrupted version of the following base data sets: 1) Synthetic, which is generated as a draw of a GP with a exponentiated quadratic kernel, and four data sets available on the UCI machine learning repository[34], in particular, 2) Boston [29], 3) Concrete [68], 4) Energy [63], and 5) Yacht [27]. The benchmark considers no corruptions ('No Outliers'), 'Asymmetric Outliers', which are uniform in x are shifted negatively in y , 'Uniform Outliers', which shift y in both directions (positively and negatively), and 'Focused Outliers', which form concentrated clusters in both x and y . Any bold entry in the table signifies a results that is within one standard-error of the best result's one standard-error confidence bound.

D.7 Comparison to Robust Gaussian Process with Huber Likelihood

In the following, we compare our method with additional variational GP baselines with Laplace and Huber likelihoods, and translated the Matlab code of the "projection statistics" of Algikar and Mili [1] to PyTorch. We then combined the projection-statistics-based weighting of the Huber loss with

a variational (referred to as Huber-Projection) to get as close as possible to a direct comparison to Algikar and Mili [1] without access to a Matlab license.

Tables 4 and 5 shows the root mean square error and negative log predictive density on the Neal, Friedman 5 and Friedman 10 test functions, as well as the Yacht Hydrodynamics [27] and California Housing [51] datasets from the UCI database [34], where 15% of the training data sets of the models were corrupted. Tables 6 and 7 below were generated in a similar way, but 100% of the data were subject to heavier-tailed noise, either Studentt or Laplace.

In summary, the Relevance Pursuit model generally outperforms the variational GPs with heavy-tailed likelihoods when the corruptions are a sparse subset of all observations. Unsurprisingly, the GPs with heavy-tailed likelihoods perform best when all observations are subject to heavy-tailed noise. While such uniformly heavy-tailed noise does exist in practice, we stress that this is a distinct setting to the common setting where datasets contain a subset of a-priori unknown outliers, while a dominant fraction of the data can be considered inliers that, once they are identified, can be used to train a model without additional treatment.

Data

Standard

Relevance Pursuit

Student-

t

Laplace

Huber

Huber + Projection

Student-

t

1.95e+0 (1.22e-1)

8.78e+1 (2.80e+1)

1.17e+0 (6.08e-2)

1.22e+0 (5.71e-2)

1.20e+0 (5.17e-2)

1.21e+0 (5.15e-2)

Laplace

1.89e+0 (1.14e-1)

1.38e+2 (3.98e+1)

1.57e+0 (9.22e-2)

1.55e+0 (9.20e-2)

1.55e+0 (9.12e-2)

1.55e+0 (9.12e-2)

Friedman 5

Student-

t

4.43e+0 (8.19e-2)

3.93e+0 (2.19e-2)

3.11e+0 (2.69e-2)

2.96e+0 (3.49e-2)

2.97e+0 (3.27e-2)

2.97e+0 (3.27e-2)

Laplace

4.55e+0 (2.81e-2)

4.56e+0 (1.88e-2)

3.64e+0 (4.03e-2)

3.46e+0 (4.45e-2)

3.48e+0 (4.31e-2)

3.48e+0 (4.31e-2)

Friedman 10

Student-

t

4.60e+0 (8.54e-2)

3.89e+0 (1.36e-2)

3.04e+0 (2.45e-2)

2.83e+0 (2.30e-2)

2.84e+0 (2.22e-2)

2.84e+0 (2.22e-2)

Laplace

4.58e+0 (1.25e-2)

4.57e+0 (1.27e-2)

3.60e+0 (2.53e-2)

3.32e+0 (3.01e-2)

3.37e+0 (2.61e-2)

3.37e+0 (2.61e-2)

Yacht

Student-

t

5.14e+0 (1.85e-1)

4.60e+0 (8.15e-2)

7.30e+0 (1.40e-1)

6.79e+0 (1.52e-1)

6.70e+0 (1.32e-1)

1.28e+1 (7.75e-1)

Laplace

4.74e+0 (4.45e-2)

5.30e+0 (8.31e-2)

5.19e+0 (1.17e-1)

5.12e+0 (1.04e-1)

9.90e+0 (6.44e-1)

CA Housing

Student-

t

2.08e+0 (1.01e-1)

1.79e+0 (6.40e-2)

7.41e+0 (2.39e-1)

6.52e+0 (2.54e-1)

6.41e+0 (2.07e-1)

6.83e+0 (3.28e-1)

Laplace

1.88e+0 (5.25e-2)

2.24e+0 (9.41e-2)

2.92e+0 (6.01e-2)

3.33e+0 (7.47e-2)

3.27e+0 (5.84e-2)

6.64e+0 (3.28e+0)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Everything that is claimed in the abstract and introduction is discussed in detail in the rest of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations of the work are mentioned throughout the paper as well as summarized in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results clearly state the required assumptions; proofs are provided in Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Code for our method as well as for the benchmarking setup is provided in the supplementary material, and core models and algorithms will be open-sourced upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code for reproducing the results in the paper is included in the supplementary material. All data sets are publicly available and referenced in the work and the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details on the experiments are provided in the experimental section as well as the Appendix. Exact implementation details are contained in the included code submission.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Variance in the results is provided throughout in the form of confidence intervals and/or violin plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Total compute resources spent are described in Section D.4 and are estimated at around 2 CPU years in total.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research fully conforms to the NeurIPS Code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper makes foundational methodological contributions to robust probabilistic regression, and we do not see any direct societal impacts of the work that would require specific discussion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All creators or original owners of the code and data sets that were used in this paper are properly cited and credited. Licenses for these assets are mentioned in Sec. D and our use complies with those licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The only new asset introduced in this paper is the code for method and benchmarks; it does not introduce any other assets (data sets or models). The code is provided as an anonymized zip file for the reviewers. Upon publication, the code released as open source and documented according to community standards.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new way to perform heteroskedastic regression using Gaussian processes via data-point-specific noise levels. These noise levels are inferred using a sequential selection procedure maximizing the log-marginal likelihood. The authors show that under a specific parametrization the log-marginal likelihood is strongly concave in the noise variances and give approximation guarantees for their proposed algorithm. Experiments on regression and Bayesian optimization tasks demonstrate the benefits of this approach.\n\nSTRENGTHS:\nMaking Gaussian processes robust to some degree of model misspecification is a worthwhile goal and this paper provides a nice addition to the existing toolbox of methods. The paper is well-written and presents largely well-executed experiments for regression and Bayesian optimization. In particular, in tasks where Bayesian optimization is applied, it seems plausible that outliers appear in the way they are synthetically generated in the evaluation. I also appreciated that the authors submitted their code.\n\nWEAKNESSES:\n### Theory\nWhile correct as far as I can tell, I question the value of the theoretical result, if in practice you optimize the hyperparameters of the covariance function and $\\rho$ jointly. Is there reason to believe, that the convexity in $\\rho$ for fixed hyperparameters is beneficial given this choice? It would be informative to see an experiment that compares the non-convex parametrization to the convex one where all hyperparameters are optimized jointly. If the convex parametrization improves convergence, this would suggest that convexity is beneficial even when optimizing all hyperparameters jointly.\n\n### Related Work\nEven though this is quite recent work, I would have liked to see a discussion of Altamirano et al. (2024) in the related work section. They introduce data-point-specific observation noise based on a generalized Bayesian inference perspective. I think a comparison to their approach in the experiments would improve the paper, but I would not expect this given the recency of the work.\n\n- Altamirano, Matias, Francois-Xavier Briol, and Jeremias Knoblauch. \"Robust and Conjugate Gaussian Process Regression.\" International Conference on Machine Learning (ICML), 2024. URL: https://arxiv.org/abs/2311.00463\n\n### Experiments\nAs the authors laudably acknowledge, the fit times were 100x slower (see Figure 8) than the simple baseline approaches. How does that impact the Bayesian optimization results if plotted as a function of wall-clock time? It seems that eventually some of the other approaches catch up to your method. If the fit time is 100x that of the baseline approaches, the reward curves might look different if plotted as a function of wall-clock time. Or is there an implicit assumption that each evaluation is significantly more expensive than fitting the GP?\nThe experiments on regression problems could be expanded in my opinion, e.g. with benchmark datasets from the UCI repository. The test problems that were considered are arguably designed to test optimization algorithms (hence their usage in BO).\nFrom looking at the code the \"real-world problems\" in Section 6 have synthetically generated corruptions. I think the paper could be made stronger if there were at least one experiment where the corruption is not synthetically generated, but simply part of the observation process. \n\n### Other Weaknesses and Improvements\n- Almost all plots are way too small and barely readable. If you reformat these, then I am willing to raise my \"Presentation\" score.\n- Link to the proofs in the appendix after each theoretical result, otherwise they are hard to find.\n- When you reference details in the appendix, link to the appropriate section (e.g. line 286).\n- Citations in lines 32 to 34 for the methods that are referenced are missing\n- Typos (l.211, l. 286, l. 362)\n\nQUESTIONS:\n- Is your assumption about the data corruption different from Huber's $\\varepsilon$-contamination model (see e.g. https://arxiv.org/pdf/1511.04144) or is it identical? Equation 2 suggests they are different, but the text and experiments often refer to only a fraction of the data being corrupted (e.g. in Section 4.3 and Section 6).\n- How does your approach compare to a vanilla GP on a dataset with no corruption? Does it \"fail gracefully\"?\n- Do you see better performance when *jointly* optimizing the hyperparameters for the convex reparametrization of the data-point-specific noise variances than if not reparametrizing?\n- How did you set the schedule to test data points for being outliers in the experiments (c.f. lines 182-184)?\n\nLIMITATIONS:\nI would have liked to see a bit more discussion on the limitations of the approach in Section 7. For example, the fact that the theoretical result does not apply when jointly optimizing the hyperparameters as is the case in the experiments, and a discussion of the wall-clock runtime.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a robust Gaussian Process regression by inferring data-point-specific noise levels with a sequential selection procedure maximising the log marginal likelihood. The authors show the good performance of their method in a mix of synthetic and real-world regression tasks, including Bayesian optimisation.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The method is clearly motivated and presented, with a good amount of related work discussed. The authors address a problem—robust regression in the context of Gaussian processes—of sufficiently broad interest to merit publication. The authors also provide a theoretical analysis to support their method.\n\nWEAKNESSES:\nIn my opinion, the weakest part of the paper is the regression problems in the experiment section. It only benchmarks against two test functions, one of which is not a standard benchmark for regression; I would like to see more standard UCI benchmarks. Additionally, the proposed method does not clearly outperform other methods, particularly in the Hartmann6 test, where adaptive Winsorization outperforms RGP-RP. The authors do not discuss the results of the regression problems, so a discussion providing guidelines on the cases where the proposed method will perform better is needed.\n\nFinally, the authors claim their method \"permits fast and robust inference\"; however, in Appendix D.1, the method is slower than the Student-t GP for the Hartmann6. It would be useful if the authors discussed this in the main paper.\n\nQUESTIONS:\n- What's the difference between RGP-RP and RGP-RP* in the experiment section?\n- What's the schedule $\\mathcal{K}$ selected for the experiments? How does this affect the performance and fitting time of the method?\n- While the Friedman and Hartmann6 test functions are well-known, the authors should provide a reference for these functions.\n- Why use Hartmann6 as a test function for regression? Hartmann6 is typically used to test optimisation due to its properties. Why not use standard UCI benchmarks?\n- Do the authors know why the student-t GP performs so poorly in the Hartmann6 function, even when the noise comes from a student-t distribution?\n\nLIMITATIONS:\nNo obvious negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a robust Gaussian process model in the sparse heteroscedastic noise setting. Orthogonal Matching Pursuit- like algorithm is proposed to infer data point specific noise levels. The negative log marginal likelihood function to be optimized is claimed to be strongly convex by reparametrizing the additive noise levels. The method is validated on synthetic as well as real-world experiments.\n\nSTRENGTHS:\n1. The authors formulated a convex $\\mathcal{L}$ in the Gaussian process regression model by reparametrizing the noise levels, which is very desirable for optimization perspective. The representation of the paper is good, however, I found the section 4.3 difficult to follow.\n2. The relevance pursuit is a novel approach to the heteroscedastic noise case of robust GP regression.\n\nWEAKNESSES:\n1. Much more work on heavy tailed outliers is done. So, more references and discussion need to be added in related work. Kersting et. al. 2007 proposed GPs for heteroscedastic noise setting. They also learned the homoscedastic part of the variance aiming to learn the remaining part using a second GP. The work is related and at least needs a mention. It would have been nice to see comparison with this model. \n\n2. While this work employs greedy algorithms for GPs, it is not the first to do so, so I would not consider it exceptionally novel.\n\n3. The experimentation is adequate but less discussed.\n \nRef: Most Likely Heteroscedastic Gaussian Process Regression\n\n\nMinor corrections: \n\n1. Consider adding references for these three categories on line 34. No reference is given for down weighting procedures even in related work.\n2. Figure 1 and 2 are not discussed at all. What regression example? \n3. broken line 150 \n4. unclosed braces figure 3\n5. no need of repeating the abbreviation RIP on line 200\n6. Broken line 264\n7. Is the x axis of right figure in Figure 4 percentage of corruptions or probability?\n8. Different notations need to be used for $\\delta$ on lines 67 and 112.\n\nQUESTIONS:\n1. Why Matern kernel was chosen in one of the baselines?\n2. What is the difference between RGP-RP*-BW and RGP-RP-BW. I see that the former is canonical one. What is the difference between canonical and non-canonical reparameterization of $\\mathbf{s}$?\n3. What is $\\mathbf{K}_{0}$ on line 218?\n\nLIMITATIONS:\nLimitations are discussed as I hoped they would be.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}