Title: Sharp Calibrated Gaussian Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian Processes

Frequentist Statistics

Kernel Methods

Model Selection and Structure Learning

Regression
Abstract: While Gaussian processes are a mainstay for various engineering and scientific applications, the uncertainty estimates don't satisfy frequentist guarantees and can be miscalibrated in practice. State-of-the-art approaches for designing calibrated models rely on inflating the Gaussian process posterior variance, which yields confidence intervals that are potentially too coarse. To remedy this, we present a calibration approach that generates predictive quantiles using a computation inspired by the vanilla Gaussian process posterior variance but using a different set of hyperparameters chosen to satisfy an empirical calibration constraint. This results in a calibration approach that is considerably more flexible than existing approaches, which we optimize to yield tight predictive quantiles. Our approach is shown to yield a calibrated model under reasonable assumptions. Furthermore, it outperforms existing approaches in sharpness when employed for calibrated regression.

.arget

★

nnlt

nnut

IXK

X

X

XX

8

C

XX

XX

8

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by the observation that the posterior variance of a Gaussian process is often poorly calibrated, the authors propose an alternative approach of attaching predictive quantiles to the posterior mean. In essence, their approach minimises the width of the predictive quantiles under an empirical calibration constraint computed on held-out validation data. By satisfying the empirical calibration constraint, the authors prove that the predictive quantiles are indeed approximately the right quantiles (Theorem 5.3). The approach is tested and compared against other calibration approaches in a toy example and on seven data sets from the UCI data set repository. The results show that it outperforms other techniques in terms of sharpness.\n\nSTRENGTHS:\nTo begin with, I would like to thank the authors for their submission. \n\n## Strengths\n\n* The paper is easy to follow and generally well written. I found only a few typos.\n\n* The problem that the posterior variance of a GP may be poorly calibrated is highly relevant, so methods that attempt to attack this problem, like the proposed approach by the authors, are certainly important.\n\n* Accompanying the method with a theoretical result that guarantees correctness (Theorem 5.3) reassures the practitioner.\n\n* I really like Section 5.2, where the authors spend some effort on reducing the computational cost in practice. They could have just left it at (7) and state that the optimisation problem would have to be computed for every confidence level $\\delta$ of interest, but of course the approach in Section 5.2 is much more elegant.\n\n* The experiments appear to show that the proposed approach is sharper than alternatives. I, however, have some doubts about the experimental section. Please see below.\n\n## General Comments and Suggestions\n\n* Should the title of the paper be \"Sharply Calibrated GPs\" instead of \"Sharp Calibrated GPs\"? I think \"Sharp(ly)\" should be an adverb, since it modifies \"calibrated\".\n\n* On line 75, you use $\\mathcal{D}\\_{\\text{tr}}$ without introducing the symbol first. This is confusing, because you have just introduced $\\mathcal{D}$, not $\\mathcal{D}\\_{\\text{tr}}$, as the training data.\n\n* On line 75, you introduce the cut-point term as $\\beta_\\delta \\sigma_{\\mathcal{D}}(\\delta, x)$. At this point, you should really better explain this term by answering the following questions: Should $\\sigma_{\\mathcal{D}}(\\delta, x)$ be interpreted as a standard deviation? If so, how do you make sense of that it depends on $\\delta$? If not, then what is $\\sigma$? If you're learning $\\sigma$, why do you also need $\\beta_\\delta$? Can you not absorb $\\beta_\\delta$ into $\\sigma$? You should also mention that $\\beta_\\delta$ may be negative. Without that information, (1) doesn't make much sense for small $\\delta$.\n\n* On line 81: Do you mean \"are small\" or \"are as small as possible\"? This nuance is very important and changes the meaning substantially.\n\n* On line 119, you say that the log-marginal likelihood does not account for calibration. Depending on what precisely you mean by calibration, I think that this is false: the log-marginal likelihood is an empirical estimate of the KL divergence, and the KL divergence certainly accounts for the \"whole distribution\" and therefore the calibration.\n\n* In Section 5.1, is $\\sigma\\_{\\mathcal{D}\\_{\\text{tr}}}(\\theta, x)$ the posterior variance of the GP where the kernel parameters are now the parameters that we optimise over? From line 146 onwards, you don't actually explain this! Since this is a crucial part of your construction, I think the exposition would be better if you were to clearly explain this somewhere around line 158.\n\n* On line 175, you state that (5) can be replaced by (6) with further explanation. Since this is an important step in the derivation of your algorithm, I think that it deserves a careful explanation. Moreover, it is not true that (5) and (6) are equivalent, since $q\\_{\\text{lin}}$ is linearly interpolated. I think the exposition would benefit from a little more care here.\n\n* Line 268: \"calibration\" -> \"calibrated\"\n\n* Could you add to the legend of Figure 2 what the squares and crosses are? Why does it look like there are two \"lines of squares/crossed\"?\n\nWEAKNESSES:\n## Weaknesses\n\n### Assumption 4.1 Not Obviously Satisfied\n\nI agree that Assumption 4.1 is obviously satisfied for the variance of a kernel. However, for the inverse length scale, I can believe that Assumption 4.1 might be satisfied, but this is not obvious at all. Would the authors be able to produce a proof that the posterior variance of a GP is monotonic in the inverse length scale?\n\n### Theorem 5.3 Might Not Be Valid\n\nThe proof of Theorem 5.3 crucially relies on Theorem 1 by Marx et al. (2022). This Theorem 1, however, operates in the setting where all pairs $(x_i, y_i)$ are sampled i.i.d. (see Section 2 of Marx et al., 2022). But in the setting of GPs, which is the setting of the submission, the pairs aren't independent, because they are correlated by the sample from the underlying GP! This means that Theorem 1 might not apply, which means that Theorem 5.3 might not be valid. Could the authors comment on this?\n\n### Result of Section 7.1 Looks Questionable\n\nIn Figure 2, you state that your approach produces a 99% confidence interval. However, if you look at the right side of Figure 2.(a), then the confidence interval completely misses the data! It hence looks like the shaded region is not actually a 99% confidence interval, which makes me wonder whether the predictions are actually well calibrated.\n\n### Bolded Results Are Not Significantly Best Results\n\nThroughout the main paper and the supplement, you bold the score with the best average. However, if $x_1 \\pm e_1$ and and $x_2 \\pm e_2$ are such that $x_1 < x_2$, but $x_1 + e_1 \\ge x_2 - e_2$, then you cannot actually conclude with confidence that $x_1 \\pm e_1$ is really lower than $x_2 \\pm e_2$, because the difference might be explained by random fluctuations. In other words, you should really only bold results that are the best results at reasonable statistical significance. Currently, because of this issue, I think that the results throughout are misrepresented.\n\n### What Are STD and NLL in Table 2.1?\n\nThe premise of the paper is that you discard the predictive variance and instead produce calibrated predictive quantiles at one or multiple given confidence levels. This means that the predictions now consist of a mean and associated intervals. Therefore, the predictions are no longer probabilistic, so I really don't understand what the STD and NLL in Table 2 are! (For given a mean and an interval, how can you compute a probability?)\n\nQUESTIONS:\n## Conclusion\n\nAlthough I think the proposed approach is very interesting, I am mainly worried about the validity of Theorem 5.3, the soundness the result in Section 7.1, the use of boldness to present the experimental results, and the STD and NLL metrics in Table 2.1. Therefore, at this point, I cannot accept this submission and must unfortunately recommend rejection. However, if the authors are able to address the following points, then I am willing to change my reject to an accept:\n\n* Please see if Theorem 5.3 is really flawed, and, if it is, whether it can be fixed.\n\n* Please correct the use of boldness throughout the main paper and the supplement.\n\n* Please investigate whether the predictions in Figure 2.(a) really are calibrated.\n\n* Please explain what the STD and NLL metrics in Table 2.1 are.\n\nEDIT\n\nThe authors have largely addressed my criticisms, so I have increased my score to a borderline accept. I believe that the submission might deserve a higher score, but I am not comfortable recommending any higher without seeing a revised version of the PDF.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel method for calibrating Gaussian process posterior variances using held-out data. In particular, they train a new, separate, GP using the held out data for the variance, using the GP trained on the original dataset for the mean. They do this in a way which approximately maximises the sharpness (roughly speaking, minimises the variance). The calibration properties of the method are supported by theory. This is supported by an empirical evaluation of the calibration error on synthetic data and of the sharpness on real-world data.\n\nPost-discussion: The authors have provided considerable improvement in their presentation of the results, and have added some extra results which further improve the clarity.\n\nSTRENGTHS:\n1. (major) the idea is novel,  and appears to be theoretically well supported, although I did not review the proofs in the supplementary material.\n2. (major) The main claimed advantage of the proposed method is that is produces sharper predictives than comparable methods, which is well supported by the real-world experimental results, not withstanding the points raise in questions.\n3. (major) Code is provided, which should improve reproducibility. It is based on a widely used framework, which increases potential for impact.\n\nWEAKNESSES:\n1. (major) The paper is not sufficiently clear. The method is fairly well explained, but the evaluation is extremely hard to follow, details of which are in the questions section. One particular area for straightforward correction is with Table 2: the values with the lowest mean are highlighted but there are several rows where the estimates are overlapping, so this should be clarified by highlighting.\n\nQUESTIONS:\n1. It's not clear to me how to interpret figure 2. I guess that the crosses are training data, what about the squares? Are they the calibration data?\n2. I guess Table 1 is for the synthetic data plotted in figure 2, but I don't think this is stated anywhere.\n2. When you train the vanilla GP, do you add the calibration data to the training data?\n3. The sharpness of the predictives is claimed to be an advantage, but really it looks like you cover less of the data as a result in the toy example?\n4. I appreciate that the main purpose of Table 2 is to compare the sharpness amongst the well-calibrated methods,  but I think it would be highly worthwhile to see how a vanilla GP compares on these same metrics.\n5. With respect to line 103, it is mentioned that the assumption holds for isotropic kernels. I think that it also works out if you transform an isotropic kernel to have different lengthscales in each dimension, and I think you've implied this elsewhere, is that right?\n\nSome suggestions and typos:\n* For the hyperparameters, could you use $θ$?\n* It would be helpful to have a bit more explanation in section 3. For example, instead of saying 'are small for every $δ$' perhaps you could say that the goal is to minimise that quantity for every $δ$.\n* line 127: the z-score -> z-score\n* Summations in equations 5, 7: provide the starting value (i=1 I guess)\n\nLIMITATIONS:\nThe main limitation of the method appears to be its applicability to different covariance functions, and the added computational cost, both of which are noted clearly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue that the posterior variance of Gaussian processes are often poorly calibrated, typically underestimating quantile estimation. They propose a new method to calibrated uncertainty bounds, by training a quantity related to the posterior variance with new hyper parameters. This method further leverages optimization of the distance between the quantiles and the predictive mean, enabling sharp calibration. The method is tested on synthetic toy examples and standard UCI benchmark datasets, and appears to perform well against existing methods.\n\nSTRENGTHS:\nThe paper is well written and presented and addresses a relevant problem, offering a principled solution.\n\nWEAKNESSES:\nThe definition of a Gaussian process should specify that any **finite** collection of points are jointly Gaussian.\n\nThe most concerning issue is that the paper appears to be dismissive of two recent competing methods (Song et al.; Kuleshov & Deshpande) in the introduction, and does not mention them again. Indeed, the current work has advantages over each of these methods in terms of its construction and theoretical guarantees, however comparisons to these methods in terms of performance, as well as some further discussion, should help demonstrate the benefits of the proposed method. For example, the current method still relies upon a reasonably large dataset size, so dismissing a competing method because it only has asymptotic guarantees is maybe premature.\n\nI would also question the presentation of the numerical results, in particular the use of both highlighting and boldface to denote best performing model in the tables. Since confidence intervals are provided, it may be advisable to bold the best performing ones, and using green to denote statistical significance that a single model is the best performing (i.e. multiple models can be bolded but none or one should be green).\n\nQUESTIONS:\nHow does the proposed method compare with those proposed in Song et al, Kuleshov & Deshpande, as well as Capone, Lederer & Hirche?\n\nLIMITATIONS:\nThere is little view for potential negative societal impact due to improving the calibration of uncertainty bounds for Gaussian processes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles calibration of Gaussian processes in regressions. The authors argue that while maximizing the evidence is a good way to choose hyperparameters to obtain an accurate posterior mean, it generally does not produces an accurate posterior variance. For this purpose, they propose a different way to obtain hyperparameters for the posterior variance which minimizes the length of centered confidence intervals, while constraining the parameters to provide accurate empirical quantiles. By replacing the constraint with a more treatable piece-wise expressions, they transform the problem into an unconstrained one. The further a practical algorithm and a theoretical result to obtain calibrated confidence intervals simultaneously over different confidence thresholds. Experimentation on toy and UCI datasets show the benefits of the method.\n\nSTRENGTHS:\nI think the paper is well written and clear. The solution proposed is reasonable and it tackles an important problem. The algorithms derived seem to be practical, and they are motivated by theoretical arguments that appear valid.\n\nWEAKNESSES:\nThe authors bring up themselves that the core methodology they propose shares the same scalability issues as standard GPs, as it involves they inversion of a very large matrix. Hence, approximation methods are needed.\n\nQUESTIONS:\n- Not immediately clear to me what is the effect of replacing equation (5) with equation (6). I understand that the problem becomes easier to solve. Is the replacement, however, detrimental in some way? It is also not straightforward why the expression is (6) is the right one to use. I would encourage the authors to expand on this section.\n\n- It is not super clear how easily the inequality in Assumption 4.1 holds. Perhaps making some examples, and discussing limitations of this assumption, would make this part clearer.\n\nLIMITATIONS:\nThe work handles calibration for regression only.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}