Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper addresses the question that considers if the marginal likelihood and the posterior predictive losses exhibit a monotone error curve (as other learning models do) and double descent in the input dimension. For that reason, the paper introduces the necessary conditions for proving the results and verify empirically on synthetic problems that this is indeed true or not. 
strength_and_weaknesses: **Strengths:** The paper adequately addresses the literature around the considered question. The formulation of the problem around the marginal likelihood of GPs and predictive posterior quantities seems correct to me and the obtained theoretical result seems clear to me. Some connections are built, for instance with Fong & Holmes, 2020 and the analysis with respect to the obtained result for the regularisation hyperparameter $\lambda$ and temperature $\gamma$ is also clear.

**Weaknesses:** The manuscript presents several weaknesses that I want to remark in the following lines:

- *On GPs, Bayesian statistics and clarity:* Despite the correctness of the question addressed and the considered ideas, I think that the presentation of the GP model plus Bayesian metrics, i.e. posterior predictive and log-posterior predictive density, is far from being clear or easily understandable. This is problematic as this has been longly studied in the recent literature since Rassmussen & Williams (2006), and there are simpler and clearer ways to make the presentation of such model. Example can easily found in the recent literature. Additionally, some use of notation is not entirely positive for me, for instance, the regularisation parameter $\lambda$ appears in the whole manuscript, when it could be simply added as a kernel scale parameter, right? 

- *Imprecise claims and sentences:* I find somehow some imprecise comments and claims in the paper that make me doubt or be unsure of the technical claims around the GP analysis. For instance:

> In practice, the hyperparameters (...) are often tuned to minimize the Bayes free energy. This is an
empirical Bayes procedure, and typically achieves excellent results (Krivoruchko & Gribov, 2019)

Exhaustive efforts for training GP models (including tuning of hyperparameters) based on the log-marginal likelihood have been done in the last 15 years. Sentences as the previous one, are somehow surprising to me. Even if the Bayes free energy is defined as the negative log-marginal likelihood...

- *Connections with CV, but no further analysis:* The results presented by Fong & Holmes (2020) were extremely revelant for the community and particularly for the connection between log-marginal likelihood, log-predictive scores and average k-fold-out cross validation (CV). This connection is mentioned before Section 2.3. and where the average $n^{-1}F_n$ is first described. However, to my eyes, this seems extremely related to the Theorem 1, but no mentions or links to Fong & Holmes (2020) are added. I really do not perceive much of a difference between the two results, and make me think if some this analysis is not partially covered in the previous work.

- *Analysis in a very reduced scenario:* The theoretical result is interesting to me, but seems quite reduced or constrained for GPs, when it seems could be considered from a wider perspective of more general Bayesian models. Additionally, I find the practical utility missing or at least not mentioned or properly discussed.


clarity,_quality,_novelty_and_reproducibility: Clear and theoretically interesting work but not entirely novel at this point.
summary_of_the_review: The paper has several flaws that make me doubt about its quality to be accepted in its current format.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: There has recently been renewed interest in further studying the theoretical foundations influencing the quality of increasingly parametric models under conditions of larger dataset sizes and higher dimensionality. This paper focus on the latter case, with particular focus on how conditions observed for learning curves on predictive performance carry over to metrics that instead describe uncertainty quantification (UQ). By focusing on non-parametric Gaussian process models, the authors illustrate how increased dimensionality impacts the quality of uncertainty estimates returned by such models. With some assumptions on the choice of kernel used and the nature of additional dimensions added to existing data, the authors are able to formalize how the aforementioned UQ metrics are expected to change as input dimensionality is increased. These claims are verified using a variety of experiments on synthetic and real-world datasets.
strength_and_weaknesses: - There is always great value in papers that contribute towards demystifying how widely-used models behave under controlled set-ups, and the insights provided in this paper are expected to be of particular interest to researchers working on Gaussian processes and Bayesian deep learning models. 
- I particularly appreciated how the authors considered different measurements of uncertainty quantification, and illustrated how the expectations for how learning improves with added dimensionality vary in each case. While this entails that some of the findings are less definitive, the work comes across as more complete as a result of their inclusion.
- Although the findings rely on making certain assumptions on the choice of kernel, I believe that the findings are still sufficiently broad, and it makes sense to leave further extensions, such as adapting to NTKs, for future work.
- One aspect I would have liked to see more of is how the findings of this submission influence future work on Gaussian processes and other Bayesian models? Are there any specific takeaways which might guide the design of models yielding improved estimates with better uncertainty calibration? 
- Furthermore, although the authors highlight the importance of additional covariates being Gaussian for some of the derived theorems to hold, it wasn’t clear whether there are also any actionable learnings practitioners could take from this.
clarity,_quality,_novelty_and_reproducibility: I admit to not being very familiar with certain aspects of random matrix theory that are here used to derive key propositions and theorems; however, the assumptions made appeared to be sensible, as well as the overall findings and conclusions. The contributions also appear to be novel as they focus on the less explored area of how input dimensionality impacts uncertainty quantification in non-parametric models such as Gaussian processes.

The paper is well-written and the structure is sensible, although I feel as though the paper would benefit from more illustrative examples, especially in the introductory sections where concepts such as double descent may be less intuitive to grasp for readers who are unfamiliar with related work on the subject.
summary_of_the_review: I consider this to be a solid paper, which delivers interesting theoretic insight that could inspire further research in this direction. I do think that the paper could benefit from certain concepts being better illustrated overall however, for which some rewriting may be necessary.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In their manuscript, "Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes", the authors consider the topic known in machine learning as 'double descent' which speaks to an important question for anyone using spectral representations of Gaussian processes for predictive modelling: how many features should I use?  A particular emphasis is given here to the performance of these models in terms of uncertainty quantification. Theoretical results are derived and numerical examples are given in some well- and miss- specified settings.

strength_and_weaknesses: The authors demonstrate a strong knowledge of the recent literature in this field.  I have not checked the proofs (though I would happily do so if a revised manuscript is submitted); however, the conditions supposed and the theorem statements themselves are consistent with what I what expect from a proof in their field (i.e., address the natural questions such as differentiability at the origin for the chosen kernels).

However, I struggle to see the value of the theorems given for understanding the real world behaviour of Gaussian process models, owing to the limitations of the problem set up: 
- foremost: Unless I have gravely misinterpreted the problem description, the "best-case scenario" (page 6) assumed is that all the data are generated under an iid noise model.  In this case, any improvement that an estimator that correlates the prediction to new locations against data from currently observed locations can give over the naive model (no correlation; modulo that potentially induced by learning a pooled value for the scale of e_i) is the domain of the original Stein effect (Stein 1956).  This has been examined previously for the case of RFF kernel approximations by (e.g.) Muandet et al. 2014 and Chang et al. arXiv:1705.08525).  For these particular Stein effect results to be valuable the authors would need to explain---either theoretically, or just heuristically but with convincing numerical examples---why the results for an iid noise model should be of value to practitioners trying to model highly correlated (essentially, 'functional') data.
- second: the restriction to a setting in which estimation of a data-driven bandwidth is not included in the empirical Bayesian solution feels like an important limitation relative to the frequency of this additional calibration step in practice.

clarity,_quality,_novelty_and_reproducibility: The exposition is overall clear and the mix of formal and heuristic discussion is appropriate.  I do find that the use of the term 'dimension' is confused/confusing in places: primarily the dimension is considered here as the number of spectral features rather than the dimension of the input domain.  The quality is high in terms of technical validity but lower in terms of transferability and applicable insights under the current presentation.  The novelty is difficult to assess as the Stein estimator papers I mentioned have not been discussed with regard to similarities and differences against the present study.  The reproducibility would seem high.

summary_of_the_review: The topic is of importance and the results seem correct, but there is missing a strength of argument for the value of the insights given by these results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper asks: Are there double descent phenomena when performing uncertainty quantification with gaussian processes? They provide both positive and negative answers. On the negative side, they show that the learning curve is monotone with respect to the input dimension. On the positive side, they show that other forms of posterior predictive loss do exhibit double descent. The reviewers were skeptical about whether this work furnishes any insights relevant for practice. 
justification_for_why_not_higher_score: This paper explores double descent phenomena in gaussian processes. There is by now a large literature on double descent. Originally the motivation was to shed light on practical findings about why increasing the dimension could actually, counter-intuitively, improve performance. However it is not clear what the practical relevance is of the particular models and findings here.
Decision: Reject