Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper suggests using the nested sampling algorithm to marginalize hyperparameters of Gaussian processes with spectral mixture kernels. The motivation is that the posterior over hyperparameters of such kernels is highly multimodal and nested sampling is known to work well with multimodal distributions. Furthermore, practical priors for the parameters of spectral mixture kernels are suggested. The proposed nested sampling driven marginalised Gaussian process is compared to NUTS driven marginalised Gaussian process and the Gaussian process with hyperparameters estimated by maximum likelihood on a set of low-dimensional benchmark problems. The comparison is in terms of negative log predictive density. It is shown that the proposed method usually outperforms other methods while also performing faster than NUTS.
main_review: The paper highlights the method of nested sampling, popular in physics, as a method for marginalizing over hyperparameters of spectral mixture kernels. This is a valuable contribution. The conducted empirical study is convincing in showing that the method is indeed promising in this new setting. The method is only benchmarked in low dimensions but may be useful in the time series modeling problems which are quite broad and important themselves.

# Originality
As far as I can tell, the contribution is novel.

# Quality
The claims are supported by an extensive empirical evaluation. On the downside, the implementation is not provided and said to be proprietary.

# Clarity
The paper is very well written. The most serious problem of the sort spotted by me in the paper is fonts being too small in some figures (especially in Figure 2).

# Significance
The method suggested by the paper and the corresponding empirical study will be valuable for machine learning practitioners who use low-dimensional Gaussian processes. Mostly, this work is relevant for time series modeling. Regrettably, this positive impact is somewhat limited by the fact that the code will not be openly released for people to be able to quickly and easily apply the method to their problems.

# Remarks
- Line 17. I suggest giving a reference or briefly discussing the abbreviation ML-II, it might not be a universally known one.
- Line 39. “lies not its”->”lies not in its”.
- Line 44. “some the”-> “some”.
- Lines 67-68. Phrase “the final form of the posterior predictive in a marginalised GOs is a mixture of Gaussians” is not rigorous because it is only approximately so, I suggest reflecting this in text.
- Line 75. “consider”->“considered”.
- Line 76. “demonstrate”->”demonstrated”.
- Line 79. “complet”->”complex”.
- All figures except Figure 3 need larger fonts.

limitations_and_societal_impact: The limitations and societal impact are adequately addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes nesting sampling techique for sampling from GP hyperparameter posteriors. Specifically, obtaining hyperparameters of the spectral mixture kernel is  discussed. Empirical results show that nesting sampling benefits GP performance while being computationally cheaper than HMC.
main_review: The paper addresses a problem of marginalizing GP hyperparameters. The proposed technique is nested sampling of the hyperparameters. Nested sampling (NS henceforth) was developed previuosly, however the application of the technique to GPs is original work. The authors thoroughly cite related work.

The work claims that NS improves GP behavior compared to a point estimate of the hyperparameters. On the other hand, NS takes significantly less computation time than exploring the posterior with HMC. Performance comparison between NS and HMC seems to depend largely on the kernel and the dataset. There is also a comparison with Neural Kernel Network. 
The claims are supported by experiments. 

The paper is very well-written and organized. The supplementary material provides settings to reproduce the results. Here are a some issues:
p. 3 l. 79: complet -- typo
It could also benefit to have larger pictures or larger fonts in the pictures, because they have to be ridiculously zoomed-in to be seen.
Also, please provide a clearer explanation of the choice of the uniform prior for frequencies larger than the fundamental.

The results are important because they provide a relatively low-cost (compared to HMC) technique to marginalize SM kernel hyperparameters which improves the performance of the GP. The SM kernel is indeed a very expressive family. 

Overall, very good paper.
limitations_and_societal_impact: The authors are aware of the limitations of their work and possible societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses, the problem of simultaneously evaluating the marginal likelihood for Bayesian model selection and the inference of the posterior of hyperparameters in Gaussian process (GP) regression. To this end the authors utilize the nested sampling (NS). They show empirically, that when little data is present or the dimensionality of the problem is high NS outperforms the more common methods, such as marginal-likelihood (ML-II) maximization and Hamiltonian Monte-Carlo (HMC) sampling.
main_review: The problem the authors consider is an important one in the Bayesian community. Apart from some minors (see below), I found the paper also well written. The experiments seem rigorously designed.

While I find the paper interesting and the experiments insightful, my main concern is the novelty of the proposed method. While it is true, that as the authors state NS isn’t well known in the ML community, it has been discussed to evaluate the marginal likelihood integral (see e.g. the thesis of Ian Murray [https://homepages.inf.ed.ac.uk/imurray2/pub/07thesis/murray_thesis_2007.pdf], which btw contains some nice illustrations to explain NS intuitively) and was also used in several works (of whose to be fair the manuscript is citing several). The main contribution of the current work comes from more rigorously comparing NS to the dominant solutions (ML-II, HMC), to solve the integral of interest, and using NS for  complicated priors (priors with spectral mixture kernels) in the GP regression setting. 

Overall, I think the paper is and edge case, where on the pro-side stands a rigour empirical study to showcase the advantages of NS as a widely used model in the community, and on the contra limited novelty. In my opinion, the former slightly dominates, hence my rating.

## Update

After reading the authors response and the other reviews, I think, I was potentially underestimating the novelty of the work slightly. Hence, I will increase my score to 7.

limitations_and_societal_impact: Minors:
+ I would appreciate if runtimes could be reported for the experiments.
+ Notation in Algorithm1:
++ N -> n_L in the exponent
++ I think, the index i is a bit overloaded in meaning (running index in the initial set of live points, then a counter of samplings steps). Maybe change i to j in the loop and make clear, that it is a counter.
+ I’d prefer to write the convergence criterion additionally as a formula, because I needed to read the sentence in l.140 several times, and was still struggling.
+ L.79 completE
+ I found the paragraph from l.137 onwards not helpful. Especially, when it came to the rslice option. It would be nice, if the authors could add some more details, at least in the supplement.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes using nested sampling to have a fully Bayesian GP. The idea is illustrated with spectral mixture kernel, and supported (to some extend) with experiments.
main_review: This is a paper to bring the technical of nested sampling that is well-used in physics and cosmology to the machine learning GP community. The paper is not very original and significant, given that nested sampling has been introduced previously in N(eur)IPS [Murray, 2005], and the authors of that paper have very deep knowledge of GP and are experts in this area. It is also widely accepted that margalised (or Bayesian) GP is *the correct* thing to to, though it can be somewhat challenging to execute in practice.

Nevertheless, this is a worthwhile paper, (re-)illustrating the significance of  Bayesian GP, esp in the context of the recently introduced spectral mixture kernel (SKM) which has multiple modes as illustrated by the authors. Also useful is to bring to attention the DYNESTY package. If the authors were to release their code, I can see that the code will be used frequently.

Section 4 is a particularly helpful advice for anyone attempting to use SKM. The paper is also well-written.

# Detailed comments
1. I prefer the term "Bayesian Gaussian Processes" rather than "Marginalised Gaussian Processes" in the title, but this is a matter of personal preference.
2. Given that the study is limited to SKM, this should be in the title.
3. Reorganised the paper to put SMK material in section 2 together with section 4.
4. It is paramount to cite and discuss [Murray, 2005].
5. I beg to differ from the sentence from line 232  to 233. The pattern for NS is too clear on along the axis in Figure 4, and then it blurs out along the edges. This is not what the generating function gives. It is nevertheless reassuring that all these are better than NKN.
6. Lines 262 to 263 tries to give something negative a positive light. What I read is that for simpler kernels/covariance-functions like the RBF/squared-exponential that are typically used in practice, there is no requirement for NS.
7. Lines  284 to 286 requires a correct heading.

Iain Murray, David MacKay, Zoubin Ghahramani, John Skilling. Nested sampling for Potts models. Part of Advances in Neural Information Processing Systems 18 (NIPS 2005)

Changes after author response
=======================
Removed the comment on NLPD from point 6.
limitations_and_societal_impact: See comment 6 above. The paper should say that while they strongly advice to use NS for SMK, the conclusion for simpler kernels like RBF/squared-exponential with limited number of hyperparameters required further study.
review_title: null

Meta-review: The paper proposes the use of nested sampling (NS) for inference in Gaussian process (GP) models with Gaussian likelihoods and shows the benefits of this fully Bayesian approach over competing methods such as HMC and type-II marginal likelihood hyper-parameter estimation. Although NS has been studied previously and the evaluation of the proposed method is mainly focused on the spectral mixture kernel and low-dimensional settings, all the reviewers recommend acceptance and I agree, as the paper further strengthens the machinery to carry out inference in GP models. I note and appreciate the authors’ commitment to release the corresponding code. 
Decision: Accept (Poster)