Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a novel active learning scheme for GPs based on a full-Bayesian solution.
strengths_and_weaknesses: The paper is well-written and it is nice to read this work. This is the the main strength, in my opinion.
The state-of-the-art discussion is the nicest part, in my opinion.

Weakness: the contribution seems a bit incremental.
questions: I have only a suggestion: to complete the  state-of-the-art discussion,
 including the acquisition functions in GP schemes (for regression of for quadrature) that considers also the gradient information as suggested in

D. H. Svendsen, et al. Active Emulation of Computer Codes with Gaussian Processes - Application to Remote Sensing, Pattern Recognition Volume 100, 2020,

F. Llorente, et al, "Adaptive quadrature schemes for Bayesian inference via active learning", IEEE Access, Volume 8, 2020. 

M. Kanagawa and P. Hennig, “Convergence Guarantees for Adaptive Bayesian Quadrature Methods,” in Advances in Neural Information Processing Systems, 2019, pp. 6234–6245.
limitations:  The contribution seems a bit incremental.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on active learning using Gaussian Processes, where the authors take a fully Bayesian approach to optimize the bias-variance trade-off, an important problem in various learning problems.
Especially, this work focuses on a small-data setting, where such bias-variance trade-off can significantly affect the learning outcomes.
The authors propose two acquisition functions based on FBGP for regression tasks, showing the potential advantages of the proposed schemes.

strengths_and_weaknesses: Taking a Bayesian approach to automatically balance the bias-variance trade-off and improve the robustness of the learning outcomes and the predictive performance of the learned models in a small-data setting is expected to be beneficial, and this work demonstrates the potential advantages of using FBGP  for active learning in regression tasks.

1. Table 2 and Figure 4 show that the two proposed acquisition functions - B-QBC and QB-MGP - can lead to better regression results for a number of test cases. However, these results also show that the proposed scheme do not consistently outperform other alternatives, nor significantly improve the learning outcomes when they do. Especially, while the comparison based on AUC (in Table 2) make the performance improvement attained by the proposed schemes relatively prominent in some cases, the actual improvement shown in the curves (Figure 4) do not appear to be very significant. This is especially so when the data size is relatively small, even though that is the setting that motivates the current work.

2. The computational burden of taking a fully Bayesian approach is not discussed in detail, and the authors simply mention that they assume that the computational burden for querying new data points would far exceed that for the Bayesian inference. While this may be the case in some applications, it would be nevertheless important and informative to compare the computational cost of different active learning schemes, which is absent in the current study.

3. The authors do not discuss the general applicability of the proposed scheme beyond the regression tasks considered in the current work. For example, BALD - shown to outperform the proposed scheme in a number of cases - is widely used for active learning for classification, and it would be meaningful to present how the proposed schemes would be applied to classification tasks and how their performance would compare to BALD and other alternatives.

4. While the authors proposed QB-MGP to consider the "predictive performance" of the learned model, when the prediction performance is of main interest, it would be more sensible to compare the performance with other active learning schemes that focus on this aspect. For example, ELR (expected loss reduction) strategies are widely used to acquire new data points that are expected to optimally reduce the error rather than reducing the uncertainty of the model parameters.

5. Recently, a number of Bayesian active learning schemes have been proposed based on the ELR strategy, whose performance has been shown to outperform BALD and various other methods with theoretical convergence guarantees. Some recent examples include:

(1) Tan et al, Diversity Enhanced Active Learning with Strictly Proper Scoring Rules, NeurIPS 2021.
(2) Zhao et al, Efficient Active Learning for Gaussian Process Classification by Error Reduction, NeurIPS 2021.

Considering that the aforementioned Bayesian active learning methods have shown to consistently outperform BALD and other existing methods with the added benefit of convergence guarantee to the optimal model, the potential benefits of the proposed schemes remain somewhat unclear. Further comparison and elaboration would be necessary to clarify the most significant advantages of the proposed methods, in comparison with the latest relevant developments in Bayesian active learning.


questions: 1. Please elaborate on the major advantages of the proposed active learning schemes in a small-data setting, especially, based on the results shown in Figure 4, since it is unclear based on the current evaluation results.

2. Please compare the computational cost of the proposed methods, in comparison with other schemes considered in this study.

3. Please clarify whether and how the proposed schemes can be generalized beyond the tasks/settings considered in the current study.

4. How would the proposed schemes (especially, QB-MGP) compare to other active learning schemes (such as ELR) that focus on predictive performance?

5. Please clarify what would be the main potential advantages of the proposed schemes compared to other recent developments in Bayesian active learning.

 



limitations: The authors note that they "create a generic active learning acquisition function" and therefore "there is no direct negative societal impacts."
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes the use of fully Bayesian GPs in the context of Bayes opt and active learning. They propose fully Bayesian versions of known acquisition functions that leverage information from the hyperparameter posterior and also suggest using the posterior predictive variance of the  GMM (non-Gaussian) process to derive an acquisition function. The claim is that the Bayesian variants yield more effective acquisition functions --  implicitly because they approximate the posterior predictive uncertainty better than ML-II. They demonstrate and analyse the performance across methods on 6 simulator tasks
strengths_and_weaknesses: Strengths:
* Interesting interpretation of the multi-modal marginal likelihood surface in terms of the bias-variance tradeoff parlance.  
* Good presentation, well-written and clear. 
* Decent experimental evaluation.
* Approximating the hyperparameter posterior - a relatively understudied area in literature.

Weakness:

*A major concern is the computational cost of running FBGP at each iteration. A naive strategy (albeit sufficient for small datasets like the ones studied) would scale as O(MN^{3}) where M is the number of samples - how exactly can this method scale? perhaps by interleaving FBGP at intervals with traditional ML-II optimisation - a more sophisticated strategy would be required. 

questions: 1) Isn't the most reasonable mode given by the mode where the marginal likelihood is maximized? and typically, the modes manifest in very starkly different marginal likelihood values. 
2) The underperformance of B-QBC on the heteroscedastic motorcycle data is not sufficiently explained, QB-MGP performs best but in terms of the criteria is not so different than B-QBC.
3) Both B-QBC and QB-MGP use hyper samples from the posterior - then \mu_GMM is the same as the mean of the models corresponding to thetas drawn from the hyper posterior \bar{\mu}? Their performance should be very similar but they are quite different - is there more insight on this? 
limitations: The authors do discuss limitations - but conspicuously miss out the obvious one which is the compute for FBGPs.
review_title: null

Meta-review: metareview: This paper took a “fully Bayesian” perspective on Gaussian process (FBGP) regression tasks. The key technical contribution builds on the argument that the optimal mode of the hyperparameter posterior (i.e. over the length-scale and the noise parameters) corresponds to the optimal bias-variance trade-off. One would expect that a fully Bayesian approach (with a reasonable prior) would indeed be beneficial; it is interesting and useful to understand how such methods perform in practice in terms of robustness and the (extra) computational burden. All reviewers agree that this work provided sufficient empirical support that demonstrates the potential advantages of using FBGP for active learning in regression tasks. During the rebuttal/discussion phase, the authors provided extra empirical evidence which makes the results more convincing (e.g., discussion and empirical analysis on the computational complexity and the factors that affect such). There were no other critical concerns in the reviews. 

There are valuable suggestions in the reviews, including improving the clarity when analyzing the results against baselines, providing details of the experimental setting and results, and an in-depth discussion of the computational cost (which appears to be a key message to convey) The authors are strongly encouraged to address the concerns raised in the reviews when preparing a revision of this paper.

confidence: Certain
Decision: Accept