Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper studied the uncertainty estimation in GBDT method. The authors described 3 methods to estimate the uncertainty. With SGB, the estimation is achieved by training multiple models using data sub-samples. With SGLB, the authors derived that we can estimate the posterior distribution of the model parameters. These two methods both have the disadvantage that the training time is multiplicative of the number of trained models. To address this issue, the authors proposed an improvement to SGLB which they call virtual SGLB. The main idea is to use a subset of trees in a GBDT as a model sample, so that we can train a single model but still able to estimate the uncertainty.

I find the paper clearly written and well organized. The theoretical formulation of the proposed methods makes sense and clear to follow. A natural concern of the vSGLB method is the correlation between the sub-trees in GBDT. Indeed, the experiments show that vSGLB is significantly worse in uncertainty estimation in real datasets. Nevertheless, I think the authors make a good case about the trade-off between a fast training time and a good uncertainty estimation.

Uncertainty estimation is an increasingly important topic in machine learning application. GBDT method is one of the most commonly used ML method in application. This paper propose a fast approach and a more accurate approach for uncertainty estimation when using GBDT. Thus I think it would be an interesting read to ML practitioners especially.
review_title: Well written with important practical contribution

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper explores a strategy of building ensembles of Gradient Boosting Decision Trees (GBDTs) to improve capturing total uncertainty and knowledge uncertainty to better detect errors and out of domain data points. Overall, I vote for accepting. I think while the results are not super impressive, they are encouraging, and I would like to see more research exploring this area.

Pros:
1. I think error detection and OOD detection are an under explored area in the context of tabular data. The problem itself is important in terms of practicality.
2. The main idea of the paper is to build ensembles of GBDTs with two flavors of gradient descent. The intuition that ensembles should improve uncertainty estimation is natural, and something worth exploring.
3. The main improvement in terms of results is the improved out-of-domain data detection with SGLBs. Other than that, the results aren’t really better than a single GBDT for error detection.
4. The number of experiments done is quite comprehensive with a good mix of synthetic and real datasets.
5. It’s disheartening that even with ensembles of GBDTs, the PRR is quite low in most datasets. This is not the authors’ fault of course. I think this is an important contribution to the literature so that the community is aware it does not work for error detection. 
  
Cons:
1. The virtual ensemble doesn’t quite give as good performances as SGLBs which the authors were emphasizing quite a bit.
2. I couldn’t find any description on how the knowledge uncertainty was calculated practically (coding wise). 
3. I think it is important to at least show how Random forests or ExtraTreeForests are performing in terms of Precision-rejection ratio (PRR), and OOD detection. These ensemble methods typically perform well, so even if they perform worse here, that would be a benchmark on how much SGLBs are improving upon.

I would request the authors to address the cons during the rebuttal period.
review_title: Good paper on a topic that needs further exploration

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work examines a probabilistic ensemble-based framework for deriving uncertainty estimates in the predictions of gradient boosting classification and regression models. As the authors have said, predictive uncertainty is sometimes a must have feature for high-risk application of machine learning techniques.

The authors conducted a range of experiments on both synthetic and real datasets and investigated the applicability of ensemble approaches to gradient boosting models that are themselves ensembles of decision trees. The results seem to suggest these ensembles were able to detect anomaly input. The authors have also introduced the idea/concept of virtual ensemble by exploiting the ‘ensemble-of-trees’ nature of GBDT models. , which is interesting too.

In general I think this paper is trying to tackle a significant problem, so it would be good if the authors would give some real world examples of possible applications when uncertainty estimates in the predictions is essential. Another aspect is, as the authors have mentioned, boosting trees itself is an ensemble, I would expect the authors explain a bit more on why we need ensemble of ensembles rather than working out uncertainty estimates just using the trees in a single boosting-tree ensemble?
review_title: UNCERTAINTY INGRADIENTBOOSTINGVIAENSEMBLES

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The authors propose to apply the idea of Bayesian ensemble methods to (tree-based) gradient boosting methods so as to be able to measure the knowledge uncertainty (e.g., to detect anomaly or out-of-domains samples) while typically only data uncertainty (e.g. related to noise in the data) is considered.  

Overal comment:
The paper is well-written and the amount of experiments is impressive. However, the proposed approach raises several concerns addressed below and regarding the novelty mainly (comment (b)) and the actual performances (comment (d)). I think the paper could be accepted if comments (b) and (d) are addressed. 

Major comments:
(a) The motivation is well explained and I agree that not knowing when a machine learnt predictor makes a prediction with certainty or not is the key of the trust in such models. 
(b) It is unclear how the approach for estimating uncertainty is novel with respect to what was made with other ensemble approaches as Section II refers to preliminary works but Section III (except virtual ensembles) refers to ensemble of GB models and I am not sure it is an actual contribution of this paper. Please clarify what's new and what already exists in Section III and how much your approach differs / improves / modifies what has already been suggested by the uncertainties estimation via Bayesian Ensembles. 
(c) As suggested by Figure 1, it appears that a virtual ensemble is built recursively by adding the one tree model to the previous ensemble and making hence a new object of the virtual ensemble. This suggests that models are not independent and intrinsically give a much higher weight on the first model for instance. The motivation behind the building of the virtual ensemble (the choice of T/2 in the text for instance) is not very clear either. Please clarify (a) if Figure 1 is not overly simplistic and (b) the motivation of the making of a virtual ensemble (especially the value T/2).
(d) It would have been interesting  (and more convincing) to have results (of Table 1) for other non-gradient boosting techniques to convince the soundness of the proposed approach and how it compares with other techniques. 

Minor comments: 
- typo : "it's (applications)" in the first section.
- Typically, loss functions are defined as function $R^2 \rightarrow R^+$ justifying the "negative" log-likelihood. Please check.
review_title: Measuring knowledge uncertainty with GBDT

Meta-review: The authors design a framework to estimate the uncertainties in the predictions of gradient boosting models, for both classification and regression. The framework contains several methods, some that use sub-sampling on data to calculate the estimation, and some that use sub-sampling on the trees within one single gradient boosting model (i.e. virtual ensemble) to calculate the estimation. The different methods reveal the trade-off between faster calculation and good uncertainty estimation. The authors conduct extensive empirical study to demonstrate the validity of the designed framework.

The reviewers agree that the paper is well-written on a very important topic of machine learning in practice. The authors have done a great job addressing the comments from the reviewers, including the comparison to random forest, and adding more motivating examples. The reviewers believe that the work marks a good starting point for addressing this important topic. Nevertheless, the reviewers have some concerns that the results are promising but not impressive yet, and the performance of the virtual ensemble is a bit discouraging. 

Decision: Accept (Poster)