Uncertainty in Gradient Boosting via Ensembles

1 INTRODUCTION . Gradient boosting ( Friedman , 2001 ) is a widely used machine learning algorithm that achieves stateof-the-art results on tasks containing heterogeneous features , complex dependencies , and noisy data : web search , recommendation systems , weather forecasting , and many others ( Burges , 2010 ; Caruana & Niculescu-Mizil , 2006 ; Richardson et al. , 2007 ; Roe et al. , 2005 ; Wu et al. , 2010 ; Zhang & Haghani , 2015 ) . Gradient boosting based on decision trees ( GBDT ) underlies such well-known libraries like XGBoost , LightGBM , and CatBoost . In this paper , we investigate the estimation of predictive uncertainty in GBDT models . Uncertainty estimation is crucial for avoiding costly mistakes in high-risk applications , such as autonomous driving , medical diagnostics , and financial forecasting . For example , in self-driving cars , it is necessary to know when the AI-pilot is confident in its ability to drive and when it is not to avoid a fatal collision . In financial forecasting and medical diagnostics , mistakes on the part of an AI forecasting or diagnostic system could either lead to large financial or reputational loss or to the loss of life . Crucially , both financial and medical data are often represented in heterogeneous tabular form — data on which GBDTs are typically applied , highlighting the relevance of our work on obtaining uncertainty estimates for GBDT models . Approximate Bayesian approaches for uncertainty estimation have been extensively studied for neural network models ( Gal , 2016 ; Malinin , 2019 ) . Bayesian methods for tree-based models ( Chipman et al. , 2010 ; Linero , 2017 ) have also been widely studied in the literature . However , this research did not explicitly focus on studying uncertainty estimation and its applications . Some related work was ∗All authors contributed equally and are listed in alphabetical order . done by Coulston et al . ( 2016 ) ; Shaker & Hüllermeier ( 2020 ) , who examined quantifying predictive uncertainty for random forests . However , the area has been otherwise relatively under-explored , especially for GBDT models that are widely used in practice and known to outperform other approaches based on tree ensembles . While for classification problems GDBT models already return a distribution over class labels , for regression tasks they typically yield only point predictions . Recently , this problem was addressed in the NGBoost algorithm ( Duan et al. , 2020 ) , where a GBDT model is trained to return the mean and variance of a normal distribution over the target variable y for a given feature vector . However , such models only capture data uncertainty ( Gal , 2016 ; Malinin , 2019 ) , also known as aleatoric uncertainty , which arises due to inherent class overlap or noise in the data . However , this does not quantify uncertainty due to the model ’ s inherent lack of knowledge about inputs from regions either far from the training data or sparsely covered by it , known as knowledge uncertainty , or epistemic uncertainty ( Gal , 2016 ; Malinin , 2019 ) . One class of approaches for capturing knowledge uncertainty are Bayesian ensemble methods , which have recently become popular for estimating predictive uncertainty in neural networks ( Depeweg et al. , 2017 ; Gal & Ghahramani , 2016 ; Kendall et al. , 2018 ; Lakshminarayanan et al. , 2017 ; Maddox et al. , 2019 ; Smith & Gal , 2018 ) . A key feature of ensemble approaches is that they allow overall uncertainty to be decomposed into data uncertainty and knowledge uncertainty within an interpretable probabilistic framework ( Depeweg et al. , 2017 ; Gal , 2016 ; Malinin , 2019 ) . Ensembles are also known to yield improvements in predictive performance . This work examines ensemble-based uncertainty-estimation for GBDT models . The contributions are as follows . First , we consider generating ensembles using both classical Stochastic Gradient Boosting ( SGB ) as well as the recently proposed Stochastic Gradient Langevin Boosting ( SGLB ) ( Ustimenko & Prokhorenkova , 2020 ) . Importantly , SGLB allows us to guarantee that the models are asymptotically sampled from a true Bayesian posterior . Second , we show that using SGLB we can construct a virtual ensemble using only one gradient boosting model , significantly reducing the computational complexity . Third , to understand the attributes of using ensembles-based uncertainty estimation in GBDT models , we conduct extensive analysis on several synthetic datasets . Finally , we evaluate the proposed approach on a range of real regression and classification datasets . Our results show that this approach successfully enables the detection of anomalous out-of-domain inputs . Importantly , our solution is easy to combine with any implementation of GBDT . Our methods have been implemented within the open-source CatBoost library . The code of our experiments is publicly available at https : //github.com/yandex-research/GBDT-uncertainty . 2 PRELIMINARIES . Uncertainty Estimation via Bayesian Ensembles In this work we consider uncertainty estimation within the standard Bayesian ensemble-based framework ( Gal , 2016 ; Malinin , 2019 ) . Here , model parameters θ are considered random variables and a prior p ( θ ) is placed over them to compute a posterior p ( θ|D ) via Bayes ’ rule : p ( θ|D ) = p ( D|θ ) p ( θ ) p ( D ) . ( 1 ) where D = { x ( i ) , y ( i ) } Ni=1 is the training dataset . Each set of parameters can be considered a hypothesis or explanation about how the world works . Samples from the posterior should yield explanations consistent with the observations of the world contained within the training data D. However , on data far from D each set of parameters can yield different predictions . Therefore , estimates of knowledge uncertainty can be obtained by examining the diversity of predictions . Consider an ensemble of probabilistic models { P ( y|x ; θ ( m ) ) } Mm=1 sampled from the posterior p ( θ|D ) . Each model P ( y|x , θ ( m ) ) yields a different estimate of data uncertainty , represented by the entropy of its predictive distribution ( Malinin , 2019 ) . Uncertainty in predictions due to knowledge uncertainty is expressed as the level of spread , or “ disagreement ” , of models in the ensemble ( Malinin , 2019 ) . Note that exact Bayesian inference is often intractable , and it is common to consider either an explicit or implicit approximation q ( θ ) to the true posterior p ( θ|D ) . While a range of approximations has been explored for neural network models ( Gal & Ghahramani , 2016 ; Lakshminarayanan et al. , 2017 ; Maddox et al. , 2019 ) 1 , to the best of our knowledge , limited work 1A full overview is available in ( Ashukha et al. , 2020 ; Ovadia et al. , 2019 ) . has explored Bayesian inference for gradient-boosted trees . Given p ( θ|D ) , the predictive posterior of the ensemble is obtained by taking the expectation with respect to the models in the ensemble : P ( y|x , D ) = Ep ( θ|D ) [ P ( y|x ; θ ) ] ≈ 1 M M∑ m=1 P ( y|x ; θ ( m ) ) , θ ( m ) ∼ p ( θ|D ) . ( 2 ) The entropy of the predictive posterior estimates total uncertainty in predictions : H [ P ( y|x , D ) ] = EP ( y|x , D ) [ − ln P ( y|x , D ) ] . ( 3 ) Total uncertainty is due to both data uncertainty and knowledge uncertainty . However , in applications like active learning ( Kirsch et al. , 2019 ) and out-of-domain detection it is desirable to estimate knowledge uncertainty separately . The sources of uncertainty can be decomposed by considering the mutual information between the parameters θ and the prediction y ( Depeweg et al. , 2017 ) : I [ y , θ|x , D ] ︸ ︷︷ ︸ Knowledge Uncertainty = H [ P ( y|x , D ) ] ︸ ︷︷ ︸ Total Uncertainty −Ep ( θ|D ) [ H [ P ( y|x ; θ ) ] ] ︸ ︷︷ ︸ Expected Data Uncertainty ≈ H [ 1 M M∑ m=1 P ( y|x ; θ ( m ) ) ] − 1 M M∑ m=1 H [ P ( y|x ; θ ( m ) ) ] . ( 4 ) This is expressed as the difference between the entropy of the predictive posterior , a measure of total uncertainty , and the expected entropy of each model in the ensemble , a measure of expected data uncertainty . Their difference is a measure of ensemble diversity and estimates knowledge uncertainty . Unfortunately , when considering ensembles of probabilistic regression models { p ( y|x ; θ ( m ) ) } Mm=1 over continuous-valued target y ∈ R , it is no longer possible to obtain tractable estimates of the ( differential ) entropy of the predictive posterior , and , by extension , mutual information . In this cases uncertainty estimates can instead derived via the law of total variation : Vp ( y|x , D ) [ y ] ︸ ︷︷ ︸ Total Uncertainty = Vp ( θ|D ) [ Ep ( y|x , θ ) [ y ] ] ︸ ︷︷ ︸ Knowledge Uncertainty +Ep ( θ|D ) [ Vp ( y|x , θ ) [ y ] ] ︸ ︷︷ ︸ Expected Data Uncertainty . ( 5 ) This is conceptually similar to the decomposition ( 4 ) obtained via mutual information . For an ensemble of probabilistic regression models which parameterize the normal distribution , and where each models yields a mean and standard-deviation , the total variance can be computed as follows : Vp ( y|x , D ) [ y ] ︸ ︷︷ ︸ Total Uncertainty ≈ 1 M M∑ m=1 [ ( M∑ m=1 µm M ) − µm ] 2 ︸ ︷︷ ︸ Knowledge Uncertainty + 1 M M∑ m=1 σ2m︸ ︷︷ ︸ Expected Data Uncertainty , { µm , σm } = f ( x ; θ ( m ) ) . ( 6 ) However , while these measures are tractable , they are based on only first and second moments , and may therefore miss high-order details in the uncertainty . They are also not scale-invariant , which can cause issues is the scale of prediction on in-domain and out-of-domain data is very different . Gradient boosting is a powerful machine learning technique especially useful on tasks containing heterogeneous features . It iteratively combines weak models , such as decision trees , to obtain more accurate predictions . Formally , given a dataset D and a loss function L : R2 → R , the gradient boosting algorithm ( Friedman , 2001 ) iteratively constructs a model F : X → R to minimize the empirical risk L ( F |D ) = ED [ L ( F ( x ) , y ) ] . At each iteration t the model is updated as : F ( t ) ( x ) = F ( t−1 ) ( x ) + ϵh ( t ) ( x ) , ( 7 ) where F ( t−1 ) is a model constructed at the previous iteration , h ( t ) ( x ) ∈ H is a weak learner chosen from some family of functionds H , and ϵ is learning rate . The weak learner h ( t ) is usually chosen to approximate the negative gradient −g ( t ) ( x , y ) : = −∂L ( y , s ) ∂s ∣∣ s=F ( t−1 ) ( x ) : h ( t ) = argmin h∈H ED [ ( − g ( t ) ( x , y ) − h ( x ) ) 2 ] . ( 8 ) A weak learner h ( t ) is associated with parameters ϕ ( t ) ∈ Rd . We write h ( t ) ( x , ϕ ( t ) ) to reflect this dependence . The set of weak learners H often consists of shallow decision trees , which are models that recursively partition the feature space into disjoint regions called leaves . Each leaf Rj of the tree is assigned to a value , which is the estimated response y in the corresponding region . We can write h ( x , ϕ ( t ) ) = ∑d j=1 ϕ ( t ) j 1 { x∈Rj } , so the decision tree is a linear function of ϕ ( t ) . The final GBDT model F is a sum of decision trees ( 7 ) and the parameters of the full model are denoted by θ . For classification tasks , a model yields estimates data uncertainty if it is trained via negative loglikelihood and provides a distribution over class labels . However , classic GBDT regression models yield point predictions , and there has been little research devoted to estimating predictive uncertainty . Recently , this issue was addressed by Duan et al . ( 2020 ) via an algorithm called NGBoost ( Natural Gradient Boosting ) , which allows estimating data uncertainty . NGBoost simultaneously estimates the parameters of a conditional distribution p ( y|x , θ ) over the target y given the features x , by optimizing a proper scoring rule . Typically , a normal distribution over y is assumed and negative log-likelihood is taken as a scoring rule . Formally , given an input x , the model F predicts two parameters of normal distribution - the mean µ and the logarithm of the standard deviation log σ . The loss function is the expected negative log-likelihood:2 p ( y|x , θ ( t ) ) = N ( y|µ ( t ) , σ ( t ) ) , { µ ( t ) , log σ ( t ) } = F ( t ) ( x ) . L ( θ|D ) = ED [ − log p ( y|x , θ ) ] = − 1 N N∑ i=1 log p ( y ( i ) |x ( i ) , θ ) . ( 9 ) ( 10 ) Note that θ denotes the concatenation of two parameter vectors used to predict µ and log σ .

Summary: This paper studied the uncertainty estimation in GBDT method. The authors described 3 methods to estimate the uncertainty. With SGB, the estimation is achieved by training multiple models using data sub-samples. With SGLB, the authors derived that we can estimate the posterior distribution of the model parameters. These two methods both have the disadvantage that the training time is multiplicative of the number of trained models. To address this issue, the authors proposed an improvement to SGLB which they call virtual SGLB. The main idea is to use a subset of trees in a GBDT as a model sample, so that we can train a single model but still able to estimate the uncertainty.