Learn Together, Stop Apart: a Novel Approach to Ensemble Pruning

1 INTRODUCTION . There are still many areas where classical machine learning algorithms prevail over deep neural networks despite the dramatic growth of their usage in artificial intelligence research . One of such classical algorithms is Gradient Boosting ( GB ) ( Friedman ( 2001 ) ) . It allows to obtain high-quality models on table data with no multimedia ( e.g. , images , audios , videos ) , with samples full of categorical features , noisy features and labels , missing data ( Zhang & Haghani , 2015 ; Li et al. , 2007 ; Babajide Mustapha & Saeed , 2016 ) . Also , the undoubted advantage of the boosting method is the low computational complexity of training and inference ( Deng et al. , 2018 ) . For these reasons , Gradient Boosting is widely used in ranking ( Chapelle & Chang , 2011 ) , recommender systems ( Cheng et al. , 2014 ) , meta-learning ( LeDell & Poirier , 2020 ) , and many other tasks ( Touzani et al. , 2018 ; Trofimov et al. , 2012 ; Ling et al. , 2017 ) . In recent years , many hyperparameters and additional options have been proposed for GB influencing the performance of the learned model ( Ke et al. , 2017 ; Ibragimov & Gusev , 2019 ) . But the learning rate ( weight of each model in the ensemble ) and the size of the ensemble are the key ones . Large models are responsible for revealing complex dependencies in the data but require more time for training and inference ( Friedman , 2002 ) . In comparison , the smaller ones are less expressive but more time-efficient . The standard approach to select an optimal number of training steps is to control the quality of the model by measuring it on a hold-out sample called validation set , which is separate from the training data . The idea is to set a large enough size of the model and find the moment ( overfitting point ) when the validation score stops growing and begins going down . Then one can prune the ensemble to the retrieved number of iterations . The described method has a significant and surprisingly understudied weakness . The approach assumes the existence of a universal ensemble size equally effective for any instance in the sample . In other words , the hypothesis is that all samples require approximately the same number of learners to fit them well . However , in practice , the learning task can consist of different subtasks , which correspond to different regions in the input space of the dataset , where examples follow different distributions with diversified complexities and functional dependencies . In particular , the data space may contain regions of both simple and complex surfaces for training . For the first ones , the ensemble needs a relatively small number of boosting rounds to be trained well , while the latter requires a way longer path until convergence . In this case , the generic boosting size selected by the least regret principle is a compromise between simple and complex areas . This approach encourages models with a composition of overfitted and underfitted regions in the dataset . To handle this issue , we propose a new method to prune large GB models based on an adaptive choice of the optimal size of the ensemble . As in the standard version of GB ( Friedman , 2001 ) , we train one sequence of learners in an ensemble but apply a different number of learned models to different regions in the dataset . Namely , we build an additional model that divides the input space into regions where the distribution of data points has homogeneous complexity and representativity . Then we optimize the ensemble size to each region individually . Our method incurs meager computational costs and can be easily incorporated into any existing learning pipeline . We apply the proposed approach to state-of-the-art open-source GB algorithms and demonstrate its ability to outperform on popular publicly available benchmarks consistently . We show that the described problem of the universal stopping moment highly affects the quality of trained models . To the best of our knowledge , this is the first research devoted to adaptive , instance–wise early stopping in GB , and we hope this paper will encourage further research of the GB algorithm . The rest of the paper is organized as follows . Section 2 introduces notations and background on GB . Previous works on early stopping and ensemble pruning are discussed in Section 3 . In Section 4 , we reveal the details of the proposed approach and present theoretical reasoning and discussions . In Section 5 , the effectiveness of the algorithm is empirically studied using several popular datasets . Section 6 makes conclusions and proposes ideas for future work . 2 BACKGROUND . In this section , we introduce necessary notations and briefly discuss basic concepts concerning gradient boosting and cross-validation for clarity and independent reading reasons . 2.1 GRADIENT BOOSTING . Let S = { xi , yi } ni=1 be a sample from some fixed but unknown distribution P ( x , y ) , where xi = ( x1i , ... , x m i ) ∈ X is an m-dimensional feature representation and yi ∈ Y is a target value of the i-th observation . The classical formulation of the learning problem consists in constructing a function F : X→ Y minimizing the expected target prediction error , which is calculated using a loss function L : Y× Y→ R+ : L ( P , F ) : = E ( x , y ) ∼P [ L ( F ( x ) , y ) ] → min F Since the distribution P is not given and the sample S is the only source of data , the task reduces to empirical risk minimization problem : L̂ ( S , F ) = Ê ( x , y ) ∼S [ L ( F ( x ) , y ) ] = 1 n n∑ i=1 L ( F ( xi ) , yi ) → min F The ability to achieve smaller value of an empirical risk is bounded by the complexity of the set F from which the desired function F ∈ F is selected . A common approach to increase the expressiveness of the learned model is to build a composition ( or an ensemble ) of functions from F . Gradient Boosting ( GB ) constructs an ensemble FB of size B as a weighted sum of base functions { f1 , f2 , ... , fB } ⊂ F : FB ( x ) = B∑ i=1 αifi ( x ) ( 1 ) When the set of available base functions F is closed under scalar multiplication , multipliers αi are usually fixed and equal : ∀i αi = α , where α is a hyperparameter of the GB algorithm called learning rate . Having constructed the first t − 1 terms , the learning algorithm aimes to select the next function ft sequentially as a solution of : L̂ ( S , Ft−1 + ft ) = 1 n n∑ i=1 [ L ( Ft−1 ( xi ) + ft ( xi ) , yi ) ] → min ft The approximate solution of the latter equation in GB is usually constructed as follows . Algorithm calculates first and second order derivatives of L̂ at the point Ft−1 w.r.t . predicted values ŷ : gti = ∂L ( ŷi , yi ) ∂ŷi ∣∣∣∣ ŷi=Ft−1 ( xi ) , hti = ∂2L ( ŷi , yi ) ∂ŷ2i ∣∣∣∣ ŷi=Ft−1 ( xi ) , and selects a least squares estimator to Newton ’ s gradient step in the functional space : ft = argmin f∈F N∑ i=1 hti ( ~xi , yi ) ( f ( ~xi ) − ( − g t i ( ~xi , yi ) hti ( ~xi , yi ) ) ) 2 , ( 2 ) see ( Chen & Guestrin , 2016 ) for details . 2.2 MODEL SELECTION VIA CROSS-VALIDATION . Since the quality estimation based on a train set used in the learning process is biased ( Prokhorenkova et al. , 2017 ) ( the inference is performed on the unseen data ) , it is conventional to use a separate independent set , called validation set , to control the generalization ability of the algorithm . The whole dataset S is split into two disjoint sets Strain and Svalid , where the first one is used for learning and the latter for quality estimation . The final result of this procedure is often highly dependent on the particular train-validation split and , therefore , quality estimation can be very noisy . To tackle this issue , one can use crossvalidation ( Stone , 1974 ) method : split the data S into k subsets of approximately equal size , or folds , ( S1 , S2 , ... , Sk ) s.t . S = k⊔ i=1 Si , and perform k rounds of training-evaluation cycle using S−i : = S/Si as the training set and Si as the validation data for each i ∈ { 1 , 2 , ... , k } . Then the estimated quality is calculated as the mean value of qualities on validation sets over all iterations of described procedure . Another source of bias in the quality estimator is mismatch of target distributions in the training and validation samples . Because the splits in the standard cross-validation procedure are generated randomly the weights of positive samples ( in binary classification tasks ) in the trained model may differ from the data on which quality control is performed . To avoid this effect stratified sampling scheme ( preserving the proportions of the target ) is usually applied . 3 RELATED WORK . 3.1 EARLY STOPPING . Early stopping is a task of controlling the learning process and interrupting it to avoid unnecessary boosting steps , which increase complexity of the model and can lead to overfitting . Since the number of learning steps is directly connected to the complexity of the model , larger ensemble sizes lead to models of smaller bias but larger variance ( bias–variance tradeoff ) . One of the ideas proposed in the literature ( Chang et al . ( 2010 ) , Mayr et al . ( 2012 ) ) is to penalize the complexity of the models , e.g. , via AIC-based methods by approximating the ensemble ’ s degrees of freedom . Some works use generalization bounds of the algorithm employing VC-dimension ( Freund & Schapire ( 1997 ) ) , Rademacher complexity ( Cortes et al . ( 2019 ) ) , or in the PAC setting ( Yao et al . ( 2007 ) , Wei et al . ( 2017 ) ) . These methods do not require separate validation control , but in most cases , are not applicable in real-world tasks since the obtained bounds are distribution-agnostic . Therefore the approximations are very rough . Standard approaches of early stopping mentioned in most of the well-known implementations of GB utilize the simple ” waiting ” idea : if the validation quality does not change for some ” reasonable ” number of iterations , then the training must be stopped ( see , e.g. , Click et al . ( 2016 ) ) . It is important to note that this kind of early stopping may be performed simultaneously with the boosting learning procedure step-by-step so that the training is stopped at the same time when a specific criterion is met . Unfortunately , this method has nothing to do with the double–descent problem ( Belkin et al. , 2019 ) , and the choice of the required number of waiting rounds remains at the researcher ’ s discretion based on experience or heuristic assumptions .

Summary: The paper proposes a novel method to set the optimal ensemble size in gradient boosting. In particular, the authors propose an adaptive strategy that sets distinct ensemble sizes for different regions of the input space. For that, they propose dividing the input space in coherent regions (whose instances are similar both in terms of features and labels) and estimating the optimal ensemble size within those regions. For clustering data, they propose using a decision tree induced from the entire training set, and the leaves of the trees are the clusters that comprise the data partition. They show the results of both a biased and a less-biased estimator for finding out the optimal ensemble size per region, and they compare their findings with the traditional strategy of simply pruning the ensembles based on a single optimal number of learners estimated from a cross-validation procedure. Results in 6 public datasets show that in at least 4 of those datasets the method seems to provide better results.