Learn Together, Stop Apart: a Novel Approach to Ensemble Pruning

1 INTRODUCTION . There are still many areas where classical machine learning algorithms prevail over deep neural networks despite the dramatic growth of their usage in artificial intelligence research . One of such classical algorithms is Gradient Boosting ( GB ) ( Friedman ( 2001 ) ) . It allows to obtain high-quality models on table data with no multimedia ( e.g. , images , audios , videos ) , with samples full of categorical features , noisy features and labels , missing data ( Zhang & Haghani , 2015 ; Li et al. , 2007 ; Babajide Mustapha & Saeed , 2016 ) . Also , the undoubted advantage of the boosting method is the low computational complexity of training and inference ( Deng et al. , 2018 ) . For these reasons , Gradient Boosting is widely used in ranking ( Chapelle & Chang , 2011 ) , recommender systems ( Cheng et al. , 2014 ) , meta-learning ( LeDell & Poirier , 2020 ) , and many other tasks ( Touzani et al. , 2018 ; Trofimov et al. , 2012 ; Ling et al. , 2017 ) . In recent years , many hyperparameters and additional options have been proposed for GB influencing the performance of the learned model ( Ke et al. , 2017 ; Ibragimov & Gusev , 2019 ) . But the learning rate ( weight of each model in the ensemble ) and the size of the ensemble are the key ones . Large models are responsible for revealing complex dependencies in the data but require more time for training and inference ( Friedman , 2002 ) . In comparison , the smaller ones are less expressive but more time-efficient . The standard approach to select an optimal number of training steps is to control the quality of the model by measuring it on a hold-out sample called validation set , which is separate from the training data . The idea is to set a large enough size of the model and find the moment ( overfitting point ) when the validation score stops growing and begins going down . Then one can prune the ensemble to the retrieved number of iterations . The described method has a significant and surprisingly understudied weakness . The approach assumes the existence of a universal ensemble size equally effective for any instance in the sample . In other words , the hypothesis is that all samples require approximately the same number of learners to fit them well . However , in practice , the learning task can consist of different subtasks , which correspond to different regions in the input space of the dataset , where examples follow different distributions with diversified complexities and functional dependencies . In particular , the data space may contain regions of both simple and complex surfaces for training . For the first ones , the ensemble needs a relatively small number of boosting rounds to be trained well , while the latter requires a way longer path until convergence . In this case , the generic boosting size selected by the least regret principle is a compromise between simple and complex areas . This approach encourages models with a composition of overfitted and underfitted regions in the dataset . To handle this issue , we propose a new method to prune large GB models based on an adaptive choice of the optimal size of the ensemble . As in the standard version of GB ( Friedman , 2001 ) , we train one sequence of learners in an ensemble but apply a different number of learned models to different regions in the dataset . Namely , we build an additional model that divides the input space into regions where the distribution of data points has homogeneous complexity and representativity . Then we optimize the ensemble size to each region individually . Our method incurs meager computational costs and can be easily incorporated into any existing learning pipeline . We apply the proposed approach to state-of-the-art open-source GB algorithms and demonstrate its ability to outperform on popular publicly available benchmarks consistently . We show that the described problem of the universal stopping moment highly affects the quality of trained models . To the best of our knowledge , this is the first research devoted to adaptive , instance–wise early stopping in GB , and we hope this paper will encourage further research of the GB algorithm . The rest of the paper is organized as follows . Section 2 introduces notations and background on GB . Previous works on early stopping and ensemble pruning are discussed in Section 3 . In Section 4 , we reveal the details of the proposed approach and present theoretical reasoning and discussions . In Section 5 , the effectiveness of the algorithm is empirically studied using several popular datasets . Section 6 makes conclusions and proposes ideas for future work . 2 BACKGROUND . In this section , we introduce necessary notations and briefly discuss basic concepts concerning gradient boosting and cross-validation for clarity and independent reading reasons . 2.1 GRADIENT BOOSTING . Let S = { xi , yi } ni=1 be a sample from some fixed but unknown distribution P ( x , y ) , where xi = ( x1i , ... , x m i ) ∈ X is an m-dimensional feature representation and yi ∈ Y is a target value of the i-th observation . The classical formulation of the learning problem consists in constructing a function F : X→ Y minimizing the expected target prediction error , which is calculated using a loss function L : Y× Y→ R+ : L ( P , F ) : = E ( x , y ) ∼P [ L ( F ( x ) , y ) ] → min F Since the distribution P is not given and the sample S is the only source of data , the task reduces to empirical risk minimization problem : L̂ ( S , F ) = Ê ( x , y ) ∼S [ L ( F ( x ) , y ) ] = 1 n n∑ i=1 L ( F ( xi ) , yi ) → min F The ability to achieve smaller value of an empirical risk is bounded by the complexity of the set F from which the desired function F ∈ F is selected . A common approach to increase the expressiveness of the learned model is to build a composition ( or an ensemble ) of functions from F . Gradient Boosting ( GB ) constructs an ensemble FB of size B as a weighted sum of base functions { f1 , f2 , ... , fB } ⊂ F : FB ( x ) = B∑ i=1 αifi ( x ) ( 1 ) When the set of available base functions F is closed under scalar multiplication , multipliers αi are usually fixed and equal : ∀i αi = α , where α is a hyperparameter of the GB algorithm called learning rate . Having constructed the first t − 1 terms , the learning algorithm aimes to select the next function ft sequentially as a solution of : L̂ ( S , Ft−1 + ft ) = 1 n n∑ i=1 [ L ( Ft−1 ( xi ) + ft ( xi ) , yi ) ] → min ft The approximate solution of the latter equation in GB is usually constructed as follows . Algorithm calculates first and second order derivatives of L̂ at the point Ft−1 w.r.t . predicted values ŷ : gti = ∂L ( ŷi , yi ) ∂ŷi ∣∣∣∣ ŷi=Ft−1 ( xi ) , hti = ∂2L ( ŷi , yi ) ∂ŷ2i ∣∣∣∣ ŷi=Ft−1 ( xi ) , and selects a least squares estimator to Newton ’ s gradient step in the functional space : ft = argmin f∈F N∑ i=1 hti ( ~xi , yi ) ( f ( ~xi ) − ( − g t i ( ~xi , yi ) hti ( ~xi , yi ) ) ) 2 , ( 2 ) see ( Chen & Guestrin , 2016 ) for details . 2.2 MODEL SELECTION VIA CROSS-VALIDATION . Since the quality estimation based on a train set used in the learning process is biased ( Prokhorenkova et al. , 2017 ) ( the inference is performed on the unseen data ) , it is conventional to use a separate independent set , called validation set , to control the generalization ability of the algorithm . The whole dataset S is split into two disjoint sets Strain and Svalid , where the first one is used for learning and the latter for quality estimation . The final result of this procedure is often highly dependent on the particular train-validation split and , therefore , quality estimation can be very noisy . To tackle this issue , one can use crossvalidation ( Stone , 1974 ) method : split the data S into k subsets of approximately equal size , or folds , ( S1 , S2 , ... , Sk ) s.t . S = k⊔ i=1 Si , and perform k rounds of training-evaluation cycle using S−i : = S/Si as the training set and Si as the validation data for each i ∈ { 1 , 2 , ... , k } . Then the estimated quality is calculated as the mean value of qualities on validation sets over all iterations of described procedure . Another source of bias in the quality estimator is mismatch of target distributions in the training and validation samples . Because the splits in the standard cross-validation procedure are generated randomly the weights of positive samples ( in binary classification tasks ) in the trained model may differ from the data on which quality control is performed . To avoid this effect stratified sampling scheme ( preserving the proportions of the target ) is usually applied . 3 RELATED WORK . 3.1 EARLY STOPPING . Early stopping is a task of controlling the learning process and interrupting it to avoid unnecessary boosting steps , which increase complexity of the model and can lead to overfitting . Since the number of learning steps is directly connected to the complexity of the model , larger ensemble sizes lead to models of smaller bias but larger variance ( bias–variance tradeoff ) . One of the ideas proposed in the literature ( Chang et al . ( 2010 ) , Mayr et al . ( 2012 ) ) is to penalize the complexity of the models , e.g. , via AIC-based methods by approximating the ensemble ’ s degrees of freedom . Some works use generalization bounds of the algorithm employing VC-dimension ( Freund & Schapire ( 1997 ) ) , Rademacher complexity ( Cortes et al . ( 2019 ) ) , or in the PAC setting ( Yao et al . ( 2007 ) , Wei et al . ( 2017 ) ) . These methods do not require separate validation control , but in most cases , are not applicable in real-world tasks since the obtained bounds are distribution-agnostic . Therefore the approximations are very rough . Standard approaches of early stopping mentioned in most of the well-known implementations of GB utilize the simple ” waiting ” idea : if the validation quality does not change for some ” reasonable ” number of iterations , then the training must be stopped ( see , e.g. , Click et al . ( 2016 ) ) . It is important to note that this kind of early stopping may be performed simultaneously with the boosting learning procedure step-by-step so that the training is stopped at the same time when a specific criterion is met . Unfortunately , this method has nothing to do with the double–descent problem ( Belkin et al. , 2019 ) , and the choice of the required number of waiting rounds remains at the researcher ’ s discretion based on experience or heuristic assumptions .

Summary: This paper proposes to tune the number of models in a boosting ensemble in an instance-wise fashion. The idea is to first cluster the samples using a decision tree and then to tune the size of the ensemble independently for each cluster, instead of doing it globally for all instances. An efficient two-level cross-validation procedure is designed to tune both the number of terms in each cluster and the number of clusters. Experiments are conducted on 6 large-scale problems that show that local pruning brings some improvement with respect to the more standard global pruning technique.