SANE: Specialization-Aware Neural Network Ensemble

1 INTRODUCTION . Real-world data distribution could be complex in most cases and people usually approximate it by a composition of several simpler distributions ( Xie et al. , 2016 ; Yang et al. , 2016 ; Tsai et al. , 2020 ) . Intuitively , to fit the complex data distribution conveniently , we can divide the complex model learning process into training several simpler models , each of which specializes in one simple distribution . In this paper , we call this idea model specialization . Ensemble learning is one of the most effective methods to leverage model specialization . Originated from decades ago ( Hansen & Salamon , 1990 ) , ensemble method ( Schapire , 1990 ; Breiman , 1996 ; Zhou et al. , 2002 ) has been proven effective for practical machine learning tasks in various scenarios including computer vision ( Huang et al. , 2017 ) , natural language processing ( Shazeer et al. , 2017 ) and tabular data mining ( Liu et al. , 2020 ) . Many works conduct ensemble learning through improving diversity across base models , which has been shown to reduce the variance of the combined predictions and improve generalization ( Lee et al. , 2015 ; Zhou et al. , 2018 ) . The recent methodologies pursuing model diversity mainly through incorporating randomization in data sampling ( Breiman , 1996 ; Ho , 1995 ) or model training ( Srivastava et al. , 2014 ; Lakshminarayanan et al. , 2016 ) . Some other works propose sequential optimization such as boosting ( Freund , 1995 ; Chen & Guestrin , 2016 ; Ke et al. , 2017 ) and snapshot ensemble ( Huang et al. , 2017 ) . Other methods , e.g. , model decorrelation ( Zhou et al. , 2018 ) and diversity enhancement ( Zhang et al. , 2020 ) , explicitly encourage model diversity through incorporating additional objectives . However , overvaluing diversity may hurt the ensemble effectiveness and the trade-off between the performance and diversity of base models still remains an open problem ( Rame & Cord , 2021 ; Fort et al. , 2019 ; Masegosa , 2019 ) . Moreover , without explicitly enforcing model specialization , pursuing model diversity does not provide guarantees of each model ’ s specialization on different simple distribution in the composition . The last two columns in Figure 1 visualize the specialty of each model by plotting the data regions where the base model makes correct predictions . It shows that neither the ensemble of ( b ) randomly initialized base models nor ( c ) diversified base models produce models with different specialties . Specifically , although the method in ( c ) enforces diversity and shows low correlation among base models ’ outputs , their data regions with correct predictions are very similar as shown in the last two columns in Figure 1 , resulting in inferior performance after ensemble . Besides , simply putting specialized base models together does not necessarily bring promising ensemble performance . We need to know each base model ’ s specialty over different samples and aggregate their outputs based on their specialty . As shown in Figure 1 ( d ) , although the overall accuracy of the two base models are low , the ensemble model performs surprisingly well ( 100 % accuracy in ( d ) ) as long as 1 ) each base model specializes in unique sub-regions of the latent space and 2 ) the ensemble model knows which sub-regions each base model specializes in , demonstrating the effectiveness of specialization in ensemble learning . More detailed analyses of Figure 1 are presented in Section 2 . In this paper , we propose an end-to-end ensemble learning method that actively enforces model specialization by training models to specialize in specific simple distributions , and aggregates them based on their specialties . Firstly , we represent the composition of simple distributions by mapping them into a unified latent space , where samples from each data distribution form a locally clustered sub-region . This paper proposes to enforce each base model to specialize in one sub-region , and we call this region the base model ’ s corresponding specialty region . The base models ’ specialty regions are defined by a set of learnable anchor points , each of which corresponds to the center of the specialty region , called specialty anchor . During training , base models ’ parameters and their corresponding specialty anchors are learned simultaneously . Each base model ’ s specialty is enforced by encouraging model to focus more on samples in the vicinity of its specialty anchor . Finally , during prediction , based on the correlation between models ’ specialty anchors and the samples in the latent space , our method is able to automatically estimate each base model ’ s specialty over testing samples , and the outputs of the base models are adaptively aggregated based on their specialty over different testing samples . To demonstrate the effectiveness of the proposed method , we conduct extensive experiments in prediction tasks on two tabular datasets and two image classification datasets . The comprehensive empirical studies show that 1 ) coupling specialized base models and specializationaware aggregation can benefit ensemble learning and 2 ) the proposed specialization-aware neural network ensemble method can achieve superior performance compared with the state-of-the-art ensemble methods in various prediction tasks . 2 PRELIMINARIES AND SYNTHETIC ANALYSIS . 2.1 PRELIMINARIES . In general , ensemble learning consists of two tasks : training multiple base models and aggregating their predictions . We use { Mk } Kk=1 to denote a collection of K base models . For a given instance xn ∈ Rd and its label yn ∈ [ 1 , C ] , each base model learns a mapping Mk : Rd 7−→ RC , which produces logits for C classes . Base models ’ predictions are then aggregated in a weighted manner . The likelihood is written as p ( yn |xn ) = ∑K k=1 wk p ( yn |xn ; θk ) s.t . ∑K k=1 wk = 1 , whereMk is parameterized by θk and wk denotes the weight for aggregating k-th model prediction . Objectives of ensemble learning . To maximize the likelihood , most previous studies attempt to promote ∑K k=1 p ( yn |xn ; θk ) ( true confidence ofMk , for brevity ) , with fixed model weights on all samples . To this end , some works train models based on random initialization to reduce prediction variance and generalization errors , while others explicitly quantify diversity to reduce the correlation of errors between models . However , there is always a problem about the trade-off between diversity and accuracy in this scenario , since the weighted majority correctness among selected models determines the correctness of the ensemble . One way to avoid the conflict between adopting diverse models and boosting the overall performance is to use sample-specific model weights to aggregate models ’ predictions . Intuitively , if a base model performs much better than any other base models over a specific sub-region of the dataset containing one or more samples ( i.e. , a base model specializes in a sub-set of the dataset ) , it should be assigned with higher weight in this sub-region . As a result , we define the specialized model as follows . Definition 2.1 . ( Specialized model ) We define the specialized modelMk of a sample ( xn , yn ) with two aspects : ( 1 ) k = argmaxk′ p ( yn |xn ; θk′ ) , and ( 2 ) yn = argmaxc p ( c |xn ; θk ) where c ∈ [ 1 , C ] denotes the class label . If no model satisfies the above conditions , then xn is not specialized by any model . That is , the specialized model is the one that works correctly and has the highest prediction confidence on a given sample among all the base models . 2.2 SYNTHETIC ANALYSIS . In this section , we conduct the experiment with three ensemble solutions including a random initialization based method and a diversity-based ensemble method , and then we illustrate how specialization can benefit ensemble . A 5× 5 checkerboard synthetic dataset containing 25 Gaussian blobs shown in Figure 1 ( a ) is used for analysis . Two widely used ensemble methods are selected for this experiment . Firstly , following the classical random ensemble , base models with different weight initialization are trained independently and aggregated averagely . On the basis of the random ensemble , we further implement a diversitydriven ensemble by incorporating a diversity loss ( Li et al. , 2012 ) into the error function of each base model to encourage base models to perform differently on the same sample . To illustrate how different base models and their ensembles make predictions , we draw their decision boundaries and graphs showing their individual specialty regions , which for a single model is where it makes the correct prediction . A multi-layer perceptron ( MLP ) with one hidden layer and 20 neurons serves as the base model of ensembles through this section . For simplicity of visualization and analysis , we maintain two base models for each ensemble method . Random ensemble v.s . diversity-driven ensemble . The results of these two methods are , respectively , shown in Figure 1 ( b ) and ( c ) . From random ensemble results in Figure 1 ( b ) , the base models are similar in performance and decision boundary thus resulting in their limited ability for fitting the complex distribution with ensemble . One question may be raised : will enforcing the base learners to predict differently on the same samples by incorporating diversity loss be sufficient to solve the above issue ? From Figure 1 ( c ) , base models ’ specialty regions for the diversity-driven ensemble are similar to that for the random ensemble , despite the fact that the prediction confidence of the base model diverges significantly across different sample regions . This suggests that diversity may not be sufficient to encourage the base model towards specialized on more samples , thus failing to improve ensemble performance . We can also observe that ensemble fails on some samples correctly predicted by a base model , for such a model we define as an unexpressed specialist . Definition 2.2 . ( Unexpressed specialist ) Given the specialized modelMk on a sample ( xn , yn ) , we defineMk as an unexpressed specialist if yn 6= argmaxc ∑K i=1 wi p ( c |xn ; θi ) where c ∈ [ 1 , C ] . Specialization-aware ensemble . In terms of diversity-driven results , the pursuit of diversity does not lead to model specialization . To verify whether a specialization-driven approach is better than the diversity-driven approach , we further use true confidence of model prediction on a sample ( i.e. , p ( c = yn|xn ; θk ) ) to explicitly quantify its specialization and conduct specialization-aware ensemble learning . As for training , we train the two base models while weighting the classification losses of each base model with the normalized true confidence . As a result , for model inference and ensemble , averaging their predictions only yields an accuracy of 71.0 % , whereas an ensemble weighted by the true confidence of the base models yields an accuracy of 100.0 % . It clearly illustrates that , as long as the specialization has been cautiously tackled , specialization-aware ensemble over weak base models will significantly improve the ensemble performance . As shown in Figure 1 ( d ) , base models ’ specialty regions overlap less , with more samples having a specialized model compared with the first two methods . Both specialization-aware training and output aggregation based on model specialty could contribute to the result that improves the accuracy from 71.0 % to 100.0 % . We verify that both factors are equally important by conducting an ablation study on the diversitydriven ensemble , where we assign the ensemble weights in ( c ) by the true confidences of each base model as we do in ( d ) . However , the test accuracy is not improved ( still 81 % ) by such a design . Therefore , ideally , our target is to have an expressed specialist for each sample by training specialized models and aggregating based on their specialty accordingly . To this end , a problem with the current specialization-driven ensemble is that the base models tend to have overconfidence in the samples they are not specialized in ( outside their known distributions ) ( Hein et al. , 2019 ) . These overconfident yet incorrect predictions tend to induce the presence of unexpressed specialists . Thus , not only do we need the models to be specialized in different specialty regions , but we also want these specialists to have a global perception to avoid overconfidence . As opposed to specialized , we also try to jointly train the sum of base models by directly minimizing the cross entropy between ground truth and the final ensemble result 1K ∑K k=1Mk ( · ) . Surprisingly , from our experiments in Section 4.3 and evidences in other study ( Allen-Zhu & Li , 2020 ) , the performance of the trained large model is even worse than that of the single model . It shows that in this way we can actually give more loss weights in back-propagation for models with higher true confidence , i.e. , it leads to specialization of models . However , without specialization-aware aggregation , the averaged ensemble result turns out to be inferior .

Summary: This paper presents a novel method for ensemble learning. By introducing a anchor scheme and specialization loss , the base learner are forced to be specialized. The method is validated on both tabular and image datasets.