SANE: Specialization-Aware Neural Network Ensemble

1 INTRODUCTION . Real-world data distribution could be complex in most cases and people usually approximate it by a composition of several simpler distributions ( Xie et al. , 2016 ; Yang et al. , 2016 ; Tsai et al. , 2020 ) . Intuitively , to fit the complex data distribution conveniently , we can divide the complex model learning process into training several simpler models , each of which specializes in one simple distribution . In this paper , we call this idea model specialization . Ensemble learning is one of the most effective methods to leverage model specialization . Originated from decades ago ( Hansen & Salamon , 1990 ) , ensemble method ( Schapire , 1990 ; Breiman , 1996 ; Zhou et al. , 2002 ) has been proven effective for practical machine learning tasks in various scenarios including computer vision ( Huang et al. , 2017 ) , natural language processing ( Shazeer et al. , 2017 ) and tabular data mining ( Liu et al. , 2020 ) . Many works conduct ensemble learning through improving diversity across base models , which has been shown to reduce the variance of the combined predictions and improve generalization ( Lee et al. , 2015 ; Zhou et al. , 2018 ) . The recent methodologies pursuing model diversity mainly through incorporating randomization in data sampling ( Breiman , 1996 ; Ho , 1995 ) or model training ( Srivastava et al. , 2014 ; Lakshminarayanan et al. , 2016 ) . Some other works propose sequential optimization such as boosting ( Freund , 1995 ; Chen & Guestrin , 2016 ; Ke et al. , 2017 ) and snapshot ensemble ( Huang et al. , 2017 ) . Other methods , e.g. , model decorrelation ( Zhou et al. , 2018 ) and diversity enhancement ( Zhang et al. , 2020 ) , explicitly encourage model diversity through incorporating additional objectives . However , overvaluing diversity may hurt the ensemble effectiveness and the trade-off between the performance and diversity of base models still remains an open problem ( Rame & Cord , 2021 ; Fort et al. , 2019 ; Masegosa , 2019 ) . Moreover , without explicitly enforcing model specialization , pursuing model diversity does not provide guarantees of each model ’ s specialization on different simple distribution in the composition . The last two columns in Figure 1 visualize the specialty of each model by plotting the data regions where the base model makes correct predictions . It shows that neither the ensemble of ( b ) randomly initialized base models nor ( c ) diversified base models produce models with different specialties . Specifically , although the method in ( c ) enforces diversity and shows low correlation among base models ’ outputs , their data regions with correct predictions are very similar as shown in the last two columns in Figure 1 , resulting in inferior performance after ensemble . Besides , simply putting specialized base models together does not necessarily bring promising ensemble performance . We need to know each base model ’ s specialty over different samples and aggregate their outputs based on their specialty . As shown in Figure 1 ( d ) , although the overall accuracy of the two base models are low , the ensemble model performs surprisingly well ( 100 % accuracy in ( d ) ) as long as 1 ) each base model specializes in unique sub-regions of the latent space and 2 ) the ensemble model knows which sub-regions each base model specializes in , demonstrating the effectiveness of specialization in ensemble learning . More detailed analyses of Figure 1 are presented in Section 2 . In this paper , we propose an end-to-end ensemble learning method that actively enforces model specialization by training models to specialize in specific simple distributions , and aggregates them based on their specialties . Firstly , we represent the composition of simple distributions by mapping them into a unified latent space , where samples from each data distribution form a locally clustered sub-region . This paper proposes to enforce each base model to specialize in one sub-region , and we call this region the base model ’ s corresponding specialty region . The base models ’ specialty regions are defined by a set of learnable anchor points , each of which corresponds to the center of the specialty region , called specialty anchor . During training , base models ’ parameters and their corresponding specialty anchors are learned simultaneously . Each base model ’ s specialty is enforced by encouraging model to focus more on samples in the vicinity of its specialty anchor . Finally , during prediction , based on the correlation between models ’ specialty anchors and the samples in the latent space , our method is able to automatically estimate each base model ’ s specialty over testing samples , and the outputs of the base models are adaptively aggregated based on their specialty over different testing samples . To demonstrate the effectiveness of the proposed method , we conduct extensive experiments in prediction tasks on two tabular datasets and two image classification datasets . The comprehensive empirical studies show that 1 ) coupling specialized base models and specializationaware aggregation can benefit ensemble learning and 2 ) the proposed specialization-aware neural network ensemble method can achieve superior performance compared with the state-of-the-art ensemble methods in various prediction tasks . 2 PRELIMINARIES AND SYNTHETIC ANALYSIS . 2.1 PRELIMINARIES . In general , ensemble learning consists of two tasks : training multiple base models and aggregating their predictions . We use { Mk } Kk=1 to denote a collection of K base models . For a given instance xn ∈ Rd and its label yn ∈ [ 1 , C ] , each base model learns a mapping Mk : Rd 7−→ RC , which produces logits for C classes . Base models ’ predictions are then aggregated in a weighted manner . The likelihood is written as p ( yn |xn ) = ∑K k=1 wk p ( yn |xn ; θk ) s.t . ∑K k=1 wk = 1 , whereMk is parameterized by θk and wk denotes the weight for aggregating k-th model prediction . Objectives of ensemble learning . To maximize the likelihood , most previous studies attempt to promote ∑K k=1 p ( yn |xn ; θk ) ( true confidence ofMk , for brevity ) , with fixed model weights on all samples . To this end , some works train models based on random initialization to reduce prediction variance and generalization errors , while others explicitly quantify diversity to reduce the correlation of errors between models . However , there is always a problem about the trade-off between diversity and accuracy in this scenario , since the weighted majority correctness among selected models determines the correctness of the ensemble . One way to avoid the conflict between adopting diverse models and boosting the overall performance is to use sample-specific model weights to aggregate models ’ predictions . Intuitively , if a base model performs much better than any other base models over a specific sub-region of the dataset containing one or more samples ( i.e. , a base model specializes in a sub-set of the dataset ) , it should be assigned with higher weight in this sub-region . As a result , we define the specialized model as follows . Definition 2.1 . ( Specialized model ) We define the specialized modelMk of a sample ( xn , yn ) with two aspects : ( 1 ) k = argmaxk′ p ( yn |xn ; θk′ ) , and ( 2 ) yn = argmaxc p ( c |xn ; θk ) where c ∈ [ 1 , C ] denotes the class label . If no model satisfies the above conditions , then xn is not specialized by any model . That is , the specialized model is the one that works correctly and has the highest prediction confidence on a given sample among all the base models . 2.2 SYNTHETIC ANALYSIS . In this section , we conduct the experiment with three ensemble solutions including a random initialization based method and a diversity-based ensemble method , and then we illustrate how specialization can benefit ensemble . A 5× 5 checkerboard synthetic dataset containing 25 Gaussian blobs shown in Figure 1 ( a ) is used for analysis . Two widely used ensemble methods are selected for this experiment . Firstly , following the classical random ensemble , base models with different weight initialization are trained independently and aggregated averagely . On the basis of the random ensemble , we further implement a diversitydriven ensemble by incorporating a diversity loss ( Li et al. , 2012 ) into the error function of each base model to encourage base models to perform differently on the same sample . To illustrate how different base models and their ensembles make predictions , we draw their decision boundaries and graphs showing their individual specialty regions , which for a single model is where it makes the correct prediction . A multi-layer perceptron ( MLP ) with one hidden layer and 20 neurons serves as the base model of ensembles through this section . For simplicity of visualization and analysis , we maintain two base models for each ensemble method . Random ensemble v.s . diversity-driven ensemble . The results of these two methods are , respectively , shown in Figure 1 ( b ) and ( c ) . From random ensemble results in Figure 1 ( b ) , the base models are similar in performance and decision boundary thus resulting in their limited ability for fitting the complex distribution with ensemble . One question may be raised : will enforcing the base learners to predict differently on the same samples by incorporating diversity loss be sufficient to solve the above issue ? From Figure 1 ( c ) , base models ’ specialty regions for the diversity-driven ensemble are similar to that for the random ensemble , despite the fact that the prediction confidence of the base model diverges significantly across different sample regions . This suggests that diversity may not be sufficient to encourage the base model towards specialized on more samples , thus failing to improve ensemble performance . We can also observe that ensemble fails on some samples correctly predicted by a base model , for such a model we define as an unexpressed specialist . Definition 2.2 . ( Unexpressed specialist ) Given the specialized modelMk on a sample ( xn , yn ) , we defineMk as an unexpressed specialist if yn 6= argmaxc ∑K i=1 wi p ( c |xn ; θi ) where c ∈ [ 1 , C ] . Specialization-aware ensemble . In terms of diversity-driven results , the pursuit of diversity does not lead to model specialization . To verify whether a specialization-driven approach is better than the diversity-driven approach , we further use true confidence of model prediction on a sample ( i.e. , p ( c = yn|xn ; θk ) ) to explicitly quantify its specialization and conduct specialization-aware ensemble learning . As for training , we train the two base models while weighting the classification losses of each base model with the normalized true confidence . As a result , for model inference and ensemble , averaging their predictions only yields an accuracy of 71.0 % , whereas an ensemble weighted by the true confidence of the base models yields an accuracy of 100.0 % . It clearly illustrates that , as long as the specialization has been cautiously tackled , specialization-aware ensemble over weak base models will significantly improve the ensemble performance . As shown in Figure 1 ( d ) , base models ’ specialty regions overlap less , with more samples having a specialized model compared with the first two methods . Both specialization-aware training and output aggregation based on model specialty could contribute to the result that improves the accuracy from 71.0 % to 100.0 % . We verify that both factors are equally important by conducting an ablation study on the diversitydriven ensemble , where we assign the ensemble weights in ( c ) by the true confidences of each base model as we do in ( d ) . However , the test accuracy is not improved ( still 81 % ) by such a design . Therefore , ideally , our target is to have an expressed specialist for each sample by training specialized models and aggregating based on their specialty accordingly . To this end , a problem with the current specialization-driven ensemble is that the base models tend to have overconfidence in the samples they are not specialized in ( outside their known distributions ) ( Hein et al. , 2019 ) . These overconfident yet incorrect predictions tend to induce the presence of unexpressed specialists . Thus , not only do we need the models to be specialized in different specialty regions , but we also want these specialists to have a global perception to avoid overconfidence . As opposed to specialized , we also try to jointly train the sum of base models by directly minimizing the cross entropy between ground truth and the final ensemble result 1K ∑K k=1Mk ( · ) . Surprisingly , from our experiments in Section 4.3 and evidences in other study ( Allen-Zhu & Li , 2020 ) , the performance of the trained large model is even worse than that of the single model . It shows that in this way we can actually give more loss weights in back-propagation for models with higher true confidence , i.e. , it leads to specialization of models . However , without specialization-aware aggregation , the averaged ensemble result turns out to be inferior .

Summary: This paper presents a SANE model to improve the ensemble learning from the perspective of model specialization. In particular, it first gives the analysis on existing ensemble strategies, i.e., random and diversity-driven, via conducting experiments on synthetic data. By this way, it points out the weakness of these strategies lies in the lack of specialty. Motivated by this, this paper presents the specialization-aware method to improve the ensemble learning. Specifically, it introduces anchor points in the latent space for guiding the model learning towards specialty. To derive the correlation between samples and anchors, it utilizes the transformer-like attention mechanism for learning the weights of base models. This paper demonstrates its effectiveness on several image and tabular benchmarks.