Deep Ensembles with Hierarchical Diversity Pruning

1 INTRODUCTION . Deep ensembles with sufficient ensemble diversity hold potential of improving both accuracy and robustness of ensembles with their combined wisdom . The improvement can be measured by three criteria : ( i ) the average ensemble accuracy of the selected ensemble teams , ( ii ) the percentage of selected ensembles that exceed the highest accuracy of individual member models ; ( iii ) the lower bound ( worst case ) and the upper bound ( best case ) accuracy of the selected ensembles . The higher these three measures , the higher quality of the ensemble teams . Ensemble learning can be broadly classified into two categories : ( 1 ) learning the ensemble of diverse models via diversity optimized joint-training , coined as the ensemble training approach , such as boosting ( Schapire , 1999 ) ; and ( 2 ) learning to compose an ensemble of base models from a pool of existing pre-trained models through ensemble teaming based on ensemble diversity metrics ( Partridge & Krzanowski , 1997 ; Liu et al. , 2019 ; McHugh , 2012 ; Skalak , 1996 ) , coined as the ensemble consensus approach . This paper is focused on improving the state-of-the-art results in the second category . Related Work and Problem Statement . Ensemble diversity metrics are by design to capture the degree of negative correlation among the member models of an ensemble team ( Brown et al. , 2005 ; Liu et al. , 2019 ; Kuncheva & Whitaker , 2003 ) such that the high diversity indicates high negative correlation among member models of an ensemble . Three orthogonal and yet complimentary threads of efforts have been engaged in ensemble learning : ( 1 ) developing mechanisms to produce diverse base neural network models , ( 2 ) developing diversity metrics to select ensembles with high ensemble diversity from the candidate ensembles over the base model pool , and ( 3 ) developing consensus voting methods . The most popular consensus voting methods include the simple averaging , the weighted averaging , the majority voting , the plurality voting ( Ju et al. , 2017 ) , and the learn to rank ( Burges et al. , 2005 ) . For the base model selection , early efforts have been devoted to training diverse weak models to form a strong ensemble on a learning task , such as bagging ( Breiman , 1996 ) , boosting ( Schapire , 1999 ) , or different ways of selecting features , e.g. , random forests ( Tin Kam Ho , 1995 ) . Several recent studies also produce diverse base models by varying the training hyper-parameters , such as snapshot ensemble ( Huang et al. , 2017 ) , which utilizes the cyclic learning rates ( Smith , 2015 ; Wu et al. , 2019 ) to converge the single DNN model at different epochs to obtain the snapshots as the ensemble member models . Alternative method is to construct the pool of base models by using pre-trained models with different neural network backbones ( Wu et al. , 2020 ; Liu et al. , 2019 ; Wei et al. , 2020 ; Chow et al. , 2019a ) . The research efforts on diversity metrics have proposed both pairwise and non-pairwise ensemble diversity measures ( Fort et al. , 2019 ; Wu et al. , 2020 ; Liu et al. , 2019 ) , among which the three representative pairwise metrics are Cohen ’ s Kappa ( CK ) ( McHugh , 2012 ) , Q Statistics ( QS ) ( Yule , 1900 ) , Binary Disagreement ( BD ) ( Skalak , 1996 ) , and the three representative non-pairwise diversity metrics are Fleiss ’ Kappa ( FK ) ( Fleiss et al. , 2013 ) , Kohavi-Wolpert Variance ( KW ) ( Kohavi & Wolpert , 1996 ; Kuncheva & Whitaker , 2003 ) and Generalized Diversity ( GD ) ( Partridge & Krzanowski , 1997 ) . These diversity metrics are widely used in several recent studies ( Fort et al. , 2019 ; Liu et al. , 2019 ; Wu et al. , 2020 ) . Some early study has shown that these diversity metrics are correlated with respect to ensemble accuracy and diversity in the context of traditional machine learning models ( Kuncheva & Whitaker , 2003 ) . However , few studies to date have provided in-depth comparative critique on the effectiveness of these diversity metrics in pruning those low quality deep ensembles from the candidate ensembles due to their high negative correlation . Scope and Contributions . In this paper , we focus on the problem of defining ensemble diversity metrics that can select diverse ensemble teams with high ensemble accuracy . We first investigate the six representative ensemble diversity metrics , coined as Q metrics . We identify and analyze their inherent limitations in capturing the negative correlation among the member models of an ensemble , and why pruning out those deep ensembles with low Q-diversity may not always guarantee to improve the ensemble accuracy . To address the inherent problems of Q metrics , we extend the existing six Q metrics with three optimizations : ( 1 ) We introduce the concept of the focal model and argue that one way to better capture the negative correlations among member models of an ensemble is to compute diversity scores for ensembles of fixed size based on the focal model . ( 2 ) We introduce the six HQ diversity metrics to optimize the six Q-diversity metrics respectively . ( 3 ) We develop a HQbased hierarchical pruning method , consisting of two stage pruning : the α filter and the K-Means filter . By combining these optimizations , the deep ensembles selected by our HQ-metrics can significantly outperform those deep ensembles selected by the corresponding Q metrics , showing that the HQ metrics based hierarchical pruning approach is efficient in identification and removal of low diversity deep ensembles . Comprehensive experiments are conducted on three benchmark datasets : CIFAR-10 ( Krizhevsky & Hinton , 2009 ) , ImageNet ( Russakovsky et al. , 2015 ) , and Cora ( Lu & Getoor , 2003 ) . The results show that our hierarchical diversity pruning approach outperforms their corresponding Q-metrics in terms of the lower bound and the upper bound of ensemble accuracy over the deep ensembles selected , exhibiting the effectiveness of our HQ approach in pruning low diversity deep ensembles . 2 HIERARCHICAL PRUNING WITH DIVERSITY METRICS . Existing studies on consensus based ensemble learning ( Huang et al. , 2017 ; Krizhevsky et al. , 2012 ; Zoph & Le , 2016 ) generate the base model pool through two channels : ( i ) deep neural network training using different network structures or different configurations of hyperparameters ( Breiman , 1996 ; Schapire , 1999 ; Zoph & Le , 2016 ; Hinton et al. , 2015 ; Wu et al. , 2018 ; 2019 ) and ( ii ) selecting the top performing pre-trained models from open-source projects ( e.g. , GitHub ) and public model zoos ( Jia et al. , 2014 ; ONNX Developers , 2020 ; GTModelZoo Developers , 2020 ) . Hence , an important technical challenge for deep ensemble learning is to define diversity metrics for producing high quality ensemble teaming strategies , aiming to boost the ensemble accuracy . Given that the number of possible ensemble teams increases exponentially with a small pool of base models , de- veloping proper ensemble diversity metrics is critical for effective pruning of deep ensembles with insufficient diversity . Consider a pool of M base models for a learning task on a given dataset D , denoted by BMSet ( D ) = { F1 , ... , FM } . Let EnsSet denote the set of all possible ensemble teams that are composed from BMSet ( D ) , with the ensemble team size S varying from 2 to M . We have a total of ∑M S=2 ( M S ) ensembles , i.e. , |EnsSet| = ( M 2 ) + ( M 3 ) + ... + ( M M ) = 2M − ( 1 + M ) . The cardinality of the set of possible ensembles EnsSet grows exponentially with M , the number of base models . For example , M = 3 , we have |EnsSet| = 4 . When M becomes larger , such as M = 5 , 10 , 20 , |EnsSet| = 26 , 1013 , 1048555 . Hence , asM increases , it is non-trivial to construct a set of high-accuracy ensemble teams ( GEnsSet ) , from the candidate set ( EnsSet ) of all possible ensembles that are composed from BMSet ( D ) . Consider a pool of M = 10 base models for ImageNet , in which the highest performing base model is 78.25 % , the lowest performing base model is 56.63 % , and the average accuracy of these 10 base models is 71.60 % ( see Table 5 in Appendix Section F ) . For a pool of 10 base models , there will be a total of 1013 ( 210 − ( 10 + 1 ) ) different ensembles with team size ranging from 2 to 10 . The performance of these ensembles vary sharply , from 61.39 % ( lower bound ) to 80.77 % ( upper bound ) . Randomly selecting an ensemble team from these 1013 teams in EnsSet ( ImageNet ) may lead to a non-trivial probability of selecting a team with the ensemble accuracy lower than the average member model accuracy of 71.60 % over the 10 base models . Clearly , an efficient ensemble diversity metric should be able to prune out those ensemble teams with insufficient ensemble diversity and thus low ensemble accuracy , increasing ( i ) the average ensemble accuracy of the selected ensemble teams , ( ii ) the percentage of selected ensembles that exceed the highest accuracy of individual member models ( i.e. , 78.25 % for the 10 base DNN models on ImageNet ) ; and ( iii ) the lower bound ( worst case ) and the upper bound ( best case ) accuracy of the selected ensembles . A number of ensemble diversity metrics have been proposed to address this challenging problem . In this section , we first provide a comparative study of the six state-of-the-art Q-diversity metrics and analyze their inherent limitations in identifying and pruning out low diversity ensembles . Then we introduce our proposed HQ-diversity metrics and analyze the effectiveness of our HQ based hierarchical diversity approach in pruning low quality ensembles . 2.1 Q-DIVERSITY METRICS AND THEIR LIMITATIONS . We outline the key notations for the six Q-diversity metrics in Table 1 : three pairwise diversity metrics : Cohen ’ s Kappa ( CK ) ( McHugh , 2012 ) , Q Statistics ( QS ) ( Yule , 1900 ) and Binary Disagreement ( BD ) ( Skalak , 1996 ) , and three non-pairwise diversity metrics : Fleiss ’ Kappa ( Fleiss et al. , 2013 ) ( FK ) , Kohavi-Wolpert Variance ( KW ) ( Kohavi & Wolpert , 1996 ; Kuncheva & Whitaker , 2003 ) and Generalized Diversity ( GD ) ( Partridge & Krzanowski , 1997 ) . The arrow column ↑ | ↓ specifies the relationship between the Q-value and the ensemble diversity . The ↑ represents positive relationship of the Q-value and the ensemble diversity , that is a high Q-value refers to high ensemble diversity . The ↓ indicates the negative relationship , that is the low Q-value corresponds to high ensemble diversity . To facilitate the comparison of the six Q-diversity metrics such that the low Q-value refers to high ensemble diversity for all six Q-metrics , we apply ( 1−Q-value ) when calculating Q-diversity score with BD , KW and GD . We refer readers to Appendix ( Section C ) for the formal definitions of the six Q-diversity metrics . Given a Q-diversity metric , we calculate the diversity score for each ensemble team in the ensemble set ( EnsSet ) using a set of negative samples ( NegSampSet ) on which one or more models in the ensemble make prediction errors . The low Q-score indicates sufficient diversity among member models of an ensemble . Upon the completion of Q-diversity score computation for all ensembles in EnsSet , the diversity threshold based pruning is employed to remove those ensembles with insufficient diversity among ensemble member models . Either a pre-defined Q-diversity threshold or a mean threshold by taking the average value of all Q-diversity scores calculated for all candidate deep ensembles in EnsSet . The mean threshold tends to work better in general than a manually defined threshold . Once a mean threshold is obtained , those ensembles in EnsSet with their Q diversity scores below the threshold will be selected and placed into the diverse ensemble set GEnsSet , and the remaining ensembles are those with their Q scores higher than the threshold and thus will be pruned out . The pseudo code of the algorithm is included in Appendix ( Algorithm 1 ) . The last three columns of Table 1 show the mean threshold for all six Q-diversity metrics calculated on the set of 1013 candidate deep ensembles for the three benchmark datasets used in this study . We make two observations . First , different Q-diversity metrics capture the ensemble diversity from different perspectives with different diversity measurement principles , resulting in different Q-scores . Second , each Q-metric , say CK , is used to compare ensembles based on their Q-CK scores . Hence , even though the Q-KW metric has relatively high KW-specific Q scores for all ensemble teams , it can select the diverse ensembles based on the mean KW-threshold , in a similar manner as any of the other five Q metrics . Limitations of Q Metrics . Figure 1a and 1b show Q-KW and Q-GD metrics and their relationship with ensemble accuracy for all 1013 deep ensembles on CIFAR-10 respectively . Each dot represents a deep ensemble team with team sizes color-coded by the color diagram on the right . The vertical red dashed line represents the Q-KW and Q-GD mean thresholds of 0.868 and 0.476 respectively . The horizontal red and black dashed lines represent the maximum single model accuracy 96.68 % and the average accuracy 94.25 % of the 10 base models respectively . We use these two accuracy bounds as important references to quantify the quality of the deep ensembles selected using a Q metric with its mean threshold . Those deep ensembles on the left of the red vertical dash line are selected and added into GEnsSet given that their Q-scores are below the mean threshold ( e.g. , QKW or Q-GD ) . The ensembles on the right of this red vertical dash line are pruned out because their Q diversity scores exceed the mean threshold . Compare Figure 1a and 1b , it is visually clear that both Q metrics can select a sufficient number of good quality ensemble teams while at the same time , both Q metrics with mean threshold pruning will miss a large portion of teams with high ensemble accuracy , indicating the inherent limitations of both Q metrics and the mean threshold pruning with respect to capturing the concrete ensemble diversity in terms of low negative correlation among member models of an ensemble . To better understand the inherent problems with the Q-diversity metrics , we performed another set of experiments by measuring the Q-GD metric over ensemble teams of fixed size S on CIFAR-10 . Figure 1c shows a visualization of the results using the Q-GD scores computed over ensembles of size S = 4 with mean threshold indicated by the vertical red dashed line , showing a visually sharper trend in terms of the relationship between ensemble diversity and ensemble accuracy when comparing the selected ensemble teams ( red dots ) with those ensembles ( black dots ) on the right of the red vertical threshold line . However , relying on separating the diversity computation and comparison over ensemble teams of the same size alone may not be sufficient , because Figure 1c shows that ( i ) some selected ensemble teams have low accuracy , affecting all three ensemble quality measures ( recall Section 2 , page 3 ) , and ( ii ) a fair number of ensemble teams with high ensemble accuracy ( black dots on the top right side ) are still missed . Similar observations are also found for other five Q-diversity metrics . We conclude our analysis with three arguments : ( 1 ) The Qdiversity metrics may not accurately capture the degree of negative correlation among the member models of an ensemble even when its ensemble Q-diversity score is below the mean threshold . ( 2 ) Comparing ensembles of different team size S using their Q scores may not be a fair measure of their true ensemble diversity in terms of the degree of negative correlation among member models of an ensemble . However , relying on ensembles of the same team size S alone is still insufficient . ( 3 ) Mean threshold is not a good Q-diversity pruning method in terms of capturing the intrinsic relationship between ensemble diversity and ensemble accuracy . This motivates us to propose the HQ diversity metrics with two phase pruning using learning algorithms .

Summary: The paper succeeds in developing diversity metrics that correlate better with ensemble accuracy than the original diversity metrics. However, this makes one wonder why one cannot just use ensemble accuracy directly. One can also use a combining scheme along the lines of (Freund, 1995) where it adds models that focus on the examples that will increase accuracy and allowing errors on examples where most of the models so far have either classified the examples correctly already or incorrectly (where there is no hope of recovery and so effort is not worthwhile). Additionally, the appendix has the algorithms and other substantive content that is central to the paper, which is not supposed to be the case.