DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation

Deep ensembles perform better than a single network thanks to the diversity among their members . Recent approaches regularize predictions to increase diversity ; however , they also drastically decrease individual members ’ performances . In this paper , we argue that learning strategies for deep ensembles need to tackle the trade-off between ensemble diversity and individual accuracies . Motivated by arguments from information theory and leveraging recent advances in neural estimation of conditional mutual information , we introduce a novel training criterion called DICE : it increases diversity by reducing spurious correlations among features . The main idea is that features extracted from pairs of members should only share information useful for target class prediction without being conditionally redundant . Therefore , besides the classification loss with information bottleneck , we adversarially prevent features from being conditionally predictable from each other . We manage to reduce simultaneous errors while protecting class information . We obtain state-of-the-art accuracy results on CIFAR-10/100 : for example , an ensemble of 5 networks trained with DICE matches an ensemble of 7 networks trained independently . We further analyze the consequences on calibration , uncertainty estimation , out-of-distribution detection and online co-distillation . 1 INTRODUCTION . Averaging the predictions of several models can significantly improve the generalization ability of a predictive system . Due to its effectiveness , ensembling has been a popular research topic ( Nilsson , 1965 ; Hansen & Salamon , 1990 ; Wolpert , 1992 ; Krogh & Vedelsby , 1995 ; Breiman , 1996 ; Dietterich , 2000 ; Zhou et al. , 2002 ; Rokach , 2010 ; Ovadia et al. , 2019 ) as a simple alternative to fully Bayesian methods ( Blundell et al. , 2015 ; Gal & Ghahramani , 2016 ) . It is currently the de facto solution for many machine learning applications and Kaggle competitions ( Hin , 2020 ) . Ensembling reduces the variance of estimators ( see Appendix E.1 ) thanks to the diversity in predictions . This reduction is most effective when errors are uncorrelated and members are diverse , i.e. , when they do not simultaneously fail on the same examples . Conversely , an ensemble of M identical networks is no better than a single one . In deep ensembles ( Lakshminarayanan et al. , 2017 ) , the weights are traditionally trained independently : diversity among members only relies on the randomness of the initialization and of the learning procedure . Figure 1 shows that the performance of this procedure quickly plateaus with additional members . To obtain more diverse ensembles , we could adapt the training samples through bagging ( Breiman , 1996 ) and bootstrapping ( Efron & Tibshirani , 1994 ) , but a reduction of training samples has a negative impact on members with multiple local minima ( Lee et al. , 2015 ) . Sequential boosting does not scale well for time-consuming deep learners that overfit their training dataset . Liu & Yao ( 1999a ; b ) ; Brown et al . ( 2005b ) explicitly quantified the diversity and regularized members into having negatively correlated errors . However , these ideas have not significantly improved accuracy when applied to deep learning ( Shui et al. , 2018 ; Pang et al. , 2019 ) : while members should predict the same target , they force disagreements among strong learners and therefore increase their bias . It highlights the main objective and challenge of our paper : finding a training strategy to reach an improved trade-off between ensemble diversity and individual accuracies ( Masegosa , 2020 ) . one input ( , ) should not share more information than features from two inputs in the same class ( , ) : i.e. , ( , - ) should not be able to differentiate ( - , ) and ( - , ) . Our core approach is to encourage all members to predict the same thing , but for different reasons . Therefore the diversity is enforced in the features space and not on predictions . Intuitively , to maximize the impact of a new member , extracted features should bring information about the target that is absent at this time so unpredictable from other members ’ features . It would remove spurious correlations , e.g . information redundantly shared among features extracted by different members but useless for class prediction . This redundancy may be caused by a detail in the image background and therefore will not be found in features extracted from other images belonging to the same class . This could make members predict badly simultaneously , as shown in Figure 2 . Our new learning framework , called DICE , is driven by Information Bottleneck ( IB ) ( Tishby , 1999 ; Alemi et al. , 2017 ) principles , that force features to be concise by forgetting the task-irrelevant factors . Specifically , DICE leverages the Minimum Necessary Information criterion ( Fischer , 2020 ) for deep ensembles , and aims at reducing the mutual information ( MI ) between features and inputs , but also information shared between features . We prevent extracted features from being redundant . As mutual information can detect arbitrary dependencies between random variables ( such as symmetry , see Figure 2 ) , we increase the distance between pairs of members : it promotes diversity by reducing predictions ’ covariance . Most importantly , DICE protects features ’ informativeness by conditioning mutual information upon the target . We build upon recent neural approaches ( Belghazi et al. , 2018 ) based on the Donsker-Varadhan representation of the KL formulation of MI . We summarize our contributions as follows : • We introduce DICE , a new adversarial learning framework to explicitly increase diversity in ensemble by minimizing the conditional redundancy between features . • We rationalize our training objective by arguments from information theory . • We propose an implementation through neural estimation of conditional redundancy . We consistently improve accuracy on CIFAR-10/100 as summarized in Figure 1 , with better uncertainty estimation and calibration . We analyze how the two components of our loss modify the accuracy-diversity trade-off . We improve out-of-distribution detection and online co-distillation . 2 DICE MODEL . Notations Given an input distribution X , a network θ is trained to extract the best possible dense features Z to model the distribution pθ ( Y |X ) over the targets , which should be close to the Dirac on the true label . Our approach is designed for ensembles with M members θi , i ∈ { 1 , . . . , M } extracting Zi . In branch-based setup , members share low-level weights to reduce computation cost . We average the M predictions in inference . We initially consider an ensemble of M = 2 members . Quick overview First , we train each member separately for classification with information bottleneck . Second , we train members together to remove spurious redundant correlations while training adversarially a discriminator . In conclusion , members learn to classify with conditionally uncorrelated features for increased diversity . Our procedure is driven by the following theoretical findings . 2.A DERIVING TRAINING OBJECTIVE 2.A.1 BASELINE : NON-CONDITIONAL OBJECTIVE The Minimum Necessary Information ( MNI ) criterion from ( Fischer , 2020 ) aims at finding minimal statistics . In deep ensembles , Z1 and Z2 should capture only minimal information from X , while preserving the necessary information about the task Y . First , we consider separately the two Markov chains Z1 ← X ↔ Y and Z2 ← X ↔ Y . As entropy measures information , entropy of Z1 and Z2 not related to Y should be minimized . We recover IB ( Alemi et al. , 2017 ) in deep ensembles : IBβib ( Z1 , Z2 ) = 1 βib [ I ( X ; Z1 ) + I ( X ; Z2 ) ] − [ I ( Y ; Z1 ) + I ( Y ; Z2 ) ] = IBβib ( Z1 ) + IBβib ( Z2 ) . Second , let ’ s consider I ( Z1 ; Z2 ) : we minimize it following the minimality constraint of the MNI . IBRβib , δr ( Z1 , Z2 ) = 1 βib Compression︷ ︸︸ ︷ [ I ( X ; Z1 ) + I ( X ; Z2 ) ] − Relevancy︷ ︸︸ ︷ [ I ( Y ; Z1 ) + I ( Y ; Z2 ) ] +δr Redundancy︷ ︸︸ ︷ I ( Z1 ; Z2 ) = IBβib ( Z1 ) + IBβib ( Z2 ) + δrI ( Z1 ; Z2 ) . ( green vertical stripes ) with no overlap with relevancy ( red stripes ) . Analysis In this baseline criterion , relevancy encouragesZ1 and Z2 to capture information about Y . Compression & redundancy ( R ) split the information from X into two compressed & independent views . The relevancy-compressionredundancy trade-off depends on the values of βib & δr . 2.A.2 DICE : CONDITIONAL OBJECTIVE The problem is that the compression and redundancy terms in IBR also reduce necessary information related to Y : it is detrimental to have Z1 and Z2 fully disentangled while training them to predict the same Y . As shown on Figure 3 , redundancy regions ( blue horizontal stripes ) overlap with relevancy regions ( red stripes ) . Indeed , the true constraints that the MNI criterion really entails are the following conditional equalities given Y : I ( X ; Z1|Y ) = I ( X ; Z2|Y ) = I ( Z1 ; Z2|Y ) = 0 . Mutual information being non-negative , we transform them into our main DICE objective : DICEβceb , δcr ( Z1 , Z2 ) = 1βceb [ I ( X ; Z1|Y ) + I ( X ; Z2|Y ) ] ︸ ︷︷ ︸ Conditional Compression − [ I ( Y ; Z1 ) + I ( Y ; Z2 ) ] ︸ ︷︷ ︸ Relevancy +δcr I ( Z1 ; Z2|Y ) ︸ ︷︷ ︸ Conditional Redundancy = CEBβceb ( Z1 ) + CEBβceb ( Z2 ) + δcrI ( Z1 ; Z2|Y ) , ( 1 ) where we recover two conditional entropy bottleneck ( CEB ) ( Fischer , 2020 ) components , CEBβceb ( Zi ) = 1 βceb I ( X ; Zi|Y ) − I ( Y ; Zi ) , with βceb > 0 and δcr > 0 . Analysis The relevancy terms force features to be informative about the task Y . But contrary to IBR , DICE bottleneck constraints only minimize irrelevant information to Y . First , the conditional compression removes in Z1 ( or Z2 ) information from X not relevant to Y . Second , the conditional redundancy ( CR ) reduces spurious correlations between members and only forces them to have independent bias , but definitely not independent features . It encourages diversity without affecting members ’ individual precision as it protects information related to the target class in Z1 and Z2 . Useless information from X to predict Y should certainly not be in Z1 or Z2 , but it is even worse if they are in Z1 and Z2 simultaneously as it would cause simultaneous errors . Even if for i ∈ { 1 , 2 } , reducing I ( Zi , X|Y ) indirectly controls I ( Z1 , Z2|Y ) ( as I ( Z1 ; Z2|Y ) ≤ I ( X ; Zi|Y ) by chain rule ) , it is more efficient to directly target this intersection region through the CR term . In a final word , DICE is to IBR for deep ensembles as CEB is to IB for a single network . We now approximate the two CEB and the CR components in DICE objective from equation 1 . 2.B APPROXIMATING DICE INTO A TRACTABLE LOSS 2.B.1 VARIATIONAL APPROXIMATION OF CONDITIONAL ENTROPY BOTTLENECK We leverage Markov assumptions in Zi ← X ↔ Y , i ∈ { 1 , 2 } and empirically estimate on the classification training dataset of N i.i.d . points D = { xn , yn } Nn=1 , yn ∈ { 1 , . . . , K } . Following Fischer ( 2020 ) , CEBβceb ( Zi ) = 1 βceb I ( X ; Zi|Y ) − I ( Y ; Zi ) is variationally upper bounded by : VCEBβceb ( { ei , bi , ci } ) = 1 N N∑ n=1 1 βceb DKL ( ei ( z|xn ) ‖bi ( z|yn ) ) − E [ log ci ( yn|ei ( xn , ) ) ] . ( 2 ) See explanation in Appendix E.4 . ei ( z|x ) is the true features distribution generated by the encoder , ci ( y|z ) is a variational approximation of true distribution p ( y|z ) by the classifier , and bi ( z|y ) is a variational approximation of true distribution p ( z|y ) by the backward encoder . This loss is applied separately on each member θi = { ei , ci , bi } , i ∈ { 1 , 2 } . Practically , we parameterize all distributions with Gaussians . The encoder ei is a traditional neural network features extractor ( e.g . ResNet-32 ) that learns distributions ( means and covariances ) rather than deterministic points in the features space . That ’ s why ei transforms an image into 2 tensors ; a features-mean eµi ( x ) and a diagonal features-covariance e σ i ( x ) each of size d ( e.g . 64 ) . The classifier ci is a dense layer that transforms a features-sample z into logits to be aligned with the target y through conditional cross entropy . z is obtained via reparameterization trick : z = ei ( x , ) = eµi ( x ) + e σ i ( x ) with ∼ N ( 0 , 1 ) . Finally , the backward encoder bi is implemented as an embedding layer of size ( K , d ) that maps the K classes to class-features-means bµi ( z|y ) of size d , as we set the class-features-covariance to 1 . The Gaussian parametrization also enables the exact computation of the DKL ( see Appendix E.3 ) , that forces ( 1 ) features-mean e µ i ( x ) to converge to the class-featuresmean bµi ( z|y ) and ( 2 ) the predicted features-covariance eσi ( x ) to be close to 1 . The advantage of VCEB versus VIB ( Alemi et al. , 2017 ) is the class conditional bµi ( z|y ) versus non-conditional bµi ( z ) which protects class information . 2.B.2 ADVERSARIAL ESTIMATION OF CONDITIONAL REDUNDANCY Theoretical Problem We now focus on estimating I ( Z1 ; Z2|Y ) , with no such Markov properties . Despite being a pivotal measure , mutual information estimation historically relied on nearest neighbors ( Singh et al. , 2003 ; Kraskov et al. , 2004 ; Gao et al. , 2018 ) or density kernels ( Kandasamy et al. , 2015 ) that do not scale well in high dimensions . We benefit from recent advances in neural estimation of mutual information ( Belghazi et al. , 2018 ) , built on optimizing Donsker & Varadhan ( 1975 ) dual representations of the KL divergence . Mukherjee et al . ( 2020 ) extended this formulation for conditional mutual information estimation . CR = I ( Z1 ; Z2|Y ) = DKL ( P ( Z1 , Z2 , Y ) ‖P ( Z1 , Y ) p ( Z2|Y ) ) = sup f Ex∼p ( z1 , z2 , y ) [ f ( x ) ] − log ( Ex∼p ( z1 , y ) p ( z2|y ) [ exp ( f ( x ) ) ] ) = Ex∼p ( z1 , z2 , y ) [ f∗ ( x ) ] − log ( Ex∼p ( z1 , y ) p ( z2|y ) [ exp ( f∗ ( x ) ) ] ) , where f∗ computes the pointwise likelihood ratio , i.e. , f∗ ( z1 , z2 , y ) = p ( z1 , z2 , y ) p ( z1 , y ) p ( z2|y ) . Empirical Neural Estimation We estimate CR ( 1 ) using the empirical data distribution and ( 2 ) replacing f∗ = w ∗ 1−w∗ by the output of a discriminator w , trained to imitate the optimal w ∗ . Let B be a batch sampled from the observed joint distribution p ( z1 , z2 , y ) = p ( e1 ( z|x ) , e2 ( z|x ) , y ) ; we select the features extracted by the two members from one input . Let Bp be sampled from the product distribution p ( z1 , y ) p ( z2|y ) = p ( e1 ( z|x ) , y ) p ( z2|y ) ; we select the features extracted by the two members from two different inputs that share the same class . We train a multi-layer network w on the binary task of distinguishing these two distributions with the standard cross-entropy loss : Lce ( w ) = − 1 |B|+ |Bp| ∑ ( z1 , z2 , y ) ∈B logw ( z1 , z2 , y ) + ∑ ( z1 , z′2 , y ) ∈Bp log ( 1− w ( z1 , z′2 , y ) ) . ( 3 ) If w is calibrated ( see Appendix B.3 ) , a consistent ( Mukherjee et al. , 2020 ) estimate of CR is : ÎCRDV = 1 |B| ∑ ( z1 , z2 , y ) ∈B log f ( z1 , z2 , y ) ︸ ︷︷ ︸ Diversity − log 1 |Bp| ∑ ( z1 , z′2 , y ) ∈Bp f ( z1 , z ′ 2 , y ) ︸ ︷︷ ︸ Fake correlations , with f = w 1− w . Intuition By training our members to minimize ÎCRDV , we force triples from the joint distribution to be indistinguishable from triples from the product distribution . Let ’ s imagine that two features are conditionally correlated , some spurious information is shared between features only when they are from the same input and not from two inputs ( from the same class ) . This correlation can be informative about a detail in the background , an unexpected shape in the image , that is rarely found in samples from this input ’ s class . In that case , the product and joint distributions are easily distinguishable by the discriminator . The first adversarial component will force the extracted features to reduce the correlation , and ideally one of the two features loses this information : it reduces redundancy and increases diversity . The second term would create fake correlations between features from different inputs . As we are not interested in a precise estimation of the CR , we get rid of this second term that , empirically , did not increase diversity , as detailed in Appendix G. L̂CRDV ( e1 , e2 ) = 1 |B| ∑ ( z1 , z2 , y ) ∈B∼p ( e1 ( z|x ) , e2 ( z|x ) , y ) log f ( z1 , z2 , y ) . ( 4 ) Summary First , we train each member for classification with VCEB from equation 2 , as shown in Step 1 from Figure 4 . Second , as shown in Step 2 from Figure 4 , the discriminator , conditioned on the class Y , learns to distinguish features sampled from one image versus features sampled from two images belonging to Y . Simultaneously , both members adversarially ( Goodfellow et al. , 2014 ) delete spurious correlations to reduce CR estimation from equation 4 with differentiable signals : it conditionally aligns features . We provide a pseudo-code in B.4 . While we derive similar losses for IBR and CEBR in Appendix E.5 , the full DICE loss is finally : LDICE ( θ1 , θ2 ) = VCEBβceb ( θ1 ) + VCEBβceb ( θ2 ) + δcrL̂CRDV ( e1 , e2 ) . ( 5 ) 2.C FULL PROCEDURE WITH M MEMBERS We expand our objective for an ensemble with M > 2 members . We only consider pairwise interactions for simplicity to keep quadratic rather than exponential growth in number of components and truncate higher order interactions , e.g . I ( Zi ; Zj , Zk|Y ) ( see Appendix F.1 ) . Driven by previous variational and neural estimations , we train θi = { ei , bi , ci } , i ∈ { 1 , . . . , M } on : LDICE ( θ1 : M ) = M∑ i=1 VCEBβceb ( θi ) + δcr ( M − 1 ) M∑ i=1 M∑ j=i+1 L̂CRDV ( ei , ej ) , ( 6 ) while training adversariallyw on Lce . Batch B is sampled from the concatenation of joint distribution p ( zi , zj , y ) where i , j ∈ { 1 , . . . , M } , i 6= j , while Bp is sampled from the product distribution , p ( zi , y ) p ( zj |y ) . We use the same discriminator w for ( M 2 ) estimates . It improves scalability by reducing the number of parameters to be learned . Indeed , an additional member in the ensemble only adds 256 ∗ d trainable weights in w , where d is the features dimension . See Appendix B.3 for additional information related to the discriminator w .

Summary: This paper proposes a method of learning ensembles that adhere to an "ensemble version" of the information bottleneck principle. Whereas the information bottleneck principle says the representation should avoid spurious correlations between the representation (Z) and the training data (X) that is not useful for predicting the labels (Y), i.e. I(X;Z) or I(X;Z|Y), this paper proposes that ensembles should additionally avoid spurious correlations between the ensemble members that aren't useful for predicting Y, i.e. I(Z_i; Z_j| Y). They show empirically that the coefficient on this term increases diversity at the expense of decreasing accuracy of individual members of the ensemble.