Interpreting and Boosting Dropout from a Game-Theoretic View

1 INTRODUCTION . Deep neural networks ( DNNs ) have exhibited significant success in various tasks , but the overfitting problem is still a considerable challenge for deep learning . Dropout is usually considered as an effective operation to alleviate the over-fitting problem of DNNs . Hinton et al . ( 2012 ) ; Srivastava et al . ( 2014 ) thought that dropout could encourage each unit in an intermediate-layer feature to model useful information without much dependence on other units . Konda et al . ( 2016 ) considered dropout as a specific method of data augmentation . Gal & Ghahramani ( 2016 ) proved that dropout was equivalent to the Bayesian approximation in a Gaussian process . Our research group led by Dr. Quanshi Zhang has proposed game-theoretic interactions , including interactions of different orders ( Zhang et al. , 2020 ) and multivariate interactions ( Zhang et al. , 2021b ) . As a basic metric , the interaction can be used to explain signal-processing behaviors in trained DNNs from different perspectives . For example , we have built up a tree structure to explain hierarchical interactions between words encoded in NLP models ( Zhang et al. , 2021a ) . We also prove a close relationship between the interaction and the adversarial robustness ( Ren et al. , 2021 ) and transferability ( Wang et al. , 2020 ) . Many previous methods of boosting adversarial transferability can be explained as the reduction of interactions , and the interaction can also explain the utility of the adversarial training ( Ren et al. , 2021 ) . As an extension of the system of game-theoretic interactions , in this paper , we aim to explain , model , and improve the utility of dropout from the following perspectives . First , we prove that the dropout ∗Correspondence . This study is conducted under the supervision of Dr. Quanshi Zhang . zqs1022 @ sjtu.edu.cn . Quanshi Zhang is with the John Hopcroft Center and the MoE Key Lab of Artificial Intelligence , AI Institute , at the Shanghai Jiao Tong University , China . operation suppresses interactions between input units encoded by DNNs . This is also verified by various experiments . To this end , the interaction is defined in game theory , as follows . Let x denote the input , and let f ( x ) denote the output of the DNN . For the i-th input variable , we can compute its importance value φ ( i ) , which measures the numerical contribution of the i-th variable to the output f ( x ) . We notice that the importance value of the i-th variable would be different when we mask the j-th variable w.r.t . the case when we do not mask the j-th variable . Thus , the interaction between input variables i and j is measured as the difference φw/ j ( i ) − φw/o j ( i ) . Second , we also discover a strong correlation between interactions of input variables and the overfitting problem of the DNN . Specifically , the over-fitted samples usually exhibit much stronger interactions than ordinary samples . Therefore , we consider that the utility of dropout is to alleviate the significance of over-fitting by decreasing the strength of interactions encoded by the DNN . Based on this understanding , we propose an interaction loss to further improve the utility of dropout . The interaction loss directly penalizes the interaction strength , in order to improve the performance of DNNs . The interaction loss exhibits the following two distinct advantages over the dropout operation . ( 1 ) The interaction loss explicitly controls the penalty of the interaction strength , which enables people to trade off between over-fitting and under-fitting . ( 2 ) Unlike dropout which is incompatible with the batch normalization operation ( Li et al. , 2019 ) , the interaction loss can work in harmony with batch normalization . Various experimental results show that the interaction loss can boost the performance of DNNs . Furthermore , we analyze interactions encoded by DNNs from the following three perspectives . ( 1 ) First , we discover the consistency between the sampling process in dropout ( when the dropout rate p = 0.5 ) and the sampling in the computation of the Banzhaf value . The Banzhaf value ( Banzhaf III , 1964 ) is another metric to measure the importance of each input variable in game theory . Unlike the Shapley value , the Banzhaf value is computed under the assumption that each input variable independently participates in the game with the probability 0.5 . We find that the frequent inference patterns in Banzhaf interactions ( Grabisch & Roubens , 1999 ) are also prone to be frequently sampled by dropout , thereby being stably learned . This ensures the DNN to encode smooth Banzhaf interactions . We also prove that the Banzhaf interaction is close to the aforementioned interaction , which also relates to the dropout operation with the interaction used in this paper . ( 2 ) Besides , we find that the interaction loss is better to be applied to low layers than being applied to high layers . ( 3 ) Furthermore , we decompose the overall interaction into interaction components of different orders . We visualize the strongly interacted regions within each input sample . We find out that interaction components of low orders take the main part of interactions and are suppressed by the dropout operation and the interaction loss . Contributions of this paper can be summarized as follows . ( 1 ) We mathematically represent the dependence of feature variables using as the game-theoretic interactions , and prove that dropout can suppress the strength of interactions encoded by a DNN . In comparison , previous studies ( Hinton et al. , 2012 ; Krizhevsky et al. , 2012 ; Srivastava et al. , 2014 ) did not mathematically model the the dependence of feature variables or theoretically proved its relationship with the dropout . ( 2 ) We find that the over-fitted samples usually contain stronger interactions than other samples . ( 3 ) Based on this , we consider the utility of dropout is to alleviate over-fitting by decreasing the interaction . We design a novel loss function to penalize the strength of interactions , which improves the performance of DNNs . ( 4 ) We analyze the properties of interactions encoded by DNNs , and conduct comparative studies to obtain new insights into interactions encoded by DNNs . 2 RELATED WORK . The dropout operation . Dropout is an effective operation to alleviate the over-fitting problem and improve the performance of DNNs ( Hinton et al. , 2012 ) . Several studies have been proposed to explain the inherent mechanism of dropout . According to ( Hinton et al. , 2012 ; Krizhevsky et al. , 2012 ; Srivastava et al. , 2014 ) , dropout could prevent complex co-adaptation between units in intermediate layers , and could encourage each unit to encode useful representations itself . However , these studies only qualitatively analyzed the utility of dropout , instead of providing quantitative results . Wager et al . ( 2013 ) showed that dropout performed as an adaptive regularization , and established a connection to the algorithm AdaGrad . Konda et al . ( 2016 ) interpreted dropout as a kind of data augmentation in the input space , and Gal & Ghahramani ( 2016 ) proved that dropout was equiva- lent to a Bayesian approximation in the Gaussian process . Gao et al . ( 2019 ) disentangled the dropout operation into the forward dropout and the backward dropout , and improved the performance by setting different dropout rates for the forward dropout and the backward dropout , respectively . Gomez et al . ( 2018 ) proposed the targeted dropout , which only randomly dropped variables with low activation values , and kept variables with high activation values . In comparison , we aim to explain the utility of dropout from the view of game theory . Our interaction metric quantified the interaction between all pairs of variables considering all potential contexts , which were randomly sampled from all variables . Furthermore , we propose a method to improve the utility of dropout . Interaction . Previous studies have explored interactions between input variables . Bien et al . ( 2013 ) developed an algorithm to learn hierarchical pairwise interactions inside an additive model . Sorokina et al . ( 2008 ) detected the statistical interaction using an additive model-based ensemble of regression trees . Murdoch et al . ( 2018 ) ; Singh et al . ( 2018 ) ; Jin et al . ( 2019 ) proposed and extended the contextual decomposition to measure the interaction encoded by DNNs in NLP tasks . Tsang et al . ( 2018 ) measured the pairwise interaction based on the learned weights of the DNN . Tsang et al . ( 2020 ) proposed a method , namely GLIDER , to detect the feature interaction modeled by a recommender system . Janizek et al . ( 2020 ) proposed the Integrated-Hessian value to measure interactions , based on Integral Gradient ( Sundararajan et al. , 2017 ) . Integral Gradient measures the importance value for each input variable w.r.t . the DNN . Given an input vector x ∈ Rn , Integrated Hessians measures the interaction between input variables ( dimensions ) xi and xj as the numerical impact of xj on the importance of xi . To this end , Janizek et al . ( 2020 ) used Integral Gradient to compute Integrated Hessians . Appendix J further compares Integral Hessians with the interaction proposed in this paper . Besides interactions measured from the above views , game theory is also a typical perspective to analyze the interaction . Several studies explored the interaction based on game theory . Lundberg et al . ( 2018 ) defined the interaction between two variables based on the Shapley value for tree ensembles . Because Shapley value was considered as the unique standard method to estimate contributions of input words to the prediction score with solid theoretic foundations ( Weber , 1988 ) , this definition of interaction can be regarded to objectively reflect the collaborative/adversarial effects between variables w.r.t the prediction score . Furthermore , Grabisch & Roubens ( 1999 ) extended this definition to interactions among different numbers of input variables . Grabisch & Roubens ( 1999 ) also proposed the interaction based on the Banzhaf value . In comparison , the target interaction used in this paper is based on the Shapley value ( Shapley , 1953 ) . Since the Shapley is the unique metric that satisfies the linearity property , the dummy property , the symmetry property , and the efficiency property ( Ancona et al. , 2019 ) , the interaction based on the Shapley value is usually considered as a more standard metric than the interaction based on the Banzhaf value . In this paper , we aim to explain the utility of dropout using the interaction defined in game theory . We reveal the close relationship between the strength of interactions and the over-fitting of DNNs . We also design an interaction loss to improve the performance of DNNs . 3 GAME-THEORETIC EXPLANATIONS OF DROPOUT . Preliminaries : Shapley values . The Shapley value was initially proposed by Shapley ( 1953 ) in game theory . It is considered as a unique unbiased metric that fairly allocates the numerical contribution of each player to the total reward . Given a set of players N = { 1 , 2 , · · · , n } , 2N def= { S|S ⊆ N } denotes all possible subsets of N . A game f : 2N → R is a function that maps from a subset to a real number . f ( S ) is the score obtained by the subset S ⊆ N . Thus , f ( N ) − f ( ∅ ) denotes the reward obtained by all players in the game . The Shapley value allocates the overall reward to each player , as its numerical contribution φ ( i|N ) , as shown in Equation ( 1 ) . The Shapley value of player i in the game f , φ ( i|N ) , is computed as follows.∑n i=1 φ ( i|N ) = f ( N ) − f ( ∅ ) , φ ( i|N ) = ∑ S⊆N\ { i } PShapley ( S ∣∣N \ { i } ) [ f ( S ∪ { i } ) − f ( S ) ] ( 1 ) where PShapley ( S|M ) = ( |M|−|S| ) ! |S| ! ( |M|+1 ) ! is the likelihood of S being sampled , S⊆M . The Shapley value is the unique metric that satisfies the linearity property , the dummy property , the symmetry property , and the efficiency property ( Ancona et al. , 2019 ) . We summarize these properties in Appendix A . Understanding DNNs via game theory . In game theory , some players may form a coalition to compete with other players , and win a reward ( Grabisch & Roubens , 1999 ) . Accordingly , a DNN f can be considered as a game , and the output of the DNN corresponds to the score f ( · ) in Equation ( 1 ) . For example , if the DNN has a scalar output , we can take this output as the score . If the DNN outputs a vector for multi-category classification , we select the classification score corresponding to the true class as the score f ( · ) . Alternatively , f can also be set as the loss value of the DNN . The set of players N corresponds to the set of input variables . We can analyze the interaction and the element-wise contribution at two different levels . ( 1 ) We can consider input variables ( players ) as the input of the entire DNN , e.g . pixels in images and words in sentences . In this case , the game f is considered as the entire DNN . ( 2 ) Alternatively , we can also consider input variables as a set of activation units before the dropout operation . In this case , the game f is considered as consequent modules of the DNN . S⊆N in Equation ( 1 ) denotes the context of the input variable i , which consists of a subset of input variables . In order to compute the network output f ( S ) , we replace variables in N \S with the baseline value ( e.g . mask them ) , and we do not change the variables in S. In particular , when we consider neural activations before dropout as input variables , such activations are usually non-negative after ReLU . Thus , their baseline values are set to 0 . Both ( Ancona et al. , 2019 ) and Appendix G introduce details about the baseline value . In this way , f ( ∅ ) measures the output score when all input variables are masked , and f ( S ) −f ( ∅ ) measures the entire reward obtained by all input variables in S. Interactions encoded by DNNs . In this section , we introduce how to use the interaction defined in game theory to explain DNNs . Two input variables may interact with each other to contribute to the output of a DNN . Let us suppose input variables i and j have an interaction . In other words , the contribution of i and j when they work jointly is different with the case when they work individually . For example , in the sentence he is a green hand , the word green and the word hand have a strong interaction , because the words green and hand contribute to the person ’ s identity jointly , rather than independently . In this case , we can consider these two input variables to form a certain inference pattern as a singleton player Sij = { i , j } . Thus , this DNN can be considered to have only ( n−1 ) input variables , N ′ = N \ { i , j } ∪Sij , i.e . Sij is supposed to be always absent or present simultaneously as a constituent . In this way , the interaction I ( i , j ) between input variables i and j is defined by Grabisch & Roubens ( 1999 ) , as the contribution increase of Sij when input variables i and j cooperate with each other w.r.t . the case when i and j work individually , as follows . I ( i , j ) def = φ ( Sij |N ′ ) − [ φ ( i|N \ { j } ) + φ ( j|N \ { i } ) ] = ∑ S⊆N\ { i , j } PShapley ( S ∣∣N \ { i , j } ) ∆f ( S , i , j ) ( 2 ) where ∆f ( S , i , j ) def= f ( S ∪ { i , j } ) − f ( S ∪ { j } ) − f ( S ∪ { i } ) + f ( S ) . φ ( i|N \ { j } ) and φ ( j|N \ { i } ) correspond to the contribution to the DNN output when i and j work individually . Theoretically , I ( i , j ) is also equal to the change of the variable i ’ s Shapley value when we mask another input variable j w.r.t . the case when we do not mask j . If I ( i , j ) > 0 , input variables i and j cooperate with each other for a higher output value . Whereas , if I ( i , j ) < 0 , i and j have a negative/adversarial effect . The strength of the interaction can be computed as the absolute value of the interaction , i.e . |I ( i , j ) | . We find that the overall interaction I ( i , j ) can be decomposed into interaction components with different orders s. We use the multi-order interaction defined in ( Zhang et al. , 2020 ) , as follows . I ( i , j ) = n−2∑ s=0 [ I ( s ) ( i , j ) n− 1 ] , I ( s ) ( i , j ) def= ES⊆N\ { i , j } , |S|=s [ ∆f ( S , i , j ) ] ( 3 ) where s denote the size of the context S for the interaction . We use I ( s ) ( i , j ) to represent the s-order interaction between input variables i and j. I ( s ) ( i , j ) reflects the average interaction between input variables i and j among all contexts S with s input variables . For example , when s is small , I ( s ) ( i , j ) measures the interaction relying on inference patterns consisting of very few input variables , i.e . the interaction depends on a small context . When s is large , I ( s ) ( i , j ) corresponds to the interaction relying on inference patterns consisting of a large number of input variables , i.e . the interaction depends on the context of a large scale . Visualization of interactions in each sample , which are encoded by the DNN : There exists a specific interaction between each pair of pixels , which boosts the difficulty of visualization . In order to simplify the visualization , we divide the original image into 16 × 16 grids , and we only visualize the strength of interactions between each grid g and its neighboring grids g′ as Color ( g ) = Eg′∈neighbor ( g ) [ |I ( g , g′ ) | ] . Figure 1 visualizes the interaction strength within the image in the CelebA 0 , 17 , 11 , 20 , 21 , 25 , 31 , 49 , 43 , 46 , 6 dataset ( Liu et al. , 2015 ) , which are normalized to the range [ 0 , 1 ] . Grids on the face usually contain more significant interactions with neighboring grids than grids in the background . Proof of the relationship between dropout and the interaction . In this section , we aim to mathematically prove that dropout is an effective method to suppress the interaction strength encoded by DNNs . Given the context S , let us consider its subset T ⊆ S , which forms a coalition to represent a specific inference pattern T ∪ { i , j } . Note that for dropout , the context refers to activation units in the intermediate-layer feature without semantic meanings . Nevertheless , we just consider i , j as pixels as a toy example to illustrate the basic idea , in order to simplify the introduction . For example , let S represent the face , and let T ∪ { i , j } represent pixels of an eye in the face . Let RT ( i , j ) quantify the marginal reward obtained from the inference pattern of an eye . All interaction effects from smaller coalitions T ′ $ T are removed from RT ( i , j ) . According to the above example , Let T ′ $ T correspond to the pupil inside the eye . Then , RT ′ ( i , j ) measures the marginal reward benefited from the existence of the pupil T ′ ∪ { i , j } , while RT ( i , j ) represents the marginal benefit from the existence of the entire eye , in which the reward from the pupil has been removed . I.e . the inference pattern T ∪ { i , j } can be exclusively triggered by the co-occurrence of all pixels in the eye , but can not be triggered by a subset of pixels in the pupil T ′ ∪ { i , j } . The benefit from the pupil pattern has been removed from RT ( i , j ) . Thus , the s-order interaction can be decomposed into components w.r.t . all inference patterns T ∪ { i , j } , T ⊆ S. I ( s ) ( i , j ) = ES⊆N\ { i , j } , |S|=s [ ∑ T⊆S RT ( i , j ) ] = ∑ 0≤q≤s ( s q ) J ( q ) ( i , j ) = ∑ 0≤q≤s Γ ( q ) ( i , j|s ) ( 4 ) where Jq ( i , j ) def= ET⊆N\ { i , j } , |T |=q [ RT ( i , j ) ] denotes the average interaction between i and j given all potential inference patterns T ∪ { i , j } with a fixed inference pattern size |T | = q ; Γ ( q ) ( i , j|s ) def= ( s q ) J ( q ) ( i , j ) . The computation of RT ( i , j ) and the proof of Equation ( 4 ) are provided in Appendices D and B , respectively . However , when input variables in N are randomly removed by the dropout operation , the computation of I ( s ) dropout ( i , j ) only involves a subset of inference patterns consisting of variables that are not dropped . Let the dropout rate be ( 1 − p ) , p ∈ [ 0 , 1 ] , and let S′ ⊆ S denote the input variables that remain in the context S after the dropout operation . Let us consider cases when r activation units remain after the dropout operation , i.e . |S′| = r. Then , the average interaction I ( s ) dropout ( i , j ) in these cases can be computed as follows . I ( s ) dropout ( i , j ) = E S⊆N\ { i , j } , |S|=s [ E S′⊆S , |S′|=r ( ∑ T⊆S′ RT ( i , j ) ) ] = ∑ 0≤q≤r Γ ( q ) ( i , j|r ) ( 5 ) The interaction only comprises the marginal reward from the inference patterns consisting of at most r ∼ B ( s , p ) variables , where B ( s , p ) is the binomial distribution with the sample number s and the sample rate p. Since r = |S′| ≤ s , we have 1 ≥ Γ ( 1 ) ( i , j|r ) Γ ( 1 ) ( i , j|s ) ≥ · · · ≥ Γ ( r ) ( i , j|r ) Γ ( r ) ( i , j|s ) ≥ 0 , I ( s ) dropout ( i , j ) I ( s ) ( i , j ) = ∑ 0≤q≤r Γ ( q ) ( i , j|r ) ∑ 0≤q≤s Γ ( q ) ( i , j|s ) ≤ 1 . ( 6 ) We assume that most Γ ( q ) ( i , j|s ) , 0 ≤ q ≤ s share the same signal . In this way , we can get Equation ( 6 , right ) based on law of large numbers , which shows that the number of inference patterns usually significantly decreases when we use dropout to remove s − r , r ∼ B ( s , p ) , activation units . Please see Appendix C for the proof . Experimental verification 1 : Inference patterns with more activation units are less likely to be sampled when we use dropout , thereby being more vulnerable to the dropout operation , according to Equation ( 6 , left ) . Inspired by this , we conducted experiments to explore the following two terms , ( 1 ) which component { I ( s ) ( i , j ) } ( s = 0 , ... , n− 2 ) took the main part of the overall interaction strength among all interaction components with different orders ; ( 2 ) which interaction component was mainly penalized by dropout . To this end , the strength of the s-order interaction component was averaged over images , i.e . I ( s ) = Eimage [ E ( i , j ) ∈image ( |I ( s ) ( i , j ) | ) ] . Specifically , when we analyzed the strength of interaction components , we selected the following orders , s = 0.1n , 0.3n , ... , 0.9n . Note that we randomly sampled the value of s from [ 0.0 , 0.2n ] for each context S to approximate the interaction component with the order s = 0.1n . We also used the similar approximation for s = 0.3n , ... , 0.9n . Figure 2 shows curves of different interaction components1 within VGG-11/19 ( Simonyan & Zisserman , 2015 ) learned on the CIFAR-10 ( Krizhevsky & Hinton , 2009 ) dataset with and without the dropout operation . We found that the interaction components with low orders took the main part of the interaction . Experimental verification 2 : We also conducted experiments to illustrate how dropout suppressed the interaction modeled by DNNs , which was a verification of Equation ( 6 , right ) . In experiments , we trained AlexNet ( Krizhevsky et al. , 2012 ) , and VGG-11/16/19 on CIFAR-10 with and without the dropout . Figure 3 ( left ) compares the strength of interactions1 encoded by DNNs , which were learned with or without the dropout operation . When we learned DNNs with dropout , we set the dropout rate as 0.5 . Please see Appendix H.1 for experiments on different dropout rates . We averaged the strength of interactions over images , i.e . I = Eimage [ E ( i , j ) ∈image ( |I ( i , j ) | ) ] , where I ( i , j ) was obtained according to Equation ( 2 ) . Note that accurately computing the interaction of two input variables was an NP-hard problem . Thus , we applied a sampling-based method ( Castro et al. , 2009 ) to approximate the strength of interactions . Furthermore , we conducted an experiment to explore the accuracy of the interactions approximated via the sampling-based method . Please see Appendix K for details . Castro et al . ( 2009 ) proposed a method to approximate the Shapley value , which can be extended to the approximation of the interaction . We found that dropout could effectively suppress the strength of the interaction , which verified the above proof . Property : The sampling process in dropout is the same as the computation in the Banzhaf value . In this section , we aim to show that the sampling process in dropout ( when the dropout rate is 0.5 ) is similar to the sampling in the computation of the Banzhaf value . Just like the Shapley value , the Banzhaf value ( Banzhaf III , 1964 ) is another typical metric to measure importance of each input variable in game theory . Unlike the Shapley value , the Banzhaf value is computed under the assumption that each input variable independently participates in the game with the probability 0.5 . The Banzhaf value is computed as ψ ( i|N ) = ∑ S⊆N\ { i } PBanzhaf ( S|N\ { i } ) [ f ( S ∪ { i } ) − f ( S ) ] , where PBanzhaf ( S|N \ { i } ) = 0.5n−1 is the likelihood of S being sampled . The form of the Banzhaf value is similar to that of the Shapley value in Equation ( 1 ) , but the sampling weight of the Banzhaf value 1For fair comparison , we normalize the value of interaction using the range of output scores of DNNs . Please see Appendix E for more details . PBanzhaf ( S|N\ { i } ) is different from PShapley ( S|N\ { i } ) of the Shapley value . For dropout with the dropout rate 0.5 , let n be the number of input variables , and let S be the units not dropped inN \ { i } . Then , the likelihood of S not being dropped is given as Pdropout ( S|N\ { i } ) = 0.5|S|0.5n−|S|−1 = PBanzhaf ( S|N\ { i } ) . In this way , dropout usually generates activation units S following PBanzhaf ( S|N\ { i } ) . Therefore , the frequent inference patterns in the computation of the Banzhaf value are also frequently generated by dropout , thereby being reliably learned by the DNN . This makes that f ( S ∪ { i } ) − f ( S ) in the computation of the Banzhaf value can be sophisticatedly modeled without lots of outlier values . Thus , the dropout can be considered as a smooth factor in terms of the Banzhaf value . Grabisch & Roubens ( 1999 ) defined the interaction based on the Banzhaf value as IBanzhaf ( i , j ) =∑ S⊆N\ { i , j } PBanzhaf ( S ∣∣N\ { i , j } ) ∆f ( S , i , j ) , just like Equation ( 2 ) . Thus , dropout can also be considered as a smooth factor in the computation of the Banzhaf interaction . Figure 3 ( right ) shows that the target interaction used in this study is closely related to the Banzhaf interaction , which potentially connects the target interaction based on the Shapley value to the dropout operation .

Summary: This paper aims to explain dropout from the lens of game theoretic interactions. Let x denote the input of a deep neural net (DNN), intuitively, the interaction between two variables x_i and x_j quantifies how much the presence/absence of the j-th variable affects the contribution of the i-th variable to the output of the DNN. With the above definition in place, the authors show theoretically and empirically that dropout reduces the interactions between input variables of DNNs. As this type of interactions turn out to be strongly correlated with overfitting, the authors suggest that dropout alleviates overfitting by reducing interactions between input variables (or activation units) of DNNs. Based on this understanding of dropout, an alternative regularization technique is proposed, which explicitly penalizes pairwise interactions between variables.