Interpreting and Boosting Dropout from a Game-Theoretic View

1 INTRODUCTION . Deep neural networks ( DNNs ) have exhibited significant success in various tasks , but the overfitting problem is still a considerable challenge for deep learning . Dropout is usually considered as an effective operation to alleviate the over-fitting problem of DNNs . Hinton et al . ( 2012 ) ; Srivastava et al . ( 2014 ) thought that dropout could encourage each unit in an intermediate-layer feature to model useful information without much dependence on other units . Konda et al . ( 2016 ) considered dropout as a specific method of data augmentation . Gal & Ghahramani ( 2016 ) proved that dropout was equivalent to the Bayesian approximation in a Gaussian process . Our research group led by Dr. Quanshi Zhang has proposed game-theoretic interactions , including interactions of different orders ( Zhang et al. , 2020 ) and multivariate interactions ( Zhang et al. , 2021b ) . As a basic metric , the interaction can be used to explain signal-processing behaviors in trained DNNs from different perspectives . For example , we have built up a tree structure to explain hierarchical interactions between words encoded in NLP models ( Zhang et al. , 2021a ) . We also prove a close relationship between the interaction and the adversarial robustness ( Ren et al. , 2021 ) and transferability ( Wang et al. , 2020 ) . Many previous methods of boosting adversarial transferability can be explained as the reduction of interactions , and the interaction can also explain the utility of the adversarial training ( Ren et al. , 2021 ) . As an extension of the system of game-theoretic interactions , in this paper , we aim to explain , model , and improve the utility of dropout from the following perspectives . First , we prove that the dropout ∗Correspondence . This study is conducted under the supervision of Dr. Quanshi Zhang . zqs1022 @ sjtu.edu.cn . Quanshi Zhang is with the John Hopcroft Center and the MoE Key Lab of Artificial Intelligence , AI Institute , at the Shanghai Jiao Tong University , China . operation suppresses interactions between input units encoded by DNNs . This is also verified by various experiments . To this end , the interaction is defined in game theory , as follows . Let x denote the input , and let f ( x ) denote the output of the DNN . For the i-th input variable , we can compute its importance value φ ( i ) , which measures the numerical contribution of the i-th variable to the output f ( x ) . We notice that the importance value of the i-th variable would be different when we mask the j-th variable w.r.t . the case when we do not mask the j-th variable . Thus , the interaction between input variables i and j is measured as the difference φw/ j ( i ) − φw/o j ( i ) . Second , we also discover a strong correlation between interactions of input variables and the overfitting problem of the DNN . Specifically , the over-fitted samples usually exhibit much stronger interactions than ordinary samples . Therefore , we consider that the utility of dropout is to alleviate the significance of over-fitting by decreasing the strength of interactions encoded by the DNN . Based on this understanding , we propose an interaction loss to further improve the utility of dropout . The interaction loss directly penalizes the interaction strength , in order to improve the performance of DNNs . The interaction loss exhibits the following two distinct advantages over the dropout operation . ( 1 ) The interaction loss explicitly controls the penalty of the interaction strength , which enables people to trade off between over-fitting and under-fitting . ( 2 ) Unlike dropout which is incompatible with the batch normalization operation ( Li et al. , 2019 ) , the interaction loss can work in harmony with batch normalization . Various experimental results show that the interaction loss can boost the performance of DNNs . Furthermore , we analyze interactions encoded by DNNs from the following three perspectives . ( 1 ) First , we discover the consistency between the sampling process in dropout ( when the dropout rate p = 0.5 ) and the sampling in the computation of the Banzhaf value . The Banzhaf value ( Banzhaf III , 1964 ) is another metric to measure the importance of each input variable in game theory . Unlike the Shapley value , the Banzhaf value is computed under the assumption that each input variable independently participates in the game with the probability 0.5 . We find that the frequent inference patterns in Banzhaf interactions ( Grabisch & Roubens , 1999 ) are also prone to be frequently sampled by dropout , thereby being stably learned . This ensures the DNN to encode smooth Banzhaf interactions . We also prove that the Banzhaf interaction is close to the aforementioned interaction , which also relates to the dropout operation with the interaction used in this paper . ( 2 ) Besides , we find that the interaction loss is better to be applied to low layers than being applied to high layers . ( 3 ) Furthermore , we decompose the overall interaction into interaction components of different orders . We visualize the strongly interacted regions within each input sample . We find out that interaction components of low orders take the main part of interactions and are suppressed by the dropout operation and the interaction loss . Contributions of this paper can be summarized as follows . ( 1 ) We mathematically represent the dependence of feature variables using as the game-theoretic interactions , and prove that dropout can suppress the strength of interactions encoded by a DNN . In comparison , previous studies ( Hinton et al. , 2012 ; Krizhevsky et al. , 2012 ; Srivastava et al. , 2014 ) did not mathematically model the the dependence of feature variables or theoretically proved its relationship with the dropout . ( 2 ) We find that the over-fitted samples usually contain stronger interactions than other samples . ( 3 ) Based on this , we consider the utility of dropout is to alleviate over-fitting by decreasing the interaction . We design a novel loss function to penalize the strength of interactions , which improves the performance of DNNs . ( 4 ) We analyze the properties of interactions encoded by DNNs , and conduct comparative studies to obtain new insights into interactions encoded by DNNs . 2 RELATED WORK . The dropout operation . Dropout is an effective operation to alleviate the over-fitting problem and improve the performance of DNNs ( Hinton et al. , 2012 ) . Several studies have been proposed to explain the inherent mechanism of dropout . According to ( Hinton et al. , 2012 ; Krizhevsky et al. , 2012 ; Srivastava et al. , 2014 ) , dropout could prevent complex co-adaptation between units in intermediate layers , and could encourage each unit to encode useful representations itself . However , these studies only qualitatively analyzed the utility of dropout , instead of providing quantitative results . Wager et al . ( 2013 ) showed that dropout performed as an adaptive regularization , and established a connection to the algorithm AdaGrad . Konda et al . ( 2016 ) interpreted dropout as a kind of data augmentation in the input space , and Gal & Ghahramani ( 2016 ) proved that dropout was equiva- lent to a Bayesian approximation in the Gaussian process . Gao et al . ( 2019 ) disentangled the dropout operation into the forward dropout and the backward dropout , and improved the performance by setting different dropout rates for the forward dropout and the backward dropout , respectively . Gomez et al . ( 2018 ) proposed the targeted dropout , which only randomly dropped variables with low activation values , and kept variables with high activation values . In comparison , we aim to explain the utility of dropout from the view of game theory . Our interaction metric quantified the interaction between all pairs of variables considering all potential contexts , which were randomly sampled from all variables . Furthermore , we propose a method to improve the utility of dropout . Interaction . Previous studies have explored interactions between input variables . Bien et al . ( 2013 ) developed an algorithm to learn hierarchical pairwise interactions inside an additive model . Sorokina et al . ( 2008 ) detected the statistical interaction using an additive model-based ensemble of regression trees . Murdoch et al . ( 2018 ) ; Singh et al . ( 2018 ) ; Jin et al . ( 2019 ) proposed and extended the contextual decomposition to measure the interaction encoded by DNNs in NLP tasks . Tsang et al . ( 2018 ) measured the pairwise interaction based on the learned weights of the DNN . Tsang et al . ( 2020 ) proposed a method , namely GLIDER , to detect the feature interaction modeled by a recommender system . Janizek et al . ( 2020 ) proposed the Integrated-Hessian value to measure interactions , based on Integral Gradient ( Sundararajan et al. , 2017 ) . Integral Gradient measures the importance value for each input variable w.r.t . the DNN . Given an input vector x ∈ Rn , Integrated Hessians measures the interaction between input variables ( dimensions ) xi and xj as the numerical impact of xj on the importance of xi . To this end , Janizek et al . ( 2020 ) used Integral Gradient to compute Integrated Hessians . Appendix J further compares Integral Hessians with the interaction proposed in this paper . Besides interactions measured from the above views , game theory is also a typical perspective to analyze the interaction . Several studies explored the interaction based on game theory . Lundberg et al . ( 2018 ) defined the interaction between two variables based on the Shapley value for tree ensembles . Because Shapley value was considered as the unique standard method to estimate contributions of input words to the prediction score with solid theoretic foundations ( Weber , 1988 ) , this definition of interaction can be regarded to objectively reflect the collaborative/adversarial effects between variables w.r.t the prediction score . Furthermore , Grabisch & Roubens ( 1999 ) extended this definition to interactions among different numbers of input variables . Grabisch & Roubens ( 1999 ) also proposed the interaction based on the Banzhaf value . In comparison , the target interaction used in this paper is based on the Shapley value ( Shapley , 1953 ) . Since the Shapley is the unique metric that satisfies the linearity property , the dummy property , the symmetry property , and the efficiency property ( Ancona et al. , 2019 ) , the interaction based on the Shapley value is usually considered as a more standard metric than the interaction based on the Banzhaf value . In this paper , we aim to explain the utility of dropout using the interaction defined in game theory . We reveal the close relationship between the strength of interactions and the over-fitting of DNNs . We also design an interaction loss to improve the performance of DNNs . 3 GAME-THEORETIC EXPLANATIONS OF DROPOUT . Preliminaries : Shapley values . The Shapley value was initially proposed by Shapley ( 1953 ) in game theory . It is considered as a unique unbiased metric that fairly allocates the numerical contribution of each player to the total reward . Given a set of players N = { 1 , 2 , · · · , n } , 2N def= { S|S ⊆ N } denotes all possible subsets of N . A game f : 2N → R is a function that maps from a subset to a real number . f ( S ) is the score obtained by the subset S ⊆ N . Thus , f ( N ) − f ( ∅ ) denotes the reward obtained by all players in the game . The Shapley value allocates the overall reward to each player , as its numerical contribution φ ( i|N ) , as shown in Equation ( 1 ) . The Shapley value of player i in the game f , φ ( i|N ) , is computed as follows.∑n i=1 φ ( i|N ) = f ( N ) − f ( ∅ ) , φ ( i|N ) = ∑ S⊆N\ { i } PShapley ( S ∣∣N \ { i } ) [ f ( S ∪ { i } ) − f ( S ) ] ( 1 ) where PShapley ( S|M ) = ( |M|−|S| ) ! |S| ! ( |M|+1 ) ! is the likelihood of S being sampled , S⊆M . The Shapley value is the unique metric that satisfies the linearity property , the dummy property , the symmetry property , and the efficiency property ( Ancona et al. , 2019 ) . We summarize these properties in Appendix A . Understanding DNNs via game theory . In game theory , some players may form a coalition to compete with other players , and win a reward ( Grabisch & Roubens , 1999 ) . Accordingly , a DNN f can be considered as a game , and the output of the DNN corresponds to the score f ( · ) in Equation ( 1 ) . For example , if the DNN has a scalar output , we can take this output as the score . If the DNN outputs a vector for multi-category classification , we select the classification score corresponding to the true class as the score f ( · ) . Alternatively , f can also be set as the loss value of the DNN . The set of players N corresponds to the set of input variables . We can analyze the interaction and the element-wise contribution at two different levels . ( 1 ) We can consider input variables ( players ) as the input of the entire DNN , e.g . pixels in images and words in sentences . In this case , the game f is considered as the entire DNN . ( 2 ) Alternatively , we can also consider input variables as a set of activation units before the dropout operation . In this case , the game f is considered as consequent modules of the DNN . S⊆N in Equation ( 1 ) denotes the context of the input variable i , which consists of a subset of input variables . In order to compute the network output f ( S ) , we replace variables in N \S with the baseline value ( e.g . mask them ) , and we do not change the variables in S. In particular , when we consider neural activations before dropout as input variables , such activations are usually non-negative after ReLU . Thus , their baseline values are set to 0 . Both ( Ancona et al. , 2019 ) and Appendix G introduce details about the baseline value . In this way , f ( ∅ ) measures the output score when all input variables are masked , and f ( S ) −f ( ∅ ) measures the entire reward obtained by all input variables in S. Interactions encoded by DNNs . In this section , we introduce how to use the interaction defined in game theory to explain DNNs . Two input variables may interact with each other to contribute to the output of a DNN . Let us suppose input variables i and j have an interaction . In other words , the contribution of i and j when they work jointly is different with the case when they work individually . For example , in the sentence he is a green hand , the word green and the word hand have a strong interaction , because the words green and hand contribute to the person ’ s identity jointly , rather than independently . In this case , we can consider these two input variables to form a certain inference pattern as a singleton player Sij = { i , j } . Thus , this DNN can be considered to have only ( n−1 ) input variables , N ′ = N \ { i , j } ∪Sij , i.e . Sij is supposed to be always absent or present simultaneously as a constituent . In this way , the interaction I ( i , j ) between input variables i and j is defined by Grabisch & Roubens ( 1999 ) , as the contribution increase of Sij when input variables i and j cooperate with each other w.r.t . the case when i and j work individually , as follows . I ( i , j ) def = φ ( Sij |N ′ ) − [ φ ( i|N \ { j } ) + φ ( j|N \ { i } ) ] = ∑ S⊆N\ { i , j } PShapley ( S ∣∣N \ { i , j } ) ∆f ( S , i , j ) ( 2 ) where ∆f ( S , i , j ) def= f ( S ∪ { i , j } ) − f ( S ∪ { j } ) − f ( S ∪ { i } ) + f ( S ) . φ ( i|N \ { j } ) and φ ( j|N \ { i } ) correspond to the contribution to the DNN output when i and j work individually . Theoretically , I ( i , j ) is also equal to the change of the variable i ’ s Shapley value when we mask another input variable j w.r.t . the case when we do not mask j . If I ( i , j ) > 0 , input variables i and j cooperate with each other for a higher output value . Whereas , if I ( i , j ) < 0 , i and j have a negative/adversarial effect . The strength of the interaction can be computed as the absolute value of the interaction , i.e . |I ( i , j ) | . We find that the overall interaction I ( i , j ) can be decomposed into interaction components with different orders s. We use the multi-order interaction defined in ( Zhang et al. , 2020 ) , as follows . I ( i , j ) = n−2∑ s=0 [ I ( s ) ( i , j ) n− 1 ] , I ( s ) ( i , j ) def= ES⊆N\ { i , j } , |S|=s [ ∆f ( S , i , j ) ] ( 3 ) where s denote the size of the context S for the interaction . We use I ( s ) ( i , j ) to represent the s-order interaction between input variables i and j. I ( s ) ( i , j ) reflects the average interaction between input variables i and j among all contexts S with s input variables . For example , when s is small , I ( s ) ( i , j ) measures the interaction relying on inference patterns consisting of very few input variables , i.e . the interaction depends on a small context . When s is large , I ( s ) ( i , j ) corresponds to the interaction relying on inference patterns consisting of a large number of input variables , i.e . the interaction depends on the context of a large scale . Visualization of interactions in each sample , which are encoded by the DNN : There exists a specific interaction between each pair of pixels , which boosts the difficulty of visualization . In order to simplify the visualization , we divide the original image into 16 × 16 grids , and we only visualize the strength of interactions between each grid g and its neighboring grids g′ as Color ( g ) = Eg′∈neighbor ( g ) [ |I ( g , g′ ) | ] . Figure 1 visualizes the interaction strength within the image in the CelebA 0 , 17 , 11 , 20 , 21 , 25 , 31 , 49 , 43 , 46 , 6 dataset ( Liu et al. , 2015 ) , which are normalized to the range [ 0 , 1 ] . Grids on the face usually contain more significant interactions with neighboring grids than grids in the background . Proof of the relationship between dropout and the interaction . In this section , we aim to mathematically prove that dropout is an effective method to suppress the interaction strength encoded by DNNs . Given the context S , let us consider its subset T ⊆ S , which forms a coalition to represent a specific inference pattern T ∪ { i , j } . Note that for dropout , the context refers to activation units in the intermediate-layer feature without semantic meanings . Nevertheless , we just consider i , j as pixels as a toy example to illustrate the basic idea , in order to simplify the introduction . For example , let S represent the face , and let T ∪ { i , j } represent pixels of an eye in the face . Let RT ( i , j ) quantify the marginal reward obtained from the inference pattern of an eye . All interaction effects from smaller coalitions T ′ $ T are removed from RT ( i , j ) . According to the above example , Let T ′ $ T correspond to the pupil inside the eye . Then , RT ′ ( i , j ) measures the marginal reward benefited from the existence of the pupil T ′ ∪ { i , j } , while RT ( i , j ) represents the marginal benefit from the existence of the entire eye , in which the reward from the pupil has been removed . I.e . the inference pattern T ∪ { i , j } can be exclusively triggered by the co-occurrence of all pixels in the eye , but can not be triggered by a subset of pixels in the pupil T ′ ∪ { i , j } . The benefit from the pupil pattern has been removed from RT ( i , j ) . Thus , the s-order interaction can be decomposed into components w.r.t . all inference patterns T ∪ { i , j } , T ⊆ S. I ( s ) ( i , j ) = ES⊆N\ { i , j } , |S|=s [ ∑ T⊆S RT ( i , j ) ] = ∑ 0≤q≤s ( s q ) J ( q ) ( i , j ) = ∑ 0≤q≤s Γ ( q ) ( i , j|s ) ( 4 ) where Jq ( i , j ) def= ET⊆N\ { i , j } , |T |=q [ RT ( i , j ) ] denotes the average interaction between i and j given all potential inference patterns T ∪ { i , j } with a fixed inference pattern size |T | = q ; Γ ( q ) ( i , j|s ) def= ( s q ) J ( q ) ( i , j ) . The computation of RT ( i , j ) and the proof of Equation ( 4 ) are provided in Appendices D and B , respectively . However , when input variables in N are randomly removed by the dropout operation , the computation of I ( s ) dropout ( i , j ) only involves a subset of inference patterns consisting of variables that are not dropped . Let the dropout rate be ( 1 − p ) , p ∈ [ 0 , 1 ] , and let S′ ⊆ S denote the input variables that remain in the context S after the dropout operation . Let us consider cases when r activation units remain after the dropout operation , i.e . |S′| = r. Then , the average interaction I ( s ) dropout ( i , j ) in these cases can be computed as follows . I ( s ) dropout ( i , j ) = E S⊆N\ { i , j } , |S|=s [ E S′⊆S , |S′|=r ( ∑ T⊆S′ RT ( i , j ) ) ] = ∑ 0≤q≤r Γ ( q ) ( i , j|r ) ( 5 ) The interaction only comprises the marginal reward from the inference patterns consisting of at most r ∼ B ( s , p ) variables , where B ( s , p ) is the binomial distribution with the sample number s and the sample rate p. Since r = |S′| ≤ s , we have 1 ≥ Γ ( 1 ) ( i , j|r ) Γ ( 1 ) ( i , j|s ) ≥ · · · ≥ Γ ( r ) ( i , j|r ) Γ ( r ) ( i , j|s ) ≥ 0 , I ( s ) dropout ( i , j ) I ( s ) ( i , j ) = ∑ 0≤q≤r Γ ( q ) ( i , j|r ) ∑ 0≤q≤s Γ ( q ) ( i , j|s ) ≤ 1 . ( 6 ) We assume that most Γ ( q ) ( i , j|s ) , 0 ≤ q ≤ s share the same signal . In this way , we can get Equation ( 6 , right ) based on law of large numbers , which shows that the number of inference patterns usually significantly decreases when we use dropout to remove s − r , r ∼ B ( s , p ) , activation units . Please see Appendix C for the proof . Experimental verification 1 : Inference patterns with more activation units are less likely to be sampled when we use dropout , thereby being more vulnerable to the dropout operation , according to Equation ( 6 , left ) . Inspired by this , we conducted experiments to explore the following two terms , ( 1 ) which component { I ( s ) ( i , j ) } ( s = 0 , ... , n− 2 ) took the main part of the overall interaction strength among all interaction components with different orders ; ( 2 ) which interaction component was mainly penalized by dropout . To this end , the strength of the s-order interaction component was averaged over images , i.e . I ( s ) = Eimage [ E ( i , j ) ∈image ( |I ( s ) ( i , j ) | ) ] . Specifically , when we analyzed the strength of interaction components , we selected the following orders , s = 0.1n , 0.3n , ... , 0.9n . Note that we randomly sampled the value of s from [ 0.0 , 0.2n ] for each context S to approximate the interaction component with the order s = 0.1n . We also used the similar approximation for s = 0.3n , ... , 0.9n . Figure 2 shows curves of different interaction components1 within VGG-11/19 ( Simonyan & Zisserman , 2015 ) learned on the CIFAR-10 ( Krizhevsky & Hinton , 2009 ) dataset with and without the dropout operation . We found that the interaction components with low orders took the main part of the interaction . Experimental verification 2 : We also conducted experiments to illustrate how dropout suppressed the interaction modeled by DNNs , which was a verification of Equation ( 6 , right ) . In experiments , we trained AlexNet ( Krizhevsky et al. , 2012 ) , and VGG-11/16/19 on CIFAR-10 with and without the dropout . Figure 3 ( left ) compares the strength of interactions1 encoded by DNNs , which were learned with or without the dropout operation . When we learned DNNs with dropout , we set the dropout rate as 0.5 . Please see Appendix H.1 for experiments on different dropout rates . We averaged the strength of interactions over images , i.e . I = Eimage [ E ( i , j ) ∈image ( |I ( i , j ) | ) ] , where I ( i , j ) was obtained according to Equation ( 2 ) . Note that accurately computing the interaction of two input variables was an NP-hard problem . Thus , we applied a sampling-based method ( Castro et al. , 2009 ) to approximate the strength of interactions . Furthermore , we conducted an experiment to explore the accuracy of the interactions approximated via the sampling-based method . Please see Appendix K for details . Castro et al . ( 2009 ) proposed a method to approximate the Shapley value , which can be extended to the approximation of the interaction . We found that dropout could effectively suppress the strength of the interaction , which verified the above proof . Property : The sampling process in dropout is the same as the computation in the Banzhaf value . In this section , we aim to show that the sampling process in dropout ( when the dropout rate is 0.5 ) is similar to the sampling in the computation of the Banzhaf value . Just like the Shapley value , the Banzhaf value ( Banzhaf III , 1964 ) is another typical metric to measure importance of each input variable in game theory . Unlike the Shapley value , the Banzhaf value is computed under the assumption that each input variable independently participates in the game with the probability 0.5 . The Banzhaf value is computed as ψ ( i|N ) = ∑ S⊆N\ { i } PBanzhaf ( S|N\ { i } ) [ f ( S ∪ { i } ) − f ( S ) ] , where PBanzhaf ( S|N \ { i } ) = 0.5n−1 is the likelihood of S being sampled . The form of the Banzhaf value is similar to that of the Shapley value in Equation ( 1 ) , but the sampling weight of the Banzhaf value 1For fair comparison , we normalize the value of interaction using the range of output scores of DNNs . Please see Appendix E for more details . PBanzhaf ( S|N\ { i } ) is different from PShapley ( S|N\ { i } ) of the Shapley value . For dropout with the dropout rate 0.5 , let n be the number of input variables , and let S be the units not dropped inN \ { i } . Then , the likelihood of S not being dropped is given as Pdropout ( S|N\ { i } ) = 0.5|S|0.5n−|S|−1 = PBanzhaf ( S|N\ { i } ) . In this way , dropout usually generates activation units S following PBanzhaf ( S|N\ { i } ) . Therefore , the frequent inference patterns in the computation of the Banzhaf value are also frequently generated by dropout , thereby being reliably learned by the DNN . This makes that f ( S ∪ { i } ) − f ( S ) in the computation of the Banzhaf value can be sophisticatedly modeled without lots of outlier values . Thus , the dropout can be considered as a smooth factor in terms of the Banzhaf value . Grabisch & Roubens ( 1999 ) defined the interaction based on the Banzhaf value as IBanzhaf ( i , j ) =∑ S⊆N\ { i , j } PBanzhaf ( S ∣∣N\ { i , j } ) ∆f ( S , i , j ) , just like Equation ( 2 ) . Thus , dropout can also be considered as a smooth factor in the computation of the Banzhaf interaction . Figure 3 ( right ) shows that the target interaction used in this study is closely related to the Banzhaf interaction , which potentially connects the target interaction based on the Shapley value to the dropout operation .

Summary: This paper analyzes the effect of dropout on interaction between units in a neural network. The strength of the interaction is measured using a metric that is used in game theory to quantify interaction between players in a co-operative game. The paper shows that dropout reduces high-order interaction (as measured by this metric), and that reduction in interaction is correlated with better generalization. The paper introduces a new regularizer that explicitly minimizes the metric and claims that using this regularizer instead of dropout has some advantages.