Interpreting and Boosting Dropout from a Game-Theoretic View
Authors: Hao Zhang
Sen Li
YinChao Ma
Mingjie Li
Yichen Xie
Quanshi Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: Dropout
Interpretability
Interactions

Abstract: This paper aims to understand and improve the utility of the dropout operation from the perspective of game-theoretical interactions. We prove that dropout can suppress the strength of interactions between input variables of deep neural networks (DNNs). The theoretical proof is also verified by various experiments. Furthermore, we find that such interactions were strongly related to the over-fitting problem in deep learning. So, the utility of dropout can be regarded as decreasing interactions to alleviating the significance of over-fitting. Based on this understanding, we propose the interaction loss to further improve the utility of dropout. Experimental results on various DNNs and datasets have shown that the interaction loss can effectively improve the utility of dropout and boost the performance of DNNs.

Published as a conference paper at ICLR 2021
INTERPRETING AND BOOSTING DROPOUT FROM A CCME-CHEORETIC
THEORETIC VIEW
Hao Zhang
Sen Li
Shanghai Jiao Tong University 1603023-280 -000023--0000 jtu. edu. en
Sun Yat-sen University 1isen6@mai12. sysu. edu.
Yinchao Ma
Mingjie Li
Huazhong University of Science and Technology u201713506@hust. edu. on
Shanghai Jiao Tong University
1 Limingjie0608e jtu. edu.
Yichen Xie
Quanshi Zhang*
Shanghai Jiao Tong University xieyichencs jtu. edu. on
Shanghai Jiao Tong University zqs 1 2222s jtu. edu. on
ABSTRACT
This paper aims to understand and improve the utility of the dropout operation from the perspective of game-theoretic interactions. We prove that dropout can suppress the strength of interactions between input variables of deep neural net- works (DNNs). The theoretic proof also verified by various experiments. Fur- thermore, find that such interactions were strongly related to the over-fitting problem in deep learning. Thus, the utility of dropout can be regarded as decreas- ing interactions to alleviate the significance of wver-fitting. Based this under- standing, we propose an interaction loss to further improve the utility of dropout. Experimental results have shown that the interaction loss effectively improve the utility of dropout and boost the performance of DNNs.
INTRODUCTION
Deep neural networks (DNNs) have exhibited significant success in various tasks, but the over- fitting problem is still a considerable challenge for deep learning. Dropout is usually considered as an effective operation to alleviate the over-fitting problem of DNNs. [Hinton al. x et al. (2014) thought that dropout could encourage each unit in an intermediate- layer feature to model useful information without much dependence on other units. Konda et al. (2016) considered dropout specific method of data augmentation. Gal & Ghahramani (2016) proved that dropout was equivalent the Bayesian approximation
Gaussian process.
Our research group led by Dr. Quanshi Zhang has proposed theeethi interactions, includ- ing interactions of different orders (Zhang al. 2020) and multivariate interactions (Zhang et al,] bb2 As basic metric, the interaction can be used to explain signal-processing behaviors in trained DNNs from different perspectives. For example, we have built up a tree structure to explain hierarchical interactions between words encoded in NLP models (Zhang al., (021a). We also prove a close relationship between the interaction and the adversarial robustness (Ren et al. (02T) and transferability (Wang et al., 2020). Many previous methods of boosting adversarial transferabil- ity can be explained as the reduction of interactions, and the interaction can also explain the utility of the adversarial training (Ren al. 202I).
As an extension of the system of game-theoretic interactions, in this paper, we aim to explain, model, and improve the utility of dropout from the following perspectives. First, we prove that the dropout
"Correspondence. This study is conducted under the supervision of Dr. Quanshi Zhang 7qs 1022@sjin.ccduccc Quanshi Zhang is with the John Hopcroft Center and the MoF Key Lab of Artificial Intelligence, AI Institute, the Shanghai Jiao Tong University, China.

---

compute its Published as a conference paper at ICLR 2021
operation suppresses interactions between imput units encoded by DNNs. This is also verified by various experiments. To this end, the interaction is defined in game theory, as follows. Let x denote the input, and let f()) denote the output of the DNN. For the i-th input variable, importance value o(i), which measures the numerical contribution of the i-th variable to the output (z)). We notice that the importance value of the i-th variable would be different when we mask the j-th variable w.E.. the case when we do not mask the j-th variable. Thus, the interaction between input variables and j is measured as the difference Put, own (i).
Second, we also discover a strong correlation between interactions of input variables and the over- fitting problem of the DNN. Specifically, the over-fitted samples usually exhibit much stronger in- teractions than ordinary samples.
Therefore, we consider that the utility of dropout alleviate the significance of over-fitting by de- creasing the strength of interactions encoded by the DNN. Based on this understanding, we propose an interaction loss to further improve the utility of dropout. The interaction loss directly penal- izes the interaction strength, in order to improve the performance of DNNs. The interaction loss exhibits the following two distinct advantages over the dropout operation. (1) The interaction loss explicitly controls the penalty of the interaction strength, which enables people to trade off between over- fittifiting and under-fitting. fitting. (2) Unlike dropout which is incompatible with the batch normaliza- tion operation (Li et al. 2019), the interaction loss can work in harmony with batch normalization. Various experimental results show that the interaction loss can boost the performance of DNNs.
Furthermore, we analyze interactions encoded by DNNs from the following three perspectives. (1) First, we discover the consistency between the sampling process in dropout (when the dropout rate p= 0.5) and the sampling in the computation of the Banzhaf value. The Banzhaf value (Banzhaf III, 1964) is another metric to measure the importance of each input variable in game theory. Unlike the Shapley value, the Banzhaf value is computed under the assumption that each input variable independently participates in the game with the probability 0.5. We find that the frequent inference patterns in Banzhaf interactions (Grabisch & Roubens 1999) are also prone to be frequently sam- pled by dropout, thereby being stably learned. This ensures the DNN to encode smooth Banzhaf interactions. We also prove that the Banzhaf interaction is close to the aforementioned interaction, which also relates to the dropout operation with the interaction used in this paper. (2) Besides, we find that the interaction loss better to be applied to low layers than being applied to high layers, (3) Furthermore, we decompose the overall interaction into interaction components of different orders. We visualize the strongly interacted regions within each input sample. We find out that interac- tion components of low orders take the main part of interactions and are suppressed by the dropout operation and the interaction loss.
Contributions of this paper be summarized as follows. (1) We mathematically represent the dependence of feature variables using as the game-theoretic interactions, and prove that dropout can suppress the strength of interactions encoded by DNN. In comparison, previous studies (Hinton) et al. 2012, Krizhevsky et al. 2012 Srivastava et al. 2014) did not mathematically model the the dependence of feature variables or theoretically proved its relationship with the dropout. (2) We find
that the over-fitted samples usually contain stronger interactions than other samples. (3) Based on this, we consider the utility of dropout is to alleviate over-fitting by decreasing the interaction. We design novel loss function to penalize the strength of interactions, which improves the performance of DNNs. (4) We analyze the properties of interactions encoded by DNNs, and conduct comparative studies to obtain new insights into interactions encoded by DNNs.
2 RELATED WORK
The dropout operation. Dropout is an effective operation to alleviate the over-fitting problem and improve the performance of DNNs (Hinton et al. 2012).. Several studies have been proposed to explain the inherent mechanism of dropout. According to (Hinton et al., 2012) Krizhevsky et al.) 2012 Srivastava et al. 2014), dropout could prevent complex co-adaptation between units in in- termediate layers, and could encourage each unit to encode useful representations itself. However, these studies only qualitatively analyzed the utility of dropout, instead of providing quantitative results. Wagger et al. (2013) showed that dropout performed an adaptive regularization, and estab- lished a connection to the algorithm AdaGrad, IKonda et al. (2016) interpreted dropout as a kind of data augmentation in the input space, and & Ghahramani
2016) proved that dropout was equiva-
2

---

Published as a conference paper at ICLR 2021
lent to a Bayesian approximation in the Gaussian process. Gao et al. (2019) disentangled the dropout operation into the forward dropout and the backward dropout, and improved the performance by set- ting different dropout rates for the forward dropout and the backward dropout, respectively. [oomzz eet al. (2018) proposed the targeted dropout, which only randomly dropped variables with low acti- vation values, and kept variables with high activation values. In comparison, utility of dropout from the view of game theory. Our interaction metric quantified the interaction between all pairs of variables considering all potential contexts, which were randomly sampled from all variables. Furthermore, we propose a method to improve the utility of dropout.
aim to explain the
Interaction. Previous studies have explored interactions between imput variables. Bien et al. (2013)
developed an algorithm to learn hierarchical pairwise interactions inside an additive model. (Srookin
at. 2008) detected the statistical interaction using an additive model-based ensemble of regression
trees. Murdoch et al. (2018); Singh et al. (2018); Jin et al. (2019) proposed and extended the contex-
tual decomposition to measure the interaction encoded by DNNS in NLP tasks. [sang et al. (2018)
measured the pairwise interaction based the learned weights of the DNN. Tsang et al. 2020
proposed a method, namely GLIDER, to detect the feature interaction modeled by a recommender
system. Janizek et al. (2020) proposed the Senggrated-Hessiane value to measure interactions, based on Integral Gradient (Sundararajan et al.] (017). Integral Gradient measures the importance value for each input variable w.s.t. the DNN. Given input vector x E R", Integrated Hessians measures the interaction between input variables (dimensions) I; and z. as the numerical impact of Tj on the importance of z4. To this end. Jaanzek al. (2020) used Integral Gradient to compute Integrated Hessians. Appendix further compares Integral Hessians with the interaction proposed in this paper.
furtthr
Besides interactions measured from the above views, game theory also a typical perspective an- alyze the interaction. Several studies explored the interaction based on game theory. Lundberg et al.) (2018) defined the interaction between two variables based on the Shapley value for tree ensembles. Because Shapley value was considered as the unique standard method to estimate contributions of input words to the prediction score with solid theoretic foundations (Weber) 1988), this definition of interaction can be regarded to objectively reflect the collaborative/adverraria effects between vari- ables w.r.f the prediction score. Furthermore, Grabisch & Roubens 1999) extended this definition to interactions among different numbers of input variables. Grabisch & Roubens (1999) also proposed the interaction based on the Banzhaf value. In comparison, the target interaction used in this paper is based on the Shapley value (Shapley 1953). Since the Shapley is the unique metric that satisfies the linearity property. the dummy property, the symmetry property, and the efficiency property (Ancona) et al. 2019), the interaction based on the Shapley value is usually considered as a more standard metric than the interaction based on the Banzhaf value. In this paper, we aim to explain the utility of dropout using the interaction defined in game theory. We reveal the close relationship between the strength of interactions and the over- wer-fitting of DNNs. We also design an interaction loss to improve the performance of DNNs.
THEEREEE EXPLANATIONS OF DROPOUT
Preliminaries: Shapley values. The Shapley value was initially proposed by Shapley (1953) in game theory. It is considered as a unique unbiased metric that fairly allocates the numerical contri- bution of each player to the total reward. Given set of players N = 2, ........ .2,2" 2" def [SIS C N} denotes all possible subsets of N. A game f : 2N + R is a function that maps from a subset to a real number. f(S) is the score obtained by the subset S C N.. Thus, f(V) - f0) denotes the reward obtained by all players in the game. The Shapley value allocates the overall reward to each player, as its numerical contribution (iilN) as shown in Equation (). The Shapley value of player i in the game f. N). is computed as follows.
Σn ((ilN) = f(N)- f(0,, d([ Σ SENVA) Pstupley (5 (SSN (0)) f(S)
(1)
where P'ssaptyy (SIM) 1SpA is the likelihood of S being sampled, SCM. The Shapley value
is the unique metric that satisfies the linearity property, the dummy property, the symmetry property,
and the efficiency property (Ancona et al., 2019). We summarize these properties in Appendix A
Understanding DNNs via game theory. In game theory, some players may form coalition to compete with other players, and win a reward (Grabisch & Roubens] 1999). Accordingly, a DNN/ / can be considered as a game, and the output of the DNN corresponds to the score f-)) in Equation (T)
3

---

Published as a conference paper at ICLR 2021
For example, if the DNN has scalar output, can take this output as the score. the DNN outputs a vector for category classification, we select the classification score corresponding to the true class as the score f((). Alternatively, f can also be set as the loss value of the DNN.
The set of players NN corresponds to the set of input variables. We can analyze the interaction and the element-wise contribution at two different levels. (1) We can consider input variables (players) as the input of the entire DNN, e.g. pixels in images and words in sentences. In this case, the game is considered as the entire DNN. (2) Alternatively, we can also consider input variables as a set of activation units before the dropout operation. In this case, the game f is considered as consequent modules of the DNN.
SCN in Equation (T) denotes the context of the input variable i, which consists of a subset of input variables. In order to compute the network output f(S), we replace variables in N S with the base- line value (e.g. mask them), and we do not change the variables in S. In particular, when we consider neural activations before dropout as input variables, such activations are usually non-negativy after ReLU. Thus, their baseline values are set to O. Both (Ancona et al. (2019) and Appendix (G introduce details about the baseline value. this way. (0) measures the output score when all input variables are masked, and f(S) f(0) measures the entire reward obtained by all input variables in S.
Interactions encoded by DNNs. In this section, we introduce how to use the interaction defined in game theory to explain DNNs. Two input variables may interact with each other to contribute to the output of DNN. Let us suppose input variables i and j have an interaction. In other words, the contribution of and j when they work jointly is different with the case when they work individually. For example, in the sentence he is a green hand, the word green and the word hand have a strong interaction, because the words green and hand contribute to the person's identity jointly, rather than independently. In this case, we can consider these two input variables to form a certain inference pattern as a singleton player Si {i.j]. Thus, this DNN can be considered to have only (n-1) input variables. N = N' (US i.e. S.y is supposed to be always absent or present simultancously as constituent. In this way, the interaction / (i. between input variables i and j is defined by Grabisch & Roubens (1999), as the contribution increase of S.y when input variables i and cooperate with each other w.r.f. the case when é and j work individually, as follows.
I(i,j) 2nd (S. NN) [6G)NND) + +0
= E ESENACE Pstmptes (S \ ((.))) (2)
where (S,) def J(SU (i.53) S(S U (1) - J(SU (+]) + S(S). 6G (7}) and @GIN (t))
correspond to the contribution to the DNN output when and j work individually. Theoretically, I(i,j) is also equal to the change of the variable i's Shapley value when we mask another input variable j w.r.f. the case when we do not mask j.
If T(i.5) > 0, input variables i and j cooperate with each other for higher output value. Whereas, if I(i,5) < O, and j have a negative/addeersaria effect. The strength of the interaction can be computed as the absolute value of the interaction, i.e. (M(i,) We find that the overall interaction [ii can be decomposed into interaction components with different orders s. We use the multi-order interaction defined in (Zhang et al. 2020), follows.
n-2 T(i,5)
Σ (i.5)
(1,) det (ec
where denote the size of the context S for the interaction. We use (i, to represent the s-order interaction between input variables i and j. (i.j) reflects the average interaction between input variables and j among all contexts S with s input variables. For example, when s is small, (i.j) measures the interaction relying on inference patterns consisting of very few input variables, i.e. the interaction depends on a small context. When a large, 7(5) (1,2) corresponds to the interaction relying inference patterns consisting of large number of input variables, i.e. the interaction depends on the context of a large scale.
Visualization of interactions in each sample, which are encoded by the DNN: There exists spe- cific interaction between each pair of pixels, which boosts the difficulty of visualization. In order divide the original image into 16 x 16 grids, and we only visu-
to simplify the visualization, alize the strength of interactions between each grid and its neighboring grids g' as Color(g) = F.y' (noighterr || (g. I-] Figure visualizes the interaction strength within the image in the CelebA
4

---

Published as a conference paper at ICLR 2021
Image
Interactions enooded by
0.50
Internctiona enoxded by
5.00
Figure 1: Visualization of the interaction strength encoded by DNNs.
dataset (Liu et al. 2015), which are normalized to the range [0, 1]. Grids on the face usually contain more sigmificant interactions with neighboring grids than grids in the background.
Proof of the relationship between dropout and the interaction. In this section, we aim to mathe- matically prove that dropout is an effective method to suppress the interaction strength encoded by DNNs. Given the context S, let us consider its subset T S S, which forms coalition to represent specific inference pattern TU (i. Note that for dropout, the context refers to activation units in the intermediate-laye feature without semantic meanings. Nevertheless, we just consider i, pixels as a toy example to illustrate the basic idea, in order to simplify the introduction. For example, let represent the face, and let T J { } represent pixels of an eye in the face. Let (i, quantify the marginal reward obtained from the inference pattern of an eye. All interaction effects from smaller coalitions T' _ T are removed from Rº (i.j).
According to the above example, Let T" Ç T correspond to the pupil inside the eye. Then, R (1,5) measures the marginal reward benefited from the existence of the pupil T' U (i,j), while n2 (i.j) represents the marginal benefit from the existence of the entire eye, in which the reward from the pupil has been removed. Lee. the inference pattern T U {i,}} can be exclusively triggered by the co-occurrence of all pixels in the eye, but cannot be triggered by a subset of pixels in the pupil I" U {i.j).. The benefit from the pupil pattern has been removed from R² (i.j). Thus, the s-order interaction can be decomposed into components w.r.t. all inference patterns T (1,5),
7(0) (1,5) ESCNN(WJHISI. [ErS R (1.11)) -1 (o:0==Eossss Eosgs. (I ( Ensgge (1, (4)
where (i. BrENY [R² (i.j)) denotes the average interaction between i and given all potential inference patterns T (1,5} with a fixed inference pattern size [T] = q; (i,s)))) del
(i, The computation of (1, and the proof of Equation (4) are provided in Appendices
and B respectively.
However, when input variables in are randomly removed by the dropout operation, the compu- tation of (1. only involves subset of inference patterns consisting of variables that not dropped. Let the dropout rate be - (1-p).p E [o, 1]. and let S' S denote the input variables that remain in the context S after the dropout operation. Let us consider cases when r activation units remain after the dropout operation, i.e. |5'| = r. Then, the average interaction Liegou (i,j) these cases can be computed as follows.
E The interaction only comprises the marginal reward from the inference patterns consisting of at most r B(s,p) variables. where B(s,p) p) the binomial distribution with the sample number s and the sample rate p. Since r = |S < s, we have
drocour (i,j)
(Ercor H ((. (1.1))==
-ossst (i.jrr)
(5)
(i, (1. 2
(
(e (1,5) Eososr (( (1.)) Eosos. p(a) (i.gil))))
12
2 O,
S1.
(6)
(1,))s)
We assume that most ((0) (i.j]s). < share the same signal. In this way, can get Equation 66) right) based on law of large numbers, which shows that the number of inference patterns usually significantly decreases when Please see Appendix [ the proof.
dropout to remove r, B(s,p), activation units.
Experimental verification 1: Inference patterns with more activation units less likely to be sampled when we use dropout. thereby being more vulnerable to the dropout operation, accord- ing to Equation e left). Inspired by this, conducted experiments explore the following two
5

---

Published as a conference paper at ICLR 2021
wb dropout
Wi dropout
WD cropoul
wl drapoul
y 0.1m
-1.
0.5m
2..
0.7n
50 Epoch
Epoch
cpech
150 Epech
0.9%L
Figure 2: Curves of strength of interaction components with different orders during the learn- ing process. All interactions were computed between activation units before dropout. A (+) denotes the change of interaction strength.
| drapoux
dropout
VCG 21
VOD LS
V6G-13
VEG-19
imtraction
trppat
159
Figure 3: (left) The decrease of interactions caused by the dropout operation. (right) Close relation- ship between the target interaction Eiinge Eimi)e [i] and the interaction based on the Banzhaf value Emaage (B(s))) / (i, j)]]. We show the change of the two types of interactions during the learning process. The target interaction is highly related to the interaction based on Banzhaf value. Here, all interactions were computed between activation units before dropout.
terms, (1) which component (i = 0, ..., n- 2) took the main part of the overall interaction strength among all interaction components with different orders; (2) which interaction component was mainly penalized by dropout.
To this end, the strength of the s-order interaction component was averaged over images, i.e. 7(0) (E.,) (Iz (i. j)]] Specifically, when we analyzed the strength of interaction components, we selected the following orders, s = 0.1m, o..,.... 0.9n. Note that we randomly sampled the value of s from [0.0, 0.2n] for each context S to approximate the interaction component with the order 8 = 0.1n. We also used the similar approximation for = 0.3n. 0.9n. Figure 2 shows curves of different interaction components" within VGG-11/19 Simonyyn & Zisserman 2015) learned on the CIFAR-10 (Krizhevsky & Hinton 2009) dataset with and without the dropout operation. We found that the interaction components with low orders took the main part of the interaction.
Experimental verification 2: We also conducted experiments to illustrate how dropout suppressed the interaction modeled by DNNs, which was verification of Equation (6) right). In experiments, we trained AlexNet (Krizhessyy et al. 2012), and VGG-11/16/19 on CIFAR-10 with and without the dropout. Figure (eftt compares the strength of interactions' encoded by DNNs, which were learned with or without the dropout operation. When we learned DNNs with dropout, we set the dropout rate as 0.5. Please see Appendix H. for experiments on different dropout rates. We averaged the strength of interactions over images, i.e. I = Emaage (E (I (i))), where [(i,j) was obtained according to Equation (2).
Note that accurately computing the interaction of two input variables was an NP-hard problem. Thus, we applied a sampling-basee based method (Castro et al. 2009) to approximate the strength of in- teractions. Furthermore, we conducted experiment to explore the accuracy of the interactions approximated via the sampling- method. Please see Appendix K for details. Castro et al.) (2009) proposed method approximate the Shapley value, which can extended the approx- imation of the interaction. We found that dropout could effectively suppress the strength of the interaction, which verified the above proof.
Property: The sampling process in dropout is the same as the computation in the Banzhaf value. In this section, we aim to show that the sampling process in dropout (when the dropout rate is 0.5) is similar to the sampling in the computation of the Banzhaf value. Just like the Shapley value, the Banzhaf value (Banzhaf III, 1964) is another typical metric to measure importance of each input variable in game theory. Unlike the Shapley value, the Banzhaf value is computed under the assumption that each input variable independently participates in the game with the probability 0.5. The Banzhaf value is computed as ((iNN = EsgNy() Poussaat NS (()) [ {i)) - f(S)). where Poombua (S N (i)) 0.5"-1 is the likelihood of S being sampled. The form of the Banzhaf value is similar to that of the Shapley value in Equation (I). but the sampling weight of the Banzhaf value
For fair comparison, we normalize the value of interaction using the range of output scores of DNNs. Please Appendix F for more details.

---

Published as a conference paper at ICLR 2021
Dataset
Model
Ordinary
Over-fitted
Dataset
low lyye
LotSuaradoon
high layers
low layers
Dropout
high layers
MNIST
RN-44
2.17 10
3.64 10 3
AlexNet
CIFAR-1O
69.6
67.2
67.5
68.4
Tiny-Imane Inn-ImmaeNNet
RN-34
2.57 x 10 3
2.89 x189x100
VGG-13
CIFAR-10
66.2
CelebA
RN-34
6.46x10->
1.17x10-2
VGG-16
CIFAR-10
64.9
62.6
63.0
59.5
VGG-19
Tiny- ImageNet
45.2
37.4
32.6
37.0
Table I: Comparison of the interaction strength between the over-fitted samples and ordinary samples. DNNs usually encode more interactions for the over-fitted samples than ordinary samples.
VGG-16
CelcbA
94.6
Table 2: Classification accuracy of DNNs when
we applied the dropout operation and the inter- action loss at different positions.
Pruszaar (S) MME) is different from Pstugiey (+))) of the Shapley value. For dropout with the dropout rate 0.5, let n be the number of input variables, and let S be the units not dropped in N Vi Then, the
likelihood of not being dropped is given as Paropou (S) (N)())) = 0.5 (S Paunchar(S (S)))))
In this way, dropout usually generates activation units S following (S) (()). Therefore, the
frequent inference patterns in the computation of the Banzhaf value are also frequently generated by dropout, thereby being reliably learned by the DNN. This makes that f(S U {i}) f(S) in the computation of the Banzhaf value can be sophisticatedly modeled without lots outlier values. Thus, the dropout can be considered as smooth factor in terms of the Banzhaf value.
Grabisch & Roubens (999) defined the interaction based on the Banzhaf value as Iaanahar [s()((EJ) Piannnat [8] N (E, Af(,))
like Equation (2). Thus, dropout can also be consid- ered as a smooth factor in the computation of the Banzhaf interaction. Figure rightt shows that the target interaction used in this study closely related to the Banzhaf interaction, which potentially
connects the target interaction based on the Shapley value to the dropout operation.
4 UTILITY OF DROPOUT & IMPROVEMENT OF THE DROPOUT UTILITY
Close relationship between the strength of interactions and over-fitting. In this section, we conducted various experiments to explore the relationship between the interaction strength and the over-fitted samples. We noticed that in the classification task, the over-fitted samples were usually outliers. To this end, we assigned 5% training samples with random incorrect labels in MNIST (Le- cun et al. 1998), CelebA et al. 2015), and Tiny ImageNet (Le & Yang 2015). When the DNN was well- trained on such training data with an almost zero training loss, then we took samples with incorrect labels the over-fitted samples. We trained ResNet-34 (RN-34) (He al. (016) for the classification task using the Tiny ImageNet dataset and the CelebA dataset, and trained ResNet-44 (RN-44) for the classification task using the MNIST dataset. For each trained DNN, we divided input images into grids, and computed the average interaction strength between grids on the over- fitted samples and ordinary samples, respectively. As Table [hoows, the over-fitted samples usually contained stronger interactions' than ordinary samples.
Understanding of dropout. We have proved that dropout can decrease the strength of interactions encoded in DNNs in Section[] Besides, previous paragraphs have shown that the over- weer-fitted samples usually encode more interactions by the DNN than other samples. Therefore, we consider the utility of the dropout operation is to decrease interaction strength to reduce the significance of wver-fitting
Further improvement of the utility of dropout using the interaction loss. Based on the above understanding, we develop the interaction loss as improvement of the utility of dropout. To this end, we apply this loss to learn DNNs, in order to boost the performance, as follows.
Loss LosSatasificaiinn AL LosSmmeraction
where À >> is the weight of the interaction loss. Let us focus on an intermediate-layee feature after the ReLU operation h E I." activation units. We aim to suppress interactions between any two units j.j E N. Thus, the interaction loss can be formulated as Lossueeacco = Je ( The baseline value in the computation of [(i.j)) set to O, as explained in the paragraph understanding DNNs via game theory, Section
(8) However, is extremely computational expensive to use the above interaction loss to train the DNN. Therefore, we propose the following approximation of the interaction loss, which is implemented in batch manner, instead of averaging all pairs of activation units. Specifically, we sample disjoint
Loss BijeN. [1 BB..),e EsgnAna) Pshaptey (S) (S) (.NN(6.)))AA(((5..))
7

---

Published as a conference paper at ICLR 2021
Inwoddin
Inlecatton
ClassBottion
Accursey
Inteucker
Iranstoon
2
2 500
2003
2- 13100
Oppr
Aladdee
Figure 4: Curves of interaction strength encoded by DNNs trained using different weights of the interaction loss. The interaction strength decreases along with the increase of the weight X. For fair comparison, we computed the interaction between activation units before the layer, in which we used the interaction loss or the dropout operation.
subsets of units A, S N, 0, instead of sampling two single units i, compute for the interaction loss. Here, we can regard A and B as batches of the sampled units {i), (i), respectively.
Accordingly, the context disjoint with A and B. i.e. S S NYAA B. We can get the following approximation of mncamnctaction Please see Appendix F for the proof.
Lossimerration B
E. E BI A, B]]]]]
where
and a is small positive constant. In this paper, we set 0=0.5 Figure leftt) verifies the proposed approximated interaction loss can successfully reduce the interaction defined in Equation 2
Advantages. Compared with dropout, the interaction loss exhibits following advantages.
Advantage 1:
The interaction loss enables people to explicitly control the penalty of the interaction
strength by adjusting the weight À in Equation (7). The explicit control of the interaction strength crucial, because is important to make a careful balance between over-fitting and under-fitting during the learning of the DNN. The weight À needs to be carefully set, since it is usually difficult to trade off between over-fitting and under-fitting in deep learning. A large value of may lead to under-fitting, while small value of À may increase the risk of ever-fitting. In comparison, the dropout rate can control the strength of the dropout, when we consider the utility of the dropout as penalty of interactions. However, reduce the interaction than the dropout. To this end, we have conducted experiments to compare the strength of the dropout with the strength of the interaction loss, which verified the conclusion on the strength of the dropout. Please see Appendix H. for details.
find that the interaction loss exhibits stronger power to
Due to the disharmony between dropout and the batch normalization (Li al. 2019), people usually cannot apply dropout in the DNN with batch normalization. In comparison, the interaction loss is compatible with batch normalization. For example, we compute LosSunenctionn using a new track, which is different from the track of computing LosSeasitiationn In other words, we do not update the parameter batch normalization layers when we compute LosS Please see Appendix for more discussions.
Advantage 2:
Experiments: We trained DNNs for the classification based on the CIFAR-10 10 dataset (Krizhevsky &
Hinton| (009), the classification based on the Tiny ImageNet dataset (Le & Yang 2015), and the face attribution estimation based on the CelebA dataset (Liu et al. 2015). We trained six types of DNNs, including AlexNet (Krizhevsky et al. 2012). VGG-11/13/16 (Simonyan & Zisserman 2015), and ResNet- 18/34 18N31)/34) (He et al. 2016). Note that we only randomly sampled 10% training data in the CIFAR- 10 dataset and the Tiny ImageNet dataset, and randomly sampled training data form the CelebA dataset. Besides, we sampled over 80% units to generate the context S in experiments using ResNets, in order to ensure the stability of training. This was because DNN usually showed a low significance of over-fitting, when it was learned on the complete dataset. We only sampled a small proportion of training images to better show the over-fitting problem. In this experiment, when we trained DNNs with dropout, we set the dropout rate as 0.5. Please see Appendix H. for experiments on different dropout rates. Furthermore, we trained the BERT (Devlin et al. (2018) using the SST-2 dataset (Socher et al. 2013) in order to verify the effectiveness of the interaction loss on linguistic data. Please see Appendix H.2 for more details.
Learning DNNs with the interaction loss. We trained the DNNs with different weights of the inter- action loss, and evaluated their testing accuracy. Table 3 shows that the accuracy of DNNs trained with different values of À and the accuracy of DNNs trained with dropout. The accuracy of DNNs
We slightly adjusted the structure of DNNs to adapt them to input images with sizes 32 x 32 and 64 x 64, according to the classic and standard implementation of (pytorch-cifar 100 2020)
8

---

Published as a conference paper at ICLR 2021
50.0
0101
A
AlesNet
66.1
69.2
VOG-TF
63.9
61.9
VOG-TF
BIL
64.0
VGO-16"
68.8
62201
Timy ImaggeNet
0.001
0/
RN-T8F RNESF
45.3
48.4
45.6
50.0
030
A
VOG-16
38.4
154
VOG-T9
38.2
17.6
00
VDG-13
94.8
54.6
VOG-16
91.7
0.001
0.00
A
RX-18
93.0
V227
1iaee OoM (wer (wir
100,0
09.6
64.5
94.7
93.1
200.0
09.6
65.3
65.9
64.7
0.01
52.1
49.6
200.0
38.2
39.0
20.0
94.9
94.1
0.01
93.0
500.0
70.0
65.9
66.2
64.9
0.03
50.4
48.8
500.
42.8
41.8
50.0
94.7
94.08
0.03
92.9
1000.0
64.3
66.3
66.0
64.5
1000.0
40.8
45.2
1000
94.7
94.3
Dropout
67.5
60.9
60.9
63.0
Dropout
47.4
46.0
Dropout
36.8
Dropou
94.6
92.4
Dropout
92.1
Table 3: Classification accuracy when the DNNs are controlled from over-fitting to under-fitting
usually increased along with the À at first, and then decreased when the value of À continued to in- crease. This could be explained as the interaction loss alleviated the significance of over-fitting when the weight À was small, and then caused under-fitting when the weight À large. Furthermore, the DNN trained using the interaction loss usually outperformed the DNN trained using the dropout, when the weight À was properly selected. This verified that the interaction loss would improve the utility of the dropout.
Figure 4 shows curves of the interaction and the accuracy of DNNs during the learning process with different weights A of the interaction loss. First, we found that the interaction decreased along with the increase of the weight X, which verified the effectiveness of the interaction loss. Second, when the weight À was properly selected, the DNN trained with the interaction loss usually outperformed the DNN trained with dropout. The interaction of the DNN trained using the interaction loss was much lower than the interaction of the DNN trained using the dropout. In other words, the interaction loss was more effective to suppress the interaction and boost the performance than the dropout. In particular, Table 3 shows that the performance of ResNets sometimes dropped when these DNNs were learned with dropout. This phenomenon also verified the conclusion in (Li et al. 2019), i.e. dropout was not compatible with batch normalization. In comparison, learning with the interaction loss did not suffer from the use of batch normalization.
Exploring the suitable position for the interaction loss and the dropout operation. We learned the AlexNet and VGG-13/16 using the CIFAR-10 - dataset, learned the VGG-19 using the Tiny ImageNet dataset, and learned the VGG-16 using the CelebA dataset. For each DNN, we put the dropout operation and the interaction loss in the low convolutional layer (before the 3rd/5th convolutional layer of the AlexNet/VGGs) and the high fully-connected layer (before the 2nd fully-connected layer), respectively, and compared their performance. As Table 2 shows, the interaction loss was better to be applied to the feature of a low convolutional layer. In comparison, we did not find out a clear principle about the suitable position for dropout.
Discussions about advantages of the interaction loss. As mentioned above, the interaction loss exhibits two advantages over dropout. Table 3 shows that the interaction loss enabled people to explicitly control the interaction strength. When the À value increased, DNNs were controlled from over-fitting to under-fitting. Both the over-fitting problem and the under-fitting problem would hurt the performance. However, this explicit control could not be achieved by dropout. Furthermore, unlike dropout, the interaction loss was compatible with batch normalization. As Table shows, the use of dropout decreased the classification accuracy of ResNet-18 (RN-18) on the Tiny ImageNet dataset by 1.4%, and decreased the classification accuracy of ResNet-18 (RN-18) on the CelebA dataset by 0.6%. In comparison, the use of the interaction loss does not suffer from the hatch normalization operation. For fair comparison, the interaction loss and dropout were put in the same position of the DNN.
5 CONCLUSION
In this paper, we explain, model, and improve the utility of the dropout operation using game theory. We prove that dropout can reduce the strength of interactions, thereby improving the performance of DNNs. Experimental results have verified this conclusion. Furthermore, based on the close rela- tionship between the interaction and over-fittin, we propose an interaction loss to directly penalize the strength of interactions, in order to improve the utility of dropout. We find that the interaction loss reduce the interaction strength effectively and further boost the performance of DNNs.
9

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work is partially supported by National Natural Science Foundation of China (61906120 and U19B2043).
REFERENCES
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a poly- nomial time algorithm for shapley values approximation. 1903. 2019.
John F Banzhaf III. Weighted voting doesn't work: A mathematical analysis. Rutgers L. Rev, 19: 317, 1964.
Jacob Bien, Jonathan Taylor, and Robert Tibshirani. A lasso for hierarchical interactions. Annals of Statistics, 41(3::111111411 2013.
Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the shapley value based on sampling. In Computers & Operations Research, 36(5):1726-1700 2009.
Jacob Devlin, Ming-Wee Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv: (810:00000000005,, 2018.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. In ICMI, pp. 1050-1059, 2016.
Hongchang Gao, Jian Pei, and Heng Huang. Demystifying dropout. In ICML, 2019.
Aidan N. Gomez, Ivan Zhang, Kevin Swersky, Yarin Gal, and Geoffrey E. Hinton. Targeted dropout. In NIPS Workshop, 2018
Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of Game Theory, 28:547-565, 1999.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv: Neural and Evolutionary Computing, 2012.
Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. arXiv:2002.044 2020.
Xisen Jin, Junyi Du, Zhongyu Wei, Xiangyang Xue, and Xiang Ren. Towards hierarchi- importance attribution: Explaining compositional semantics for neural sequence models. arXiv: 1XN1v:991..06/99 2019.
Kishore Reddy Konda, Xavier Bouthillier, Roland Memisevic, and Pascal Vincent. Dropout data augmentation. In ICLR, 2016
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. In Computer Science Department, University of Toronto, Tech. Rep, 1, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 2015.
Y. Lecun, L. Bottou, Y. Bengio. and P. Haffner. Gradient- based learning applied to document recog- nition. Proceedings of the IEEE, 86(11) 86(11):227-23244 1998.
Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the disharmony between dropout and batch normalization by variance shift. In CVPR, pp. 2682-2690, 2019.
10

---

Published as a conference paper at ICLR 2021
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Scott Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv: Learning. 2018.
W James Murdoch, Peter J Liu, and Bin Yu. Beyond word importance: Contextual decomposition to extract interactions from Istms. A.Kiv:8801.05555 2018.
pytorch-cifar100. Proper implementation of DNNs CIFAR-10 using PyTorch. https: 2020.
Jie Ren, Die Zhang, Yisen Wang, Lu Chen, Zhanpeng Zhou, Xu Cheng, Xin Wang, Yiting Chen, Jie Shi, and Quanshi Zhang. Game-theoretie understanding of adversarially learned features. arXiv, 2103.07364, 2021.
Lloyd S Shapley. A value for n-person games. In Contributions to the Theory of Games, 2(28): 307-317, 1953.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Chandan Singh, W James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. arXiv:1806.053:0 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642. 2013.
Daria Sorokina. Rich Caruana, Mirek Riedewald. and Daniel Fink. Detecting statistical interactions with additive groves of trees. In ICML, pp. 1000-1007, 2008.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1955 1958, 2014.
suinleelab. Implementation of the integrated hessian. httpps///gituub..cm////slllee bath_explaing 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, pp. 3319-3328, 2017.
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In ICLR, 2018
Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. Feature inter- action interpretability: A case for explaining ad-recommendation systems via neural interaction detection. In ICLR, 2020.
Stefan Wager, Sida Wang, and Percy Liang. Dropout training as adaptive regularization. In NIPS, pp. 351-359, 2013.
Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach interpreting and boosting adversarial transferability. arXiv, 2010.04055, 2020.
Robert J Weber. Probabilistic values for games. The Shapley Value. Essays in Honor of Lloyd S. Shapley, pp. 101-119, 1988.
Die Zhang, Huilin Zhou, Hao Zhang, Xiaoyi Bao, Da Huo, Ruizhao Chen, Xu Cheng, Mengyue Wu, and Quanshi Zhang. Building interpretable interaction trees for deep nlp models. In AAAI, 2021a.
Hao Zhang, Xu Cheng, Yiting Chen, and Quanshi Zhang. Technical note: Game-theoretic interac- tions of different orders. ArXiv, 2010.14978. 2020.
Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dons In AAAI, 2021b.
11

---

Published as a conference paper at ICLR 2021
A FOUR DESIRABLE PROPERTIES OF THE SHAPLEY VALUE
In this section, we mainly introduce the four desirable properties of the Shapley value mentioned in Section 3 including the linearity property, the dummy property, the symmetry property and the efficiency property (Ancona et al. 2019)
Linearity property: Given three games f. 9, and h. If the reward of the game f satisfies f(S) = ((S) + h(S), then the Shapley value of each player i E N in the game f is the sum of Shapley values of the player i in the game ! and h, i.e 6y(i]N) = (i) + on (i)N).
Dummy property: In game S, If a player i satisfies VS C N \ (3), ((SU()) ((S) + f(()),
then this player is defined as the dummy player. The dummy player i satisfies ((i = s((i))
f(0), Le. the dummy player has no interaction with other players in N.
Symmetry property: Given two player i and j in a game f. if SA C N (i.j). (i)) U (i)). then c(i = M(j
Efficiency property: For a game f. the overall reward can be distributed to each player in the game. Le. EiEN ((i] J(N) f(0), where f(0) is the obtained reward when no player participates in the game.
B PROOF OF EQUATION
This section provides the detailed proof for Equation 44 in Section be decomposed follows.
The s-order interaction can
(1.))
Σ (
(10)
SCN((L)) 151=0
TCS
Σ TCS
H (1.1)
(11)
(CN) 18|=0
Σ
R (1,5)
(12)
1
8 (in'
LoSasa
TCS
18|=*
Ti=9
Σ
Σ
RT (1.9)
(13)
(SC)) TCS T|=9
[s|=s
Σ
E
Rd (1.5)
%et = T
(14)
and = 0.
0Sy5*
[T|=]
IT'1=009
8 8-9 (m=2-9) 9
nt (1.0))
Σ
(15)
0S45*
(CN)
n 9--
n
R² (i.j)
(16)
5-0
q
TCNMid)
0595=
1T
J9(L,)
Σ
(( = Σ (1,
(17)
05955
US950
In Equation 17). Jd (i,r) = ErrgM [RR (i. denotes the average interaction between i and given all potential inference patterns TÚ 5i with fixed inference pattern size [T] q-
12

---

Published as a conference paper at ICLR 2021
C PROOF OF EQUATION (5) AND EQUATION 66 LEFT)
This section gives the detailed proof for Equation ( and Equation (6 left) in Section 3 We first
prove Equation 5p. The interaction of i and with dropout, I (t, can be computed as
Lingoal (2,5)
E
E
Σ T
R (2.5)
(18)
8 (c,)
5'58 8 15" my
19 -
E (c.) 18|
E s'Cs |S =r
Σ T5.8"
n² (i.3)
(19)
Σ T
R² (2,3)
(20)
SC (CN) =8|=*
S'ss =5
Σ
E
Σ TES'
Re (1.1)
% similar to Equation
(21)
13 Equation (66).
8" (6,,)
=S-
n2 2-2-1
Σ
Σ TCS'
R² (1.))
(22)
8 8-r
S'EN(Ed) 1r
Σ
Σ
R (1,1))
(23)
(s.)
TCS
1s"|=r
Σ TS8'
Re (1.i) (1.))
(24)
s s'CNV6J) 1r
(1.
% similar to the proof of Equation 14
(25)
Thus, Equation 55) holds.
In order to prove Equation 6 left), we aim prove Yr S and
rio (i.jir) ri
1 S 1, we have [a x) step by step.
0. Next, prove the three inequalities
((
((j))))
rlss-) (r-g)
r((s-))! ((1-1)=-((1-+1) (r- g]]s! ((-1))-(s--tt) I 1.
First, ((.(e.)))))
((0()))
((o()r))))
p(+ (1rlr)
r-(r 1) (r- (r-++))
/r(r-1) ((--))-(ا(( s- > 1.
Next,
(1))))))))
-(--I)--- (+++++++ r-9
((.rr)
Then, (1.Jlc)))))
1
x-q
Thus, we prove Equation
6 left).
D THE COMPUTATION OF Rd (i.j)
This section gives recursive formulation of R (1, mentioned in Section 3 R (t, measures
the marginal reward of the inference pattern T J (1 where all interaction effects from smaller inference patterns T' T are removed from Rd (i,]). In this way, R R (i, can be computed as follows.
Rºd (i,j) del AS(T,
- Σ R² (i.i) TCT
(26)
13

---

Published as a conference paper at ICLR 2021
Next, prove the correctness of the formulation of R" (i, j). In other words, aim to prove (i.)) = EsgN(( Ergs (1,,)) in Equation (4
RHS = EsC
S (i.j)
Σ (1,5) TCS
(27)
the marginal reward ohtained from the inference pattern S {},
the marginal reward ohtained from inference patterss smaller than (1,
according to Equation (26).
(28)
(i,) = LHS
% according Equation 3
(29)
Thus, we prove the correctness the recursive formulation of R RT (i.
E THE COMPUTATION OF THE STRENGTH OF INTERACTIONS
In implementation, we randomly sampled 10 images for the computation. We divided each image into 16 x 16 grids in experiments to further boost the computation efficiency. We consider each grid as an input variable, and computed the interaction between two grids. We randomly sampled 80 pairs of grids for computation of the interaction. We averaged the strength of interaction over all sampled pairs of grids as the result of the interaction. Besides, for fair comparison of interactions among different DNNs, we normalized the value of interaction using the range of output score of DNNs. Le. mnom = I I/Y, where Y was the normalization term that reflected the range of the output score of the DNN. If the DNN was learned for multi- multi-category classification, then Y |E (xx), Eyssu' [f(x))]]], where was sampled from the training set X, y was the fround-trutie label of æ. and f(x), denoted the value of the y-th dimension in the output score f(x). If the DNN was learned for binary classification and the output score was a scalar, then Y = x [f() BueX [x))). where Xpostee and Xaegatie represent the set of positive samples and the set of negative samples, respectively.
F PROOF OF THE APPROXIMATION OF THE INTERACTION Loss
This section gives the proof for the trustworthiness of the approximated interaction loss Loss in Equation (.. The relationship between Equation (8) and Equation (9) is proved as follows.
According to Equation (3). Equation (8), and Equation (??), the interaction loss can approximated by sampling-based method, i.e.
LosSinteraction EijeN (1) = E.jeN [[,E,N[[((( [I,))
-B. Bi,jEN F. B (.s.::ss..SNN/A
(30)
Let denote the strength of the interaction / (ii, [). i.e. [[ii]]] / 0. We assume that Af(S
follows the Gaussian distribution, i.e. (f i,)) ~ Gaussian (f, f I(i,j). In this way, given specific pair of units i and j. we have
P(AJ(S,L,)).)) C- exp( (AF(S,i,)) 11)2 )p(p)
2a²
(31)
(Af(S,i,) i. + 11)2
P(Af(S, - -
}p(-p),
202
14

---

Published as a conference paper at ICLR 2021
where C
V2To is constant value.
Thus, given the value of Af(S, the likelihood of
I(i,5) = +u given as
P(AS(S.L,)).))
P(uIAf(S.L,5)) = P(Af(S.L.)).) + P(A
expp (A(18A)) 10) }p(u)
20
% p(u) = p(-p)
e'pxp( (A((S,)))) 1 (p()) + C- exp( (AS(S...))=)) )p(
20
2
expl 20 (AS(S, (S) 11)2]
expl 20
(A (S, (. 2=) + expl 20
1
+ exp(
(2Af(S,i,)))
sigmoid
d²
(32)
To simplify the story, let us first consider some easy cases. During the learning process, P(ulAf(S,i,)) i,j)) = 1, then it indicates [ii,] = p > 0. In this case, we need to decrease AS(S, j) to decrease [(iij)), To this end, we need to set the gradient of (f to be 1. In comparison, if P[n|A[(S (S,,,)) Rs 0, then it indicates I(i,j)
- 0, and we need to in- crease Af(S, to reduce [((, To this end, we need to set the gradient of i. to be consider the probability Pf (S,
-1. In this way, for other cases, of setting the gradient of A i, to 1, and consider - P(u (S,, (S,, j) as the probability of setting the gradient of AS(S, to -1. Therefore, the expectation of the gradient can be computed as P( f(S, j) + (1 - (( i,))) (-1) = ff - 1. Furthermore, the gradient of A i, can be approximated
the probability
2Af(S,i.))p
2P(M) (,)) 2 sigmoid
g²
2
% We assume that Af(S, is close to zero
+ exp( 23,($,
2 2
1
+
#
A((S,
-
2 20
02 i
(33)
Thus, the interaction loss can be rewritten follows.
Lostinteraction RJ EEJEN
Egg Ess
Af(S,
20
E A
P EigAjEB E, Bist=A,SCN 2
AS(S
E BABBSNAnB=0/AABBBaa
Eg = 20 AS(S, A, B)²
(34)
where
n0 is positive constant. Therefore, we can use Equation to approximate the inter-
action loss.
G DETAILS OF THE INTERACTION AND THE INTERACTION Loss
Implementation of the interaction loss: In this section, we aim to introduce the implementation details of the interaction loss. Since the computation of the interaction loss in Equation X is expen- sive, we propose Equation
9 as an approximation of the interaction loss.
As the Equation shows, need to sample subsets of units A, B, and S for the computation of the interaction loss. For and B, we randomly select 5% units from the intermediaae-layer feature, respectively, and A n B 0. As for the subset S, we sampled units from N B.
15

---

Published as a conference paper at ICLR 2021
Specifically, we first uniformly select a sampling rate from the range [0. 1], and sample the subset with this sampling rate. When we apply the interaction loss to DNNs with batch normalization layers (such ResNets), need to compute the interaction loss using different track with the computation of the classification loss. I.e. when we compute the classification loss, parameters in the batch normalization layers are normally updated. In comparison, when we compute the interaction loss, we directly use the parameters in batch normalization layers without updating their values. In this way, we can apply the interaction loss to DNNs without causing disharmony with batch normalization layers.
The selection of the output score: In this paper, we consider DNN f as a game, and take the output score of the DNN as the reward score. Since the reward score is usually a scalar value, for the DNN with a vectorized output, we usually take the output score before the softmax layer corresponding to the true class as the reward score. Specifically, for DNNs with batch normalization layers, such as ResNets, we take the output score after the softmax layer corresponding to the true class as the reward score.
Details of the datasets and training settings Section 4 the first experiment of Section f we used datasets with incorrect labels to reveal the relationship between the interaction and the over- fitting of DNNs. For the Tiny ImageNet dataset, we used all images from the first ten categories for the classification task, and each category contained 500 images. For the MNIST dataset, we randomly selected 10% images for training the DNN. In this case, each category contains 600 im- ages. For the CelebA dataset, we randomly selected 1 1% training samples for the estimation of the gender. In order to build datasets with the over-fitted samples, we randomly selected 5% samples of the training samples, and replaced their labels with a randomly selected incorrect labels. In this case, when the DNN was well-learned and had a training loss close to zero, samples with incorrect labels could be considered as the over-fitted samples. comparison, samples with correct labels were ordinary samples. Besides, in this experiment, the strength of interaction was normalized by the normalization term of each individual image.
In the following experiment of Section 4 we explored the utility of the interaction loss. We trained DNNs using the CIFAR-I0 dataset, the Tiny ImageNet dataset and the CelebA dataset. In order to make the DNN suffer from the over-fitting problem, for the CIFAR- 10 dataset and the Tiny ImageNet dataset, we randomly generated 10% training samples to train the DNN. More specifically, we only used the first 10 categories for the classification task. Besides, for the CelebA dataset, used 1% training samples to train the DNN for gender estimation. In these experiments, the strength of interactions was normalized by the normalization term over all training samples.
For fair comparison, the interaction loss and dropout were put in the same position of the DNN. Specifically, for AlexNet, we put the interaction loss dropout before the third convolutional layer. For VGGs, we put the interaction loss or dropout behind the second block, i.e. behind the second We put the interaction
ReLU layer of VGG-11 and behind the forth ReLU layer of loss or dropout behind the first block for ResNet-18 18 (RN-18), and behind the second block for ResNet-34 (RN-34). When we put the interaction loss or dropout at a high layer, we always put the interaction loss or dropout behind the first fully-connected layer.
The baseline value: In this section, we introduce details about the baseline value, which was pro- posed by (Ancona et al., 2019). Ancona et al. (2019) proposed polynomial-time algorithm for Shapley values approximation to explain deep neural networks. When we estimate the attribution of input variables and the interaction between input variables, we usually use baseline value to simulate the absence of variables. Many methods of estimating attributions require the definition of a baseline value as the anchor value corresponding to the absence of the variable. Setting an input variable to the baseline value indicates this input variable is toggled off. Let us focus on the task of object classification. We follow the settings in (Ancona et al., 2019). If we aim to measure the interactions between input pixels, the baseline value is defined as the average pixel value among all samples. When we measure the interaction between activation units before the dropout operation, the baseline value is defined as zero
16

---

Published as a conference paper at ICLR 2021
Dropout rates
Weights of Lossuettiion
0.1
0..
100.0
500.0
1000.0
Accuracy
VGG-11
AlexNet
66.2
61.9
67.1
61.7
66.6
61.5
67.5
60.9
66.1
58.1
57.0
28.5
69.2
63.9
69.6
64.3
69.6
65.3
70.0
65.9
64.3
66.3
Interactionxx1077
AlexNet
21.73
15.90
8.77
8.39
5.24
3.93
1.40
1.18
0.76
0.50
0.23
VGG-11
9.26
9.21
9.15
8.01
6.17
2.69
0.08
0.04
0.03
0.02
0.01
Table 4: (left) Classification accuracy and the interaction of DNNs with dropouts when tried different dropout rates; (right) classification accuracy and the interaction of DNNs trained with different weights Ả of the interaction loss.
Weights I LoSSmerrction
Dropout
0.0
0.5
1.0
5.0
10.0
Accuracy
77.1
76.3
77.1
78.0
77.0
76.5
Interaction (x10->)
24.06
36.89
0.17
0.15
0.09
0.08
Table 5: Classification accuracy and the interaction the BERT with the dropout (dropout rate=0.5), and Classification accuracy and the interaction of BERTs with different weights À of the interaction loss.
Dropout
LoSSimteractin
3rd conv
1st fc
2nd fc
3rd fc
AlexNet
I1min
23min
AlexNet VGG-11
67.5
69.4
68.4
67.6
VGG-11
12min
29min
60.9
58.1
57.6
60.9
Table 7: The time cost training DNNs using LoSSinteration and the time cost of training DNNs with the dropout.
Table 6: applied the dropout operation before different layers.
Classification accuracy of DNNs when we
H MORE RESULTS
H.1 THE COMPARISON OF DROPOUT USING DIFFERENT DROPOUT RATES.
In this section, we conducted new experiments to further study the performance of different dropout rates. We trained AlexNet and VGG-11 using the CIFAR-10 dataset. We set the dropout rate as 0.1, 0.3, 0.5, 0.7, 0.9, respectively, and applied the dropout operation before the 3rd convolutional layers of the AlexNe//VGG-11 which was the same as in experiments corresponding to Table 3 Table 4 shows the classification accuracy of different DNNs, well the strength of interactions modeled by DNNs. We found that the dropout rate of 0.9 usually penalized more interactions than the dropout operations with other dropout rates. Nevertheless, DNNs trained using the interaction loss still decrease more interactions than DNNs trained with the dropout in most cases. In addition, the DNN trained with a proper setting of outperformed all DNNs with the dropout.
H.2 EXPERIMENTS ON LINGUISTIC DATA
In this section, applied our interaction loss to another type of data, i.e. the linguistic data. We trained the BERT model (Devlin et al., 2018) on the SST-2 (Socher et al.) 2013) dataset for the binary sentiment classification task. Just like experiments ImageNet dataset, we randomly sampled 10% training samples. This was because the DNN showed a low over-fitting level, when was trained using the complete dataset. We set the dropout rate as 0.5, and applied the dropout/interaction loss after embedding layers of the BERT. We designed two baselines. The first baseline was the DNN trained with the traditional dropout. The second baseline was the DNN trained without the dropout the interaction loss. Table shows that the interaction loss successfully decreased the interaction modeled by the BERT, and outperformed two baselines. Therefore, the interaction loss more effective method to control the interaction, and boost the performance of DNNs than the dropout.
the CIFAR-10 dataset and the Tiny
H.3 THE COMPARISON BETWEEN DIFFERENT POSITIONS TO APPLY THE DROPOUT
In order to verify that the dropout is applicable to convolutional layers, in this section, we conducted experiment to apply the dropout various fully-connected layers and the convolutional layer.
17

---

Published as a conference paper at ICLR 2021
Interaction with DNN autput
17
Interaction with
Interaction with
50.0
Interaction with DNN oulput
1.0
f Loss
Loss
100.0
0.8
= 1000.0
0.2
0.0
0.0
dropout
150
250 Spach
Spoch
Figure Curves of interactions when respectively, The interaction with f Loss had similar behaviors with the interaction with f DNN output.
DNN output and when we set f Loss,
Specifically, applied the dropout before the 1st/2nd/3rd fully-connected layers and before the 3rd convolutional layer of the AlexNet and VGG-11. this experiment, we set the dropout rate 0.5. Table 6 shows the accuracy of DNNs with different positions, to which the dropout was applicd. We found that DNNs trained with dropout on the convolutional layer had similar perfor- mance with DNNs trained with dropout on fully-connected layers. Thus, dropout applicable convolutional layers.
H.4 THE COMPARISON OF COMPUTATIONAL TIME BETWEEN DNNs TRAINED USING DROPOUT AND IOSSATTEATTION
In this section, we compared the time of learning DNNs with dropout and the time of learning DNNs using the interaction loss. We trained AlexNet and VGG-11 using the CIFAR-10 dataset on GPU of GeForce GTX-1080Ti. The experimental setting was the same as that in experiments corresponding to Table 3 and Figure As Table 7 shows, our interactions loss increased the time cost, but was still of specific value considering its good performance and the harmony with the batch-normalization operation.
H.5 THE COMPARISON BETWEEN TAKING THE OUTPUT AS THE SCORE AND TAKING THE LOSS VALUE AS THE SCORE f
In this section, we aim to measure the interaction when we used the loss value of the DNN as the score f. We trained the VGG-11 using the CIFAR-I0 dataset. Then, we compared the interaction when we set f Loss with the interaction when we set f as the output score of the DNN. As Figure 5 shows, the interaction with f = Loss had similar behaviors with the interaction with the network output as Therefore, the choice of the score f did not affect the conclusion that the interaction loss could effectively decrease interactions encoded the DNN.
I DETAILS FOR VISUALIZING INTERACTIONS
This section provides the implementation details visualize interactions modeled by DNNs. We used the CelebA dataset for visualization. There were 224 x 224 pixels in images of CelebA, which made it expensive to compute and visualize the interaction between any pairs of pixels. Thus, we divided the image into 16 x 16 grids, each grid contained 14 x 14 pixels. We considered these grids as input variables, and computed the interaction between any two adjacent grids. We sampled 100 times to approximate the interaction between two grids. For specific grid, we averaged the interactions between the grid with its adjacent grids, and used the strength of the averaged interaction as the attribution value of this grid.
J COMPARISON BETWEEN OUR INTERACTION AND THE INTEGRAL HESSIANS
The Hessian matrix is another typical way to define interactions. However, the standard way of using the Hessian matrix to define interactions is not to directly use the raw Hessian value V but the Integrated- Hessian value that has been proposed by (Janizek et al. 2020)
1. L
18

---

Published as a conference paper at ICLR 2021
Interaction x10
Insiablitt
Raw Hessian
5.09
0 0.
Integrated Tesssan
0.81
VGG-11
Our interaction
5.76
0.02
sampling
Comparison of different types of interactions, including the raw Hessian value, the Eneggaaed-Hessiann value, and the interaction value defined in this paper.
Table 8:
Figure 6: Instability of the interaction approximated with different sampling numbers. The instability of the approximated interaction decreased along with the in- crease of the sampling number.
Janizek et al. (2020) proposed the Enegrated-Hessiaan value measure interactions, based on Inte- gral Gradient (Sundararajan et al. 2017). Integral Gradient is used to compute an importance value for each imput variable w.r.t. the DNN. Given input vector x E IR", Integrated Hessians measures the interaction between input variables (dimensions) I; and z, as the numerical impact of I on the importance of 24- To this end, Jaanzekk al.) (2020) used Integral Gradient to compute Integrated Hessians. More specifically, given DNN and two input variables and zz. the interaction between I; and øy is measured as follows.
[i. = (z. - , - 2//) f l' S' aB f(x' + +a3(x - x')) dad3,
J a==
02,02,
where x' is baseline image.
Theoretically, the Integrated- value is not equal to our interaction metric, and has different theoretic foundations with interaction metric. Our interaction metric defines the interaction based on the Shapley value (a game-theoretic importance). In comparison, the importance value used value is computed using the Integral Gradient. Strietly speaking, the
to define the antegrated-Hessiie Integral-Grade Gradient importance is highly related but not equal to the Shapley value (importance). The Shapley value is the unique importance that satisfies the linearity property, the dummy property, the symmetry property, and the efficiency property (Shapley 1953 Weber 1988
Ancona et al. 2019).
Compared to the raw Hessian value V2, I., the Eitegrated-Hessian value integrates high-order gradients to generate the final Integrated- -esssan value. Unlike our interaction metric, the computation of Integrated Hessssan values requires the DNN use the SoftPlus operation to replace the ReLU operation. The computation of our interaction metric does not have such requirement.
multiple
To this end, conducted an experiment to show the difference between the raw Hessian value Li the Integrated- Hessian value (Janizek et al. 2020), and our interaction metric. We imple- mented the Integrated Hessian using the code released by (suinleelab) 2020 We directly used the code and the DNN architecture provided by (suinleclab) 2020) to train the DNN on the MNIST dataset, where all ReLUU layers in the DNN were replaced with the SoftPlus operation (this was required by (Janizek et al. 2020) for the computation of the Integrated- -eessaa value). Given the trained DNN, we computed the strength interactions based on the raw Hessian value, the Integrated-Hessinn value, and the interaction value defined in this paper, respectively. As Table R shows, these metrics generated fully different interaction values, because these interaction metrics defined different types of interactions from fully different perspectives.
K EVALUATION OF THE ACCURACY OF THE APPROXIMATED INTERACTION
In this section, we conducted an experiment to explore the accuracy of the interactions approxi- mated via the sampling-basse based method (Castro et al.] 2009). More specifically, given a certain im- age and certain sampling number, we measured the instability of the average interaction strength Timage Eimgge) ()),, when we repeatedly computed the interaction for multiple times considering the randomness in the sampling process. This instability value indicated whether obtained similar interactions with different sampling states. The instability w.r.f the interaction
Limage is computed as E,, sruge [r()] (1m)
where ((()) denotes the interaction result computed in
E (())
the w-th time. Then, we computed the average instability value over interactions of 100 pairs of
input variables in 10 images to measure the accuracy of the approximated interaction. A high insta-
19

---

Published as a conference paper at ICLR 2021
bility indicated low accuracy. We used the trained AlexNet and VGG-11 I to compute the instability of interactions. We computed the instability with different sampling numbers. Figure shows that the instability of the approximated interaction decreased along with the increase of the sampling number. We found that when the sampling number was large than 500, the instability of the approx- imated interaction
less than 0.1, which indicated that the approximated interaction and accurate enough. Thus, we set the sampling number as 500 in all other experiments in this paper. Meanwhile, computing the interaction using such a sampling number was far more efficient than the original NP-hard computation of the interaction.
stable
20


Reviews:
[{"review_id": "jFQN64PMJYa", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary.\n\nThis paper aims to explain dropout from the lens of game theoretic interactions. Let x denote the input of a deep neural net (DNN), intuitively, the interaction between two variables x_i and x_j quantifies how much the presence/absence of the j-th variable affects the contribution of the i-th variable to the output of the DNN. With the above definition in place, the authors show theoretically and empirically that dropout reduces the interactions between input variables of DNNs. As this type of interactions turn out to be strongly correlated with overfitting, the authors suggest that dropout alleviates overfitting by reducing interactions between input variables (or activation units) of DNNs. Based on this understanding of dropout, an alternative regularization technique is proposed, which explicitly penalizes pairwise interactions between variables. \n\nStrengths.\n\n1. The paper is well written and clearly presented.\n\n2. Although it is already well known (or at least widely accepted) in the community that dropout reduces dependencies among activation units in DNNs, the explanation of dropout from the perspective of game theoretic interactions is interesting, and it is supported both theoretically as well as empirically.\n\nWeakness.\n\n1. Hyperparameter settings (e.g., optimization-related ones) to reproduce the results are not provided. It is not clear what dropout rate was used in the experiments. Is it 0.5? In all cases, different dropout rates should be investigated before claiming the superiority of the proposed interaction loss (regularization) over dropout.\n\n2. Experiments are curried out on only one task (classification), one type of data (images), and one family of DNNs (convolutional neural nets). However, the paper draws quite general conclusions regarding the understanding of dropout from the perspective of game theoretic interactions. Therefore, considering at least one more task involving a different type of data and another family of DNNs would reinforce the findings of this paper.\n\n3. Computational time analysis of the proposed interaction loss and training time comparisons with dropout are lacking.\n\nAdditional comments\n\n1. Dropout is used both at convolutional and at fully connected layers. However, one can argue that applying dropout to convolutional layers does not make sense owing to the sparsity of connections in this type of layers.\n \n2. I would recommend revising the title of the paper. What is proposed is more of an alternative regularization form to dropout than an improvement for the latter. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JizU1JDUyKm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary：\nThe paper proves that dropout can suppress the strength of interactions between input variables from the perspective of game theory. It further improves the utility of dropout by introducing an explicit interaction loss. Experimental results verify the theoretic proof and the effectiveness of the proposed loss.\n\nStrengths:\n1. The paper introduces a new perspective of game theory to understand dropout.\n2. Experiments are conducted on various datasets to support the theoretic proof and the proposed interaction loss\n\nConcerns:\n1. Although I have no background in game theory, I try my best to understand the terminology and the analysis. However, I do not have the ability to verify the correctness of its proof. Thus, I cannot evaluate the main contribution of this paper. For experimental results, the conclusion that dropout suppressing the input interactions is not a new story.\n2. It would be more interesting if the author can further explain the disharmony between dropout and bn from the perspective of game theory. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "APHGjfARov", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Paper Summary*\nThe authors provide a novel interpretation of dropout regularization using Banzhaf interactions, a tool from game theory. \n\n*Pros*\n* The authors are able to mathematically prove that dropout is capable of suppressing neural co-adaptations, the latter being one of the reasons for overfitting. Visualizations are also provided in this respect on a dataset for face analysis.\n* Through their mathematical analysis, authors are able to improve upon the classical dropout training, by making it more compatible with batch normalization, so that these two classical regularization strategies show a better complementarity.\n\n*Cons*\n* Some of the results does not read well, like Table 3 or Figure 4, but this is really minor and fixable\n\n*Preliminary Evaluation*\nI believe that the overall analysis provided by authors is complete and interesting, so I am akin to call for a full acceptance of the paper which I deem suitable for such a venue like ICLR. In order to improve their paper, I would encourage authors to better investigate over the following aspect: since many times authors established a principled connections between dropout and neural activations, it would be very interesting to discuss the relationship with the present work and another paper [Gomez et al, Targeted Dropout, NeurIPS Workshops 2018] in which a computational variant of dropout is proposed, such that the dropout rate depends upon neural activations. \n\n*Post-Rebuttal Evaluation*\nI have carefully read the response provided by authors and checked the revised manuscript. I confirm my preliminary acceptance rate.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6qJ6mqTcDe9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper analyzes the effect of dropout on interaction between units in a neural network. The strength of the interaction is measured using a metric that is used in game theory to quantify interaction between players in a co-operative game. The paper shows that dropout reduces high-order interaction (as measured by this metric), and that reduction in interaction is correlated with better generalization. The paper introduces a new regularizer that explicitly minimizes the metric and claims that using this regularizer instead of dropout has some advantages.\n\nPros:\n- The idea that dropout reduces overfitting by breaking up complex co-adaptations and regularizing interactions is widely believed to be true. However, this paper tries to explicitly quantify the amount of interaction and presents theoretical and experimental evidence that interaction reduces as a result of having dropout.\n\nCons:\n- The proposed metric is hard to compute exactly since it requires summing over exponentially many terms, each term requiring a forward prop through the network.\n- The assumptions made in computing this metric approximately seem unclear to me (Appendix H). I could not understand what probability distributions are being expressed and why. In particular, how is the term in Eq 38 approximated by the one in the first line of Eq 41. The paragraph after Eq 40 was also unclear.\n- It is not discussed how this metric for evaluating interaction strength compares to something conceptually simpler like the Hessian \\\\(\\nabla^2_{i,j} L\\\\) which directly measures the dependence of the network's loss on pairs of input variables, and its magnitude is proportional to the interaction strength.\n- The paper mentions that an advantage of the proposed loss is that the weight \\\\(\\lambda\\\\) applied to the interaction loss can be explicit controlled, whereas the strength of dropout cannot be controlled (Section 4 \"advantages\", \"Unlike the interaction loss, people cannot explicitly control the strength of dropout ..\"). This does not seem correct. The dropout probability provides such as control mechanism for dropout.\n- For the experimental results in Table 3, it is not mentioned what value of the dropout probability was used, whether this value was tuned for each architecture, and which network layers was dropout applied in. These factors can have a significant impact on overall performance. On the other hand, the \\\\(\\lambda\\\\) parameter for the proposed interaction loss is tuned. So the resulting comparison is not fair.\n- It is not clear what additional insight this metric provides about dropout, beyond confirming what is intuitively apparent : that having randomly dropped neurons will make it harder for the network to learn high-order interactions.\n\nOther comments and suggestions:\n- The introduction includes a discussion around Banzhaf value, without describing what it means. The concept of Banzhaf value might be new to many readers in the ML community. I would suggest including a short explanation to give some intuition about what it means, before discussing it in more detail.\n- \" the output of the DNN corresponds to the score f\" : would it make sense to say that (negative) loss corresponds to the score f, rather than the output of the network ?\n- \"award\" -> \"reward\" or \"utility\" ? (I'm not familiar with game theory literature, so I'm not sure if \"award\" is a commonly used term there).\n- The title of the paper is a bit misleading as it seems to suggest that the paper is about using dropout in Game theory (i.e. solving problems in game theory using dropout).\n\nPost rebuttal\nThe authors addressed the concerns around the clarity of the paper and added useful additional experiments. I will increase my score to 7.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a game-theoretic framework to improve our understanding of dropout. All reviewers appreciated the contribution of the paper. While they had a number of questions/suggestions, almost all of them were adequately addressed. Three reviewers are satisfied and recommend acceptance, while a lone reviewer is on the fence, he/she admits he/she is less knowledgeable about game theory. Overall, I think this paper makes a solid contribution to ICLR.
Decision: Accept (Poster)