DISCOVERING AND EXPLAINING THE REPRESENTATION BOTTLENECK OF DNNS

1 INTRODUCTION . The revolution from shallow to deep models is a crucial step in the development of artificial intelligence . Deep neural networks ( DNNs ) usually exhibit superior performance to shallow models , which is generally believed as a result of the improvement of the representation power ( Pascanu et al. , 2013 ; Montúfar et al. , 2014 ) . To this end , instead of considering previous issues of the accuracy and the generalization ability of DNNs , we focus on the following two questions about the representation capacity : • Are there any common tendencies of DNNs in representing specific types of features ? • Does a DNN encode similar visual concepts as human beings for image classification ? In order to answer the above two questions , our research first investigates the bottleneck of knowledge representation , i.e. , which types of knowledge points ( e.g. , certain visual concepts ) are likely to be encoded by a DNN , and which types of knowledge points are difficult to be learned . To this end , we discover that the interaction between input variables is an effective tool to analyze the knowledge representation . It is because the DNN does not consider input variables working independently , but it encodes the interaction between input variables to form an interaction pattern for inference . For example , the inference of a face image can be explained as the interaction between left and right eyes , the interaction between nose and mouth , and etc . As the answers to the above questions , we discover a common representation bottleneck of DNNs in encoding interactions , i.e. , a DNN is more likely to encode both too complex and too simple interactions , instead of encoding interactions of intermediate complexity . This also indicates a dramatic difference from human cognition . The interaction can be understood as follows . Let us take the face recognition task for example . Let φi=mouth measure the numerical importance of the mouth region i to the classification score . Then , the interaction utility between the mouth region i and the nose region j is measured as the change of the φi=mouth value by the presence or absence of the nose region j . If the presence of j increases the importance φi=mouth by 0.1 , then , we consider 0.1 as the utility of the interaction between i and j. Multi-order interactions . In order to represent the interaction complexity mentioned in the representation bottleneck , we use the multi-order interaction utility between variables i , j proposed by Zhang et al . ( 2020 ) . The interaction of the m-th order I ( m ) ( i , j ) measures the average interaction utility between variables i , j on all contexts consisting of m variables . In this way , the order m � �� � � ( a ) Interaction pattern ( b ) Representation bottleneck ( c ) Whether humans/DNNs extract new information from patches . much new information little new information little new information much new information much new information little new information 5 % 10 % 30 % 70 % 90 % 95 % patches In te ra ct io n st re ng th Figure 1 : ( a ) Five pixels ( g , h , i , j , k ) interact with each other , forming an edge pattern for classification . ( b ) Representation bottleneck . A DNN is likely to encode low-order and high-order interactions , but usually fails to learn middle-order interactions . ( c ) The cognition gap between DNNs and humans . Humans extract little information from a few image patches ( e.g. , 5 % patches ) . Also , given almost all patches ( e.g. , 90 % patches ) , the information is already redundant for human cognition . Then , people learn little new information from the additional 5 % patches . In comparison , the DNN encodes most interactions when the DNN is given very few patches or most patches . reflects the contextual complexity of the interaction . A low-order I ( m ) ( i , j ) measures the relatively simple collaboration between variables i , j and a few m contextual variables , while a high-order I ( m ′ ) ( i , j ) corresponds to the complex collaboration between i , j and m′ massive contextual variables , where m′ m. Moreover , we prove that the multi-order interaction is a trustworthy tool to analyze the representation capacity of DNNs . Specifically , the output score of a DNN can be decomposed into utilities of compositional multi-order interactions between different pairs of variables , i.e. , model output =∑n−2 m=0 ∑ i , j∈N , i6=j w ( m ) I ( m ) ( i , j ) + ∑ i∈N local utility of i + bias . For example , the inference score of a face can be decomposed into the interaction utility between left and right eyes , between mouth and nose , and etc . Therefore , we can take the utility I ( m ) ( i , j ) as the underlying reason to explain the DNN , because each interaction makes a compositional contribution to the output . Representation bottleneck of DNNs . Surprisingly , the above decomposition of multi-order interactions enables us to discover a representation bottleneck of DNNs . As Figure 1 ( b ) shows , low-order and high-order interaction utilities I ( m ) ( i , j ) usually have high absolute values , while middle-order interaction utilities I ( m ) ( i , j ) usually have low absolute values . In other words , a DNN is more likely to encode the interaction between variables i and j , when i , j interact with a few contextual variables . Similarly , it is also easy for the DNN to learn the interaction , when i , j interact with most contextual variables . However , it is difficult for the DNN to learn the interaction , when i , j cooperate with a medium number of contextual variables . The difficulty of learning middle-order interactions reflects a representation bottleneck of DNNs . Cognitive gap between DNNs and humans . Such a representation bottleneck also indicates a significant gap between the knowledge representation of DNNs and the visual cognition of humans . As Figure 1 ( c ) shows , people usually can not extract meaningful information from a few image patches . Besides , if people are given almost all patches , then the information is already too redundant and inserting additional patches will bring in little new information . In contrast , the DNN encodes most information , when the DNN is given only a few patches or is given most patches . Theoretical proof . In this paper , we theoretically prove the mechanism that is responsible for the representation bottleneck . Such a proof also enables us to simulate the distribution of interactions of different orders , which well matches the distribution of interactions in real applications . Beyond the theoretical proof , another important issue is how to guide the learning of knowledge representation in DNNs by learning interactions of specific orders . We propose two losses to encourage/penalize the DNN to make inferences by interactions of specific orders , thereby boosting/preventing the learning of such interactions . Experimental results have validated the effectiveness of the two losses . Next , we investigate the representation capacities of several DNNs which encoded interactions of different orders . We find that the DNNs mainly encoding high-order interactions represent more structural information than the normally trained DNNs . In addition , high-order interactions were vulnerable to adversarial attacks . In summary , this paper makes three contributions : • This study discovers a representation bottleneck phenomenon of DNNs , i.e. , it is difficult for a DNN to learn middle-order interactions . It also clearly proves that DNNs and humans use different types of visual concepts for inference . •We theoretically prove the underlying reason for the representation bottleneck . • We design two losses to encourage/penalize the DNN to learn interactions of specific orders . Experiments have validated the effectiveness of the proposed losses . Besides , we investigate the representation capacities of DNNs which encode interactions of different orders . 2 RELATED WORK . The representation capacity of DNNs . The evaluation of the representation capacity of DNNs provides a new perspective to explain and analyze DNNs . Pascanu et al . ( 2013 ) and Montúfar et al . ( 2014 ) used the number of linear response regions in a deep rectifier MLP to evaluate its representation capacity . The information bottleneck theory ( Shwartz-Ziv & Tishby , 2017 ) used the mutual information to explain how DNNs gradually learned the information during the training process . Achille & Soatto ( 2018 ) , Amjad & Geiger ( 2019 ) and Hjelm et al . ( 2019 ) further improved the representation capacity by optimizing mutual information . Arpit et al . ( 2017 ) studied the memorization behavior of DNNs during training to analyze the feature representations . Xu ( 2018 ) proposed Fourier analysis to understand the generalization . In addition , several metrics were proposed to analyze the generalization capacity or robustness of DNNs , including the stiffness ( Fort et al. , 2019 ) , the sensitivity ( Novak et al. , 2018 ) , and the CLEVER score ( Weng et al. , 2018 ) . Neyshabur et al . ( 2017 ) examined whether existing complexity measures can guarantee generalization . Previous researches mainly studied the theoretically maximum complexity , generalization ability , and robustness of DNNs . In comparison , our research focuses on the limitation of DNNs in feature representations , i.e. , which types of interactions are unlikely to be encoded . Interactions . Interactions between input variables of a DNN have been widely investigated in recent years . Based on the Shapley value ( Shapley , 1951 ) , Grabisch & Roubens ( 1999 ) proposed the Shapley interaction index to define the interaction in a cooperative game . Lundberg et al . ( 2018 ) used the interaction effect to build tree ensemble explanations for the decision of DNNs . Janizek et al . ( 2021 ) extended Integrated Gradients ( Sundararajan et al. , 2017 ) to explain the pairwise feature interaction in DNNs . Sundararajan et al . ( 2020 ) proposed the Shapley Taylor interaction to measure interactions among multiple variables . Tsang et al . ( 2020 ) and Tsang et al . ( 2017 ) interpreted DNNs by detecting statistical interactions between input variables and interactions between network weights respectively . Peebles et al . ( 2020 ) and Tsang et al . ( 2018 ) achieved the disentanglement of features by restricting interactions . Song et al . ( 2019 ) and Lian et al . ( 2018 ) designed network architectures to effectively learn feature interactions in the recommendation system . Lengerich et al . ( 2020 ) applied the ANOVA technique to model interaction effects and further explored the relationship between dropout and interactions . Zhang et al . ( 2020 ) proposed the multi-order interaction , and used it to understand and boost dropout . In this paper , we used the multi-order interaction to study the complexity of interactions encoded in DNNs . 3 REPRESENTATION BOTTLENECK . Before the analysis of the representation bottleneck , let us first introduce multi-order interactions between input variables , which are encoded in a DNN . Given a pre-trained DNN v and an input sample with a set of n variables N = { 1 , . . . , n } ( e.g. , an input image with n pixels ) , v ( N ) denotes the network output of all input variables . Input variables of DNNs usually interact with each other to make inferences , instead of working individually . In this study , we mainly discuss the pairwise interactions . For example , as Figure 1 ( a ) shows , pixels i , j ∈ N interact with each other , forming an edge pattern for classification . If the existence of this pattern increases the network output by 0.01 , we consider the utility of this pattern as 0.01 . Similarly , if the existence of this pattern decreases the network output , we consider this pattern has a negative utility . Furthermore , the multi-order interaction I ( m ) ( i , j ) between two input variables i , j ∈ N , 0 ≤ m ≤ n−2 , is proposed to measure interactions of different complexities ( Zhang et al. , 2020 ) . Specifically , the m-th order interaction I ( m ) ( i , j ) measures the average interaction utility between variables i , j under all possible contexts consisting of m variables . Therefore , the order m can be considered to represent the contextual complexity of the interaction . For example , as Figure 1 ( a ) shows , five pixels ( i , j , g , h , k ) collaborate with each other and form an edge pattern for classification . Thus , the pair- wise interaction between pixels i , j also depends on three contextual pixels g , h , k. Mathematically , the multi-order interaction I ( m ) ( i , j ) is defined as follows : I ( m ) ( i , j ) = ES⊆N\ { i , j } , |S|=m [ ∆v ( i , j , S ) ] , ( 1 ) where ∆v ( i , j , S ) = v ( S ∪ { i , j } ) − v ( S ∪ { i } ) − v ( S ∪ { j } ) + v ( S ) . Here , v ( S ) is the output score when we keep variables in S ⊆ N unchanged but replace variables in N \ S by the baseline value . The baseline value follows the widely-used setting in Ancona et al . ( 2019 ) , which is set as the average value of the variable over different samples . Let us take the multi-category image classification for example . Given an input image x , v ( S ) = v ( xS ) can be implemented as any scalar output of the DNN ( e.g. , log P ( ŷ=y truth|xS ) 1−P ( ŷ=ytruth|xS ) of the true category ) , where we replace the pixel values inN \S of original input x by the baseline value ( the average pixel value over images ) to construct a masked image xS . Then , ∆v ( i , j , S ) = [ v ( S∪ { i , j } ) −v ( S∪ { i } ) ] − [ v ( S∪ { j } ) −v ( S ) ] measures the marginal interaction effects of the collaboration between i , j in a context S. Generic metric . The proposed multi-order interaction I ( m ) ( i , j ) is a generic metric , which has a strong connection with the Shapley value ( Shapley , 1951 ) and the Shapley interaction index ( Sundararajan et al. , 2020 ) in game theory . In addition , it has been proven that I ( m ) ( i , j ) satisfies the following five desirable properties , i.e. , linearity , nullity , commutativity , symmetry , and efficiency properties . The connections with existing metrics and five properties are introduced in Appendix A .

Summary: The authors study the complexity of interactions between input variables in deep neural networks (DNNs). They use a multi-order interaction utility between pairs of variables to represent the interaction complexity between input variables encoded in DNNs. The results can be summarized as follows:  1) DNNs are more likely to encode simple and complex interactions between pairs of input variables but aren't good enough to encode the intermediate ones. The authors define the simplicity of interaction depending on its contextual complexity.  2) The authors develop novel loss functions to encourage/penalize the interactions of specific complexities and then, investigate the representation capacities of the learned DNNs.