Towards Interpreting Deep Neural Networks via Understanding Layer Behaviors

Deep neural networks ( DNNs ) have achieved unprecedented practical success in many applications . However , how to interpret DNNs is still an open problem . In particular , what do hidden layers behave is not clearly understood . In this paper , relying on a teacher-student paradigm , we seek to understand the layer behaviors of DNNs by “ monitoring ” both across-layer and single-layer distribution evolution . Here , the “ across-layer ” and “ single-layer ” considers the layer behavior along the depth and a specific layer along training epochs , respectively . Relying on optimal transport theory , we employ the Wasserstein distance ( W-distance ) to measure the divergence between the layer distribution and the target distribution . Theoretically , we prove that i ) the W-distance between the distribution of any layer and the target distribution tends to decrease along the depth . ii ) For a specific layer , the W-distance between the distribution in an iteration and the target distribution tends to decrease along training iterations . iii ) However , a deep layer is not always better than a shallow layer for some samples . Moreover , our results helps to analyze the stability of layer distributions and explains why auxiliary losses help the training of DNNs . Extensive experiments justify our theoretical findings . 1 INTRODUCTION . Deep neural networks ( DNNs ) have been successfully applied in computer vision , such as image classification ( Chen et al. , 2019b ; Hsu et al. , 2019 ) , image generation ( Cao et al. , 2019 ; Brock et al. , 2019 ; Chrysos et al. , 2019 ) and speech recognition ( Yeh et al. , 2019 ; Chen et al. , 2019a ) . Despite their success , the internal mechanism of DNNs is still a black box . In particular , understanding what do hidden layers do and how to achieve remarkable performance remain persistently elusive . To answer these questions , we seek to understand across-layer and single-layer behavior . For the across-layer behavior , we study the learning process by measuring theW-distance between the distribution of any layer and the target distribution . Recently , most methods only focus on final predictions in different tasks ( He et al. , 2016 ) . Due to the end-to-end training , interpreting each intermediate layer behaviors of a DNN , which , however , is still not clear . To provide interpretability , existing works try to produce a single prediction and observe the classification performance of each layer ( Papernot & McDaniel , 2018 ; Szegedy et al. , 2013 ; Kaya et al. , 2019 ) . For example , ( Alain & Bengio , 2016 ) experimentally observe that the linear separability of features increases monotonically along the depth of a DNN . Unfortunately , there is no theoretical analysis to support the experimental finding . To understand the learning process of DNNs , one can explore the across-layer behavior by monitoring how the distributions propagate across different layers . However , how to open the internal mechanism of DNNs and investigate the across-layer behavior of a DNN remains an open question . For the single-layer behavior , we seek to measure the W-distance between the distribution in an iteration and the target distribution . It is important to analyze the distributional stability of one layer by measuring the change of the distributions . Recently , the distributional stability of a deep neural network attracts extensive attention ( Santurkar et al. , 2018 ) . Some studies try to visualize the training behavior of one layer by plotting the mean and variance of features ( Santurkar et al. , 2018 ) . Unfortunately , the visualizations are subjective and lack of necessary theoretical justifications . To this end , ( Sonoda & Murata , 2019 ) propose a transport analysis method and state that a denoising Autoencoder transports mass to decrease the Shannon entropy of the data distribution . However , the Under review as a conference paper at ICLR 2020 method is limited and inflexible for analyzing a general case of deep neural networks . Therefore , it is very necessary and important to develop a new analytical method to interpret the stability of layers . In this paper , we apply optimal transport theory to analyze the behavior of distributions . Specifically , we exploit Wasserstein divergence ( W-distance ) to measure the difference between the distribution of any layer and the target distribution . By monitoring the change of theW-distance , we are able to study both across-layer and single-layer behaviors . Our contributions are summarized as follows . • We analyze the across-layer behavior and prove that theW-distance between the distribution of any layer and the target distribution decreases along the depth of a DNN . It means that every layer of the network can express the target distribution progressively . • We analyze the single-layer behavior and prove that for a specific layer , theW-distance between the distribution in an iteration and the target distribution decreases across the training iterations when introducing a loss in the layer . • Moreover , we provide experiments and theoretical justifications on these findings . The proposed analytical framework provides a different view of understanding and interpreting neural networks . 2 RELATED WORK . Many studies analyze the features of intermediate layers . ( Bau et al. , 2017 ) evaluates how hidden units align a set of semantic concepts to quantifies the interpretability of latent representations of a CNN . ( Dosovitskiy & Brox , 2016 ) inverts image representations with up-convolutional networks for studying image representations . ( Zeiler & Fergus , 2014 ) presents a visualization technique to investigate the function of intermediate feature layers and the classifier . ( Zhang et al. , 2018 ) learns an explanatory graph to reveal the knowledge hierarchy hidden inside a pre-trained CNN . ( Zhang et al. , 2019 ) learns a decision tree to clarify the specific reason for each semantic prediction of a CNN . Some studies analyze the classification accuracy of a DNN . ( Lee et al. , 2015 ) introduces a classification objective to the individual hidden layers , in addition to the overall objective at the output layer . ( Szegedy et al. , 2015 ) uses auxiliary classifiers to improve the classification performance . ( Kaya et al. , 2019 ) introduces internal classifiers into off-the-shelf DNNs to understand overthinking of networks by studying how the prediction changes during a DNN ’ s forward pass . ( Gupta & Schütze , 2018 ) explains recurrent neural networks by understanding the layer-wise semantic accumulation behavior . ( Sonoda & Murata , 2019 ) investigates the feature map inside a DNN by tracking the transport map , and prove that a deep Gaussian DAE transports mass to decrease Shannon entropy of the data distribution . ( Alain & Bengio , 2016 ) experimentally observes that the linear separability of features increases monotonically along the depth of a DNN . However , there is no theoretical result to analyze this phenomenon . Existing studies using information bottleneck methods ( Tishby & Zaslavsky , 2015 ; Bang et al. , 2019 ) mainly analyze the dynamics of across different layers . However , it is hard for these methods to analyze the dynamics of a specific layer through different iterations . In contrast , our proposed method is able to analyze both single-layer and across-layer behaviors . Under review as a conference paper at ICLR 2020 3 PROBLEM SETTING . Notation . We use bold lower-case letters ( e.g. , x ) to denote vectors , and bold upper-case letters ( e.g. , X ) to denote matrices . We denote the transpose of a vector ( e.g. , xT ) or matrix ( e.g. , XT ) by the superscript T. Let 1 : = [ 1 , . . . , 1 ] T and 1S ( · ) be an indicator function of a set S , and let [ n ] = { 0 , 1 , . . . , n } . Let ∗ be the convolution operator , grad be gradient operator . Label distribution learning . In this paper , we consider a label distribution learning problem ( Geng , 2016 ) , especially multi-label classification . In this setting , the label distribution is defined as a probability distribution to cover a certain number of labels , representing the degree to which each label describes the instance ( see Figure 1 ( c ) ) . Note that the sum of the label distribution is equal to 1 . Specifically , given training samples S , we seek to learn a model f from data space X into the label space Y . In Figure 1 ( a ) , for L-layered neural networks , we denote f̃0 : l=f̃l◦ · · · ◦f̃0 , l≤L as the output of the l-th layers , then the label distribution mapping can be defined as fl : =hl◦f̃0 : l , where hl is a probability function ( e.g. , FC+softmax ( Frogner et al. , 2015 ) ) . Note that fl , l∈ [ L ] have the same input domains and the same output domains . Our goal is to learn a model fl to let the predicted distribution µ̂l close to the target distribution µ . Then , we optimize the empirical risk : minfl L ( fl ) : = ES [ d ( µ̂l , µ ) ] , where µ̂l=fl # ( µ0 ) , l≤L , ( 1 ) where ES [ · ] is the expectation w.r.t . the training set S , fl # denotes the pushforward operator of µ0 , and d ( · , · ) is some distribution divergence , such as Cross-entropy ( CE ) loss and Wasserstein loss . In practice , the label distribution is obtained by training with CE loss . Because of the convexity of CE loss , we derive the optimal label distribution for given features of every layer ( Alain & Bengio , 2016 ) . In this sense , the label distribution reflects the actual distribution of feature maps in a specific layer . Label distribution propagation . In this paper , our goal is to explore how the layer distributions propagate in the across-layer and single-layer of a DNN . Specifically , we measure the divergence ( e.g. , Wasserstein distance ( Villani , 2008 ) ) to observe how the layer distributions change across different layers or iterations . In Figure 1 ( b ) , we explain the behaviors of layer from the following two perspectives . For the across layers , measuring the distance between the distribution of any layer and the target distribution helps to understand the learning process and distribution expression ability . For a single layer , measuring the distance between the distribution in an training iteration and the target distribution helps to understand the dynamic behavior and training process of a DNN . 4 WASSERSTEIN DISTRIBUTION PROPAGATION QUANTIFICATION . Optimal transport ( OT ) . Recently , the performance of multi-label classification can be improved by usingW-distance ( Frogner et al. , 2015 ; Zhao & Zhou , 2018 ) . However , DNNs on the multi-label classification task still lack theoretical understanding . With the help of OT theory ( Villani , 2008 ) , W-distance can be used to interpret DNNs . In contrast , some measures , such as Jensen–Shannon divergence , which , however , often ignore any metric structure of distribution ( Frogner et al. , 2015 ) . Definition 1 ( W-distance ( Villani , 2008 ) ) Given a prediction distribution µ̂ and the target distribution µ , and the cost matrix C defined as Cκ , κ′=d p K ( κ , κ ′ ) with the metric dK , where κ , κ′ are label tags , then theW-distance seeks to find a transportation matrix T by transporting a mass µ̂ to µ , W2 ( µ̂ , µ ) = infT∈Π ( µ̂ , µ ) 〈T , C〉 , ( 2 ) where Π ( µ̂ , µ ) is the set of couplings and defined as Π ( µ̂ , µ ) = { T∈RK×K+ : T1=µ̂ , T T1=µ } . In practice , the distance of two label tags in the cost matrix can be calculated by using word2vec ( Mikolov et al. , 2013 ) in the Euclidean space . We calculateW-distance with entropic regularization by Sinkhorn algorithm ( Knight , 2008 ) with p=2 . See more details in Supplementary materials . Distribution propagation measure . In this paper , we use W-distance to measure the acrosslayer and single-layer label distribution propagation , as shown in Figure 1 ( b ) . For an acrosslayer , we measure W-distance between the distribution of any layer and the target distribution , i.e. , W2 ( µ̂1 , µ ) , . . . , W2 ( µ̂L , µ ) . In this way , we can measure how the distributions propagate across different layers . For the single-layer , we measureW-distance between the distribution in an training iteration and the target distribution , i.e. , W2 ( µ̂ ( 0 ) l , µ ) , . . . , W 2 ( µ̂ ( t ) l , µ ) . Therefore , we can study the behavior of distributions in the same layer . Under review as a conference paper at ICLR 2020 Teacher-student analytical framework

Summary: This paper seeks to understand both across-layer and single-layer behavior within neural networks (i.e. layer behavior along the depth of a network, and behavior of a single layer along training epochs). Therefore, they resort to the optimal transport framework to compare predicted and target distributions. Theoretically, they show that the Wasserstein distance between predicted and target distributions is decreasing along the depth and for a single layer, along training iterations. They also give intuition on how this analysis can help the learning process in practice.