Title: TOWARDS INTERPRETING DEEP NEURAL NETWORKS VIA UNDERSTANDING LAYER BEHAVIORS

Abstract: Deep neural networks (DNNs) have achieved unprecedented practical success in many applications. However, how to interpret DNNs is still an open problem. In particular, what do hidden layers behave is not clearly understood. In this paper, relying on a teacher-student paradigm, we seek to understand the layer behaviors of DNNs by "monitoring" both across-layer and single-layer distribution evolution. Here, the "across-layer" and "single-layer" considers the layer behavior along the depth and a specific layer along training epochs, respectively. Relying on optimal transport theory, we employ the Wasserstein distance (W-distance) to measure the divergence between the layer distribution and the target distribution. Theoretically, we prove that i) the W-distance between the distribution of any layer and the target distribution tends to decrease along the depth. ii) For a specific layer, the W-distance between the distribution in an iteration and the target distribution tends to decrease along training iterations. iii) However, a deep layer is not always better than a shallow layer for some samples. Moreover, our results helps to analyze the stability of layer distributions and explains why auxiliary losses help the training of DNNs. Extensive experiments justify our theoretical findings.

1. INTRODUCTION:

Deep neural networks (DNNs) have been successfully applied in computer vision, such as image classification (Chen et al., 2019b; Hsu et al., 2019) , image generation (Cao et al., 2019; Brock et al., 2019; Chrysos et al., 2019) and speech recognition (Yeh et al., 2019; Chen et al., 2019a) . Despite their success, the internal mechanism of DNNs is still a black box. In particular, understanding what do hidden layers do and how to achieve remarkable performance remain persistently elusive. To answer these questions, we seek to understand across-layer and single-layer behavior.

For the across-layer behavior, we study the learning process by measuring the W-distance between the distribution of any layer and the target distribution. Recently, most methods only focus on final predictions in different tasks (He et al., 2016) . Due to the end-to-end training, interpreting each intermediate layer behaviors of a DNN, which, however, is still not clear. To provide interpretability, existing works try to produce a single prediction and observe the classification performance of each layer (Papernot & McDaniel, 2018; Szegedy et al., 2013; Kaya et al., 2019) . For example, (Alain & Bengio, 2016) experimentally observe that the linear separability of features increases monotonically along the depth of a DNN. Unfortunately, there is no theoretical analysis to support the experimental finding. To understand the learning process of DNNs, one can explore the across-layer behavior by monitoring how the distributions propagate across different layers. However, how to open the internal mechanism of DNNs and investigate the across-layer behavior of a DNN remains an open question.

For the single-layer behavior, we seek to measure the W-distance between the distribution in an iteration and the target distribution. It is important to analyze the distributional stability of one layer by measuring the change of the distributions. Recently, the distributional stability of a deep neural network attracts extensive attention (Santurkar et al., 2018) . Some studies try to visualize the training behavior of one layer by plotting the mean and variance of features (Santurkar et al., 2018) . Unfortunately, the visualizations are subjective and lack of necessary theoretical justifications. To this end, (Sonoda & Murata, 2019) propose a transport analysis method and state that a denoising Autoencoder transports mass to decrease the Shannon entropy of the data distribution. However, the Under review as a conference paper at ICLR 2020 (b) Across-layer propagation: the label distribution µ (t) 0 propagate to µ (t) L . Single-layer propagation: the label distribution µ (0) k propagate to µ (t) k during the epochs in one layer. (c) Given an image with two labels (i.e., cat and dog), the label distribution depicts their corresponding probabilities of 0.5. method is limited and inflexible for analyzing a general case of deep neural networks. Therefore, it is very necessary and important to develop a new analytical method to interpret the stability of layers.

In this paper, we apply optimal transport theory to analyze the behavior of distributions. Specifically, we exploit Wasserstein divergence (W-distance) to measure the difference between the distribution of any layer and the target distribution. By monitoring the change of the W-distance, we are able to study both across-layer and single-layer behaviors. Our contributions are summarized as follows.

• We analyze the across-layer behavior and prove that the W-distance between the distribution of any layer and the target distribution decreases along the depth of a DNN. It means that every layer of the network can express the target distribution progressively.

• We analyze the single-layer behavior and prove that for a specific layer, the W-distance between the distribution in an iteration and the target distribution decreases across the training iterations when introducing a loss in the layer. • Moreover, we provide experiments and theoretical justifications on these findings. The proposed analytical framework provides a different view of understanding and interpreting neural networks.

2. RELATED WORK:

Many studies analyze the features of intermediate layers. (Bau et al., 2017) evaluates how hidden units align a set of semantic concepts to quantifies the interpretability of latent representations of a CNN. (Dosovitskiy & Brox, 2016) inverts image representations with up-convolutional networks for studying image representations. (Zeiler & Fergus, 2014) presents a visualization technique to investigate the function of intermediate feature layers and the classifier. (Zhang et al., 2018) learns an explanatory graph to reveal the knowledge hierarchy hidden inside a pre-trained CNN. (Zhang et al., 2019) learns a decision tree to clarify the specific reason for each semantic prediction of a CNN.

Some studies analyze the classification accuracy of a DNN. (Lee et al., 2015) introduces a classification objective to the individual hidden layers, in addition to the overall objective at the output layer. (Szegedy et al., 2015) uses auxiliary classifiers to improve the classification performance. (Kaya et al., 2019) introduces internal classifiers into off-the-shelf DNNs to understand overthinking of networks by studying how the prediction changes during a DNN's forward pass. (Gupta & Schütze, 2018) explains recurrent neural networks by understanding the layer-wise semantic accumulation behavior. (Sonoda & Murata, 2019) investigates the feature map inside a DNN by tracking the transport map, and prove that a deep Gaussian DAE transports mass to decrease Shannon entropy of the data distribution. (Alain & Bengio, 2016) experimentally observes that the linear separability of features increases monotonically along the depth of a DNN. However, there is no theoretical result to analyze this phenomenon. Existing studies using information bottleneck methods (Tishby & Zaslavsky, 2015; Bang et al., 2019) mainly analyze the dynamics of across different layers. However, it is hard for these methods to analyze the dynamics of a specific layer through different iterations. In contrast, our proposed method is able to analyze both single-layer and across-layer behaviors.

3. PROBLEM SETTING:

Notation. We use bold lower-case letters (e.g., x) to denote vectors, and bold upper-case letters (e.g., X) to denote matrices. We denote the transpose of a vector (e.g., x T ) or matrix (e.g., X T ) by the superscript T . Let 1:=[1, . . . , 1] T and 1 S (•) be an indicator function of a set S, and let

[n]={0, 1, . . . , n}. Let * be the convolution operator, grad be gradient operator. Label distribution learning. In this paper, we consider a label distribution learning problem (Geng, 2016) , especially multi-label classification. In this setting, the label distribution is defined as a probability distribution to cover a certain number of labels, representing the degree to which each label describes the instance (see Figure 1 (c)). Note that the sum of the label distribution is equal to 1. Specifically, given training samples S, we seek to learn a model f from data space X into the label space Y. In Figure 1 (a), for L-layered neural networks, we denote f0:l = fl • • • • • f0 , l≤L as the output of the l-th layers, then the label distribution mapping can be defined as f l :=h l • f0:l , where h l is a probability function (e.g., FC+softmax (Frogner et al., 2015) ). Note that f l , l∈[L] have the same input domains and the same output domains. Our goal is to learn a model f l to let the predicted distribution µ l close to the target distribution µ. Then, we optimize the empirical risk:

EQUATION

where E S [•] is the expectation w.r.t. the training set S, f l# denotes the pushforward operator of µ 0 , and d(•, •) is some distribution divergence, such as Cross-entropy (CE) loss and Wasserstein loss. In practice, the label distribution is obtained by training with CE loss. Because of the convexity of CE loss, we derive the optimal label distribution for given features of every layer (Alain & Bengio, 2016) .

In this sense, the label distribution reflects the actual distribution of feature maps in a specific layer.

Label distribution propagation. In this paper, our goal is to explore how the layer distributions propagate in the across-layer and single-layer of a DNN. Specifically, we measure the divergence (e.g., Wasserstein distance (Villani, 2008) ) to observe how the layer distributions change across different layers or iterations. In Figure 1 (b), we explain the behaviors of layer from the following two perspectives. For the across layers, measuring the distance between the distribution of any layer and the target distribution helps to understand the learning process and distribution expression ability. For a single layer, measuring the distance between the distribution in an training iteration and the target distribution helps to understand the dynamic behavior and training process of a DNN.

4. WASSERSTEIN DISTRIBUTION PROPAGATION QUANTIFICATION:

Optimal transport (OT). Recently, the performance of multi-label classification can be improved by using W-distance (Frogner et al., 2015; Zhao & Zhou, 2018) . However, DNNs on the multi-label classification task still lack theoretical understanding. With the help of OT theory (Villani, 2008) , W-distance can be used to interpret DNNs. In contrast, some measures, such as Jensen-Shannon divergence, which, however, often ignore any metric structure of distribution (Frogner et al., 2015) .

Definition 1 (W-distance (Villani, 2008) ) Given a prediction distribution µ and the target distribution µ, and the cost matrix C defined as C κ,κ =d p K (κ, κ ) with the metric d K , where κ, κ are label tags, then the W-distance seeks to find a transportation matrix T by transporting a mass µ to µ,

EQUATION

where Π( µ, µ) is the set of couplings and defined as Π( µ, µ)={T ∈R K×K + : T 1= µ, T T 1=µ}.

In practice, the distance of two label tags in the cost matrix can be calculated by using word2vec (Mikolov et al., 2013) in the Euclidean space. We calculate W-distance with entropic regularization by Sinkhorn algorithm (Knight, 2008) with p=2. See more details in Supplementary materials.

Distribution propagation measure. In this paper, we use W-distance to measure the acrosslayer and single-layer label distribution propagation, as shown in Figure 1 (b) . For an acrosslayer, we measure W-distance between the distribution of any layer and the target distribution, i.e., W 2 ( µ1, µ), . . . , W 2 ( µL, µ). In this way, we can measure how the distributions propagate across different layers. For the single-layer, we measure W-distance between the distribution in an training iteration and the target distribution, i.e., W 2 ( µ (0) l , µ), . . . , W 2 ( µ

(t)

l , µ). Therefore, we can study the behavior of distributions in the same layer. 

… 𝑓 ∞ T-net S-net … … … … … … … … … … … 𝑓 1 ′ 𝑓 2 ′ 𝑓 ∞ ′ 𝑓 1 ′ 𝑓 2 ′ 𝑓 𝐿 ′ 𝑓 1 𝑓 2 𝑓 1 𝑓 2 𝑓 𝑙

5. TEACHER-STUDENT FRAMEWORK FOR LAYER BEHAVIOR ANALYSIS:

Teacher-student analysis framework is widely used to analyze and understand DNNs (Tian, 2017; Du et al., 2018; Cuturi, 2013) . For the one-layer behavior, this framework helps to understand the dynamic of student network from the teacher network. For the multi-layer behavior, this framework helps to study the ability of the student network to express distributions of the teacher network. Therefore, teacher-student analysis framework is important and necessary to understand DNNs.

5.1. ONE-LAYER BEHAVIOR ANALYSIS:

In our problem setting, the target distribution in the training loss (1) can be represented by an output of a teacher network (T-net). Our goal is to make the output of the student network (S-net) to be close to the output of T-net, as shown in Figure 2 (a). Specifically, given a S-net f and T-net f , our goal is to find a function f that minimizes the following training loss,

EQUATION

where x=x+ε, ε∼ν t is the corruption of data, and ν t is a noise distribution w.r.t. the variance tI, i.e., ν t =N (0, tI). By optimizing the training loss, the distribution in one layer can be monitored by using a transport map. Next, we provide the definition of transport map below.

Definition 2 (Transport map) A transport map f t : R m →R m can be defined as f t (x)=x+g t (x) with an update direction g t when t>0, and f 0 (x)=x when t=0.

Based on the definition of the transport map, we analyze the layer behavior of ResNet (He et al., 2016) , i.e., T-net is a residual block. Note that our analysis method can be extended to a more general case. Then, we solve the optimal transport map as follow.

Theorem 1 (Optimal transport map) If the teacher network is a residual unit f ( x)=x+σ( x), where σ contains FC layer and an activation function. When ν t :=N (0, tI), the minimum

f * t is f * t ( x) = x + t∇ log(ν t * µ 0 )( x) + σ( x) := x + g t ( x).

The minimizer f * t updates x along the update direction g t (x). Therefore, the optimal transport map f * t with time t transports the mass at the activation x toward x+g t (x). To associate with continuity equation (Sonoda & Murata, 2019) , we introduce Wasserstein gradient flow (WGF) as follows.

Proposition 1 (Wasserstein gradient flow) Assume µ t satisfies the continuity equation with the gradient vector field

∂ t µ t =-∇•[µ t ∇V t ] of a potential function V t , and if the function F satisfies ∂ t F (µ t )= R m -V t (x)[∂ t µ t ](x)dx, then WGF coincides with ∂ t µ t =-gradF (µ t ).

Using the definition of WGF, we explore how the distribution propagate in one layer as follows.

Theorem 2 Based on the equivalent condition in Theorem 2 (Belavkin, 2016) , and let ∆ be the Laplacian operator. At the initial moment t→0, the pushforward f # µ t with Gaussian distribution satisfies the backward heat equation (BHE), and evolves according to Wasserstein gradient flow:

∂ t µ t=0 (x)= -∆µ 0 (x)= -grad W 2 (µ t , µ), x∈R m .

Note that W is the potential function. Theorem 2 states that by introducing an auxiliary loss in a layer, the W-distance can be decreased in the layer. It means that the label distribution can be close to the target distribution across training iterations.

Under review as a conference paper at ICLR 2020 5.2 MULTI-LAYER BEHAVIOR ANALYSIS (i) Infinitely deep neural networks. For an infinitely DNN, we assume the number of layers and hidden units and the size of dataset is infinite. Specifically, let 0=t 0 <t 1 < • • • <t L+1 =t, and let f 0 be trained on µ 0 with the variance t 1 -t 0 , then x 1 :=f 0 (x 0 ) and its distribution is µ 1 :=f 0# µ 0 . Then, we train f 1 on µ 1 with the variance t 2 -t 1 . In the same way, we obtain f l (x l ) and µ l :=f l-1 # µ l-1 . The composition of residual blocks can be written as

f t 0:L (x):=f L •• • ••f 0 (x)

, where t is total time. Then, the velocity of the composition coincides with the vector field ∂ t f t=t l 0:l (x)=∇V t l (x). Furthermore, we extend one-layer behavior analysis to the case of infinite layers, as shown in Figure 2 (b). Based on Theorem 1, we have the dynamical systems as follows.

Lemma 1 When the noise distribution is a Gaussian distribution, the continuous residual units is defined as a flow ϕ t : R m →R m of the following dynamical systems associated with the vector field ∇V t , i.e., d dt x(t)=∇ log(µ t (x(t)))+σ(x(t)), where t≥0 and µ t :=ϕ t# µ 0 . Based on the proofs of Theorem 2, we can achieve a similar conclusion as follows.

Theorem 3 Based on the equivalent condition in Theorem 2 (Belavkin, 2016), and when the noise distribution is a Gaussian distribution, then the pushforward measure µ t :=ϕ t# µ 0 evolves according to Wasserstein gradient flow as follows: d dt µ t (x)=-gradW 2 [µ t , µ], µ t=0 =µ 0 . Theorem 3 states that the W-distance can be decreased across different layers. It means that the label distribution can be close to the target distribution along the depth of the layer.

(ii) Finitely deep neural networks. When the number of layers and the hidden unit number are finite, the above analysis is not available. As shown in Figure 2 Based on the definition of Barron function, (Barron, 1993) states that a Barron function can be approximated by a neural network with one hidden layer. Given a fixed composition of L Barron functions, we derive the error bound of the approximation between such composite function and a neural network with l hidden layers. Note that l is not necessarily equal to L. The approximation bound of w.r.t. Wasserstein distance can be provided as follows.

Theorem 4 (Wasserstein distribution approximation) Given a data distribution µ 0 and a function ϕ i : R mi-1 →R mi , and let

L l = log(l)+1, if support(µ 0 )⊂K 0 and ϕ i (K i-1 )⊆K i , 1≤i≤l, the function ϕ i is L l-1 -1 L l -Lipschitz and is a Barron function, i.e., ϕ 1 ∈Ω K0 (C 0 ) and ϕ i ∈Ω Ki-1+sBm i-1 (C i ),

then there exists a network f with l hidden layers with

EQUATION

)

where D l is the diameter of the set K l , and , δ, s>0 are parameters.

Theorem 4 provides an error bound when using a neural network with l hidden layers to approximate a fixed composition of L Barron functions. Moreover, the upper bound decreases with the increasing of layers under certain conditions. Here, we can conclude that deep layers are not always better than shallow layer for some specific samples. Such samples are often classified correctly in the shallow layer rather than the deep layer, then the diameter D l can be large for this set of samples. Hence, shallow layers can behave better than deep layers for these kinds of samples. Under review as a conference paper at ICLR 2020 0 1 2 3 4 5 6 7 8 9  0 object labels per image. For the evaluation metrics, we report the classification accuracy, CF1, OF1 and mean average precision (mAP) in the experiments.

Implementation Details. All experiments are implemented by PyTorch on a single NVIDIA TITAN X GPU. Specifically, we use SGD optimizer with the learning rate as 0.001. We experiment with ResNet (?) and VGG (?) with Binary Cross Entropy (BCE). We train the model 100 epochs with the batch size of 128. Then, we train the internal classifiers (i.e., the network h l in Figure 1 (a)) for 30 epochs.

6.1. INTER-LAYER BEHAVIOR EXPLORATION:

In this section, we provide theoretical and experimental analysis to study the behaviors of distributions in different layers. Next, we provide a new method to improve the performance of DNNs.

How every layer express the distribution? In this experiment, we study the ability of every layer to express the target distribution for ResNet and VGG. In this paper, the expression ability can be measured by Wasserstein distance (W-distance) between the label distribution of every layer and the target distribution. W-distance of different layers are shown in Figure 4 . As suggested in Theorem 4, deep layers have smaller error bound than shallow layers, meaning that they have sufficient expression ability. This experiment verifies this conclusion by observing that the deeper layers have larger W-distance than shallow layers. Moreover, W-distance tends to decrease from layer to layer. The front most layers have a little different ability to express distribution since they have similar value of W-distance. When approaching to the last layer, W-distance drops to a very small value. It implies that sufficient layers is able to express the target distribution.

How the distribution propagate across layers? In this experiment, we investigate how distributions propagate across different layers, which help to understand the learning process from layer to layer in a deep neural network. Note that we normalize the prediction of layer as the probability distribution (that is the sum is 1). Figure 5 shows that the label distribution of one sample propagates from the first layer to the final layer. At the early learning stage, the label distribution is far away from the target distribution, then it can be close to the target distribution at the latter learning stage. Different from the decreasing tendency in Figure 4 , the label distribution of one sample may not close to the target distribution progressively. For examples, on the first line of Figure 5 , the probability of the 15-th label is large in the shallow layer, but goes down in the middle layer. 

6. EXPERIMENT:

Implementation Details. All experiments are implemented on PyTorch. In the training, we use SGD optimizer with the initial learning rate as 0.01. The learning rate decays by a factor of 10 for every 40 epochs. The momentum and weight decay is set to be 0.9 and 10 -4 , respectively. We obtain models (including ResNet-18, ResNet-50 (He et al., 2016) and VGG-16 (Simonyan & Zisserman, 2014 )) pre-trained on ImageNet (Deng et al., 2009) and train them for 100 epochs with the batch size of 128.

Then, we freeze the model weights and train the internal classifiers (i.e., the network h l in Figure 1(a) ) for 100 epochs. Each internal classifier contains fully connected layer and Sigmoid function. We train the network using Binary Cross Entropy. We use training set to explore the distribution propagation of the across-and single-layer. In addition, we set α=0.01 in Eqn. 17 to achieve the balanced results.

For simplicity, we use ResNet to represent the ResNet-18 model and VGG to represent the VGG-16 model in the main paper.

Data Sets and Evaluation Metrics. We conduct experiments on two benchmark multi-label classification datasets, i.e., VOC2007 (Everingham et al., 2010) How does the distribution propagate across layers? In this experiment, we investigate how the label distribution propagate across layers to understand the learning process from layer to layer in a DNN. Note that we normalize the prediction of classifier as the probability distribution (i.e., the sum of label prediction probabilities of one sample is 1.). Figure 4 shows that the label distribution of one sample propagates from the first layer to the last layer. In the shallow layers, the label distribution is far away from the target distribution, then it can be close to the target distribution in the deep layers. Different from the decreasing tendency in Figure 3 , the label distribution of one sample may not close to the target distribution progressively. For example, in the first row of Figure 4 , the probability of the 10-th class is large in the 0-th layer, but goes down in the 3-th layer.

6.1.2. THE EVALUATION OF ACROSS-LAYER'S BEHAVIORS:

How to evaluate the quality of each layer? Existing methods attempt to visualize the feature maps to evaluate the quality of each layer. However, observing the feature maps is difficult because of its high dimensionality and complexity. Relying on the W-distance, we can evaluate the quality of each layer by measuring the ability of expression. If the W-distance decreases rapidly in a layer, it suggests that the quality of this layer is good. Taking ResNet on VOC in Figure 3 as an example, 0∼5 layers have similar value of the W-distance, while 6∼8 layers have quite different value of the W-distance. It means that these deep layers (i.e., 6∼8) are better than the shallow layers (i.e., 0∼5).

6.1.3. HOW TO EXPLOIT THE ACROSS-LAYER BEHAVIORS:

Does every sample contribute equally? In practice, a deep neural network construct more complex features progressively throughout the layers (Lee et al., 2011 ). An interesting question arises: does the contribution of each sample vary across different layers and training iterations? To answer this question, we first define the concept of sample difficulty in terms of the W-distance. As shown in Figure 5 , the samples can be divided into three categories, including easy, hard and confused. The intuition is that: 1) Easy samples should have small the W-distance in the first few layers, i.e., , they should be classified correctly with high confidence in a shallow layer. 2) Hard samples would have large the W-distance in a deeper layer, i.e., , it cannot be resolved at all, or only near the last layer. 3) Confused samples have small the W-distance in a shallow layer, while have large the W-distance in a deep layer. It means that although these samples can be classified correctly in the shallow layer, they still become misclassified at the last layer. Considering the difficulty of samples would help to interpret the training of models and design the training loss, which may speed up convergence and improve the performance eventually. We show more results in Section I in Supplementary materials.

Under review as a conference paper at ICLR 2020

Under review as a conference paper at ICLR 2020 networks often occur the over-thinking issue. In Table ???, out of *% and *% of the samples for ResNet and VGG misclassifies, respectively. For these samples, there are *% samples are correctly classified in the *-th layer of ResNet, and *% samples are correctly classified in the *-th layer of VGG. Using this phenomenon, we try to improve the performance of classification by early existing such confused samples. Different from the strategy of SDN (?), how to early exist in a DNN for a multi-label task is more challenging. Therefore, we design a new early-exist strategy for multi-label predictions. In Table 1 , our proposed method consistently outperforms the other methods.

6.2. INNER-LAYER BEHAVIOR EXPLORATION:

In this section, we study how the distributions evolute in one layer for different training iterations. We consider to use Wasserstein distance to measure the stability of distribution when training the neural network. We propose theoretical and empirical analysis for the distribution stability in one layer.

How stable is the distribution in one layer? Recently, the stability of a deep neural network attracts extensive attention. Batch Normalization is proposed to stabilize the distributions of layer distribution. The popular belief is that success of Batch Normalization stems from controlling the stability of distribution during the training. However, we prove that the stability of distribution is hard to be guaranteed, the detail proofs are put in Supplementary. In this paper, we use W-distance to measure the discrepancy between two distributions in one layer across different training iterations.

Motivated by this, we further measure stability of label distribution in different training iterations. The conventional understanding of Batch Normalization suggests that W-distance between distributions in two successive iterations should decrease. In Figure 7 , the first layer of ResNet or VGG is hard to decrease W-distance, indicating that the label distributions alway change in different iterations.

How the distribution propagate across iterations? In this experiment, we investigate how distributions propagate across different training iterations. The label distribution propagation is shown in Figure 5 . The label distribution in the first layers of ResNet or VGG often has great fluctuation in different training iterations because the supervision is insufficient when the neural network is deep. When approaching the final layer, the supervision is sufficient to decrease W-distance. These experiments justify Theorem 2 that W-distance can be decreased with sufficient supervision.

Why add auxiliary loss in some layer? When the neural network is very deep, shallow layers tend to receive insufficient supervision. Recently, some methods are proposed to introduce auxiliary loss to improve the performance of DNNs (?). However, why introduce loss in some layers and what an important role does it play in the training are not clear. From Theorem 2, when adding the supervision in one intermediate layer, W-distance can be decreased across the training iterations.

7. CONCLUSION:

In this paper, we have interpreted deep neural networks (DNNs) via understanding layer behaviors.

With the help of optimal transport theory, we propose a new analytical framework using Wasserstein 8 How to improve classification performance? We will discuss how to exploit the behaviors of different layers to improve the model performance. In practice, we observe that in deep neural networks, some samples are correctly classified in the intermediate layers but misclassified in the last layer. For example, 35.52% and 31.04% of the samples for ResNet and VGG are misclassified on the test set of VOC2007, respectively. For these samples, 5.96% and 7.76% samples are correctly classified in all intermediate layers of ResNet and VGG, respectively. Based on this phenomenon, we can improve the performance by early exiting such confused samples. Different from the strategy of SDN (Kaya et al., 2019) , how to early exit for multi-label classification DNNs is very challenging. Therefore, we design a new early-exits strategy for multi-label prediction. In Table 1 , our proposed method consistently outperforms the baseline methods. We give more details about early-exits strategy and show more results in Section H in Supplementary materials.

6.2. SINGLE-LAYER BEHAVIOR EXPLORATION:

In this section, we study how the distribution of an intermediate layer change during the training process. We consider to use W-distance to measure the stability of distribution. We propose theoretical and empirical analysis for the distribution stability in one layer.

How the distribution propagate during updating DNNs? In this experiment, we investigate how distributions propagate when training DNNs. From Figure 6 , the label distribution in the first layers of ResNet or VGG often fluctuates significantly due to the limited discriminative power of very shallow layers. When approaching the last layer, the supervision is sufficient to decrease the W-distance. These experiments justify Theorem 2 that the W-distance can be decreased with sufficient supervision. More results are shown in Section G in Supplementary materials.

How stable is the distribution in one layer? The training stability is a key problem in DNNs and can be largely addressed by the widely used Batch Normalization (BN) (Ioffe & Szegedy, 2015) . However, we prove that the stability of distribution is hard to be guaranteed even with BN (See proof in supplementary). In this paper, we use the W-distance to measure the discrepancy between the distribution in an iteration and the target distribution across different training iterations in the specific layer. Motivated by this, we further measure stability of label distribution during training DNNs. The conventional understanding of BN suggests that the W-distance should decrease. Note that, from Figure 6 , the first layer of ResNet or VGG may fluctuate during the training process, which indicates that deep layers often have better ability than shallow layers to express the target distribution.

In this paper, we have interpreted deep neural networks (DNNs) via understanding layer behaviors.

With the help of optimal transport theory, we propose a unified teacher-student analysis framework to investigate the across-layer and single-layer behaviors of a DNN. Theoretically, we prove that i) the W-distance between the distribution of any layer and the target distribution decreases along the depth. ii) the W-distance of a specific layer between the distribution in an iteration and the target distribution decreases along training iterations. Extensive experiments justify our theoretical findings. The proposed analytical framework can facilitate future researches to interpret a deep neural network.

Under review as a conference paper at ICLR 2020 functions is restricted in the set of Ω B (C), then there exists

EQUATION

Proof Directly using Lemma 1 and Theorem 2 of (Barron, 1993) can complete the conclusion.

Theorem 6 (Extend Barron theorem) Let B be a bounded set, µ be any probability measure and f (x)= n i=1 c i σ( w i , x +b i ), where σ(•) be a sigmoidal function. If support(µ 0 )⊂K 0 and ϕ i (K i-1 )⊆K i , 1≤i≤l, the function ϕ i is log(l-1) log(l)+1 -Lipschitz and Barron function, i.e., ϕ 1 ∈Ω K0 (C 0 ) and ϕ i ∈Ω Ki-1+sBm i-1 (C i ), then there exists a neural network f with l hidden layers and S ⊂ R d0 satisfying µ 0 (S)≤1 -2 s 2 (log(l-1)+1) 2 so that

EQUATION

)

Proof For simplicity, let ϕ:=ϕ l:

1 =ϕ l • • • • •ϕ 1 be the composition of Barron functions, especially ϕ 1 = ϕ k:1 , where ϕ l is defined in Corollary 1. Let f :=f l:1 =f l •f l-1 • • • • • f 1 be the composition of k-layered neural network

, where f i : R di-1 →R di-1 are functions defined as

EQUATION

where c ijk , b ijk ∈R and w ijk ∈R di-1 are parameters, and r i is the number of nodes in the i-th layer. Note that the function f i is represented by a neural network with one hidden layer and a linear output layer. We prove the theorem by induction on l.

i) For l=1, using assumptions of the support of initial distribution (i.e., support(µ 0 )⊂K 0 ) and the definition of Barron function. Then, =(2C 1 ) 2 /r 1 and r1 i=1 |c i | ≤ 2C 1 , we complete the results. ii) For l>1, by the induction step, we assume the functions f 1 , . . . , f l-1 satisfy the conclusion for ϕ 1 , . . . , ϕ l-1 . Let S l-1 be the set in the conclusion. Applying Corollary 1 to f l to get that for each 1 ≤ j ≤ m j , for any µ supported on a set K l-1 ⊆ R m l-1 and any r l ∈ N, there exists a neural net f l,j with 1 hidden layer with r l nodes such that

R m l-1 (ϕ l,j -f l,j ) 2 dµ 1 2 ≤ 2C f l ,K l-1 √ r l .

Note that Theorem 1 applies to any distribution µ supported on the set

K l-1 . Let S l = S l-1 ∩ {x : f l-1:1 (x)∈K l-1 +sB m l-1 }.

Applying Theorem 1 with

K l =K l +sB m l , r l = 4C 2 l m l (log(l-1)+1) 2 (log(l)+1) 2 2

and µ=f l-1:1 # (1 S l µ 0 ). We have that µ is supported on g l-1:1 (S l ) ⊆ K l-1 +sB m l-1 =K l-1 , and the function ϕ l is C l -Barron function on this set by the assumption. The conclusion of Theorem 1 gives (f l ) j such that

R m l-1 (ϕ l,j -f l,j ) 2 d(f l-1:1 # (1 S l µ 0 )) 1 2 ≤ 2C l √ r l ≤ √ m l (log(l-1)+1)(log(l)+1) .

For all elements of ϕ l and f l , we have

R m l-1 ϕ l -f l 2 d(f l-1:1 # (1 S l µ 0 )) 1 2

≤ (log(l-1)+1)(log(l)+1) .

Under review as a conference paper at ICLR 2020

We bound by the triangle inequality

R m 1 S l ϕ l:1 -f l:1 2 dµ 0 1 2 ≤ R m 1 S l ϕ l • ϕ l-1:1 -ϕ l • f l-1:1 2 dµ 0 1 2 + R m 1 S l ϕ l • f l-1:1 -f l • f l-1:1 2 dµ 0 1 2 ≤ R m 1 S l ϕ l • ϕ l-1:1 -ϕ l • f l-1:1 2 dµ 0 1 2 + R m 1 S l ϕ l -f l 2 df l-1:1 # (1 S l µ 0 ) 1 2 ≤Lip(ϕ l ) R m 1 S l ϕ l-1:1 -f l-1:1 2 dµ 0 1 2 + (log(l-1)+1)(log(l)+1) ≤Lip(ϕ l ) R m 1 S l-1 ϕ l-1:1 -f l-1:1 2 dµ 0 1 2 + (log(l-1)+1)(log(l)+1) ≤ log(l-1) log(l)+1 • log(l-1)+1 + (log(l-1)+1)(log(l)+1) ≤ log(l)+1 ,

where the last inequality holds by the assumption (i.e., Lip(ϕ l )= log(l-1) log(l)+1 ) and the induction hypothesis.

As above analysis, we have

1 S l-1 ϕ l-1:1 -f l-1:1 dµ 0 2 dµ 0 ≤ 2 (log(l-1)+1) 2

, by the induction hypothesis. Also, ϕ l-1:1 ∈ K l-1 for all x ∈ support(µ 0 ) by the assumption of the theorem. Therefore, by Markov's inequality and the induction hypothesis on S l-1 ,

µ 0 S l-1 ∩ x : x / ∈ K l-1 + sf l-1:1 (B m l-1 ) ≤µ 0 (S l-1 ∩ {x : ϕ l-1:1 -f l-1:1 ≥ s}) ≤ 2 s 2 (log(l-1)+1) 2 . Therefore, µ 0 (S l ) ≤ µ 0 (S l-1 ) - 2 s 2 (log(l-1)+1) 2 ,

the last equality holds by a fact that there exists a constant δ to satisfy µ 0 (S l ) = 1-

δ 2 s 2 (log(l-1)+1) 2 .

Based on the above conclusions, we prove Theorem 4 as follow.

Theorem 4 Given a data distribution µ 0 and a function ϕ i : R mi-1 →R mi , and let

L l = log(l)+1, if support(µ 0 )⊂K 0 and ϕ i (K i-1 )⊆K i , 1≤i≤l, the function ϕ i is L l-1 -1 L l

-Lipschitz and is a Barron function, i.e., ϕ 1 ∈Ω K0 (C 0 ) and ϕ i ∈Ω Ki-1+sBm i-1 (C i ), then there exists a network f with l hidden layers with 4C 2 l m l L 2 l-1 L 2 l / 2 neurons on the i-th layer,

EQUATION

)

where D l is the diameter of the set K l , and , δ, s>0 are parameters.

Proof The functions f 1 , • • • , f l in Theorem 6 satisfy that

1 S l-1 ϕ l-1:1 -f l-1:1 dµ 0 2 dµ 0 ≤ 2 (log(l-1)+1) 2 . The range of f l = ((f l ) 1 , • • • , (f l ) m l ) is contained in a set of diameter 2C l

√ m l because the activation in a neural network is ranged in [0, 1] and the weights c ijk in Theorem 6 satisfy

r k=1 |c ijk | ≤ 2C l .

Under review as a conference paper at ICLR 2020 Choose a constant vector k to minimize S l ϕ l:1 (x)f l:1k 2 dµ 0 and replace f l with f l + k. Note that the range of f l and ϕ l necessarily overlap. We still have S l ϕ l:1 (x)f l:1 2 dµ 0 ≤ 2 (log(l)+1) 2 . Moreover, we have

ϕ l (x) -f l (x) ≤ 2C l √ m l + D l ,

for x ∈ K 0 , where D l is the diameter of K l . Let S c l be a complementary of S l , we have

µ(S c l ) = δ 2 s 2 (log(l-1)+1) 2 , then we have K0 ϕ l:1 -f l:1 2 dµ 0 ≤ S l ϕ l:1 -f l:1 2 dµ 0 + S c l ϕ l:1 -f l:1 2 dµ 0 ≤ 2 (log(l) + 1) 2 + (2C l √ m l + D l ) 2 δ 2 s 2 (log(l-1)+1) 2 = 2 (log(l) + 1) 2 (2C l √ m l + D l ) 2 δ s 2 + 1 . For (ϕ l:1 (X), f l:1 (X)), X ∼ µ 0 define a coupling between distributions. Based on the definition of W-distance, we have W 2 (μ k , µ) =E X∼µ0 ϕ l:1 -f l:1 2 = K0 ϕ l:1 -f l:1 2 dµ 0 ≤ 2 (log(l) + 1) 2 (2C l √ m l + D l ) 2 δ s 2 + 1 .

C PROOFS OF LAYER BEHAVIOR ANALYSIS OF DEEP NEURAL NETWORKS:

Definition 9 (Flow) A flow ϕ t is given by an ordinary differential equation (ODE), φt (x)=v t (ϕ t (x)) with a velocity field v t when t>0, and ϕ 0 (x)=x when t=0.

Proposition 2 (Continuity equation (Sonoda & Murata, 2019) ) Let ϕ t be the flow of an ODE with vector field v t , then the distribution µ t evolves according to the continuity equation

∂ t µ t (x)=-∇•[µ t (x)v t (x)], x∈R m , t≥0

, where ∇• denotes the divergence operator.

Theorem 7 (Optimal transport map) The global minimum

f * t of L(f ) is attained at f * t ( x) = x - 1 ν t * µ 0 ( x) R m εν t (ε)µ 0 ( x -ε)dε,

where * denotes the convolution operator.

Proof The proof follows from the calculus of variations.

L(f ) = R m E ε f (x + ε) -f (x) 2 µ 0 (x)dx = R m E ε f (x ) -f (x -ε) 2 µ 0 (x -ε) dx ,

where the second line holds by the calculus of variations, i.e., x = x + ε. Then, for an arbitrary function h, the first variation δL(h) is given by

δL(h) = d ds L(f + sh) s=0 = R m ∂ ∂s E ε f (x) + sh(x) -f (x -ε) 2 µ 0 (x -ε) dx s=0 =2 R m E ε [(f (x) -f (x -ε)) µ 0 (x -ε)] h(x)dx.

Under review as a conference paper at ICLR 2020 At a critical point f * of L, δL(h) = 0 for every h. Hence,

E ε [(f (x) -f (x -ε)) µ 0 (x -ε)] = 0, a.e.

x, by the fundamental lemma of calculus of variations for integrable functions, and we have

f * (x) = E ε [f (x -ε)µ 0 (x -ε)] E ε [µ 0 (x -ε)] .

Note that f * attains the global minimum, because, for every function h,

L(f * + h) = R m E ε f * (x) -f (x -ε) + h(x) 2 µ 0 (x -ε) dx = R m E ε f * (x) -f (x -ε) 2 µ 0 (x -ε) dx + R m E ε h(x) 2 µ 0 (x -ε) dx + 2 R m E ε h(x) T (f * (x) -f (x -ε)) µ 0 (x -ε) dx =L(f * ) + L(h) + 2 • 0 ≥ L(f * ).

Theorem 1 If the target network is a residual unit f ( x)=x+σ( x), where σ contains fully connected layer and activation function. When ν t := N (0, tI), then the global minimum

f * t is f * t ( x) = x + t∇ log(ν t * µ 0 )( x) + σ( x) := x + g t ( x).

Proof Using Stein's identity (Liu, 1994) , -t∇ν t (ε) = εν t (ε), which is known to hold only for Gaussians.

f * t ( x) = x - 1 ν t * µ 0 ( x) R m εν t (ε)µ 0 ( x -ε)dε + σ( x) = x + 1 ν t * µ 0 ( x) R m t∇ν t (ε)µ 0 ( x -ε)dε + σ( x) = x + t∇ν t * µ 0 ( x) ν t * µ 0 ( x) + σ( x) = x + t∇ log(ν t * µ 0 ( x)) + σ( x).

Theorem 2 Based on the equivalent condition in Theorem 2 (Belavkin, 2016), and let ∆ be the Laplacian operator. At the initial moment t→0, the pushforward f # µ t with Gaussian distribution satisfies the backward heat equation (BHE), and evolves according to Wasserstein gradient flow:

∂ t µ t=0 (x)=-∆µ 0 (x)=-grad W 2 (µ 0 , µ), x∈R m .

Proof The initial velocity vector is given by

EQUATION

Hence, by substituting the score (13) in the continuity equation of Proposition 2, we have

∂ t µ t=0 (x) = -∇•[µ 0 (x)(∇ log µ 0 (x) + σ(x))] = -∆µ 0 (x).

We leave the proof of ∆µ 0 (x) = W 2 (µ t , µ) in Theorem 3.

Theorem 3 Based on the equivalent condition in Theorem 2 (Belavkin, 2016), and when the noise distribution is a Gaussian distribution, then the pushforward measure µ t :=ϕ t# µ 0 evolves according to Wasserstein gradient flow as follows:

EQUATION

D OPTIMAL TRANSPORT:

Definition 10 (Optimal Transport (Villani, 2008) ) Given a cost function c: K×K→R, the optimal transport distance measures the optimal plan to transport the mass from a probability measure µ to another probability measure µ:

EQUATION

where Π( µ, µ) is the set of joint probability measures π on K×K with the marginals µ and µ.

In this paper, we consider the multi-label classification problem. We first introduce the definition of Wasserstein distance. For the case of probability measures, they are histograms in the simplex ∆ K .

Definition 11 (W-distance) Given an input x∈X and any f : X →∆ K , let f (x) be the predicted label distribution µ, and let y be the target distribution µ, then the W-distance between µ and µ is

EQUATION

where C∈R K×K + is the cost matrix whose the element can be defined as C κ,κ =d p K (κ, κ ), and the set of couplings is composed of joint probability distributions with their marginals μ and µ, i.e., Π( µ, µ)={T ∈R K×K + : T 1= µ, T T 1=µ}.

Entropic Wasserstein loss. Cuturi (2013) introduces an entropic regularization such that the nonconvex problem (2) can be turned to a strictly convex problem:

EQUATION

where

H(T )=-κ,κ T κ,κ (log(T κ,κ )-1).

With the help of the entropic regularization, we solve optimal transport problem efficiently using Sinkhorn-Knopp matrix scaling algorithm (Knight, 2008) .

Proposition 5 The transport matrix T * optimizing Problem (17) satisfies T * = diag(u) Kdiag(v) , where K = e -αM , u = e αa and v = e αb , where a and b are the Lagrange dual variables.

Theorem 8 (Genevay et al., 2017) Wasserstein sinkhorn loss between the predicted label distribution µ and the target label distribution µ is defined as:

EQUATION

with the following limiting behavior as α → 0: W2 ( µ, µ) → 2 W 2 α ( µ, µ).

Note that normalized Wasserstein loss is non-negative and W2 α ( µ, µ) = 0 if and only is µ = µ. In the quantification, we use normalized Wasserstein loss to measure the divergence between distributions.

E INTERMEDIATE LAYER:

We add auxiliary classifiers for the intermediate layers as mentioned below. For ResNet-18, we truncate each block and add a classifier after its feature map. We label these intermediate layers from 0-th layer to 8-th layer. The truncation of ResNet-50 is analogous to ResNet-18. For VGG-16, we add a classifier after each convolution layer and label them from 0-th layer to 12-th layer. and divide it into training set and test set following the ratio of original dataset. We shuffle twice and define them as "trial_0" and "trial_1", respectively. Besides, the "orig." means that we use the original dataset. From Figure 9 , different experiments consistently have the same decreasing tendency through the depth of a neural network.

Under review as a conference paper at ICLR 2020 (Frogner et al., 2015) . Therefore, we use Wasserstein distance to quantify the discrepancy of label distributions. 

H EARLY-EXITS STRATEGY:

Deep neural networks often occur the over-thinking issue, that is, the samples are correctly classified in the intermediate layer but misclassified in the last layer. We call these samples as confused samples.

From Table 2 and 3, the accuracy of ResNet-18 (VOC) and VGG-16 (VOC) are 64.48% and 68.96%, respectively. In other words, out of 35.52% and 31.04% of the samples for ResNet-18 and VGG-16 are misclassified on the test set of VOC2007, respectively. For these misclassified samples, 1.17% are actually correctly classified at 0-th layer, 0.44% are at 1-th layer and so on. Ideally, if we early exit these confused samples, the cumulative accuracy of ResNet-18 is 70.44% on VOC dataset. Early-exits strategy. We propose a simple probability mechanism to exit confused samples. Specifically, we add an auxiliary classifier for a selected intermediate layer to get the prediction probability distribution. Each auxiliary classifier contains fully connected layer and Sigmoid function. For each sample, we denote the number of exceeding the threshold p = 0.5 as N . We also denote the number of exceeding another threshold q as n. We denote the ratio as γ and γ = n/N . The threshold q and ratio γ are range from 0.5 to 1. We search for the best values of them to improve classification performance and show the results on Table 4 . 

:



Figures and Tables:

Figure 1: Demonstration of a DNN forward, distributions propagation and the intuition of the label distribution. (a) Given an input x, we train hidden layers and classifiers to output label distributions µ l . (b) Across-layer propagation: the label distribution µ (t) 0 propagate to µ (t) L . Single-layer propagation: the label distribution µ (0)k propagate to µ (t) k during the epochs in one layer. (c) Given an image with two labels (i.e., cat and dog), the label distribution depicts their corresponding probabilities of 0.5.

finite-layer behavior analysis (b) infinite-layer behavior analysis

Figure 2: The teacher-student paradigm for analyzing layer behaviors of DNNs. We analyze one-layer behavior in (a), infinite-layer behavior in (b) and finite-layer behavior in (c).

(c), given a composition of L T-nets, we try to analyze the ability of each layer of S-net to express the distribution of T-net. Hence, we derive an approximation bound for finitely deep neural networks. Here, Barron function is present a T-net and its definition is provided as follows. Definition 3 (Barron function (Sonoda & Murata, 2019)) The function ϕ is Barron function on B if ϕ satisfies Ω B (C)={ϕ: B→R: ∃φ, φ| B =ϕ, φ * B ≤C, φ∈F B }, φ * B := R m w B | φ(w)|dw, where w B = sup x∈B | w, x | and φ is the Fourier transform of φ, and F B is the set of functions with the Fourier inversion formula holds on B.

Figure 3: Wasserstein distance across different layers for different networks.

Figure 4: Wasserstein distance across different layers of ResNet (left) and VGG (right).

Figure 4: Distribution propagation across different layers for different networks.

Figure 5: Demonstration of properties on easy, hard and confused samples.

Figure 7: Wasserstein distance across different training iterations for ResNet and VGG on VOC and COCO datasets. We choose the 1, 4, 9-th layer for ResNet and the 1, 6, 13-layer for VGG.

Figure 6: Wasserstein distance between the distribution in an epoch and the target distribution across different training epochs for different networks. We choose the 1, 4, 9-th layer for ResNet and the 1, 6, 13-layer for VGG.

Figure 7: Definition of intermediate layer of ResNet-18.

Figure 8: Definition of intermediate layer of VGG-16.

Figure 9: Wasserstein distance across different layers of ResNet-18 (left) and VGG-16 (right) on different test set. Specifically, we shuffle the VOC2007 dataset (including training set and test set)and divide it into training set and test set following the ratio of original dataset. We shuffle twice and define them as "trial_0" and "trial_1", respectively. Besides, the "orig." means that we use the original dataset. From Figure9, different experiments consistently have the same decreasing tendency through the depth of a neural network.

Figure 9: Definition of intermediate layer of VGG-16.

Figure 10: Wasserstein distance across different layers of ResNet-18 (left) and VGG-16 (right) on test set. The tendency of distribution propagation on test set is the same as the training set(Figure3), suggesting that the distribution propagation is irrelevant to the property of the dataset.

Figure 9: Definition of intermediate layer of VGG-16.

Figure 12: Jensen-Shannon divergence across different layers of ResNet-18 (left) and VGG-16 (right) on training set.

Figure 15: Wasserstein distance between distributions of adjacent training epoch.

and MS-COCO(Maas et al., 2013). VOC2007(Everingham et al., 2010) contains 9,963 images from 20 object categories, which is divided into train, val and test sets. MS-COCO(Maas et al., 2013) contains 82,783 images as the training set and 40,504 images as the validation set. The objects are categorized into 80 classes with about 2.9 object labels per image. We use the classification accuracy, average per-class F1 (CF1), average overall F1 (OF1) and mean average precision (mAP) as the evaluation metrics. Improve performance of ResNet and VGG.

Table content: <table><tr><td>6.1 ACROSS-LAYER BEHAVIOR EXPLORATION</td></tr><tr><td>6.1.1 THE BEHAVIORS OF LABEL DISTRIBUTIONS IN DIFFERENT LAYERS</td></tr><tr><td>How does every layer express the distribution? Here, we study the expression ability of every layer, which is measured by Wasserstein distance (W-distance) between the label distribution of any layer and the target distribution. As suggested in Theorem 4, deep layers have smaller error bound</td></tr></table>

Improve performance of ResNet-50.

Table content: <table><tr><td>#</td><td>#</td></tr><tr><td>#</td><td>#</td></tr><tr><td>Method ResNet ResNet+DSN ResNet+EarlyExits</td><td>VOC accuracy CF1 69.79 60.93 60.23 88.84 OF1 mAP accuracy CF1 COCO 33.81 61.25 64.01 71.77 OF1 mAP 70.40 61.29 60.39 89.04 34.19 61.33 64.27 72.41 70.98 62.84 60.21 89.14 35.70 62.39 64.76 73.68</td></tr><tr><td/><td>16</td></tr></table>

Improve performance of ResNet-50.Figure 11: distance across different layers of ResNet-18 (left) and VGG-16 (right) on training set. Although the tendency of Chebyshev distance is analogous to Wasserstein distance, Chebyshev distance ignores any metric structure

Table content: <table><tr><td>#</td><td>#</td></tr><tr><td>#</td><td>#</td></tr><tr><td>Method ResNet ResNet+DSN ResNet+EarlyExits</td><td>VOC accuracy CF1 69.79 60.93 60.23 88.84 OF1 mAP accuracy CF1 COCO 33.81 61.25 64.01 71.77 OF1 mAP 70.40 61.29 60.39 89.04 34.19 61.33 64.27 72.41 70.98 62.84 60.21 89.14 35.70 62.39 64.76 73.68</td></tr><tr><td/><td>16</td></tr></table>

Over-thinking results of ResNet-18.

Table content: <table><tr><td>layer VOC2007 1.17% 0.44% 0.69% 0.71% 0.55% 0.50% 0.53% 0.85% 0.53% 64.48% 0 1 2 3 4 5 6 7 8 9 MS-COCO 0.49% 0.43% 0.47% 0.28% 0.30% 0.31% 0.37% 0.78% 0.50% 27.33%</td><td>ideal result 70.44% 31.26%</td></tr></table>

Over-thinking results of ResNet-18.Figure 16: Distribution propagation across different training epochs of ResNet-18 and VGG-16.

Table content: <table><tr><td>layer VOC2007 1.17% 0.44% 0.69% 0.71% 0.55% 0.50% 0.53% 0.85% 0.53% 64.48% 0 1 2 3 4 5 6 7 8 9 MS-COCO 0.49% 0.43% 0.47% 0.28% 0.30% 0.31% 0.37% 0.78% 0.50% 27.33%</td><td>ideal result 70.44% 31.26%</td></tr><tr><td>17</td><td/></tr></table>

Over-thinking results of ResNet-18.

Table content: <table><tr><td>layer VOC COCO 0.49% 0.43% 0.47% 0.28% 0.30% 0.31% 0.37% 0.78% 0.50% 27.33% 0 1 2 3 4 5 6 7 8 9 1.17% 0.44% 0.69% 0.71% 0.55% 0.50% 0.53% 0.85% 0.53% 64.48%</td><td>ideal result 70.44% 31.26%</td></tr></table>

Over-thinking results of VGG-16.

Table content: <table><tr><td>layer VOC COCO 0.35% 0.35% 0.27% 0.38% 0.33% 0.19% 0.21% 0.17% 0.18% 0.13% 0.38% 0.37% 1.63% 32.41% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 2.34% 0.00% 0.71% 0.67% 0.50% 0.24% 0.48% 0.22% 0.20% 0.14% 0.28% 0.34% 1.62% 68.96%</td><td>ideal result 76.72% 37.35%</td></tr></table>

Improve performance of ResNet and VGG.

Table content: <table><tr><td>Method ResNet-18 ResNet-18+EarlyExits VGG-16 VGG-16+EarlyExits ResNet-50 ResNet-50+EarlyExits</td><td>VOC accuracy CF1 64.48 58.02 59.10 85.18 OF1 mAP accuracy CF1 COCO 27.33 55.65 60.30 64.36 OF1 mAP 66.01 58.67 58.76 85.49 30.03 57.49 61.38 67.28 68.96 58.58 59.71 88.48 32.41 59.37 62.94 70.64 69.85 59.49 59.94 88.57 33.95 60.32 63.73 71.95 69.79 60.93 60.23 88.84 33.81 61.25 64.01 71.77 70.98 62.84 60.21 89.14 35.70 62.39 64.76 73.68</td></tr></table>

Peer review (iclr2020):
This paper seeks to understand both across-layer and single-layer behavior within neural networks (i.e. layer behavior along the depth of a network, and behavior of a single layer along training epochs). Therefore, they resort to the optimal transport framework to compare predicted and target distributions. Theoretically, they show that the Wasserstein distance between predicted and target distributions is decreasing along the depth and for a single layer, along training iterations. They also give intuition on how this analysis can help the learning process in practice. This paper gives an interesting contribution to the in-depth analysis of neural networks. However, some elements remain unclear: 1. The setting of multi-label classification does not really motivate the use of measures. 2. It is unclear why the use of teacher/student networks are pertinent or necessary. 3. There is no detail on the regularization strength of the Wasserstein distance, or what p (in definition 1) is chosen either in the experiments or in the theorems. 4. I believe it is understated that all 	ilde{f}_i have the same input and output domains (as well as h=h_i in figure 1a), which is restrictive and should have been made clearer. - Post rebuttal: I thank the authors for their response. On this basis, I am maintaining my weak reject rating.