Year: 2019

Abstract:
Deep neural networks (DNNs) have achieved unprecedented practical success in many applications. However, how to interpret DNNs is still an open problem. In particular, what do hidden layers behave is not clearly understood. In this paper, relying on a teacher-student paradigm, we seek to understand the layer behaviors of DNNs by “monitoring” both across-layer and single-layer distribution evolution. Here, the “across-layer” and “single-layer” considers the layer behavior along the depth and a specific layer along training epochs, respectively. Relying on optimal transport theory, we employ the Wasserstein distance (W-distance) to measure the divergence between the layer distribution and the target distribution. Theoretically, we prove that i) the W-distance between the distribution of any layer and the target distribution tends to decrease along the depth. ii) For a specific layer, the W-distance between the distribution in an iteration and the target distribution tends to decrease along training iterations. iii) However, a deep layer is not always better than a shallow layer for some samples. Moreover, our results helps to analyze the stability of layer distributions and explains why auxiliary losses help the training of DNNs. Extensive experiments justify our theoretical findings.

text: Deep neural networks (DNNs) have achieved unprecedented practical success in many applications. However, how to interpret DNNs is still an open problem. In particular, what do hidden layers behave is not clearly understood. In this paper, relying on a teacher-student paradigm, we seek to understand the layer behaviors of DNNs by “monitoring” both across-layer and single-layer distribution evolution. Here, the “across-layer” and “single-layer” considers the layer behavior along the depth and a specific layer along training epochs, respectively. Relying on optimal transport theory, we employ the Wasserstein distance (W-distance) to measure the divergence between the layer distribution and the target distribution. Theoretically, we prove that i) the W-distance between the distribution of any layer and the target distribution tends to decrease along the depth. ii) For a specific layer, the W-distance between the distribution in an iteration and the target distribution tends to decrease along training iterations. iii) However, a deep layer is not always better than a shallow layer for some samples. Moreover, our results helps to analyze the stability of layer distributions and explains why auxiliary losses help the training of DNNs. Extensive experiments justify our theoretical findings.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) have been successfully applied in computer vision, such as image classification (Chen et al., 2019b; Hsu et al., 2019), image generation (Cao et al., 2019; Brock et al., 2019; Chrysos et al., 2019) and speech recognition (Yeh et al., 2019; Chen et al., 2019a). Despite their success, the internal mechanism of DNNs is still a black box. In particular, understanding what do hidden layers do and how to achieve remarkable performance remain persistently elusive. To answer these questions, we seek to understand across-layer and single-layer behavior.
For the across-layer behavior, we study the learning process by measuring theW-distance between the distribution of any layer and the target distribution. Recently, most methods only focus on final predictions in different tasks (He et al., 2016). Due to the end-to-end training, interpreting each intermediate layer behaviors of a DNN, which, however, is still not clear. To provide interpretability, existing works try to produce a single prediction and observe the classification performance of each layer (Papernot & McDaniel, 2018; Szegedy et al., 2013; Kaya et al., 2019). For example, (Alain & Bengio, 2016) experimentally observe that the linear separability of features increases monotonically along the depth of a DNN. Unfortunately, there is no theoretical analysis to support the experimental finding. To understand the learning process of DNNs, one can explore the across-layer behavior by monitoring how the distributions propagate across different layers. However, how to open the internal mechanism of DNNs and investigate the across-layer behavior of a DNN remains an open question.
For the single-layer behavior, we seek to measure the W-distance between the distribution in an iteration and the target distribution. It is important to analyze the distributional stability of one layer by measuring the change of the distributions. Recently, the distributional stability of a deep neural network attracts extensive attention (Santurkar et al., 2018). Some studies try to visualize the training behavior of one layer by plotting the mean and variance of features (Santurkar et al., 2018). Unfortunately, the visualizations are subjective and lack of necessary theoretical justifications. To this end, (Sonoda & Murata, 2019) propose a transport analysis method and state that a denoising Autoencoder transports mass to decrease the Shannon entropy of the data distribution. However, the
Under review as a conference paper at ICLR 2020
method is limited and inflexible for analyzing a general case of deep neural networks. Therefore, it is very necessary and important to develop a new analytical method to interpret the stability of layers.
In this paper, we apply optimal transport theory to analyze the behavior of distributions. Specifically, we exploit Wasserstein divergence (W-distance) to measure the difference between the distribution of any layer and the target distribution. By monitoring the change of theW-distance, we are able to study both across-layer and single-layer behaviors. Our contributions are summarized as follows.
• We analyze the across-layer behavior and prove that theW-distance between the distribution of any layer and the target distribution decreases along the depth of a DNN. It means that every layer of the network can express the target distribution progressively. • We analyze the single-layer behavior and prove that for a specific layer, theW-distance between the distribution in an iteration and the target distribution decreases across the training iterations when introducing a loss in the layer. • Moreover, we provide experiments and theoretical justifications on these findings. The proposed
analytical framework provides a different view of understanding and interpreting neural networks.

heading: 2 RELATED WORK
text: Many studies analyze the features of intermediate layers. (Bau et al., 2017) evaluates how hidden units align a set of semantic concepts to quantifies the interpretability of latent representations of a CNN. (Dosovitskiy & Brox, 2016) inverts image representations with up-convolutional networks for studying image representations. (Zeiler & Fergus, 2014) presents a visualization technique to investigate the function of intermediate feature layers and the classifier. (Zhang et al., 2018) learns an explanatory graph to reveal the knowledge hierarchy hidden inside a pre-trained CNN. (Zhang et al., 2019) learns a decision tree to clarify the specific reason for each semantic prediction of a CNN.
Some studies analyze the classification accuracy of a DNN. (Lee et al., 2015) introduces a classification objective to the individual hidden layers, in addition to the overall objective at the output layer. (Szegedy et al., 2015) uses auxiliary classifiers to improve the classification performance. (Kaya et al., 2019) introduces internal classifiers into off-the-shelf DNNs to understand overthinking of networks by studying how the prediction changes during a DNN’s forward pass. (Gupta & Schütze, 2018) explains recurrent neural networks by understanding the layer-wise semantic accumulation behavior. (Sonoda & Murata, 2019) investigates the feature map inside a DNN by tracking the transport map, and prove that a deep Gaussian DAE transports mass to decrease Shannon entropy of the data distribution. (Alain & Bengio, 2016) experimentally observes that the linear separability of features increases monotonically along the depth of a DNN. However, there is no theoretical result to analyze this phenomenon. Existing studies using information bottleneck methods (Tishby & Zaslavsky, 2015; Bang et al., 2019) mainly analyze the dynamics of across different layers. However, it is hard for these methods to analyze the dynamics of a specific layer through different iterations. In contrast, our proposed method is able to analyze both single-layer and across-layer behaviors.
Under review as a conference paper at ICLR 2020

heading: 3 PROBLEM SETTING
text: Notation. We use bold lower-case letters (e.g., x) to denote vectors, and bold upper-case letters (e.g.,X) to denote matrices. We denote the transpose of a vector (e.g., xT) or matrix (e.g.,XT) by the superscript T. Let 1:=[1, . . . , 1]T and 1S(·) be an indicator function of a set S, and let [n]={0, 1, . . . , n}. Let ∗ be the convolution operator, grad be gradient operator. Label distribution learning. In this paper, we consider a label distribution learning problem (Geng, 2016), especially multi-label classification. In this setting, the label distribution is defined as a probability distribution to cover a certain number of labels, representing the degree to which each label describes the instance (see Figure 1 (c)). Note that the sum of the label distribution is equal to 1. Specifically, given training samples S, we seek to learn a model f from data space X into the label space Y . In Figure 1(a), for L-layered neural networks, we denote f̃0:l=f̃l◦ · · · ◦f̃0, l≤L as the output of the l-th layers, then the label distribution mapping can be defined as fl:=hl◦f̃0:l, where hl is a probability function (e.g., FC+softmax (Frogner et al., 2015)). Note that fl, l∈[L] have the same input domains and the same output domains. Our goal is to learn a model fl to let the predicted distribution µ̂l close to the target distribution µ. Then, we optimize the empirical risk: minfl L(fl) := ES [d(µ̂l, µ)] , where µ̂l=fl#(µ0), l≤L, (1) where ES [·] is the expectation w.r.t. the training set S, fl# denotes the pushforward operator of µ0, and d(·, ·) is some distribution divergence, such as Cross-entropy (CE) loss and Wasserstein loss. In practice, the label distribution is obtained by training with CE loss. Because of the convexity of CE loss, we derive the optimal label distribution for given features of every layer (Alain & Bengio, 2016). In this sense, the label distribution reflects the actual distribution of feature maps in a specific layer.
Label distribution propagation. In this paper, our goal is to explore how the layer distributions propagate in the across-layer and single-layer of a DNN. Specifically, we measure the divergence (e.g., Wasserstein distance (Villani, 2008)) to observe how the layer distributions change across different layers or iterations. In Figure 1(b), we explain the behaviors of layer from the following two perspectives. For the across layers, measuring the distance between the distribution of any layer and the target distribution helps to understand the learning process and distribution expression ability. For a single layer, measuring the distance between the distribution in an training iteration and the target distribution helps to understand the dynamic behavior and training process of a DNN.

heading: 4 WASSERSTEIN DISTRIBUTION PROPAGATION QUANTIFICATION
text: Optimal transport (OT). Recently, the performance of multi-label classification can be improved by usingW-distance (Frogner et al., 2015; Zhao & Zhou, 2018). However, DNNs on the multi-label classification task still lack theoretical understanding. With the help of OT theory (Villani, 2008), W-distance can be used to interpret DNNs. In contrast, some measures, such as Jensen–Shannon divergence, which, however, often ignore any metric structure of distribution (Frogner et al., 2015).
Definition 1 (W-distance (Villani, 2008)) Given a prediction distribution µ̂ and the target distribution µ, and the cost matrix C defined as Cκ,κ′=d p K(κ, κ
′) with the metric dK, where κ, κ′ are label tags, then theW-distance seeks to find a transportation matrix T by transporting a mass µ̂ to µ,
W2(µ̂, µ) = infT∈Π(µ̂,µ)〈T ,C〉, (2) where Π(µ̂, µ) is the set of couplings and defined as Π(µ̂, µ)={T∈RK×K+ : T1=µ̂,T T1=µ}.
In practice, the distance of two label tags in the cost matrix can be calculated by using word2vec (Mikolov et al., 2013) in the Euclidean space. We calculateW-distance with entropic regularization by Sinkhorn algorithm (Knight, 2008) with p=2. See more details in Supplementary materials.
Distribution propagation measure. In this paper, we use W-distance to measure the acrosslayer and single-layer label distribution propagation, as shown in Figure 1(b). For an acrosslayer, we measure W-distance between the distribution of any layer and the target distribution, i.e.,W2(µ̂1, µ), . . . ,W2(µ̂L, µ). In this way, we can measure how the distributions propagate across different layers. For the single-layer, we measureW-distance between the distribution in an training iteration and the target distribution, i.e.,W2(µ̂(0)l , µ), . . . ,W 2(µ̂ (t) l , µ). Therefore, we can study the behavior of distributions in the same layer.
Under review as a conference paper at ICLR 2020
Teacher-student analytical framework

heading: 5 TEACHER-STUDENT FRAMEWORK FOR LAYER BEHAVIOR ANALYSIS
text: Teacher-student analysis framework is widely used to analyze and understand DNNs (Tian, 2017; Du et al., 2018; Cuturi, 2013). For the one-layer behavior, this framework helps to understand the dynamic of student network from the teacher network. For the multi-layer behavior, this framework helps to study the ability of the student network to express distributions of the teacher network. Therefore, teacher-student analysis framework is important and necessary to understand DNNs.

heading: 5.1 ONE-LAYER BEHAVIOR ANALYSIS
text: In our problem setting, the target distribution in the training loss (1) can be represented by an output of a teacher network (T-net). Our goal is to make the output of the student network (S-net) to be close to the output of T-net, as shown in Figure 2 (a). Specifically, given a S-net f and T-net f ′, our goal is to find a function f that minimizes the following training loss,
L(f) := ES [d(µ̂l, µ)] := Ex∼S [ ‖f(x̃)− f ′(x)‖2 ] , (3)
where x̃=x+ε, ε∼νt is the corruption of data, and νt is a noise distribution w.r.t. the variance tI , i.e., νt=N (0, tI). By optimizing the training loss, the distribution in one layer can be monitored by using a transport map. Next, we provide the definition of transport map below.
Definition 2 (Transport map) A transport map ft: Rm→Rm can be defined as ft(x)=x+gt(x) with an update direction gt when t>0, and f0(x)=x when t=0.
Based on the definition of the transport map, we analyze the layer behavior of ResNet (He et al., 2016), i.e., T-net is a residual block. Note that our analysis method can be extended to a more general case. Then, we solve the optimal transport map as follow.
Theorem 1 (Optimal transport map) If the teacher network is a residual unit f ′(x̃)=x+σ(x̃), where σ contains FC layer and an activation function. When νt:=N (0, tI), the minimum f∗t is
f∗t (x̃) =x̃+ t∇ log(νt ∗ µ0)(x̃) + σ(x̃) := x̃+ gt(x̃).
The minimizer f∗t updates x along the update direction gt(x). Therefore, the optimal transport map f∗t with time t transports the mass at the activation x toward x+gt(x). To associate with continuity equation (Sonoda & Murata, 2019), we introduce Wasserstein gradient flow (WGF) as follows.
Proposition 1 (Wasserstein gradient flow) Assume µt satisfies the continuity equation with the gradient vector field ∂tµt=−∇·[µt∇Vt] of a potential function Vt, and if the function F satisfies ∂tF (µt)= ∫ Rm −Vt(x)[∂tµt](x)dx, then WGF coincides with ∂tµt=−gradF (µt).
Using the definition of WGF, we explore how the distribution propagate in one layer as follows.
Theorem 2 Based on the equivalent condition in Theorem 2 (Belavkin, 2016), and let ∆ be the Laplacian operator. At the initial moment t→0, the pushforward f#µt with Gaussian distribution satisfies the backward heat equation (BHE), and evolves according to Wasserstein gradient flow:
∂tµt=0(x)=−∆µ0(x)=− gradW2(µt, µ),x∈Rm.
Note thatW is the potential function. Theorem 2 states that by introducing an auxiliary loss in a layer, theW-distance can be decreased in the layer. It means that the label distribution can be close to the target distribution across training iterations.
Under review as a conference paper at ICLR 2020

heading: 5.2 MULTI-LAYER BEHAVIOR ANALYSIS
text: (i) Infinitely deep neural networks. For an infinitely DNN, we assume the number of layers and hidden units and the size of dataset is infinite. Specifically, let 0=t0<t1< · · ·<tL+1=t, and let f0 be trained on µ0 with the variance t1−t0, then x1:=f0(x0) and its distribution is µ1:=f0#µ0. Then, we train f1 on µ1 with the variance t2−t1. In the same way, we obtain fl(xl) and µl:=fl−1#µl−1. The composition of residual blocks can be written as f t0:L(x):=fL◦· · ·◦f0(x), where t is total time. Then, the velocity of the composition coincides with the vector field ∂tf t=tl0:l (x)=∇Vtl(x). Furthermore, we extend one-layer behavior analysis to the case of infinite layers, as shown in Figure 2 (b). Based on Theorem 1, we have the dynamical systems as follows.
Lemma 1 When the noise distribution is a Gaussian distribution, the continuous residual units is defined as a flow ϕt: Rm→Rm of the following dynamical systems associated with the vector field ∇Vt, i.e., ddtx(t)=∇ log(µt(x(t)))+σ(x(t)), where t≥0 and µt:=ϕt#µ0.
Based on the proofs of Theorem 2, we can achieve a similar conclusion as follows.
Theorem 3 Based on the equivalent condition in Theorem 2 (Belavkin, 2016), and when the noise distribution is a Gaussian distribution, then the pushforward measure µt:=ϕt#µ0 evolves according to Wasserstein gradient flow as follows: ddtµt(x)=−gradW2[µt, µ], µt=0=µ0.
Theorem 3 states that theW-distance can be decreased across different layers. It means that the label distribution can be close to the target distribution along the depth of the layer.
(ii) Finitely deep neural networks. When the number of layers and the hidden unit number are finite, the above analysis is not available. As shown in Figure 2 (c), given a composition of L T-nets, we try to analyze the ability of each layer of S-net to express the distribution of T-net. Hence, we derive an approximation bound for finitely deep neural networks. Here, Barron function is present a T-net and its definition is provided as follows.
Definition 3 (Barron function (Sonoda & Murata, 2019)) The function ϕ is Barron function on B if ϕ satisfies ΩB(C)={ϕ: B→R: ∃φ, φ|B=ϕ, ‖φ‖∗B≤C, φ∈FB}, ‖φ‖∗B:= ∫ Rm ‖w‖B|φ̂(w)|dw, where ‖w‖B= supx∈B |〈w,x〉| and φ̂ is the Fourier transform of φ, and FB is the set of functions with the Fourier inversion formula holds on B.
Based on the definition of Barron function, (Barron, 1993) states that a Barron function can be approximated by a neural network with one hidden layer. Given a fixed composition of L Barron functions, we derive the error bound of the approximation between such composite function and a neural network with l hidden layers. Note that l is not necessarily equal to L. The approximation bound of w.r.t. Wasserstein distance can be provided as follows.
Theorem 4 (Wasserstein distribution approximation) Given a data distribution µ0 and a function ϕi: Rmi−1→Rmi , and letLl= log(l)+1, if support(µ0)⊂K0 andϕi(Ki−1)⊆Ki, 1≤i≤l, the function ϕi is ( Ll−1−1
Ll ) -Lipschitz and is a Barron function, i.e., ϕ1∈ΩK0(C0) and ϕi∈ΩKi−1+sBmi−1 (Ci),
then there exists a network f with l hidden layers with ⌈ 4C2l mlL 2 l−1L 2 l / 2 ⌉ neurons on the i-th layer,
W2(µ̂l, µ) ≤ 2
L2l
( (2Cl √ ml +Dl) 2 δ
s2 + 1
) , l ≤ L. (4)
where Dl is the diameter of the set Kl, and , δ, s>0 are parameters.
Theorem 4 provides an error bound when using a neural network with l hidden layers to approximate a fixed composition of L Barron functions. Moreover, the upper bound decreases with the increasing of layers under certain conditions. Here, we can conclude that deep layers are not always better than shallow layer for some specific samples. Such samples are often classified correctly in the shallow layer rather than the deep layer, then the diameter Dl can be large for this set of samples. Hence, shallow layers can behave better than deep layers for these kinds of samples.
Under review as a conference paper at ICLR 2020
Figure 3: Wasserstein distance across different layers for different networks.
Under review as a conference paper at ICLR 2020
1 5 10 15 200
0.5
Re sN et (V OC ) layer0
1 5 10 15 20
layer3
1 5 10 15 20
layer7
1 5 10 15 20
layer9
1 5 10 20
target
1 5 10 15 200
0.5
VG G (V OC
) layer2
1 5 10 15 20
layer6
1 5 10 15 20
layer11
1 5 10 15 20
layer13
1 5 10 20
target
1 20 40 60 800
0.5
Re sN et (C OC O) layer0
1 20 40 60 80
layer3
1 20 40 60 80
layer7
1 20 40 60 80
layer9
1 20 40 60 80
target
1 20 40 60 80 # label
0
0.5
VG G (C OC
O) layer2
1 20 40 60 80 # label
layer6
1 20 40 60 80 # label
layer11
1 20 40 60 80 # label
layer13
1 20 40 60 80 # label
target
15
15
Figure 5: Distribution propagation across different layers of ResNet and VGG.
object labels per image. For the evaluation metrics, we report the classification accuracy, CF1, OF1 and mean average precision (mAP) in the experiments.
Implementation Details. All experiments are implemented by PyTorch on a single NVIDIA TITAN X GPU. Specifically, we use SGD optimizer with the learning rate as 0.001. We experiment with ResNet (?) and VGG (?) with Binary Cross Entropy (BCE). We train the model 100 epochs with the batch size of 128. Then, we train the internal classifiers (i.e., the network hl in Figure 1(a)) for 30 epochs.

heading: 6.1 INTER-LAYER BEHAVIOR EXPLORATION
text: In this section, we provide theoretical and experimental analysis to study the behaviors of distributions in different layers. Next, we provide a new method to improve the performance of DNNs.
How every layer express the distribution? In this experiment, we study the ability of every layer to express the target distribution for ResNet and VGG. In this paper, the expression ability can be measured by Wasserstein distance (W-distance) between the label distribution of every layer and the target distribution. W-distance of different layers are shown in Figure 4. As suggested in Theorem 4, deep layers have smaller error bound than shallow layers, meaning that they have sufficient expression ability. This experiment verifies this conclusion by observing that the deeper layers have largerW-distance than shallow layers. Moreover,W-distance tends to decrease from layer to layer. The front most layers have a little different ability to express distribution since they have similar value ofW-distance. When approaching to the last layer,W-distance drops to a very small value. It implies that sufficient layers is able to express the target distribution.
How the distribution propagate across layers? In this experiment, we investigate how distributions propagate across different layers, which help to understand the learning process from layer to layer in a deep neural network. Note that we normalize the prediction of layer as the probability distribution (that is the sum is 1). Figure 5 shows that the label distribution of one sample propagates from the first layer to the final layer. At the early learning stage, the label distribution is far away from the target distribution, then it can be close to the target distribution at the latter learning stage. Different from the decreasing tendency in Figure 4, the label distribution of one sample may not close to the target distribution progressively. For examples, on the first line of Figure 5, the probability of the 15-th label is large in the shallow layer, but goes down in the middle layer.
Figure 4: Distribution propagation across different layers for different networks.

heading: 6 EXPERIMENT
text: Implementation Details. All experiments are implemented on PyTorch. In the training, we use SGD optimizer with the initial learning rate as 0.01. The learning rate decays by a factor of 10 for every 40 epochs. The momentum and weight decay is set to be 0.9 and 10−4, respectively. We obtain models (including ResNet-18, ResNet-50 (He et al., 2016) and VGG-16 (Simonyan & Zisserman, 2014)) pre-trained on ImageNet(Deng et al., 2009) and train them for 100 epochs with the batch size of 128. Then, we freeze the model weights and train the internal classifiers (i.e., the network hl in Figure 1(a)) for 100 epochs. Each internal classifier contains fully connected layer and Sigmoid function. We train the network using Binary Cross Entropy. We use training set to explore the distribution propagation of the across- and single-layer. In addition, we set α=0.01 in Eqn. 17 to achieve the balanced results. For simplicity, we use ResNet to represent the ResNet-18 model and VGG to represent the VGG-16 model in the main paper.
Data Sets and Evaluation Metrics. We conduct experiments on two benchmark multi-label classification datasets, i.e., VOC2007 (Everingham et al., 2010) and MS-COCO (Maas et al., 2013). VOC2007 (Everingham et al., 2010) contains 9,963 images from 20 object categories, which is divided into train, val and test sets. MS-COCO (Maas et al., 2013) contains 82,783 images as the training set and 40,504 images as the validation set. The objects are categorized into 80 classes with about 2.9 object labels per image. We use the classification accuracy, average per-class F1 (CF1), average overall F1 (OF1) and mean average precision (mAP) as the evaluation metrics.

heading: 6.1 ACROSS-LAYER BEHAVIOR EXPLORATION

heading: 6.1.1 THE BEHAVIORS OF LABEL DISTRIBUTIONS IN DIFFERENT LAYERS
text: does every layer ex ress the di tribution? Here, we study the expression ability of every layer, which is measure by Wasserstein distanc (W-distance) be ween the label distribution of any a d th target distribution. As suggested in Theorem 4, de p layers have smaller err r bound than shallow layer , meaning that they have better expression ability. From Figure 3, deep layers h ve smaller theW-distance than shallow layers, which verifies the conclusion in Theorem 4. Mo eover, theW-distance ec eases from shallow layers to deep layers. The shall w layers have similar ability to express the target distribution since they have similar values of theW-dista ce. Wh n approaching l st layer, theW-distance drops to a very small value. It impli s that s fficient layers are able to express th target distribution. More r sults are shown in Section F in Supplementary materials.
Under review as a conference paper at ICLR 2020
How does the distribution propagate across layers? In this experiment, we investigate how the label distribution propagate across layers to understand the learning process from layer to layer in a DNN. Note that we normalize the prediction of classifier as the probability distribution (i.e., the sum of label prediction probabilities of one sample is 1.). Figure 4 shows that the label distribution of one sample propagates from the first layer to the last layer. In the shallow layers, the label distribution is far away from the target distribution, then it can be close to the target distribution in the deep layers. Different from the decreasing tendency in Figure 3, the label distribution of one sample may not close to the target distribution progressively. For example, in the first row of Figure 4, the probability of the 10-th class is large in the 0-th layer, but goes down in the 3-th layer.

heading: 6.1.2 THE EVALUATION OF ACROSS-LAYER’S BEHAVIORS
text: How to evaluate the quality of each layer? Existing methods attempt to visualize the feature maps to evaluate the quality of each layer. However, observing the feature maps is difficult because of its high dimensionality and complexity. Relying on theW-distance, we can evaluate the quality of each layer by measuring the ability of expression. If theW-distance decreases rapidly in a layer, it suggests that the quality of this layer is good. Taking ResNet on VOC in Figure 3 as an example, 0∼5 layers have similar value of theW-distance, while 6∼8 layers have quite different value of the W-distance. It means that these deep layers (i.e., 6∼8) are better than the shallow layers (i.e., 0∼5).

heading: 6.1.3 HOW TO EXPLOIT THE ACROSS-LAYER BEHAVIORS
text: Does every sample contribute equally? In practice, a deep neural network construct more complex features progressively throughout the layers (Lee et al., 2011). An interesting question arises: does the contribution of each sample vary across different layers and training iterations? To answer this question, we first define the concept of sample difficulty in terms of theW-distance. As shown in Figure 5, the samples can be divided into three categories, including easy, hard and confused. The intuition is that: 1) Easy samples should have small theW-distance in the first few layers, i.e., , they should be classified correctly with high confidence in a shallow layer. 2) Hard samples would have large theW-distance in a deeper layer, i.e., , it cannot be resolved at all, or only near the last layer. 3) Confused samples have small theW-distance in a shallow layer, while have large theW-distance in a deep layer. It means that although these samples can be classified correctly in the shallow layer, they still become misclassified at the last layer. Considering the difficulty of samples would help to interpret the training of models and design the training loss, which may speed up convergence and improve the performance eventually. We show more results in Section I in Supplementary materials.
Under review as a conference paper at ICLR 2020Under review as a conference paper at ICLR 2020
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.2
1.0
1.8
W -d
ist an ce Re sN et (V OC
)
layer 1 layer 4 layer 9
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.2
1.0
1.8
W -d
ist an ce VG G (V OC )
layer1 layer 6 layer 13
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.4
0.8 1.2 W -d ist an ce
Re sN
et (C
OC O)
layer 1 layer 4 layer 9
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.4
0.8
1.2
W -d
ist an ce VG G (C OC O) layer1 layer 6 layer 13
Figure 7: Wasserstein distance across different training iterations for ResNet and VGG on VOC and COCO datasets. We choose the 1, 4, 9-th layer for ResNet and the 1, 6, 13-layer for VGG.
networks often occur the over-thinking issue. In Table ???, out of *% and *% of the samples for ResNet and VGG misclassifies, respectively. For these samples, there are *% samples are correctly classified in the *-th layer of ResNet, and *% samples are correctly classified in the *-th layer of VGG. Using this phenomenon, we try to improve the performance of classification by early existing such confused samples. Different from the strategy of SDN (?), how to early exist in a DNN for a multi-label task is more challenging. Therefore, we design a new early-exist strategy for multi-label predictions. In Table 1, our proposed method consistently outperforms the other methods.
6.2 INNER-LAYER BEHAVIOR EXPLORATION
In this section, we study how the distributions evolute in one layer for different training iterations. We consider to use Wasserstein distance to measure the stability of distribution when training the neural network. We propose theoretical and empirical analysis for the distribution stability in one layer.
How stable is the distribution in one layer? Recently, the stability of a deep neural network attracts extensive attention. Batch Normalization is proposed to stabilize the distributions of layer distribution. The popular belief is that success of Batch Normalization stems from controlling the stability of distribution during the training. However, we prove that the stability of distribution is hard to be guaranteed, the detail proofs are put in Supplementary. In this paper, we useW-distance to measure the discrepancy between two distributions in one layer across different training iterations. Motivated by this, we further measure stability of label distribution in different training iterations. The conventional understanding of Batch Normalization suggests thatW-distance between distributions in two successive iterations should decrease. In Figure 7, the first layer of ResNet or VGG is hard to decreaseW-distance, indicating that the label distributions alway change in different iterations.
How the distribution propagate across iterations? In this experiment, we investigate how distributions propagate across different training iterations. The label distribution propagation is shown in Figure 5. The label distribution in the first layers of ResNet or VGG often has great fluctuation in different training iterations because the supervision is insufficient when the neural network is deep. When approaching the final layer, the supervision is sufficient to decreaseW-distance. These experiments justify Theorem 2 thatW-distance can be decreased with sufficient supervision.
Why add auxiliary loss in some layer? When the neural network is very deep, shallow layers tend to receive insufficient supervision. Recently, some methods are proposed to introduce auxiliary loss to improve the performance of DNNs (?). However, why introduce loss in some layers and what an important role does it play in the training are not clear. From Theorem 2, when adding the supervision in one intermediate layer,W-distance can be decreased across the training iterations.
7 CONCLUSION
In this paper, we have interpreted deep neural networks (DNNs) via understanding layer behaviors. With the help of optimal transport theory, we propose a new analytical framework using Wasserstein
8
Figure 6: Wasserstein distance between the distribution in an epoch and the target distribution across different training epochs for different networks. We choose the 1, 4, 9-th layer for ResNet and the 1, 6, 13-layer for VGG.
How to improve classification performance? We will discuss how to exploit the behaviors of different layers to improve the model performance. In practice, we observe that in deep neural networks, some samples are correctly classified in the intermediate layers but misclassified in the last layer. For example, 35.52% and 31.04% of the samples for ResNet and VGG are misclassified on the test set of VOC2007, respectively. Fo these samples, 5.96% and 7.76% samples are correctly classifi in all intermediate layers of ResNet and VGG, respectively. Based on this phenomenon, we can improve the performance by early exiting such confused samples. Different from the strategy of SDN (Kaya et al., 2019), how to early exit for multi-label classification DNNs is very challenging. Therefore, we design a new early-exits strategy for multi-label prediction. In Table 1, our proposed method consistently outperforms the baseline methods. We give more details about early-exits strategy and show more results in Section H in Supplementary materials.

heading: 6.2 SINGLE-LAYER BEHAVIOR EXPLORATION
text: In this section, we study how the distribution of an intermediate layer change during the training process. We consider to use W-dista ce to measure the stability of distribution. We propose theoretical and empirical analysis for the distribution stability in one layer.
How the distribution propagate during updating DNNs? In this experiment, we investigate how distributions propagate when training D Ns. From Figure 6, the label distribution in the first layers of ResNet or VGG oft n fluctuates signifi antly due to the limited discriminative p wer of very shallow layers. Wh n approa hing the las layer, th supervision is sufficient to ecr ase theW-distance. These experiments justify Theorem 2 that theW-distance can be decreased with sufficient supervision. More results are shown in Section G in Supplementary materials.
How stable is the distribution in one layer? The training stability is a key problem in DNNs and can be largely addressed by the widely used Batch Normalization (BN) (Ioffe & Szegedy, 2015). Howev r, we prove th t the stability of distribution i hard to be guaranteed even with BN (See proof in suppl mentary). In t is paper, we use theW-distance to measure the discrepancy between the distribution in an iteration and the target distribution across different training iterations in the specific layer. Motivated by this, we further measure stability of label distribution during training DNNs. The conve tional understanding of BN uggests that theW-distance sh uld ecrease. Note that, from Figure 6, the first layer of ResNet or VGG may fluctuate during the training process, which indicates that deep layers often have better ability than shallow layers to express the target distribution.

heading: 7 CONCLUSION
text: In this paper, we have interpreted deep neural net orks (DNNs) via understanding layer behaviors. With the help of ptimal transport theor , ropose a unified teacher-student a alysi f amework to investigate the across-layer and single-layer behaviors of a DNN. Theoretically, we prove that i) theW-distance between the distribution of any layer and the target distribution decreases along the depth. ii) theW-distance of a specific layer between the distribution in an iteration and the target distribution decreases along training iterations. Extensive experiments justify our theoretical findings. The proposed analytical framework can facilitate future researches to interpret a deep neural network.
Under review as a conference paper at ICLR 2020

heading: A PRELIMINARY
text: Notation. Specifically, we use bold lower-case letters (e.g., x) to denote vectors, and bold uppercase letters (e.g.,X) to denote matrices. We denote the transpose of a vector (e.g., xT) or matrix (e.g.,XT) by the superscript T. Let 1 := [1, . . . , 1]T . For two matrices A and B of the same size, their inner-product is 〈A,B〉=tr(ATB). Let ∗ be the convolution operator. Let ‖ · ‖=‖ · ‖2 denote Euclidean norm on vectors in Rn. For a function f , let f∨:=f(−x). Let ∆f=∑ni=1 ∂2f∂2xi denote the Laplacian. Let Bn be the unit ball in n dimension, i.e., Bn={x∈Rn : ‖x‖≤1}. For two sets A and B and a scalar r ∈ R, let A+B={x+y : x∈A,y∈B} and rA={rx : x∈A}.
Definition 4 (Lipschitz continuous) A function ϕ: Rm→Rn is called L-Lipschitz continuous if there exists a real constant L≥0 such that, for all x and y w.r.t. the L2 norm,
‖ϕ(x)− ϕ(y)‖2 ≤ L‖x− y‖2.
Definition 5 (Fourier transform) For ϕ∈L1(R), Fourier transform of ϕ: Rn→R is defined as:
ϕ̂(w) := 1
(2π) n 2 ∫ Rn ϕ(x)e−i〈w,x〉dx. (5)
For vector-valued functions ϕ: Rn→Rm, we conduct component-wise Fourier transform. Let ϕ̂∨(x) := ϕ̂(−x), the inverse Fourier transform is
(F−1ϕ)(x) := ∫ Rn ϕ(w)ei〈w,x〉dx = (2π) n 2 ϕ̂∨. (6)
Definition 6 For a bounded set B, the norm of a function f : Rn→R can be defined on a set B, ‖f‖∗B:= ∫ Rn ‖w‖B|f̂(w)|dw, where ‖w‖B= supx∈B |〈w,x〉|.

heading: B PROOF OF THEOREM 4
text: Definition 7 (Barron function (Sonoda & Murata, 2019)) The function ϕ is Barron function on B if ϕ satisfies ΩB(C)={ϕ: B→R: ∃φ, φ|B=ϕ, ‖φ‖∗B≤C, φ∈FB}, where ‖φ‖∗B:= ∫ Rm ‖w‖B|φ̂(w)|dw, where ‖w‖B= supx∈B |〈w,x〉| and φ̂ is the Fourier transform of φ, and FB is the set of functions with the Fourier inversion formula holds on B,i.e.,
FB = { φ : Rm → R : ∀x ∈ B, φ(x) = φ(0) + ∫ ( ei〈w,x〉 − 1 ) φ̂(w)dw } . (7)
Definition 8 (Sigmoidal function) A sigmoidal function is a bounded measurable function σ: R→R such that limx→−∞ σ(x)=0 and limx→+∞ σ(x)=1.
Theorem 5 (Barron theorem (Barron, 1993)) Let B⊆Rd be a bounded set, and µ be any probability measure on B, ϕ∈ΩB(C) and f(x)= ∑n i=1 ciσ(〈wi,x〉 + bi), where σ(·) be a sigmoidal
function, then there exist wi∈Rd, bi∈R, ci∈R with ∑n i=1 |ci|≤2C such that
‖ϕ(x)− f(x)‖2µ := ∫ B (ϕ(x)− f(x))2µ(dx) ≤ (2C) 2 n . (8)
We extend the above theorem to the case of the composition of Barron functions denoted as ϕj:i := ϕj ◦ ϕj−1 ◦ · · · ◦ ϕi.
Corollary 1 Let B be a bounded set, µ be any probability measure and the neural network f(x)= ∑n i=1 ciσ(〈wi,x〉+bi), where σ(·) is a sigmoidal function. If the composition of Barron
Under review as a conference paper at ICLR 2020
functions is restricted in the set of ΩB(C), then there existswi∈Rd, bi∈R, ci∈R with ∑n i=1 |ci|≤2C such that
‖ϕk:1(x)− f(x)‖2µ := ∫ B (ϕk:1(x)− f(x))2 µ(dx) ≤ (2C)2 n . (9)
Proof Directly using Lemma 1 and Theorem 2 of (Barron, 1993) can complete the conclusion.
Theorem 6 (Extend Barron theorem) Let B be a bounded set, µ be any probability measure and f(x)= ∑n i=1 ciσ(〈wi,x〉+bi), where σ(·) be a sigmoidal function. If support(µ0)⊂K0 and ϕi(Ki−1)⊆Ki, 1≤i≤l, the function ϕi is log(l−1)log(l)+1 -Lipschitz and Barron function, i.e., ϕ1∈ΩK0(C0) and ϕi∈ΩKi−1+sBmi−1 (Ci), then there exists a neural network f with l hidden layers and S ⊂ Rd0 satisfying µ0(S)≤1− 2
s2(log(l−1)+1)2 so that(∫ 1S‖ϕ− f‖2dµ0 ) 1 2
≤ log(l)+1 . (10)
Proof For simplicity, let ϕ:=ϕl:1=ϕl◦ · · · ◦ϕ1 be the composition of Barron functions, especially ϕ1=ϕ̃k:1, where ϕ̃l is defined in Corollary 1. Let f :=fl:1=fl◦fl−1◦ · · · ◦ f1 be the composition of k-layered neural network, where fi : Rdi−1→Rdi−1 are functions defined as
(fi(x))j = ri∑ k=1 cijk · σ(〈wijk,x〉+ bijk), (11)
where cijk, bijk∈R and wijk∈Rdi−1 are parameters, and ri is the number of nodes in the i-th layer. Note that the function fi is represented by a neural network with one hidden layer and a linear output layer. We prove the theorem by induction on l.
i) For l=1, using assumptions of the support of initial distribution (i.e., support(µ0)⊂K0) and the definition of Barron function. Then, =(2C1)2/r1 and ∑r1 i=1 |ci| ≤ 2C1, we complete the results.
ii) For l>1, by the induction step, we assume the functions f1, . . . , fl−1 satisfy the conclusion for ϕ̃1, . . . , ϕ̃l−1. Let Sl−1 be the set in the conclusion. Applying Corollary 1 to fl to get that for each 1 ≤ j ≤ mj , for any µ supported on a set K ′l−1 ⊆ Rml−1 and any rl ∈ N, there exists a neural net fl,j with 1 hidden layer with rl nodes such that(∫
Rml−1 (ϕl,j − fl,j)2dµ
) 1 2
≤ 2Cfl,K′l−1√
rl .
Note that Theorem 1 applies to any distribution µ supported on the set K′l−1. Let Sl = Sl−1 ∩ {x : fl−1:1(x)∈Kl−1+sBml−1}. Applying Theorem 1 with K′l=Kl+sBml , rl= ⌈ 4C2l ml(log(l−1)+1) 2(log(l)+1)2 2 ⌉ and µ=fl−1:1#(1Slµ0). We have that µ is supported on gl−1:1(Sl) ⊆ Kl−1+sBml−1=K′l−1, and the function ϕl is Cl-Barron function on this set by the assumption. The conclusion of Theorem 1 gives (fl)j such that(∫
Rml−1 (ϕl,j − fl,j)2d(fl−1:1#(1Slµ0))
) 1 2
≤ 2Cl√ rl ≤ √ ml(log(l−1)+1)(log(l)+1) .
For all elements of ϕl and fl, we have(∫ Rml−1 ‖ϕl − fl‖2d(fl−1:1#(1Slµ0)) ) 1 2 ≤ (log(l−1)+1)(log(l)+1) .
Under review as a conference paper at ICLR 2020
We bound by the triangle inequality(∫ Rm 1Sl‖ϕl:1 − fl:1‖2dµ0 ) 1 2
≤ (∫
Rm 1Sl‖ϕl ◦ ϕl−1:1 − ϕl ◦ fl−1:1‖2dµ0
) 1 2
+ (∫ Rm 1Sl‖ϕl ◦ fl−1:1 − fl ◦ fl−1:1‖2dµ0 ) 1 2
≤ (∫
Rm 1Sl‖ϕl ◦ ϕl−1:1 − ϕl ◦ fl−1:1‖2dµ0
) 1 2
+ (∫ Rm 1Sl‖ϕl − fl‖2dfl−1:1#(1Slµ0) ) 1 2
≤Lip(ϕl) (∫
Rm 1Sl‖ϕl−1:1 − fl−1:1‖2dµ0
) 1 2
+
(log(l−1)+1)(log(l)+1) ≤Lip(ϕl) (∫
Rm 1Sl−1‖ϕl−1:1 − fl−1:1‖2dµ0
) 1 2
+
(log(l−1)+1)(log(l)+1)
≤ log(l−1) log(l)+1 · log(l−1)+1 +
(log(l−1)+1)(log(l)+1) ≤
log(l)+1 ,
where the last inequality holds by the assumption (i.e., Lip(ϕl)= log(l−1) log(l)+1 ) and the induction hypothesis.
As above analysis, we have∫ 1Sl−1‖ϕl−1:1 − fl−1:1dµ0‖2dµ0 ≤
2
(log(l−1)+1)2 ,
by the induction hypothesis. Also, ϕl−1:1 ∈ Kl−1 for all x ∈ support(µ0) by the assumption of the theorem. Therefore, by Markov’s inequality and the induction hypothesis on Sl−1,
µ0 ( Sl−1 ∩ { x : x /∈ Kl−1 + sfl−1:1(Bml−1) }) ≤µ0 (Sl−1 ∩ {x : ‖ϕl−1:1 − fl−1:1‖ ≥ s}) ≤
2
s2(log(l−1)+1)2 .
Therefore,
µ0(Sl) ≤ µ0(Sl−1)− 2
s2(log(l−1)+1)2 ,
the last equality holds by a fact that there exists a constant δ to satisfy µ0(Sl) = 1− δ 2
s2(log(l−1)+1)2 .
Based on the above conclusions, we prove Theorem 4 as follow.
Theorem 4 Given a data distribution µ0 and a function ϕi: Rmi−1→Rmi , and let Ll= log(l)+1, if support(µ0)⊂K0 and ϕi(Ki−1)⊆Ki, 1≤i≤l, the function ϕi is
( Ll−1−1
Ll
) -Lipschitz and is a Barron
function, i.e., ϕ1∈ΩK0(C0) and ϕi∈ΩKi−1+sBmi−1 (Ci), then there exists a network f with l hidden layers with ⌈ 4C2l mlL 2 l−1L 2 l / 2 ⌉ neurons on the i-th layer,
W2(µ̂l, µ) ≤ 2
L2l
( (2Cl √ ml +Dl) 2 δ
s2 + 1
) , l ≤ L. (12)
where Dl is the diameter of the set Kl, and , δ, s>0 are parameters. Proof The functions f1, · · · , fl in Theorem 6 satisfy that∫
1Sl−1‖ϕl−1:1 − fl−1:1dµ0‖2dµ0 ≤ 2
(log(l−1)+1)2 .
The range of fl = ((fl)1, · · · , (fl)ml) is contained in a set of diameter 2Cl √ ml because the activation in a neural network is ranged in [0, 1] and the weights cijk in Theorem 6 satisfy ∑r k=1 |cijk| ≤ 2Cl.
Under review as a conference paper at ICLR 2020
Choose a constant vector k to minimize ∫ Sl ‖ϕl:1(x)−fl:1−k‖ 2dµ0 and replace fl with fl+k. Note
that the range of fl and ϕl necessarily overlap. We still have ∫ Sl ‖ϕl:1(x)− fl:1‖ 2dµ0 ≤ 2
(log(l)+1)2 . Moreover, we have
‖ϕl(x)− fl(x)‖ ≤ 2Cl √ ml +Dl,
for x ∈ K0, where Dl is the diameter of Kl. Let Scl be a complementary of Sl, we have
µ(Scl ) = δ 2
s2(log(l−1)+1)2 ,
then we have∫ K0 ‖ϕl:1 − fl:1‖2dµ0 ≤ ∫ Sl ‖ϕl:1 − fl:1‖2dµ0 + ∫ Scl ‖ϕl:1 − fl:1‖2dµ0
≤ 2
(log(l) + 1)2 + (2Cl
√ ml +Dl) 2 δ 2
s2(log(l−1)+1)2
= 2
(log(l) + 1)2
( (2Cl √ ml +Dl) 2 δ
s2 + 1
) .
For (ϕl:1(X), fl:1(X)), X ∼ µ0 define a coupling between the distributions. Based on the definition ofW-distance, we have
W2(µ̂k, µ) =EX∼µ0‖ϕl:1 − fl:1‖2
= ∫ K0 ‖ϕl:1 − fl:1‖2dµ0 ≤ 2
(log(l) + 1)2
( (2Cl √ ml +Dl) 2 δ
s2 + 1
) .

heading: C PROOFS OF LAYER BEHAVIOR ANALYSIS OF DEEP NEURAL NETWORKS
text: Definition 9 (Flow) A flowϕt is given by an ordinary differential equation (ODE), ϕ̇t(x)=vt(ϕt(x)) with a velocity field vt when t>0, and ϕ0(x)=x when t=0.
Proposition 2 (Continuity equation (Sonoda & Murata, 2019)) Let ϕt be the flow of an ODE with vector field vt, then the distribution µt evolves according to the continuity equation ∂tµt(x)=−∇·[µt(x)vt(x)], x∈Rm, t≥0, where∇· denotes the divergence operator.
Theorem 7 (Optimal transport map) The global minimum f∗t of L(f) is attained at
f∗t (x̃) = x̃− 1
νt ∗ µ0(x̃) ∫ Rm ενt(ε)µ0(x̃− ε)dε,
where ∗ denotes the convolution operator.
Proof The proof follows from the calculus of variations. L(f) = ∫ Rm Eε [ ‖f(x+ ε)− f ′(x)‖2 ] µ0(x)dx
= ∫ Rm Eε [ ‖f(x′)− f ′(x′ − ε)‖2 µ0(x′ − ε) ] dx′,
where the second line holds by the calculus of variations, i.e., x′ = x + ε. Then, for an arbitrary function h, the first variation δL(h) is given by
δL(h) = d ds L(f + sh) ∣∣∣∣ s=0
= ∫ Rm ∂ ∂s Eε [ ‖f(x) + sh(x)− f ′(x− ε)‖2 µ0(x− ε) ] dx ∣∣∣∣ s=0
=2 ∫ Rm Eε [(f(x)− f ′(x− ε))µ0(x− ε)]h(x)dx.
Under review as a conference paper at ICLR 2020
At a critical point f∗ of L, δL(h) = 0 for every h. Hence,

heading: Eε [(f(x)− f ′(x− ε))µ0(x− ε)] = 0, a.e. x,
text: by the fundamental lemma of calculus of variations for integrable functions, and we have
f∗(x) = Eε [f ′(x− ε)µ0(x− ε)]
Eε [µ0(x− ε)] .
Note that f∗ attains the global minimum, because, for every function h, L(f∗ + h) = ∫ Rm Eε [ ‖f∗(x)− f ′(x− ε) + h(x)‖2 µ0(x− ε) ] dx
= ∫ Rm Eε [ ‖f∗(x)− f ′(x− ε)‖2 µ0(x− ε) ] dx+ ∫ Rm Eε [ ‖h(x)‖2 µ0(x− ε) ] dx
+ 2 ∫ Rm Eε [ h(x)T (f∗(x)− f ′(x− ε))µ0(x− ε) ] dx
=L(f∗) + L(h) + 2 · 0 ≥ L(f∗).
Theorem 1 If the target network is a residual unit f ′(x̃)=x+σ(x̃), where σ contains fully connected layer and activation function. When νt := N (0, tI), then the global minimum f∗t is
f∗t (x̃) =x̃+ t∇ log(νt ∗ µ0)(x̃) + σ(x̃) := x̃+ gt(x̃).
Proof Using Stein’s identity (Liu, 1994), −t∇νt(ε) = ενt(ε), which is known to hold only for Gaussians.
f∗t (x̃) =x̃− 1
νt ∗ µ0(x̃) ∫ Rm ενt(ε)µ0(x̃− ε)dε+ σ(x̃)
=x̃+ 1
νt ∗ µ0(x̃) ∫ Rm t∇νt(ε)µ0(x̃− ε)dε+ σ(x̃)
=x̃+ t∇νt ∗ µ0(x̃) νt ∗ µ0(x̃) + σ(x̃) =x̃+ t∇ log(νt ∗ µ0(x̃)) + σ(x̃).
Theorem 2 Based on the equivalent condition in Theorem 2 (Belavkin, 2016), and let ∆ be the Laplacian operator. At the initial moment t→0, the pushforward f#µt with Gaussian distribution satisfies the backward heat equation (BHE), and evolves according to Wasserstein gradient flow: ∂tµt=0(x)=−∆µ0(x)=−gradW2(µ0, µ),x∈Rm. Proof The initial velocity vector is given by
∂tf ∗ t=0(x) = lim
t→0 f∗t (x)− x t = ∇ logµ0(x) + σ(x). (13)
Hence, by substituting the score (13) in the continuity equation of Proposition 2, we have
∂tµt=0(x) = −∇·[µ0(x)(∇ logµ0(x) + σ(x))] = −∆µ0(x). We leave the proof of ∆µ0(x) =W2(µt, µ) in Theorem 3.
Theorem 3 Based on the equivalent condition in Theorem 2 (Belavkin, 2016), and when the noise distribution is a Gaussian distribution, then the pushforward measure µt:=ϕt#µ0 evolves according to Wasserstein gradient flow as follows:
d dt µt(x)=−∆µt(x) = −gradW2(µt, µ), µt=0(x)=µ0(x). (14)
Under review as a conference paper at ICLR 2020
Proof Let the potential function be equal to Kullback–Leibler divergence be KL(µt, µ) =∫ µt(x) log µt(x) µ(x) − µt(x) + µ(x)dx, then Vt = −(log(µt)− log(µ)), then
gradF (µt) = ∇·(µt∇(log(µt)− log(µ))) = ∆µt. The above continuity also satisfies the case of Wasserstein distance, then using the equivalent condition in Theorem 2 (Belavkin, 2016), we have
∂tµt=0(x)=−gradW2(µt, µ),x∈Rm.
We assume the distribution µ(t)l is Gaussian distribution. TheW-distance of two Gaussian distribution is defined as follow.
Proposition 3 (Gelbrich, 1990) Given two Gaussian distributions µ1∼N (u1,Σ1) and µ2∼N (u2,Σ2), then Wasserstein distance between µ1 and µ2 is
W2(µ1, µ2)=‖u1 − u2‖2 + tr ( Σ1+Σ2 − 2 ( Σ 1 2 1 Σ2Σ 1 2 1 ) 1 2 ) .
Batch Normalization is proposed to stabilize the distributions of layer distribution by introducing the parameters γt and βt in the t-th optimization step. Let Γ(t)l =diag(γ (t) l ) and Γ (t+1) l =diag(γ (t+1) l ).
Proposition 4 Assume the distributions in two iterations in the l-th layer are Gaussian distributions, i.e., µ(t)l ∼N (β (t) l ,Γ (t) l ) and µ (t+1) l ∼N (β (t+1) l ,Γ (t+1) l ), then theW-distance between µ(t)l and µ(t+1)l isW2(µ(t)l , µ (t+1) l )=‖β (t) l −β (t+1) l ‖ 2+‖(Γ(t)l −Γ (t+1) l ) 2‖2.
Proof Directly from 3, and let u1=β (t) l ,u2=β (t+1) l and Σ1=Γ (t) l ,Σ2=Γ (t+1) l , we have
W2(µ(t)l , µ (t+1) l )=‖β (t) l −β (t+1) l ‖2+‖(Γ (t) l −Γ (t+1) l ) 2‖2.
Under review as a conference paper at ICLR 2020

heading: D OPTIMAL TRANSPORT
text: Definition 10 (Optimal Transport (Villani, 2008)) Given a cost function c: K×K→R, the optimal transport distance measures the optimal plan to transport the mass from a probability measure µ̂ to another probability measure µ:
W̃2(µ̂, µ) = inf π∈Π(µ̂,µ) ∫∫ K×K c(κ̂,κ)π(dκ̂, dκ), (15)
where Π(µ̂, µ) is the set of joint probability measures π on K×K with the marginals µ̂ and µ.
In this paper, we consider the multi-label classification problem. We first introduce the definition of Wasserstein distance. For the case of probability measures, they are histograms in the simplex ∆K .
Definition 11 (W-distance) Given an input x∈X and any f : X→∆K , let f(x) be the predicted label distribution µ̂, and let y be the target distribution µ, then theW-distance between µ̂ and µ is
W2(µ̂, µ) = inf T∈Π(µ̂,µ) 〈T ,C〉, (16)
where C∈RK×K+ is the cost matrix whose the element can be defined as Cκ,κ′=dpK(κ, κ′), and the set of couplings is composed of joint probability distributions with their marginals µ̂ and µ, i.e., Π(µ̂, µ)={T∈RK×K+ : T1=µ̂,T T1=µ}.
Entropic Wasserstein loss. Cuturi (2013) introduces an entropic regularization such that the nonconvex problem (2) can be turned to a strictly convex problem:
Ŵ2α(µ̂, µ) := inf T∈Π(µ̂,µ) 〈T ,C〉 − 1 α H(T ), (17)
where H(T )=−∑κ,κ′ Tκ,κ′(log(Tκ,κ′)−1). With the help of the entropic regularization, we solve optimal transport problem efficiently using Sinkhorn-Knopp matrix scaling algorithm (Knight, 2008).
Proposition 5 The transport matrix T ∗ optimizing Problem (17) satisfies T ∗ = diag(u)Kdiag(v), whereK = e−αM , u = eαa and v = eαb, where a and b are the Lagrange dual variables.
Theorem 8 (Genevay et al., 2017)Wasserstein sinkhorn loss between the predicted label distribution µ̂ and the target label distribution µ is defined as:
W̄2(µ̂, µ) := 2Ŵ2α(µ̂, µ)− Ŵ2α(µ̂, µ̂)− Ŵ2α(µ, µ), (18)
with the following limiting behavior as α→ 0: W̄2(µ̂, µ)→ 2Ŵ2α(µ̂, µ).
Note that normalized Wasserstein loss is non-negative and W̄2α(µ̂, µ) = 0 if and only is µ̂ = µ. In the quantification, we use normalized Wasserstein loss to measure the divergence between distributions.
Under review as a conference paper at ICLR 2020
E INTERMEDIATE LAYER
We add auxiliary classifiers for the intermediate layers as mentioned below. For ResNet-18, we truncate each block and add a classifier after its feature map. We label these intermediate layers from 0-th layer to 8-th layer. The truncation of ResNet-50 is analogous to ResNet-18. For VGG-16, we add a classifier after each convolution layer and label them from 0-th layer to 12-th layer.

heading: F ACROSS-LAYER DISTRIBUTION PROPAGATION
text: We show more results for different datasets, measures and networks across different layers.
Under review as a conference paper at ICLR 2020
Under review as a conference paper at ICLR 2020 Figure 8: Definition of intermediate layer of ResNet-18.
Figure 9: Definition of intermediate layer of VGG-16.
Figure 10: Wasserstein distance across different layers of ResNet-18 (left) and VGG-16 (right) on test set.
Figure 11: Chebyshev distance across different layers of ResNet-18 (left) and VGG-16 (right) on training set.
Table 2: Improve performance of ResNet-50.
Method VOC COCOaccuracy CF1 OF1 mAP accuracy CF1 OF1 mAP ResNet 69.79 60.93 60.23 88.84 33.81 61.25 64.01 71.77 ResNet+DSN 70.40 61.29 60.39 89.04 34.19 61.33 64.27 72.41 ResNet+EarlyExits 70.98 62.84 60.21 89.14 35.70 62.39 64.76 73.68
16
The tendency of distribution propagation on test set is the same as the training set(Figure 3), suggesting that the distribution propagation is irrelevant to the property of the dataset.
Under review as a conference paper at ICLR 2020 Figure 8: Definition of intermediate layer of ResNet-18.
Figure 9: Definition of intermediate layer of VGG-16.
Table 2: Improve performance of ResNet-50.
Method VOC COCOaccuracy CF1 OF1 mAP accuracy CF1 OF1 mAP ResNet 69.79 60.93 60.23 88.84 33.81 61.25 64.01 71.77 ResNet+DSN 70.40 61.29 60.39 89.04 34.19 61.33 64.27 72.41 ResNet+EarlyExits 70.98 62.84 60.21 89.14 35.70 62.39 64.76 73.68
16
t. Although the tendency of Chebyshev distance is analogous to Wasserstein distance, Chebyshev distance ignores any metric st uctur (Frogner et al., 2015). Therefore, we use Wasserstein distance to quantify the discrepancy of label distributions.
Under review as a conference paper at ICLR 2020
0 1 2 3 4 5 6 7 8 9 layer (ResNet)
0.00
0.25
0.50
0.75
1.00
JS -d
iv er
ge nc
e VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 layer (VGG)
0.00
0.25
0.50
0.75
1.00 VOC COCO
Figure 12: Jensen–Shannon divergence across different layers of ResNet-18 (left) and VGG-16 (right) on training set.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 layer (ResNet)
0.0
0.5
1.0
1.5
W -d
ist an
ce
VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 layer (ResNet)
0.0
0.5
1.0
1.5 VOC COCO
Figure 13: Wasserstein distance across different layers of ResNet-50 on training set (left) and test set (right).
Figure 14: Chebyshev distance (left) and Jensen–Shannon divergence (right) across different layers of ResNet-50 on training set.
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce Re sN et (V OC
)
layer 1 layer 4 layer 9
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce VG G (V OC )
layer1 layer 6 layer 13
Figure 15: Wasserstein distance between distributions in adjacent training epoch.
1 5 10 15 200
0.5
Re sN et (V OC ) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 15 20
target
1 5 10 15 200
0.5
VG G (V OC
) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 20
target
1 20 40 60 800
0.5
Re sN et (C OC O) epoch=1
1 20 40 60 80
epoch=30
1 20 40 60 80
epoch=70
1 20 40 60 80
epoch=100
1 20 40 60 80
target
1 20 40 60 80 # label
0
0.5
VG G (C OC
O) epoch=1
1 20 40 60 80 # label
epoch=30
1 20 40 60 80 # label
epoch=70
1 20 40 60 80 # label
epoch=100
1 20 40 60 80 # label
target
15
Figure 16: Distribution propagation across different training epochs of ResNet-18 and VGG-16.
Table 3: Over-thinking results of ResNet-18.
layer 0 1 2 3 4 5 6 7 8 9 ideal result VOC2007 1.17% 0.44% 0.69% 0.71% 0.55% 0.50% 0.53% 0.85% 0.53% 64.48% 70.44% MS-COCO 0.49% 0.43% 0.47% 0.28% 0.30% 0.31% 0.37% 0.78% 0.50% 27.33% 31.26%
17
# #
# #
i r : i r r iff r t l r f t- (l ft) - (ri t) tr i i t. The conclusion of Jensen–Shannon divergence is the same as the Chebyshev distance.
nder revie as a conference paper at I L 2020
0 1 2 3 4 5 6 7 8 9 layer (ResNet)
0.00
0.25
0.50
0.75
1.00
JS -d
iv er
ge nc
e VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 layer (VGG)
0.00
0.25
0.50
0.75
1.00 VOC COCO
Figure 12: Jensen–Shannon divergence across different layers of es et-18 (left) and -16 (right) on training set.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 layer (ResNet)
0.0
0.5
1.0
1.5
W -d
ist an
ce
VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 layer (ResNet)
0.0
0.5
1.0
1.5 VOC COCO
Figure 13: asserstein distance across different layers of es et-50 on training set (left) and test set (right).
igure 14: hebyshev distance (left) and Jensen– hannon divergence (right) across different layers of es et-50 on training set.
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce Re sN et (V OC
)
layer 1 layer 4 layer 9
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce VG G (V OC )
layer1 layer 6 layer 13
igure 15: asserstein distance bet een distributions in adjacent training epoch.
1 5 10 15 200
0.5
Re sN et (V OC ) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 15 20
target
1 5 10 15 200
0.5
VG G (V OC
) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 20
target
1 20 40 60 800
0.5
Re sN et (C OC O) epoch=1
1 20 40 60 80
epoch=30
1 20 40 60 80
epoch=70
1 20 40 60 80
epoch=100
1 20 40 60 80
target
1 20 40 60 80 # label
0
0.5
VG G (C OC
O) epoch=1
1 20 40 60 80 # label
epoch=30
1 20 40 60 80 # label
epoch=70
1 20 40 60 80 # label
epoch=100
1 20 40 60 80 # label
target
15
i re : istri ti r a ati acr ss iffere t trai i e c s f es et- a - .
a le : er-t i i res lts f es et- .
layer 0 1 2 3 4 5 6 7 8 9 ideal result 2007 1.17 0.44 0.69 0.71 0.55 0.50 0.53 0.85 0.53 64.48 70.44
S- 0.49 0.43 0.47 0.28 0.30 0.31 0.37 0.78 0.50 27.33 31.26
# #
# #
We get the same conclusion on ResNet-50 that Wasserstein distance between the distribution
of any layer and the target distribution tends to decreases along the depth of a DNN.
Under review as a conference paper at ICLR 2020
0 1 2 3 4 5 6 7 8 9 layer (ResNet)
0.00
.25
.50
.75
1.00
JS -d
iv er
ge nc
e VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 layer (VGG)
0.00
25
50
75
1 00 VOC COCO
Figure 12: Jensen–Shannon divergence across different layers of ResNet-18 (left) and VGG-16 (right) on training set.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
layer (ResNet)
0.0
0.5
1.0
1.5
W -d
ist an
ce
VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
layer (ResNet)
0.0
0.5
1.0
1.5 VOC COCO

heading: Fi ure 13: Wasserstein distance across different layers of ResNet-50 on training set (left) and test set (right).
text: 1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce Re sN et (V OC
)
layer 1 layer 4 layer 9
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce VG G (V OC )
layer1 layer 6 layer 13
Figure 15: Wasserstein distance between distributions in adjacent training epoch.
1 5 10 15 200
0.5
Re sN et (V OC ) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 15 20
target
1 5 10 15 200
0.5
VG G (V OC
) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 20
target
1 20 40 60 800
0.5
Re sN et (C OC O) epoch=1
1 20 40 60 80
epoch=30
1 20 40 60 80
epoch=70
1 20 40 60 80
epoch=100
1 20 40 60 80
target
1 20 40 60 80 # label
0
0.5
VG G (C OC
O) epoch=1
1 20 40 60 80 # label
epoch=30
1 20 40 60 80 # label
epoch=70
1 20 40 60 80 # label
epoch=100
1 20 40 60 80 # label
target
15
Figure 16: Distribution propagation across different training epochs of ResNet-18 and VGG-16.
Table 3: Over-thinking results of ResNet-18.
layer 0 1 2 3 4 5 6 7 8 9 ideal result VOC2007 1.17% 0.44% 0.69% 0.71% 0.55% 0.50% 0.53% 0.85% 0.53% 64.48% 70.44% MS-COCO 0.49% 0.43% 0.47% 0.28% 0.30% 0.31% 0.37% 0.78% 0.50% 27.33% 31.26%
# #
# #
20
Under review as a conference paper at ICLR 2020

heading: G SINGLE-LAYER DISTRIBUTION PROPAGATION
text: We present more results about the single-layer distribution propagation when training DNNs. We calculate the Wasserstein distance between the prediction distribution of a selected epoch i and the next epoch i + 1 and show in Figure 15. Analogous to Figure 4, Figure 16 shows that the label distribution of one sample propagates from the first epoch to the last epoch.
Under review as a conference paper at ICLR 2020 0 1 2 3 4 5 6 7 8 9 layer (ResNet) 0.00 0.25 0.50 0.75 1.00 JS -d iv er ge nc e VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 layer (VGG) 0.00 0.25 0.50 0.75 1.00 VOC COCO Figure 12: Jensen–Shannon divergence across different layers of ResNet-18 (left) and VGG-16 (right) on training set. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 layer (ResNet) 0.0 0.5 1.0 1.5 W -d ist an ce VOC COCO 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 layer (ResNet) 0.0 0.5 1.0 1.5 VOC COCO
Figure 13: Wasserstein distance across different layers of ResNet-50 on training set (left) and test set (right).
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # layer (ResNet)
0.00
0.25
0.50
0.75
1.00
Cdi
st an
ce
VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # layer (ResNet)
0.00
0.25
0.50
0.75
1.00
JS -d
iv er
ge nc
e VOC COCO
Figur 14: Chebyshev distance (left) and Jensen–Shann n divergence (right) across different layers of ResNet-50 on training set.
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce Re sN et (V OC
)
layer 1 layer 4 layer 9
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce VG G (V OC )
layer1 layer 6 layer 13
Figure 15: Wasserstein distance between distributions in adjacent training epoch.
1 5 10 15 200
0.5
Re sN et (V OC ) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 15 20
target
1 5 10 15 200
0.5
VG G (V OC
) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 20
target
1 20 40 60 800
0.5
Re sN et (C OC O) epoch=1
1 20 40 60 80
epoch=30
1 20 40 60 80
epoch=70
1 20 40 60 80
epoch=100
1 20 40 60 80
target
1 20 40 60 80 # label
0
0.5
VG G (C OC
O) epoch=1
1 20 40 60 80 # label
epoch=30
1 20 40 60 80 # label
epoch=70
1 20 40 60 80 # label
epoch=100
1 20 40 60 80 # label
target
15
Figure 16: Distribution propagation across different training epochs of ResNet-18 and VGG-16.
Table 3: Over-thinking results of ResNet-18.
layer 0 1 2 3 4 5 6 7 8 9 ideal result VOC2007 1.17% 0.44% 0.69% 0.71% 0.55% 0.50% 0.53% 0.85% 0.53% 64.48% 70.44% MS-COCO 0.49% 0.43% 0.47% 0.28% 0.30% 0.31% 0.37% 0.78% 0.50% 27.33% 31.26%
17
# #
# #
Figure 15: Wasserstein distance between distributions of adjacent training epoch.
Under review as a conference paper at ICLR 2020 0 1 2 3 4 5 6 7 8 9 layer (ResNet) 0.00 0.25 0.50 0.75 1.00 JS -d iv er ge nc e VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 layer (VGG) 0.00 0.25 0.50 0.75 1.00 VOC COCO Figure 12: Jensen–Shannon divergence across different layers of ResNet-18 (left) and VGG-16 (right) on training set.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 layer (ResNet)
0.0 0.5 1.0 1.5 W -d ist an ce
VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 layer (ResNet)
0.0 0.5 1.0 1.5 VOC COCO

heading: Fi ure 13: Wasserstein distance across different layers of ResNet-50 on training set (left) and test set (right).
text: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # layer (ResNet)
0.00
0.25
0.50
0.75
1.00
Cdi
st an
ce
VOC COCO
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 # layer (ResNet)
0.00
0.25
0.50
0.75
1.00
JS -d
iv er
ge nc
e VOC COCO

heading: F gure 14: Chebyshev distance (left) and Jens n–Shannon divergenc (rig t) across different layers of ResNet-50 on training set.
text: 1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce Re sN et (V OC
)
layer 1 layer 4 layer 9
1 10 20 30 40 50 60 70 80 90 100
# epoch
0.0 0.4 0.8 1.2 1.6 W -d ist an ce VG G (V OC )
layer1 layer 6 layer 13
Figure 15: Wasserstein distance between distributions in adjacent training epoch.
1 5 10 15 200
0.5
Re sN et (V OC ) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 15 20
target
1 5 10 15 200
0.5
VG G (V OC
) epoch=1
1 5 10 15 20
epoch=30
1 5 10 15 20
epoch=70
1 5 10 15 20
epoch=100
1 5 10 20
target
1 20 40 60 800
0.5
Re sN et (C OC O) epoch=1
1 20 40 60 80
epoch=30
1 20 40 60 80
epoch=70
1 20 40 60 80
epoch=100
1 20 40 60 80
target
1 20 40 60 80 # label
0
0.5
VG G (C OC
O) epoch=1
1 20 40 60 80 # label
epoch=30
1 20 40 60 80 # label
epoch=70
1 20 40 60 80 # label
epoch=100
1 20 40 60 80 # label
target
15
Figure 16: Distribution propagation across different training epochs of ResNet-18 and VGG-16.
Table 3: Over-thinking results of ResNet-18.
layer 0 1 2 3 4 5 6 7 8 9 ideal result VOC2007 1.17% 0.44% 0.69% 0.71% 0.55% 0.50% 0.53% 0.85% 0.53% 64.48% 70.44% MS-COCO 0.49% 0.43% 0.47% 0.28% 0.30% 0.31% 0.37% 0.78% 0.50% 27.33% 31.26%
17
# #
# #
Figure 16: Distribution propagation across different training epochs of ResNet-18 and VGG-16.
Under review as a conference paper at ICLR 2020

heading: H EARLY-EXITS STRATEGY
text: Deep neural networks often occur the over-thinking issue, that is, the samples are correctly classified in the intermediate layer but misclassified in the last layer. We call these samples as confused samples. From Table 2 and 3, the accuracy of ResNet-18 (VOC) and VGG-16 (VOC) are 64.48% and 68.96%, respectively. In other words, out of 35.52% and 31.04% of the samples for ResNet-18 and VGG-16 are misclassified on the test set of VOC2007, respectively. For these misclassified samples, 1.17% are actually correctly classified at 0-th layer, 0.44% are at 1-th layer and so on. Ideally, if we early exit these confused samples, the cumulative accuracy of ResNet-18 is 70.44% on VOC dataset.
Early-exits strategy. We propose a simple probability mechanism to exit confused samples. Specifically, we add an auxiliary classifier for a selected intermediate layer to get the prediction probability distribution. Each auxiliary classifier contains fully connected layer and Sigmoid function. For each sample, we denote the number of exceeding the threshold p = 0.5 as N . We also denote the number of exceeding another threshold q as n. We denote the ratio as γ and γ = n/N . The threshold q and ratio γ are range from 0.5 to 1. We search for the best values of them to improve classification performance and show the results on Table 4.
Under review as a conference paper at ICLR 2020

heading: I MORE QUANTITATIVE RESULTS
text: We demonstrate more results on easy, hard and confused samples. For each sample, we show the image in the upper left, distribution corresponding to the selected layer in the upper right and the distribution propagation across different layers in the bottom.

Review summaries:
This paper seeks to understand both across-layer and single-layer behavior within neural networks (i.e. layer behavior along the depth of a network, and behavior of a single layer along training epochs). Therefore, they resort to the optimal transport framework to compare predicted and target distributions. Theoretically, they show that the Wasserstein distance between predicted and target distributions is decreasing along the depth and for a single layer, along training iterations. They also give intuition on how this analysis can help the learning process in practice.