Towards Interpreting Deep Neural Networks via Understanding Layer Behaviors
Authors: Jiezhang Cao
Jincheng Li
Xiping Hu
Peilin Zhao
Mingkui Tan
Venue:  (2020)
Keywords: Interpretability of DNNs
Wasserstein distance
Layer behavior

Abstract: Deep neural networks (DNNs) have achieved unprecedented practical success in many applications.
However, how to interpret DNNs is still an open problem.
In particular, what do hidden layers behave is not clearly understood. 
In this paper, relying on a teacher-student paradigm, we seek to understand the layer behaviors of DNNs by ``monitoring" both across-layer and single-layer distribution evolution to some target distribution in the training. Here, the ``across-layer" and ``single-layer" considers the layer behavior \emph{along the depth} and  a specific layer \emph{along training epochs}, respectively. 
Relying on optimal transport theory, we employ the Wasserstein distance ($W$-distance)  to measure the divergence between the layer distribution and the target distribution. 
Theoretically, we prove that i) the $W$-distance of across layers to the target distribution tends to decrease along the depth. ii) the $W$-distance of a specific layer to the target distribution tends to decrease along training iterations. iii) 
However, a deep layer is not always better than a shallow layer for some samples. Moreover, our results helps to analyze the stability of layer distributions and explains why auxiliary losses helps the training of DNNs. Extensive experiments on real-world datasets justify our theoretical findings.

Under review as a conference paper at ICLR 2020
TOWARDS INTERPRETING DEEP NEURAL NETWORKS VIA UNDERSTANDING LAYER BEHAVIORS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks (DNNs) have achieved unprecedented practical success in many applications. However, how interpret DNNs is still an open problem. In particular, what do hidden layers behave is not clearly understood. In this paper, relying on teacher-student paradigm, we seek to understand the layer behaviors of DNNs by "monitoring" both across-layer and single-layer distribution evolution.
Here, the "across-layer" and "single-layer" considers the layer behavior along the depth and a specific layer along training epochs, respectively. Relying on optimal transport theory, we employ the Wasserstein distance (W-distance) measure the divergence between the layer distribution and the target distribution. Theoretically, we prove that i) the W-distance between the distribution of any layer and the target distribution tends to decrease along the depth. ii) For a specific layer, the W-distance between the distribution in an iteration and the target distribution tends to decrease along training iterations. iii) However, a deep layer not always better than a shallow layer for some samples. Moreover, our results helps to analyze the stability of layer distributions and explains why auxiliary losses help the training of DNNs. Extensive experiments justify our theoretical findings.
INTRODUCTION
Deep neural networks (DNNs) have been successfully applied in computer vision, such as image classification (Chen et al. 2019h Hsu et al., 2019), image generation (Cao et al., 2019 Brock et al., 2 Chrysos al. 2019) and speech recognition (Yeh et al., 2019, Chen al. 2019a). Despite their success, the internal mechanism of DNNs is still black box. In particular, understanding what do hidden layers do and how to achieve remarkable performance remain persistently elusive. To answer these questions, we seek to understand across-layer and single-layer behavior.
For the across-layer behavior, we study the learning process by measuring the W-distance between the distribution of any layer and the target distribution. Recently, most methods only focus final predictions in different tasks (He et al. 2016). Due to the end-to-end training. interpreting each intermediate layer behaviors of DNN, which, however, is still not clear. To provide interpretahility. existing works try to produce a single le prediction and observe the classification performance of each layer (Papernot& & McDaniel 2018 Szegedy et al.) [2013 Kaya al. 2019). For example, (Alain & Bengio 2016) experimentally observe that the linear separability of features increases monotonically along the depth of a DNN. Unfortunately, there no theoretical analysis to support the experimental finding. To understand the learning process of DNNs, one can explore the across-layer behavior by monitoring how the distributions propagate across different layers. However, how to open the internal mechanism of DNNs and investigate the across-layer behavior of a DNN remains open question.
For the single-layer behavior, we seek to measure the W-distance between the distribution in an iteration and the target distribution. It is important to analyze the distributional stability of one layer by measuring the change of the distributions. Recently, the distributional stability of deep neural network attracts extensive attention (Santurkar et al. 2018). Some studies try to visualize the training behavior of one layer by plotting the mean and variance of features (Santurkar et al. 2018). Unfortunately, the visualizations are subjective and lack of necessary theoretical justifications. To this end, (Sonoda & Murata 2019) propose a transport analysis method and state that a denoising Autoencoder transports mass decrease the Shannon entropy of the data distribution. However, the

---

Under review as a conference paper at ICLR 2020
hidden layer
x
T.
f
loss
single layer propegation
probabliny
classifier
hg
h2
F6
COM
dog
bus
label distribution
across-layer propegation
labal
(a) Network forward propagation
(b) Label distribution propagation
(c) Label distribution
Figure I: Demonstration of DNN forward, distributions propagation and the intuition of the label distribution. (a) Given input æ, we train hidden layers and classifiers output label distributions fi- (b) Across-layer propagation: the label distribution A propagate (( Single-layer propagation: the label distribution P propagate _ during the epochs in one layer. (c) Given image with two labels (i.e., cat and dog), the label distribution depicts their corresponding probabilities of 0.5.
method is limited and inflexible for analyzing a general case of deep neural networks. Therefore, is very necessary and important to develop new analytical method interpret the stability of layers.
In this paper, we apply optimal transport theory to analyze the behavior distributions. Specifically, exploit Wasserstein divergence (W-distance) to measure the difference between the distribution of any layer and the target distribution. By monitoring the change of the W-distance, we are able to study both across-layer and single-layer behaviors. Our contributions summarized as follows.
We analyze the across-layer behavior and prove that the W-distance between the distribution of any layer and the target distribution decreases along the depth of a DNN. It means that every layer of the network
express the target distribution progressively.
We analyze the single-layer behavior and prove that for specific layer, the W-distance between the distribution in an iteration and the target distribution decreases across the training iterations when introducing loss in the layer.
Moreover, we provide experiments and theoretical justifications on these findings. The proposed analytical framework provides a different view of understanding and interpreting neural networks.
RELATED WORK
Many studies analyze the features of intermediate layers. (Bau et al.) 2017 evaluates how hidden units align set of semantic concepts to quantifies the interpretability of latent representations of CNN. (Dosovitskiy & Brox 2016) inverts image representations with up-convolutional networks for studying image representations. (Zeiler & Fergus 2014) presents a visualization technique to investigate the function of intermediate feature layers and the classifier. (Zhang et al., (2088) learns an explanatory graph reveal the knowledge hierarchy hidden inside a pre-trained CNN. (Zhang et al.) 2019) learns a decision tree to clarify the specific reason for each semantic prediction a CNN.
Some studies analyze the classification accuracy of DNN. (Lee et al. 2015 introduces classifica- tion objective to the individual hidden layers, in addition to the overall objective at the output layer. (Szegedy al. 2015) uses auxiliary classifiers improve the classification performance. (Kayal et al. 2019) introduces internal classifiers into off-the-shelf DNNs to understand overthinking of networks by studying how the prediction changes during a DNN's forward pass. (Gupta & Schütze 2018) explains recurrent neural networks by understanding the layer-wise semantic accumulation behavior. (Sonoda & Murata 2019) investigates the feature map inside a DNN by tracking the transport map, and prove that deep Gaussian DAE transports mass to decrease Shannon entropy of the data distribution. (Alain & Bengio 2016) experimentally observes that the linear separability of features increases monotonically along the depth of a DNN. However, there is no theoretical result to analyze this phenomenon. Existing studies using information bottleneck methods (Tishby & Zaslavsky| 2015 Bang al.) 2019) mainly analyze the dynamics of across different layers. However, it is hard for these methods to analyze the dynamics of a specific layer through different iterations. In contrast, proposed method able analyze both single-layer and across-layer behaviors.
2

---

Under review as a conference paper at ICLR 2020
3 PROBLEM SETTING
Notation. We use bold lower-case letters (e.g., x) to denote vectors, and bold upper-case letters (e.g., X) to denote matrices. We denote the transpose of vector (e.g., æ matrix (e.g. X") by the superscript [n]= 1,.0.,......... n.. Let be the convolution operator, grad be gradient operator.
Let 1:=[l..... 1]" and (s(-) be an indicator function of a set S, and let
Label distribution learning. In this paper, we consider a label distribution learning problem (Geng 2016), especially multi-label classification. In this setting, the label distribution is defined probability distribution to cover a certain number of labels, representing the degree to which each label describes the instance (see Tigure[[(c)) Note that the sum of the label distribution is equal to 1. Specifically, given training samples& S, we seek to learn model f from data space X' into the label space y. In Figure ((a) for L-layered neural networks, we denote sou-fto ... fo, ISL as the output of the /-th layers, then the label distribution mapping can be defined as fu==ho foa. where hy a probability function (e.g., FC+softmax (Frogner et al. 2015)). Note that fi, 1 [4] have the same input domains and the same output domains. Our goal is to learn model fi to let the predicted distribution üú close to the target distribution μs. Then, we optimize the empirical risk:
(I) where [ is the expectation w.r.t. the training set Se fiy denotes the pushforward operator of (4o, and d(.-.) .) is some distribution divergence, such Cross-entropp (CE) loss and Wasserstein loss. In
minf;
C(fi) == Es [d((t). H)], where (4=
ISL,
practice, the label distribution is obtained by training with CE loss. Because of the convexity of CE
loss, we derive the optimal label distribution for given features of every layer (Alain & Bengio 2016). In this sense, the label distribution reflects the actual distribution of feature maps in a specific layer.
Label distribution propagation. In this paper, our goal is to explore how the layer distributions propagate in the across-layer and single-layer of DNN. Specifically, we measure the divergence (e.g., Wasserstein distance (Villani 2008) to observe how the layer distributions change across different layers or iterations. In Figure I(b) we explain the behaviors of layer from the following two perspectives. For the across layers, measuring the distance between the distribution of any layer and the target distribution helps to understand the learning process and distribution expression ability. For a single layer, measuring the distance between the distribution in an training iteration and the target distribution helps to understand the dynamic behavior and training process of DNN.
4 WASSERSTEIN DISTRIBUTION PROPAGATION QUANTIFICATION
Optimal transport (OT). Recently the performance of multi-label classification can be improved by using W-distance (Frogner et al. 2015 Zhao & Zhou| 2018). However, DNNs on the lulti-label classification task still lack theoretical understanding. With the help of OT theory (Villani 2008).
W-distance can be used to interpret DNNs. In contrast, some measures, such Jensen Shannon divergence, which, however, often ignore any metric structure of distribution (Frogner et al. 2015).
Definition 1 (W-distacce (Villani 2008) Given a prediction distribution ça and the target distribu- tion and the cost matrix C defined as dkc k²) with the metric dx, where K, K' are label tags, then the W-distance seeks to find a transportation matrix T by transporting a mass ç to μt,
W2 (p. u) infrença, (T, C)2
where [1 u) is the set of couplings and defined as Ui ( TI-2,T"1-N11
In practice. the distance of two label tags in the cost matrix can be calculated by using word2vec (Mikolov et al. 2013) in the Euclidean space. We calculate W-distance with entropic regularization by Sinkhorn algorithm (Knight 2008) with p=2. See more details in Supplementary materials.
propagation measure. In this paper, we use W-distance to measure the across- layer and single-layer label distribution propagation, shown in Figure [(b) For an across- layer, we measure W-distance between the distribution of any layer and the target distribution, i.e.,
Distribution
wD (Ft,.)) In this way, we can measure how the distributions propagate across different layers. For the single-layer, we measure W-distance between the distribution in an training iteration and the target distribution, i.e., (²( :)...., W2 (²(2()) 11). Therefore, we can study the behavior of distributions in the same layer.
3

---

Under review as a conference paper at ICLR 2020
T-net
f
f2
Fo
fi
S-net
fa
fi
(a) one-looeee layer behavior analysis (b) infinite layer behavior analysis (c) finite-layer layer behavior analysis
Figure The teacher-student paradigm for analyzing layer behaviors of DNNs. We analyze one-layer
behavior in (a), infinite-layer behavior in (b) and finite-layer behavior in (c).
MACHHERSTTUDNT FRAMEWORK FOR LAYER BEHAVIOR ANALYSIS
Teacher-student analysis framework is widely used analyze and understand DNNs (Tian 2017 Du et al., 2018 Cuturi 2013). For the one-layer behavior, this framework helps to understand the dynamic of student network from the teacher network. For the multi-layer behavior, this framework helps to study the ability of the student network to express distributions of the teacher network. Therefore, stadeet-student analysis framework is important and necessary understand DNNs.
5.1 ONE-LAYER BEHAVIOR ANALYSIS
In our problem setting, the target distribution in the training loss (I can be represented by an output of teacher network (T-net). Our goal is make the output of the student network (S-net) to be close to the output of TTnet, as shown in Figure 2 (a). Specifically, given a S-net f and '--net f", our goal is find function f that minimizes the following training loss,
[(J)= = Es [d( n)] = En-S [1](*)) f' [())
where ã=m+ gnay the corruption of data, and v4 is noise distribution w.r.f. the variance #I, Le., DO (T) By optimizing the training loss, the distribution one layer can be monitored by using a transport map. Next, we provide the definition of transport map below.
Definition (Transport map) A transport map fe: IR -IR can be defined as f((x)=x+te((t) with update direction 9t when t>0, and fo(æ)=x when t=0.
Based on the definition of the transport map. we analyze the layer behavior of ResNet (He al. 2016), i... T-net is residual block. Note that our analysis method can be extended to more general case. Then, we solve the optimal transport map as follow.
Theorem (Optimal transport map) If the teacher network is a residual unit f where contains FC layer and an activation function. When n,==N((0, IT), the minimum f is
fz) = + tV log(v; * po) o (z) == x + (z).
The minimizer ft updates æ along the update direction gr (a). Therefore, the optimal transport map Jč with time / transports the mass at the activation æ toward g.r)). To associate with continuity equation (Sonoda & Murata 2019), we introduce Wasserstein gradient flow (WGF) as follows.
Proposition 1 (Wasserstein gradient flow) Assume 14t satisfies the continuity equation with the gradient vector field 0, V- V-Iu, of a potential function V,, and if the function F satisfies (,F())= S (z(x)) (a [,, then WGF coincides with g-addF
Using the definition of WGF, we explore how the distribution propagate in one layer as follows.
Theorem 2 Based on the equivalent condition in Theorem (Belavkin 2016), and let A be the Laplacian operator. At the initial moment 1-+0, the pushforward I The with Gaussian distribution satisfies the backward heat equation (BHE), and evolves according to Wasserstein gradient flow:
0,p:=o(z)=-- ( grad W2 (1u p)), xER"
Note that W is the potential function. Theorem 2 states that by introducing auxiliary loss in layer, the W-distance can be decreased in the layer. It means that the label distribution can close to the target distribution across training iterations.
4

---

Under review as a conference paper at ICLR 2020
5.2 MULTI-LAYER BEHAVIOR ANALYSIS
(i) Infinitely deep neural networks. For infinitely DNN, we assume the number of layers and hidden units and the size of dataset infinite. Specifically, let 0 <1,+1=1, trained on Ao with the variance to. then æ foo== and its distribution is =1:= we train fi on dl with the variance f2-t1. In the same way, we obtain fi(w;) and 1 The composition of residual blocks be written (x)::=)=1100 / (o().) where / is total time. Then, the velocity of the composition coincides with the vector field 0z ( VV) (x).
and let fo be
Then,
Furthermore, we extend one-layer behavior analysis to the case of infinite layers, as shown in Figure 2 (b). Based Theorem
have the dynamical systems follows.
Lemma When the noise distribution is a Gaussian distribution, the continuous residual units is defined as a flow Py: RT- of the following dynamical systems associated with the vector field VV,, i.e, d (xt (x())) +o (=(((t)))))))))))))))))))()))( where t20 and (y:=
Based on the proofs of Theorem we can achieve a similar conclusion follows.
Theorem Based on the equivalent condition in Theorem (Belavkin 2016). and when the noise
distribution is Gaussian distribution, then the pushforward measure i No evolves according
Wasserstein gradient flow as follows: 4. (
ud], pt=0=po
Theorem states that the W-distance can be decreased across different layers. means that the label distribution can be close to the target distribution along the depth of the layer.
(ii) Finitely deep neural networks. When the number layers and the hidden unit number are finite, the above analysis is not available. As shown in Figure (c), given composition of 7. T-nets, we try to analyze the ability of each layer of S-net express the distribution of T-net. Hence, we derive approximation bound for finitely deep neural networks. Here, Barron function is present T-net and its definition is provided as follows.
The function yp is Burron function B if ip satisfies Qg(()) (s<(C)((p: B-R: 36, Φ1B=p, | FEFB), || B Jgm || as Φ w) duc, where w/ sup B (w, æ)) and 6 is the Fourier transform of 6. and FB is the set of functions with the Fourier inversion formula holds on B.
Definition 3 (Barron function (Sonoda & Murata 2019
Based on the definition of Barron function, (Barron 1993) states that Barron function can be approximated by neural network with one hidden layer. Given fixed composition of Barron functions, we derive the error bound of the approximation between such composite function and neural network with hidden layers. Note that / not necessarily equal to I.. The approximation bound of w.r.f. Wasserstein distance can be provided as follows.
Theorem (Wasserstein distribution approximation) Given a data distribution μig and a function Pp: Rm--1 mI and let L-= (;=log([)+1] sfpport) CKo and (K; Sisl, the function sp -)) Lipschitz and is a Barron function, i.e., spi SplESrx. and p,E5l (C.). neurons on the i-th layer,
then there exists a network f with hidden layers with [4C7
6 z2 +1). 1S Wi( S (PECNTT D.)²
² T
where Dj is the diameter of the set Kr, and 5, 8>0 are parameters.
Theorem provides an error bound when using a neural network with hidden layers to approximate fixed composition of L Barron functions. Moreover, the upper bound decreases with the increasing of layers under certain conditions. Here, we can conclude that deep layers are not always better than shallow layer for some specific samples. Such samples layer rather than the deep layer, then the diameter D, can be large for this set of samples. Hence, shallow layers can behave better than deep layers for these kinds of samples.
often classified correctly in the shallow
5

---

Under review as a conference paper at ICLR 2020
1.5
1.5
W-distance
VOC
COCO
voc
COCO
1.0
1.0
0.5
0.5
0.0
0
0.0
1
2
3
4
5
7
10
11 12 13
#layer (ResNet)
#layer (VGG)
Figure 3: Wasserstein distance across different layers for different networks.
(VOC)
0.5
layer2
sver?
layer
Sarget
15
10
10
10
VGG ReaNet VGG
(COCO) (COCO) (VOC)
0.5
layel
layort
layaz21
tergat
10
15
70
5
18
15
20
10
15
20
10
15
20
10
15
20
20
40
60
80
20
40
60
80
20
40
80
20
40
60
80
20
40
60
80
0.5
byer)
layerd
layeril
40
40
label
label
llael
label
#label
Figure 4: Distribution propagation across different layers for different networks.
EXPERIMENT
Implementation Details. All experiments are implemented on PyTorch. In the training, we use SGD optimizer with the initial learning rate 0.01. The learning rate decays by a factor of 10 for every 40 epochs. The momentum and weight decay is set to be 0.9 and 10-4 respectively. We obtain models (including ResNet-18, ResNet-50 (He al. 2016) and VGG-16 (Simonyan & Zisserman 2014)
pre-trained on ImageNet(Deng et al., 2009) and train them for 100 epochs with the batch size of 128.
Then, we freeze the model weights and train the internal classifiers (i.e., the network h, Figure (aa for 100 epochs. Each internal classifier contains fully connected layer and Sigmoid function. We train the network using Binary Cross Entropy. We use training set to explore the distribution propagation of the across- and single-layer. In addition, set a=0.01 in Eqn. t7 achieve the balanced results. For simplicity, we use ResNet to represent the ResNet-18 model and VGG to represent the VGG-16 model in the main paper.
Data Sets and Evaluation Metrics. We conduct experiments on two benchmark multi-label clas- sification datasets. i.e., VOC2007 (Everingham et al. 2010) and MS-COCO (Maas et al.) 2013). VOC2007 (Everingham et al. 2010) contains 9,963 images from 20 object categories, which is divided into train, val and test sets. MS-COCO (Maas al. 2013) contains 82,783 images as the training set and 40,504 images as the validation set. The objects are categorized into 80 classes with about 2.9 object labels per image. We use the classification accuracy, average per-class Fl (CFI), average overall F1 (OFI) and mean average precision (mAP) as the evaluation metrics.
6.1 ACROSS-LAYER BEHAVIOR EXPLORATION
6.1.1 THE BEHAVIORS OF LABEL DISTRIBUTIONS IN DIFFERENT LAYERS
How does every layer express the distribution? Here, we study the expression ability of every layer, which is measured by Wasserstein distance (W-distance) between the label distribution of any layer and the target distribution. As suggested in Theorem 4 deep layers have smaller error bound than shallow layers, meaning that they have better expression ability. From Figure 3 deep layers have smaller the W-distance than shallow layers, which verifies the conclusion in Theorem 4 Moreover, the W-distance decreases from shallow layers to deep layers. The shallow layers have similar ability to express the target distribution since they have similar values of the W-distance. When approaching to the last layer, the W-distance drops to a very small value. implics that sufficient layers are able express the target distribution. More results shown in Section F in Supplementary materials.

---

Under review as a conference paper at ICLR 2020
Table 1: Improve performance of ResNet and VGG.
VOC
COCO
Method
accuracy
CFI
OFT
mAP
accuracy
CF1
OFI
MAP
ResNet
64.48
58.02
59.10
85.18
27.33
55.65
60.30
64.36
ResNet+Farllyxits
66.01
58.67
58.76
85.49
30.03
57.49
61.38
67.28
VGG
68.96
58.58
59.71
88.48
32.41
59.37
62.94
70.64
EGG-EarlyExits
69.85
59.49
59.94
88.57
33.95
60.32
63.73
71.95
Target
Inrget
target presiction
dpr apribntt diyes
prediction
predict
0.0
CD
tabel
1.0
W-dstancs
W-- widccaance
0.0
0.0
[a] easy sample
(b) hard sample
(c) confused sample
Figure Demonstration of properties on easy, hard and confused samples.
How does the distribution propagate across layers? In this experiment, we investigate how the label distribution propagate across layers understand the learning process from layer layer in DNN. Note that we normalize the prediction of classifier as the probability distribution (i.e. the sum of label prediction probabilities one sample is 1.). Figure shows that the label distribution of one sample propagates from the first layer to the last layer. In the shallow layers, the label distribution is far away from the target distribution, then can be close to the target distribution in the deep layers. Different from the decreasing tendency in Figure 33 the label distribution of one sample may not close the target distribution progressively. For example, in the first row of Figure the probability of the 10-th class is large in the 0-th layer, but goes down in the 3-th layer.
6.1.2 THE EVALUATION OF ACROSS-LAYER' BEHAVIORS
How to evaluate the quality of each layer? Existing methods attempt visualize the feature maps to evaluate the quality of each layer. However, observing the feature maps is difficult because of its high dimensionality and complexity. Relying on the W-distance, we can evaluate the quality of each layer by measuring the ability of expression. If the W-distance decreases rapidly in a layer, it suggests that the quality of this layer is good. Taking ResNet on VOC in Figure as an example, 0~5 layers have similar value of the W-distance, while 6~88 layers have quite different value of the W-distance, means that these deep layers (i.e., 6-~8) are better than the shallow layers (i.e., 0~55).
1.3 HOw TO EXPLOIT THE ACROSS-LAYER BEHAVIORS
Does every sample contribute equally? In practice, a deep neural network construct more complex features progressively throughout the layers (Lee et al. 2011). An interesting question arises: does the contribution of each sample vary across different layers and training iterations? To answer this question, we first define the concept of sample difficulty in terms of the W-distance. As shown in Figure the samples can be divided into three categories. including easy, hard and confused. The intuition is that: 1) Easy samples should have small the W-distance the first few layers, i.e., they should be classified correctly with high confidence in a shallow layer. 2) Hard samples would have large the W-distance in a deeper layer, i.e., it cannot be resolved all, or only near the last layer. 3) Confused samples have small the W-distance in shallow layer, while have large the W-distance in a deep layer. It means that although these samples can be classified correctly in the shallow layer, they still become misclassified at the last layer. Considering the difficulty of samples would help to interpret the training of models and design the training loss, which may speed up convergence and improve the performance eventually. We show more results in Section [ in Supplementary materials.
7

---

Under review as a conference paper at ICLR 2020
Pastett (VOC)
W Wisistnnce
1.8
layer
laya
layer 9
W-distance
VGG (VOC)
1.8
layer2
-
byar 6
layer
30
70
Resset (COOO)
sepcch
4-epcch
Waistance
1.2
W-distancee VGG (COGO)
1.2
layor
bayor
layer
layer2
baye 6
+ layor
0.4
20
30
40
50
60
70
80
90
100
10
20
30
40
50
60
70
80
90
100
epoch
eepcch
Figure 6: Wasserstein distance between the distribution in epoch and the target distribution across different training epochs for different networks. We choose the 1, 4, 9-th layer for ResNet and the 1, 13-layer for VGG.
How to improve classification performance? We will discuss how to exploit the behaviors of different layers to improve the model performance. In practice, we observe that in deep neural networks, some samples are correctly classified in the intermediate layers but misclassified in the last layer. For example, 35.52% and 31.04% of the samples for ResNet and VGG misclassified the test set of VOC2007, respectively. For these samples, 5.96% and 7.76% samples are correctly classified in all intermediate layers of ResNet and VGG, respectively. Based on this phenomenon, we can improve the performance by early exiting such confused samples. Different from the strategy of SDN (aay et al., 2019). how early exit for multi- classification DNNs very challenging. Therefore, we design new early-exits strategy for multi-label prediction. In Table I] our proposed method consistently outperforms the baseline methods. We give more details about carly-exits strategy and show more results in Section H in Supplementary materials.
6.2 SINGLE-LAYER BEHAVIOR EXPLORATION
In this section, we study how the distribution of an intermediate layer change during the training process. We consider use W-distance measure the stability of distribution. We propose theoretical and empirical analysis for the distribution stability in one layer.
How the distribution propagate during updating DNNs? In this experiment, we investigate how distributions propagate when training DNNs. From Figure the label distribution in the first layers of ResNet or VGG often fluctuates significantly due to the limited discriminative power of very shallow layers. When approaching the last layer, the supervision is sufficient to decrease the W-distance. These experiments justify Theorem 2 that the W-distance can be decreased with sufficient supervision. More results shown in Section G in Supplementary materials.
How stable the distribution in one layer? The training stability is a key problem in DNNs and can be largely addressed by the widely used Batch Normalization (BN) (loffe & Szegedy 2015). However, we prove that the stability of distribution is hard to be guaranteed even with BN (See proof in supplementary). In this paper, distribution in an iteration and the target distribution across different training iterations in the specific layer. Motivated by this, we further measure stability of label distribution during training DNNs. The conventional understanding of BN suggests that the W-distance should decrease. Note that, from Figure 6 the first layer of ResNet or VGG may fluctuate during the training process, which indicates that deep layers often have better ability than shallow layers to express the target distribution.
the W-distance to measure the discrepancy between the
CONCLUSION
In this paper, we have interpreted deep neural networks (DNNs) via understanding layer behaviors. With the help of optimal transport theory, we propose a unified teacher-student analysis framework investigate the across-layer and lingle-layer behaviors of DNN. Theoretically, we prove that the W-distance between the distribution of any layer and the target distribution decreases along the depth. ii) the W-distance of a specific layer between the distribution an iteration and the target distribution decreases along training iterations. Extensive experiments justify our theoretical findings. The proposed analytical framework can facilitate future researches to interpret a deep neural network.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: 1610.01644, 2016.
Seojin Bang, Pengtao Xie, Wei Wu, and Eric Xing. Explaining black-box using deep variational information bottleneck approach. arXiv preprint 1902.0690 2019.
Andrew R Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions Information theory, 39(3):930-99) 945, 1993.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, pp. 6541-6549, 2017.
Roman V Belavkin. Relation between the kantorovich--waserstein metric and the kullback-leibler divergence. In Information Geomeiry and its Applications IV, pp. 363-373, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Jiezhang Cao, Langyuan Mo, Yifan Zhang, Kui Jia, Chunhua Shen, and Mingkui Tan. Multi-marginal wasserstein gan. In Advances in Neural Information Processing Systems, 2019.
Chun-Fu (Richard) Chen, Quanfu Fan, Neil Mallinar, Tom Sercu, and Rogerio Feris. Big-little An efficient multi-scale feature representation for visual and speech recognition. In International Conference on Learning Representations, 2019a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In International Conference on Learning Representations, 2019b.
Grigorios G. Chrysos, Jean Kossaifi, and Stefanos Zafeiriou. Roc-GAN: Robust conditional GAN. In
International Conference Learning Representations, 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems, pp. 2292-2300, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp.
2009.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Computer Vision and Pattern Recognition, pp. 4829 -837, 2016.
Simon S Du, Jason D Lee, and Yuandong Tian. When is convolutional filter easy to leam? 2018.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International joueraalionaauuralll of computer vision, 88(2): 303-338, 2010.
Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learning with a wasserstein loss. Advances in Neural Information Processing Systems, pp. 2053-2061,
2015.
Matthias Gelbrich. On formula for the 12 wasserstein metric between measures on euclidean and hilbert spaces. Mathematische Nachrichten, 147(1):185-203 1990.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn diver- gences. arXiv preprint 1706:007700 2017.
Xin Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 28 (7):1734-1748, 2016.
Pankaj Gupta and Hinrich Schütze. Lisa: Explaining recurrent neural network judgments via layer- wise semantic accumulation and example to pattern transformation. Empirical Methods Natural Language Processing Workshop BlackboxNI.P. 2018

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. In International Conference on Learning Representations, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In International Conference on Machine Learning, volume 97, pp. 3301-3310 2019.
Philip A Knight. The sinkhorn knopp algorithm: convergence and applications. SIAM Journal on Matrix Analysis and Applications, 50(1):261-275. 2008.
Chen-Yo Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In Artificial intelligence and statistics, pp. 562-570, 2015.
Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. Unsupervised learning of hierarchical representations with convolutional deep belief networks. Communications of the ACM, 54(10))95-003, 2011.
Jun S Liu. Siegel's formula via stein's ' identities. Statistics & Probability Letters, 21(3):247-251, 1994.
Andrew L. Maas, Awni Y Hannun, and Andrew Ng. Rectifier nonlinearities improve neural network acoustic models. In International Conference on Machine Learning, 2013.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 111111119, 2013.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: 1803.04765 2018.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In Advances Neural Information Processing Systems, pp. 2493, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1553 2014.
Sho Sonoda and Noboru Murata. Transport analysis of infinitely deep neural network. The Journal of Machine Learning Research, 20(1):31-82, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing propertics neural networks. arXiv preprint arXiv:1312.6199, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Computer Vision and Pattern Recognition, pp. 1-9, 2015.
Yuandong Tian. An analytical formula of population gradient for two-layered relu network and its applications in convergence and critical point analysis. In International Conference Machine Learning, 2017.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
10

---

Under review as a conference paper at ICLR 2020
Chih- Kuan Yeh, Jianshu Chen, Chengzhu Yu, and Dong Yu. Unsupervised speech recognition via segmental empirical output distribution matching. In International Conference on Learning Representations, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833, 2014.
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun Chug-Chunnnu Zhu. Interpreting cnn knowledge an explanatory graph. AAAI Conference on Artificial Intelligence, 2018.
Quanshi Zhang, Yu Yang. Haotian Ma, and Ying Nian Wu. Interpreting cnns via decision trees, In Computer Vision and Pattern Recognition, pp. 6261-6276 2019.
Peng Zhao and Zui-Hua Zhou. Label distribution learning by optimal transport. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

---

Under review as a conference paper at ICLR 2020
SUPPLEMENTARY ON "TOWARDS INTERPRETING DEEP NEURAL NETWORKS VIA UNDERSTANDING LAYER BEHAVIORS"
A PRELIMINARY
Notation. Specifically, we use bold lower-case letters (e.g., z) to denote vectors, and bold upper- case letters (e.g., X) denote matrices. We denote the transpose of vector (e.g., ( matrix (e.g., )) by the superscript Let == [1....,1] .1"². For two matrices and B of the same size,
denote their inner-product (A, -tr B). Let be the convolution operator. Let || ||=|| a' denote
Euclidean norm on vectors in IR. For a function f. let f===(-=x). Let A >
the Laplacian. Let B, be the unit ball in dimension, i.e., B,= ||||||||||| <1). For two sets A and B and a scalar r R, let A+B-(x+y æEA, yEB) and rA-(rw: xEA).
Definition (Lipschitz continuous) A function yo: R -R" called L-Lipschitz continuous if there exists real constant T,20 such that, for all and y w.E.. the T? norm,
[1p(a)) <0()))2 L|l y112.
Definition (Fourier transform) For L (RR) Fourier transform of po: R" defined as: )(,,,))
((w) = (2 p(x)) -( dx.
(5)
For vector-valued functions yo: RT-R=m conduct component-wise Fourier transform.
Let ( : ( the inverse Fourier transform
())) = / ((w))e do = (2T) (2x)# GV.
(6)
Definition For a bounded set B, the norm of a function f: can be defined on a set B, || Jaz [| (())(ww))) ldw, where tul|s=supree (ww x.)l
B PROOF OF THEOREM 4
Definition (Barron function (Sonoda & Murata 2019)) The function yp is Barron function on B if
satisfies (x(C)=((p: B- 30, @| @ i CC, FEFB). where 0l JRm []]l|/@(w) (dw. where ||w]] l|- suPweB (w, æ>) and Φ is the Fourier transform of d. and Fs is the set of functions with the Fourier inversion formula holds on B,i.e,,
FB (0: - + Vx E B(x()) + (cttuaa)-
+
((w))d
Definition 8 (Sigmoidal function) A sigmoidal function is bounded measurable IR+ such that lim (x)=0 and limz-+t0 =1)=
Theorem (Barron theorem (Barron 1993)) Let BCR be bounded set, and μ be any prob- ability measure B, (C) and [(x) Ei qa (w, x) + bi), where af-) be sigmoidal function, then there exist w.ERd b,E., o; ER with En_ |cal |2C such that
[|+((x) f(x) = (((x)) - f(x)) (p(ar) S (2C)²
(8)
We extend the above theorem the case of the composition of Barron functions denoted as Pp Pi (Pj-1 .... sp..
Corollary Let be bounded set, be any probability measure and the neural network f(x)= Ei= Go ((w;, x) +b)), where (-) is sigmoidal function. the composition of Barron
12

---

Under review as a conference paper at ICLR 2020
functions is restricted in the set of (CB(C), then there exists W,ER", b,ERR with Σ 1a|520 such that
1x)) - f(x))1 ( u(dx) _ (2C)²
(9)
Proof Directly using Lemma and Theorem of (Barron 1993) can complete the conclusion.
Theorem (Extend Barron theorem) Let be a bounded set, i be any probability measure and f(x)= Ei=1 (ow bb). where o(-) be sigmoidal function. (o and (;(KC i),,, Isisi, the function pp is log(t-1) Lipschitz and Barron function, iee, 91 E5exx (Co) and - B (C:), then there exists neural network f with hidden layers and S C Rd satisfying No(S) (2 -ip 1)-112 so that
Tog(U]+
( |sslso- -1P S log(1)+1
(10)
Proof For simplicity, let p== op1 be the composition of Barron functions, especially P1= where 3 defined in Corollary Let fi= o fi be the composition of k-layered neural network, where fi RRd Rdd are functions defined as
(f(())); = Σ Cifk o(wij, x) + bigk),
(11)
where Cijk: ER and w. parameters, and is the number of nodes in the i-th layer. Note that the function fü is represented by a neural network with one hidden layer and a linear output layer. We prove the theorem by induction on I.
i) For /=1, using assumptions of the support initial distribution (i.e., support((n) definition of Barron function. Then, c= 2 //r and Σ |cal S 2C1, we complete the results.
C KG)) and the
ii) For 1>1, by the induction step, we assume the functions f1,..., fi-1 satisfy the conclusion for P1,...,, PI-11 Let S,-1 be the set in the conclusion. Applying Corollary to fi get that for each I j S. my, for any μ supported on a set Ki_, C Rmm-- and any rı E N. there exists a neural net Jtg with 1 hidden layer with rı nodes such that
Uern (PiS- - f(.))
20 20
vii
Note that Theorem applies to any distribution a supported on the set KG-1- Let S S
fi-1a (x) Ko-Km r= 4C ml 11-111111-11 1)+ (1) and f- (1s, We have that μ
Applying Theorem with
supported on g1-1:1 (Si) S Ki-rtsmm_ LirBBB_,KL,,, and the function şi is Cy-Barron function on this set by the assumption. The conclusion of Theorem gives (fi) such that
(915 (O
fi. f(.) (i. (1s,
20 Vii
(nj(lo((--1)+ (log)))))))))))))))))))))())))))
For all elements of ipı and fi, we have
1 (gr.., |l - d(s (1
(log(I-1)+ (()g=-1)))((og(()t}))
13

---

Under review as a conference paper at ICLR 2020
We bound by the triangle inequality
( 1s fiall/duo)
(1
(fm 1sillsi ft-1a f
1
- -p /1-1:1 01-1:1112440
St S1-1.1 (1-::1/duun 1pd
+
(L 1sill900 - f1-1:111
( Rm
1s fill dsllli--ill'dfiin (Is,
<Lip(si) (Le 11s111111111 - 11-1:1 122
E
(log(1 (og(l-1)+H) ()0])))))(((o(()))))))))))))))))
7/4 +
<Lip((i)
(fe
1s-.1ll91- f1-1.1 duo
6
(0g(l-1)+1)((og(()))
log(l-1)
1og(1)+1
log(1-1)4
(log((=1)+1)(og(())
log(()+1
where the last inequality holds by the assumption (i.e., Lip((p))=
log(t- 1) and the induction hypothe- log()++
sis.
As above analysis, we have
/ 1s ft-radpoll2ppa S (log(I-1++1)2'
22
by the induction hypothesis. Also, P1-11:1 E Ki-1 for all x E support(un) by the assumption of the theorem. Therefore, by Markov's inequality and the induction hypothesis on Si-1.
#40 (S n fx: x & Ki-1 + sfi (Bm..)(mm.))
Suo (S {x: ||ę1-14 fi-1. 8}) s
2(log(1-1)+1)
Therefore,
2
(o(8)) S (o(()-1) - 8² (log(1-1)+1]3
the last equality holds by fact that there exists constant 5 to satisfy (a(()) = ([log((-1)+1])
dea
Based on the above conclusions, we prove Theorem aas follow.
Theorem Given data distribution pn and a function yp I and let log)o)())+ +1, if support(po (P)) CKo and Pi(K,-1)KK,, 1sis,, 1 the function (p; is ( (---)--) Lipsschtt and is Barron
function, i.e,, sllk, (Co) and prE00,, (C,), then there exists a network f with hidden layers with [4C 12/// neurons the i-th layer,
(i, ((CINTT Di)² (222+1) 15 L.
è² T.
(12)
where Dj is the diameter of the set Kr, and e, Ö, 8>0 are parameters.
Proof The functions fi,..., Si in Theorem 6 satisfy that
f 1s_,1p91111 f- S (log(1-1)+11)
The range of fi (..),,.... (J)) is contained in set of diameter 20, mimm because the activation
in neural network is ranged in [0, 1] and the weights Ctyte in Theorem 6 satisfy Er lijkl S 2C1.
14

---

Under review as a conference paper at ICLR 2020
Choose constant vector k to minimize Js, (1 - fi.1 - kll2duo and replace fi with fi + k. Note that the range of fi and (ir necessarily overlap. We still have Js, (x:)(())))))))))))()) fr.1 duo (log())+1)2- Moreover, have
e²
((()(x)) - fi(2:))l & 201ym + Dr. for æ E Ko, where Dj is the diameter of KG. Let Si be a complementary of St, we have Sc
(aS)) = s² ([log(11)++1))
then we have
f(. 11 ftalldpo s
S 11911 fiall fiall2dpo S 11 ftall
6²
8e2
(2C1/m + Di)²
(log(1) 1)2
s² (log((-1))
e²
(2Civm) D2) +1+
(log(1) + 1)²
For (X), ~ 140 define a coupling between the distributions. Based on the definition of W-distncce have
(1x, =Ixx=ao I141:1 - fiall2
=/e .Koo
l1 - fi.1 diu
6²
(2Civmi + Di)² & z2 +1)
(log(1) + +12
C PROOFS OF LAYER BEHAVIOR ANALYSIS OF DEEP NEURAL NETWORKS
Definition 9 (Flow) A flow is given by an ordinary differential equation (ODE), (xx) (x))=))(((((( with velocity field V. when 1>00, and < when 1=0.
Proposition 2 (Continuity equation (Sonoda & Murata 2019 Let be the flow of an ODE with vector field vG then the distribution μr evolves according to the continuity equation Ospt (x)== =- V. (x))v xER" t20. where V. denotes the divergence operator.
Theorem (Optimal transport map) The global minimum fł of f(f) is attained at
ft
1/2 * Po (x) J EVi(E))o(E E)de,
where * denotes the convolution operator.
Proof
The proof follows from the calculus of variations.
C(f)
Er
[1f(x +e) - f((x))1]) po(x))d
Rmm
Ez
[1](x())- f' E)] po(el - $) dæ',
mmm
where the second line holds by the calculus of variations, i.e, x' + Then, for arbitrary function h, the first variation 8C(h) is given by
d - C(f + sh)
8L(h) ds
==
i Ee
[1((z) + sh(x) fl =)ll² no( z) dx
Rm
Os
==0
=2
Ee ((f(x) f' E)) no( h(z)dx.
15

---

Under review as a conference paper at ICLR 2020
At critical point f of L, (h(h) for every h. Hence,
Eg [[f(z) f E)) fo(æ e)]= =0 o.e. I,
by the fundamental lemma of calculus of variations for integrable functions, and we have
f*)x) E8g (f =)/o - =)]
Eg [ E)]
Note that f attains the global minimum, because, for every function h.
L( + h) =
/
Eg (x:) fa E) + h(xx)1l no(a E)] dx
Rm
f
Ee [x)) f' s)(? no (0 - e)] dx { Ess [[l(2))] uo(o - e)] dx
+ 2
Ee [<(x)) (f) f E)) no(x - E)] dx
R"
= + L(h) + 2-02 L(
Theorem 1 the target network is residual unit f to (σ)) where contains fully connected layer and activation function. When Vt == N (0, t1), then the global minimum ft is
(z) a(x) = ã + (3)).
Proof Gaussians.
Using Stein's 's identity
Liu 1994
- -IVu)(E) = EV(E), which is known to hold only for
(7)
evi(E))o(@ - e)dle + (σ))
Ut * no a))
Rm
(Vu(e)) e)de (
V/ * no a
Jmm
IV44 (1( +
Vi (
= tV log(u (o())) + (
Theorem Based on the equivalent condition Theorem 2 (Belavkin 2016p, and let & be the Laplacian operator. At the initial moment t-+0, the pushforward J ue with Gaussian distribution satisfies the backward heat equation (BHE), and evolves according to Wasserstein gradient flow: W NER"
Proof
The initial velocity vector given by
(+x -
(z) = -im
V log (o() + ((x)).
(13)
t-50
t
Hence, by substituting the score 13 in the continuity equation of Proposition 2 we have
011+=((x:) V (po()) log (x) + axx))) = Ax)(x))
We leave the proof of (u)(x) = (pr, (4) in Theorem 3
Theorem Based on the equivalent condition in Theorem 2 (Belakkin 2016), and when the noise
distribution is Gaussian distribution, then the pushforward measure pa: = Mn evolves according to Wasserstein gradient flow as follows:
d
di - = -gradW"(uu, (o=)(2))Ho(a))
(14)
16

---

Under review as a conference paper at ICLR 2020
Proof Let the potential function be equal to Kullback-Leible divergence be KL p)) pe(x) + ((x)) then V = (1og((1)) - log(u)), then
Pe(x)) log Me(a) u(æ)
gradF( = V (14 (1og((4) log(())) = A14-
The above continuity also satisfies the case of Wasserstein distance, then using the equivalent condition in Theorem 2 (Belavkin) 2016), we have
Orfir 0() --rad (4,,
assume the distribution ii defined as follow.
(9) Gaussian distribution. The W-distance two Gaussian distribution
Proposition 3 (Gelbrich 1990 Given two Gaussian distributions M1~N(u1, En) and
N22~ En), then Wasserstein distance between μi and P2 is
- u2ll + tr -2+=2-2(E(EE3))
Batch Normalization is proposed stabilize the distributions of layer distribution by introducing the parameters y' and 3 in the f-th optimization step. Let =daagia"( (() and p(ttt)=diag(( ((+1)).
Proposition 4 Assume the distributions two iterations in the I-th layer are Gaussian distributions, i.e, 1h (t) (N(( r(()) and μ ((+1) (N( r(f+)), then the W-distance between (0) and pi
1e+1)
W (ML ((+1))= -113 3 +++ ( 111
Proof
Directly from and let un 31 u2 u2 (( ((+1) and - - have
W (0) ii
(++1)) -+ -B² - (=+))((())) -r(tt))
17

---

Under review as a conference paper at ICLR 2020
D OPTIMAL TRANSPORT
Definition 10 (Optimal Transport (Villani 2008)) Given a cost function c: KxK--R, the optimal transport distance measures the optimal plan transport the mass from a probability measure p to another probability measure μui
SK c(c, (.) drs).
inf
(15)
=En(A.))
where [1 pu) is the set of joint probability measures т on KxX with the marginals p and μ..
In this paper, we consider the multi-label classification problem. We first introduce the definition of Wasserstein distance. For the case of probability measures, they are histograms in the simplex ÅK-
Definition 11 (W-distance) Given an input xEX and any f: X let f(x) be the predicted label distribution ft, and let y be the target distribution ps, then the W-distance between zi and p is
w (P.
inf TEL((p,))
(T.C),
(16)
where CERKXK is the cost matrix whose the element can be defined as C (R., R²), and the set of couplings composed of joint probability distributions with their marginals is and p.
i.e, [I. (TE Kx,, 11 1=p]-
Entropic Wasserstein loss. convex problem can be turned to strictly convex problem:
Cuturi (2033) introduces entropic regularization such that the non-
(T.C) -CC--H(T)) H(T).
W a 14) TEL((..))
inf
(17)
where ((T)=-Ex... T
With the help of the entropic regularization, we solve optimal transport problem efficiently using Sinkhorn- Knopp matrix scaling algorithm (Knight 2008).
Proposition 5 The transport matrix T optimizing Problem (17) satisfies T diag u)) K diag(v), ddiag(v)) where K -a- u = c and = eb where a and b are the Lagrange dual variables.
Theorem 8 (Genevay et al. 2017) Wasserstein sinkhorn loss between the predicted label distribution Tú and the target label distribution μs is defined as:
( = 2W2 ( n)) ( W2 (p.p),
(18)
with the following limiting behavior as 0: W u) 2 (p.
Note that normalized Wasserstein loss negatiiee and 2 a (ū,, μa) = Of and only is īi = μ.. the quantification, normalized Wasserstein loss to measure the divergence between distributions.
18

---

Under review as a conference paper at ICLR 2020
E INTERMEDIATE LAYER
We add auxiliary classifiers for the intermediate layers as mentioned below. For ResNet-18, we truncate each block and add classifier after its feature map. We label these intermediate layers from O-th layer to 8-th layer. The truncation of ResNet-50 is analogous to ResNet-18 18. For VGG-16, we add classifier after each convolution layer and label them from 0-th layer to 12-th layer.
313 conv 128, /2
22 2
7x7 cony, 54,
pool f2
3x3 conv. 64
3x3 comy, 64
3x3 Rond &
3x Coon 128
128
Cood 246
S5h
Ut
DXc Cole 512
ave pos
fic 1000
3xc Pon
E33
3x
layer0
layer1
layer2
layer3
layer4
layer5
layer6
layer7
layer8
layer9
Figure 7: Definition of intermediate layer of ResNet- 18.
layer1
layer3
layer5
layer7
layer9
layer11
3x3 corw, 64
comy. 128
3x3 conv, 128
256
256
256
S12
512
512
S12
Image
pook F
popa
3x3 comy
3x3 comn
4
comy
pood
3x3 comy.
3x3 corn
coryy
pool,
3x3 cony
pool, /2
4096
fc 4096
fc 1000
3xd
layer0
layer2
layer4
layer6
layer8
layer10
layer12
layer13
Figure 8: Definition of intermediate layer of VGG-16.
F ACROSS-LAYER DISTRIBUTION PROPAGATION
We show more results for different datasets, measures and networks across different layers.
1.5
1.5
trial
trial
orig.
trial
orig.
W-distance
1.0
0.0
0
1
2
8
0.0
9 10 11 12 13
#layer (ResNet)
#layer (VGG)
Figure 9: Wasserstein distance across different layers of ResNet-13 18 (left) and VGG-16 (right) on different test set. Specifically, we shuffle the VOC2007 dataset (including training set and test set) and divide it into training set and and define them "trial 0" and "trial_"", respectively. Besides, the "orig." means that we use the original dataset. From Figure different experiments consistently have the same decreasing tendency through the depth of neural network.
following the ratio of original dataset. We shuffle twice
19

---

Under review as a conference paper at ICLR 2020
1.5
1.5
W-distance
COCO
1.0
1.0
0.0
0
2 3
4 5
6
10
11
12
13
#layer (ResNet)
#layer (VGG)
Figure 10: Wasserstein distance across different layers of ResNet-18 (left) and VGG-16 (right) on test set. The tendency of distribution propagation on test set is the same as the training set set(Figure suggesting that the distribution propagation is irrelevant to the property of the dataset.
VOC
COCO
VOC
COCO
C-distance
0.75
0.75
0.50
0.50
0.00
0.00
2
10
12
13
#laer (ResNet)
# layer (VGG)
Figure 11: Chebyshev distance across different layers of ResNet-18 (left) and VGG-16 (right) on training set. Although the tendency of Chebyshev distance analogous to Wasserstein distance, Chebyshev distance ignores any metric structure (Frogner et al. 2015). Therefore, we Wasserstein distance to quantify the discrepancy of label distributions.
(S-divergence
1.00
1.00
VOC
COCO
VOC
COCO
0.75
0.75
0.25
0.25
0.00
0.00
0
1
2
3
9
10
11
12
13
#layer (ResNet)
layer (VGG)
Figure 12: Jensen Shannon divergence across different layers of ResNet-18 (left) and VGG-16 (right)
on training set. The conclusion of MaasennShannon divergence is the same as the Chebyshev distance.
W-distance
VOC
COCO
VOC
COCO
1.0
1.0
0.5
0.5
0.0
0.0
14 16
layer (ResNet)
# layer (ResNet)
Figure 13: Wasserstein distance across different layers of ResNet-50 training set (left) and test set (right). We get the same conclusion on ResNet-50 that Wasserstein distance between the distribution of any layer and the target distribution tends to decreases along the depth of DNN.
1.00
dsvvierreence
VOC
COCO
VOC
COCO
C-distance
0.75
0.75
0.50
0.50
0.00
2
3
10
12
1617
0.00
10
1617
layer (ResNet)
layer (ResNet)
Figure 14: Chebyshev distance (left) and divergence (right) across different layers of ResNet-50 on training set.
20

---

Under review as a conference paper at ICLR 2020
G SINGLE-LAYER DISTRIBUTION PROPAGATION
We present more results about the single-layer distribution propagation when training DNNs. We calculate the Wasserstein distance between the prediction distribution of selected epoch and the next epoch + 1 and show in Figure T3 Analogous to Figure Figure 16 shows that the label distribution one sample propagates from the first epoch to the last epoch.
ResNet (
W-dirtance
W- W-distance
VGG (VOC 0
layer
1.2
0.8
0.0
0.4
0.0
30
70
100
0.0
50
80
epoch
# epoch
Figure 15: Wasserstein distance between distributions of adjacent training epoch.
VGG ResNet
(COCO) (COCO) (VOC) (VOC)
10
20
20
15
15
15
10
VGG ReaNet
0.5
opoche:
002ch=30
scochw72
spachmic0
targe:
0.5
00:01=30
scodbw7:
spochm220
20
60
20
00
80
40
80
# label
# label
label
label
label
Figure 16: Distribution propagation across different training epochs of ResNet-18 18 and VGG-16.
21

---

Under review as a conference paper at ICLR 2020
H EARLY-EXITS STRATEGY
Deep neural networks often occur the over-thinking issue, that is, the samples are correctly classified in the intermediate layer but misclassified in the last layer. We call these samples as confused samples. From Table and the accuracy of ResNet-18 (VOC) and VGG-16 (VOC) are 64.48% and 68, 68.96%, respectively. In other words, out of 35.52% and 31.04% of the samples for ResNet- 18 and VGG-16 are misclassified on the test set of VOC2007, respectively. For these misclassified samples, 1.17% are actually correctly classified at 0-th layer, 0.44% are 1-th layer and so on. Ideally, we early exit these confused samples, the cumulative accuracy of ResNet- 18 is 70.44% on VOC dataset.
Table Over-thinking results ResNet-18.
layer
0
9 Ideal result
VOC
1.17%
0.44%
0.69%
0.71%
0.55%
0.50%
0.53%
0.85% 0.78%
0.53%
64.48%
70.44%
COCO
0.49%
0.43%
0.47%
0.28%
0.30%
0.31 %
0.37%
0.50%
27.33%
31.26%
Table 3: Over-thinking results of VGG-16.
layer
VOC
2.34%
0.00%
0.71%
0.67%
0.30%
0.24%
0.48%
0.22%
0.20%
0.14%
0.28%
0.34%
1.22%
68 9698
76.72%
00CO
0.15%
0.35%
0.27%
0.38%
0.33%
0.19%
0.21%
0.17%
0.18%
0.13%
0.38%
0.37%
1.3%
32.41%
37.35%
Early-exits strategy. We propose simple probability mechanism to exit confused samples. Specifi- cally, we add auxiliary classifier for selected intermediate layer get the prediction probability distribution. Each auxiliary classifier contains fully connected layer and Sigmoid function. For each sample, we denote the number of exceeding the threshold p 0.5 as N. We also denote the number of exceeding another threshold as n. We denote the ratio as and 7 n/N. The threshold and ratio y are range from 0.5 1. We search for the best values of them improve classification performance and show the results on Table
Table 4: Improve performance of ResNet and VGG.
VOC
COCO
Method
accuracy
CFI
OF1
mAP
accuracy
CF.
OF1
mAP
ResNet-18
64.48
58.02
59.10
85.18
27.33
55.65
60.30
64.36
66.01
58.67
58.76
85,49
30.03
$7.49
61.38
67.28
VGG-16
68.96
58.58
59.71
88.48
32.41
59.37
62.94
70.64
VGG-16+EarlyExits
69.85
59.49
59.94
88.57
33.95
60.32
63.73
ResNet-50
69.79
60.93
60.23
88.84
33.81
61.25
64.01
71.77
ResNet-500+Kariyiiitt
70.98
62.84
60.21
89.14
35.70
62.39
64.76
73.68
22

---

Under review as a conference paper at ICLR 2020
I MORE QUANTITATIVE RESULTS
We demonstrate more results on easy, hard and confused samples. For each sample, we show the image in the upper left, distribution corresponding to the selected layer in the upper right and the distribution propagation across different layers in the bottom.
50cl
Garget
sample
sfaed
0.0
bbc
1sbel
W-dslance
1.4
ocarcution
prediction
prediction
prediction
10
w-disance
0.6
0.0
Dyper
larget
target prodktion
largei prod ction
Bertutttnn
prediction
Tinas
c.
0.0
18bel
1.8
label
0.0
(a) easy sample
(b) hard sample
(c) confused sample
Figure 17: More results of easy, hard and confused samples.
23


Reviews:
[{"review_id": "HkxW9dLptr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors intuitively, and then analytically, explain the behavior in the hidden layers of deep convolutional networks and show how the behavior can be used to improve performance by \"early exiting.\"\n\nI give this paper a weak reject. I believe this paper does well by connecting the intuitive explanation with the proofs, and then by confirming their results through experimentation. I also applaud the authors for their rigorous explanation of the hyper-parameters and experimentation methods. However, from what I can tell, there was no cross-fold validation or even repeat trials with different partitioning to see whether the differences in performance were just random perturbations or a consistent effect. The increase in accuracy isn't large enough across experiments to allay my concerns.\n\nI think the authors have some very compelling work here, but the lack of a large difference in accuracy combined with insufficient testing methodology causes me to reject this paper... but only barely. I can be convinced otherwise with a compelling set of arguments.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJe8i8JLqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to compute the distance of distribution of two layers in neural networks by using the label distribution mapping (e.g., Frogner et al., 2015). With the tool, authors could see how individual layers could related each other across-layer (along the depth) and single layer (training epoch). \n\nI believe that the contributions of this paper are week in analyzing individual layers across-layer since there are many extensive studies are conducted on information bottleneck methods with mutual information. I believe that those methods are better to analyze the dynamics of learning even without the additional label distribution mapping.\n\nHowever, authors of this paper presents a way to utilize the label distribution mapping to compare the distance of individual layers when an input image come as shown in Figure 5 which I believe the main contribution of this paper. \n\nThe (somehow artificial and ambiguous) term, label distribution is used several places before it is defined. Even in Section 3, the label distribution mapping is not clearly explained except for the description of FC+softmax. Thus, it would be better to clarify the definition. Also, it is not clear that the label distribution reflect the actual distribution of (nodes or feature maps) in a specific layer. It would be good to spend more space and resources (e.g., image and/or running examples) to explain the definition of label distribution.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lfD3Ahtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper seeks to understand both across-layer and single-layer behavior within neural networks (i.e. layer behavior along the depth of a network, and behavior of a single layer along training epochs). Therefore, they resort to the optimal transport framework to compare predicted and target distributions. Theoretically, they show that the Wasserstein distance between predicted and target distributions is decreasing along the depth and for a single layer, along training iterations. They also give intuition on how this analysis can help the learning process in practice.\n\nThis paper gives an interesting contribution to the in-depth analysis of neural networks. However, some elements remain unclear:\n\n1.\tThe setting of multi-label classification does not really motivate the use of measures.\n2.\tIt is unclear why the use of teacher/student networks are pertinent or necessary.\n3.\tThere is no detail on the regularization strength of the Wasserstein distance, or what p (in definition 1) is chosen either in the experiments or in the theorems.\n4.\tI believe it is understated that all \\tilde{f}_i have the same input and output domains (as well as h=h_i in figure 1a), which is restrictive and should have been made clearer. \n\n- Post rebuttal: I thank the authors for their response. On this basis, I am maintaining my weak reject rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the transfer of representations learned by deep neural networks across various datasets and tasks when the network is pre-trained on some dataset and subsequently fine-tuned on the target dataset. On the theoretical side the authors analyse two-layer fully connected networks. In an extensive empirical evaluation the authors argue that an appropriately pre-trained networks enable better loss landscapes (improved Lipschitzness). Understanding the transferability of representations is an important problem and the reviewers appreciated some aspects of the extensive empirical evaluation and the initial theoretical investigation. However, we feel that the manuscript needs a major revision and that there is not enough empirical evidence to support the stated conclusions. As a result, I will recommend rejecting this paper in the current form. Nevertheless, as the problem is extremely important I encourage the authors to improve the clarity and provide more convincing arguments towards the stated conclusions by addressing the issues raised during the discussion phase.
Decision: Reject