Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a method to compute the distance of distribution of two layers in neural networks by using the label distribution mapping (e.g., Frogner et al., 2015). With the tool, authors could see how individual layers could related each other across-layer (along the depth) and single layer (training epoch). 

I believe that the contributions of this paper are week in analyzing individual layers across-layer since there are many extensive studies are conducted on information bottleneck methods with mutual information. I believe that those methods are better to analyze the dynamics of learning even without the additional label distribution mapping.

However, authors of this paper presents a way to utilize the label distribution mapping to compare the distance of individual layers when an input image come as shown in Figure 5 which I believe the main contribution of this paper. 

The (somehow artificial and ambiguous) term, label distribution is used several places before it is defined. Even in Section 3, the label distribution mapping is not clearly explained except for the description of FC+softmax. Thus, it would be better to clarify the definition. Also, it is not clear that the label distribution reflect the actual distribution of (nodes or feature maps) in a specific layer. It would be good to spend more space and resources (e.g., image and/or running examples) to explain the definition of label distribution.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors intuitively, and then analytically, explain the behavior in the hidden layers of deep convolutional networks and show how the behavior can be used to improve performance by "early exiting."

I give this paper a weak reject. I believe this paper does well by connecting the intuitive explanation with the proofs, and then by confirming their results through experimentation. I also applaud the authors for their rigorous explanation of the hyper-parameters and experimentation methods. However, from what I can tell, there was no cross-fold validation or even repeat trials with different partitioning to see whether the differences in performance were just random perturbations or a consistent effect. The increase in accuracy isn't large enough across experiments to allay my concerns.

I think the authors have some very compelling work here, but the lack of a large difference in accuracy combined with insufficient testing methodology causes me to reject this paper... but only barely. I can be convinced otherwise with a compelling set of arguments.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper seeks to understand both across-layer and single-layer behavior within neural networks (i.e. layer behavior along the depth of a network, and behavior of a single layer along training epochs). Therefore, they resort to the optimal transport framework to compare predicted and target distributions. Theoretically, they show that the Wasserstein distance between predicted and target distributions is decreasing along the depth and for a single layer, along training iterations. They also give intuition on how this analysis can help the learning process in practice.

This paper gives an interesting contribution to the in-depth analysis of neural networks. However, some elements remain unclear:

1.	The setting of multi-label classification does not really motivate the use of measures.
2.	It is unclear why the use of teacher/student networks are pertinent or necessary.
3.	There is no detail on the regularization strength of the Wasserstein distance, or what p (in definition 1) is chosen either in the experiments or in the theorems.
4.	I believe it is understated that all \tilde{f}_i have the same input and output domains (as well as h=h_i in figure 1a), which is restrictive and should have been made clearer. 

- Post rebuttal: I thank the authors for their response. On this basis, I am maintaining my weak reject rating.
review_title: Official Blind Review #3

Meta-review: This paper studies the transfer of representations learned by deep neural networks across various datasets and tasks when the network is pre-trained on some dataset and subsequently fine-tuned on the target dataset. On the theoretical side the authors analyse two-layer fully connected networks. In an extensive empirical evaluation the authors argue that an appropriately pre-trained networks enable better loss landscapes (improved Lipschitzness). Understanding the transferability of representations is an important problem and the reviewers appreciated some aspects of the extensive empirical evaluation and the initial theoretical investigation. However, we feel that the manuscript needs a major revision and that there is not enough empirical evidence to support the stated conclusions. As a result, I will recommend rejecting this paper in the current form. Nevertheless, as the problem is extremely important I encourage the authors to improve the clarity and provide more convincing arguments towards the stated conclusions by addressing the issues raised during the discussion phase.
Decision: Reject