Towards Understanding the Transferability of Deep Representations
Authors: Hong Liu
Mingsheng Long
Jianmin Wang
Michael I. Jordan
Venue:  (2020)
Keywords: Transfer Learning
Fine-tuning
Deep Neural Networks

Abstract: Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can significantly boost generalization and accelerate training if fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort has been devoted to uncovering the reason of transferability in deep feature representations. This paper tries to understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability. We demonstrate that 1) Transferred models tend to find flatter minima, since their weight matrices stay close to the original flat region of pretrained parameters when transferred to a similar target dataset; 2) Transferred representations make the loss landscape more favorable with improved Lipschitzness, which accelerates and stabilizes training substantially. The improvement largely attributes to the fact that the principal component of gradient is suppressed in the pretrained parameters, thus stabilizing the magnitude of gradient in back-propagation. 3) The feasibility of transferability is related to the similarity of both input and label. And a surprising discovery is that the feasibility is also impacted by the training stages in that the transferability first increases during training, and then declines. We further provide a theoretical analysis to verify our observations.

Under review as a conference paper at ICLR 2020
TOWARDS UNDERSTANDING THE TRANSFERABILITY OF DEEP REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks trained wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can significantly boost generalization and accelerate training if fine-tuned Despite its pervasiveness, few effort has been devoted to uncovering the reason of transferability in deep feature representations. This paper tries to understand transferability from the perspectives of improved generalization, optimization and the feasibility transferability. We demonstrate that 1) Transferred models tend find flatter minima, since their weight matrices stay close the original flat region pretrained parameters when transferred to similar target dataset; 2) Transferred representations make the loss landscape more favorable with improved Lipschitz- ness, which accelerates and stabilizes training substantially. The improvement largely attributes the fact that the principal component of gradient is suppressed in the pretrained parameters, thus stabilizing the magnitude of gradient in back- propagation. 3) The feasibility of transferability is related to the similarity of both input and label. And a surprising discovery is that the feasibility is also impacted by the training stages that the transferability first increases during training, and then declines. We further provide theoretical analysis to verify our observations.
smaller target dataset.
INTRODUCTION
The last decade has witnessed the enormous success of deep neural networks in wide range of applications. Deep learning has made unprecedented advances in many rescarch fields, including computer vision, natural language processing, and robotics. Such great achievement largely attributes to several desirable properties of deep neural networks. One of the most prominent properties is the transferability of deep feature representations.
Transferability is basically the desirable phenomenon that deep feature representations learned from one dataset can benefit optimization and generalization on different datasets or even different tasks, e.g. from real images synthesized images, and from image recognition to object detection (Yosinski et al., 2014). This is essentially different from traditional learning techniques and is often regarded as one of the parallels between deep neural networks and human learning mechanisms.
In real-world applications, practitioners harness transferability to overcome various difficulties. Deep networks pretrained on large datasets are in prevalent use as general- general-purpose feature extractors for downstream tasks (Donahue et al., 2014). For small datasets, a standard practice is to fine-tune a model transferred from large-scale dataset such ImageNet (Russakovsky et al., 2015) to avoid over-fitting. For complicated tasks such as object detection, semantic segmentation and landmark localization, ImageNet pretrained networks accelerate training process substantially (Oquab et al., 2014; He al., 2018). In the NLP field, advances in unsupervised pretrained representations have enabled remarkable improvement in downstream tasks (Vaswani et al., 2017; Devlin et al., 2019).
Despite its practical success, few efforts have been devoted to uncovering the underlying mechanism of transferability. Intuitively, deep neural networks are capable of preserving the knowledge learned one dataset after training another similar dataset (Yosinski et al., 2014; Li et al., 2018b; 18;; 2019). This is even true for notably different datasets or apparently different tasks. Another line of works have observed several detailed phenomena in the transfer learning of deep networks (Kirkpatrick

---

Under review as a conference paper at ICLR 2020
Imagetet
Caltech-011
CUB-200
Stanford cars
Figure 1: Examples of object recognition datasets. Caltech- M0ttchh-111- (Fei-Fei al., 2004) is more similar to ImageNet with 101 classes of common objects. CUB-200 (Welinder et al., 2010) and Stanford Cars (Krause al., 2013) essentially different with various kinds of birds and vehicles, respectively.
et al., 2016; Kornblith et al., 2019), yet it remains unclear why and how the transferred representations beneficial to the generalization and optimization perspectives of deep networks.
The present study addresses this important problem from several new perspectives. We first probe into how pretrained knowledge benefits generalization. Results indicate that models fine-tuned target datasets similar to the pretrained dataset tend to stay close to the transferred parameters. In this sense, transferring from similar dataset makes fine-tuned parameters stay in the flat region around the pretrained parameters, leading to flatter minima than training from scratch.
Another key transferability is that transferred features make the optimization landscape significantly improved with better Lipschitzness, which eases optimization. Results show that the landscapes with transferred features are smoother and more predictable, fundamentally stabilizing and accelerating training especially at the early stages of training. This is further enhanced by the proper scaling of gradient in back- prock-poppggtion.... The principal component of gradient is suppressed the transferred weight matrices, controlling the magnitude of gradient and smoothing the loss landscapes.
We also investigate a common concern raised by practitioners: when transfer learning helpful to target tasks? We test the transferability of pretrained networks with varying inputs and labels. Instead of the similarity between pretrained and target inputs, what really matters the similarity between the pretrained and target tasks, i.e. both inputs and labels required to be sufficiently similar. We also investigate the relationship between pretraining epoch and transferabilityy Surprisingly, although accuracy on the pretrained dataset increases throughout training, transferability first increases the beginning and then decreases significantly as pretraining proceeds.
Finally, this paper gives theoretical analysis based two-layer fully connected networks. Theoreti- cal results consistently justify our empirical discoveries. The analysis here also casts light on deeper networks. We believe the mechanism of transferability is the fundamental property of deep neural networks and the in-depth understanding presented here may stimulate further algorithmic advances.
RELATED WORK
There exists extensive literature on transferring pretrained representations to learn an accurate model on target dataset. Donahue et al. (2014) employed brand-new label predictor to classify features extracted by the pre-trained feature extractor different layers of AlexNet (Krizhevsky et al., 2012). Oquab et al. (2014) showed deep features can benefit object detection tasks despite the fact that they are trained for image classification. Ge Yu (2017) introduced a selective joint fine-tuning scheme for improving the performance of deep learning tasks under the scenario of insufficient training data.
The enormous success of the transferability of deep networks in applications stimulates empirical studies on fine-tuning and transferability. Yosinski et al. (2014) observed the transferability of deep feature representations decreases as the discrepancy between pretrained task and target task increases and gets worse in higher layers. Another phenomenon of catastrophic forgetting as discovered by Kirkpatrick et al. (2016) describes the loss of pretrained knowledge when fitting to distant tasks. Huh et al. (2016) delved into the influence of ImageNet pretrained features by pretraining on various
2

---

Under review as a conference paper at ICLR 2020
subsets of the ImageNet dataset. Kornblith et al. (2019) further demonstrated that deep models with better ImageNet pretraining performance can transfer better to target tasks.
As for the techniques used in our analysis, Li et al. (2018a) proposed the impact of the scaling of weight matrices on the visualization of loss landscapes. Santurkar et al. (2018) proposed to measure the variation of loss to demonstrate the stability of loss function. Du et al. (2019) provided powerful framework of analyzing two-layer over werrpparmeerrieed neural networks, with elegant results and no strong assumptions on input distributions, which is flexible for our extensions transfer learning.
3 TRANSFERRED KNOWLEDGE INDUCES BETTER GENERALIZATION
basic observation transferability is that tasks on target datasets more similar to the pretrained dataset have better performance. We delve deeper into this phenomenon by experimenting on variety of target datasets (Figure 1), carried out with two common settings: I) train only the last layer by fixing the pretrained network as the feature extractor and 2) train the whole network by fine- fine-tuning from the pretrained representations. Results in Table clearly demonstrate that, for both settings and for all target datasets, the training error converges nearly zero while the generalization error varies significantly. In particular, a network pretrained on more similar dataset tends to generalize better and converge faster on the target dataset. A natural implication is that the knowledge learned from the pretrained networks can only be preserved to different extents for different target datasets.
Table I: Transferring to different datasets with ImageNet pretrained networks fixed or fine-tuned.
Dataset
training error (fixed)
test error
W
Woll. training error test error
Σ, W (
Wo II
(fixed)
Vn
(fine-tuned) (fine-tuned) vn
Webcam
0.00110
0,45:10.0
0,0971100007
0.001:0
0.45:10.09
0.94
Stanford Cars
0.00=
24.2++0.73
0.0000000033
0.00+
13.95+:0.44
3.70+:0.26
Caltcch- 101
0.00+t0
6.24
0.059++0.005
0.00
4.57+t0.40
1.22++0.21
Synthetic CUB-200
0.03=+0.01
0.81+:019
0.015+:0.003
0.00+10
0.75++0.11
0.65+
0.15.10.03
35.10.1..50
0.262.10.006
0.04
21.04.10.36
2.38.10.33
We substantiate this implication with the following experiments. To analyze to what extent the knowl- edge learned from pretrained dataset is preserved, for the fixing setting, we compute the Frobenius norm of the deviation between fine-tuned weight W and pretrained weight Wo as v W -ulle
where n denotes the number of target examples (for the fine-tuning setting, we compute the sum of deviations in all layers vn Σ, (/ Wuucyll p.). Results are shown in Figure 2. It is surprising that
although accuracy may oscillate, V IW Wollr increases monotonously throughout the training process and eventually converges. Datasets with larger visual similarity to ImageNet have smaller - W Wolf also converges faster. Another observation
deviations from pretrained weight, and V
that
W
Woll is approximately proportional to the generalization error shown in Table
Vn
1/ WWW Welr
CUB-200
W-lr
CUB Webcam
Webcam
Synthetic
Syrthwbic
Stanford Cars
400
Epoch
W
Wollr the last layer.
Wo p in all layers.
Figure 2: The deviation of the weight parameters from the pretrained ones in the transfer process to different target datasets. For all datasets, Ae || -Wollr increases monotonously. More knowledge
can be preserved on target datasets more similar to ImageNet, yielding smaller 1 W Wollr.
v
3

---

Under review as a conference paper at ICLR 2020
Why is preserving pretrained knowledge related to better generalization? From the experiments above, we can observe that models preserving more transferred knowledge (i.e. yielding smaller W-Wlllla generalize better. It is reasonable hypothesize that J W Woll is implicitly bounded in the transfer process, and that the bound is related to the similarity between pretrained and target datasets (We will formally study this conjecture in the theoretical analysis). Intuitively, a neural network attempts to fit the training data by twisting itself from the initialization point. For similar datasets the twist will be mild, with the weight parameters staying closer to the pretrained parameters.
Imagehiet Retrained
Handarey Inilst ood
50
100
325
350
100
129
350
175
(a) Eigenvalues of Hessian.
(b) Randomly initialized.
(c) ImageNet pretrained.
Figure 3: Generalization w.r.t. landscape at convergence on Stanford Cars dataset with ResNet-50. (a) Comparison of the top 20 eigenvalues of the Hessian matrix. ImageNet pretrained networks have smaller eigenvalues, indicating flatter minima. (b) (c) Comparison of landscapes centered at the minima. We use the filter normalization (Li et al., 2018a) to avoid the impact of weight scale. Randomly initialized networks end up with sharper minima. Pretrained networks stay in flat regions.
Such property of staying near the pretrained weight is crucial for understanding the improvement of generalization. Since optimizing deep networks inevitably runs into local minima, a common belief of deep networks is that the optimization trajectories of weight parameters on different datasets will be essentially different, leading to distant local minima. To justify whether this is true, we compare the weight matrices of training from scratch and using ImageNet pretrained representations in Figure Results quite counterintuitive. The local minima of different datasets using ImageNet pretraining are closed to each other, all concentrating around ImageNet pretrained weight. However, the local minima of training from scratch and ImageNet pretraining way distant, even the same dataset.
ImageNet Pretrained
Places Pretrained
Random Inttialized
Figure 4: The t-SNE (van der Maaten & Hinton, 2008) visualization of the weight matrices of the pretrained and randomly initialized networks different datasets. Surprisingly, weight matrices the same dataset may be distant at convergence when using different initializations. On the contrary, for discrepant datasets, the weight matrices stay close to the initialization when using the same pretrained parameters.
This provides with clear picture of how transferred representations improve generalization target datasets. Rich studies have indicated that the properties local minima are directly related to generalization (Keskar al., 2017; Izmailov et al., 2018). Using pretrained representations restricts weight matrices to stay near the pretrained weight. Since the pretrained dataset usually sufficiently large and of high-quality, transferring their representations will lead to flatter minima located in large flat basins. On the contrary, training from scratch may find sharper minima. To observe this, we adopt filter normalization (Li al., 2018a) the visualization tool, and illustrate the loss landscapes around the minima in Figure 3. This observation concurs well with the experiments above. The weight matrices for datasets similar to pretrained ones deviate less from pretrained weights and stay in the flat region. On more different datasets, the weight matrices have to go further from pretrained weights fit the data and may run out of the flat region.
4

---

Under review as a conference paper at ICLR 2020
4 PROPERLY PRETRAINED NETWORKS ENABLE BETTER LOSS LANDSCAPES
A common belief of modern deep networks is the improvement of loss landscapes with techniques such as BatchNorm (loffe & Szegedy, 2015) and residual structures (He al., 2016). Li et al. (2018a); Santurkar et al. (2018) validated this improvement when the model is close to convergence. However, it is often overlooked that loss landscapes can still be messy at the initialization point. To verify this conjecture, we visualize the loss landscapes centered at the initialization point of the 25th layer of ResNet-50 in Figure (Visualizations of the other layers can be found in Appendix B.4.) ImageNet pretrained networks have much smoother landscape than networks trained with random initialization. The improvement of loss landscapes at the initialization point directly gives rise to the acceleration of training. Concretely, transferred features help ameliorate the chaos of loss landscape with improved Lipschitzness in the early stages of training. Thus, gradient- based optimization method can easily escape from the initial region where the loss is very large.
Kandom y Indi magolett Pretrained
100
125
350 375
100
135
350
(a) Faster convergence.
(b) Randomly initialized.
(c) ImageNet pretrained.
Figure 5: Optimization landscape at initialization. (a) A figure of convergence on fine-grained classification task Food-101. The convergence of fine-tuning from pretrained networks is significantly faster than training from scratch. (b) (c) Visualizations of loss landscapes of the 25th layer in ResNet-50 centered at initialization on Stanford Cars dataset. ImageNet pretrained landscape much smoother, indicating better Lipschitzness and predictability of the loss function. Randomly initialized landscape is more bumpy, making optimization unstable and inefficient.
The properties of loss landscapes influence the optimization fundamentally, In randomly initialized networks, going in the direction of gradient may lead large variation in the loss function. On the contrary, ImageNet pretrained features make the geometry of loss landscape much more predictable, and step in gradient direction will lead to mild decrease of loss function. To demonstrate the impact of transferred features on the stability of loss function, we further analyze the variation of loss in the direction of gradient in Figure 6. For each step the training process, we compute the gradient of the loss and measure how the loss changes as we move the weight matrix in that direction. We can clearly observe that in contrast to networks with transferred features, randomly initialized networks have larger variation along the gradient, where a step along the gradient leads to drastic change in the loss. Why can transferred features control the magnitude of gradient and smooth the loss landscape?
magelett
8008
1000
1000
800
5000
1880
11080
11588
4000
1800
1000
1750
$250
1050
18550
11800
1396
(a) Randomly initialized.
(b) ImageNet pretrained.
Figure 6: Variation of the loss in ResNet-50 with ImageNet pretrained weight and random initial- ization. We compare the variation of loss function in the direction of gradient during the training process on CUB-200 dataset. The variation of pretrained networks is substantially smaller than the randomly initialized one, implying a more desirable loss landscape and more stable optimization.
A natural explanation is that transferred weight matrices provide appropriate transform of gradient
5

---

Under review as a conference paper at ICLR 2020
Sancom tribalized
(requerc)
Frequency
X3D
(a) Randomly initialized.
(b) ImageNet pretrained.
Random magaNett
ImageNet Pretrained Random intialization
Projection on singule
Index singular
(c) Projection of weight gradient components.
(d) Scale of gradient in different layers.
Figure 7: The stabilization of gradient by pretrained weights on CUB-200. (a) (b) Distribution of the magnitude of gradient in the 25th layer of ResNet-50. (c) Magnitude of the projection of weight on the singular vectors of gradient. (d) Scaling of the gradient in different layers in back- proppppaaaation.
in each layer and help stabilize its magnitude. Note that in deep neural networks, the gradient w.r.t.
each layer is computed through pack-propagation by ax W&Il
aL
where I denotes oxx
aL
the activation of x; layer k. The weight matrices We function as the scaling factor of gradient Basically, a randomly initialized weight matrix will multiply the magnitude of gradient by its norm. In pretrained weight matrices, situation completely different. To delve into this, we decompose the gradient into singular vectors and measure the projections of weight matrices in these principal directions. Results are shown in Figure 7(c). During pretraining, the singular vectors of the gradient with large singular values are shrunk in the weight matrices. Thus, the magnitude of gradient pook-propagated through a pretrained layer is controlled. In this sense, pretrained weight matrices stabilize the magnitude of gradient especially in lower layers. We visualize the magnitude and scaling of gradient of different layers in ResNet-50 in Figure 7. The gradient of randomly initialized networks grows fast with layer numbers during back-propagation while the gradient of ImageNet pretrained networks remains stable. Note that ResNet-50 already incorporates BatchNorm and coineconnetso improve the gradient flow, and pretrained representations can stabilize the magnitude of gradient substantially even in these modern networks. We complete this analysis by visualizing the change of landscapes during back-propagation in Section B.4.
in
WHEN IS TRANSFER LEARNING FEASIBLE IN DEEP NETWORKS?
Transferring from pretrained representations boosts performance in wide range of applications. However, as discovered by He et al. (2018); Kornblith et al. (2019), there still exist cases when pretrained representations provide no help for target tasks even downgrade test accuracy. Hence, the conditions on which transfer learning is feasible is an important open problem to be explored. In this section, we delve into the feasibility of transfer learning with extensive experiments, while the theoretical perspectives are presented in the next section. We hope our analysis will provide insights into how to adopt transfer learning by practitioners.
5.1 CHOICES OF A PRETRAINED DATASET
As common practice, people choose datasets similar to the target dataset for pretraining. However, how can we determine whether a dataset is sufficiently similar to a target dataset? We verify with experiments that the similarity depends the nature of tasks, i.e. both inputs and labels matter.

---

Under review as a conference paper at ICLR 2020
Iagelet Protreinod
Test ACC. No-9.001 M) -1-000p
WWW
(a) Performance SVHIN
Webcam.
(c) MIT-indoors.
(d) CUB-200.
Figure& 8: Performance by varying either the inputs or labels. (a) Accuracy and deviation of transferring from different datasets MNIST, where v and th denote the convolutional kernels in the first and the second layers, and W denotes the weight of the fully-connected layer. (b) Accuracy on Wehcam with Caltech and Caltech-color pretraining. (c) (d) Test accuracy and convergence of transferring to MIT-indoors and CUB-200 from models pretrained on large-scale ImageNet and Places, respectively.
Varying input with fixed labels. We randomly sample 600 images from the original SVHN dataset, and fine-tune the MNIST pretrained LeNet (LeCun et al., 1998) to this SVHN subset. For comparison, we pretrain other two models on MNIST with images upside down and Fashion-MNIST (Xiao et al., 2017), respectively. Note that for all three pretrained models, the dataset sizes, labels, and the number of images per class are kept exactly the same, and thus the only difference lies in the image pixels themselves. Results shown in Figure 8(a). Compared to training from scratch, MNIST pretrained features improve generalization significantly. Upside-down MNIST shows slightly worse generalization performance than the original one. In contrast, fine-tuning from Fashion-MNIST barely improves generalization. We also compute the deviation from pretrained weight of each layer. The weight matrices and convolutional kernel deviation of Fashion-MNIST pretraining show no improvement over training from scratch. A reasonable implication here is that choosing model pretrained on a more similar dataset in the inputs yields larger performance gain.
Varying labels with fixed input. We train ResNet-50 model Caltech- 101 and then fine-tune it to Webcam (Saenko et al., 2010). We train another ResNet-50 to recognize the color of the upper part of Caltech-100 101 images and fine-tune it to Webcam. Results in Figure 8(b) indicate that the latter one provides no improvement over training on Webcam from scratch, while pretraining on standard Caltech- 101 significantly boosts performance. Models generalizing very well similar images not transferable to the target dataset with totally different labels. These experiments challenge the common perspective of similarity between datasets. The description of similarity using the input (images) themselves is just one point. Another key factor of similarity is the relationship between the nature of tasks (labels). This observation is further line with our theoretical analysis in Section 6.
Similar results are observed on datasets of larger scale for the above two cases (See Figure 8(c) We fine-tune tune deep representations pretrained on ImageNet and Places (Zhou al., 2018) (a large- scale dataset of scene understanding) to CUB-200 and MIT-indoors datasets. As fine- ane-grained object recognition task, CUB-20K benefits more from ImageNet pretrained features. In contrast, MIT-indoors is scene dataset more similar to Places, thus it benefits more from Places pretraining.
Performance in prearaining
Target Performance
Epoch of pretraining
Epoch of pretraining
(a) Accuracy of pretraining.
(b) Accuracy the target dataset.
Figure 9: Transfer performance w.r.t. the pretraining epochs from Food-101 to CUB-200. (a) Accuracy pretraining task and (b) Accuracy on target dataset by fine-tuning from models pretrained for different numbers of epochs. Although accuracy of pretraining increases stably, transfcrability increases only during the early epochs and decreases afterwards.
7

---

Under review as a conference paper at ICLR 2020
5.2 CHOICES OF PRETRAINING EPOCHS
Currently, people usually train a model on ImageNet until it converges and use it as the pretrained parameters. However, the final model do not necessarily have the highest transferability. To see this, we pretrain a ResNet-50 model on Food-101 (Bossard et al., 2014) and transfer to CUB-200, with results shown in Figure 9. During the early epochs, the transferability increases sharply. As we continue pretraining, although the test accuracy on the pretraining dataset continues increasing, the test aceuracy on the target dataset starts to decline, indicating downgraded transferability. Intuitively, during the early epochs, the model learns general knowledge that is informative to many datasets. As training goes on, however, the model starts to fit the specific knowledge of the pretrained dataset and even fit noise. Such dataset-specific knowledge is usually detrimental to the transfer performance. This interesting finding implies a promising direction for improving the de facto pretraining method: Instead of seeking for model with higher accuracy only on the pretraining dataset, more transferable model can be pretrained with appropriate epochs such that the fine- ttning accuracies on a diverse set of target tasks are advantageous. Algorithms for pretraining should take this point into consideration.
6 THEORETICAL ANALYSIS
We have shown through extensive empirical analysis that transferred features exert fundamental impact on generalization and optimization performance, and provided some insights for the feasibility of transfer learning. In this section, we analyze some of our empirical observations from theoretical perspective. We base our analysis on two-layer fully connected networks with ReLU activation and sufficiently many hidden units. Our theoretical results are in line with the experimental findings.
6.1 SETUP
Denote by o(.) the ReLU activation function, a (z) max(z, 0}. I{A} is the indicator function, Le. [{A] if A is and 0 otherwise. [m]] is the set of integers ranging from 1 to m. Consider
a two-layer ReLU network of m hidden units fx.a(x) = vm a o(W x), with x E R2 as input
and W = -(W1,- Wm) E Rdxm as the weight matrix. We are provided with nq samples no
{xq 1 drawn i..dd from the target distribution Q as the target dataset and a weight matrix W (P) pretrained on n samples (xr,,, ) drawn i.i.d. from pretrained distribution P. Suppose | = and |y| S 1. Our goal is transferring the pretrained W(P) to learn an accurate model W (Q) for the target distribution Q. When training the model on the pretraining dataset, we initialize the weight as: (0) ~ ~O, K "I),, ar unif (1)1,1)), where Vr E [ma] and k is constant.
For both pretraining and fine-tuning, the objective function of the model is the squared loss L(W)- (y- - Jw.a (X)). Note that a is fixed throughout training and W is updated with gradient descent. The learning rate is set 17.
fwy-wwaa
We base our analysis on the theoretical framework of Du et al. (2019), since it provides elegant results on convergence of two-layer ReLU networks without strong assumptions on the input distributions, facilitating our extension to the transfer learning scenarios. In our analysis, we use the Gram matrices Hр E R xpp and H E R xn to measure the quality of pretrained input and target input as
H Paj = E No. Xp, 2 O, w XP. 2 2H1 = xp, (T- (xccos(xp,xP.))
2mT
-arccossss Q, (x..)) xQ.i))
Q,y = =o &o, xq. 2 0, w xQ. 2 03} =
2m
To quantify the relationship between pretrained input and target input, we define the following Gram matrix H E Raexing across samples drawn from P and Q:
T (wT xP3 2 O, w xQ. 2 0}] xp, - axccos HPQ. -E- Xq.
2x
Assume Gram matrices H3S and H are invertible with smallest eigenvalue Ap and Ao greater than
zero. HP "'yp
yy characterizes the labeling function of pretrained tasks. yP-Q A PQ H Hp-'yp ' further transforms the pretrained labeling function to the target labels. A critical point in our analysis yo yr-Q. which measures the task similarity between target label and transformed label.
8

---

Under review as a conference paper at ICLR 2020
6.2 IMPROVED LIPSCHITZNESS OF LOSS FUNCTION
To analyze the Lipschitzness of loss function, a reasonable objective is the magnitude of gradient, which is direct manifestation of the Lipschitz constant. We analyze the gradient w.r.. the activations. For the magnitude of gradient w.r.t. the activations, we show that the Lipschitz constant is significantly reduced when the pretrained and target datasets are similar in both inputs and labels.
Theorem (The effect of transferred features on the Lipschitzness of the loss). Denote by X! the activations in the target dataset. For a two-layer networks with sufficiently large number of
hidden unit defined in Section 6.1, if m 2 poly(np, no, 0 1 Ao KK k)
AZ5
with probability no less than 1 - over the random initialization,
ÖT (L())()) 1 = AT. (T())
0x 12 -----o4 + (yQ - -r--) Tyr - -vr-Q)
0x
poly(np, nq 5- xpppp', ,1=1)
(1)
+0
m
Xz6
This provides us with theoretical explanation of experimental results in Section 4. The control of Lipschitz constant relies the similarity between tasks in both input and labels. If the original target label is similar to the label transformed from the pretrained label, i.e. llyo yr- ll is small, the Lipschitzness of loss function will be significantly improved. On the contrary, if the pretrained and target tasks are completely different, the transformed label will be discrepant with target label, resulting larger Lipschitz constant of the loss function and worse landscape in the fine-tuned model.
6.3 IMPROVED GENERALIZATION
Recall in Section 3 that we have investigated the weight change [[()) W(P)| during training and point out the role it plays in understanding the generalization. this section, we show that [[(()- W(P) p can be bounded with terms depicting the similarity between pretrained and target tasks. Note that the Rademacher complexity of the function class is bounded with [[(Q)) (P))W((P) F as shown in the seminal work (Arora et al., 2019), thus the generalization error is directly related to [[W - (W) We still use the Gram matrices defined in Section 6.
Theorem 2 (The effect of transferred features the generalization error). For two-layer
A
networks with m 2 poly no, nn, 5- App AoO K )) K x=
with probability no less
than 1 8 over the random initialization,
[W(() - WP)) Vivo= - QP
Hx- (yo - -V
poly(np, no, nq. 8-1 :xp' Ap',n')
(2)
+0
ApdQot
m
This result is directly related to the generalization error and casts light on experiments in Section 5.1. Note that when training on the target dataset from scratch, the upper bound of is H -lyo-lyo. to target label, the generalization error of the function class is hopefully reduced. On the contrary, features pretrained on discrepant tasks do not transfer to classification task in spite of similar images since they have disparate labeling functions. Another example is fine-tuning to Food-101 as in the experiment of Kornblith et al. (2019). Since it is a fine- fine-grained dataset with many similar images, H will be more singular than common tasks, resulting a larger deviation from the pretrained weight. Hence even transferring from ImageNet, the performance on Food-10 101 is still far from satisfactory.
r "yo By fine-tuning from a similar pretrained dataset where the transformed label is close
7 CONCLUSION: BEHIND TRANSFERABILITY
Why are deep representations pretrained from modern neural networks generally transferable to novel tasks? When is transfer learning feasible enough to consistently improve the target task performance?

---

Under review as a conference paper at ICLR 2020
These are the key questions in the way of understanding modern neural networks and applying them to variety of real tasks. This paper performs the first in-depth analysis the transferability of deep representations from both empirical and theoretical perspectives. The results reveal that pretrained representations will improve both generalization and optimization performance of a target network provided that the pretrained and target datasets are sufficiently similar in both input and labels. With this paper, we show that transfer learning, as an initialization technique of neural networks, exerts implicit regularization to restrict the networks from escaping the flat region of pretrained landscape.
REFERENCES
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 322-332, 2019.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101 minnngg discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Jacob Devlin, Ming-West Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume / (Long and Short Papers), pp. 4171-4186, 2019.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31st International Conference on Machine Learning, volume 32, pp. 647- 655, 2014.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes pee-pparamttrrized neural networks. In International Conference on Learning Representations, 2019.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference Computer Vision and Pattern Recognition Workshop, pp. 178 178, 2004.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1086-1095, 2017.
Kaiming. He, Xiangyu. Zhang, Shaoqing. Ren, and Jian. Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Ross B. Girshick, and Piotr Dollár. 0/5/1811.08888 2018
Rethinking imagenet pre-training. arxiv,
Mi-Young Hub, Pulkit Agrawal, and Alexei A. Efros. What makes imagenet good for transfer learning? arxiv, abs/1608.00 08614, 2016.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, pp. 448-456, 2015.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Uncertainty in Artificial Intelligence (UAI). 2018
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On batch training for deep learning: Generalization gap and sharp minima. 2017.
10

---

Under review as a conference paper at ICLR 2020
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John. Quan, Tiago Ramalho, Agnieszka Grabska-Barwinskk. Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Science USA, 1 114( (14(13)3551-3352 2016.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2661-2671, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3), Sydney, Australia, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances Neural Information Processing Systems 25, pp. 1097-1105. 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2288-2224, 1998.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems 31, pp. 3389-6399 2018a.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. DELTA: deep learning transfer using feature map with attention for convolutional networks. International Conference on Learning Representations, 2019.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 2825-2834, 2018b.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivie. Learning and transferring mid-level image representations using convolutional neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 113()): 211-252, 2015.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision, pp. 213-226, 2010.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Advances in Neural Information Processing Systems 31, pp. 2483-2493. 2018.
Laurens J.P. van der Maaten and Geoffrey E. Hinton. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9(2):2579-266) 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp. 5998-5999990 0008. 2017.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, Califoria Institute of Technology, 2010.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmin mnist: a novel image dataset for benchmarking machine learning algorithms. arxiv, htt//708..07477 2017.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. In Advances Neural Information Processing Systems 31, pp. 4959. 2018.
11

---

Under review as a conference paper at ICLR 2020
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 27, pp. 3320-3328 3328. 2014.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Antonio Oliva, and Aude Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452- 2018.
12

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
In this section, we provide details of the architectures, setup, methods of visualizations in our analysis. The codes and visualizations are attached with the submission and will be made available online.
Models. We implement all models on PyTorch with 2080Ti GPUs. For object recognition and scene recognition tasks, we use standard ResNet-50 from torchvision. ImageNet pretrained models can be found in torchvision, and Places pretrained models are provided by Zhou al. (2018). During fine-tuning we use a batch size of 32 and set the initial learning rate to 0.01 with 0.9 momentum following the protocol of (Li al., 2018b). For fine-tuning, we train the model for 200 epochs. We decay the learning rate by 0.1 with the time of decay set by cross validation. In Figure 2(a) where the pretrained ResNet-50 functions as feature extractor, the downstream classifier is a two-layer ReLU network with Batch-Norm and Leaky-Rel.U non-linearity. The number of hidden unit is 512. For this task, the backbone ResNet-50 is fixed, with the downstream two-layer classifier trained with momentum SGD. The learning rate is set to 0.01 with 0.9 momentum, and remains constant throughout training.
For digit recognition tasks, we use LeNet (LeCun et al., 1998). The learning rate is also set to 0.01, with 5 x 10 weight decay. The batch-size is set to 64. We train the model for 100 epochs.
Fine-tuning. We follow the protocol of fine-tuning as in the previous paragraphs. In Tables and 2, we run all the experiments for 3 times and report their mean and variance. For Table 2, the improvement of fine-tuing is calculated with the generalization error of fine-tuning divided by the generalization error of training from scratch.
Visualization of loss landscapes. We use techniques similar to filter normalization to provide an accurate analysis of loss landscapes (Li et al., 2018a). Note that ReLU networks are invariant to the scaling of weight parameters. To remove this scaling effect, the direction used in visualization should be normalized in a filter-wise way. Concretely, the axes of each landscape figure are two random Gaussian orthogonal vectors normalized by the scale of each filter in the convolutional layers. Concretely, suppose the parameter of the center point is 0. 0, denotes the j-th filter of the i-th layer. Suppose the two unit orthogonal vectors
and 6. Then with filter normalization. by. |0ta For each point (i.e. pixel) (p, q) in the plot, the value is evaluated with g(p, g) = L(f(0 + nppa + qb))). where L denotes the loss function, f denotes the neural networks. 77 is a parameter to control the scale of the plot. In all visualization images of ResNet- 50, the resolution is 200 x 200, i.e.
aijj f Taisyll ||,r]| and b + Tb..l
98, 99 and = 100. -99, ... 98, 99. For additional details of filter normalization, please refer to (Li et al., 201 8a). 77 set to 0.001, which is of the same order as 10 times the step size in training. This is reasonable scale if we want to study the local loss landscape of model using SGD. For fair comparison between the pretrained landscapes and randomly initialized landscapes, the scale of loss variation in each plot is exactly the same. The difference of loss value between each contour is 0.05. When we compute the loss landscape of one layer, the parameters of other layers are fixed. The gradient is computed based on 256 fixed samples since the gradient w.r.t. full dataset requires too much computation. Figure 3 and Figure 10 are centered the final weight parameters, while others are centered at the initialization point to show the situation when training just starts. We visualize the loss landscapes on CUB-200 Stanford Cars and Food-101 for multiple times and reach consistent conclusions. But due to space limitation, we only show the results on one dataset for each experiment in the main paper. Other results are deferred to Section B.
- 100, 999
Computing the eigenvalue of Hessian. We compute the eigenvalues of Hessian with Hessian- vector product and power methods based on the autograd of PyTorch. A similar implementation is provided by Yao et al. (2018). We only list top 20 eigenvalues in limited space.
t-SNE embedding of model parameters. We put the weight matrices of ResNet-50 in one vector as input. For faster computation, we pre-compute the distance between parameters of every two models with PyTone Torch and then use the distance matrix to compute the t-SNE embedding with scikit- learn. Note that we use the same ImageNet model from torchvision and the same Places model from Zhou et al. (2018) 8) for fine-tuning.
13

---

Under review as a conference paper at ICLR 2020
Variation of loss function in the direction of gradient. Based on the original trajectory of training, we take steps in the direction of gradient from parameters at different steps during training to calculate the maximum changes of loss in that direction. The step size is set the size of gradient. We take 100 steps from the original trajectories to measure the local property of loss landscapes. We aim to quantify the stability of loss functions and directly show the magnitude of gradient with this experiments different datasets. Results on CUB-200 are provided in the main paper, with additional results further provided in Section Not that this experiment is inspired by Santurkar et al. (2018), We use the similar protocol as Section 3.2 in Santurkar et al. (2018), 8). Another protocol is to fix the step size along the gradient and compute the maximum variation of loss. Results on Stanford Cars with this protocol are provided in Section B.2. Results for both scenarios are similar.
B ADDITIONAL EXPERIMENTAL RESULTS
B.1
VISUALIZATION OF LANDSCAPES AROUND CONVERGENCE FOR DIFFERENT DATASETS
175
175
150
150
125
125
100
25
100
125
150
175
50
75
100
125
150
175
(a) Randomly initialized.
(b) ImageNet pretrained.
Figure 10: Comparison of landscapes centered at the minima fo1 datasets with ResNet-50. We use the filter normalization (Li et al., 2a) to avoid the impact of weight scale. Randomly initialized networks end up with sharper minima, while pretrained networks stay in flat regions.
B.2 VARIATION OF LOSS FUNCTION IN THE DIRECTION OF GRADIENT WITH FIXED STEP SIZE
Imageker pretroined
Veristion of 005
Sandom Infoliteed
4000
17880
12000
2200
400
5016
(a) Randomly initialized.
(b) ImageNet pretrained.
Figure 11: Variation of the loss in ResNet-50 with ImageNet pretrained weight and random initial- ization. We compare the variation of loss function in the direction of gradient during the training process on Stanford Cars dataset. The variation of pretrained networks substantially smaller than the randomly initialized one, implying a more desirable loss landscape and more stable optimization.
B.3 COMPARISON OF TRAINING FROM SCRATCH
To validate that the generalization error is indeed improved with pretraining, for each dataset, list the generalization error and the norm of deviation from the pretrained parameters in Table The decreased percentage is calculated by dividing the error reduced in fine-tuning with the error of training from seratch. Compared to the results of fine-tuning, we observe that ImageNet pretraining improves the generalization performance of general coarse-grann grained classification tasks significantly,
14

---

Under review as a conference paper at ICLR 2020
yet the performance boost is smaller for fine-graincd tasks which dissimilar in the sense of task with ImageNet. Note that, although Stanford Cars and CUB-200 are visually similar to ImageNet, what really matters is the similarity between the nature of tasks, i.c. both images and labels matter.
Table 2: Performance of training from scratch.
Dataset
lest
line-tuning improvement
Σ [@o)
Woosll
Webcam
26.39-+00:4
98.29%
3.54+0..3
Stanford Cars
46.25-+0.76
48.75%
5.95++0.3
Caltech- 101
22.14:10.63
79.36%
2.34.10.34
CUB-200
52.70.10.77
60.15%
3.90110.61
B.4 COMPARISON OF THE LOSS LANDSCAPES IN EACH LAYER
We visualize the loss landscape of 25-48th layers in ResNet-50 on Food-101 dataset. We compare the landscapes centered at the initialization point of randomly initialized and ImageNet pretrained networks- see Figure and Figure 13. Results are in line with our observations of the magnitude of gradient in Figure 7. At higher layers, the landscapes of random initialization and ImageNet pretraining are similar. However, as the gradient is tack-propagated through lower layers, the landscapes of pretrained networks remain smooth as the higher layers. In sharp contrast. the landscapes of randomly initialized networks worsen through the lower layers, indicating that the magnitude of gradient is substantially worsened in back- propppropaattion.
15

---

Under review as a conference paper at ICLR 2020
(a) layer3.0.conv
(b) layer3.0.conv2
(c) layer3.0.comv.
(d) layer3.1.conv
10
60
(e) layer3.1.conv.
(1) layer3.1.comv.
(g) layer3.2.comy
(h) 1yyer3.2.comp
1ayer3.2.conv3
layer3.3.convl
layer3.3.comv2
(I) layer3.3.conv.
(m) 1ayer3.4.convl
(n) layer3.4.conv2
(o) layer3.4.c 3.oovv
(p) 1ayer3.5.com 1.covvl
30
(q) layer3.5.com'
(x) layer3. 3.covvvv
(s) layer4.0.conv
(1) 0.0.co
104
1n
(u) layer4.0.conv3
(v) layer4.1.com/ L.covv
(w) 1ayer4.1.conv2
(x) layer4.1.comv
Figure 12: Landscapes centered at the initialization point of each layer in ResNet-50 using ImageNet pretrained weight. The smoothness of landscapes each layer are nearly identical, indicating proper scaling of gradient.
16

---

Under review as a conference paper at ICLR 2020
00
(a) layer3.0.conv
(b) layer3.0.conv.
(c) layer3.0.comv3
(d) layer3.1.comv
10
00
(e) layer3.1.conv.
(1) layer3.1.comg
(g) layer3.2.comy
(h) layer3.2.00 2
174
1ayer3.2.conv3
(j) layer3.3.conv
layer3.3.conv2
(I) layer3.3.conv3
20
(m) 1ayer3.4.conv1
(n) layer3.4.comy
(o) layer3.4.com 4.conv3
(p) 1ayer3.5.com
10
(q) layer3.5.conv2
(x) layer3.5 3.conv
(s) layer4.0.comv
(1) Jayer4.0.come
m
(u) layer4.0.conv3
(v) layer4.1.convl
(w) layer4.1.conv2
(x) layer4.1.comv.
Figure 13: Landscapes centered at the initialization point of each layer in ResNet-50 initialized randomly. At the higher layers, the landscapes tend to be smooth. However, as the gradient is propagated to lower layers, the landscapes are becoming full of ridges and trenches in spite of the presence of Batch-Norm and skip connections.
17

---

Under review as a conference paper at ICLR 2020
C PROOFS OF THEOREMS IN SECTION 6
To study how transferring pretrained knowledge helps target tasks, we first study the trajectories of weight matrices during pretraining and then analyze its effect Our analysis is based on Du et al. (2019)'s framework for over-parametrized networks. For the weight matrix W. W()) denotes the random initialization. W (k) denotes W at the kth step of pretraining. (P) denotes the pretrained weight matrix after training K steps. Wo (k) denotes the weight matrix after K steps of fine-tuning from (PP). For other terms, the notation each step is similar.
initialization in target tasks.
We first analyze the pretraining process the source datasets based Arora al. (2019). Define a matrix Zp E Rmndxng which is crucial to analyzing the trajectories of the weight matrix during pretraining.
1
Zp
R mdxnp
(3)
Vm
1m_10mXpp1
where TT.; = [I{ w, Xr,j 2 0}. Zk-(k: denotes the matrix corresponding to W pk(k). Note that the
gradient descent is carried out
vec(W + 1)) = vec( (P())) - ())((())(() - yp),
(4)
where vee (+) denotes concatenating column of matrice into single vector. Then in the K iterations of pretraining on the source dataset,
vec (W()) - (0)(
vec (W 1)) (Pc())
K-1 Σ
Zr(())(ur(t)) - yr)
=
k=0
K-1 1)
Σ
k=0
k=
K-1
K-1
K-1 Σ
Σ
yy
- (P) (H
(k)((k))p(k).
(5)
The first term is the primary component in the pretrained matrix, while the second and third terms is small under the over-pate wer-parametrized conditions. Now following Arora et al. (2019), the magnitude of these terms can be bounded with probability no less than 1 -ō,
1Z (k)) =0 0 |el|2 = Σ (000
K-1
K-1
00 1Ppk Apd
n
+0
k=
k=0
I
X
Here we also provide lemmas from Du et al. (2019) which are extensively used later.
Lemma 1. With Ap Ammn (H?) > 0, m St
no and 1j =
Ap
with probability at nz
24
least 1 8 over the random initialization have
Ww (k) (,)() 4VHpllyp up
Vr [m], 2
Vmdp
Lemma 2. If Wi,...., Wm are i.i.d. generated from N0, I), then with probability at least 8, from
the following holds. For any set of weight vectors Wi...., Wm Rdd that satisfy for any r [m]. [w) w, 1, < o62o A R. some small positive constant c, then the matrix E 1Raxn defined by
n²
1
m
H
x
[w. x 2 20, 20
m
satisfies
H -H(0)] Ag and Amiin (H)
18

---

Under review as a conference paper at ICLR 2020
C.I PROOF OF THEOREM 1
Now we start to analyze the influence of pretrained weight on target tasks. 1) We show that during
pretraining. Hig close Zr (Z(0)) (P)) 2) Then we analyze ug (P) -u uQ)()) (0) with the properties of PQ. 3) Standard calculation shows the magnitude gradient relates closely up(P) uQ(
and we are able to find out how is the magnitude of gradient improved.
To start with, we analyze the properties of the matrix HyQ We show that under over-parametrized conditions, HyQ is close to the randomly initialized Gram matrix denote (p (0), ZO))) and (P) to denote (0) (P)
Use Hro (0) to
Lemma 3. With the same condition as lemma 1, with probability no less than 1 - 6,
aimo
H PQ (P) - HiQ |F 5o
(vmApn53/2
(8)
Hrq,u(()) - (rQ,i)(0))
XP XQ. Σ
((PR0)) -
- (((O)))))))
r=1
1 m
+
(9)
m
( (0) x R) + I(I]w,(P) w,(0))l2 R)),
where cy with small c. Since (0) independent of xQ. and |x 1,
/mdr
the distribution of w()) xQ.: and w,( are the same Gaussian. B[[( (-( xQ,1| S R)]
[[u([w,(() S R)] - 2R
V2tk*
2R 8 v 2TK
(PP HPQ,i <
+
(10)
m
Applying Markov's inequality, and noting that Ilyp - u()) = O (nf) no less than 6,
we have with probability
nPRQ
2R
ninQ
HPQ(P) - HPQ(0)]]= <
O
(11)
12TK
m
VmXpk6332
Also note that B[HPQ. (0)] = HTQ By Hoeffding's inequality, we have with probability at least
1 - 8,
HPQ HPQ npnq log(2npnq/0)
(12)
2m
Combining equation 12 and equation 11, we have with probability at least 5,
(Pro(P)- - lr So (
ninnQ
(VmApR53/2
Denote by uP(P), (0) the output on the target dataset using weight matrix W(P) and W respectively, First, we compute the gradient with respect the activations,
aL(W(P))
1
aa(u(()) yo),
(13)
OX
OL (L(W(P) (P))
112 --a - ((Ta(uq(P)) y
0x1
171
= =0lup(0) vellă + (P) uu + (ue(P) -uo(0), (0) yQ).
(14)
19

---

Under review as a conference paper at ICLR 2020
It is obvious from equation 14 that ug uo( should become the focus of our analysis. To calculate up(P) - uo()), we need to sort out how the activations change by initializing the target networks with ((P) instead of W(0).
1
uQ(P) uQ(0)=
(a
vm
(15)
1
(a(w Xo) o(w (0)xa)
For each XQ divide into two sets to quantify the change variation in activations on the target dataset.
S E [mm] (0)
(0),(x))) R), 3, {r E [m]]//([[/] (0), xqxl S R),
(16)
where
For in S. we can estimate the size of S. Note that I (Iw((),,xq..l)) $ R)] For each and r. B[I([ since the distribution of w(0,, is Gaussian with mean 0 and covariance matrix x² Therefore, taking sum over all and m and using Markov inequality, with probability at least 1 over the random initialization have
R 1 AvEplyys--uon
Vmdp
[ = E
E m =1 28
s R) =
E w (0), S R)
V2Ek*
2mmpR 8ymmpllyz up 2TTR
Vr [m], 4k O.
(17)
V2TkApō
Thus, this part of activations is the same for W(O) and W(P) on the target dataset. For each XQ.
m
uQ -u),(00)=
a
(o(W (w, (0)
vm
a (0) xQ.) - 1,, (0)xQ,))
Vm
re(m]
(18)
Σ a, - (0,
Vm
rE3,
- (w) (0)xq.))
rES,
where (0) denotes 1(w, the second and the third term can be bounded with S [w) w,( la since ||x 1. Putting all xQ.; together,
(0)x0,+ [ww The first term is the primary part, while we can show that
uQ(P) - (1(
1
(a (aTo (oT(oo((P)))W) -WP)) + +e)
Vm
(19)
ZQ( ved (W()) - (0) +61 +
ZZ (Zr(0)(H)) +) + +
where and 62 correspond to each of the second term and third term in equation 18. Thus, using lemma and the estimation of 15)1 with probability no less than 8,
nam 1/2 (20)
/nQ Vm S || | (P)
|e'||2 = llei + E2/12 <
w,, (0)) =
(VmA3.0R (
20

---

Under review as a conference paper at ICLR 2020
Now equipped with equation 6, equation 19, equation 20 and lemma 3, are ready calculate exactly how much pretrained wight matrix ((P) help reduce the magnitude of gradient over (()),
aL
1 a (P m
ll2
0x
=
- -luu(0)-<ol2++ + (lue (P) u (0) + (uQ - u2 u2(0) - yo)
= aLo 112 + y H Hp Hp
HPQ
yr 25 ZZ(P)'
Hp Hy
yy
+ ||e'112 + 2E" Zo(P) yr
yr +
+ Hy (p) (z H ) |cl3 + +2c (P)
+2cT Z (p(H) 'yr
+ unQ (P) (0) yy + (0) (P) c
T uQ(0)
+ 2y (Z (p(()))))))) H Qq yp.
(21)
In equation 21, note that ||c||2 ||cl (P)
(r) - HPQ PQ || - (ro H lr. and
(1)(2) are all small values we have estimated above. Therefore, using [|z](0]| I /op and
[Z)(P)lFF S /nq. we can control the magnitude of the perturbation terms under over-parametrized
conditions. Concretely, with probability at least 1 - å over random initialization,
||<'11== = = ( manQ
(22)
mXp88423
1/2 n
noma
e'
yy==0
npp nTQ Xp
Vnp =0
(23)
Vmxppnn
VInAp02k
(1
ninn (vmipo'
no.nQ
<TZq(P)Tc=O)
+
(24)
m3/4 53/3/22
(vnp 1
-ZZp())Zr(0)) ZZ(P) H =
Ap /mdpk63/2
mXpxx283
(25)
nsun?
np
|e[12 0
+0
(26)
1P203
m 12 X
ZQ(P) yp==
n
1/2
Ki
n 1/2
(27)
12.0
11/1x 5/2 vid
n2(0) ( (0) yr "ypp---y=- yr=
npnox
(28)
Apvo
npnok²
nēpno
ue(0) up(D) =
(29)
App3/22
3/2 / P VR83/2
niAQ
u =0
(30)
m1/2 11/2X288/22
5/2 3/2
(PL - H -
typ = 0
up nQ
(31)
Vm
Substituting these estimations into equation 21 completes the proof of Theorem 1.
C.2 PROOF OF THEOREM 2
In this subsection, we analyze the impact of pretrained weight matrix on the generalization perfor- mance. First, we show that model will converge if initialized with pretrained weight matrix. Based on this, we further investigate the trajectories during transfer learning and bound || |W- - ([])] with the relationship between source and target datasets.
21

---

Under review as a conference paper at ICLR 2020
C.2.1 CONVERGENCE TRANSFERRING FROM PRETRAINED REPRESENTATIONS
Similar to Du et al. (2019), the proof is done with induction, but since start from W instead of randomly initialized Wo in the transferring process, we should use the randomness of Wo in an indirect way. Concretely, we should use lemma 1 to bound the difference between each column of (P)) and randomly initialized (()) when proving the induction hypothesis.
Theorem (Convergence of Transfer Learning). Under the same conditions as in Theorem 1, if
we set the number of hidden nodes m 22
Ap (1 Ap
ningg
=0
and the learning n2pm
X
then with probability at least 1 8 over the random initialization we have for k 1,,
rate 17 = =0 ng
nde) [lu - vell3 - (1 (1--222 2 [lug(P) - yall..
(32)
The following lemma a direct corollary of Theorem and lamma 1, and crucial analysis to follow.
Lemma 4, Under the same conditions as Theorem 3, with probability at least ō over the random initialization have Vrr [m]. vk 2 0,
(k) w.(0)]l2 4/Rpllys (0]ll2 4 A - un(P)l22 =0
nang
Vmdp
vmiaQ
Vmipppo02
(33)
We have the estimation of (w.r(()) w, la from lemma 1. From |/ (k) wr <
(k) (P)P)l2 + (w (, we can proove lemma 4 by estimating (Q. (P)
(k) -w,(P)l (L(W(k))
1|
Ow,(k')
yQQ
un (b)
Vm
T
mdq 2
1/12
<1
lyQ
uo(k)]l2
Vm
yQ
un
(P)l22
Vmiq
We also have Ilyq (0)l) = 0 (1: /ng and (P uq( = (Zr +
v8
<) e e2. Substituting lemma 3, equation 6, and equation 22 into |uo (P) (0))))22 completes the proof.
Now we start to prove Theorem 3 by induction.
Condition 1. At the k-th iteration, we have |nll (A: - -ylll 5 (1- 5
xdgce & [lu( - voll.
We have the following corollary condition 1 holds,
Corollary 1. condition holds for k' O.... ... k for every r [m]) with probability at least 8.
(.. - (0)12 s AVupllyp - up(22 AVToollve uP()lll2 & R'.
VmAr
Vmdq
If = 0. by definition Condition holds. Suppose for k' = 0, D.....A k. condition 1 holds and we want to show it still holds for K' = 1. The strategy is similar the proof of convergence
22

---

Under review as a conference paper at ICLR 2020
on training from scratch. By classifying the change of activations into two categories, able to deal with the ReLU networks as a perturbed version of linear regression. We define the event Air = (0) x I Rl,, where R sde for some small positive constant to control the
no
magnitude of perturbation. Similar to the analysis above, we let S; = {r E [m] [(Air} 0} and 5; - [m]] \ S, Since the distribution of (0) is Gaussian, we can bound the value of each Asr and then bound the size of S; just as we have estimated in equation 17 above.
Lemma 5. With probability at least 1 - 6 over the initialization, we have Σ S.1 S some positive constant C > O.
Cmng for
The following analysis is identical to the situation of training from scratch.
uq + 1) (k)
Vm Σ a, (o (wQ, 1)" ()qx)).)= (we.r(k) (k)xi)))
By dividing [m] into St and Si, we have,
I A Σ or (o (wo, + 1) +1)xxx)) - ( xq:))
rES:
1
Σ ar (o (wq. +1) +),xxxx))))))))) - (kQ xq,)). (k)x)))
rE3.
We view I as a perturbation and bound its magnitude. Because ReLU is a I-Lipschitz function and |ar| 1, we have
(l((Wo(k))
S/Sa mm
@L( (1+) (k)
n/Ssinn + (Iu vollz
7 Σ
(wQ..(1)
xQ.15
max rE(m)
112 S
m
rES;
By Corollary 1, we know (k) w, < R' for all r E [m]. Furthermore, with the conditions on m, we have R' < R. Thus I (kQ + 1) xQ 2 0} = [[( xQ. 2 0} for E S...
ng
11
7
x x xei (ues - ()..))
Σ
I H[ (T))
xo2 2 0, wo. (b) xe. 2 0}
m
j=1
reS,
7
xQi (uqs yQ.)) r=1
xxx+ 2 0, (k) xQi 2 20} Σ (k) xQ,
m
nQ
1
xq xQx (uq -VQ.)
Σ ((worr(k) xQ. 2, (k) (xx 2 0}
m
j=1
re
where m Ex xQrll (k) xq. 2 wQ. (x)x 2 0} is just the (i.j)-th entry of discrete version of Gram matrix H defined in Section 6.1 and
(k) xQ.
no
7
x xQs (uq.s - (Q.)
xQ. 2 0, (k) 2 2 If (x)x rEB.
Σ
1=1
7
no
11 n/nQ (S.]l(u((k) - yell2-
<
S
(k) - 3o.l <
For ease of notations, denote by [H) the matrix whose i,j entry is m
XQx (uQs VQ.s) Eres, Il (wee((k) xq,i 2 0, (k) xQ. 2 0}. Therefore, the L2 norm of ||H(A)
(1(k))>12 1|2 is bounded with Crg To bound the quadratic term, we use the same techniques training from scratch.
oL(Wo(k))
Ol ((k)) 112 awQ,-(k) l1)
|u + 1) - (1))1 <
S .inngllo (k)))
23

---

Under review as a conference paper at ICLR 2020
With these estimates at hand, are ready to prove the induction hypothesis.
Q uk + 1)
-lly u²(4) (uo(k: + 1) (
=IlyQ (k)kl 2yQ uu(k)) (uo +1 uo(k:) + Ilu + 1 uk(kk)l
=Il -Ilve - dn (k) 2n (yQ - uk():) (k( (rq - uk(k:)
+2n (ye uk)) Ho(k) (yQ uk(()) 2yy uQ(k)) I2
+ +luug (k + 1) un({))1
3/2 R <(1 - -1120 2Cmma 20mm
+ 1 nrri)llye - (
8
<(1 n7e )lla u.) 2
The third equality we used the decomposition of u(k + 1) u(k). The first inequality we used the Lemma 2, the bound on the step size, the bound on I2, the bound on (H I|2 and the bound on |lu[k + 1l The last inequality we used the bound of the step size and the bound of R. Therefore Condition holds for = k + 1. Now by induction, we prove Theorem 3.
Similar to the analysis of lemma 3, we can show the change of ZQ(k) and HQ(k) is negligible under the conditions of sufficiently large m.
Lemma 6. Under the same conditions as Theorem 3 in the transferring process, with probabilility no less than - 8,
3/2 5/4 Up nQ
[Z)(0) - Zo(k)]l =0
m X7.. A21 16281/2
7/2
n
(0) (+)
This is a direct corollary of lemma 4 and can be proved by the same techniques as lemma 3. Now we can continue analyze the trajectory of transferring W(P) to the target dataset by dividing the activations in to the two categories as in the proof of Theorem 3.
u2 (k: + 1) (k(k) (H) - yy) + cs(k),
(34)
n2
[es((k)]l2
17
x XQS (uq.s yQ.))
(T))))xxx Σ I (kQ xQ,1 2 (k. xQ. 2 20} + +1
m j=1
rEB,
ng
2m 2 TQ
S [k(k)) -yell2.
m
vmnning Vr [m)] Vk 2 0. Then substituting Hk(k) with Hō
where [S
OmagR R
0
13,2088
using the bound on [H - HQ(k)) (k) P. we further have,
ue (I: + 1) u2 (A:) = nH (ua(k) - -y) + cz()) + ((K),
(35)
[((())]l2 Sn1 (HQ(0) HQ (k)]l /u(()))(()))))))))()) - yQllz
En 51(1-11 (1-111 n110 h-1
mPyTQ
7/2
(36)
.
Ap
mm
where the second inequality holds with Theorem 3. Taking over each iteration.
k-1
(k)) = (1-HH))* (uo(P)-vo))
Σ (I
* (c) (37)
1=0
24

---

Under review as a conference paper at ICLR 2020
R-1
[e(k)]l2 =
Σ
n
(5(1: 1-
-
t=0
(38)
h-1
7/2
nPy nq 0
nang
s k 1
0
0
4
Xp
Vmm 2g x3
where we notice max (1- nda the target dataset,
=0
Then in the K iterations of transferring to
vec (Q(()) - vec(W((P)
k-1 ved (o + 1)) vec(W((()))
Zo (k) -yQ)
n
K
K-1
(39)
- (K) (yQ - u2(())
Σ ZZZ
km
1 -
K-1
- - up(())
Σ
())
The first term is the primary part, while the second and the third are considered perturbations and could be controlled using lemma 6 and equation 38.
np 5 3/4
Σ 7 (ZQ(k) 0=0
(40)
since
[Zokk) (0)ll is bounded, the maximum eigenvalue of H is A²¹
K
nZ (1))
K-1
Σ (k)()
Σ
II - ) (C(k - 1- 1
+ cs(k - 11 t))llz
k=0
1=0
nTT@ k (1
ndQ
1-1
mmpmn 7/2
n npvnQQ
0
4
Xp
7/2
nanm 9/2
=nan 0
10
mmpmo
nP nQ
= =0
xx²
Ap
(41)
25

---

Under review as a conference paper at ICLR 2020
where we use Σk=1 k (11 _dee
k-1
0
With these estimations hand, we are ready
7F
to calculate the final results
[Wo())- W(
= =|/voc(WQ(()) - voc( (W)))
K-1
K-
= -rv up(P))
Σ
(I - ))) &T Zo(0) (g()))((())
(0)
Σ
(I - n H2) uQ(P))
0=0
k=0
n
+0
nolog a1/
nnng
lyq - (P)) H (yQ uq(P)) +
+0
m!/4>q
(42)
Using equation 19, we have
yQ un (P)
=yQ (Z)(0)
E2) - u2(0)
yQ HiQ (HP) 'yp (HPQ Zo(0) (p)) yy Zo(0) c' + uz(
With lemma 3, we have
m2no
[HPQQ Zp 20 yplls 50
(43)
Combine the estimation above with equation 22 and equation 26. We also have ||uo(0)]
O
g V8
yQ - uu( = yQ
HJQ (HA)
yp t
(44)
n2n 4
t
nono
n
||e 1 =
+0
+0
+0
+
m N2,
Apo
m
md
A
m 17,1502
V8
ni.nn
/P
=0
+
(
Apō
Substitute equation 44 into equation 42, we have
[Wo(K) -(P))) -yQ - PQ
H
H (yQ
HpQ HH (yp)
+2 da
|eallelye
HPQ
Hp ypllz 12 + \o
||al2
nolog
11/4
npn
+0
m1/1x0
m-/4X
X 2 2/122
(45)
-(yQ
HpQ
H xpp-)yr)
Yr)
H Q
lyo
H Hp
Hp -yp)
nPnoR
n
+0
+0
17,100
m
23222186 x2x²
which completes the proof.
26


Reviews:
[{"review_id": "r1eGgJ6h5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I thank the authors for the clarifications and the modifications to the paper. However, I still lean towards rejection. While the authors provided detailed explanations on some of my points here, most of these are still not in the paper, so the reader would still probably be confused. There are new figures in the appendix, but they are not referred to from the text. The paper feels like a hastily compiled collection of weakly related, somewhat anecdotal and not necessarily clearly explained and motivated experiments. While I believe there is interesting content, with the current presentation style it is really difficult to digest. As an advice, I think it may be better to not squeeze in all the results into one paper, but rather focus on some aspects and analyze them really well, with clear explanation, motivation, and preferably with demonstrating practical gains that result from the analysis - not just hypothetical, but supported by experiments.\n\n---\n\nThe paper studies transfer of representations learned by deep networks across datasets and tasks. Namely, the paper analzes the standard setup with pre-training on one dataset and further fine-tuning on a different  dataset. Results include both experiments and theory. On the experimental side, the focus is mainly on TODO. On the theory side, an analysis of transfer in two-layer fully connected networks, based on the ICML-2019 work of Arora et al., is proposed. \n\nI lean towards rejecting the paper, since the presentation and the technical quality are somewhat substandard. This is mainly based on evaluation of the experimental results, since I am not an expert in this subfield of theory and was therefore not able to check the statements and the proofs thoroughly. On the presentation side, many details are missing, which often makes understanding difficult and will likely lead to the results not being reproducible. On the experiments side, the issue is that they are quite anecdotal and have multiple confusing or questionable details, as specified below.\n\nPros:\n1) An interesting topic of trying to understand generalization and transfer in deep learning\n2) Multiple types of experiments, including visualizations of loss landscapes at convregence and at initialization, plots of Hessian eigenvalues, measuring the deviation of the weights from their initial values, measuring the variance of the gradients of the weights, measuring the transfer between different datasets, measuring the transfer performance depending on the durarion of pre-training.\n3) Theoretical analysis. As mentioned, I cannot really judge about the quality of the results.\n\nCons:\n1) The presentation is quite confusing. \n1a) The paper includes many different experiments as well as theory, and it is not very clear how these all come together and what message do they give to the reader. The paper states at one point that it \"may stimulate further algorithmic advances\", and it would be great if there was a bit more elaboration on this. \n1b) Experimental methodology is not presented in the main paper and not referred to. Some of it is described in the appendix, but also not too detailed, for instance the duration of ResNet training is not clear, the details of loss landscape visualization are confusing (for instance, the phrase \"... i.e. 0.1× gradient and a total of 200 × 200 grids\"), \n1c) The paper is over 9 pages, which is more than the recommended length of 8 pages.\n1d) Scaling in Figure 8(b) is quite suboptimal, it is impossible to read the test accuracy results.\n1e) Minor issues:\n - No color-code in Figure 3 (unclear what values do the colors correspond to) and it does not seem to be referred to in the text. \n - Page 5: \"predictive\" -> \"predictable\"?\n - Page 6 \"While pretraining on sufficiently large datasets...\" - I do not think the experiments tell anything about the dependence of the effect on the size of the dataset, so this phrasing is not justified\n\n2) I found many of the experiments confusing or unconvincing. This is partially affected by the aforementioned issues with presentation.\n2a) In Table 1 and Figure 2, it is unclear if difference in generalization between the dataesets is due to similarity to ImageNet (by the way, ImageNet is only mentioned in the caption of Table 1, but not in the text) or due to the inherent properties of the datasets (perhaps some are naturally more difficult or prone to overfitting). I think numbers for training from scratch would be helpful for disambiguating the two cases.\n2b) It is unclear why is the norm difference normalized by the square root of the number of target examples. This choice is not justified and it seems it can very noticeably affect the results in counter-intuitive way. For instance, if I understand right, if one duplicates each example in the training set, these values will change, which seems somewhat counter-intuitive. Would it make more sense to normalize by the initial norm of the weights?\n2c) In Figure 4, it is unclear if the permutations of the weights are accounted for. The output of a deep network is invariant under a wide variety of weight permutations, and it is natural that networks trained from different random initializations may converge to weights permuted in different ways. In order to meaningfully compare the weight vectors of these networks, one would have to first \"align\" them, which is a non-trivial task. Another issue I have with the results on Figure 4 is that I don't find them all that counter-intuitive: it seems natural that the weights stay relatively close to the pre-trained network when fine-tuned for that (paritally because the aforementioned permutation symmetry is already largely broken during pre-trianing).\n2d) It is unclear which downstream task is analyzed in Figure 6. Moreover, the plot seems to mix two factors: magnitue of the gradient and the smoothness of the loss landscape. Would it not make more sense to make fixed-size steps in the direction of the gradient? Moreover, I am wondering if the difference between the loss values is simply because the overall loss magnitude is larger for the randomly initialized network?\n2e) A few parts of subsection 5.1 are confusing. It is not clear where does the interpretation of Figure 8 follow from (perhaps partially because figure 8(b) is partially difficult to read). \\psi and W are not defined. What does it move that the weight matrices \"show no improvement\"? Where are the results on Caltech-101 and Webcam? Why is Food-101 mentioned in the text, but CUB-200 shown in the plots?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "H1gSNMbatr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper gives an extensive empirical and somewhat limited theoretical analysis for the transferability in DNNs. It is shown that transferred models tend to have flatter minima with improved Lipschitzness in the loss function when good choices of pretraining are made.\n\n- The paper is well written and well-organized. Notations and claims are clear.\n\n- This paper presents an interesting line of research, that in my opinion, would be interesting to many researchers in the field, and could attract many follow up works. \n\n- Empirical analysis in sections 3 and 4 are interesting, and give good sense of generalization and optimization landscape. Analyzing the Frobenius norm of the deviation between fine-tuned network and fixed network is reasonable. \n\n- The theoretical analysis seems like a good start, but it is not sufficient in general. There seems to be gap between the network architectures used in empirical evaluations and the theoretical results. However, analyzing transferability is an important topic that needs to be evaluated more. This paper presents interesting new steps towards that goal. That being said, I would be interested to see theoretical results for more general cases alongside experiments on different types of applications.\n\n- Overall, the paper presents a novel approach for evaluating transferability, that I think would be interesting to many researchers in this field. The theoretical results are still limited, and should be investigated more. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkx40G0oFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "N.B Since this paper exceeds 8 pages, it was held to a higher standard as per the instructions. \n\nThe main goal of this paper is to concretely demonstrate some key properties of transfer in Deep Neural Networks through experiment and theory. By transfer, here they specifically mean the ability for pre-trained networks to be 'updated' to new, similar datasets either completely (all parameters are updated while being initialised by the pre-trained network parameters) or partially (all but the last few layers are kept constant at pre-trained parameter values). There are of course other ways of carrying out transfer learning, but this paper focusses on these methods.They attempt to assess the viability of such a process in its improvements to generalisation and improvements to the loss landscape. In addition, the authors attempt to assess when this type of transfer is viable. The majority of the paper focus on experimental results, while the final 2 pages present some theoretical work that explains those results. \n\nI believe that this work is well motivated. As the paper suggests, there have been several advances in the use of transfer learning that showcase its benefits. That being said, there is a lack of work that systematically tries to explain why these benefits are seen, and how we can better make use of them. This paper tries to fill that gap. \n\nHowever, while it goes a way in trying to do so, I am not convinced that this paper sufficiently addresses what it sets out to do. This is why I recommend a weak reject, and a summary of my reasons for this are as follows:\n\n1) Section 3: In this section, the authors try to show that transferred networks tend to have better generalisation when the dataset being transferred to is similar to that the network was pre-trained on (ImageNet dataset). The results are shown in Table 1, and they assess this by showing that for new datasets that are more visually similar to the ImageNet dataset, the generalisation error is lower (e.g the Webcam dataset shows lower gen. error than the Stanford Car dataset). I believe that the authors are saying that they are 'visually' similar. However, the ImageNet dataset has subsets that are similar to the Webcam dataset (geological formation), and the Cars dataset (wheeled vehicle). As such, while the goal of the experiment is interesting, it is not clear how interpretable the results are, nor the validity of the conclusions raised. \n\nIn addition to this, I would have liked to see the generalisation error of a randomly initialised network for each of the datasets. This would have been an interesting control to see whether the pre-training does indeed improve generalisation. \n\nFurther, the authors use the Frobenius norm between the original pre-trained parameters and the final parameters as a measure of how much knowledge is preserved. I am not convinced that this is a sufficiently representative measure of this. I think the extent to which knowledge is preserved is indicated by how well the new, transferred network performs at old tasks from the original dataset. Simply measuring a distance between the parameters doesn't show this. Also consider the fact that it isn't true that networks with parameters a fixed distance away from the original parameters will have similar behaviour. \n\nFigure 3 isn't mentioned anywhere in the text!\n\n2) Section 4: Here, the authors show that pretrained networks lead to flatter, smoother loss landscapes when compared to randomly initialising. This is shown in Figure 5 and Figure 6 mainly. Figure 5 depicts the loss landscapes, and directly shows what this section is claiming. Figure 6 further solidifies this claim by showing that the change in loss at each gradient update is smaller when compared to a randomly initialised network. That being said, the experimental details of this setup is quite sketchy; what dataset is Figure 5 and 6 transferring to, having been pre-trained with ImageNet? Has this been tested between multiple different datasets, multiple times, to show that the conclusions are consistent? Further, it looks like Figure 5a was taken from another paper, why?\n\n3) Section 5: This section tries to answer the question of when transfer learning (as defined by this paper) is viable. Section 5.1 was quite difficult for me to read because I could not understand how the experimental setup described in the text. For example, Section 5.1 says that the network was pretrained on the MNIST dataset and transferred to the SVHN dataset, whereas Figure 8b states the complete opposite. If I assume that the text is correct, the generalisation error in Figure 8b is very difficult to read. In addition to this, I am not sure what the norm of phi-phi(0) is. I also still have my reservations to the use of the norm between parameters, as mentioned above. \n\nIn the the section 'Varying Labels with fixed input', the authors mention the use of Caltech-101 and Webcam data, but this isn't mentioned in Figure 8, instead, it mentioned CUB-200, which isn't mentioned in the text. They also mention conclusions from experiments using the Food-101 and Places datasets, but don't show these results anywhere. \n\nSection 5.1 asks important questions, but the authors haven't shown results that can properly answer them. \n\nThat being said, Section 5.2 shows the very interesting result that pre-training after a certain number of epochs starts showing diminishing returns in terms of performance of the transferred network. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the transfer of representations learned by deep neural networks across various datasets and tasks when the network is pre-trained on some dataset and subsequently fine-tuned on the target dataset. The authors theoretically analyse two-layer fully connected networks and provide an extensive empirical evaluation arguing that the loss landscape of  appropriately pre-trained networks is easier to optimise (improved Lipschitzness). 
Understanding the transferability of representations is an important problem and the reviewers appreciated some aspects of the extensive empirical evaluation and the initial theoretical investigation. However, we feel that the manuscript needs a major revision and that there is not enough empirical evidence to support the stated conclusions. As a result, I will recommend rejecting this paper in the current form. 
Nevertheless, as the problem is extremely important I encourage the authors to improve the clarity and provide more convincing arguments towards the stated conclusions by addressing the issues raised during the discussion phase.
Decision: Reject