Title: TOWARDS UNDERSTANDING THE TRANSFERABILITY OF DEEP REPRESENTATIONS

Abstract: Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can significantly boost generalization and accelerate training if fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort has been devoted to uncovering the reason of transferability in deep feature representations. This paper tries to understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability. We demonstrate that 1) Transferred models tend to find flatter minima, since their weight matrices stay close to the original flat region of pretrained parameters when transferred to a similar target dataset; 2) Transferred representations make the loss landscape more favorable with improved Lipschitzness, which accelerates and stabilizes training substantially. The improvement largely attributes to the fact that the principal component of gradient is suppressed in the pretrained parameters, thus stabilizing the magnitude of gradient in backpropagation.
3) The feasibility of transferability is related to the similarity of both input and label. And a surprising discovery is that the feasibility is also impacted by the training stages in that the transferability first increases during training, and then declines. We further provide a theoretical analysis to verify our observations.

1. INTRODUCTION:

The last decade has witnessed the enormous success of deep neural networks in a wide range of applications. Deep learning has made unprecedented advances in many research fields, including computer vision, natural language processing, and robotics. Such great achievement largely attributes to several desirable properties of deep neural networks. One of the most prominent properties is the transferability of deep feature representations.

Transferability is basically the desirable phenomenon that deep feature representations learned from one dataset can benefit optimization and generalization on different datasets or even different tasks, e.g. from real images to synthesized images, and from image recognition to object detection (Yosinski et al., 2014) . This is essentially different from traditional learning techniques and is often regarded as one of the parallels between deep neural networks and human learning mechanisms.

In real-world applications, practitioners harness transferability to overcome various difficulties. Deep networks pretrained on large datasets are in prevalent use as general-purpose feature extractors for downstream tasks (Donahue et al., 2014) . For small datasets, a standard practice is to fine-tune a model transferred from large-scale dataset such as ImageNet (Russakovsky et al., 2015) to avoid over-fitting. For complicated tasks such as object detection, semantic segmentation and landmark localization, ImageNet pretrained networks accelerate training process substantially (Oquab et al., 2014; He et al., 2018) . In the NLP field, advances in unsupervised pretrained representations have enabled remarkable improvement in downstream tasks (Vaswani et al., 2017; Devlin et al., 2019) .

Despite its practical success, few efforts have been devoted to uncovering the underlying mechanism of transferability. Intuitively, deep neural networks are capable of preserving the knowledge learned on one dataset after training on another similar dataset (Yosinski et al., 2014; Li et al., 2018b; 2019) . This is even true for notably different datasets or apparently different tasks. Another line of works have observed several detailed phenomena in the transfer learning of deep networks (Kirkpatrick Figure 1 : Examples of object recognition datasets. Caltech-101 (Fei-Fei et al., 2004 ) is more similar to ImageNet with 101 classes of common objects. CUB-200 (Welinder et al., 2010) and Stanford Cars (Krause et al., 2013) are essentially different with various kinds of birds and vehicles, respectively. et al., 2016; Kornblith et al., 2019 ), yet it remains unclear why and how the transferred representations are beneficial to the generalization and optimization perspectives of deep networks.

The present study addresses this important problem from several new perspectives. We first probe into how pretrained knowledge benefits generalization. Results indicate that models fine-tuned on target datasets similar to the pretrained dataset tend to stay close to the transferred parameters. In this sense, transferring from a similar dataset makes fine-tuned parameters stay in the flat region around the pretrained parameters, leading to flatter minima than training from scratch.

Another key to transferability is that transferred features make the optimization landscape significantly improved with better Lipschitzness, which eases optimization. Results show that the landscapes with transferred features are smoother and more predictable, fundamentally stabilizing and accelerating training especially at the early stages of training. This is further enhanced by the proper scaling of gradient in back-propagation. The principal component of gradient is suppressed in the transferred weight matrices, controlling the magnitude of gradient and smoothing the loss landscapes.

We also investigate a common concern raised by practitioners: when is transfer learning helpful to target tasks? We test the transferability of pretrained networks with varying inputs and labels. Instead of the similarity between pretrained and target inputs, what really matters is the similarity between the pretrained and target tasks, i.e. both inputs and labels are required to be sufficiently similar. We also investigate the relationship between pretraining epoch and transferability. Surprisingly, although accuracy on the pretrained dataset increases throughout training, transferability first increases at the beginning and then decreases significantly as pretraining proceeds.

Finally, this paper gives a theoretical analysis based on two-layer fully connected networks. Theoretical results consistently justify our empirical discoveries. The analysis here also casts light on deeper networks. We believe the mechanism of transferability is the fundamental property of deep neural networks and the in-depth understanding presented here may stimulate further algorithmic advances.

2. RELATED WORK:

There exists extensive literature on transferring pretrained representations to learn an accurate model on a target dataset. Donahue et al. (2014) employed a brand-new label predictor to classify features extracted by the pre-trained feature extractor at different layers of AlexNet (Krizhevsky et al., 2012) . Oquab et al. (2014) showed deep features can benefit object detection tasks despite the fact that they are trained for image classification. Ge & Yu (2017) introduced a selective joint fine-tuning scheme for improving the performance of deep learning tasks under the scenario of insufficient training data.

The enormous success of the transferability of deep networks in applications stimulates empirical studies on fine-tuning and transferability. Yosinski et al. (2014) observed the transferability of deep feature representations decreases as the discrepancy between pretrained task and target task increases and gets worse in higher layers. Another phenomenon of catastrophic forgetting as discovered by Kirkpatrick et al. (2016) describes the loss of pretrained knowledge when fitting to distant tasks. Huh et al. (2016) delved into the influence of ImageNet pretrained features by pretraining on various subsets of the ImageNet dataset. Kornblith et al. (2019) further demonstrated that deep models with better ImageNet pretraining performance can transfer better to target tasks.

As for the techniques used in our analysis, Li et al. (2018a) proposed the impact of the scaling of weight matrices on the visualization of loss landscapes. Santurkar et al. (2018) proposed to measure the variation of loss to demonstrate the stability of loss function. Du et al. (2019) provided a powerful framework of analyzing two-layer over-parametrized neural networks, with elegant results and no strong assumptions on input distributions, which is flexible for our extensions to transfer learning.

3. TRANSFERRED KNOWLEDGE INDUCES BETTER GENERALIZATION:

A basic observation of transferability is that tasks on target datasets more similar to the pretrained dataset have better performance. We delve deeper into this phenomenon by experimenting on a variety of target datasets (Figure 1 ), carried out with two common settings: 1) train only the last layer by fixing the pretrained network as the feature extractor and 2) train the whole network by fine-tuning from the pretrained representations. Results in Table 1 clearly demonstrate that, for both settings and for all target datasets, the training error converges to nearly zero while the generalization error varies significantly. In particular, a network pretrained on more similar dataset tends to generalize better and converge faster on the target dataset. A natural implication is that the knowledge learned from the pretrained networks can only be preserved to different extents for different target datasets. We substantiate this implication with the following experiments. To analyze to what extent the knowledge learned from pretrained dataset is preserved, for the fixing setting, we compute the Frobenius norm of the deviation between fine-tuned weight W and pretrained weight W 0 as 1 √ n W -W 0 F , where n denotes the number of target examples (for the fine-tuning setting, we compute the sum of deviations in all layers 1 √ n l W (l) -W 0(l) F ). Results are shown in Figure 2 . It is surprising that although accuracy may oscillate, 1 √ n W -W 0 F increases monotonously throughout the training process and eventually converges. Datasets with larger visual similarity to ImageNet have smaller deviations from pretrained weight, and

1 √ n W -W 0 F also converges faster. Another observation is that 1 √ n W -W 0 F is approximately proportional to the generalization error shown in Table 1. (a) 1 √ n W -W0 F in the last layer. (b) 1 √ n l W (l) -W 0(l) F in all layers.

Figure 2 : The deviation of the weight parameters from the pretrained ones in the transfer process to different target datasets. For all datasets, 1 √ n W -W 0 F increases monotonously. More knowledge can be preserved on target datasets more similar to ImageNet, yielding smaller 1

√ n W -W 0 F .

Why is preserving pretrained knowledge related to better generalization? From the experiments above, we can observe that models preserving more transferred knowledge (i.e. yielding smaller

1 √ n W -W 0 F ) generalize better. It is reasonable to hypothesize that 1 √ n W -W 0 F is

implicitly bounded in the transfer process, and that the bound is related to the similarity between pretrained and target datasets (We will formally study this conjecture in the theoretical analysis). Intuitively, a neural network attempts to fit the training data by twisting itself from the initialization point. For similar datasets the twist will be mild, with the weight parameters staying closer to the pretrained parameters. We use the filter normalization (Li et al., 2018a) to avoid the impact of weight scale.

Randomly initialized networks end up with sharper minima. Pretrained networks stay in flat regions.

Such property of staying near the pretrained weight is crucial for understanding the improvement of generalization. Since optimizing deep networks inevitably runs into local minima, a common belief of deep networks is that the optimization trajectories of weight parameters on different datasets will be essentially different, leading to distant local minima. To justify whether this is true, we compare the weight matrices of training from scratch and using ImageNet pretrained representations in Figure 4 .

Results are quite counterintuitive. The local minima of different datasets using ImageNet pretraining are closed to each other, all concentrating around ImageNet pretrained weight. However, the local minima of training from scratch and ImageNet pretraining are way distant, even on the same dataset.

Figure 4 : The t-SNE (van der Maaten & Hinton, 2008) visualization of the weight matrices of the pretrained and randomly initialized networks on different datasets. Surprisingly, weight matrices on the same dataset may be distant at convergence when using different initializations. On the contrary, even for discrepant datasets, the weight matrices stay close to the initialization when using the same pretrained parameters.

This provides us with a clear picture of how transferred representations improve generalization on target datasets. Rich studies have indicated that the properties of local minima are directly related to generalization (Keskar et al., 2017; Izmailov et al., 2018) . Using pretrained representations restricts weight matrices to stay near the pretrained weight. Since the pretrained dataset is usually sufficiently large and of high-quality, transferring their representations will lead to flatter minima located in large flat basins. On the contrary, training from scratch may find sharper minima. To observe this, we adopt filter normalization (Li et al., 2018a) as the visualization tool, and illustrate the loss landscapes around the minima in Figure 3 . This observation concurs well with the experiments above. The weight matrices for datasets similar to pretrained ones deviate less from pretrained weights and stay in the flat region. On more different datasets, the weight matrices have to go further from pretrained weights to fit the data and may run out of the flat region.

4. PROPERLY PRETRAINED NETWORKS ENABLE BETTER LOSS LANDSCAPES:

A common belief of modern deep networks is the improvement of loss landscapes with techniques such as BatchNorm (Ioffe & Szegedy, 2015) and residual structures (He et al., 2016) . Li et al. (2018a) ; Santurkar et al. (2018) validated this improvement when the model is close to convergence. However, it is often overlooked that loss landscapes can still be messy at the initialization point. To verify this conjecture, we visualize the loss landscapes centered at the initialization point of the 25th layer of ResNet-50 in Figure 5 . (Visualizations of the other layers can be found in Appendix B.4.) ImageNet pretrained networks have much smoother landscape than networks trained with random initialization. The improvement of loss landscapes at the initialization point directly gives rise to the acceleration of training. Concretely, transferred features help ameliorate the chaos of loss landscape with improved Lipschitzness in the early stages of training. Thus, gradient-based optimization method can easily escape from the initial region where the loss is very large. The properties of loss landscapes influence the optimization fundamentally. In randomly initialized networks, going in the direction of gradient may lead to large variation in the loss function. On the contrary, ImageNet pretrained features make the geometry of loss landscape much more predictable, and a step in gradient direction will lead to mild decrease of loss function. To demonstrate the impact of transferred features on the stability of loss function, we further analyze the variation of loss in the direction of gradient in Figure 6 . For each step in the training process, we compute the gradient of the loss and measure how the loss changes as we move the weight matrix in that direction. We can clearly observe that in contrast to networks with transferred features, randomly initialized networks have larger variation along the gradient, where a step along the gradient leads to drastic change in the loss. Why can transferred features control the magnitude of gradient and smooth the loss landscape? Figure 6 : Variation of the loss in ResNet-50 with ImageNet pretrained weight and random initialization. We compare the variation of loss function in the direction of gradient during the training process on CUB-200 dataset. The variation of pretrained networks is substantially smaller than the randomly initialized one, implying a more desirable loss landscape and more stable optimization.

A natural explanation is that transferred weight matrices provide appropriate transform of gradient in each layer and help stabilize its magnitude. Note that in deep neural networks, the gradient w.r.t. each layer is computed through back-propagation by

∂L ∂x k-1 i = W k I k i ∂L ∂x k i

, where I k i denotes the activation of x i at layer k. The weight matrices W k function as the scaling factor of gradient in back-propagation. Basically, a randomly initialized weight matrix will multiply the magnitude of gradient by its norm. In pretrained weight matrices, situation is completely different. To delve into this, we decompose the gradient into singular vectors and measure the projections of weight matrices in these principal directions. Results are shown in Figure 7 (c). During pretraining, the singular vectors of the gradient with large singular values are shrunk in the weight matrices. Thus, the magnitude of gradient back-propagated through a pretrained layer is controlled. In this sense, pretrained weight matrices stabilize the magnitude of gradient especially in lower layers. We visualize the magnitude and scaling of gradient of different layers in ResNet-50 in Figure 7 . The gradient of randomly initialized networks grows fast with layer numbers during back-propagation while the gradient of ImageNet pretrained networks remains stable. Note that ResNet-50 already incorporates BatchNorm and skip-connections to improve the gradient flow, and pretrained representations can stabilize the magnitude of gradient substantially even in these modern networks. We complete this analysis by visualizing the change of landscapes during back-propagation in Section B.4.

5. WHEN IS TRANSFER LEARNING FEASIBLE IN DEEP NETWORKS?:

Transferring from pretrained representations boosts performance in a wide range of applications. However, as discovered by He et al. (2018) ; Kornblith et al. (2019) , there still exist cases when pretrained representations provide no help for target tasks or even downgrade test accuracy. Hence, the conditions on which transfer learning is feasible is an important open problem to be explored. In this section, we delve into the feasibility of transfer learning with extensive experiments, while the theoretical perspectives are presented in the next section. We hope our analysis will provide insights into how to adopt transfer learning by practitioners.

5.1. CHOICES OF A PRETRAINED DATASET:

As a common practice, people choose datasets similar to the target dataset for pretraining. However, how can we determine whether a dataset is sufficiently similar to a target dataset? We verify with experiments that the similarity depends on the nature of tasks, i.e. both inputs and labels matter. Varying input with fixed labels. We randomly sample 600 images from the original SVHN dataset, and fine-tune the MNIST pretrained LeNet (LeCun et al., 1998) to this SVHN subset. For comparison, we pretrain other two models on MNIST with images upside down and Fashion-MNIST (Xiao et al., 2017) , respectively. Note that for all three pretrained models, the dataset sizes, labels, and the number of images per class are kept exactly the same, and thus the only difference lies in the image pixels themselves. Results are shown in Figure 8 (a). Compared to training from scratch, MNIST pretrained features improve generalization significantly. Upside-down MNIST shows slightly worse generalization performance than the original one. In contrast, fine-tuning from Fashion-MNIST barely improves generalization. We also compute the deviation from pretrained weight of each layer.

The weight matrices and convolutional kernel deviation of Fashion-MNIST pretraining show no improvement over training from scratch. A reasonable implication here is that choosing a model pretrained on a more similar dataset in the inputs yields a larger performance gain.

Varying labels with fixed input. We train a ResNet-50 model on Caltech-101 and then fine-tune it to Webcam (Saenko et al., 2010) . We train another ResNet-50 to recognize the color of the upper part of Caltech-101 images and fine-tune it to Webcam. Results in Figure 8 (b) indicate that the latter one provides no improvement over training on Webcam from scratch, while pretraining on standard Caltech-101 significantly boosts performance. Models generalizing very well on similar images are not transferable to the target dataset with totally different labels. These experiments challenge the common perspective of similarity between datasets. The description of similarity using the input (images) themselves is just one point. Another key factor of similarity is the relationship between the nature of tasks (labels). This observation is further in line with our theoretical analysis in Section 6.

Similar results are observed on datasets of larger scale for the above two cases (See Figure 8 (c)8(d)). We fine-tune deep representations pretrained on ImageNet and Places (Zhou et al., 2018 ) (a largescale dataset of scene understanding) to CUB-200 and MIT-indoors datasets. As a fine-grained object recognition task, CUB-200 benefits more from ImageNet pretrained features. In contrast, MIT-indoors is a scene dataset more similar to Places, thus it benefits more from Places pretraining. Under review as a conference paper at ICLR 2020

5.2. CHOICES OF PRETRAINING EPOCHS:

Currently, people usually train a model on ImageNet until it converges and use it as the pretrained parameters. However, the final model do not necessarily have the highest transferability. To see this, we pretrain a ResNet-50 model on Food-101 (Bossard et al., 2014) and transfer it to CUB-200, with results shown in Figure 9 . During the early epochs, the transferability increases sharply. As we continue pretraining, although the test accuracy on the pretraining dataset continues increasing, the test accuracy on the target dataset starts to decline, indicating downgraded transferability. Intuitively, during the early epochs, the model learns general knowledge that is informative to many datasets. As training goes on, however, the model starts to fit the specific knowledge of the pretrained dataset and even fit noise. Such dataset-specific knowledge is usually detrimental to the transfer performance. This interesting finding implies a promising direction for improving the de facto pretraining method: Instead of seeking for a model with higher accuracy only on the pretraining dataset, a more transferable model can be pretrained with appropriate epochs such that the fine-tuning accuracies on a diverse set of target tasks are advantageous. Algorithms for pretraining should take this point into consideration.

6. THEORETICAL ANALYSIS:

We have shown through extensive empirical analysis that transferred features exert a fundamental impact on generalization and optimization performance, and provided some insights for the feasibility of transfer learning. In this section, we analyze some of our empirical observations from a theoretical perspective. We base our analysis on two-layer fully connected networks with ReLU activation and sufficiently many hidden units. Our theoretical results are in line with the experimental findings. 

Q samples {x Q,i , y Q,i } n Q

i=1 drawn i.i.d. from the target distribution Q as the target dataset and a weight matrix W(P ) pretrained on n P samples {x P,i , y P,i } n P i=1 drawn i.i.d. from pretrained distribution P . Suppose x 2 = 1 and |y| ≤ 1. Our goal is transferring the pretrained W(P ) to learn an accurate model W(Q) for the target distribution Q. When training the model on the pretraining dataset, we initialize the weight as: w r (0) ∼ N (0, κ 2 I), a r ∼unif ({-1, 1}), where ∀r ∈ [m] and κ is a constant.

For both pretraining and fine-tuning, the objective function of the model is the squared loss

L(W) = 1 2 (y -f W,a (X)) (y -f W,a (X))

. Note that a is fixed throughout training and W is updated with gradient descent. The learning rate is set to η.

We base our analysis on the theoretical framework of Du et al. (2019) , since it provides elegant results on convergence of two-layer ReLU networks without strong assumptions on the input distributions, facilitating our extension to the transfer learning scenarios. In our analysis, we use the Gram matrices

H ∞ P ∈ R n P ×n P and H ∞ Q ∈ R n Q ×n Q

to measure the quality of pretrained input and target input as

H ∞ P,ij = E w∼N (0,I) [x P,i x P,j I{w x P,i ≥ 0, w x P,j ≥ 0}] = x P,i x P,j (π -arccos(x P,i x P,j )) 2π , H ∞ Q,ij = E w∼N (0,I) [x Q,i x Q,j I{w x Q,i ≥ 0, w x Q,j ≥ 0}] = x Q,i x Q,j (π -arccos(x Q,i x Q,j )) 2π .

To quantify the relationship between pretrained input and target input, we define the following Gram matrix H ∞ P Q ∈ R n P ×n Q across samples drawn from P and Q:

H ∞ P Q,ij = E w∼N (0,I) [x P,i x Q,j I{w x P,i ≥ 0, w x Q,j ≥ 0}] = x P,i x Q,j (π -arccos(x P,i x Q,j )) 2π .

Assume Gram matrices H ∞ P and H ∞ Q are invertible with smallest eigenvalue λ P and λ Q greater than zero. H ∞ P -1 y P characterizes the labeling function of pretrained tasks. y P →Q H ∞ P Q H ∞ P -1 y P further transforms the pretrained labeling function to the target labels. A critical point in our analysis is y Q -y P →Q , which measures the task similarity between target label and transformed label.

6.2. IMPROVED LIPSCHITZNESS OF LOSS FUNCTION:

To analyze the Lipschitzness of loss function, a reasonable objective is the magnitude of gradient, which is a direct manifestation of the Lipschitz constant. We analyze the gradient w.r.t. the activations. For the magnitude of gradient w.r.t. the activations, we show that the Lipschitz constant is significantly reduced when the pretrained and target datasets are similar in both inputs and labels. Theorem 1 (The effect of transferred features on the Lipschitzness of the loss). Denote by X 1 the activations in the target dataset. For a two-layer networks with sufficiently large number of hidden unit m defined in Section 6.1, if m ≥ poly(n

P , n Q , δ -1 , λ -1 P , λ -1 Q , κ -1 ), κ = O λ 2 P δ n 2 P n 1 2 Q

, with probability no less than 1 -δ over the random initialization, ∂L(W(P ))

∂X 1 2 = ∂L(W(0)) ∂X 1 2 -y Q y Q + (y Q -y P →Q ) (y Q -y P →Q ) + poly(n P , n Q , δ -1 , λ -1 P , κ -1 ) m 1 4 + O   n 2 P n 1 2 Q κ λ 2 P δ   .

(

EQUATION

This provides us with theoretical explanation of experimental results in Section 4. The control of Lipschitz constant relies on the similarity between tasks in both input and labels. If the original target label is similar to the label transformed from the pretrained label, i.e. y Q -y P →Q 2 2 is small, the Lipschitzness of loss function will be significantly improved. On the contrary, if the pretrained and target tasks are completely different, the transformed label will be discrepant with target label, resulting in larger Lipschitz constant of the loss function and worse landscape in the fine-tuned model.

6.3. IMPROVED GENERALIZATION:

Recall in Section 3 that we have investigated the weight change W(Q) -W(P ) F during training and point out the role it plays in understanding the generalization. In this section, we show that W(Q)-W(P ) F can be bounded with terms depicting the similarity between pretrained and target tasks. Note that the Rademacher complexity of the function class is bounded with W(Q)-W(P ) F as shown in the seminal work (Arora et al., 2019) , thus the generalization error is directly related to W(Q) -W(P ) F . We still use the Gram matrices defined in Section 6.1. Theorem 2 (The effect of transferred features on the generalization error). For a two-layer networks with m ≥ poly(n

P , n Q , δ -1 , λ -1 P , λ -1 Q , κ -1 ), κ = O λ 2 P λ 2 Q δ n 2 P n 1 2 Q

, with probability no less than 1 -δ over the random initialization,

EQUATION

)

This result is directly related to the generalization error and casts light on our experiments in Section 5.1. Note that when training on the target dataset from scratch, the upper bound of

W(Q)-W(0) F is y Q H ∞ Q -1 y Q .

By fine-tuning from a similar pretrained dataset where the transformed label is close to target label, the generalization error of the function class is hopefully reduced. On the contrary, features pretrained on discrepant tasks do not transfer to classification task in spite of similar images since they have disparate labeling functions. Another example is fine-tuning to Food-101 as in the experiment of Kornblith et al. (2019) . Since it is a fine-grained dataset with many similar images, H ∞ Q will be more singular than common tasks, resulting in a larger deviation from the pretrained weight. Hence even transferring from ImageNet, the performance on Food-101 is still far from satisfactory.

7. CONCLUSION: BEHIND TRANSFERABILITY:

Why are deep representations pretrained from modern neural networks generally transferable to novel tasks? When is transfer learning feasible enough to consistently improve the target task performance?

These are the key questions in the way of understanding modern neural networks and applying them to a variety of real tasks. This paper performs the first in-depth analysis of the transferability of deep representations from both empirical and theoretical perspectives. The results reveal that pretrained representations will improve both generalization and optimization performance of a target network provided that the pretrained and target datasets are sufficiently similar in both input and labels. With this paper, we show that transfer learning, as an initialization technique of neural networks, exerts implicit regularization to restrict the networks from escaping the flat region of pretrained landscape.

A IMPLEMENTATION DETAILS:

In this section, we provide details of the architectures, setup, methods of visualizations in our analysis. The codes and visualizations are attached with the submission and will be made available online.

Models. We implement all models on PyTorch with 2080Ti GPUs. For object recognition and scene recognition tasks, we use standard ResNet-50 from torchvision. ImageNet pretrained models can be found in torchvision, and Places pretrained models are provided by Zhou et al. (2018) . During fine-tuning we use a batch size of 32 and set the initial learning rate to 0.01 with 0.9 momentum following the protocol of (Li et al., 2018b) . For fine-tuning, we train the model for 200 epochs. We decay the learning rate by 0.1 with the time of decay set by cross validation. In Figure 2 (a) where the pretrained ResNet-50 functions as feature extractor, the downstream classifier is a two-layer ReLU network with Batch-Norm and Leaky-ReLU non-linearity. The number of hidden unit is 512. For this task, the backbone ResNet-50 is fixed, with the downstream two-layer classifier trained with momentum SGD. The learning rate is set to 0.01 with 0.9 momentum, and remains constant throughout training.

For digit recognition tasks, we use LeNet (LeCun et al., 1998) . The learning rate is also set to 0.01, with 5 × 10 -4 weight decay. The batch-size is set to 64. We train the model for 100 epochs.

Fine-tuning. We follow the protocol of fine-tuning as in the previous paragraphs. In Tables 1 and 2 , we run all the experiments for 3 times and report their mean and variance. For Table 2 , the improvement of fine-tuing is calculated with the generalization error of fine-tuning divided by the generalization error of training from scratch.

Visualization of loss landscapes. We use techniques similar to filter normalization to provide an accurate analysis of loss landscapes (Li et al., 2018a) . Note that ReLU networks are invariant to the scaling of weight parameters. To remove this scaling effect, the direction used in visualization should be normalized in a filter-wise way. Concretely, the axes of each landscape figure are two random Gaussian orthogonal vectors normalized by the scale of each filter in the convolutional layers. Concretely, suppose the parameter of the center point is θ. θ i,j denotes the j-th filter of the i-th layer. Suppose the two unit orthogonal vectors are a and b. Then with filter normalization, a i,j ← ai,j ai,j θ i,j and b i,j ← bi,j bi,j θ i,j . For each point (i.e. pixel) (p, q) in the plot, the value is evaluated with g(p, q) = L(f (θ + η(pa + qb))), where L denotes the loss function, f denotes the neural networks. η is a parameter to control the scale of the plot. In all visualization images of ResNet-50, the resolution is 200 × 200, i.e. p = -100, -99, • • • , 98, 99 and q = -100, -99, • • • , 98, 99. For additional details of filter normalization, please refer to (Li et al., 2018a) . η is set to 0.001, which is of the same order as 10 times the step size in training. This is a reasonable scale if we want to study the local loss landscape of model using SGD. For fair comparison between the pretrained landscapes and randomly initialized landscapes, the scale of loss variation in each plot is exactly the same. The difference of loss value between each contour is 0.05. When we compute the loss landscape of one layer, the parameters of other layers are fixed. The gradient is computed based on 256 fixed samples since the gradient w.r.t. full dataset requires too much computation. Figure 3 and Figure 10 are centered at the final weight parameters, while others are centered at the initialization point to show the situation when training just starts. We visualize the loss landscapes on CUB-200, Stanford Cars and Food-101 for multiple times and reach consistent conclusions. But due to space limitation, we only show the results on one dataset for each experiment in the main paper. Other results are deferred to Section B.

Computing the eigenvalue of Hessian. We compute the eigenvalues of Hessian with Hessianvector product and power methods based on the autograd of PyTorch. A similar implementation is provided by Yao et al. (2018) . We only list top 20 eigenvalues in limited space.

t-SNE embedding of model parameters. We put the weight matrices of ResNet-50 in one vector as input. For faster computation, we pre-compute the distance between parameters of every two models with PyTorch and then use the distance matrix to compute the t-SNE embedding with scikitlearn. Note that we use the same ImageNet model from torchvision and the same Places model from Zhou et al. (2018) for fine-tuning.

Variation of loss function in the direction of gradient. Based on the original trajectory of training, we take steps in the direction of gradient from parameters at different steps during training to calculate the maximum changes of loss in that direction. The step size is set to the size of gradient. We take 100 steps from the original trajectories to measure the local property of loss landscapes. We aim to quantify the stability of loss functions and directly show the magnitude of gradient with this experiments on different datasets. Results on CUB-200 are provided in the main paper, with additional results further provided in Section B. Not that this experiment is inspired by Santurkar et al. (2018) . We use the similar protocol as Section 3.2 in Santurkar et al. (2018) . Another protocol is to fix the step size along the gradient and compute the maximum variation of loss. Results on Stanford Cars with this protocol are provided in Section B.2. Results for both scenarios are similar. Figure 11 : Variation of the loss in ResNet-50 with ImageNet pretrained weight and random initialization. We compare the variation of loss function in the direction of gradient during the training process on Stanford Cars dataset. The variation of pretrained networks is substantially smaller than the randomly initialized one, implying a more desirable loss landscape and more stable optimization.

B ADDITIONAL EXPERIMENTAL RESULTS:



B.3 COMPARISON OF TRAINING FROM SCRATCH:

To validate that the generalization error is indeed improved with pretraining, for each dataset, we list the generalization error and the norm of deviation from the pretrained parameters in Table 2 . The decreased percentage is calculated by dividing the error reduced in fine-tuning with the error of training from scratch. Compared to the results of fine-tuning, we observe that ImageNet pretraining improves the generalization performance of general coarse-grained classification tasks significantly, yet the performance boost is smaller for fine-grained tasks which are dissimilar in the sense of task with ImageNet. Note that, although Stanford Cars and CUB-200 are visually similar to ImageNet, what really matters is the similarity between the nature of tasks, i.e. both images and labels matter. We visualize the loss landscape of 25-48th layers in ResNet-50 on Food-101 dataset. We compare the landscapes centered at the initialization point of randomly initialized and ImageNet pretrained networks -see Figure 12 and Figure 13 . Results are in line with our observations of the magnitude of gradient in Figure 7 . At higher layers, the landscapes of random initialization and ImageNet pretraining are similar. However, as the gradient is back-propagated through lower layers, the landscapes of pretrained networks remain as smooth as the higher layers. In sharp contrast, the landscapes of randomly initialized networks worsen through the lower layers, indicating that the magnitude of gradient is substantially worsened in back-propagation. Figure 13 : Landscapes centered at the initialization point of each layer in ResNet-50 initialized randomly. At the higher layers, the landscapes tend to be smooth. However, as the gradient is propagated to lower layers, the landscapes are becoming full of ridges and trenches in spite of the presence of Batch-Norm and skip connections.

C PROOFS OF THEOREMS IN SECTION 6:

To study how transferring pretrained knowledge helps target tasks, we first study the trajectories of weight matrices during pretraining and then analyze its effect as an initialization in target tasks.

Our analysis is based on Du et al. (2019) 's framework for over-parametrized networks. For the weight matrix W, W(0) denotes the random initialization. W P (k) denotes W at the kth step of pretraining. W(P ) denotes the pretrained weight matrix after training K steps. W Q (k) denotes the weight matrix after K steps of fine-tuning from W(P ). For other terms, the notation at each step is similar.

We first analyze the pretraining process on the source datasets based on Arora et al. (2019) . Define a matrix Z P ∈ R md×n P which is crucial to analyzing the trajectories of the weight matrix during pretraining,

EQUATION

where I P i,j = I{w i x P,j ≥ 0}. Z P (k) denotes the matrix corresponding to W P (k). Note that the gradient descent is carried out as vec(W

EQUATION

where vec (•) denotes concatenating a column of a matrice into a single vector. Then in the K iterations of pretraining on the source dataset,

vec(W(P )) -vec(W(0)) = K-1 k=0 vec(W P (k + 1)) -vec(W P (k)) = -η K-1 k=0 Z P (k)(u P (k) -y P ) = K-1 k=0 ηZ P (k)(I -ηH ∞ P ) k y P - K-1 k=0 ηZ P (k)e P (k) = K-1 k=0 ηZ P (0)(I -ηH ∞ P ) k y + K-1 k=0 η(Z P (k) -Z P (0))(I -ηH ∞ P ) k y - K-1 k=0 ηZ P (k)e P (k).

(5) The first term is the primary component in the pretrained matrix, while the second and third terms is small under the over-parametrized conditions. Now following Arora et al. (2019) , the magnitude of these terms can be bounded with probability no less than 1 -δ,

2 = K-1 k=0 η(Z P (k)-Z P (0))(I-ηH ∞ ) k y- K-1 k=0 ηZ P (k)e P (k) 2 = O n P κ λ P δ +O n 2 P m 1 4 λ 3 2 P κ 1 2 δ .

(6) Here we also provide lemmas from Du et al. (2019) which are extensively used later.

Lemma 1. With λ P = λ min (H ∞ P ) > 0, m = Ω n 6 P λ 4 P κ 2 δ 3 and η = O λ P n 2 P

, with probability at least 1 -δ over the random initialization we have

EQUATION

Lemma 2. If w 1 , . . . , w m are i.i.d. generated from N (0, I), then with probability at least 1 -δ, the following holds. For any set of weight vectors w 1 , . . . , w m ∈ R d that satisfy for any r ∈

[m], w r (0) -w r 2 ≤ cδλ0 n 2

R for some small positive constant c, then the matrix H ∈ R n×n defined by

H ij = 1 m x i x j m r=1 I w r x i ≥ 0, w r x j ≥ 0 satisfies H -H(0) 2 < λ0 4 and λ min (H) > λ0 2 .

C.1 PROOF OF THEOREM 1

Now we start to analyze the influence of pretrained weight on target tasks. 1) We show that during pretraining, H ∞ P Q is close to Z P (0) Z Q (P ). 2) Then we analyze u Q (P )-u Q (0) with the properties of H ∞ P Q . 3) Standard calculation shows the magnitude of gradient relates closely to u Q (P ) -u Q (0), and we are able to find out how is the magnitude of gradient improved.

To start with, we analyze the properties of the matrix H ∞ P Q . We show that under over-parametrized conditions, H ∞ P Q is close to the randomly initialized Gram matrix Z P (0) Z Q (P ). Use H P Q (0) to denote Z P (0) Z Q (0), and H P Q (P ) to denote Z P (0) Z Q (P ). Lemma 3. With the same condition as lemma 1, with probability no less than 1 -δ,

EQUATION

)

EQUATION

where

EQUATION

Applying Markov's inequality, and noting that y P -u(0) 2 = O n P δ we have with probability no less than 1 -δ,

EQUATION

) Also note that E[H P Q,ij (0)] = H ∞ P Q,ij

. By Hoeffding's inequality, we have with probability at least 1 -δ,

EQUATION

Combining equation 12 and equation 11, we have with probability at least 1 -δ,

H P Q (P ) -H ∞ P Q F ≤ O n 2 P n Q √ mλ P κδ 3/2 .

Denote by u Q (P ), u Q (0) the output on the target dataset using weight matrix W(P ) and W 0 respectively. First, we compute the gradient with respect to the activations,

EQUATION

EQUATION

)

It is obvious from equation 14 that u Q (P ) -u Q (0) should become the focus of our analysis. To calculate u Q (P ) -u Q (0), we need to sort out how the activations change by initializing the target networks with W(P ) instead of W(0).

EQUATION

For each x Q,i , divide r into two sets to quantify the change of variation in activations on the target dataset.

EQUATION

where R = 4 √ n P y P -u(0) 2 √ mλ P . For r in S i , we can estimate the size of S i . Note that

E S i = E n i=1 m r=1 I |w(0) r x Q,i | ≤ R . For each i and r, E[I( w(0) r x Q,i ≤ R)] = E[I(|w(0) r | ≤ R)] ≤ 2R √ 2πκ

, since the distribution of w(0) r is Gaussian with mean 0 and covariance matrix κ 2 I. Therefore, taking sum over all i and m and using Markov inequality, with probability at least 1 -δ over the random initialization we have

EQUATION

Thus, this part of activations is the same for W(0) and W(P ) on the target dataset. For each x Q,i ,

EQUATION

where

I Q r,i (0) denotes I{w r (0)x Q,i ≥ 0}.

The first term is the primary part, while we can show that the second and the third term can be bounded with 1

EQUATION

where 1 and 2 correspond to each of the second term and third term in equation 18. Thus, using lemma 1 and the estimation of |S i |, with probability no less than 1 -δ,

EQUATION

)

Now equipped with equation 6, equation 19, equation 20 and lemma 3, we are ready to calculate exactly how much pretrained wight matrix W(P ) help reduce the magnitude of gradient over W(0),

∂L ∂X 1 2 2 = 1 m a a(u Q (P ) -y Q ) 2 = u Q (0) -y Q 2 2 + u Q (P ) -u Q (0) 2 2 + 2 u Q (P ) -u Q (0), u Q (0) -y Q = ∂L 0 ∂X 1 2 2 + y p H ∞ P -1 H ∞ P Q H ∞ P Q H ∞ P -1 y P -2y Q H ∞ P Q H ∞ P -1 y P + 2 2 + 2 Z Q (P ) Z P (0)H ∞ P -1 y P + 2 Z Q (P ) + y P H ∞ P -1 (Z P (0) Z Q (P ) -H ∞ P Q ) 2 2 + 2 2 + 2 Z Q (P ) Z P (0)H ∞ P -1 y P + u Q (0) Z Q (P ) Z P (0)H ∞ P -1 y P + u Q (0) Z Q (P ) + u Q (0) + 2y Q (Z Q (P ) Z P (0) -H ∞ P Q )H ∞ P -1 y P . (21) In equation 21, note that 2 , 2 , Z Q (P ) Z P (0) -H ∞ P Q F = H P Q (P ) -H ∞ P Q F

, and u Q (0) 2 are all small values we have estimated above. Therefore, using Z P (0) F ≤ √ n P and Z Q (P ) F ≤ √ n Q , we can control the magnitude of the perturbation terms under over-parametrized conditions. Concretely, with probability at least 1 -δ over random initialization,

2 2 = O n 4 P n Q mλ 4 P δ 4 κ 2 (22) Z Q (P ) Z P (0)H ∞ P -1 y P = O n 2 P n 1/2 Q √ mλ 2 P δκ √ n P √ n Q 1 λ P √ n P = O n 3 P n Q √ mλ 2 P δ 2 κ (23) Z Q (P ) = O n 3 P n Q √ mλ 3 P δ 3 + O n 3 P n Q m 3/4 λ 7/2 P δ 3 κ 3/2 (24) y P H ∞ P -1 (Z P (0) Z Q (P ) -H ∞ P Q ) 2 2 = O √ n P 1 λ P n 2 P n Q √ mλ P κδ 3/2 2 = O n 5 P n 2 Q mλ 4 P κ 2 δ 3 (25) 2 2 = O n 2 P κ 2 λ 2 P δ 2 + O n 4 P m 1/2 λ 3 P κδ 2 (26) Z Q (P ) Z P (0)H ∞ P -1 y P = O n 2 P n 1/2 Q κ λ 2 P δ + O n 3 P n 1/2 Q m 1/4 λ 5/2 P √ κδ (27) u Q (0) Z Q (P ) Z P (0)H ∞ P -1 y P = O n P n Q κ λ P √ δ (28) u Q (0) Z Q (P ) = O n P n Q κ 2 λ P δ 3/2 + O n 2 P n Q m 1/4 λ 3/2 P √ κδ 3/2 (29) u Q (0) = O n 2 P n Q m 1/2 λ 2 P δ 5/2 (30) y Q (Z Q (P ) Z P (0) -H ∞ P Q )H ∞ P -1 y P = O n 5/2 P n 3/2 Q √ mλ 2 P κδ 3/2 (31)

Substituting these estimations into equation 21 completes the proof of Theorem 1.

C.2 PROOF OF THEOREM 2:

In this subsection, we analyze the impact of pretrained weight matrix on the generalization performance. First, we show that a model will converge if initialized with pretrained weight matrix. Based on this, we further investigate the trajectories during transfer learning and bound W -W(P ) F with the relationship between source and target datasets.

C.2.1 CONVERGENCE OF TRANSFERRING FROM PRETRAINED REPRESENTATIONS:

Similar to Du et al. (2019) , the proof is done with induction, but since we start from W(P ) instead of randomly initialized W 0 in the transferring process, we should use the randomness of W 0 in an indirect way. Concretely, we should use lemma 1 to bound the difference between each column of W(P ) and randomly initialized W(0) when proving the induction hypothesis.

Theorem 3 (Convergence of Transfer Learning). Under the same conditions as in Theorem 1, if we set the number of hidden nodes m = Ω

n 8 P n 6 Q λ 16 P λ 4 P κ 2 δ 10 , κ = O λ 2 P δ n 2 P n 1 2 Q

, and the learning

rate η = O λ Q n 2 Q

then with probability at least 1 -δ over the random initialization we have for k = 0, 1, 2, . . .

EQUATION

)

The following lemma is a direct corollary of Theorem 3 and lamma 1, and is crucial to analysis to follow. Lemma 4. Under the same conditions as Theorem 3, with probability at least 1 -δ over the random initialization we have ∀r ∈

EQUATION

We have the estimation of w

Q,r (k) -w r (0) 2 from lemma 1. From w Q,r (k) -w r (0) 2 ≤ w Q,r (k) -w r (P ) 2 + w r (P ) -w r (0) 2 , we can proove lemma 4 by estimating w Q,r (k) -w r (P ) 2 . w Q,r (k) -w r (P ) 2 =η k k =0 ∂L(W(k )) ∂w r (k ) 2 ≤η k k =0 √ n Q y Q -u Q (k ) 2 √ m ≤η ∞ k =0 √ n Q (1 - ηλ Q 2 ) k /2 √ m y Q -u Q (k ) 2 = 4 √ n Q y Q -u Q (P ) 2 √ mλ Q We also have y Q -u Q (0) 2 = O κ √ n Q √ δ , and u Q (P ) -u Q (0) = Z Q (0) (Z P (0)H ∞ P -1 y P +

) + 1 + 2 . Substituting lemma 3, equation 6, and equation 22 into u Q (P ) -u Q (0) 2 completes the proof. Now we start to prove Theorem 3 by induction.

Condition 1. At the k-th iteration, we have u Q (k) -y Q 2 2 ≤ 1 - ηλ 0Q 2 k u Q (P ) -y Q 2 2 .

We have the following corollary if condition 1 holds, Corollary 1. If condition 1 holds for k = 0, . . . , k, for every r ∈ [m], with probability at least 1 -δ,

w Q,r (k) -w r (0) 2 ≤ 4 √ n P y P -u P (0) 2 √ mλ P + 4 √ n Q y Q -u Q (P ) 2 √ mλ Q R .

If k = 0, by definition Condition 1 holds. Suppose for k = 0, . . . , k, condition 1 holds and we want to show it still holds for k = k + 1. The strategy is similar to the proof of convergence on training from scratch. By classifying the change of activations into two categories, we are able to deal with the ReLU networks as a perturbed version of linear regression. We define the event

A ir = w r (0) x Q,i ≤ R , where R = cλ Q n 2 Q

for some small positive constant c to control the magnitude of perturbation. Similar to the analysis above, we let S i = {r ∈ [m] : I{A ir } = 0} and S i = [m] \ S i . Since the distribution of w r (0) is Gaussian, we can bound the value of each A ir and then bound the size of S i just as we have estimated in equation 17 above.

Lemma 5. With probability at least 1 -δ over the initialization, we have

n i=1 |S i | ≤ Cmn Q R δ for some positive constant C > 0.

The following analysis is identical to the situation of training from scratch.

u Q,i (k + 1) -u Q,i (k) = 1 √ m m r=1 a r σ w Q,r (k + 1) x Q,i -σ w Q,r (k) x Q,i .

By dividing [m] into S i and S i , we have,

I i 1 1 √ m r∈Si a r σ w Q,r (k + 1) x Q,i -σ w Q,r (k) x Q,i I i 2 1 √ m r∈Si a r σ w Q,r (k + 1) x Q,i -σ w Q,r (k) x Q,i .

We view I i 2 as a perturbation and bound its magnitude. Because ReLU is a 1-Lipschitz function and |a r | = 1, we have

|I i 2 | ≤ η m 1 2 r∈Si | ∂L(W Q (k)) ∂w Q,r (k) x Q,i | ≤ η|S i | √ m max r∈[m] ∂L(W Q (k)) ∂w Q,r (k) 2 ≤ η|S i |n Q 1 2 u Q (k) -y Q 2 m .

By Corollary 1, we know w Q,r (k) -w r (0) ≤ R for all r ∈ [m]. Furthermore, with the conditions on m, we have R < R.

Thus I w Q,r (k + 1) x Q,i ≥ 0 = I w Q,r (k) x Q,i ≥ 0 for r ∈ S i .. I i 1 = - η m n Q j=1 x Q,i x Q,j (u Q,j -y Q,j ) r∈Si I w Q,r (k) x Q,i ≥ 0, w Q,r (k) x Q,j ≥ 0 = - η m n Q j=1 x Q,i x Q,j (u Q,j -y Q,j ) m r=1 I w Q,r (k) x Q,i ≥ 0, w Q,r (k) x Q,j ≥ 0 + η m n Q j=1 x Q,i x Q,j (u Q,j -y Q,j ) r∈Si I w Q,r (k) x Q,i ≥ 0, w Q,r (k) x Q,j ≥ 0 where 1 m m r=1 x Q,i x Q,j I w Q,r (k) x Q,i ≥ 0, w Q,r (k) x Q,j ≥ 0 is just the (i, j)-th entry of a discrete version of Gram matrix H ∞ Q defined in Section 6.1 and | η m n Q j=1 x Q,i x Q,j (u Q,j -y Q,j ) r∈Si I w Q,r (k) x Q,i ≥ 0, w Q,r (k) x Q,j ≥ 0 | ≤ η m |S i | n Q j=1 |u Q,j (k) -y Q,j | ≤ η √ n Q m |S i | u Q (k) -y Q 2 .

For ease of notations, denote by

H Q (k) ⊥ 2 the matrix whose i, j entry is 1 m x Q,i x Q,j (u Q,j -y Q,j ) r∈Si I w Q,r (k) x Q,i ≥ 0, w Q,r (k) x Q,j ≥ 0 . Therefore, the L2 norm of H(k) ⊥ 2 is bounded with Cn 2 Q R δ

. To bound the quadratic term, we use the same techniques as training from scratch.

u Q (k + 1) -u Q (k) 2 2 ≤ η 2 n Q i=1 1 m m r=1 ∂L(W Q (k)) ∂w Q,r (k) 2 2 ≤ η 2 n 2 Q y Q -u Q (k) 2 2 .

With these estimates at hand, we are ready to prove the induction hypothesis.

y Q -u Q (k + 1) 2 2 = y Q -u Q (k) -(u Q (k + 1) -u Q (k)) 2 2 = y Q -u Q (k) 2 2 -2 (y Q -u Q (k)) (u Q (k + 1) -u Q (k)) + u Q (k + 1) -u Q (k) 2 2 = y Q -u Q (k) 2 2 -2η (y Q -u Q (k)) H Q (k) (y Q -u Q (k)) +2η (y Q -u Q (k)) H Q (k) ⊥ (y Q -u Q (k)) -2 (y Q -u Q (k)) I 2 + u Q (k + 1) -u Q (k) 2 2 ≤(1 -ηλ Q + 2Cηn Q R δ + 2Cηn 3/2 Q R δ + η 2 n 2 Q ) y Q -u Q (k) 2 2 ≤(1 - ηλ Q 2 ) y Q -u Q (k) 2 2 .

The third equality we used the decomposition of u(k + 1) -u(k). The first inequality we used the Lemma 2, the bound on the step size, the bound on I 2 , the bound on H(k) ⊥ 2 and the bound on u(k + 1) -u(k) 2 2 . The last inequality we used the bound of the step size and the bound of R. Therefore Condition 1 holds for k = k + 1. Now by induction, we prove Theorem 3.

Similar to the analysis of lemma 3, we can show the change of Z Q (k) and H Q (k) is negligible under the conditions of sufficiently large m. Lemma 6. Under the same conditions as Theorem 3 in the transferring process, with probabilility no less than 1 -δ,

Z Q (0) -Z Q (k) F = O   n 3/2 P n 5/4 Q m 1 2 λ 2 P λ Q κ 2 δ 3/2   H Q (0) -H Q (k) F = O n 3 P n 7/2 Q √ mλ 2 P λ Q κ 2 δ 3 2

This is a direct corollary of lemma 4 and can be proved by the same techniques as lemma 3. Now we can continue to analyze the trajectory of transferring W(P ) to the target dataset by dividing the activations in to the two categories as in the proof of Theorem 3.

EQUATION

3 (k) 2 = n Q i=1 η m n Q j=1 x Q,i x Q,j (u Q,j -y Q,j ) r∈Si I w Q,r (k) x Q,i ≥ 0, w Q,r (k) x Q,j ≥ 0 + I i 2 ≤ n Q i=1 2η √ n Q m |S i | u Q (k) -y Q 2 ,

where S i ≤

EQUATION

EQUATION

)

where the second inequality holds with Theorem 3. Taking sum over each iteration,

EQUATION

)

where we notice max k>0 k 1 -

ηλ Q 4 k-1

= O 1 ηλ Q . Then in the K iterations of transferring to the target dataset,

EQUATION

The first term is the primary part, while the second and the third are considered perturbations and could be controlled using lemma 6 and equation 38.

EQUATION

)

where we use

K k=1 k 1 - ηλ Q 4 k-1 = O 1 η 2 λ 2 Q

. With these estimations at hand, we are ready to calculate the final results

W Q (K) -W(P ) 2 F = vec(W Q (K)) -vec(W(P )) 2 2 = η 2 (y Q -u Q (P )) K-1 k=0 (I -ηH ∞ Q ) k Z Q (0) Z Q (0) K-1 k=0 (I -ηH ∞ Q ) k (y Q -u Q (P )) + O n 5 P n 5 Q m 1/4 λ 4 P λ 4 Q κ 2 δ 3/2 = (y Q -u Q (P )) H ∞ Q -1 (y Q -u Q (P )) + O n Q log n δ 1/4 m 1/4 λ Q + O n 5 P n 5 Q m 1/4 λ 4 P λ 4 Q κ 2 δ 3/2

(42) Using equation 19, we have

EQUATION

Combine the estimation above with equation 22 and equation 26. We also have u 

EQUATION

EQUATION

)

which completes the proof.

:



Figures and Tables:

(a) Eigenvalues of Hessian. (b) Randomly initialized. (c) ImageNet pretrained.

Figure 3: Generalization w.r.t. landscape at convergence on Stanford Cars dataset with ResNet-50. (a) Comparison of the top 20 eigenvalues of the Hessian matrix. ImageNet pretrained networks have smaller eigenvalues, indicating flatter minima. (b) (c) Comparison of landscapes centered at the minima. We use the filter normalization(Li et al., 2018a)  to avoid the impact of weight scale. Randomly initialized networks end up with sharper minima. Pretrained networks stay in flat regions.

Figure 5: Optimization landscape at initialization. (a) A figure of convergence on fine-grained classification task Food-101. The convergence of fine-tuning from pretrained networks is significantly faster than training from scratch. (b) (c) Visualizations of loss landscapes of the 25th layer in ResNet-50 centered at initialization on Stanford Cars dataset. ImageNet pretrained landscape is much smoother, indicating better Lipschitzness and predictability of the loss function. Randomly initialized landscape is more bumpy, making optimization unstable and inefficient.

Projection of weight on gradient components. (d) Scale of gradient in different layers.

Figure 7: The stabilization of gradient by pretrained weights on CUB-200. (a) (b) Distribution of the magnitude of gradient in the 25th layer of ResNet-50. (c) Magnitude of the projection of weight on the singular vectors of gradient. (d) Scaling of the gradient in different layers in back-propagation.

Figure 8: Performance by varying either the inputs or labels. (a) Accuracy and deviation of transferring from different datasets to MNIST, where ψ 1 and ψ 2 denote the convolutional kernels in the first and the second layers, and W denotes the weight of the fully-connected layer. (b) Accuracy on Webcam with Caltech and Caltech-color pretraining. (c) (d) Test accuracy and convergence of transferring to MIT-indoors and CUB-200 from models pretrained on large-scale ImageNet and Places, respectively.

(a) Accuracy of pretraining.(b) Accuracy on the target dataset.

Figure 9: Transfer performance w.r.t. the pretraining epochs from Food-101 to CUB-200. (a) Accuracy of pretraining task and (b) Accuracy on target dataset by fine-tuning from models pretrained for different numbers of epochs. Although accuracy of pretraining increases stably, transferability increases only during the early epochs and decreases afterwards.

SETUP Denote by σ(•) the ReLU activation function, σ (z) = max{z, 0}. I{A} is the indicator function, i.e. I{A} = 1 if A is true and 0 otherwise. [m] is the set of integers ranging from 1 to m. Consider a two-layer ReLU network of m hidden units f W,a (x) = 1 √ m a σ(W x), with x ∈ R d as input and W = (w 1 , • • • , w m ) ∈ R d×m as the weight matrix. We are provided with n

VISUALIZATION OF LANDSCAPES AROUND CONVERGENCE FOR DIFFERENT DATASETS (a) Randomly initialized. (b) ImageNet pretrained.

Figure 10: Comparison of landscapes centered at the minima on Food-101 datasets with ResNet-50.We use the filter normalization(Li et al., 2018a)  to avoid the impact of weight scale. Randomly initialized networks end up with sharper minima, while pretrained networks stay in flat regions.

Substitute equation 44 into equation 42, we have

Transferring to different datasets with ImageNet pretrained networks fixed or fine-tuned.

Table content: <table><tr><td/><td>(fixed)</td><td>(fixed)</td><td>(fine-tuned) (fine-tuned)</td></tr><tr><td>Webcam</td><td colspan="2">0.00±0 0.45±0.05 0.096±0.007</td><td>0.00±0 0.45±0.09</td><td>0.94±0.10</td></tr><tr><td colspan="3">Stanford Cars 0.00±0 24.2±0.73 0.165±0.003</td><td>0.00±0 13.95±0.44</td><td>3.70±0.26</td></tr><tr><td>Caltech-101</td><td colspan="2">0.00±0 6.24±0.37 0.059±0.005</td><td>0.00±0 4.57±0.40</td><td>1.22±0.21</td></tr><tr><td>Synthetic</td><td colspan="2">0.03±0.01 0.81±0.19 0.015±0.003</td><td>0.00±0 0.75±0.11</td><td>0.65±0.12</td></tr><tr><td>CUB-200</td><td colspan="3">0.15±0.03 35.10±0.50 0.262±0.006 0.04±0.01 21.04±0.36</td><td>2.38±0.33</td></tr></table>

Performance of training from scratch.

Table content: <table><tr><td>Dataset</td><td colspan="2">test error fine-tuning improvement 1 √ n</td><td>l W (l) -W 0(l) F</td></tr><tr><td>Webcam</td><td>26.39±0.41</td><td>98.29%</td><td>3.54±0.53</td></tr><tr><td colspan="2">Stanford Cars 46.25±0.76</td><td>48.75%</td><td>5.95±0.38</td></tr><tr><td>Caltech-101</td><td>22.14±0.63</td><td>79.36%</td><td>2.34±0.34</td></tr><tr><td>CUB-200</td><td>52.70±0.77</td><td>60.15%</td><td>3.90±0.61</td></tr><tr><td colspan="3">B.4 COMPARISON OF THE LOSS LANDSCAPES IN EACH LAYER</td><td/></tr></table>

Peer review (iclr2020):
I thank the authors for the clarifications and the modifications to the paper. However, I still lean towards rejection. While the authors provided detailed explanations on some of my points here, most of these are still not in the paper, so the reader would still probably be confused. There are new figures in the appendix, but they are not referred to from the text. The paper feels like a hastily compiled collection of weakly related, somewhat anecdotal and not necessarily clearly explained and motivated experiments. While I believe there is interesting content, with the current presentation style it is really difficult to digest. As an advice, I think it may be better to not squeeze in all the results into one paper, but rather focus on some aspects and analyze them really well, with clear explanation, motivation, and preferably with demonstrating practical gains that result from the analysis - not just hypothetical, but supported by experiments. --- The paper studies transfer of representations learned by deep networks across datasets and tasks. Namely, the paper analzes the standard setup with pre-training on one dataset and further fine-tuning on a different dataset. Results include both experiments and theory. On the experimental side, the focus is mainly on TODO. On the theory side, an analysis of transfer in two-layer fully connected networks, based on the ICML-2019 work of Arora et al., is proposed. I lean towards rejecting the paper, since the presentation and the technical quality are somewhat substandard. This is mainly based on evaluation of the experimental results, since I am not an expert in this subfield of theory and was therefore not able to check the statements and the proofs thoroughly. On the presentation side, many details are missing, which often makes understanding difficult and will likely lead to the results not being reproducible. On the experiments side, the issue is that they are quite anecdotal and have multiple confusing or questionable details, as specified below. Pros: 1) An interesting topic of trying to understand generalization and transfer in deep learning 2) Multiple types of experiments, including visualizations of loss landscapes at convregence and at initialization, plots of Hessian eigenvalues, measuring the deviation of the weights from their initial values, measuring the variance of the gradients of the weights, measuring the transfer between different datasets, measuring the transfer performance depending on the durarion of pre-training. 3) Theoretical analysis. As mentioned, I cannot really judge about the quality of the results. Cons: 1) The presentation is quite confusing. 1a) The paper includes many different experiments as well as theory, and it is not very clear how these all come together and what message do they give to the reader. The paper states at one point that it *may stimulate further algorithmic advances*, and it would be great if there was a bit more elaboration on this. 1b) Experimental methodology is not presented in the main paper and not referred to. Some of it is described in the appendix, but also not too detailed, for instance the duration of ResNet training is not clear, the details of loss landscape visualization are confusing (for instance, the phrase *... i.e. 0.1× gradient and a total of 200 × 200 grids*), 1c) The paper is over 9 pages, which is more than the recommended length of 8 pages. 1d) Scaling in Figure 8(b) is quite suboptimal, it is impossible to read the test accuracy results. 1e) Minor issues: - No color-code in Figure 3 (unclear what values do the colors correspond to) and it does not seem to be referred to in the text. - Page 5: *predictive* -> *predictable*? - Page 6 *While pretraining on sufficiently large datasets...* - I do not think the experiments tell anything about the dependence of the effect on the size of the dataset, so this phrasing is not justified 2) I found many of the experiments confusing or unconvincing. This is partially affected by the aforementioned issues with presentation. 2a) In Table 1 and Figure 2, it is unclear if difference in generalization between the dataesets is due to similarity to ImageNet (by the way, ImageNet is only mentioned in the caption of Table 1, but not in the text) or due to the inherent properties of the datasets (perhaps some are naturally more difficult or prone to overfitting). I think numbers for training from scratch would be helpful for disambiguating the two cases. 2b) It is unclear why is the norm difference normalized by the square root of the number of target examples. This choice is not justified and it seems it can very noticeably affect the results in counter-intuitive way. For instance, if I understand right, if one duplicates each example in the training set, these values will change, which seems somewhat counter-intuitive. Would it make more sense to normalize by the initial norm of the weights? 2c) In Figure 4, it is unclear if the permutations of the weights are accounted for. The output of a deep network is invariant under a wide variety of weight permutations, and it is natural that networks trained from different random initializations may converge to weights permuted in different ways. In order to meaningfully compare the weight vectors of these networks, one would have to first *align* them, which is a non-trivial task. Another issue I have with the results on Figure 4 is that I don*t find them all that counter-intuitive: it seems natural that the weights stay relatively close to the pre-trained network when fine-tuned for that (paritally because the aforementioned permutation symmetry is already largely broken during pre-trianing). 2d) It is unclear which downstream task is analyzed in Figure 6. Moreover, the plot seems to mix two factors: magnitue of the gradient and the smoothness of the loss landscape. Would it not make more sense to make fixed-size steps in the direction of the gradient? Moreover, I am wondering if the difference between the loss values is simply because the overall loss magnitude is larger for the randomly initialized network? 2e) A few parts of subsection 5.1 are confusing. It is not clear where does the interpretation of Figure 8 follow from (perhaps partially because figure 8(b) is partially difficult to read). psi and W are not defined. What does it move that the weight matrices *show no improvement*? Where are the results on Caltech-101 and Webcam? Why is Food-101 mentioned in the text, but CUB-200 shown in the plots?