How to choose the right transfer learning protocol? A qualitative analysis in a controlled set-up
Authors: Federica Gerace
Diego Doimo
Stefano Sarao Mannelli
Luca Saglietti
Alessandro Laio
Venue: Accepted by TMLR ()

Abstract: Transfer learning is a powerful technique that enables model training with limited amounts of data, making it crucial in many data-scarce real-world applications. Typically, transfer learning protocols require first to transfer all the feature-extractor layers of a network pre-trained on a data-rich source task, and then to adapt only the task-specific readout layers to a data-poor target task. This workflow is based on two main assumptions: first, the feature maps of the pre-trained model are qualitatively similar to the ones that would have been learned with enough data on the target task; second, the source representations of the last hidden layers are always the most expressive. In this work, we demonstrate that this is not always the case and that the largest performance gain may be achieved when smaller portions of the pre-trained network are transferred. In particular, we perform a set of numerical experiments in a controlled setting, showing how the optimal transfer depth depends non-trivially on the amount of available training data and on the degree of source-target task similarity, and it is often convenient to transfer only the first layers. We then propose a strategy to detect the most promising source task among the available candidates. This approach compares the internal representations of a network trained entirely from scratch on the target task with those of the networks pre-trained on the potential source tasks.

Published in Transactions on Machine Learning Research (07/2024)
How to choose the right transfer learning protocol? A qual- itative analysis in a controlled set-up
Federica Gerace
federica.gemacee
Scuola Internazionale Superiore di Studi Avanzati (SISSA) Department of Mathematics, University of Bologna
Diego Doimo Area Science Park
ddego.Löinn@@uraaciineeee...
Stefano Sarao Mannelli
s.s.momameelii@@ddl.ccccc
Gotsby Computational Neuroscience Unit & Sainsbury Wellcome Centre University College London
Luca Saglictti
Department of Computing Sciences
Bocconi University
Alessandro Laio
laio Wsisa.
Scuola Internazionale Superiore di Studi Avanzati (SISSA)
Abdus Salam International Centre for Theoretical Physics (ICTP)
Reviewed on OpenReview: https: // openreview. net/ forum? id- XWQgXLYuu2
Abstract
Transfer learning a powerful technique that enables model training with limited amounts of data, making it crucial in many dala-scarce real-world applications. Typically, transfer learning protocols require first to transfer all the feature-extractor layers of network pre- trained data-rich source task, and then to adapt only the task-speeific readout layers to data-poor target task. This workflow is based two main assumptions: first, the feature maps of the pre-trained model are qualitatively similar to the ones that would have been learned with enough data on the target task; second, the source representations of the last hidden layers are always the most expressive. In this work, we demonstrate that this is not always the case and that the largest performance gain may be achieved when smaller portions the pro-trained network are transferred. In particular, we perform set of numerical experiments in controlled setting, showing how the optimal transfer depth depends non-trivially the amount of available training data and on the degree of source- target task similarity, and it is often convenient to transfer only the first layers. We then proposc a strategy to detect the most promising source task among the available candidates. This approach compares the internal representations of a network trained entirely from scratch on the target task with those the networks pre-trained on the potential source tasks.
1 Introduction
Machine learning models show a remarkable capacity to extrapolate rules and predict the behavior complex systems. Still, this ability often comes at the cost of training with large amounts of data. In various domains such as in medical applications data collection is a slow and costly process. This goes at odds with the typical vast variability across data, which inevitably requires the usage of huge datasets to achieve satisfying

---

Published in Transactions on Machine Learning Research (07/2024)
generalization performance on desired task Beam & Kohane (2018): Rajkomar et al. 2019 Data efficiency
is thus a necessary condition.
Transfer learning emerged as an efficacious mitigation strategy to this problem Thrun & Pratt 2012); Shin
et al. p Raghu et al. (2019). Transferring the already meaningful representations learned on the source task to a network that has to solve given target task allows it to work with dramatically smaller dataset sizes while keeping a comparable level of accuracy, With little data at disposal, one of the major risks is to incur overfitting Geirhos et al. (2020). However, transferring the layers directly from another network allows, in principle, to filter out irrelevant information present in the input of both the source and the target data sets and work with a representation of the data of reduced efficacious dimension. This allows a significant increase in data efficiency.
Unfortunately, deep neural networks tend to be sensitive even to tiny distribution shifts Gama et al. (0144) Therefore, it is essential to perform transfer Icarning starting from the most suitable pre-trained network or, in other words, from the most convenient parameter initialization. This becomes even more relevant when the transfer learning pipeline is followed by a further stage of fine-tuning, during which the transferred feature map is unfrozen, and the whole network is trained on the target set Bengio (2012). This step is performed under the assumption that the features are already appropriate, and one can use smaller learning rates to avoid overfitting
Fig. [ embodies the main motivation of this work: it shows that in a realistic data scenario transferring all the layers up to the last hidden representation does not always lead to the best performance. In particular, the two panels show the optimal test. accuracy as a function of the transfer depth, namely the total number of frozen layers, when the source task is Imagenct and the target task is either a datasct of Breast Cancer images (panel (a)) or the Retinal Fundus Multi-Disease Image dataset (panel (b)). As we can see, while in the first case the optimal accuracy can be achieved by keeping almost all the layers frozen, in the second case it is instead necessary to retrain larger portions of the network. In the following, we will refer to these curves as defrosting profiles.
Since transfer learning can be so sensitive to parameter initialization, it is then natural to ask: How could we detect which is the source pre-trained network whose features are the most promising to transfer?, how should one choose the layers that should be kept frozen? In other words, is it more convenient to transfer all the layers or just a portion of them? Tdeally, one should be able to identily, in the source network, layer in which the representation is general enough to be meaningful also for the target task but not too general, as otherwise training it to a specialized task might be hindered by the scarcity of data in the target set.
In this work, we start addressing these questions with the following main contributions:
We design a controlled experimental framework through which we show that the optimal transfer depth strongly depends on the source-target distribution shift and on the size of the target set;
We propose a simple protocol for the identification of the optimal transfer depth (see sec. 3.1.1
We propose a strategy detect the most promising source task by measuring the topological simi- larity between the internal representations of a network pre-trained on candidate source task with a second one trained fully from scratch on the target set.
In particular, in sec. 3.1 we describe the experimental framework leading to the defrosting profiles as the ones displayed in Fig. In sec. 3.1.1] we see how this framework can also be considered as protocol for identifying the optimal transfer depth, which call incremental defrosting. As we will see, one of the key advantages of this protocol is its simplicity. Indeed, no learning heuristics need to be employed to ensure the retention of useful information obtained in the pre-training phase. In sec. 3.2 and sec. 3.3 we also show how the shape of the resulting defrosting profiles is strongly affected by the amount of data in the target training set but also on the specific choice of the source task respectively. Given that the degree of source- target similarity can consistently impact transfer learning performances, up to the point of inducing negative transfer effects Gerace et al. (002)), in see. 3.4 we also propose strategy to select the most promising source task among all the possible candidate ones.
2

---

Published in Transactions on Machine Learning Research (07 (022)
ImageNet
ImageNet Retina
0.850
0,855
6.800
0.835
6.830
13
30
60
depth
scashore
benign
mall ghalt
labrador
wastbaon
berign
Figure 1: Diverse effectiveness of freezing protocols The figure shows instances of transfer learning tasks: from Imagenet Deng al. (2099) to dataset composed of Breast Cancer images JJnowezyk &
Madabhushi 2016:; Cruz-Roa al. (2014) (pancl (a)) and from ImageNct to the Retinal Fundus Multi- Disease Image Datasel Pachade et al. (2021) (panel (b)) (see SM for further details concerning the two datasets). In the first transfer experiment, we used a ResNet-18; in the second one, we used a ResNet-50. More details can be found in the Architecture and training protocol paragraph. We can see samples each dataset the lower part the panels, while the curves above the defrosting profiles represent the test performance of the network. Each point of the curve the accuracy reached when the first. layers have been transferred. The simulations are averaged over 20 (leftmost panel) and 5 (rigthmost panel) different realizations.
1.1
Related Works
Deep neural networks (DNNs) can build surprisingly general representations in various ways Deng et al.) 2009 Golorio-Fernánder al. 20200); Jaiswal et al. (2020). A possible explanation for their effectiveness comes from their 'implicit bias" towards rogularized solutions Neyshabur al. (2014). Numerous works provided evidence of preference for simple solulions Rahaman el. al. (2018); Pérez el al. (2019): Shah et al. (220)); Abbe et al. (2021), and it was shown that learning complex functions may require training the networks ks for longer times
Goldt & Ingrosso [022]: Refinetti et al. 2023
The layered nalure DNN architectures seems to play a crucial role, as appears that increasingly complex features are progressively encoded in deeper layers of the network Kornblith et al. (2019); Abbe al. (2021). Interostingly, the interaction between input data and labels has been shown to lead to non-trivial and often abrupt changes in the nature of the extracted features throughout the layers Tishby & Zaslavsky (2015 Doimo et al. (2020). Recent investigations highlighted the importance of intermediate representations Yosinski et al. (2014). 4), but the question on how to select transferable DNN representations Raina et al. 20070:s Long et al. (2015) remains largely open, given the complex dependence on the specific source-target distribur ion shifts Wenzel et al. (2022 and on the data scarcity Lee et al. (2022). The theoretical insight on transfer lcarning phenomenology in DNNs remains limited. Except for some results on training convergence rales Du et al. (2020), the simplifying assumption of linear activation function was necessary to obtain exact results on the generalization properties after transfer, both in the asymptotic Dhifallah & Lu (2221 Dar & Baraniuk (2022) and in the dynamical Lampinen & Ganguli (2099) regimes. Currently, systematic exploration of the generalization gain as a function of the oource-Larged distribution shift in non-linear neural networks only possible in the case of fully connected 2-layer networks Gerace el al. (202)).
The present work explores the less charted area. between theory and real-world transfer learning applications through carefully designed framework that allows some degree of control over the data distributions. The main goal is to investigate the connection between data and architecture, pinpointing the connection between
3

---

Published in Transactions on Machine Learning Research (07/2024)
distribution shift and sample size one side and the quality of intermediate representations for transfer task on the other. In particular, we show that, depending on these two factors, it is possible to identify an optimal number of layers to transfer. We provide evidence that this preliminary step can provide better initial condition for subsequent finc-tuning stages, as shown in sec. 3.1 Our findings are complementary to what already known about fine-tuning. For instance, it has been shown that fine-tuning the whole architecture seems to be beneficial only the data-rich regime Wenzel et al. (2022). Similar obscrvations can be made in the context of differentially private models Tobaben et al. (2023), where the authors notice how FiLM adapters are often competitive with learning just the final classifier layers or the entire network. Therefore, it is fundamental to understand whether the whole transferred information is always needed. For instance, from the perspective of knowledge distillation in a self-supervised learning context, it has been shown that distilled models are lypically capable of surpassing the best available general purposed features of larger models in computer vision tasks Oquab et al. (023)).
In this work, we corroborate these results by investigating the case where either the whole network or just the head is adapted the larget set, and when a non-trivial number of layers is kept frozen to the source configurations. We believe this can provide a better starting point for later fine-tuning and knowledge distillation protocols. We conduct part of the experiments while having full control of the moments of the input data distribution shared between the source and target tasks. This allows us to directly relate the statistics of the input data distribution to the effectiveness of Transfer Learning protocols.
2 The Experimental setup
Architecture and training protocol We consider four different types of architectures. The first and mostly employed in our experiments is Wede-Resseet-2844 architecture Zagoruyko & Komodakis (2016). organized in 3 identical groups, each containing 4 blocks of alternating convolutional, batch-norm, and downsampling layers, plus a final batch norm and a fully connected one. This architecture is pretty simple but still reaches accuracies in image classifications close to the state-of-the-art Cubuk et al. (2020); Han et al. 2017p. Therefore, it can be considered a good starting point for a generic image classification task. Moreover, wide versions of ResNet have been shown to foster a good degree of compatibility between model reprosentations, whose corresponding similarity was demonstrated to increase with the width of the layers Kornblith et al. (2019). For the other lests, we also use base version of the Vision Transformer Dosovilskiy et al. (2020) with an imput patch size of 16, a ResNet-18 18 and a ResNet-50 from the PyTorrh model zoo, all pre-trained on the ImageNet dataset He ct al. (0016).
We train the Wide-ResNNet-2844 and the ResNet-18 for 200 epochs with stochastic gradient descent with momentum at 0.9, batch size of 128, weight decay at 5 10-4, a cosine annealing scheduler for the learning rate, starting from a value of 0.1 Zagoruyko & Komodakis (2016); Loshchilov & Hutter (2016) and no data augmentation. Concerning the experiments involving the ResNet-50 architecture,
instead use the same learning protocol as in Dominik Müller & Kramer 2021). We train the ViT for 50 epochs, with batch size of 64, learning rate 10-4, weight decay of 10-1, and Adamn optimizer Kingma& & Ba (2014). J. We provide the code to reproduce our experiments and analysis at htttps://anonymous.. For additional details, see supplementary material (SM) B.1
wcwence/ Taaicce////// I/TL gradient--DDe. DD27//
Synthetic datasets To investigate the factors at play in determining the transfer learning behaviors in- depth, we consider three types of CIFAR-10 clones. These clones are meant to form hicrarchical family of datasets, approximating the true underlying distribution of CIFAR-10 with increasing fidelity.
Following Refinetti et al. 2023), the first level of the hierarchy (IsoGM) is obtained by fitting CIFAR-10 with a mixture of isotropic Gaussians and matching the first moments with the CIFAR-10 distribution. At the second level of the hierarchy (GM), also the covariances are learned from CIF To go beyond the second moment, we propose to obtain finer approximations of CIFAR-10 10 images by using a deep autoencoder architecture consisting of convolutional encoder network and a deconvolutional decoder, both connected to a bottleneck layer of variable size (we refer to SM for additional details). Note that, by construction, this setup is devoid of the confounding effect of misalignments between labeling rules Gerace et al. 2022
4

---

Published in Transactions on Machine Learning Research (07/2024)
Lee et al. (022) in the source and target tasks since the synthetic datasets share the same label structure, independently of the alterations induced in the input distributions.
Real-world datasets We analyze the defrosting profiles for transfer and fine-tuning setups for three real- world datasets for image classification: the Food101 dataset Bossard et al. (2014). a dataset of Retinal Images Pachade ct al.] (2021), and a Breast Cancer dataset Janowczyk & Madabhushi (0016); Cruz-Roa et al. (2014). Food101 comprises 101 classes of different lypes of foods; the Retinal Dataset has 46 different types of retinal pathologies, but we treat the task as a binary classification of the presence and absence of
discase. Similarly also, in the Breast Cancer datasct, the task is binary classification task. In all cases, report the accuracies of the defrosting profiles the test sets. More information about the dataset construction can be found in Sec. B.1 of the Appendix.
Information imbalance One of the main factors affecting whether layer can be effectively transferred to target network is the similarity between the source and target feature maps. To measure this quantity, we use the Information Imbalance (II), a topological mcasure recently proposed in Gliclmo et al. (2022), and estimate if a given feature map can be used to predict another feature map. To compute the II, we first take both a pre-trained network on a given source task and a network trained entirely from scratch on the target task. We then make the two networks process the target test set by extracting the corresponding internal representations at each layer of both networks and for each test example. Since feature maps are typically very high-dimensional, estimating the conditional entropy between the two representations would be impractical. The II can be seen as computationally viable proxy for entropy and other information- thcoretical quantitics.
The IT estimated as follows. Given a layer 1 and a test data point μt, we first find the closest data point p' according to their Euclidean distance in the source representation. Then, in the target representation of the same data set, we count the number of data points that are closer to μ than j². The II is proportional to the average of this number. Formally
A (S = (r (IT =1)
with M being the total number of samples in the test set, TJ and S being the rank matrices associated with the representations in target and source networks respectively (therefore r is equal to 1 if μ and μ²' are first neighbors). The average (.) is taken over all the data points in the larget test set. The II is, therefore, a measure of the local similarity between the internal representations space of the source and target task. Tn particular, if two local neighborhoods are approximately equivalent, one finds A ~ In general, the more the first representation is predictive with respect to the second, the smaller the II is. In the limit where the source local neighborhood does not provide any information on the target one, one will find A 1 Glielmo et al. 2002)). Note that, crucially, the II circumvents all the known ambiguities caused by re-scaling, permutations, and symmetries in neural network functions since this measure relies solely on the distance ranks, which by definition are invariant to the transformations implemented by neural networks.
3 Results
3.1 Experimental framework to identify the optimal transfer depth detection
To identify the optimal transfer depth for given source-target pair, perform a series of transfer learning cycles, exemplified in the cartoon of Fig. 2 In particular, we start from a completely frozen setting, except the output layer. We then gradually increase the number of layers to be defrosted and ro-trained, starting from the very last. layer up to the early layers in the network. At each fleration of this procedure, to avoid vanishing gradient scenarios, we re-initialize the defrost layers to random values (following typical heuristics He et al. (2015)), and we then train them from scratch on the target training data, with the hyper- parameter setting described the architocture and training protocol paragraph. The defrosting profiles in Fig. 2 show the validation accuracy recorded each iteration of this procedure, starting from the transfer
5

---

Published in Transactions on Machine Learning Research (07 (07/2024) /222)
0.01
0.05
0.62
2.00
0:c0
twet
0.05
10
20
10 borsftr sopth
26
tarafer degth
Figure 2: Incremental defrosting procedure. The cartoon illustrates the transfer procedure employed in the experiments, where the first layers are transferred from the architecture trained on the source, and only the last layers are fine-tuned to the target set. On the right, with a star, we show the accuracy reached on the larget set by network trained up to the kth layer.
scenario where all layers are kept frozen except the final read-out (leftmost pancl), up to the transfer sctting where the whole network is let lo adapt to the larget lask (rightmost panel).
While it is common practice to transfer all extrator layers except for the fully-connected segment (or even just the readout layer), in many scenarios transferring a smaller number of layers may largely improve the downstream accuracy. This is, for instance, the case of the example shown in Fig. 2 where we highlight the optimal transfer depth with a red star. As we can see, in this transfer setting where the source task is GM while the target task is CIFAR-10, it is neither convenient to transfer all the feature-extractor layers nor to train the entire network entirely from scratch.
As shown the left panel of Fig. 3 this behavior is observed even with a Transformer. The plot shows the defrosting profile of a Vision Transformer pre-trained on Imagenet and then adapted to the Retinal Fundus Multi-Diseass Image Dataset (Pachade et al. (2021), top) and the Food101 dataset (Bossard et al. (2014). bottom). As for convolutional neural networks, even in this case, training from scratch or transferring all the blocks is not the optimal transfer learning strategy. The optimal performance is obtained by keeping the first. 7-9 layers frozen.
There are at least two reasons for the performance deterioration when sub-optimal number of layers is frozen. On the one hand, one can see defrosting profile that decreases when too many layers are kept frozen. This is natural when the input /ouppu mappings associated with the two tasks are too dissimilar: trained network will not retain all the information contained in the data points it learns to select the meaningful features for classification. Some information that is relevant to the target task might be dropped completely by the pre-trained model, given the presence of non- non-linearities and down-sampling operations throughout the layers. On the other hand, when the target dataset is small, one can see a performance deterioration if too many layers are defrosted and re-trained on the new data. If the ratio of training samples and tunable parameters becomes too small, one can trace over-fitting behaviors, as the model can not choose a good generalizing solution among the vast number of zero training error models.
The effect of fine-tuning. Identifying the optimal number of layers transfer can be beneficial for later fine-tuning stages. This is shown in Fig. 3 In particular, we display the test accuracy achieved by Vision Transformer the Retina (top) and on the Food101 (bottom) datasets when increasing number of layers is kept frozen to the ImageNet feature map, while the remaining ones are re-trained on the target set starting from random initialization (transfer only on the left) or fine-tuning the source feature-map (fine-tuning on the right). In both cases, fine-tuning drastically improves the generalization performances at early frozen layers if we compare the resulting defrosting profile with the one obtained at random initialization. However, we can still identify for both datasets an optimal transfer depth in the same region of the optimal depth detected at random initialization (layer 6 for Retina and layer 7 for Food101). At the optimal transfer depth,

---

Published in Transactions on Machine Learning Research (07 2024)
ImageNet - Retina: Transfer Only
ImageNet + Retina: Fine-Tuning
0.90
0.90
best accuracy
best accuracy
0.85
0.85
0.80
0.80
0.75
0.75
0.70
0
1
2
4
5
7
8
10111213
0
1
2
3
4
5
6
8
9
10111213
transfer depth
transfer depth
ImageNet Food101: Transfer Only
ImageNet Food101: Fine-Tuning
0.750
best accuracy
0.6
best accuracy
0.725
0.4
0.700
0.675
0.2
0.650
1
5
10111213
2
5
10111213
transfer depth
transfer depth
Figure 3: Optimal transfer depth and fine-tuning. The figure shows Vision Transformer pre-trained ImageNet and then adapted to the Retinal Fundus Multi-Disease Image Dataset (top) and Food101 dataset (bottom). Each point of the curve is the validation accuracy reached when the first / layers have been frozen, and the remaining ones are re-trained from a random initial condition (leftmost panel), finc-tuning architecture from
from the ImageNet configurations (middle panel) these experiments, we use Dosovitskiy al.] (2020). More details can be found in the Architecture and training protocol paragraph. The curves the defrosting profiles represent the validation performance of the network. The simulations are averaged over 3 different realizations.
the accuracy achieved higher than the one obtained starting from random initialization, meaning that the detection of the optimal number of layer transfer, combined with subsequent fine-tuning stage, allows achieve better test scores. For instance, we can that fine-tuning the optimal transfer depth improves the generalization performances that one would achieve by directly fine-tuning all the layers (first point each fine-tuning defrosting profiles), as standard practice in deep learning applications.
3.1.1 Incremental Defrosting Algorithm.
The vanilla transfer learning protocol, where only the read-out layers are retrained on the target data, may represent a cost-cffective and casily implementable stratogy. However, as we can see from the example Fig. [ for different architectures and source-target pairs, transferring all the fealure-extractor layers does not always represent the optimal strategy. Indeed, there exist transfer learning scenarios where considering different depths in the defrosting process may allow large performance gains in the downstream task.
The multiple defrosting and re-training process exemplified in the previous paragraph can be equivalently thought of as a preliminary and simple protocol for identifying the optimal transfer depth. We call this algo- rithm incremental defrosting, whose psoudocode is provided in Algorithm Since transfer lcarning mostly relevant when the target dataset is effectively small, this procedure should not require high computational costs compared with typical large-scale deep work training. Moreover, the computational cost of training

---

Published in Transactions on Machine Learning Research (07/2024)
is further reduced by pre-processing the dataset single time, up until the last frozen layer, and then just training the remaining part of the architecture a separate network receiving the frozen representations inputs. For instance, training a wide ResNet-28 on all the 50000 examples of CIFARIO from scratch takes approximately 2 hours, while training just the last layer only a couple of minutes.
Algorithm Incremental Defrosting
Target task, Source pre-trained network. Load the network pre-trained on the source task first layer do: Train on the target task freezing all layers up to layer and intializing the remaining ones at random;
Input:
For == last layer up
Evaluate the test metrics.
Return: Defrosting profile
The salient trait of the defrosting profiles is that they show a shape that generally quite regular and predictable from few points in the profile. Indeed, the main drops in the representation similarities happen after the lossy down-sampling layers (see sec. 3.4). At the same time, due to the smoothness of the curve, once the maximum has been identified, it is not necessary to compute the points in the branch between the maximum and the input layer. Therefore, in cases of limited computational budgets, the position of the maximum could also be inferred by sampling just a few transfer depths, ranging from next to last to after the first down-sampling operation.
The ident ifcatiiiiation of the optimal transfer depth by means of the incremental defrosting algorithm will then provide the best initialization for all the possible subsequent fine- tie-tuming strategies, whose first step is the choice of a source pre-trained network.
3.2 Dependency of the optimal transfer depth on the training set size
At a qualitative level, the interplay between data scarcity and transfer learning seems clear: With enough samples in the target dataset, all the relevant data features can be extracted directly from the task at hand. Instead, with a reduced sample size, learning good representations from scratch becomes impossible and transfer learning becomes thus convenient option.
We conduct a quantitative investigation of the transition between these two scenarios by considering two distinct datasets as test cases. The first dataset is the GM CIFAR-10 clone (described in sec. 2 The second one is balanced sub-sample of a publicly available medical dataset for the classification of breast cancor Janowczyk & Madablushi (2016); Cruz-Roa et al. (2014) (more on the Breast Cancer datasct in SM sec. B. Given these two da datasels, we then consider two different transfer learning experiments. In the first one, we train a Wide-RRssett-28-4 by selecting GM as the source task and CIFAR-10 as the target task. In the sccond experiment, we take instcad ResNet-18 pre-trained
ImageNet from the PyTorch model zoo, and we select as the target task the Breast Cancer dataset (this is common choice in medical applications Khan et al. (2019): Chouhan et al. (2020)). For both transfer learning experiments, we sub-sample the target data, thus spanning over a wide range of data-scarce scenarios.
Fig. shows the outcome of the experiments involving CIFAR-10 (left) and Breast Cancer (right), respec- tively. The defrosting profiles the two cases show some common features. As expected, the generalization accuracy on the target task generally increases with larger amounts of training data. However, nontrivial observation is that the optimum of the defrosting profile shifts backward in the layers as the size of the target training set is increased (darker colors). Instead of just transitioning from regime of convenient transfers (with some fixed optimal transfer depth) to one of sub-optimal transfer, we find that the number of trans- ferred layers needs to be adjusted according to the sample size, as early layers maintain a higher degree of compatibility when more data made available. Note also that optimally defrosting the pre-trained network can be more effective than employing the vanilla freezing protocol after acquiring more data. For example, in the right panel of Fig. 4 the optimal transfer with 5 x 103 samples achieves comparable performance with vanilla freezing with 10 times more data. The fact that this accuracy boost can be obtained almost
8

---

Published in Transactions on Machine Learning Research (07 2024)
GM CIFAR1O
ImageNet + Breast Cancer
best accuracy
best accuracy
0.86
0.8
0.84
0.6
0.82
0.4
0.80
20 transfer depth
10
01
9 11 15 transfer depth
Figure 4: Effect of dataset size. The figures show the defrosting curves obtained in two transfer learning tasks for different target set sizes: from GM to CIFAR-10 and from ImageNet to Breast Cancer. In the left panel,
a Wide-ResNet-28-4, while in the right panel, ResNet-18. More details can be found in the paragraph Architecture and training protocol. The darkness of the curve represents the set size, the darker the larger. In particular, in the GM to CIFAR10 transfer, the different shades of red correspond to the training set sizes 22 2*, 2,,, x 10º} per class, while, in the ImageNet to Breast Cancer transfer, the different shades of green correspond to the training sizes (5 x 102, 5 103, 10 x 10*) per class. The siruulations are averaged over 20 and different realizations the top and bottom panels, respectively.
IsoGM + CIFAR10
GM CIFAR10
L256 ~ CIFAR10
0.65
0.70
0.85
best accuracy
0.60
best bestacco accuracy
best accuracy
0.80
0.65
0.55
0.75
0.50
Ship
0.60
Ship
0.70
Ship
0.45
0.40
0.65
10
20
10
20
10
20
transfer depth
transfer depth
transfer depth
The four pancls show three different CIFAR-10 clones to CIFAR-10 transfer directions in increasing levels of CIFAR-10 approximation: IsoGM, GM, and L256. The figure shows the corresponding defrosting profiles at 384 training samples per class. It provides evidence of how better approximating clones convey richer features, and therefore, is best transfer larger portion of the architecture. Each of the three insets displays prototypical example of how the images look in the corresponding clone. The network architecture is be found in the paragraph Architecture and training protocols. The simulations are averaged over 5 different realizations.
Figure 5: Defrosting profile for CIFAR-10 clones.
More details
for free, just by estimating a defrosting profile for a given transfer task, is one the main messages this work.
3.3 Dependency of the optimal transfer depth on source-target similarity
We now investigate the role played by the statistical similarity between the input distributions the source and target tasks determining the optimal transfer depth. Comparing experiments with no control over the input statistics and with different labeling rules may be insufficient answer this question, as these factors cannot be easily disentangled. With the hierarchy of generative models described in Section 2 instead,

---

Published in Transactions on Machine Learning Research (07/2024)
can isolate this crucial aspect of transfer learning. In this controlled synthetic framework, we can gradually move from a scenario where only the first moment of the source and target task distribution are matched (IsoGM), to the case in which the first two moments are matched (GM), up to the scenario where the source and target distributions almost perfectly match (L256).
Our results are summarised by the incremental defrosting profiles in Fig. at the same training set size. As expected, if the similarity of the two data sources is insufficient (left panel), fully retraining the network and giving up the transfer learning approach is preferable, even in the data-scarce regime. At pre-training, the early layers of the network have adapted to significantly different input distributions and are thus unsuitable to extract the information needed to solve the target task. As the similarity between the two tasks increases (middle and right panel), the optimal cut for the defrosting protocol shifts towards the later layers of the network. In the SM, Fig. | complements this picture by showing how the defrosting profiles at different sizes of the training set can significantly change depending on the relatedness between the source and the target task. Indoed, when the source and the target task are poorly statistically correlated, we can observe negative transfer effects even in an extreme data scarcily regime. On the other hand, when the two lasks are extremely correlated, it is almost always convenient to transfer up to the very last layer, except when one can count on a huge number of training examples.
From this analysis, two relevant and related aspects emerge. First of all, depending on the complexity of the source task and its relatedness to the target one, we can observe different scenarios, ranging from always negative to almost always positive transfer. It is then natural to ask whether it is possible to design a strategy to identify the most promising source task among several other candidates. The second aspect is that the experiments in Fig. strongly hint at the fact that there is precise order in which the layers learn the different moments of the input data distribution. Indeed, if we consider the way CIFAR10-clones have been designed, it seems that the first layers are just focusing on capturing the first and second-order statistics of the data distribution, thus playing a more generic image pre-processing role Yosinski et al. (2014), compatible with multiple downstream classification tasks on similar input distributions. On the contrary, deeper layers are more sensitive to high-order moments. We discuss these two points in the next section.
3.4 Source Task Detection
The experiments in Fig. 5 highlight how crucial is the choice of the source task in transfer learning protocols. Indeed, the generalization performances of the target network can consistently improve or downgrade, simply depending on the initial conditions or, in other words, trained network is. It is then natural to ask: How can we choose the most promising source task among several candidates? In principle, we would expect that similar tasks induce similar internal representations in the noural networks trying to solve them. To select the most promising source task, we propose quantifying the similarity level between the internal representations of the network pre-trained on the source task and the one trained from scratch on the few examples in the target set with the Information Imbalance (IT) described in Sec. 2
how closcly related to the target the source pre-
Fig. G shows how the II can be used as good proxy to identify the most promising source task when the target task is CIFAR-10 and the available source candidates are the three clones IsoGM, GM, and L,256. In particular, the left panel of Fig, G shows the II computed at each layer | in the context where the target task is CIFAR10 and the source task is one of its clones, e.g. IsoGM (magenta curve), GM (blue curve) and I,256 (green curve). As can be seen, the CIFAR-10 clone closer to CTFAR-10 data distribution (e.g. T.256 green curve) is also the one closer in terms of internal network representation and, at the same time, the one leading to the best defrosting profile, as shown in the right panel of Fig. G By simply looking the II profile, can then predict which source candidate leads to the more informative feature map for solving the target task.
A further thing to notice is that II generally increases as a function of the depth, but its rate of increase depends on the similarity between the datasets on which the networks to be compared have been trained on. The main II jumps happen correspondence with the down-sampling layers in the network, where some information that might be relevant for one of the tasks is filtered out in the other task, inevitably inducing different local neighborhoods in the representations. As already pointed out in sec. 3.1.1 this
10

---

Published in Transactions on Machine Learning Research (07 (07/2024) 0024)
0.8
Information imbalance
0.6
0.8
best accuracy
0.7
0.4
0.6
0.2
0.5
S: IsoGM, T: CIFAR1O
0.4
S: L256, T: CIFARIO
0
20 convolutional layer
10
30
10 transfer depth
20
Figure Role of the topological similarity of representations In the (top) panel, we display the Information Imbalance (II) between CIFAR10 and three CIFAR10 clones to be employed source datasct: IsoGM, GM, and the largest autoencoder with bottleneck 256. On the (botlom), we display the incremental defrosting profile obtained in these three transfer learning scenarios, assuming the source and target datasets are full (50K images each). The simulations are averaged over 5 different realizations. In the bottom panel, the error bars are smaller than the point size.
allows us to infer the optimal transfer depth by sampling just, few transfer depths rather than identifying the whole defrosting profile. Finally, the II seems to suggest the existence of a hierarchical order according to which deep neural network layers learn the increasingly higher-order moments of the input data distribution. Indeed, the topological similarity in the case of L256 is almost perfect until the very last layers, indicating the presence of small differences in the high-order features of the two data distributions. Instead, in the case of GM,, we see that the representations are almost equivalent up to the 10-th layer and become increasingly different thereafter. Finally, the IsoGM case shows that the difference training data affects the learning process dramatically, inducing visibly dissimilar representations that are incompatible with good generalization performance. This finding complements the picture presented in Goldt & Ingrosso 2022), where the learning dynamics is shown to learn the different moments of the input distribution in a precise temporal order.
the statistical distribution of
There are many alternative similarity indices that can be measured in place of the II, each one clearly unveiling the connection belween the topological similarity the representations and the transfer learning behavior observed in the incremental defrosting profile. In SM C we also show the results for the CKA Kornblith et al. (2019), the neighborhood overlap Doimo al. (2020) and the Spearman correlation of local neighborhoods, which provide qualitatively similar information.
4 Discussion
In the present work, we propose an experimental framework able to identify the key factors determining the number of layers to keep frozen in transfer learning settings. This framework can also be considered a simple algorithm to improve the accuracy of networks trained via transfer learning. We call this algorithm incremental defrosting. For data-poor target tasks, incremental defrosting can be run a negligible compu- lational cost. However, we further propose strategy that can be employed to reduce its cost also for larger target data sets. Thanks to the controlled experimental framework, depth depends non-trivially on the amount of training data and on the similarity between source and target tasks and that its identification can be more impactful on downstream performance than acquiring 10 times more training data. Finally, we have proposed a strategy to identify the most. promising source task based on topological similarity measures of network representations.
have shown that the optimal transfer
We show that the incremental defrosting algorithm and the source task detection strategy can make a difference in real-world transfer learning applications, as it allows squeezing the best accuracy for given
11

---

Published in Transactions on Machine Learning Research (07/2024)
architecture in elementary and intuitive manner. Moreover, when used in conjunction with subsequent fine-tuning stages, both protocols can overall improve fine-tuning performance, especially if we consider that the small learning rates employed in fine-tuning stages prevent the learning dynamics from deviating too much from the initial condition dictated by the source network. Therefore starting from the most convenient initialization is definitely crucial for transfer learning effectiveness.
Recent developments in the theory of neural networks Jacot et al. (2018); Canatar al. (2021) achieved exact descriptions of learning neural networks in different regimes. However, there is still lack of math- ematical tools for analyzing the role played by representations in the Peature-learning regime relevant to transfer learning. Our work provides calibrated experiments that showed the topological similarity between representations obtained via independent training on source and target tasks is a qualitative indication of the suitability of a representation transfer between tasks. This highlights the strong interplay among feature learning, dataset similarity and transfer learning effectiveness. On the theoretical side, first stop in the direction of understanding this interplay has been already done, for two-layer neural networks, Gerace el. al. (2022); where, using tools from statistical physics of learning, the authors show that negative transfer effects occur if the source and target task are poorly related, up to the point that it is more convenient to randomly chose the network paramcters rather than transferring feature map from the source task. Extending these results to a generic number of layers in the
phase of infinite-widthl networks Yang et al. (2020) or in the proportional limit of deep neural networks Pacelli et al. (2023 Aiudi et al. (2023); Cui et al. (2023) Seroussi et al. (2023), is a promising future direction which could help to theoretically rationalize the efficacy of transfer learning in deep learning settings.
In perspective, we believe that understanding both the theoretical and the practical sides how to disentangle input similarity from the labeling rule might be interesting direction for future investigations in this research field.
Acknowledgments
The authors acknowledge Sebastian Goldt for useful discussions. SSM acknowledges support from the Sains- bury Wellcome Centre Core Grant from Wellcome (219627/2/19/27 and the Gatsby Charitable Foundation (GAT3755). AL acknowledges the Italian National Centre for HPC, Big Data, and Quantum Computing (grant number CN00000013).
References
Emmanuel Abbe, Enric Boix-Adsera, Matthew S Bremnan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hicrarchical structure can guide deep learning. Advances in Neural Information Processing Systems, 34:26989 27002, 2021.
Aiudi, R. Pacelli, A. Vezzani, R. Burioni, and Rotondo. Local Kernel Renormalization as mech- anism for feature learning in overparametrized Convolutional Neural Networks. arXic e-prints, art. urXiv:2307.11807 July 2023. doi: (188500/arxi..237....
Andrew Beam and Isaac S Kohane. Big data and machine learning in health care, Jama, 319(;3): 1317-1318, 2018.
Yoshua. Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceedings of TCML, workshop on unsupervised and transfer learning, pp. 17-36. JMLR Workshop and Conference Proccedings, 2012.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In European Conference Computer Vision, 2014.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1):
1-12, 2021.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Vikash Chouhan, Sanjay Kumar Singh, Aditya Khamparia, Deepak Gupta, Prayag Tiwari, Catarina Moreira, Robertas Damaševičius, and Victor Hugo C De Albuquerque. A novel transfer learning based approach for pneumonia detection in chest x-ray images. Applied Sciences, 10(2):559, 2020.
Angel Craz-Roa, Ajay Basavanhally, Fabio González, Hannah Gilmore, Michael Feldman, Shridar Gane- san, Natalie Shih, John Tomaszewski, and Anant Madabhushi. Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks. In Medical Imaging 2014: Digital Pathology, volume 9041, pp- 904103. SPIE, 2014.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. Tn Proceedings of the IERE/CVF conference on computer vision and pattern recognition workshops, pp. 702 703, 2020.
Hugo Cui, Florent Krzakala, and Lenka Zdeborová. Optimal learning of deep random networks of extensive- width. Xrvvv preprint #xXx:2302.00377 2023.
Ychuda Dar and Richard Baraniuk. Double double descent: on generalization errors in transfer learning between linear regression tasks. SIAM Journal on Mathemalics of Dala Science, 4(4):1447 1472, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fci-Fci. Imagenct: A large-scale hicrarchical image database. 2009 IEEE conference on computer vision and pullern recognition, pp. 248 255. leee, 2009.
Oussama Dhifallah and Yue M. Lu. Phase transitions in transfer learning for high-dimennsooall perceptrons. Entropy, (3)4)4400, 2021. doi: 10.3390/e23040400.. URL https://doi. org/ 33000//3300300000
Diego Doimo, Aldo Glielmo, Alessio Ansuini, and Alessandro Taio. Hierarchical nucleation in deep neural networks. Advances in Newral Information Processing Systems, 33:7526 7536, 2020.
Iñaki Soto-Rey Dominík Müller and Frank Kramer. Multi-disease detection in retinal imaging based ensembling heterogeneous deep learning models. Studies in Health Technology and Informatics, 283, 2021.
doi: 10.3233/shu221553 URL https://doi org/10. httgg/...33/////0
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXic:2010. 1 11929, 2020.
Simon S Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.000 09434, 2020.
João Gama, Indré Žliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1 37, 2014.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665-677 673, 2020.
Federica Gerace, Luca Saglictti, Stefano Sarao Mannelli, Andrew Saxe, and Lenka Zdeborová. Probing transfer learning with model of synthetic correlated datasets. Machine Learning: Science and Techmology, 3(1):015030, 2022.
Aldo Glielmo, Claudio Zeni, Bingqing Cheng, Gábor Csányi, and Alessandro Laio. Ranking the information content of distance measures. PNAS Nezus, 1(2), 04 2022. ISSN 2752-6542. doi: 10.1093, 00.1093/pnasncxxx pnasnexus/ pgac039. URL https://doi ftttp:://do..or//1... htt.11//paaaeexss/ppaaeao
pgac039.
Scbastian Goldt and Alessandro Ingrosso. Data-driven emergence of convolutional structure in neural net- works. Proceedings of the National Academy of Sciences, 119(40):2201854111, 2022.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 5927-5935, 2017.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEF internalional conference computer vision, pp. 1026 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp. 770 778, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Andrew Janowezyk and Anant Madabhushi, Deep learning for digital pathology image analysis: A compre- hensive tutorial with selected use cases. Journal of pothology informatics, 7(1):29, 2016.
SanaUllah Khan, Naveed Islam, Zahoor Jan, Ikram Ud Din, and Joel JP C Rodrigues. A novel deep learning based framework for the detection and classification of breast cancer using transfer learning. Pattern Recognilion Lellers, 125:1 6, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14 wet.:112..6000 6980, 2014.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519-3529. PMLR, 2019.
Andrew K. Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::/opeerrriis..eff///oo ryfMLoCqtQ
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11408 2022.
Phillip Lippe. UvA Deep Learning Tutorials. http:///aaaccreeedddrrrraaaaaaes 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-344 3440, 2015.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ar preprint arXiv:1608.034 2016.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. search the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.661, 2014.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304, 007199,, 2023.
R Pacelli, S Ariosto, M Pastore, F Ginelli, M Gherardi, and P Rotondo. A statistical mechanics framework for bayesian deep neural networks beyond the infinite-width limit. Nature Machine Intelligence, 5(12): 1497-1507, 2023.
Samiksha Pachade, Prasanna Porwal, Dhanshree Thulkar, Manesh Kokare, Girish Deshmukh, Vivek Sa- hasrabuddhe, Luca Giancardo, Gwenolé Quellec, and Fabrice Mériaudean. Retinal fundus multi-discase image dataset (rfmid): A dataset for multi-disease detection research. Data, 6(2):14, 2021. URL
14

---

Published in Transactions on Machine Learning Research (07/2024)
Guillermo Valle Pérez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. In 7th. International Conference on Learn- ing Representaaions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net net, 2019. URL
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural informalion processing systems, 32, 2019.
Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxier, Min Lin, Fred A Hamprecht, Yoshua Bengio, and Aaron C Courville. On the spectral bias of deep neural networks. 2018.
Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. Self-taught learning: transfer learning from unlabeled data. In Proceedings of the 24th international conference on Machine learning, pp. 759-766, 2007.
Alvin Rajkomar, Jeffrey Dean, and Isaac Kohane. Machine learning in medicine. New England Journal of Medicine, 180(11):1347-1335) 2019.
Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with sgd learn distri- butions of increasing complexity. In International Conference Machine Learning, pp. 28843-28863
PMLR, 2023.
Inbar Seroussi, Gadi Naveh, and Zohar Ringel. Separation of scales and a thermodynamic description of feature learning in some cnns. Nature Communicatiinss, 14(1):908, 02 2023. ISSN 2041-1723. doi: 00.1088//41146--023-36366
URL htttp:://dol.org/0... 11tp://110..110111110104100001 -23---3336----33-1------------
Harshay Shah, Kaustav Tamuly, Aditi Raghunsthan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
Hoo- Coo-Chang Shin, Holger R Roth, Mingchen Gao, Le Lu, Ziyue Xu, Isabella Nogues, Jianhua Yao, Daniel Mollura, and Ronald M Summers, Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE transactions medical imaging, 35(5): 1285 1298, 2016.
Saúl
J Ariel Carrasco-Ochoa, and José Fco Martinez-Trinn Trinidad. A review unsupervised feature selection methods. Artificial Intelligence Review, 53(2):907-99) 948, 2020.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Naftali Tishby and Noga Zaslavsky, Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Marlon Tobaben, Aliaksandra Shyshoya, John Bronskill, Andrew Paverd, Shruti Tople, Santiago Zanella- Beguelin, Richard E Turner, and Antti Honkela. On the efficacy of differentially private few-shot image classification. arXiv preprint arXiv:2302.01 01190, 2023.
Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distribution generaliza- tion in transfer learning. Advances in Newral Information Processing Systems, 35:7181 7198, 2022.
Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Transfer learning. Cambridge University Press, 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv: 1111160001111 2016.
15

---

Published in Transactions on Machine Learning Research (07 (07/2024) 2204)
A
Appendix
B Supplementary Material
B.1 Numerical Details
In this soction, we provide details concerning the autocncoder clones and the Brcast Cancer medical dataset.
Autoencoder Clones We follow Lippe (022) to build an encoder with 5 3 x 3-convolutional layers and GeLU activations, alternating stride 2 and stride 1 filters to reduce the dimensionality and ending with a single dense layer, whose size determines the bottleneck. The decoder has the same structure with the stride-2 convolutional layers replaced by deconvolutional ones. Training performed over the CTFAR-10 training set in batches of size 256 over 500 epochs using Adam Kingma Ba (2014). We used warmup schedule, peaking the learning rate up to 10-3 after 100 epochs and then annealing 10-5. At the end of the training, we collect the produced images for each bottleneck size in new dataset. In the experiments, the so-produced CIFAR-10 autoencoder clones are standardized in such a way to have zero mcan and standard deviation equal to one for each channel.
Medical Dataset Concerning the transfer experiments in the leftmost panel of Fig. I we consider the Retinal Fundus Multi-Discase Image Datasct Pachade Paccaae et al. (2021) which is composed of 3200 fundus images, each one associated to given retinal condition, including various examples of rare and challenging to detect diseases for total of 46 different classes. This dataset has been associated with the Retinal Image Analysis for Multi-Disease Classification (RIADD) challenge from the 1SB1 2021, where the main goal was to classify the 46 different types of retina conditions through sized retinal microscope images. In our experiments, to make the task more similar to the one of breast cancer detection (rightmost panel of Fig. [ we simply train the ResNet50 to recognize whether the retina is affected by some disease or not, thus switching from a multi-class to a binary classification task. Concerning the transfer exporiments in the bottom pancl of Fig. + we instead consider as a test case publicly available medical dataset for the classification of breast cancer, consisting of 277.524 50x50 patches extracted from 162 scans Janowezyk & Madabhushi (2016); Cruz-Roa et al. (0114). The images are labeled as positive negative according to the presence absence of invasive ductal carcinoma (IDC) tissue in the patch. Among all the examples, we use the 80% of them for the training set and the 20% for the test set. With the purpose of speeding up the simulations and describing more realistic transfer learning scenarios, we further randomly sub-sampled the test set up to 104 images at each simulation run, preserving class balance. In the transfer experiment, the images have been rescaled to 224 224 pixels and then standardized as required in the PyTorrc documentation for pre-trained ResNet-18.
Food101 dataset In the bottom panel of Fig. 3. show the defrosting profiles for transfer and finc-tuning setups for the Food101 dataset [Bossard el al. (2014). This dalaset has 101 classes of different kinds of foods, with 1000 images each. The images are divided into training and validation sets containing 750 and 250 examples. In our experiments, we seloct 30 images per class to create our training set and report the accuracy the defrosting profiled on the full validation set.
B.2 More on CIFAR10 Clones: Defrosting Profile and Information Imbalance
As pointed out in the main text, the optimal transfer depth strongly depends on the size of the target training set and the statistical similarity between the source and the target task.
Fig. 7 complements the picture described in sec 3.2 and sec 3.3 by showing the defrosting profile for different and CIFAR-10 training set sizes (lighter colors identify smaller training set size), when the source task is IsoGM one of the autoencoder clones at increasingly higher bottleneck size, e.g. L4, L32, L256. As we can see, in the transfer direction IsoGM to CIFAR-10, transferring up to the very layer is never beneficial, no matter the size of the larget lask. IsoGM and CIFAR-10 share only the first moment of the underlying CIFAR-10 distribution; therefore, negative transfer effects can occur due to the scarce statistical similarity between the two datascts. By including higher-order moments of the CIFAR-10 distribution and refining their approximations through the bottleneck size of the CIFAR-10 autoencoder clones, transfer learning becomes increasingly more effective. Indeed, in the transfer direction T.256 to CIFAR-10, we can see it is
16

---

Published in Transactions on Machine Learning Research (07 (2742)
always convenient to transfer up to the very last layer, except at very huge training set sizes. Almost the same picture occurs when transferring the pre-trained feature map I,32 to CIFAR10. However, similarly to the picture that emerged in the transfer experiment concerning the Breast Cancer dataset, in this case, a peak starts appearing in correspondence with the sccond-to-lan defrosted layer. The optimal transfer depth then shifts to the left in the L44 to CIFAR-10 transfer experiment with 64 images per class. This phenomenon is probably due the smaller degrec of similarity between the two datascts, even is less exacerbated than what observable in the GM CIFAR-10 transfer direction.
ssoM +
CIFAR10
L4 + CIFAR10
best accuracy
best accuracy
0.6
0.75
0.50
0.4
0.25
0.2
0
20
0
20
transfer depth
transfer depth
L32 + CIFAR10
L256 + CIFAR10
best accuracy
best accuracy
0.6
0.8
0.5
0.6
0.4
0
20
0
20
transfer depth
transfer depth
Figure 7: Effect of dataset size and source-target relatedness. The plot shows the defrosting profiles at varying training set sizes of CIFAR-10 for four different transfer couples, similar to what was shown in Fig. 4 (top panel) of the main text. The transfers reported are IsoGM to CIFAR-10 (top left pancl), L4 to CIFAR-10 (top right panel), L32 to CIFAR-10 (bottom left panel), and L256 to CIFAR-10 (bottom right panel). In particular, in the Tso-GM to CIFAR-10 transfer direction, the curves are relative from the lighter to the darker colors to the following CIFAR-10 training set sizes (8, 64, 256, 1024, full dataset size); in the L4 to CIFAR-10 transfer direction, the curves are relative from the lighter to the darker colors the following CIFAR-10 training set sizes (4, 16, 64, 256, 512); in the L32 to CIFAR-10 transfer direction, the curves are relative from the lighter to the darker colors to the following CIFAR-10 training set sizes (64, 128, 256, 384, 512}; in the L256 to CIFAR-10 transfer direction, the curves are relative from the lighter to the darker colors to the following CIFAR-10 training set sizes {128, 256, 512, 2048, full dataset size}. The simulations are averaged over 5 different realizations and the error bars are smaller than the point size.
Fig. g shows the II across layers, quantifying the representation similarity of two networks trained from scratch on CIFAR-10 and on one the clones among IsoGM, L4, L32 and L256. The representations are extracted on the CTFAR-10 test set. As we can see, the profile of the reflects the hicrarchical nature of the CIFAR-10 clone family. Indeed, the larger the size of the bottleneck, the more and the deeper similar the representations are across the layers.
C Similarity measures
In the main manuscript, we have analyzed the similarity among representations while relying on the II metries. However, as already pointed out in Sec 3.4. there other similarity measures that can be taken
17

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
0.7
0.5
information
0.3
0.1
30
convolutional layer
Figure 8: Representation Similarity. The plot displays the information imbalance across the layers to increase the fidelity the CIFAR-10 clones. particular, lighter colors refer to more similar clones (in sequence IsoGM, L4, L32, L256). The source and target network are trained at full dataset size (50K images). The simulations are averaged over 5 different realizations.
into account and that are equivalently blind to some of the re-scaling, permutation, and symmetries of the parametric function implemented by a neural network model.
Fig. shows, from left right, the CKA Kornblith et al. (2019), the Spearman correlation of the first 100 neighborhoods and the neighborhood overlap among the first 30 Doimo al. (2020) neighbors a function of the network depth. The representations correspond to the feature map of two different networks, and the other one one of the CIFAR-10 clones, e-g-, TsoGM (magenta curve), GM (blue curve) and T.256 (green curve). As can be seen, the behavior of the three metrics is qualitatively similar to the one already observed for the II in Fig. 6 (top pancl). Indeed, even in this case, the reprosentation similarity is stronger at early layers, providing a further hint at the generality of the feature maps extracted at the bottom of the networks. Moreover, the hierarchical structure of the CIFAR10-clones is reflected the profiles of the similarity measured, signaling which task is the most promising one as the source task for the CIFAR-10 classification target task.
extracted once again the CIFAR-10 test set and
trained from scratch the CIFAR-10 dataset
1.0
1.00
neighborhood overlap
0.8
0.8
spearman
0.75
0.6
0.6
0.4
0.4
0.25
0.2
0.2
0.0
10
20
30
10
20
30
0
10
20
30
convolutional layer
convolutional layer
convolutional layer #
Figure 9: Other Similarity Measures. From left to right, CKA, Spearman Correlation, and Neighborhood Overlap across layers. The different colors refer to different souree tasks, c.g., IsoGM (magenta curve), GM (blue curve), and L,256 (green curve). The setting is the same as reported in the main text. Fig. 6. i.e. source and target networks are trained the full training set size (50K images). The simulations averaged over different realizations.
18


Reviews:
[{"review_id": "Jw50quVGuQ", "reviewer": "Reviewer_yNuW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nPaper is easy to follow and the authors did a good job carefully controlling the data and experimental conditions to isolate the specific effects of dataset size and source-target similarity on transfer learning.\nTheir incremental defrosting approach provides a simple and intuitive method for finding the optimal number of layers to transfer, potentially saving computational costs. Also, the Information Imbalance (II) metric seems to be a promising tool to select the most compatible source task, helping ensure positive transfer effects.Furthermore, the results provide evidence suggesting that neural network layers learn features in a hierarchical order, from lower-order statistics to higher-order moments, based on dataset similarity.\n\nWeaknesses:\n\nLimited Scope of Datasets: The analysis largely relies on synthetic datasets (CIFAR-10 clones) and one medical imaging dataset. Results might not fully generalize to more diverse real-world scenarios.\nHeuristic Re-initialization: Retraining defrosted layers with random initialization is a common heuristic but might not always be the most optimal strategy. More sophisticated fine-tuning methods could potentially improve performance.\nLimited Exploration of Architectures: The study uses a few specific architectures. The findings might not fully translate to very different network types (e.g., transformers).\n\nComputational Cost: While the incremental defrosting protocol could be computationally cheaper than full training, it still involves multiple training cycles. Exploring more efficient search strategies for the optimal depth would be beneficial for practical settings.\nTheoretical Underpinnings: The results suggest interesting connections between feature learning, data distribution similarity, and transfer learning success. Developing a more rigorous theoretical understanding of these connections would benefit the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary: Often transfer learning helps train models with limited data. However, selecting the optimal layers to transfer from a pre-trained source model and the best source task itself is crucial for success. The paper investigates this problem using a controlled experimental setup, analyzing how factors like data size and similarity between source and target tasks impact the ideal number of layers to transfer.\n\nThe paper finds that the optimal number of transferred layers is not fixed and depends on the amount of target data and source-target similarity. Transferring too few or too many layers can hinder performance and also in some cases, only transferring the initial layers of a pre-trained model achieves the best results. In short: The best transfer learning strategy isn't one-size-fits-all. It depends on how much data you have and how similar the original task is to the new one. The paper offers a way to find the most promising starting point for transfer learning among different pre-trained models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Computational Cost Analysis: Provide a more detailed analysis of the computational cost associated with the incremental defrosting protocol, especially for larger datasets.\n\nTransformer results: Please provide results given pre-trained Transformers, if possible in language. Since the protocol is so easy, this should be quiet easily producable. It would be nice to include fine-tuning with LORA.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3c5owvESkg", "reviewer": "Reviewer_LU9V", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\nOn the positive side, the authors conduct experiments on a number of architectures and dataset combinations. The experiments are well conducted, with error bars reported for all results. Moreover, the text is well-written and the figures well-illustrated with example images where appropriate.\n\n**Weaknesses**\nGiven that this work does not provide any theoretical contributions, the empirical results are critical to its significance. Unfortunately, there are no comparisons with any previously proposed transfer learning methods. Moreover, there are no alternative metrics to compare with for the information imbalance metric. This lack of baselines makes it challenging to evaluate the paper's significance.\n\nIn addition, the paper itself does not propose a new methodology; the defrosting procedures used in the paper have been used in prior literature as well. Rather, the paper proposes using these procedures in a more effective way. Unfortunately, this reduces the paper's novelty. If the authors could use an existing methodology to produce very strong or state-of-the-art empirical results, that would improve the paper's significance.\n\nFinally, it's not clear what exactly the authors propose as their transfer learning method since it is described only in words. It would be helpful if the authors provided an algorithm for the incremental defrosting procedure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper conducts an empirical study into how the choice of representation in transfer learning affects performance. In particular, the authors consider freezing the layers of a network trained on a source task at different points and retraining the remainder of the network on the target task. The authors show that the optimal transfer depth depends on dataset size and architecture. Moreover, the authors use an Information Imbalance metric to predict the accuracy of transfer learning for use in source task selection.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical for acceptance**\nComparison with existing transfer learning methods\nComparison of II with an alternative metric\n\n**Would strengthen**\nProvide an algorithm for the incremental defrosting procedure", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xd82IiYxkw", "reviewer": "Reviewer_7HF2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- the paper does have audience as it is important to justify, both theoretically and empirically, the choice of transfer learning protocols\n- the paper is well written\n- the experiments on source-target similarity in Section 3.3 are very interesting and are a good asset to the paper\n\nWeaknesses:\n- the reviewer is unsure that the questions asked in the experimental section give a detailed enough picture to draw conclusions about designing the transfer learning protocol (see Questions below in the requested section page)\n- related literature on empirical studies of transfer learning should be discussed in more detail (see, e.g. [1, 2, 3]), so it shows the link with other transfer learning studies\n\n[1] Wenzel et al (2022) Assaying Out-Of-Distribution Generalization in Transfer Learning,\nAdvances in Neural Information Processing Systems, 2022, https://openreview.net/forum?id=57Ryl7lLD4h\n\n[2] Oquab et al (2024) DINOv2: Learning Robust Visual Features without Supervision, Transactions on Machine Learning Research,\nissn=2835-8856, https://openreview.net/forum?id=a68SUt6zFt\n\n[3] Totaben et al (2024) On the Efficacy of Differentially Private Few-shot Image Classification, Transactions on Machine Learning Research, 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors provide an empirical study of transfer learning, controlling for the optimal transfer depth, training-testing set similarity and training set size.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The design of the empirical study, as it stands at the moment, may not be able to give the systematic answer on how to design transfer learning protocols for supervised learning:\n- on one hand, the review only looks at a selective number of fairly old ResNet-family models. It has been shown (e.g., Oquab et al, 2024) that the contemporary vision-transformer-based models achieve better performance on transfer learning. This may limit the scope, so studying vision transformers' transfer learning behaviour, in comparison with ResNet, would be quite important\n- on the other hand, there are multiple factors which could affect the performance of transfer learning such as notably, use of data augmentations, model and training&finetuning hyperparameters  (see e.g. Wenzel et al (2022)) and they need to be taken into account\n- given the empirical nature of the paper, it might be that  the experiments focus on CIFAR-10 is insufficient (although some of the results feature ImageNet and Breast Cancer). Due to the limited number of datasets it might be therefore biased, so it might be that a few other image datasets, at least for the key findings, would be needed to confirm the conclusion (see e.g., Figure 7 from Tobaben et al (2024)).", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the role of pre-training in transfer learning via empirical (numerical) investigation), the authors show that last layer of the network is not always the most expressive and and that the largest performance gain may be achieved when smaller portions of the pre-trained network are transferred.  They also propose a strategy to detect the most promising source task among the available candidates. This approach compares the internal representations of a network trained entirely from scratch on the target task with those of the networks pre-trained on the potential source tasks.
The experiments are thoroughly conducted and the authors addressed reviewers concerns by adding various experiments.
Decision: Accept (poster)