Revisiting Hidden Representations in Transfer Learning for Medical Imaging
Authors: Dovile Juodelyte
Amelia Jiménez-Sánchez
Veronika Cheplygina
Venue: Accepted by TMLR ()

Abstract: While a key component to the success of deep learning is the availability of massive amounts of training data, medical image datasets are often limited in diversity and size. Transfer learning has the potential to bridge the gap between related yet different domains. For medical applications, however, it remains unclear whether it is more beneficial to pre-train on natural or medical images. We aim to shed light on this problem by comparing initialization on ImageNet and RadImageNet on seven medical classification tasks. Our work includes a replication study, which yields results contrary to previously published findings. In our experiments, ResNet50 models pre-trained on ImageNet tend to outperform those trained on RadImageNet. To gain further insights, we investigate the learned representations using Canonical Correlation Analysis (CCA) and compare the predictions of the different models. Our results indicate that, contrary to intuition, ImageNet and RadImageNet may converge to distinct intermediate representations, which appear to diverge further during fine-tuning. Despite these distinct representations, the predictions of the models remain similar. Our findings show that the similarity between networks before and after fine-tuning does not correlate with performance gains, suggesting that the advantages of transfer learning might not solely originate from the reuse of features in the early layers of a convolutional neural network.

Published in Transactions on Machine Learning Research (09/2023)
Revisiting Hidden Representations in Transfer Learning for Medical Imaging
Dovile Juodelyte
doju@itu. dk
IT University of Copenhagen, Denmark
Amelia Jiméncz-Sánchez
amjishitu. dk
IT University of Copenhagen, Denmark
Veronika Cheplygina
vech dk
University of Copenhagen, Denmark
Reviewed on OpenReview: https:// openeevee.. net/ forum? id= ScrEUZLaPr
Abstract
While key component the success of deep learning is the availability of massive amounts of training data, medical image datasets are often limited in diversity and size. Transfer learning has the potential bridge the gap between related yet different domains. For medical applications, however, it remains unclear whether it is more beneficial to pre-train on natural or medical images. We aim to shed light on this problem by comparing initialization on ImageNet and RadImageNet on seven medical classification tasks. Our work includes replication study, which yields results contrary to previously published findings. In our experiments, ResNet50 models pre-trained on ImageNet tend to outperform those trained on RadImageNet. To gain further insights, we investigate the learned representations using Canonical Correlation Analysis (CCA) and compare the predictions of the different models. Our results indicate that, contrary to intuition, ImageNet and RadImageNet may converge to distinct intermediate representations, which appear to diverge further during fine-tuning. Despite these distinct representations, the predictions of the models remain similar. Our findings show that the similarity between networks before and after fine-tuning does not correlate with performance gains, suggesting that the advantages of transfer learning might not solely originate from the reuse of features in the early layers of convolutional neural network.
Introduction
Transfer learning has become increasingly popular approach in medical imaging, as it offers a solution to the challenge of training models with limited dataset sizes. The ability to leverage knowledge from pre- trained models has proven to be beneficial in varions medical imaging applications (Raghu et al.) 2019: Cheplygina et al. 2019 Shin et al., 2016). Despite its widespread use, the precise effects of transfer learning on medical image classification are still hcavily understudied.
While pre-training ImageNet has become common practice in medical image classification, there have been growing concerns within the medical imaging community regarding its suitability for medical imaging tasks. Medical images differ from natural images in several ways, including local texture variations as an indication of pathology rather than clear global subject present in natural images (Raghu et al. 2019). Additionally, medical datasets are smaller size, have fewer classes, have higher resolution compared to ImageNet, and go beyond 2D. These differences between natural and medical image datascts have led to the argument that ImageNet may not be the optimal solution for pre-training in medical imaging due the well-known performance degradation effect caused by domain shift (Kondrateva et al. 2021 This has led

---

Published in Transactions on Machine Learning Research (09/2023)
to increased efforts to explore alternative solutions for pre-training, such using existing medical datasets
(El-Nouby et al. 2021), , their alterations (Kataoka et al. 2020; Asano et. al., 2020), and creating new medical
image datasets specifically designed for pre-training, such as RadImageNet (Mei et al. 2022).
Recent studies have challenged the conventional wisdom that the source dataset used for pre-training must be closely related to the target task in order to achieve good performance. Evidence has emerged suggesting that the source dataset may not have a significant impact on the performance of the target task and we pre-train any real large-scale diverse data (Gavrikov & Keuper] 2022a b). Further, more evidence suggests that ImageNet leads to the best transfer performance in terms of accuracy, as ImageNet not only boosts the performance (Ke et al. 2021) but also is a better source than medical image datasets (Brandt et al. 2021). This is likely due to the focus on texture in ImageNet models (Geirhos et al. 2022), which has been hypothesized to be an important cue for medical image classification.
While RadImageNet has demonstratod ImageNet-level accuracy (Mei et al. 2022) on radiology image clas- sification, it remains uncertain whether it leads to improved representations when applied medical target datasets. Furthermore, it is essential to understand the broader implications of source datasets beyond their effect on target task performance, in order to enable practitioners to make more informed decisions when selecting source dataset.
In light of the ongoing debate on the choice of source dataset for medical pre-training, we set out to inves- tigate this with a series of systematic experiments on the difference of representations learned from natural (ImageNet) and medical (RadImageNet) source datasets on range of (seven) medical targets. Our main contributions are:
We extend the work presented in Mei et al. (2022) by doing a replication study of four of their seven experiments (derived from three small medical targets: breast, thyroid, and knee datasots) and adding four additional medical imaging target datasets. Contrary to the findings in (Mei et al. 2022), we observe that most cases, models pre-trained on ImageNet tend to perform better than those trained on RadImageNet. However, it is important to note that this discrepancy does not necessarily indicate the superiority of one source dataset over the other. Rather, it emphasizes the sensitivity of transfer performance to the choice of model architecture and hyperparameters.
We investigate the learned intermediate representations of the models pre-trained on ImageNet and RadImageNet using Canonical Correlation Analysis (CCA) (Raghu et al., 2017 2019). Our results indicate that the networks may converge to distinct intermediate reprosentations, and these representations appear to become even more dissimilar after fine-tuning. Surprisingly, despite the dissimilarity in representations, the predictions of these networks are similar. This suggests that when using transfer learning, it important to evaluate other desirable model qualitics for medical imaging applications beyond performance, such as robustness to distribution shift or adversarial attacks.
Our findings demonstrate that model similarity before and after fine-tuning is not correlated with the improvement in performance across all layers. This suggests that the benefits of transfer learning may not arise from the reuse features in the early layers convolutional neural network.
We make our code and experiments publically available on Githut
2
Related work
2.1 Pre-training on different data
Transfer performance degradation due to distribution shift is a known problem. This issue is particularly relevant in scenarios where the availability of large-scale in-domain supervised data for pre-training is limited. In light of this, a line of research has emerged that challenges the common practice of pre-training on ImageNet and instead explores pre-training on different in-domain source datasets:
2

---

Published in Transactions on Machine Learning Research (09/2023)
Training target data. El-Nouby et al. (2021) have shown the potential of using denoising autoencoders for pre-training on target data in a self-supervised manner. Although this approach has yielded promising results, the experiments were conducted using natural image target datasets. Even the smallest target dataset consisted of 8,000 images, making it unclear how well this approach would translate to the domain of medical imaging where the availability of large-scale target data is often limited.
Synthetic data has been shown to be a viable alternative to real-world data for pre-training, particularly in domains where labeled data is scarce. Kataoka et al. (2020) have explored pre-training on gray-scale automatically generated fractals with labels and showed that this approach can generate unlimited amounts of synthetic labeled images, although it does not surpass the performance of pre-training on ImageNet in all cases.
Self-supervised learning is a strategy employed to learn data representations. He et al. 2022) proposed to mask random patches of the input image and reconstruct the missing pixels. MAE reconstruct missing local patches but lacks the global understanding of the image. To overcome this shortcoming, Supervised MAE (SupMAE) (Liang et al. 2022) were introduced. SupMAE include an additional supervised classification branch to learn global features from golden labels. Recently, MAE has been leveraged for medical image classification and segmentation (Zhou et al. 2022).
Data augmentation is often used to increase dataset size. Asano et al. (2020) showed that it can be used to scale a single image for self-supervised pre-training. However, this approach still falls short of using real diverse data. Even with millions of unlabeled images, it cannot fully bridge the gap between fully-supervised and self-supervised pre-training for deeper layers of a CNN.
Although aforementioned work in general computer vision has demonstrated the potential of synthetic and augmented data, the importance of large-scale labeled source datasets remains strong, particularly ImageNet in medical imaging (Brandt et al. 2021 Wen et al. (021) despite its -ot-oi- nature for medical targets. Recently, Mei et al. (2022) have demonstrated that RadImageNet, a largo-scale datasct of radiology images similar in size to ImageNet, outperforms ImageNet on radiology target datasets. To further these findings, we investigate the potential of pre-training on the RadImageNet on a range of modalities that were not included in Mei et al. 2022) experiments, such as X-rays, dermoscopic imagos, and histopathological scans.
2.2 Effects of transfer learning
Transfer learning is useful method for studying representations and generalization in deep neural networks. Yosinski et al.] (2014p defined the generality of features learned by convolutional layer based on their transferability between tasks, They analyzed representations learned in ImageNet models and found that the carly layers form general features resembling Gabor filters and color blobs, while deeper layers become more lask-specifie. More recently, Raghu et al. (2099) studied feature reuse in medical imaging using transfer learning and found that this reuse is limited to the lowest two convolutional layers. Besides feature reuse, they demonstrated that the scaling of pre-trained weights can result in significant improvement in convergence speed.
Instead of investigating the transferability of weights at different layers, Gavrikov & Keuper 2022b) inves- tigated the distributions of convolution filters learned by computer vision models and found that they only exhibit minor variations across varions tasks, image domains, and datasets. They noted that models based on the same architect tend to learn similar distributions when compared to each other, but differ signifi- cantly when compared to other architectures. The authors also discovered that modical imaging models do not learn fundamentally different filter distributions compared to models for other image domains. Based these findings, they concluded that medical imaging models can be pre-trained with diverse image data from any domain.
Our work exlends previous studies on the effects of pre-training by characterizing the representations learned from ImageNet and RadImageNet and investigating the implications of the source dataset on the learned representations. Our results provide additional evidence that the source domain may not be of high impor- tance for pro-training medical imaging models, as we observe that even though ImageNet and RadImageNet pre-trained models converge to distinct hidden representations, their predictions are still similar.
3

---

Published in Transactions on Machine Learning Research (09/2023)
3 Method
phone'
"iddeyy ultrasound'
'dog'
'chronic infarct
'cat'
'abnoral entire organ'
Pre-trained ImageNet
Pre-trained RadimageNet
Layer-wise (Seciin 3.1)
Fine-tuning
Fine-tuning
'pneumonia'
Prediction similarity (Section 3.2)
'normall
Figure 1: Overview of the experimental setup. Publicly available pre-trained ImageNet and RadImageNet weights are fine-tuned on medical targets. Model similarity is evaluated by comparing network activations over sampled stimuli images from target datasets using both CCA (described in Section 3.1 and prediction similarity (Section 3.2
We outline our overall method in Figure We fine-tune publicly available pre-trained ImageNet and Rad ImageNet weights on medical target datasets and quantify the model similarily by comparing the net- work activations over a sample images from the target datasets using two similarity measures, Canonical Correlation Analysis (CCA, Soction 3.1 and prediction similarity (Section 3.2
3.1 Canonical Correlation Analysis
CCA (Hotelling 1936 which is a statistical method used to analyze the relationship between two sets of variables.
Let X be a dataset X1, X2...,, X,, of n data points, all consisting of p variables, and Y a dataset a data points Y2.Y2.... "n and q variables. CCA seeks to find the transformation matrices A and B that linearly combine the initial variables p and in the datasets X and Y into min q) canonical variables Xa² and Yb' such that the correlation between these canonical variables is maxinized:
b' argmax corr Xa', Yb") n'.b' subject to Vjsi Xa² 1 Xa
Mjsi Yb 1 Yb=
The restrictions ensure that the canonical variables are orthogonal. This can be solved by defining substitu-
tions A 2 A and B " obtaining:
4

---

Published in Transactions on Machine Learning Research (09/2023)
A, B argmax tr(A
Σx
Ex"2Exx> Exy Ey
1/2
A.B
subject to A A =I
B
Because of the orthogonality constraints, the solution found by decomposing Σx ExyEy 1/2 into left and right singular vectors using singular value decomposition.
Layer-wise model similarity. Raahu et al. (017) proposed the use of CCA for comparing representations learned by neural networks. CCA's invariance to linear combinations makes it suitable for comparing the representations learned by different models as the layer weights in neural networks are combined before being passed on (Morcos et al. 2018).
Raghu et al. (0199) used CCA to examine representations in medical imaging models. In order to maintain consistency with Raghu et al. (2019) results, we adopt the same approach of applying CCA CNNs and use an open source CCA implementation by Raghu al. (2017) available on GitHut
In case, X and Y same-level layer activation vectors dataset, in two models with different initializations. We extract these intermediate representations and use them as input to CCA to project the representations onto a common space, where the correlation between the projections the common patterns of activity across the compared networks. Then, layer-wise similarity at layer L the average of the corrclations between the canonical variables:
stimuli images sampled from target
maximized. This common space can be thought of as a shared representation that captures
Pr. corr(Xa', Yb')
(1)
Intermediate representations extracted from CNNs of shape (n, h1, wL, PL), where hL, wz the layer spatial dimensions and pr is the number of channels in the layer. These representations are reshaped into X and Y matrices of shape (n x hy PL). As CCA is sensitive to the shape of the input matrices and the shapes vary across the layers within a network, we sample n and pr, such that n x h x wt. Rs 20, 000 and pL = 64, and then calculate layer similarity PL. This is repeated five times and the final layer similarity is obtained by averaging layer similarities pr.
3.2 Prediction similarity
We calculate prediction similarity as described in Mania et al. (2019). A model mistake is defined as gf(x, v) = {x) where x an image from the test set, y is its label and J is a network fine-tuned on the target training set. Then the prediction similarity of two networks finagolitt and fnadInagetiet: fine-tuned on the same target dataset,
P( (x,y) mastragate (x, B))
Therefore, the prediction similarity is the probability that two networks will make the same errors. To gauge the prediction similaritics between ImageNet and RadImageNet models, compare them to the prediction similarity of two classifiers with the same accuracy as ImageNet and RadImageNet models but otherwise random predictions. If the mistakes made by two models with accuracy an and are independent, the similarity of their predictions is equal to a a2 + (1 - a1)(1 (1-()11)))) a2). 02).
http://gthubbb.c/gg/ogl/////ce
5

---

Published in Transactions on Machine Learning Research (09/2023)
4 Experimental setup
4.1 Datasets
Source. We use publicly available pre-trained ImageNet (Deng et al. 2009) and RadImageNett 2022) weights as source tasks our experiments.
(Mei et al.
Target. We investigate transferability several medical target datasets. In particular, five radiol- ogy RadImageNiet in-domain datasets, and two out-of-domain datasets in the fields of dermatology and microscopy. A representative image from each dataset can be seen in Figure 2
(b)
(d)
Figure 2: Example images of (a) chest, (b) breast, (c) thyroid, (d) mammograms, (c) knee, (f) pcam peam-small, and (g) ISIC datasets.
1) Chest. Chest X-rays /Kermaay el al. 2018) datasel contains chest X-ray images from pediatric palients aged one five years old, labeled by expert physicians with binary labels 'normal' or 'pneumonia', The dataset has 5,856 images, with 1,583 labeled as 'normal' and 4,273 laboled as 'pncumonia". The image size varies, with dimensions ranging from x 2, 916 x 2, 583 pixels.
2) Breast Breast ultrasound (Al-Dhabyani et al, 2020 dataset is collected for the detection of breast cancer. The images have range of sizes, from 190 335 to 1, 0,04 x 578 pixels. The dataset is divided into
three classes: normal, benign, and malignant images. However, following Mei et al. 2022) we use a binary classification of "benign' and 'malignant" for our analysis.
3) Thyroid. The Digital Database Thyroid Ultrasound Images (DDTI) (Pedraza et al., 2013 contains 480 images of size 569 x 360 pixels, extracted from thyroid ultrasound videos. The images have been annotated radiologists into five categories. Following Mei et al. (2022) 's study, these categories
transformed into binary labels: 'normal' for categorics (1) normal thyroid, (2) benign and (3) no suspicious ultrasound
(US) feature, and 'malignant" for categories (4a) one suspicious US fealure, (4b) two suspicious US fealures, (4c) three or four suspicious US features and (5) five suspicious features.
4) Mammograms. Curated Breast Imaging Subset of Digital Databasc for Screening Mammography (CBIS- DDSM) (Saweer-Lee el al. 2016 Lee el al.] 2017 Clark et al., 2013) dataset that largets breast cancer detection. It contains scanned film mammograms with pathologically confirmed labels: 'benign' (2,111 images) or 'malignant" (1,457 images) with image sizes ranging from 1, 846 4, 4,006 to 5, 5,1 6, 871 pixcls.
5) Knee. MRNet. (Bien et al. 2018) is collection of 3D knee MRT scans. The labels for the dataset were obtained through manual extraction from clinical reports. Following Mei et al. (2022) 's study, we use extracted 2D sagittal views (1 to 3 samples per scan) amounting to a total of 4,235 'normal', 569 'ACL' (anterior cruciate ligament), and 418 'meniscal tear' images, all size 256 x 256 pixels.
6) PCam-small. PatchCamelyon (Vocling al., 2018 is a metastatic tissue classification datasct consisting of 237,680 680 colored patches extracted from histopathological scans of lymph node sections. The images are labeled 'positive' or 'negative' based on the presence of metastatic tissue. To simulate realistic target dataset size, random subset of 10,000 images was created, with 5,026 positive and 4,974 negative samples.
7) ISIC. ISIC 2018 Challenge Task 3: Lesion Diagnosis (Codella el al. 2019 Tschandl et al. 2018 dermoscopic lesion image dataset released for the task of skin lesion classification. The dataset comprises images of 600 450 pixels, which are split into seven discase categorics. The datasct unbalanced, with the class 'melanocytic nevus' having the most samples at 6,705, and the class 'dermatofibroma' having the least number of samples at. 115.

---

Published in Transactions on Machine Learning Research (09/2023)
Table Target datasets with number of images, number of classes, image size and batch size used to fine-tune the pre-trained ImageNet and RadImageNNet weights.
Dataset
Size
Classes
Image size
Batch size
Chest
5,856
112 112
128
Breast
780
2
256 x 256
16
Thyroid
480
256 256
16
Nammograms
3,568
2
224 x 224
32
Knee
5,222
112 x 112
PCam-small
10,000
2
96 x 96
128
ISIC
10,015
112 x 112
128
Due to memory constraints, we reduced the original image sizes for most of the target datasets using inter- polation without image cropping. Table [ provides details of the image sizes and number of images used for fine-tuning on each target dataset. As used publicly available pre-trained weights images were pre- processed to align with the pre-trained weights. As per the approach in Mei et al. 2022 the images with respect lo the ImageNet dataset. To increase the diversity and variability of the training data images were augmented during fine-tuning with the following parameters: rotation range of 10 degrees, width shift range of 0.1, height shift range of 0.1, shear range of 0.1, zoom range of 0.1, fill mode set to "nearest", and horizontal flip set to false if the target is chest, otherwise set to true.
normalized
4.2 Fine-tuning
We select ResNet50 (He al. 2016) the standard model architecture for our experiments. This architec- ture is widely adopted in the field of medical imaging and has been demonstrated to be strong performer in various image classification tasks. We fune-tund pre-trained networks using average pooling layer and a dropout layer with a probability of 0.5. The hyperparameters were not tuned on any of the target datasets. Since we are targeting several tasks, we decided to fix the initial learning rate to small value (le-5) for all experiments, and used the Adam optimizer to adapt to each dataset. The models were trained for a maximum 200 epochs, with early stopping after 30 epochs of no decrease validation loss, saving the models that achieved the lowest validation loss. This was done to prevent overfitting and ensure that the models generalize well to unseen data.
In addition to full fine-tuning, used freezing strategy where we froze all the pre-trained weights to train the classification layer first and then fine-tuned the whole network with the same hyperparameters as above.
Models were implemented using Keras Chollot et al. 2015) library and finc-tuned on 3 NVIDIA GeForce RTX 2070 GPU cards.
4.3
Evaluation
We fine-tune the pre-trained networks on each larget dataset using five-fold cross-validation approach. The datasets was split into training (80%), validation (5%), and test (15%) sets. To ensure patient-independeen validation where patient information is available (chest, thyroid, mammograms, knee), the target data is split such that the same patient is only present in either the training, validation or test split. We evaluate fine-tuned network performance on test set using AUC (area under the receiver operating characteristic curve). Model similarity is evaluated using CCA and prediction similarity as described in Section 3
5
Results
We carried out series of experiments evaluate the effect of pro-training on ImageNet and RadImageNet model accuracy and learned representations after fine-tuning medical targets. In the following section,

---

Published in Transactions on Machine Learning Research (09/2023)
Table 2: Mean AUC std (both x x100) after fine- suiettnnng on target datasets. Underlined is the highest mean AUC per dataset.
ImageNet
RadImageNett
Random init
Target dataset
No Freeze
Freeze
No Freeze
Freeze
No Freeze
Thyroid
64.9 + 7.2
67.8 + 6.2
62.7 + 9.1
63.7 + 5.1
64.3 + 7.8
Breast
94.3 + 1.7
95.1 + 3.6
91.0 + 5.2
89.4 3.8
85.2 1 1.4
Chest
98.7 + 0.5
99.0 = 0.3
98.7 + 0.3
98.2 + 0.3
97.9 + 0.6
Mammograms
75.4 3.1
77.3 - 1.0
74.3 + 2.0
70.4 + 5.0
68.3 + 4.4
Knee
96.5 + 0.7
97.1 + 1.3
97.3 + 0.7
95.4 + 0.7
93.2 + 1.6
ISIC
97.4 0.3
97.6 4: 0.3
96.2 + 0.4
95.8 + 0.3
95.8 + 0.3
Pcam-small
92.9 = 1.5
94.4 = 0.6
87.5 + 1.5
89.7 = 0.8
83.2 + 1.1
we present the results of our experiments and provide a thorough analysis of the findings. The results offer new insights into the effects of transfer learning and provide foundation for future research in this field.
5.1 ImageNet tends to outperform RadImageNet
Overall, ImageNet fine-tuned after freezing the classification layer leads to the highest AUCs in six out the seven datasets, Only knee reaches the highest performance with pre-traind RadImageNet weights, though we note that both ImageNet and RadImageNet performances were comparable for this dataset, as well as for the chest dataset.
We show the AUC performances in Table 2
Comparod to Mei al. (022), we obtained similar AUC values for the knee and breast datasets. However, observed significantly lower AUC for the thyroid dataset. We note that Mei et al. (2022) used subset of 349 images from the thyroid dataset, compared to 480 images available. Furthermore, they treated the classification of ACI. and meniscal tear as separate tasks for the knee dataset.
We decided to include all images the thyroid dataset for our experiments. Nonetheless, we were able to replicate the results reported by Mei et al. (2022) and achieved improved performance with our chosen hyperparameters for ResNet50. Specifically,
trained the models for 200 epochs with a learning rate of 1c-5, whereas Mei et al. (2022) trained their model for 30 epochs with learning rate of 1c-4 (Appendix A). This highlights the sensitivity of transfer performance to the choice of model architecture and hyperparameters.
5.2 Layer-wise representations become more different after fine-tuning
In this experimental setting, we compare the similarity between ImageNet and RadImageNet against several baselines. Our baselines include the similarity of two randomly initialized networks and the similarity of fine-tuned models to randomly initialized networks.
In Figure 3 we show layer-wise ImageNet and Radgeeet CCA similarity to themselves after fine-tuning, ImageNet and RadImageNet FT respectively (Figure 3al), as well as layer-wise ImageNet and RadImageNet CCA similarity before and after fine-tuning (Figure 3b). ImageNet weights change less during fine-tuning, soe Figure 3a (orange line). The two networks converge to distinct solutions after fine-tuning (both with freesing, red line on the right, and freezing, green line), even more distinct than before fine-tuning, and their similarity is significantly lower when compared to the similarity of two random initialization. The similarity between fine-tuned networks becomes comparable to the similarity between fine-tuned networks and randomly initialized networks, particularly in the higher layers. Here we only provide results on knee, however we observed similar patterns for the other target datasets (Appendix B
For experiments where pre-trained weights were initially frozen and fino-tuned after training the classification layer, we observed essentially similar trends of weight similarity as shown Figure but with higher variability between the folds of the cross- validation...
8

---

Published in Transactions on Machine Learning Research (09/2023)
1.0
Imageiee""Radddngeeeee""
InageNe (froces (aaeeeaa theae
0.8
Inagelett Radingelle
CCA similarity
Random Random
Inagelle RfI-Rardmm
0.6
RadInageNet" " R-Raddom
0.2
SrggeNet-Innggllie
RadImageNet-RRad Rangneee
0.0
conv1
block1
block2
block3
block4
convl
block1
block2
block3
block4
(a) Before versus after line-tuning
(b) ImageNet versus RadImageliet
Figure 3: Layer-wist CCA similarity (Equation I of (a) network to itself before and after fine-tuning on knee and (b) ImageNet RadImageNet. MmageNeet-ImageNee similarity (orange line) higher (ImageNet weights change less during fine-tuning) than RadImagallet-RddIngggltee Radmeaadammmgee similarity (blue line). ImageNet and ImageNet are highly dissinilar after fine-tuning on the same dataset both "No Freeze" (green line), and "Freeze" (red line), even more dissimilar than before finettning (purple line). Similarity of two randomly initialized networks (brown line) and ImageNet (pink linc) as well as RadImageNet (gray line) similarity to randomly initialized networks are provided as baselines. Error bars present mean std over five-fold cross- validation.
The results the layer-wise CCA similarity analysis reveal that ImageNet and RadImageNet converge to distinet solutions after fine-tuning on the same target dataset, the extent that they become even more dissimilar than before fine-tuning. This outcome contradicts two networks would become more similar after training the same target dataset. This discrepancy may be due to memorization of the larget dataset by one or both of the networks, suggested by the findings of MMrcos et al. (2018). They found that networks trained to classify randomized labels, hence memorizing the data, tend to converge to more distinct solutions compared to networks that generalize to unseen data.
expectation that the representations of the
(a) ImageNet
(b) RadImageNNet
(c) Synthetic Gabor filters
Figure 4: First 36 convl filters of ResNet50 pre-trained on (a) ImageNet and (b) RadImageNet. Observe that the filters ImageNet have a more pronounced resemblance to (c) Gabor filters.
The stability of the representations in early layers during fine-tuning is often attributed to their capture of general features, such as edge detectors (Raghu et al. 2019 Yosinski et al. 2014), which necessary 2018 argued that reprosentation similarity and regardless of the target domain and task. Magill et al.
generality of a layer are related, suggesting that "if certain representation leads to good performance across a variety of tasks, then well-trained networks learning any of those tasks will discover similar representations" Contrary to this hypothesis, our findings indicate that the carly layers of both ImageNet and RadImageNet exhibit similarity comparable two randomly initialized layers. We would expect that after fine-tuning on the same task, the layers of ImageNet and RadImageNet would display greater similarity to each other than

---

Published in Transactions on Machine Learning Research (09/2023)
to randomly initialized layers. Further research into general features could shed light to the learning process in early layers.
When we examine the first convolutional layer filters pre-trained on ImageNet and RadImageNet (Figure we observe that the filters in ImageNet more closely resemble Gabor filters, while those in RadImageNet are more fuzzy. This expected, as natural images often contain regular structures, such as degree angles and edges, that are typically less prominent in some of radiology images, resulting in less distinct edges in the filters learned from radiology images. Interestingly, these different first layer features in both ImageNet and Rad ImageNee,, without changing significantly during fine-tuning (Figure 3), lead to comparable performance in most cases (Table
5.3 Networks make similar mistakes after fine-tuning
In order to further understand the similarity between ImageNet and RadImageNet pre-trained nelworks, we compared their predictions before and after fine-tuning on medical targets, as shown in Figure 5 Despite the networks converging to different hidden representations after fine-tuning, as evidenced Figure their predictions were found be more similar than expected for independent predictions. The predictions before fine-tuning less similar than after fine-tuning. We found similar behavior for both "Freeze" and "No Freeze" networks. Dataset characteristies may affect the similarity of predictions, than two classes (such knee and ISIC) exhibit higher similarity in predictions before fine-tuning than independent predictions. The high variance in prediction similarity observed in the thyroid dataset before fine-tuning could potentially be attributed to the limited size of the dataset. is plausible that the fine- tuning of both ImageNet and RadImage on the same target dataset contributes to the observed prediction similarity. However, it also suggests the possibility that the networks are learning similar misleading cues in the data, resulting in similar misclassifications.
datasets with more
1.0
0.9
Prediction similarity
0.8
0.7
0.6
0.5
0.4
Predictions after fine-tune
Predictions before fine-ture -tue
Independent predictions
0.2
Thyroid
Breast
Chest
Mammogran rams
Knee
ISIC
PCam seall L
Figure 5: Prediction similarity (Equation 2 between ImageNet and RadImageNet (blue box plot), compared to prediction similarity of ImageNet and RadImageNiet (orange box plot) and of two networks that would make independent mistakes (green box plot). ImageNet and Rad predictions are more correlated than expocted for independent prodictions on average across all target datasets, with notable variation observed for predictions before fine-tuning thyroid dataset.
5.4 Higher weight similarity associated with less AUC improvement
Our findings suggest that the benefits of transfer learning in deep neural networks may not solely stem from feature reuse, defined as layer-wise ropresentational similarity before and after finc-tuning the carly layers [Raghu et al. 2019p. Figure 0 shows that the improvement AUC resulting from pre-training does not correlate with the layer-wise CCA similarity between the pre-trained and fine-tuned networks. Thus, models
10

---

Published in Transactions on Machine Learning Research (09/2023)
conv1
block1
block2
block3
block4
0.2
AUC difference
0.0
0.85
0.9
0.8
0.9
0.6
0.7
0.8
0.4
0.5
0.6
0.7
0.2
0.3
0.4
(a) Imagetlet
0.2
AUC difference
0.1
96
-0.1
0.6
0.7
0.8
0.9
0.4
0.5
0.3
0.4
0.2
0.2
0.3
0.4
0.2
0.3
0.4
CCA
CCA
CCA
CCA
Isic
Knee
Namoggaas
Pcan rcan-snalL
Thyroid
(b) RadImageNet
Figure AUC pre-training gains over random initialization for seven target datasets vs CCA similarity before and after fine-tuning on those targets, for (a) ImageNet and (b) Radgmage Higher CCA similarity after fine-tuning is not associated with higher AUC gains, observed across all layers. Note that the scaling the x-axes are different in each plot for visibility, and for RadImageNet the CCA similarity is lower overall.
that relied on reusing pre-trained features without adapting the representations during fine-tuning did not get higher gains performance compared models that underwent representation adaptation. This trend persisted across all layers for both models trained with freezing and no froezing.
Our findings align with recent results in the Natural Language Processing field which demonstrate that the benefits of pre-training are not related to knowledge transfer (Krishna et al. 2021). Additionally, our results complement the findings of Raghu et al. (2019) who showed that there are feature-independent benefits of using pre-trained weights, such as better scaling compared to random initialization. These results highlight the complex nature of transfer learning and the need for further investigation into the underlying mechanisms that drive its performance benefits.
6 Discussion
6.1 Results
In our experiments, we found that ImageNet initialization generally outperformed RadImageNet on the med- ical target datasets, in contrast to the earlier results reported in Mei et al. (2022). However, this highlights the sonsitivity of source dataset transfer performance to the model architecture and hyperparameters, rather than the inherent superiority of one source dataset over the other. We investigated the effect of hyperparam- eters such as learning rate, early stopping and training epochs, and found differences in model performance, While our study did not comprehensively analyze this sensitivity, interested readers can refer to related studies, such as Raghu et al. (2019), Wenn et al. (2021), and Cherti & Jitsev (0022) which offer valuable insights into the impact of model architecture and hyperparameters on transfer performance.
Contrary to our intuition about transfer lcarning, our analysis with CCA found that the models converged distinct intermediate representations and that these representations are even more dissimilar after fine- tuning the same target dataset. Despite distinct intermediate representations, model predictions
11

---

Published in Transactions on Machine Learning Research (09/2023)
instance level show a significant degree of similarity. This extends Mania et al. (2019) findings which showed that ImageNet models are similar in their predictions even with different architectures
6.2 Limitations and future work
We investigated transfer learning for two large source datasets and seven medical target datasets, and ResNet50, an architecture that is widely used for medical image analysis. Extending our experiments to further datasets and architectures, well as other similarity measures, such as centered kernel alignment [Korbblih et al. 2019), would be valuable to further test the generalizability of our findings.
With regard to source datasets, we used ImageNet and RadImageNet because RadImageNet's comparable size to ImageNet allowed for a unique opportunity to compare natural and medical source datasels. However, the two datasets have several differences beyond their domains. For instance, RadImageNet has differences in color, number of classes, and diversity in data, due to its limited number of patients. To further explore the impact of these differences in greater detail, future research could consider including Ecoset (Mehrer et al. 2021), natural image dataset with 565 basic-level categories selected to better reflect the human perceptual and cognitive experience.
Another limitation of our study is the use of a single classification metric (AUC) for evaluating performance. AUC a commonly used metric for classification tasks in medical imaging, and useful to compare to related work. However, there might be nuances across applications where it could be important to consider alternative metrics, such as calibration (Reinke et al. 2021.
6.3 Recommendations
In our experiments, we only used 2D images, but these lasks are not fully representative of the medical imaging field as a whole. Researchers have hypothesized that for 3D target tasks such as CT or MRI, 3D pre-training might be better alternative to 2D pro-training (Wong et al. 2018 Brandt et al.) 2021 and studies have shown that incorporating information from the third dimension might be beneficial for
performance (Yang et al. 2021 Taleb et al. 2020 However, RadImageNet only has 2D images, even though some of the original images are 3D. For 3D target tasks, consider comparing both 2D pre-training (e.g. via a 2.5D approach) and 3D pre-training. Pre-trained weights for 3D models are less common, but previous research has successfully used pre-training on for example YouTuls videos (Malik Bzdok 2022).
Regarding fine-tuning with RadImageNet, we recommend using higher learning rates when fine-tuning com- pared to ImageNet. Furthermore, suggest allocating more epochs in order to achieve optimal model performance.
Our results suggest that the implications of using ImageNet in medical image classification go beyond per- formance alone. In particular, there might be an issue with the memorization of spurious patterns in the data. This can potentially have consequences with respect to algorithmic bias and fairness. For example, see Gichoya et al.] (2022) where ImageNet pre-trained networks memorize patient race. Memorization also makes a network more vulnerable to adversarial attacks (Bortsova et al. 2021]. In the event that for a target application, ImageNet and RadImageNet weights are expected to lead to similar performances, it might be an advantage to select the weights which are less associated with these negative properties.
7 Conclusions
Transfer learning is key strategy to leverage knowledge from the models pre-trained large-scale datasets to deal with the challenge of small medical datasets. In this study, investigated the transferability of two different domain sources (natural: ImageNet and medical: RadImageNet) to seven target medical image classification lasks with limited dataset size. Our results show that pre-training ResNet50 on ImageNet outperformed RadImageNet in most cases. Furthermore, we delved deeper into the learned representations after fine-tuning by using CCA and comparing the similarity of predictions. Although the models appear to converge to distinct representations, we found they made similar prodictions. Lastly, we observed that higher model similarity before and after fine-tuning did not result in higher performance gains.
12

---

Published in Transactions on Machine Learning Research (09/2023)
References
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in Brief, 28:104863, February 2020. ISSN 2352-3409. doi: 2di..0019.104663. URL
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi.
critical analysis of self-supervision, or 13132,
what we can learn from single image, February 2020.
URL
arXiv:1904.13132 [cs].
Nicholas Bien, Pranav Rajpurkar, Robyn L. Ball, Jeremy Irvin, Allison Park, Erik Jones, Michael Bereket, Bhavik N. Patel, Kristen W. Yeom, Katie Shpanskaya, Safwan Halabi, Evan Zucker, Gary Fanton, Derek F. Amanatullah, Christopher F. Beaulieu, Geoffrey M. Riley, Russell Stewart, Francis G. Blankenberg, David B. Larson, Ricky Н. Jones, Curtis P. Langlotz, Andrew Y. Ng, and Matthew P. Lungren. Deep- Dearning-assisten diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. PLoS medicine, 15(11)e November 2018. ISSN 1549-1676. doi: 10.1371/journal. pmed. 1002699.
Gerda Bortsova, Cristina González- Gonzalez-Gonzalo, Suzanne C. Wetstein, Florian Dubost, loannis Katramados, Laurens Hogeweg, Bart Liefers, Bram van Ginneken, Josien P. W. Pluim, Mitko Veta, Clara Sánchez, and Marleen de Bruijne. Adversarial attack vulnerability of medical image analysis systems: Unexplored factors. Medical Image Analysis, 73:102141, October 2021. ISSN 1361-8415, doi: 10.1016/j. 2021. 102141. URL
Irma van den Brandt, Floris Fok, Bas Mulders, Joaquin Vanschoren, and Veronika Cheplygina. Cats, not CAT scans: study of dataset similarity in transfer learning for 2D medical image classification, July 2021. URL http::/arxiv.....////// http://axxxvvv.aaabbee/ wttp://wwrgaaab//000005555
arXiv:2107.055: [cs].
Veronika Cheplygina, Marleen de Bruijne, and Josien P. W. Pluim. Not-so-supervise:: A survey of semi- supervised, multi-instance, and transfer learning in medical image analysis. Medical Image Analysis, 54: 280-296, May 2019. ISSN 1361-8423. doi:
2019.03.009.
Mehdi Cherti and Jenia Jitsev. Effect of Pre- Tre-Trainnn Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X- X-Ray Chest Images. In 2022 International Joint Conference on Neurra Networks (IJCNN), pp. 1-9, July 2022. doi: D0.119//ICNNN5500020022000900229
URL
http:
//arxiv. hwrrg/abs22106.. 00116 arXiv:2106.00116 [cs].
Francois Chollet et al. Keras, 2015. URL htttps://kerss:
Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, Lawrence Tarbox, and Fred Prior. The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 1646) Docember 2013. ISSN 1618-727X. doi: 10.1007/110278--03--99227 URL https: btr/10..107//1102---03---22
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC), March 2019. URL httt:://tt::iii wwtp:/rr/////...00003033 arXiv:1902.0336: [cs).
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEFE Conference on Computer Vision and Pattern Recognition, pp. 248-255. June 2009. doi: 10.1109 10.1109/CVPP.2000 CVPR.2009.5208.282
ISSN: 1063-6919.
Alaacldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are Large-scale Datasets Necessary for Self-Supervised Pre-training", December 2021. URL http://arxiv. arg/abs/2112.10 10740 arXiv:2112. (cs).
13

---

Published in Transactions on Machine Learning Research (09/2023)
Paul Gavrikov and Janis Keuper. CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters. In 2022 IEEE/CVF Conference on Computer Vision and Pallern Recognation (CVPR), pp. 19044- 19054, New Orleans, LA, USA, June 2022a. IEEE. ISBN 1778---66546-946-- 2022.01848. URL
doi: 10.1109 10.1109/CVPP CVVP55688.
Paul Gavrikov and Janis Keuper. Does Medical Imaging learn different Convolution Filters?, October 2022b.
URL http://arxiv. wwwaiiv..rr/aaae orwoooog//...eo htt///bb/221.... 13799 arXiv:2210.1377 [cs, ocss).
Robert
Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNeetttaineed CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, November 2022. URL htttp://aaxxi..org/bb///////// 12231 arXiv:1811. 12231 [cs, q-bio, stat].
Judy Wawira Gichoya, Imon Banerjec, Ananth Reddy Bhimiroddy, John I. Burns, Leo Anthony Celi, Li- Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, Po-Chih Kuo, Matthew P Lungren, Lyle J Palmer, Brandon J Price, Saptarshi Purkayastha, Ayis T Pyrros, Lauren Oakden-Rayner, Chima Okechukwu, Laleh Seyyed-Kalantari, and Haoran Zhang. AI recognition of patient race in medical imaging: modelling study. The Lancet Digital Health, 000xe4006-4144 June 2022. TSSN 2589-7500. doi: (01000/////255007000222002000000
Kalantari, Hari Trivedi, Ryan Wang, Zachary Zaiman,
URL
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Las Vegas, doi: /..1099/CVPR.2016.990 URL http://
NV, USA, June 2016. IEEE. ISBN 978-1-4633-8811-1. ieeexplore. ieeeeerrreeeeee....... oee...rge/eeeeeee...........eo.e wtw.ee.rg//oocmoo////////// ddccmmnn//7804559/
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Harold Hotelling. Relations Between Two Sets of Variates. Biometrika, 28(3 377, 1936. ISSN 0006-3444. doi: 10.2307 /33339533395. URL https://www jstor...w......o................ org/sttable/233355 Publisher: [Oxford University Press, Biometrika Trust].
Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Tnoue, Akio Nakamura, and Yutaka Satoh. Pre-training without Natural Images. Asian Conference on Computer Vision (ACCV), 2020.
Alexander Ke, William Ellsworth, Oishi Banerjee, Andrew Y. Ng, and Pranav Rajpurkar. CheXtransSer: performance and parameter efficiency of ImageNet models for chest X-Ray interpretation. In Proceedings of the Conference on Health, Inference, and Learning, pp. 116 124, Virtual Event USA, April 2021. ACM. ISBN 978-1-4503-8335
doi: 10.1145/345043934451666 URL https://dll acm. org/doi/100
1145/3450439 3451867.
Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C. S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Magdalene Y. L. Ting, Jie Zhu, Christina Li, Sicrra Howett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A. N. Huu, Cindy Wen, Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia, and Kang Zhang. Identifying Medical Diagnoses and Treatable Discases by Image-Based Deep Learning. Cell, 172(5):1122-1133) February 2018. ISSN 1097-4172. doi: 10. celll//..lll..11.....
Ekaterina Kondrateva, Marina Pominova, Elena Popova, Maxim Sharaev, Alexander Bernstein, and Evgeny Burnaev. Domain shift in computer vision models for MRI data analysis: an overview. In Thirleenth International Conference on Machine Vision, volume 11605, pp. 126 133. SPIE, January 2021. doi: 10. 1117/12.2587872. URL
11606/116050H/Domainn shift- in- www.-oooootrreeiiiioooooooooooee for- MRI CARR--ata-enn awwllsiis/oo.1117/// o00a 2587872. full
14

---

Published in Transactions on Machine Learning Research (09/2023)
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network
Representations Revisited, July 2019. URL http://arxiv http://aaxxiv...g/bbbbas htrpa//ww..r////a///000000000 000111001000001444 arXiv:1905.00414 [es.
q-bio, stat].
Kundan Krishna, Jeffrey Bigham, and Zachary C. Lipton. Does Pretraining for Summarization Require Knowledge Transfer? In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3178-3189 Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18633/v1/202
findings-emnlp2275 http:///////22..ddnng-eeeemm/p
URL https://aclanthoology wrtpol/loooyyo//22//121/ findings-emnll
273
Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel Rubin. A curatod mammography data set for use in computer-aided detection and diagnosis rescarch. Scienlific Dala, 4(1):170177, December 2017. ISSN 2052-4463. doi: 10.1038/ 10.1038/sdtaas /data..011..17.. URL https:
//www.nature. nature com/ aartacelessaaate2111111111 Number: 1 Publisher: Nature Publishing Group.
Feng Liang, Yangguang Li, and Diana Marculescu. SupMAE: Supervised masked autoencoders are efficient vision learners. arXiv preprint ar arXiv:2205.1425 14540, 2022.
Martin Magill, Faisal Qureshi, and Hendrick de Haan. Neural Networks Trained to Solve Differential Equations Learn General Representations. In Advances in Newral Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https://proceeddnngs www.nnwu.rccooo http://ewe.r.rpppppp/eee//////2e
C16802660645F1122222667777777770 absssa/ html
Nahiyan Malik and Danilo Bzdok. From youtube to the brain: Transfer learning can improve brain- main-imaging prodictions with deep learning. Neural Networks, 153:325-338, 2022.
Horia Mania, John Miller, Ludwig Schmidt,, Moritz Hardt, and Benjamin Recht. Model Similarity Mitigates 12580 arXiv:1905.1 12580 [cs, stat].
Test Set Overuse, May 2019. URL
Johannes Mehrer, Courtney J. Spoerer, Emer C. Jones, Nikolaus Kriegeskorte, and Tim C. Kietzmann. An ccologically motivated image datasct for deep learning yields better models of human vision. Proceedings of the National Academy of Science, 118:e2011417118, February 2021. ISSN 0027-8424. doi: 10.1073/ ADS Bibcode:
2014410114 URL hhtpps://ut http::://u..aaabb... adsabs. harvard. edu/ htlu/abs222211/U/ 2021PNAS..118114177
11811417M
Xueyan Mei, Zelong Liu, Philip M. Robson, Brett Marinelli, Mingqian Huang, Amish Doshi, Adam Jacobi, Chendi Cao, Katherine E. Link, Thomas Yang, Ying Wang, Hayit Greenspan, Timothy Deyer, Zahi A. Fayad, and Yang Yang. RadImageNet: An Open Radiologic Deep Learning Research Datasct for Effective Transfer Learning. Radiology: Arlificial Intelligence, 4(5)=12 September 2022. doi: 10.1148/ryai, 210315. URL https: rsna. ott///g//////oooooo///o/ North America.
Publisher: Radiological Society of /tt////i/1111111 1148/ 210315
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. In Aduances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https://prooeeedings new.beedddnnnnnnuiss....... httprripss.pppppp//1////////// AattfffAbbttttttthc html
a
Lina Pedraza, Carlos Vargas, Fabián Narváez, Oscar Durán, Emma Muñoz, and Eduardo Romero. An open access thyroid ultrasound image database. In 10th International Symposium on Medical In- formation Processing and Analysis, volume 9287, pp. 188-193. SPIE, Jamuary 2015. doi: 10.1117/
12.2073532. URL httpp///www.pppaaaaaaaaee//////s
92870W/An hwwroo//nnononaaaae haw.opgw.ccccgtttooooooooororrnn http://000.11111122227333333111.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability, November 2017. URL
http://arri..orr//bb///66.. arXiv:1706.05880 [es, stat).
15

---

Published in Transactions on Machine Learning Research (09/2023)
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: understanding transfer
learning for medical imaging. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, number 301, pp. 3347 3357. Curran Associates Inc., Red Hook, NY, USA, December 2019.
Annika Reinke, Minu D Tizabi, Carole H Sudre, Matthias Eisenmann, Tim Rädsch, Michael Baumgartner, Laura Acion, Michela Antonelli, Tal Arbel, Spyridon Bakas, et al. Common limitations of image processing metrics: A picture story. arXic preprint orXiv:2104. 05642, 2021.
Rebecca Sawyer-Lee, Francisco Gimenez, Assaf Hoogi, and Daniel Rubin. Curated Breast Imaging
Subset of Digital Database for Screening Mammography (CBIS-DDSM), 2016. URL https://wiki. Version Number: 1 Type: dataset.
cancerinaginngacchive htt.aaagiiaaaaciieeeeeeee///Zaaa
Hoo-Chang Shin, Holger R. Roth, Mingchen Gao, Le Lu, Ziyue Xu, Isabella Nogues, Jianhua Yao, Danicl Mollura, and Ronald M. Summers. Deep Convolutional Neural Networks for Computer-Aidel Detection: CNN Architectures, Dataset Characteristics and Transfer Learning. IEEE transactions medical imaging, 35(5):1285 1298, May 2016. ISSN 1558-254X. doi: 10.1109/TTML 201. 2228122..
Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert. 3D Self-Supervised Methods for Medical Imaging. In Advances in Neural Informa- tion Processing Systems, volume 33, pp. 18158- 18172. Curran Associates, Inc., 2020. URL https:// http://www.......oooooooooo//ata htt.:/2220eee2200001122-eaaaaaa .html
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi-
source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1::18016,, August 2018. ISSN 2052-4463. doi: 10.1088/slatt.22118.11.. URL
wttp://///....///aaaaaaaeea..... 1038/ 2018. 161
Bastiaan S. Vecling, Jasper Linmans, Jim Winkens, Taco Cohon, and Max Welling. Rotation Equivariant CNNs for Digital Pathology, June 2018. URL
wttp://oo//ab/////eoe/ 03962 arXiv:1806.03962
[cs, stat].
Yang Wen, Leiting Chen, Yu Deng, and Chuan Zhou. Rethinking pre-training on medical imaging. Journal of Visual Communication and Image Representation, 78:103145, July 2021. ISSN 1047-3203. doi: 10.1016/j. jycir. 2ocir.0221.0311455 URL
Ken CL Wong, Tanveer Mahaoodah and Mchdi Moradi. Building medical image classifiers with very limited data using segmentation networks. Medical image analysis, 49:105-116, 2018.
Jiancheng Yang, Xiaoyang Huang, Yi He, Jingwei Xu, Canqian Yang, Guozheng Xu, and Bingbing Ni. Reinventing 2D Convolutions for 3D Images. IEEE Journal of Biomedical and Health Informatics, 25 (8):3009-301 3018, August 2021. ISSN 2168-2194, 2168-2208. doi: arXiv:1911.10477 [cs, ccss).
204144444... URL http:
//arxiv.org/ www.loggggbbbbbbeeeeee
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are fea- tures in deep neural networks? In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https: htttps:/ppapere www..iiisc........ cctpapppre/21//haasll
376671339bb291122000000000000000 2at.:/2200000AAAtaaattta Abstract. html
Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, and Prateek Prasanna. Self pre-training with masked autoencoders for medical image analysis. orXiv preprint arx.0:2003. 05579, 2022.
16

---

Published in Transactions on Machine Learning Research (09/2023)
A Replication of results from Mei et al. (2022)
Table 3: Mcan AUC + std (both x100) after fine-tuning on two different versions of the Thyroid dataset: subset used Mei et al. (2022). and full. For the results the first row, we trained ResNet50 with the parameters Mei al. (2022), specifically learning rate le-4 and 30 epochs. The results the second row correspond to ResNet55 trained with our inpeeraaaameters specified in Section 4.2 In the third row, we show the results averaged over all models reported in the paper by Mei et al. 2022
Thyroid subset
Thyroid full
Experiment
ImageNet
RadImageNet
ImageNet
RadImageNet
ResNet50 (Mei et al. (2022)
81.7 + 5.5
85.4 4.7
62.8 + 6.9
64.3 + 7.7
ResNet50 (Our parameters
87.6 + 4.1
85.9 + 3.6
64.9 + 7.2
62.7 + 9.1
Average over all models MMei et al. (0222)
76 + 14
85 +
B Layer-wise CCA similarity
1.0
InageNet Inageleet
RedinegeeittRReoInnneellll
Inag fieee RadInagefict
0.8
Inag Inageeiit
Random-Random
CCA similarity
0.6
RadIne "
0.4
0.2
block3
block!
block1
block2
block4
(a) Before versus after fine-tuning
(b) ImageNet versus RadImageNet
Figure 7: Layer-wise CCA similarity of networks fine-tuned on thyroid
1.0
Inag RaagaNieeTRAAIIagggleeli "ad
Inag freeee RadInagelict
0.8
Inag Rada--aadT
Random-Random
CCA similarity
Inag "Nann"-Random
0.6
!T
0.4
0.2
RedIngge
0.0
block2
block3
block!
block1
block2
bleck4
(a) Before versus after fine-tuning
(b) ImageNet versus RadImageNet
Figure 8: Liser-wise CCA similarity of networks fine-tuned on breast.
17

---

Published in Transactions on Machine Learning Research (09/2023)
1.0
Inag RaddeeeeT
Inag fieee RadinageNett freees
0.8
Inag elie RadInagetiet
Random-Random
CCA similarity
Inagelert Randdem
nageNet "rageeee""RRandooo Random
Inagelet-Inagellet Nnt
RadInageNie- RadInageliet"
0.0
conv1
block1
block2
block3
black4
conv1
block1
block2
block3
block4
(a) Before versus after line-tuning
(b) ImageNet versus RadImageNet
Figure Layer-wise CCA similarity of networks fine-tuned on chest.
1.0
ImageNet-Inagellet
Inag -
RadIna t"
Inag wlethrous taddassnaannaaaeeeeeett
0.8
Inag eaeeg-RadInggelle
Handam-RRndome
CCA similarity
Inag Randdmd
0.6
Rageeet
0.4
0.2
block2
block4
(a) Before versus after fine-tuning
(b) ImageNet versus RadImageNett
Figure 10: Layer-wise CCA similarity of networks finc-tuned on mammograms.
InageNet- ImageNettInngakkat"
Inag BlageNetff.a Rad
Rad InageNieet
Inag Frogoe RadInageNet
Inag eliet Rad
Random-Random
CCA similarity
Inag Randoom
0.6
0.4
0.2
0.0
conv1
block1
block2
block3
block4
conv1
block1
block2
black3
block4
(a) Before versus after fine-tuning
(b) ImageNet versus RadImagelett
Figure 11: Layer-wise CCA similarity networks fine-tuned on isic.
18

---

Published in Transactions on Machine Learning Research (09/2023)
1.0
ImageNet-Inagelaae geNe t"
Inag I-ad Raddragtii""
Inag
0.8
Inag eleet-RadInagehie
Random-Randdm
CCA similarity
Inag Randdmd
0.6
nageNet "T
0.4
0.2
block3
block4
convi
block2
block4
(a) Before versus after fine-tuning
(b) ImageNet versus RadImageNet
Figure 12: Layer-wise CCA similarity of networks finc-tuned on pcam maam-mmalll 1..
19


Reviews:
[{"review_id": "q8ql1dmjX1", "reviewer": "Reviewer_LDEB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper provides a comprehensive investigation of the difference in representations learned from natural (ImageNet) and medical (RadImageNet) source datasets for diverse medical image classification tasks.\nThe paper compares intermediate representations before and after fine-tuning, providing insights into how transfer learning affects learned representations and subsequent performance on medical image classification tasks. Additionally, the analysis on Conv1 filter visualization and its connection with the Gobar filter is interesting.\nThe paper is written in a clear and concise manner, making it easy to understand the research objectives and findings.\n\nWeaknesses:\nApplying CCA for analyzing representation similarity is not considered highly innovative and may have certain limitations. For instance, CCA is unable to accurately measure meaningful similarities when dealing with representations of higher dimensions than the number of data points [1]. As an alternative, it is recommended to incorporate additional feature similarity metrics, such as CKA (Centered Kernel Alignment) [1], which is widely recognized as one of the most popular metrics capable of addressing some of the limitations associated with CCA.\nThe study only investigated representation similarity within a single model architecture. It is proper to incorporate additional model architectures, such as the publicly available DenseNet and Inception with RadImageNet pre-trained weights, to measure representation similarity. This will enhance the persuasiveness of the conclusions drawn in the paper.\nThe argument challenging the feature reuse claim lacks persuasiveness. The correlation between CCA similarity and AUC gain is not apparent (Fig. 6), and this lack of clarity is even more pronounced in the early layers (Fig. 6 conv1).\n\n[1] Kornblith, Simon, et al. \"Similarity of neural network representations revisited.\" International Conference on Machine Learning. PMLR, 2019.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study examines the impact of pre-training on medical image classification using both natural (ImageNet) and medical (RadImageNet) source datasets. The authors perform experiments on seven medical tasks and analyze the learned representations using Canonical Correlation Analysis (CCA). The results indicate that the models trained on ImageNet and RadImageNet datasets develop different intermediate representations, which further diverge after fine-tuning. However, despite these differences, the models still yield similar predictions. This research challenges the notion of feature reuse in transfer learning and demonstrates a negative correlation between model similarity and performance improvements.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The reliability of feature similarity is the foundation for many results in this article. However, this paper only utilizes CCA to analyze feature similarity and examines features from only one network architecture, which weakens the persuasiveness of the conclusions. \nSome data in Table 2 exhibits significant variance (e.g., the \"RadImagenet Freeze\" entry for the Breast dataset), suggesting that it may not have been properly trained. It is also necessary to provide detailed explanations of the specific hyperparameters used for data augmentation and regularization terms. \nTable 4 should include visualizations of the Gabor filters for comparison purposes. \nThe contribution section should extract more specific and guiding conclusions regarding fine-tuning in medical tasks, rather than solely refuting previous arguments. Additionally, it would be insightful for the research community if the shortcomings of fine-tuning based on RadImageNet were alleviated with corresponding improvements.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q3Ig2yA1HG", "reviewer": "Reviewer_b757", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe research question is highly relevant and interesting, and the authors' message would be impactful if it were correct, since it directly contradicts those reported in RadImageNet. However, I strongly suspect the results in this paper are incorrect.\n\nWeaknesses:\n\nThere appears to be a critical issue with implementation of the models that use the RadImageNet pretrained weights. For example, the authors report 0.52 mean AUROC on the Thyroid task when using the RadImageNet pretrained weights. The authors of RadImageNet used the same initialization to evaluate on the same Thyroid task, and report 0.85 AUROC.\n\nThe authors acknowledge that their reproduced RadImageNet-pretrained results are worse than the originals, and note some differences in the experimental setup that make a direct comparison difficult (Section 5.1). However, the disparity between 0.85 and 0.52 AUROC is too significant to attribute to these differences in experimental setup, and I strongly suspect that the results in this paper are incorrect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors argue that for transfer learning in medical diagnosis, there is no advantage in pretraining on medical images over natural images. While pretraining on medical and natural images yield dissimilar representations in terms of CCA, the models tend to make similar errors.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Since the authors of RadImageNet made their dataset public and used publicly available models, I request that the authors make their reproduced results more comparable to the originally reported ones. There will be some minor discrepancies, but they should not be on the order of 0.52 vs. 0.85 AUROC.\n\nI understand that it may be computationally infeasible to fully reproduce the results from RadImageNet, but I believe compromises can be made. E.g. for the thyroid task I highlighted above, the dataset only consists of 480 images, so fine-tuning on this dataset would not take long, especially since the authors of the original paper only fine-tuned for 30 epochs.\n\nI would start by exactly matching every aspect of the original author's experimental setup that they reported (under computational constraints). I suspect something may have gone with image preprocessing, so I would double check that first. Also, this paper reports early stopping with a patience of three epochs. This is much too aggressive on small datasets where the loss has high variance, and will most certainly lead to underfitting. Mirroring the original authors' setup as much as you can will eliminate potential issues such as these.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "apu4gD7xEh", "reviewer": "Reviewer_3DKR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n - Overall, the manuscript is well written and easy to follow.\n - This work appears to be nicely embedded into existing work.\n   However, it might be that I am unaware of some of the literature.\n - The focus of this paper lies on the analysis of transfer learning.\n   There are no methodological novelties and all of the used techniques are readily available.\n\n### Weaknesses\n\n - There seems to be little context on self-supervised pre-training in related work.\n   To the best of my knowledge, the vision community prefers self-supervised pre-training.\n   Especially masked auto-encoders seem to be superior to supervised pre-training.\n   Section&nbsp;2.1, §4 (data augmentation) seems to imply the opposite.\n - There is no information on how hyper-parameters were chosen.\n - The results of the experiments are rather inconclusive:\n    1. _\"ImageNet tends to outperform RadImageNet\"_ appears to be in direct contradition to the results in Mei et al. (2022).\n       This is a strong claim, but the text (section&nbsp;5.1, §2) suggests that no proper attempt was made to reproduce the results from Mei et al. (2022).\n       Without a proper reproduction, the worse results might as well be due to a bug.\n    2. _\"representations become more different after fine-tuning\"_ is hard to interpret without a proper reference.\n       I believe the (dis)similarity of representations for two networks pre-trained with different seeds would be interesting.\n       After all, there are multiple reasons why representations could be different, apart from the dataset on which the models were pretrained.\n    3. I do not understant how the observations are _\"inconsistent with the intuition that early layers capture general features\"_.\n       I would argue that is possible to have two different sets of features that are both general, but dissimilar.\n    4. _\"The fact that both [...] were fine-tuned on the same target dataset may explain this observation\"_ comes without evidence.\n       What do the prediction similarities before fine-tuning look like?\n    5. The following statement appears to be overly stong: _\"Our results challenge the [...] belief that the benefits [...] come from feature reuse.\"_\n       If we define feature reuse as the reuse of convolutional filters, there are probably a lot of unused filters that can be removed or changed arbitrarily (e.g. Frankle et al., 2019).\n       Therefore, the results merely illustrate that some features are not reused.\n       However, this is just a weak form of the lottery ticket hypothesis (Frankle et al., 2019).\n       Moreover, it is unclear whether the decrease in similarity effectively corresponds to not reusing features.\n       E.g. if the difference in similarity is caused by changes in bias parameters or just the scale of the values,\n       I would argue that the features are still being reused.\n\n### Minor Comments\n\n - The statement _\"performance degradation effect caused by domain shift\"_ (intro, §2) is somewhat obvious, but it would be good to include a citation for reference.\n - How does sampling $n, p_L$ help with the sensitivity to shape (in section 3.1)?\n   Are they sampled such that $n \\cdot p_L$ is a constant?\n   If yes, how do you account for changes in the spatial dimensions ($h_L$ and $w_L$)?\n - What is meant with weight CCA (section&nbsp;5.4, §1)? \n   Is this CCA on the weights instead of features?\n   Should this have been weighted CCA and if yes, weighted by what?\n - The overall trends in figure&nbsp;6 are clearly not linear.\n   Therefore, I would refrain from including the suggestive linear fits.\n\n### References\n\n - Frankle, J., & Carbin, M. (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. ICLR 7. https://openreview.net/forum?id=rJl-b3RcF7", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents an analysis of transfer learning in the context of medical imaging.\nIt aims to provide insights on the effect of pre-training datasets on the features.\nThis is done by studying the similarities in predictions and activations of networks that are pre-trained on imagenet and a medical imagenet variant.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": " - Add discussion on self-supervised pre-training.\n - Explain how hyper-parameters were chosen and what dataset(s) were used to choose hyper-parameters.\n - Add details to the experiments:\n    1. Report the steps that were taken to reproduce the results from (Mei et al., 2022).\n    The goal should be to reproduce the aggregated results, but it should be easy to extract the performance of the individual models once results have been reproduced.\n    (also, see https://github.com/BMEII-AI/RadImageNet)\n    2. Provide baseline similarities to assess effects due to initialisation for pre-training or normalisation of the data.\n       Also, the frozen models might be interesting for establishing baselines.\n    3. Revisit the interpretation/explanation of the inconsistency results.\n    4. Include prediction similarity for networks before fine-tuning (e.g. the frozen networks).\n    5. The statement on feature reuse needs to be significantly weakened.\n       Alternatively, more experiments and better argumentation are necesessary to justify the statement.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline case as the reviewers are leaning towards rejection given the lack of support for some of the conclusions. However, the TMLR acceptance criteria are satisfied.
Decision: Accept (poster)