Self-supervised pretraining in the wild imparts image acquisition robustness to medical image transformers: an application to lung cancer segmentation

Authors: jue  jiang
harini  veeraraghavan
Venue: MIDL_io_2024_Conference
Published: 2024-01-29 17:47:33

Abstract: Self-supervised learning (SSL) is an approach to pretrain models with unlabeled datasets and extract useful feature representations such that these models can be easily fine-tuned for various downstream tasks. \textcolor{blue}{Self-pretraining applies SSL on curated task-specific datasets.} Increasing availability of public data repositories has now made it possible to utilize diverse and large task unrelated datasets to pretrain models in the "wild" using SSL. However, the benefit of such wild-pretraining over self-pretraining has not been studied in the context of medical image analysis. Hence, we analyzed transformers (Swin and ViT) and a convolutional neural network created using wild- and self-pretraining trained to segment lung tumors from 3D-computed tomography (CT) scans in terms of: (a) accuracy, (b) fine-tuning epoch efficiency, and (c) robustness to image acquisition differences (contrast versus non-contrast, slice thickness, and image reconstruction kernels). We also studied feature reuse using centered kernel alignment (CKA) with the Swin networks. Our analysis with two independent testing (public N = 139; internal N = 196) datasets showed that wild-pretrained Swin models significantly outperformed self-pretrained Swin for the various imaging acquisitions. Fine-tuning epoch efficiency was higher for both wild-pretrained Swin and ViT models compared to their self-pretrained counterparts. Feature reuse close to the final encoder layers was lower than in the early layers for wild-pretrained models irrespective of the pretext tasks used in SSL. Models and code will be made available through GitHub upon manuscript acceptance.

Main paper: 
 introduction: 
 Self-supervised learning (SSL) is an approach to extract useful feature representations from unlabeled images by minimizing a supervised objective through pretext tasks such as jigsaw puzzles (Zhu et al., 2020) , contrastive losses (Taleb et al., 2020) , image reconstruction (Zhou et al., 2023) , and masked image prediction (Jiang et al., 2022) . Hence, SSL pretraining followed by fine-tuning on modest sized labeled datasets has demonstrated capability to achieve highly accurate segmentation in both medical and natural image analysis tasks (Nguyen et al., 2023; Jiang et al., 2022; Tang et al., 2022; Yan et al., 2023; Zhou et al., 2021) . Prior works have shown that sequential SSL pretraining on natural images followed by pretraining on curated medical datasets improved 2D medical image analysis accuracy (Hosseinzadeh et al., 2021) and that transformers benefit more from SSL than convolutional neural networks (CNN) for image classification tasks (Hosseinzadeh et al., 2021) . More frequently, medical image applications use self-pretraining, an approach wherein SSL pretraining is applied on the same curated task dataset that is subsequently used for supervised fine-tuning. In contrast, SSL pretraining in the "wild" with large and diverse datasets that are uncurated and unrelated to task have shown to be an effective approach for natural image analysis (Matsoukas et al., 2022) . The rationale for wild-pretraining is to leverage the imaging variations inherent in large and diverse sets of images to extract universally applicable feature representations for the downstream tasks. However, supervised pretraining using ImageNet has shown only variable feature reuse depending on the distance of the medical image domain from natural images (Raghu et al., 2019) . In this context, the benefits of wild-pretraining performed on medical images over selfpretraining with curated, task-specific datasets has not been studied. Hence, we studied the impact of wild-pretraining on a relatively large and uncurated 3D medical datasets (> 10,000 CTs) vs. self-pretraining with curated, 3D CT dataset for segmenting lung tumors along with evaluation of robustness to imaging variations. We chose volumetric lung tumor segmentation because SSL pretext tasks focus on learning universal feature representations, which is likely to capture common elements like organs but not tumors. Hence, our chosen application of tumor segmentation allows us to study whether wild-pretraining from images encompassing wider variations benefits over self-pretraining on the task dataset. Our contributions include: (a) Comparative analysis of SSL-based wild-pretraining and selfpretraining applied to three common architectures, a vision transformer (ViT), hierarchical shifted window transformer (Swin), and a Unet-based convolutional network for lung tumor segmentation, (b) analysis of robustness to CT acquisition differences due to the two SSL pretraining for the individual architectures, (c) evaluation of pretext tasks on the feature reuse with SSL. Understanding the relative merits of the SSL approach for commonly used networks could inform the development of pretrained models. 
datasets: 
 Analyzed datasets with imaging acquisitions, and disease details are in Table 1 . Wild-pretraining: A total of 10,412 3D CT scans covering head to pelvis sourced from datasets provided publicly for variety of tasks including lesion detection (Xiao et al., 2023) , classification (Harmon et al., 2020) , and multi-organ and abdominal tumor segmentations were used without additional curation for pretraining. Retrospectively collected and anonymized institutional datasets were used as is from patients treated for lung, esophageal (Internal 1) and head and neck (Internal 2) cancers treated with radiotherapy (RT). Task dataset for self-pretraining and fine-tuning: A publicly available dataset of patients with locally advanced non-small cell lung cancer (LA-NSCLC) scanned with contrast and non-contrast CTs, smooth reconstruction kernels (≤ B30), 3 mm slices, and provided with tumor contours was analyzed (Aerts et al., 2015) . Tumor sizes ranged with a median of 33.68 cc and interquartile range (IQR) of 8.29 cc to 90.31 cc. A random set of 316 CTs were used in self-pretraining without tumor labels. Testing: Two independent datasets totalling 335 CTs, consisting of a public dataset of patients with early stage (stage I-II) NSCLC (Bakr et al., 2018 ) (median 7.91 cc, IQR of 3.60 cc to 28.23 cc) and institutional dataset of patients with stage (II-IV) NSCLC (median 19.54 cc, IQR of 6.64 cc to 66.87 cc) were evaluated. The LRad dataset used contrast and non-contrast CTs, a range of slice thicknesses, and a wide variety of image reconstruction kernels, which were categorized as smooth, medium, and sharp kernels for meaningful analysis of robustness. The institutional LC dataset was homogeneous in terms of CT acquisition (convolutional kernel used GE lung reconstruction and a slice thickness of 5mm). In addition, a subset of 20 patients reconstructed with both sharp (GE lung) and smooth (GE standard) kernel as well as with 2.5mm and 5mm slices were used for paired comparison of accuracy differences that controlled for tumor and patient anatomy differences.  
problem formulation and methodology: 
 The aim of this work is to understand the benefits of using SSL-based wild-pretraining with large, diverse and uncurated medical images compared to SSL-based self-pretraining with curated, in-domain task-datasets for segmenting lung tumors from CT scans. Specifically, we analyzed under what conditions wild-pretraining improves over self-pretraining by studying three common networks with varying inductive bias and CT imaging differences.ViT (Dosovitskiy et al., 2021) , which removes image specific inductive biases including locality and translational equivariance has the least inductive bias. Swin (Liu et al., 2021) adds back hierarchical scale and windowed attention to improve inductive bias. A CNN model such as non-skip Unet (nsUnet) (Zhou et al., 2023) has the highest inductive bias. Feature reuse has been attributed to success in transfer learning applied to supervised pretraining with ImageNet (Matsoukas et al., 2022; Raghu et al., 2019) . We studied how feature reuse is impacted by the SSL strategy as well as the pretext task used for SSL.Transformer encoders connected to 3D Unet decoder shown to be effective for multiorgan (Hatamizadeh et al., 2022b; Tang et al., 2022; Jiang et al., 2021) and brain tumor (Hatamizadeh et al., 2022a)  
ssl pretext tasks: 
 Wild-and self-pretraining was performed with an identical set of 5 pretext tasks that consisted of contrastive pretraining (Chen et al., 2020) , masked image prediction (MIP), two self-distillation learning tasks called masked patch token distillation (MPD) and image token distillation (ITD), as well as a combination of MIP, MPD, and ITD tasks as used in the SMIT method (Jiang et al., 2022) . We also studied the impact of sequential SSL by using wild-pretraining followed by self-pretraining using SMIT.Contrastive pretext task minimized the cosine similarity of feature embeddings from positive pairs (augmented 3D views from same patient) while maximizing the same distance between negative pairs (augmented 3D image views from different patients). ITD and MPD tasks are implemented through self-distillation performed between a student and an exponentially moving average teacher network (Jiang et al., 2022) , whereby, the two networks were presented with different 3D augmented views of input images. The view provided to the student was randomly masked using a default masking ratio of 0.75 (Jiang et al., 2022) . MIP forces the student to correctly predict the image regions underlying the masked image patches by utilizing the visual context from the visible image patches. MIP was implemented by minimizing the L1-norm between predicted and unmasked image. MPD and ITD were implemented using cross-entropy losses with temperature scaling to match the patch feature token and image token distributions, respectively.All images intensity clipped [-500 HU to 500 HU], normalized to [0,1] and resampled to a uniform voxel size of 2mm × 2mm × 2mm and then randomly cropped to 128 × 128 × 128 voxels to generate the 3D views. Images were resampled to a uniform voxel size of 1.5 mm × 1.5 mm × 2.0 mm voxels for fine-tuning and testing. The networks were optimized using ADAMw (Loshchilov and Hutter, 2017) with a cosine learning rate scheduling (Ilya and Frank, 2016) , and trained for 500 epochs with an initial learning rate of 8e -4 and warmup for 50 epochs. Self-pretraining mitigated the issue of learning from fewer examples compared to wild-pretraining by using online data augmentation and training for 2,000 epochs with a warmup for 200 epochs. A path drop rate of 0.1 was applied to the student model, and all SSL tasks were conducted on 4 NVIDIA A100 GPUs (4× 80GB memory) using a batch size of 32 for Swin, 32 for nsUnet, and 8 for ViT.Fine-tuning was performed on NVIDIA 4×A100 GPU. All analyzed networks were trained with a learning rate of 2e -4 for 1,000 epochs. Swin and nsUnet models were fine-tuned with a batch size of 24 and the ViT models used a batch size of 4 due to memory limitations.Early stopping was used to select the model with highest accuracy on validation set.Tumor segmentations were compared against manual delineations using the Dice similarity coefficient (DSC). Fine-tuning epoch efficiency was measured as the relative difference in the number of epochs at which fine-tuning was stopped with respect to the number of epochs required for training the model from scratch and expressed as a percentage. Statistical comparisons measured tumor segmentation accuracy differences between fine-tuned models produced with wild-pretrained and self-pretrained models for individual networks (e.g. ViT wild-pretrained vs. ViT self-pretrained; Swin wild-pretrained vs. Swin self-pretrained) using paired, two-sided, Wilcoxon signed rank tests at 95% significance level. Feature reuse from wild-pretrained to fine-tuned as well as self-pretrained to fine-tuned models were measured using centered kernel alignment (CKA) as detailed in Supplementary section A. 
segmentation accuracy: 
 As shown in Table . 2, wild-pretrained ViT and Swin models were more accurate their selfpretrained counterparts on both datasets. Wild-and self-pretrained CNN models were similarly accurate. Further analysis showed that wild-pretraining reduced dependency of tumor segmentation accuracy to volume compared to other training strategies for all three networks. Transformer methods showed smaller dependency of accuracy to tumor volume (Swin R 2 ranged from 0.11 for wild-pretrained to 0.14 for scratch trained; ViT R 2 ranged from 0.11 for wild-pretrained to 0.18 for scratch trained) when compared to CNN (R 2 of 0.37 for wild-pretraining to 0.41 for scratch training) as shown in Supplementary Figure . A.1.Example segmentations produced by the wild-and self-pretrained Swin following fine-tuning are shown in Figure . 1, which clearly show better performance of the wild-pretrained model. Individual pretext tasks did not lead to large differences in accuracy. However, a combination of pretext tasks as done in SMIT showed a larger accuracy improvement. Two-stage pretraining did not improve accuracy compared to wild-pretraining. Analysis of accuracy robustness to CT contrast differences showed that all three network architectures resulted in higher accuracy for contrast-enhanced CTs when using wildpretraining compared to self-pretraining (Table . 3). Wild-pretrained Swin was significantly more accurate than self-pretrained Swin for both contrast and non-contrast CTs. There was no difference in accuracy for ViT or CNN models using the two SSL strategies. Figure . 2 shows the impact of CT reconstruction kernels on tumor segmentation accuracy using the public LRad dataset (n=139 CTs). The wild-pretrained Swin was significantly more accurate than self-pretrained Swin (p < 0.001) with smooth kernel and scratch trained Swin with smooth (p < 0.001) and sharp kernels (p = 0.006).Table 3 . Tumor segmentation accuracy differences due to CT contrast. Significance comparisons of models were performed with respect to pretrained models on the public LRad dataset. Abbreviations 'Self' refers to self-pretraining and 'Pre' refers to wild-pretraining.There was a considerable variation in the reuse of the features for the same network architecture (Swin) when wild-pretrained with different pretext tasks as shown in Figure . 3 (A) to (E). Concretely, contrastive task resulted in the highest feature reuse even across different feature layers (off-diagnoal entries in the CKA matrix). ITD, a global image feature matching loss, resulted in the lowest feature reuse, followed by MPD, MIP, and SMIT (Figure . 3). SMIT, which uses a combination of ITD, MIP and MPD, the latter two are spatial locality context losses, resulted in higher feature reuse in the lower (1 to 4) and middle level features (5 to 9) compared to higher level (10 to 14) layer features. In addition, the features across different layers (off-diagonal entries of the CKA matrix) were different between wild-pretrained and fine-tuned features for SMIT, MPD, and ITD tasks but not for the contrastive learning task. Self-pretraining (Figure . 3 F) with SMIT resulted in more differentiation of off-diagnoal features (layers 5 to 14 compared to lower level features 1 to 4) but such differentiation was to a lesser degree than wild-pretrained SMIT. In general, wild-pretraining resulted in feature changes especially close to the later stage encoder layers (13 and 14) for all the pretext tasks when compared to self-pretraining.The trend of higher feature reuse for self-pretraining compared to wild-pretrained models was also observed for contrast and non-contrast CT scans ( Wild-pretrained models required fewer GPU hours for fine-tuning the models compared to self-pretrained models for Swin, ViT, and CNN networks (Table 4 ). Wild-pretrained Swin models were most efficient in terms of the number of epochs required for fine-tuning.Validation curves for the various architectures shows faster convergence of wild-pretrained models compared to self-pretrained counterparts (Supplementary Figure A.2).Analysis of feature reuse between the wild-pretrained and fine-tuned features as well as self-pretrained and fine-tuned features showed that lower level features (1 to 4) were similar  
discussion and conclusions: 
 We performed a comprehensive analysis of SSL wild-pretraining and self-pretraining applied to two transformer and one CNN model in terms of accuracy, robustness to imaging differences, as well as feature reuse. Our results are consistent with findings from natural images that demonstrated improvements in accuracy and fine-tuning epoch efficiency with wild-pretraining (Goyal et al., 2021) . Our analysis also showed that wild-pretrained Swin models were significantly more robust to CT contrast and acquisitions compared to their self-pretrained counterparts. However, the wild-pretraining approach was less beneficial for ViT as well as CNN models. Prior work with natural images (Matsoukas et al., 2022) and medical images (Hosseinzadeh et al., 2021) showed that SSL pretraining is less beneficial for CNN networks. Our analysis with multiple pretext tasks showed higher feature reuse in the lower stages and feature differentiation especially in the later stages for wild-pretrained models. This trend in lower-level feature reuse but differentiation close to higher levels was also observed for CT imaging variations with wild-pretrained models. Further analysis of feature self-similarity showed larger differentiation of features across different layers for wild-pretrained models compared to self-pretrained models, which allows the former models to extract a wider variety of features, which may contribute to higher accuracy and robustness to imaging variations. 
 
 Appendix:  


Decision: All reviewers agree that the work on SSL proposed here is worthwile and the experiments well-conducted, on a variety of datasets. The reviewers and authors engaged in the rebuttal and scores were improved after rebuttal.

Reviews: review: paper_summary: paper_summary: The manuscript examines the impact of applying self-supervised learning (SSL) to datasets that are distinct from the downstream tasks in comparison to its application on the same downstream dataset. It evaluates two backbone architectures, specifically ViT and Swin Transformer, across two SSL tasks—MIP and ITD—as well as their combined implementations. For the empirical evaluation, the authors employ one public dataset (N=139) and one internal dataset (N=196), both aimed at lung cancer segmentation. The study analyzes several key aspects: segmentation accuracy, fine-tuning efficiency, robustness to data variability, and the feature reusage.
main_review: main_review: The authors' detailed responses are appreciated. Although concerns regarding the novelty of this work remain, it is acknowledged that most existing research has predominantly focused on 2D rather than 3D, which holds its own practical value. Furthermore, the additional analysis and clarification regarding feature similarity have enhanced this study. Therefore, I have revised my rating to Borderline Accept.
strength_weakness: strength_weakness: + Most of the observations and analyses presented in the manuscript have been done in existing studies within both natural and medical imaging fields. The novelty and depth of the methodology and insights offered by this work are limited. 
+ Some empirical evaluation conducted with a constrained dataset size raises concerns regarding the robustness of the statistical analysis, potentially undermining the validity of the findings.
questions: questions: 1. The terminology of "Pretraining" and "self-pretraining" may raise confusion since both approaches leverage SSL but on differing datasets. The use of "Pretraining" might be associated with "supervised pretraining," potentially leading to confusion. It would be beneficial to differentiate these terms more distinctly or provide a clear definition to avoid ambiguity.

2. On Page 2, the description of Zhou et al., 2021 is not accurate. In the cited work, the datasets used for pretraining and the downstream tasks differ.

3. The sentence on page 7 regarding self-pretraining is not clear to me: "On the other hand, self-pretraining results in highly similar features with no separation even at higher feature levels for CT contrasts and large variations in low and mid-level features for reconstruction kernels, but not in the higher levels." This statement requires clarification for better understanding.

4. In section 5.3, the observation that datasets pertaining to the task at hand show less feature reuse but higher performance compared to self-pertaining introduces ambiguity regarding the relationship between feature reuse and downstream task accuracy. Clarifying this relationship would enhance understanding. Furthermore, analyzing the performance of pretraining and self-pretraining models used with "off-the-shelf", instead of fine-tuning, may provide valuable insights.
paper_summary: paper_summary: The manuscript explores the efficacy of pretraining and self-pretraining strategies in medical image analysis, specifically focusing on the segmentation of lung tumors from 3D-computed Tomography (CT) scans using transformer models (Swin and ViT). The authors compare the two methods in terms of accuracy gains, fine-tuning efficiency, and robustness to variations in image acquisition. The experiments show the superiority of self-pretraining methods to pretraining on different publicly available datasets. This research presents a comparative analysis of pretraining approaches in medical image analysis, providing valuable insights for developing more efficient and adaptable medical imaging models.
main_review: main_review: After reviewing the authors' responses and revisions to the manuscript, I have adjusted my evaluation from "weak reject" to "borderline accept." The authors added CNN-based architecture that broadens the study's generalizability beyond the original transformer models (ViT and Swin). The experiment enhances the paper's relevance to more straightforward types of architectures that are yet the most used in medical image analysis. Moreover, the newly added subsection 5.5, which delves into the reasons behind the superior performance of wild-pretraining over self-pretraining, addresses my concern about a "more comprehensive discussion of the outcomes". The authors have improved the English language usage and enhanced figure presentations. These changes address my initial concerns, making the findings more accessible and the paper's contributions clearer.
strength_weakness: strength_weakness: 1) The study's focus on Swin and ViT models may limit the findings' generalizability. Including a broader range of architectures could provide comprehensive insights into pretraining benefits.

2) The results section requires a more comprehensive discussion of the outcomes, particularly regarding the impact of the pretext task and an in-depth investigation into why pretraining outperforms self-training.

3)The manuscript would benefit from improved English language usage, enhancing overall readability and clarity. Moreover, a clearer link between the main paper and the supplementary materials could improve the coherence and utility of the provided resources.
questions: questions: I recommend the authors demonstrate the generalizability of their findings by including a wider variety of pre-trained networks in their study. Additionally, enhancing the manuscript's clarity through improved English language quality and deepening the discussion on the result.
paper_summary: paper_summary: The paper conducts an analysis of pretraining and self-pretraining strategies on transformer models, evaluating their performance on CT segmentation tasks. The experiments demonstrate that pretrained models significantly outperform self-pretrained models for Swin. Additionally, pretrained models require fewer fine-tuning iterations and exhibit greater accuracy and robustness in handling contrast and imaging differences.
main_review: main_review: Following a thorough review of the authors' responses and the revisions made to the manuscript, I have revised my evaluation to "weak accept." I anticipate that the improved version will provide enhanced readability and offer deeper insights to the community.
strength_weakness: strength_weakness: The definition of pretraining and self-pretraining is unclear.

The comparison between ViT and Swin may not be fair due to different number of parameters (46,405,874 V.S. 64,698,114)

The paper lacks CNN baseline and hybrid (CNN-Transformer) baselines.

The paper lacks ablation study. For example, Self-pretraining + pretraining
questions: questions: It would be better to receive a response to the comments provided above in the rebuttal.

Comments: title: Invitation for discussion
Invitation for discussion
invitation for discussion
Rebuttal
Rebuttal
Rebuttal
comment: Dear Reviewer **wLgp**,

Thanks so much for all your comments. We have addressed your comments as best as we can and updated the manuscript accordingly.

Please let us know if you have any questions and we are happy to discuss with you.

Best Regards,
Dear Reviewer **8vBC**,

Thanks so much for all your comments.
We have addressed your comments as best as we can and updated the manuscript accordingly.

Please let us know if you have any questions and we are happy to discuss with you.

Best Regards,
Dear Reviewer **CfFt**,

Thanks so much for all your comments.
We have addressed your comments as best as we can and updated the manuscript accordingly.

Please let us know if you have any questions and we are happy to discuss with you.

Best Regards,
Thank you for the positive comments regarding the study of SSL robustness to data variability and also the provided additional references [2] and [3]. We have clarified the novelty and the explanation of results and discussion in response to the reviewer’s suggestions. 

***Our answers to the weaknesses and other stated concerns are below:***

**Limited novelty, influence of curated data on… downstream tasks”** Our goal is to study the impact and the relative benefits of performing SSL wild-pretraining from large, unlabeled, diverse, and uncurated 3D medical image datasets as well as SSL-based unsupervised self-pretraining on curated 3D medical image datasets applied to 3D-based cancer segmentation. Prior works including those suggested by reviewer **CfFt** focused on 2D analysis for detection, classification, normal tissues (lung, blood vessels) and pneumothorax segmentation. We also evaluated how SSL-based wild-pretraining and self-pretraining impacted robustness to CT imaging differences on two independent testing datasets, totalling 335 examples. The prior works [1-4] as provided by reviewer **CfFt** all deal with 2D image analysis and distinctly different problems: [1] dealt with SSL performed for 2D analysis with natural images using transformers, [2] dealt with supervised training with two natural image datasets followed by fine-tuning, SSL performed with natural images followed by SSL on curated medical datasets applied to 2D medical image analysis using ResNet, [3] compared need for SSL with transformers compared to CNNs for 2D medical image classification, [4]  used 2D models supervised pretrained on ImageNet and transfer learned on 2D medical datasets. 

**Use of constrained dataset for robustness evaluation.** In total, 335 cases from two independent datasets were tested and used for statistical comparisons (Table 2), 139 from LRad dataset used to study influence of image reconstruction kernels (Figure 2) and contrast differences (Table 3). The 20 examples are additional cases that contained patients reconstructed with two different reconstruction kernels and slice thicknesses and were used to perform paired comparison to study the impact of the SSL pretraining approach on the accuracy by controlling for the tumor and patient anatomy. These results are shown in Supplementary Table A.1. 

**Clarification of novelty.**  We ***did*** ***not*** compare the performance of Swin against ViT. Instead, our statistical analyses focused on comparing wild-pretraining and self-pretraining applied to the same network configurations. We have further clarified our explanations. We also added a CNN based method using PRCLv2 as the pretraining pretext.

***Our response to the concerns in the rebuttal are:***

**1. Clarification between terminology of "Pretraining" and "self-pretraining".** We further clarified the differences in pretraining and self-pretraining. Both of these methods used unsupervised pretraining but with different data instances. Whereas self-pretraining is performed on the curated task-data without exposing any labels using pretext tasks, pretraining applies the same pretext tasks on uncurated, large, and diverse datasets that are distinct from the task-data. To further clarify this, we therefore use the terminology wild-pretraining and self-pretraining in the updated manuscript. Our approach is focused on unsupervised pretraining as used in model genesis. Supervision with labeled data instances is only performed for fine-tuning. 

**2.   About reference Zhou et al., 2021.** Thanks for pointing this out. We have now rectified it in Line 7 of introduction.

**3.  Clarification on sentence:** Thank you! We have now clarified in the revised paper。 
“The trend of higher feature reuse for self-pretraining compared to wild-pretrained models was also observed for contrast and non-contrast CT scans (Figure 4). In particular, wild-pretraining resulted in larger deviations of features close to the later encoder layer (13 and 14) for contrast compared to non-contrast CTs (Figure.4  A and B)”

**4. the relationship between feature reuse and downstream task accuracy:**
Wild-pretraining (Figure 3 E) has higher feature reuse compared to self-pretraining (Figure 3 F) in early and middle layers (1-10). In the deeper layers (especially layer 13 and 14), wild-pretraining tends to have fewer feature reuse compared to self-pretraining, which suggested that wild-pretraining adopted those deeper layers for the segmentation tasks more.
On the other hand, we also investigate the feature similarity between wild-pretraining and self-pretraining as well as their self-feature similarity, shown in Figure 5. We found that wild-pretrained feature and self-pretrained feature have less similarity (especially after layer 3), shown in Figure (C).  Wild-pretraining features have less similarity compared to self-pretraining, which indicates that wild-pretraining ability to extract a wider range of pretrained feature.
Thank you for the positive comments. We appreciate the suggestions related to “including a broader range of architectures”, “comprehensive discussion”, “impact of pretext task”, etc and have included additional experiments and added more discussion to address these concerns and improved the English to better clarify findings. 

**Generalizability of findings.** We now included a CNN based architecture, pretrained using PRCLv2 (published in TPAMI, 2023), which implemented a multi-scale pixel restoration and siamese feature comparison pretext task. We compared its performance when using training from scratch, self-pretraining vs. wild-pretraining on the exact same datasets. 

**Comprehensive discussion and pretext tasks.** We added contrastive learning-based pretext task and compared against 4 additional pretext tasks in terms of accuracy on the two testing datasets and studied feature reuse using CKA. We also studied the role of adding self-pretraining on the curated in-domain dataset to the pre-trained model to further understand how these two SSL approaches enhance the performance on downstream tumor segmentation task (two-stage pretraining). We also added an additional CKA analysis to study the similarity of pretrained features in the same versus different layers of the Swin network when it was subjected to pretraining and self-pretraining to understand how the features varied (Figure 5). Our analysis showed larger differentiation of features across different layers with wild-pretraining compared to self-pretraining, which could allow the network to extract different types of features. We added these results in Sec. 5.3, 5.5 and improved our discussion. 

ImageNet-based pretraining with SwaV and InceptionV3 are basically 2D based networks and is different from the analysis of 3D CT medical image datasets used in this work. Nonetheless, we added a contrastive learning based pretext task. 

**English and clarity.** We apologize for lack of sufficient clarity. We thoroughly revised the paper and added clearer links to the supplementary materials in the main document. We also improved the Figures with larger font sizes for readability (e.g. Figure 4) and improved the explanation. We also revised the introduction and have now included a paragraph on the contributions.


PRCLv2: Zhou, Hong-Yu, Chixiang Lu, Chaoqi Chen, Sibei Yang, and Yizhou Yu. "A unified visual information preservation framework for self-supervised pre-training in medical image analysis." IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
We thank the reviewer for their positive comments and the helpful suggestions. We have revised the manuscript and added experiments as suggested by the reviewer, which we believe has further improved the manuscript. 

**Distinction between pretraining and self-pretraining.** 
We have now use “**wild-pretraining**” and “**self-pretraining**” in the paper. We have now clarified the difference between the two approaches in the introduction. As stated in the paper, these are both self-supervised learning (SSL) approaches that differ primarily in the datasets used for pretraining the model. Pretraining makes use of large and uncurated data for the specific task, and diverse datasets for SSL. Self-pretraining is applied on the curated task-specific data. Hence, the number of examples used in the two methods is different, with pretraining often using a much larger number of examples, in our case > 10,000 3D CT volumes for pretraining and 316 3D CT volumes for self-pretraining, respectively.

**Comparison of ViT and Swin**.   As stated in our response to reviewer **CfFt**, our goal was ***not*** to compare Swin and ViT networks as this doesn’t add any new insights than what has been shown in other prior works. We instead focused our statistical comparisons on understanding the differences of using pretraining with uncurated and task-unrelated datasets versus self-pretraining on the curated task-specific dataset applied to the same network architecture. Hence, ViT was compared against itself when using self-pretraining versus pretraining and so on. We have further clarified this in the paper. 

**Lack of CNN baseline and hybrid CNN-Transformer.**  We included a CNN-based network called non-skip Unet (nsUnet) developed with SSL techniques (PRCLv2) for medical image segmentation and compared this approach to itself when subjected to pretraining and self-pretraining. Whereas it is interesting to add additional baselines like hybrid CNN-Transformer networks, these require additional and separate pretraining of the individual networks and it’s unclear if they add any additional insights than what is gained by studying the individual network approaches with different inductive biases when using the two different SSL methods. In order to get more insights on the impact of pretraining, we also included additional pretext tasks as suggested by reviewer **8vBC**. Given the page restrictions of the conference paper and to avoid putting too many results in the supplemental section, we chose to keep additional baselines for the journal version of the manuscript. 

**Ablation study.** 
We agree and added an analysis that used sequential SSL involving pretraining followed by self-pretraining.
We thank the PC and AC for the opportunity to present our rebuttal and the reviewers for their insightful comments and suggestions. In response to the suggestions, we have made the following changes. 

**(1)** further clarified our contributions. 

**(2)** improved our explanation of self-pretraining (self supervised learning [SSL] performed on the same curated downstream task dataset without labels) and wild-pretraining (SSL performed on large, diverse, unlabeled and uncurated data that is unrelated to task). 

**(3)** further clarified that the performance comparisons were performed between the SSL strategies for the same architecture (e.g. ViT wild-pretrained vs. ViT self-pretrained) instead of comparing ViT versus Swin, 

**(4)** added CNN baseline as requested by reviewers. 

**(5)** added experiments to study impact of pretext tasks on downstream task accuracy. 

**(6)** ablation analysis with two-step pretraining. 

**(7)** additional analysis using centered kernel alignment to understand the feature reuse from wild- and self- pretraining to fine-tuning due to the pretext tasks used in SSL as well as to analyze differences in the extracted features across model layers with self-pretraining and wild-pretraining methods in order to improve the discussion of why the accuracies with these approaches differ. 

**(8)** deep discussion of the CKA analysis of different pretexts using Swin backbone. 

**(9)** further clarified that the evaluation of the robustness imparted from SSL to imaging acquisition variations was applied to 335 test cases and that the additional subset analysis of 20 cases with paired acquisitions available for same patients was done to further understand differences in accuracy with these methods that controlled for the differences due to tumors. We also thoroughly revised the manuscript to rectify typographical errors and improve the English language. All changes made in response to reviewers’ comments are indicated in blue text. 

All the revisions are highlighted with blue text. We believe our manuscript is much improved from the prior version and hope the reviewers, AC, and PC would find the manuscript interesting for publication in MIDL.