Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training

Authors: jeya  maria
jose  valanarasu
dong  yang
ziyue  xu
wenqi  li
vishal m patel
bennett  landman
vishwesh  nath
Venue: MIDL_io_2024_Conference
Published: 2024-01-26 06:34:51

Abstract: Harnessing the power of pre-training on large-scale datasets like ImageNet forms a funda- mental building block for the progress of representation learning-driven solutions in com- puter vision. Medical images are inherently different from natural images as they are acquired in the form of many modalities (CT, MR, PET, Ultrasound etc.) and contain granulated information like tissue, lesion, organs etc. These characteristics of medical im- ages require special attention towards learning features representative of local context. In this work, we focus on designing an effective pre-training framework for 3D radiology im- ages. First, we propose a new masking strategy called local masking where the masking is performed across channel embeddings instead of tokens to improve the learning of local feature representations. We combine this with classical low-level perturbations like adding noise and downsampling to further enable low-level representation learning. To this end, we introduce Disruptive Autoencoders, a pre-training framework that attempts to re- construct the original image from disruptions created by a combination of local masking and low-level perturbations. We curate a large-scale dataset to enable pre-training of 3D medical radiology images (MRI and CT). The proposed pre-training framework is tested across multiple downstream tasks and achieves state-of-the-art performance. Notably, our proposed method tops the public test leaderboard of BTCV multi-organ segmentation chal- lenge. Our code can be found here: https://github.com/Project-MONAI/research-contributions/tree/main/DAE.

Main paper: 
 introduction: 
 Inception of transformers (Vaswani et al., 2017; Dosovitskiy et al., 2020) has led to a significant shift from convolution neural network (ConvNet) based methods (He et al., 2016; Ronneberger et al., 2015; Krizhevsky et al., 2017) to transformerbased methods (Dosovitskiy et al., 2020; Liu et al., 2021; Valanarasu et al., 2021; Valanarasu and Patel, 2022; Chen et al., 2021) for many computer vision applications. However, the fact that pre-training plays an irreplaceable role in model development has not changed in the past decade (Radford et al., 2021) . Model weight initialization is an important step in training deep neural networks (Kumar, 2017) as good starting weights are necessary for efficient training towards a particular task.Pre-training for natural computer vision tasks is usually not constricted by the availability of data as natural images are abundant and there is no scarcity and less restrictions in obtaining them. Unfortunately, the same does not translate to medical images as they are scarce (acquisition cost is high) and also difficult to obtain (requires specialized hardware). There is complexity involved to release them publicly due to heavy privacy regulations (Saliba et al., 2012) .Recently, masked image modelling methods (He et al., 2022) have gained significant traction as an efficient self-supervised pre-training framework. They are used to develop robust pre-trained models that can generalize well to the downstream tasks. Masked Auto-Encoders MAEs learn a feature representation by trying to reconstruct the original image while masking out randomly selected tokens in the input space. MAEs are designed specifically for transformers as masked tokens help reduce the computation. However, improvements are needed in medical imaging domain: while trying to adopt vanilla MAEs for medical images, we observed that although MAEs do lead to a performance boost for further finetuning, the reconstructions were poor and most of the anatomical structures were missing after reconstruction. This has also been observed in some recent works (Hatamizadeh et al., 2022; Zhou et al., 2022) . Unlike natural images, most of the vital information in medical images are in the fine details (e.g. small lesions, finer boundaries of organs, tiny structures of bones that need to be delineated etc).In this work, we focus on designing an effective pipeline for pre-training on 3D medical volumes. First, we design a new pre-training strategy that is better than MAEs at extracting low-level details. We introduce local masking where we do not mask at the token dimension but at the channel embedding dimension. Unlike MAEs, certain amount of masking is done to all tokens as only the channel embeddings are perturbed (visualized in Fig. 1 ), helping the network reconstruct sharp details and learn better local context. We also explore using various low-level vision tasks like denoising and super-resolution for pre-training. We observe that these tasks help extract better low-level features and result in sharper reconstructions (as seen in Fig. 2 it can be seen that MAEs cannot reconstruct the bones and other fine structures while the low-level techniques do) . In summary, we introduce Disruptive Autoencoders (DAE) where we first create a combination of these perturbations (local masking, downsampling, and adding noise) to disrupt the tokens (visualized in Fig. 1 ). Then, an autoencoder is trained to reconstruct the original medical volume from these disrupted tokens. DAEs result in sharper reconstructions, and a better performance on downstream tasks. We also devise a cross modal contrastive loss for our framework in such a way it can discriminate between the features extracted from different modalities.  
disruptive autoencoders: 
 We propose DAE, a pre-training strategy that focuses on learning strong low-level representative features of the medical image to reconstruct local context. Here, we first take a cubic patch from a 3D medical volume, perturb and tokenize it to get disrupted tokens in 3D. The disrupted tokens are then passed through a transformer encoder to learn a feature representation. The latent features are passed through a decoder and are learned to reconstruct the original image back. The tokens are disrupted using a combination of different low-level perturbations: i) Local Masking ii) Adding Noise and iii) Downsampling. Local Masking is a novel masking strategy proposed by this work. Denoising and super-resolution (recovering downsampled images) are classic low-level tasks, we are one of the first to explore them as pre-training tasks for medical imaging motivated by the tasks ability to affect low-level finer features of the images. In the next sections, we discuss these in greater detail.Local Masking The 3D input images are of dimension (H, W, Ch) where H, W , and Ch denote the height, width, and number of channels in the image respectively. After tokenization, the tokens are of the dimensions (N, C) where N represents the number of tokens and C denotes the embedding dimension. Masked image modelling methods like MAE and SimMIM (Simple Mask Image Modelling) (Xie et al., 2022b) follow a token masking approach where some tokens X out of N are set to zero and the network tries to reconstruct the original image back. The percentage of tokens masked here is a hyperparameter. Token masking approach done in MAEs can be considered a global masking approach as the entire token chosen to be masked is set to zero. To be more specific, the entire C dimension of the chosen X tokens that are to be masked are set to zero.Data Masked The entire C dimension, when set to zero, disrupts the image globally, thereby directing the network to learn global context with the objective to reconstruct the original image. This globally disruptive process of setting C does not always help in obtaining a good reconstruction for medical images as most of the information in medical images are not global but in the finer local details. Not being able to learn features representative of the local details like anatomy also affects the fine-tuning performance of MAEs. To this end, instead of masking X tokens out of N , we propose to mask X channels out of C channel embedding dimension. This ensures that there is some information preserved for each token so that local details do not get completely destroyed. The perturbation is done locally as we set certain embeddings of each token as zero. We call this approach local masking as the masking is done to local details. Local masking has been visualized in Fig. 3 . The masking ratio r here is a hyper-parameter which defines the percentage of C embeddings being masked.Other Low-level Perturbations While local masking helps us extract features representative of local context, we further try to employ other tasks like denoising and superresolution for pre-training which would help learn more low-level information. Noise and low resolution are commonly found issues in realistic clinical medical acquisition pipelines and hence having them in the pre-training pipeline is meaningful (Zhao et al., 2019) . Adding Noise: To obtain a good denoised image, a model must be able to restore all local details of the image like edges and corners. To enable denoising as a pre-training task, we first add noise to the original input and try to restore the original image from the noisy input using the network. Given that the most common additive Gaussian noise, we define the perturbed input x as follows:xEQUATIONwhere x is the original input and N is the normal distribution. For each sample, we randomly sample from a normal distribution to get the noise. This way, there is no specific pattern for the network to easily restore the input. The mean and variance are hyper-parameters which can be used to control the noise level injected. Downsampling: Super resolution helps generate high-resolution scans from otherwise lowresolution images. To use super-resolution as a pre-training strategy, we first downsample the input image x to get the LR image x. We formulate this downsampling process as follows:x = D(x, ϵ),(2) where D is the downsampling function and ϵ is the downsampling ratio. We use linear interpolation for downsampling. We follow a pre-upsampling super-resolution setup where we upsample the LR to same spatial dimension before converting it to HR. The model is trained to recover the HR image x from the LR input x. Super-resolution is a low-level vision task as we need a features representing fine details to get a HR estimate from blurry LR inputs. Thus, to produce a good HR image, the network needs to learn features that provide rich information to recover all the fine details of the image.In DAEs, we use a combination of all the above perturbations. We first add noise and downsample the image, then add the two perturbations. Then, the resultant 3D image is tokenized with a local masking strategy. We name these tokens as disrupted tokens and pass it to the transformer encoder. These features are then passed through a decoder for reconstruction and for cross-modal contrastive learning.We use Swin-UNETR (Tang et al., 2022) as our backbone architecture for all the experiments. For pre-training we use a combination of L 1 reconstruction loss and the cross modal contrastive learning loss L CM CL .First, we explain how to obtain L CM CL . Inspired from (Radford et al., 2021) we train the network to predict which of the B × B possible pairings across a batch B. The goal is to maximize the cosine similarity of the embeddings of the true pairs in the batch while minimizing the cosine similarity of the embeddings of the incorrect pairings. We optimize a symmetric cross entropy loss over these similarity scores.EQUATIONwhere z i is the mini-batch feature vector, z sim is the similarity matrix and i is the mini-batch index. t here is the temperature parameter and is set to 0.07. Note that the total number of data points is N and each mini-batch has B data points which means the mini-batch index i goes from 1 to N/B. Now, we apply a simple binary cross entropy loss on the similarity matrix z sim to perform contrastive learning.We define CMCL loss as follows:EQUATION) where CE represents the binary cross entropy loss, α represents the scale and z label denotes the labels. CE loss is applied across each axis in the z sim matrix. z label is the label matrix that is created based on the positive pairs and negative pairs as per the meta-data.The total pre-training loss can be defined as:L pretrain = L 1 + L CM CL .(5) The α in CMCL is set to 0.05 to match the range of both losses. We pre-train Swin-UNETR on a curated set of medical volumes without any labels then use those weights as starting weights for our downstream fine-tuning experiments. For finetuning experiments, we use a Dice loss to train the model. 
datasets: 
 Pre-training Dataset: We various public radiology CT and MRI datasets BraTS21 (Bakas et al., 2018 ), LUNA16 (Setio et al., 2015) , TCIA Covid19 (Desai et al., 2020) The data were provided with histogram based matching and zero-padded for 256×256×256 voxels. Data of both sites was also sampled at 0.5 mm isotropic spacing as per challenge design. The dataset was split into 5-folds of 80/20 for training and validation.Our deep learning models were implemented in PyTorch (Paszke et al., 2019) and MONAI. For pre-training experiments, we used a batch-size of 2 per GPU. The volumes were randomly cropped into 96×96×96 cubes while pre-training. We used an initial learning rate of 4e -4 , momentum of 0.9 and decay of 1e -5 for 20K iterations. We trained the model using an AdamW (Loshchilov and Hutter, 2017) optimizer with a warm-up cosine scheduler of 500 iterations. We use hyper-parameters r = 60%, σ = 0.1, ϵ = 4 when training the DAE.For BTCV fine-tuning experiments, a five-fold cross validation strategy is used to train the models. The models were trained for 600 epochs with a learning rate of 4e -4 and the batch-size was set to 4 per GPU. Multiple augmentations such as gaussian noise, contrast scaling, zoom and random flipping across the axis were utilized. We select the best model in each fold and ensemble their outputs for final segmentation predictions. For FeTA, the intensities were normalized to a scale of 0 to 1. The learning rate was set to 4e -4 and batch size was set to 4 per GPU. All models were trained for 600 epochs, which was determined by convergence for the full dataset. Augmentations like random flipping on all 3 axes, Gaussian noise etc. were utilized during the training process. The final layer of the network is also changed from the pre-training configuration to accommodate the fine-tuning task at hand. All pre-training and fine-tuning models are trained using NVIDIA DGX-1 V100 servers with 8 and 4 GPUs, respectively. We compare our proposed method with previous self-supervised methods like contrastive coding (Chen et al., 2020; Tang et al., 2022) , rotation prediction (Taleb et al., 2020; Tang et al., 2022) , and masked image modelling methods (He et al., 2022; Xie et al., 2022b) . We use Swin-UNETR as our network backbone for all these experiments. We note that MAE and SimMIM are very similar to each other, the only difference being MAEs discard masked tokens while Sim-MIM includes them. So, we just utilize a masked image modelling configuration but with Swin-UNETR as the backbone and call this configuration MAE in all upcoming discussions. For BTCV, we directly validated our predictions in the public leaderboard and so that we can compare our method with all previous backbone methods. For the leaderboard submissions, we submit to the free competition (no specific registration process required). We train all our models with 80 subjects (20% as validation set), and evaluates on the 20 images test set with spacing resolution of 1 × 1 × 1mm. Within the 80 images, 30 scans are from the public challenge data and 50 extra CT scans annotated by radiologists are used to boost the training performance. We perform 4 rounds of five-fold cross validation experiments and ensemble models to obtain the final prediction.The ensemble process are effective to exclude outliers. In addition, test time augmentation, boundary smoothing and connected component analysis are used for post-processing the labels. Note that this pipe-line for BTCV leaderboard submission is similar to the previous approaches like (Tang et al., 2022) for fair comparison. These results are tabulated in Table 2 . We note that our proposed method performs the best and outperforms all the previous baselines. Specifically, we note that we outperform Swin-UNETR (Tang et al., 2022) which also uses SSL pre-training consisting of 3 different SSL pretext tasks. In particular, we obtain a significant improvement in terms of Hausdorff Distance (HD) and Mean Surface Distance (MSD) compared to previous methods. We also conduct a paired t-test between our BTCV test Dice scores and Swin-UNETR's results.We obtain a two-tailed p-value of 0.0318 which shows our improvement is statistically significant (p ≤ 0.05). We also note we obtain better results than even recent methods like UniMiSS (Xie et al., 2022a) which works on the same dataset. Table 3 shows the results of DAE compared with MAE and from scratch on FeTA dataset. In Table 4 , we show the comparison of DAE with MAE as well as previous SSL techniques on the five-fold BTCV cross validation dataset. It can be seen that DAE is better than the previous methods. Ablation Study: In Table 5 , we conduct an ablation study on DAE. We compare with each of the disruptions separately and then compare against a combination of them. These experiments are conducted on the five-fold BTCV cross validation dataset. For this task, we observe that all three methods perform better than scratch while local masking performs the best out of the three. The combination of the disruptions obtains better performance. We note that this trend totally depends on the downstream task but with a combination of these perturbations, the pre-trained weights are always better at performance than random initialization. Impact of CMCL: To understand the impact of CMCL, we conduct an experiment on DAE with and without CMCL loss. This experiment is conducted on single fold of the BTCV dataset. It can be observed in Table 6 that CMCL provides a benefit in terms of fine-tuning performance.  
conclusion: 
 In this work, we proposed a new pre-training framework for 3D medical images called Disruptive Autoencoders, where tokens are disrupted using a combination of perturbations: local masking, additive noise, and downsampling. In particular, local masking is a new masking strategy where the masking is performed across channel embeddings instead of tokens to improve the learning of local feature representations. DAE as a pre-training framework performs better than other pre-training strategies across multiple segmentation datasets. Notably, using our proposed method we also achieve the best performance in a public multi-organ segmentation challenge leaderboard and state-of-the-art results on the other public datasets.annotations or labels is not used for pre-training, pretext tasks like solving jigsaw puzzles (Noroozi and Favaro, 2016) or predicting rotation (Gidaris et al., 2018) are used to extract meaningful information. It is also worthy to note CLIP (Radford et al., 2021) uses a contrastive loss for multi-modal data to learn robust visual representations from image and text pairs. Contrastive methods have been shown to be useful in many other multi-modal contexts (Zhang et al., 2022) .Pre-training for Medical Images: Model Genesis (Zhou et al., 2019) proposes a unified self-supervised learning framework using the recurrent anatomy in medical images. Azizi et al. (Azizi et al., 2021) perform a stage-wise pre-training on natural images followed by task specific medical images. A Multi-Instance Contrastive Learning based approach is proposed to perform self-supervised pre-training. Several other methods (Kalapos and Gyires-Tóth, 2022 ) also follow similar contrastive strategies for specific medical imaging tasks. In (Tang et al., 2022) , a self-supervised framework using a combination of contrastive coding, rotation prediction, and inpainting was proposed for pretraining on CT volumes. MAE-based pre-training methods have been quickly adopted for self-supervised pre-training on medical images (Zhou et al., 2022; Chen et al., 2022) . These works show that masked image modelling methods provides better performance than previous contrastive methods. Unlike these works, we propose a new pre-training setup which efficiently pre-trains on multiple modalities contrastively while also learning all the low-level anatomical details using an autoencoder to have a better representative power. Dira (Haghighi et al., 2022) (Zhou et al., 2022) . Although using these pre-trained weights do improve downstream tasks, we argue that there is significant potential for further improvement in learning better representations as compared to MAEs for medical images. Coarse reconstructions might be sufficient to understand a high-level semantic understanding which is useful for classifying natural images. For tasks like segmentation of medical images, we postulate that coarse features result in poor reconstructions and are not sufficient to enable efficient finetuning. MAEs lack in the aspect of learning features that reflect a deeper understanding of the medical image as the tokens are masked globally and no special attention is given to learn the local details. This is a complete version of a proof sketched in the main text.Figure 5 : BTCV Leaderboard. It can be observed that our method performs the best when compared to all the previous methods.study theimpact of hyperparameters σ (denoising), ϵ (downsampling), and r (masking ratio of local masking) on the downstream performance. All the experiments are conducted on a single fold of Beyond the Cranial Vault Abdominal Segmentation (BTCV) dataset. 
 
 Appendix: In Table 8 , we study the impact of noise level during pre-training on the fine-tuning performance. The parameter σ is the variance of the gaussian noise. Note that in this experiment, we do not include other perturbations of local masking and downsampling. Upon visualizing the degraded image across different noise levels, we performed experiments on three different variances σ = 0.05, σ = 0.1, σ = 0.2. It can be observed that the downstream performance is the best while σ = 0.1. The results can be interpreted in an intuitive manner as it's known that a higher amount of noise hyper-parameter will degrade the image significantly and thus makes it more difficult for the model to learn any useful low-level representations. While a low amount of noise does not help in learning good representations as the image hardly goes through any kind of degradation and is likely to be directly copied by the network to get the reconstruction. In Table 9 , we study the impact of downsampling ratio during pre-training on the finetuning performance. Note that in this experiment, we do not include other perturbations of local masking and adding noise. The parameter ϵ is the factor of downsampling operation done. Upon visualizing the degraded image across ϵ, we performed experiments on three different variances ϵ = 2, ϵ = 4, ϵ = 8. It can be observed that the best performance is when ϵ = 4. The reason here is also similar as that of noise level as a higher perturbation makes it difficult to learn useful representation while lower perturbation does not distort the image much hence the network just learns to copy information from input to reconstruction.  


Decision: The work has wide applicability to medical imaging due to its ability to deal with multiple modalities. A large dataset is used and results in significant improvements. The authors engaged in the rebuttal with convincing answers.

Reviews: review: paper_summary: paper_summary: This paper proposes a generic framework to pretrain neural networks for 3D MRI and 3D CT, which consists in a transformer-based auto-encoder, coined Disruptive Autoencoder (DAE).
The tokens are disrupted with different perturbations, including a proposed Local Masking.
The usefulness of the DAE is assessed with a segmentation task, and compared to other methods.
main_review: main_review: The structure of the paper has clearly improved, and is now much more easy to follow.
The method is interesting, the results are clearly presented, and show some improvement compared to previous methods.
strength_weakness: strength_weakness: The paper is not well structured, and a bit hard to follow.
Some figures are not used (5 and 6), only the first line of figure 2 is used.
The order of the figures is a bit odd.
The Local Masking is properly explained in section 2. Before that two references to Fig 1 and 3 are made to explain it, but the figures are not clear enough by themselves.
questions: questions: Please improve the structure of the paper to make it easier to follow.
paper_summary: paper_summary: This paper introduces a self-supervised pretraining approach for 3D medical images. The proposed method is primarily based on masked image modeling (MAE), and uses different image augmentations, including channel-wise masking, downsampling, and noise injection to perturb the images. The proposed method also utilizes a contrastive learning loss. Evaluation of the proposed method has been conducted on BTCV and FeTA datasets.
main_review: main_review: The paper’s novelty is limited and its contributions may not be significant. The proposed method lacks clear motivation, and the rationale behind its design choices is not adequately justified or supported by results. Additionally, the experimental setups have flaws and fail to demonstrate the efficacy and generalizability of the proposed method. Furthermore, the related works are outdated and overlook the most relevant and recent studies, which has also led to the selection of less competitive baselines.
strength_weakness: strength_weakness: 1. The novelty of the paper is limited as the proposed method largely overlaps with existing reconstruction techniques, such as [1,6], and merely introduces minor alterations concerning image augmentation. Additionally, the integration of contrastive learning with reconstruction has been previously explored in existing self-supervised learning methods, such as  [4] and Swin-UNETR (Tang et al.).

2. The proposed method lacks clear motivation, and the rationale behind its design choices is not adequately justified or supported by results.

   * The paper advocates for local masking to enhance the reconstruction capability of MAE, even though achieving optimal reconstruction is not the primary objective of MAE or other reconstruction-based SSL methods. Furthermore, there are no experimental results provided to demonstrate the efficacy of local masking standalone over conventional masking in MAE.
   *  The rationale behind the cross-modal contrastive loss is unclear. Despite being labeled as cross-modal, it operates across augmented versions of the same images rather than different modalities, making it similar to conventional contrastive learning. Moreover, there is a lack of experimental results demonstrating the superiority of this approach over contrastive loss.
 	
3. The literature review is outdated and overlooks the most relevant and recent works, including [1,2,3,4,5]. I recommend the authors conduct a thorough literature review to provide a comprehensive and up-to-date review of  the latest advancements in self-supervised learning in medical imaging, which is necessary for identifying the original contributions of the paper and its advantages over existing methods, and selection of more appropriate baselines.

4. The experimental results fail to sufficiently demonstrate the effectiveness of the proposed method and support the claim of state-of-the-art performance.

   * The authors state that the proposed method outperform Swin-UNETR (Tang et al., 2022), which also utilizes SSL pre-training. However, it's important to note that Swin-UNETR leverages 5K pretraining data, whereas the proposed method benefits from more than 10K data. Therefore, this comparison may not be entirely fair and does not convincingly demonstrate the superiority of the proposed pretraining method over the  Swin-UNETR’s pretraing schema.
   * Continuing from the previous point, there is a lack of comparison with Swin-UNETR (Tang et al., 2022) in the FeTA dataset. It is necessary to compare the proposed method with Swin-UNETR in the same pretraining setting in terms of pretraining data in both target tasks.
   * The authors consider MAE as the main baseline; however, it does not appear to be a competitive baseline. For instance, as depicted in Figure 6, the performance of MAE is even inferior to training from scratch.
   * Continuing from previous point, the performance boost over MAE in FeTA dataset seems to be marginal, and there is no statistical analysis to demonstrate the significance of the results.
   *  The proposed method has not been compared with state-of-the-art self-supervised methods specifically designed for medical imaging, such as [1,2,3,4,5]. Instead, it has been compared with basic SSL methods such as rotation, and just MAE in FeTA dataset. 

5. The results in Table 4 in the Appendix raises concerns regarding the effectiveness of the proposed method in learning features suitable for downstream applications. The reported CKA scores are notably low, indicating potential limitations in learning reusable features. Additionally, the proposed DAE provides lower reusable features across all layers, with the exception of layer 1. Consequently, this result does not convincingly establish the superiority of the proposed DAE over MAE in capturing useful features for medical images. 

6. There is no reference in the text to Figure 6, and its results are not discussed in the paper. 

7. The performance of the proposed DAE varies across Tables 1, 2, and 3, which is confusing. There is a lack of explanation regarding the cause of this discrepancy.
 

[1] Masked Image Modeling Advances 3D Medical Image Analysis, 2023

[2] Towards Foundation Models Learned from Anatomy in Medical Imaging via Self-Supervision, 2023

[3] Joint Self-Supervised Image-Volume Representation Learning with Intra-inter Contrastive Clustering, 2023

[4] DiRA: Discriminative, Restorative, and Adversarial Learning for Self-supervised Medical Image Analysis, 2022

[5] LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching, 2023

[6] Models Genesis, 2021.
questions: questions: Please refer to the weaknesses section.
paper_summary: paper_summary: This work presents a pre-training framework for 3D radiology images. They propose a local masking strategy, and combine it with low-level perturbations to further enable low-level representation learning. The authors curate a large-scale dataset of 10,000 3D medical radiology images (MRI and CT) for pre-training. Multiple downstream tasks are tested achieving state-of-the-art.
main_review: main_review: The authors review the relevant related work, they motivate and propose a novel method. The manuscript is well structured and well written. Their work is extensively validated. Some minor improvements could be considered.
strength_weakness: strength_weakness: - Ablation study. Could you provide clarification regarding the dataset to which the results in Table 2 correspond?
- The main paper lacks discussion of limitations of the proposed method and ideas for future work.
questions: questions: I believe this work has been extensively validated and the manuscript is well structured and well written. I would suggest a discussion about the limitations of the proposed method, future work and availability of the curated dataset.

Comments: title: General Comment to the AC and Reviewers
Official Reply to Reviewer wG9x
Continuation of Official Reply to Reviewer 5JqC
Official Reply to Reviewer 5JqC
Official Reply to Reviewer 8m97
comment: Thank you for your answers.
Thanks to the authors for the revised manuscript. Unfortunately, the changes or additions haven't been highlighted or colored, so this makes it difficult to review them. I am satisfied with their answers to my minor comments in OpenReview and I don't have any further comments.
We believe that we have thoroughly addressed the reviewer's concerns and kindly request the reviewer to let us of any additional issues following our response. We thank the reviewer for their time.
We would like to sincerely thank all the reviewers for their clear, detailed, and genuine comments on our paper. In the comments below, we have addressed all the issued the reviewers had. We also made necessary changes in the revised edition of the paper. We believe the inputs from the reviewers has made our revised version better, and we again thank the reviewers for that.
***1) Ablation Study***

The experiments of ablation study were conducted on a five-fold cross validation of BTCV dataset. We have made sure to clearly state this in the revised version of the paper.

***2) Limitations and future work***

We would like to point the reviewer to the Appendix E for limitations and future works. We had included this in our main submission but have now added more points to it. 

There is  scope for improvement in picking the best combination of $\sigma, \epsilon, r$ for the pre-trained weights. A grid search on these parameters would be time-consuming but would lead to better pre-trained weights. Moreover, it can be noted that it takes a lot of time and computation to conduct pre-training experiments with large-scale datasets on dense medical volumes. There is also scope for improvement in increasing the size of the pre-training dataset to obtain more generalizable weights. For future work, we plan on expanding to more modalities like US, XRs, etc.

***3) Availability of Dataset***

The dataset names and links will be provided along with our code-repository. Due to some policies with TCIA datasets that prevent use from re-sharing it with a new link, we would not be able to share our curated dataset as a single link. However, we will make sure have all the separate links and a readme to help future researchers easily use the pre-training data we used.

***4) Figure/Table and references to them***

We thank the reviewer for this comment and in the revised edition, we have used only Table captions for Tables and Figure captions for Figure. We have also re-structured the paper and cross-links to improve readability.
***8) On CKA values being less in deeper layers***

Contrary to the reviewer’s opinion, we believe that this observation on CKA values being less in deeper layers actually helps strengthen our claim that i) low level features are more important for  medical image pre-training ii) DAE is better than MAE at extracting low-level details. 

To break this down, the early layers have a higher CKA meaning more low-level feature representations were retained from the pre-trained weights to the fine-tuned model. As high-level features undergo a relatively heavier change for both MAE and DAE compared to low-level features after fine-tuning, therefore it is important to focus on learning stronger low-level features during pre-training. Second, we can observe that the CKA values of DAE are larger than MAE, especially in the early layers. This proves that more low-level features are retained for DAE, showing that DAE is better at learning low-level feature representations.



***9) Contrastive loss***

The reviewer also misunderstands the contrastive loss proposed in our work and compares them with [4] and Swin-UNETR (Tang et al.). While the concept of contrastive loss is not new, our loss is devised to distinguish modalities in the latent space to make the pre-trained encoder modality aware.


***10) Organization of the paper and cross-links***

In the revised edition, we have made the structure of the paper better and made sure all the Figures and Tables are cross linked in the text.
***1) Novelty***

The reviewer mentions that our work “merely introduces minor alterations” which we respectfully disagree with. Our method is novel in the following ways: 

**a) Generalized starting weights for multiple modalities:** Unlike any previous medical pre-training works, we provide starting weights that could be used for downstream medical tasks of different modalities proved by the fact we use the same pre-trained weight for all our results.

**b) Local Masking:** We would like to highlight that local masking is not similar to MAE. MAE masks out around 75\%  patches, which will inevitably remove all information across all the regions of the image, and that information needs to be recovered from far away patches and forcing it to learn more global information. Our local masking keeps part of the information for every patch thus forcing the model to see more locally. This forces the model to learn fine-grained information. This is a fundamental change and we showed that it is useful for medical imaging through empirical results as well as sharper reconstructions than MAE as seen in Fig 2. We further direct the reviewer to read Appendix B.

**c) Specialized  3D medical image pre-training:** We develop a new pre-training pipeline considering the reconstruction of fine-grained details of medical images by utilizing low-level perturbations and local masking. Noise and low resolution are commonly found issues in realistic clinical medical acquisition pipelines and we make use of it to develop an effective medical image pre-training pipeline and this has not been done in previous medical pre-training works.


***2) Motivation***

We agree with the reviewer that achieving optimal reconstruction is not the primary objective of MAE or other reconstruction-based SSL methods. However, please note that most reconstructions not really capturing the fine-grained details could potentially cause the encoder to not learn useful information.  In our work, we observed that if the reconstruction quality is good, the downstream performance is also good given the input image is perturbed enough. There is a sweet spot between the perturbation level and reconstruction where the downstream performance is optimal. We find this from experiments reported in Table 8,9,10 of appendix.

***3) DAE Results across Tables 1,2, and 3 (revised: 4,5, and 6)***

To maintain consistency, now we have reported the results on Table 4 and 5 on the five-fold BTCV cross validation dataset. For Table 6, we just did the experiment on a single fold of BTCV validation due to time constraints.

***4) Local Masking vs MAE***

The reviewer also mentions that “there are no experimental results provided to demonstrate the efficacy of local masking standalone over conventional masking in MAE”. For this, we now add MAE comparison to Table 5. It can be seen that Local Masking is directly better than MAE.


***5) Literature Review*** 

We thank the reviewer for pointing out the most recent works. We have added them to the revised version. We would like to point out that [1] was cited in our paper and is actually compared to as well in our experiments (as MAE).  We also cited [6] in the paper.

***6) MAE as a baseline***

MAE is in fact one of the strongest baselines for pre-training in computer vision. Also, it can be argued masking-based techniques are more efficient than [2,3,4,5] as it is faster (due to masking x% of tokens) leading to quadratically reducing the computation while training with masked tokens. That being said, one main aspect of our paper is to show the MAE is not an optimal pre-training strategy for medical image pre-training. We did a fair job in training MAE-based pre-training and we do report our findings that MAE is sometimes worse than training from scratch.

***7) Statistical Significance of MAE boost in FeTA dataset***

While our improvement is 2.6%, 0.9%, and 0.4% in terms of dice accuracy across different distributions of data, we conduct a paired t-test between our FETA test Dice scores and MAE's results. We obtain a two-tailed p-value of 0.0258 which shows our improvement is statistically significant (p < 0.05).
***1) Combining LR Patch to HR noisy Patch***

We apologize to the reviewer for the confusion. By “combine” we still meant “add”. Please note that the LR Patch is added to the noisy HR patch. To avoid any confusion, we have corrected "combine" to "add" in the revised edition. 

***2) Figure 2 and the point it makes***

In Figure 2, we compare the reconstructions obtained from various pre-training techniques like masking, denoising, super-resolution, local masking, and DAE. This figure shows how masked image modeling produces a  coarse reconstruction for radiology without local context while the proposed disruptions in this work obtain sharper reconstructions recovering meaningful fine details.

***3) Clarification of N being data points and the number of tokens***

We apologize to the reviewer for the confusion. In 2, N corresponds to the number of tokens.  In training, we have now changed it to K to correspond to the number of data points to maintain consistency.

***4) About z and Equation 4***

We have removed the typo and have corrected it: zlabel corresponds to the labels.

***5) About Results***

We show results on BTCV (Table 1) as well as FeTA datasets (Table 2). 

***6) Structure and order of the figures***

We apologize to the reviewer for the presentation's mishaps and have made necessary changes in the revised edition to improve readability, structure and the order of figures match the order we talk about them in the text. We have also added details to Figures/Tables for which explanation was missing in the text.

***7) Other Typos***

We have also corrected all the other typos mentioned by the reviewer in the revised edition.