UltraMAE: Multi-modal Masked Autoencoder for Ultrasound Pre-training

Authors: aimon  rahman
Venue: MIDL_io_2024_Conference
Published: 2024-01-05 23:35:12

Abstract: Pre-training on a large dataset such as ImageNet followed by supervised fine-tuning has brought success in various deep learning-based tasks. However, the modalities of natural images and ultrasound images have considerable differences, making pre-training on natural images ineffective for ultrasound-related tasks. In this paper, we introduce a unified masking-based model for both ultrasound images and videos that learns better visual representation than the network with single-modality representations. This is the first large-scale generalized ultrasound pre-training network that simultaneously utilizes 100,000+ videos and images of different parts of the human anatomy such as the liver, bones, heart, thyroids, nerves, etc, making the network an effective benchmark pretrained model for any ultrasound-specific downstream tasks. We propose a novel method for ultrasound image analysis that utilizes an ultrasound-specific confidence map to guide low-level representation learning through masked feature acquisition. Our pre-trained network has demonstrated remarkable efficacy and versatility in tackling both classification and segmentation tasks across a range of ultrasound pathologies, highlighting its potential for widespread adoption and impact in the ultrasound field. In addition, we show that our pre-training model can be leveraged to learn efficiently with a small number of labeled ultrasound images.

Main paper: 
 introduction: 
 The cost-effective, portable, radiation-free, and non-invasive nature of ultrasound (US) imaging has made it a popular choice for clinical diagnosis (Hacihaliloglu et al., 2014) . Recent advancements in deep learning for medical diagnosis, combined with ultrasound's ability to detect various pathologies with reasonable sensitivity, have created opportunities to develop powerful automated tools for medical diagnosis (Rahman et al., 2022a,b; Wang et al., 2018 Wang et al., , 2020)) . These tools can be leveraged to mitigate sonographers' reading burden and improve efficiency as well as the accuracy of the diagnosis. However, one of the major challenges in developing these kinds of networks is the problem of data scarcity, which is typically dealt with by transferring parameters from pre-trained networks (Azizi et al., 2021) . Pre-training on a large natural dataset has been shown to yield a performance boost for the downstream task, however, the significant domain gap between natural and medical images makes pre-training on natural images for downstream medical-related tasks less effective (Chen et al., 2021; Azizi et al., 2021 ). It's worth noting that ultrasound imaging, compared to other medical imaging modalities, encompasses a wide range of clinical applications and specialties, indicating a significant domain gap between ultrasound and other imaging techniques. Furthermore, the use of different transducers, modes, machines, and other factors, such as inter-individual variability and transducer pressures, can result in further variability within the same anatomy (Cruz-Lemini et al., 2016) . As a result, it becomes imperative to use pre-trained models specifically tailored for ultrasound imaging.In recent years, there has been a growing trend to leverage unlabelled datasets for pretraining large-scale models in a self-supervised manner, particularly in medical imaging, where labeling can be costly and require domain-specific knowledge (Azizi et al., 2021; Chen et al., 2021; Basu et al., 2022) . With the rise of transformer networks, one of the most effective pre-training strategies involves masking out certain pixels in input images and predicting their values, which helps the network to learn robust feature representations (He et al., 2022; Tong et al., 2022) . Although recent literature on ultrasound pretraining has demonstrated promising results using various unsupervised techniques for different ultrasound-related tasks (Chen et al., 2021; Basu et al., 2022) , most of these studies have focused on pre-training on a single or a few anatomies, such as liver or lungs, and evaluating on similar downstream tasks. Furthermore, these works have not explored the use of masked autoencoder architectures, which have been shown to outperform other selfsupervised techniques.Ultrasound can be in either video or image format. For example, temporal information is necessary for echocardiography, whereas bone scans may only require images. Hence, a generalized pre-trained model that works both for images and videos is necessary for the ultrasound domain. In this paper, we propose UltraMAE, the first unified masked pretraining model for ultrasound videos and images. Additionally, instead of predicting the input image, UltraMAE predicts the certainty weighted ultrasound image (CWUS) that enables the model to provide special attention to local details in ultrasound scans. CWUS utilizes a confidence map to selectively enhance the areas of high confidence, resulting in a reduction of noise and shadow artifacts that may accumulate during image acquisition. By training a network to reconstruct CWUS, the model is encouraged to learn from relevant local contexts, leading to improved accuracy and precision in downstream diagnosis. In summary, we present the following contributions in this paper:• We are the first to propose a simple yet powerful unified masked pre-training model that integrates both ultrasound images and videos into a single framework. This makes the pre-training model scalable to different ultrasound-related downstream tasks with minimal changes.• We propose a novel framework that aims to reconstruct high-quality ultrasound images/videos enhanced by confidence maps unique to ultrasound -referred to as Certainty Weighted Ultrasound Images (CWUS). This allows the decoder to learn relevant local details in noisy ultrasound images.• Our pre-trained network is trained using ultrasound data of diverse anatomies, including but not limited to 100,000+ images and videos of the heart, liver, lungs, brain, bones, thyroid, nerves., etc., making it the largest pre-trained ultrasound model that is ideally suited for a broad range of ultrasound-related downstream tasks. 
method: 
 To extend the model for both ultrasound images and videos, we pre-train a single unified model which is built on a Vision Transformer (ViT) (Dosovitskiy et al., 2020) . For pretraining, we utilize the basic self-supervised technique of masked auto-encoding (MAE) (He et al., 2022) and build upon it. We use an encoder-decoder architecture where the decoder predicts the Certainty of Weighted Ultrasound Images (CWUS) (see Fig. 2 ) of the noisy input. The network is trained to minimize the reconstruction loss for the masked pixels of the input ultrasound. Following pretraining, we assess the encoder's performance through transfer learning, while disregarding the decoder. The proposed framework is illustrated in Fig. 1 . In the following section, we describe each stage of the pre-training pipeline in detail. Although there has been a substantial improvement in ultrasound image quality, ultrasound attenuation, and shadowing artifacts are still challenging parts of the image acquisition (Hacihaliloglu, 2017) . Various approaches have been discussed in the literature regarding the concept of an ultrasound confidence map. One of these approaches involves using methods that estimate ultrasound attenuation to identify areas that are shadowed or attenuated (Yu and Wang, 2010) . By doing so, these methods can provide valuable information about regions in the image that may be unreliable or noisy. We leverage this confidence map to compute a less noisy version of the original ultrasound image, namely Certainty Weighted Ultrasound Images (CWUS). In order to compute CWUS, we construct per-pixel-based confidence maps, which emphasize the uncertainty in attenuated areas of the ultrasound images. (Karamalis et al., 2012) . The random walk-based framework with ultrasound domain-specific constraint estimates the probability of a random walk's ability to reach the transducer starting from a pixel, assuming that the likelihood of ultrasound transmission is proportional to the image information (Karamalis et al., 2012) . The constraints consider transmission modeling, depth-dependent attenuation, and ultrasound scanline constraints.For the random walk algorithm, the ultrasound image is defined as an undirected weighted graph G = (V, E) with nodes v ∈ V and edges E ∈ e. The probability of a random walk starting from a pixel to reach the first one of the labeled seeds is calculated using the graph Laplacian matrix as follows,EQUATION)where d i = j w ij . The Laplacian matrix can be formulated by utilizing both the graph incidence matrix A and the diagonal matrix of edge weights C, resulting in matrix L = A T CA.Then, L is re-ordered and decomposed into blocks of marked nodes M and unmarked nodes U as followsEQUATIONThe final probabilities are then obtained by solving,EQUATIONwhere, x U and x M are the unknown and known probabilities for the unmarked and seed nodes respectively. The attenuation of the signal is defined using the Beer-Lambert law, where we define the attenuated signal I by,EQUATIONwhere, I 0 is the initial intensity, α is the attenuation coefficient and d is the distance from the source. Ultrasound image formation uses a narrow, focused beam of sound to acquire each scanline. The random walk framework models the beam width, allowing for sound to contribute from structures close to a scanline and limiting bending around reflectors. These two constraints are integrated into the algorithm by incorporating a new weighting function (Karamalis et al., 2012) ,EQUATIONwhereEQUATIONEQUATIONw H ij = exp(-β(|c i -c j | + √ 2γ)) (8) w H ij = g i exp(-αl i ) (9)and, E H , E V , and E D are edges along the horizontal, vertical, and diagonal graph direction, g i is the image intensity respectively. α is the parameter that indicates the likelihood of vertical random walk, β affects the randomness and accuracy of the confidence estimation and γ penalizes horizontal and diagonal random walks in the graph away from the starting scan line. Finally, we obtain CWUS by multiplying the original input US with the confidence.We follow the same pipeline as (Girdhar et al., 2022a) for the network architecture while using ViT-B (Dosovitskiy et al., 2020) as the backbone. The input ultrasound or video is considered as a 4D tensor of T × H × W × 3, where H, W represents height and widths and T is the temporal dimension, which is set equal to 1 in the case of images. To process both ultrasound images and videos, an omnivorous (Girdhar et al., 2022b) encoder is utilized that takes variable N spatiotemporal patches. The patches are randomly masked with a factor of M and non-masked patches with positions are fed to the encoder to produce the patch embeddings. Then the output from the encoder is concatenated with M learnable tokens to get N embeddings. The positional encodings to N embeddings are added and fed to the decoder that learns to produce the certainty-weighted ultrasound images. Then, we minimize the l 2 distance between the decoder predictions and the CWUS scans over the M masked patches. The transformer decoder consists of 4 layers with 384 embedding dimensions (Girdhar et al., 2022a ) that output the CWUS pixels in all input patches. 
datasets: 
 Pre-training Datasets. We combine various public and private ultrasound images and videos of various anatomies such as lungs (Chen et al., 2021) , bones (Wang et al., 2018; Mohabir, 2020) , breasts (Li et al., 2022) (Pedraza et al., 2015) : The dataset contains the ultrasound images and their corresponding segmentation masks of thyroid nodules. The dataset contains 870 images in total and is split into 90:10 ratios for training and testing.The proposed framework is implemented using PyTorch. We utilize the ViT-B backbone without [CLS] token with an omnivorous encoder. Both the encoder and the decoder utilize sinusoidal positional encoding for the patches. For pre-training, we mask out 90% and 95% of patches respectively in images and videos, as this ratio has been found to yield optimal results (Girdhar et al., 2022a) . To process video ultrasound, we capture a segment of T = 16 frames with a frame rate of 6 FPS. The dimensions of both input images and videos are resized into 224 × 224 pixels. The rest of the parameters for the pre-trained model are the same as (Girdhar et al., 2022a) . The model is trained for 800 epochs on the pretraining datasets. For the pre-training process, we utilized 4 NVIDIA RTX A5000 GPUs, each with a capacity of 24 GB. For the downstream classification task, we discard the original decoder and add a linear layer of 768 input dimesions (Girdhar et al., 2022a) which maps this feature vector to a number of classes-dimensional output vector. For the downstream segmentation task, we utilize the same decoder architecture as pre-trained network, except the last layer produces pixel-wise probabilities of the binary segmentation masks. 1 . We pre-train a vanilla masked-autoencoder with the pre-train dataset and consider the model as a baseline for the masking-based pre-training method. Additionally, we compare our pre-training framework with USCL (Chen et al., 2021) and SUSCL (Basu et al., 2022) , which are specialized networks for pretraining on US datasets and are the current state-of-the-art. We chose USCL and SUSCL because they are ultrasound-specific pre-training models. We can observe that UltraMAE outperforms the previous state-of-the-art pretrained network as the model utilizes a masking-based approach. It's also evident that UltraMAE outperforms vanilla MAE, as the network leverages an omnivorous encoder that captures better data representation.In this section, we report the fine-tuning results on all the datasets to validate the different contributions of our proposed approach. We pre-train our backbone network in five different settings as shown in Table 2 . The network trained on both videos and images performed better than the network trained on a single modality. Simply turning all the video into image frames and applying MAE result in a lot of redundancy, hence the performance is not on par with the combined model. The performance improves even more across all the datasets when the UltraMAE utilizes the confidence map for reconstruction, as adding a deblurring component has proven to help while reconstructing masked tokens (Kang et al., 2023) .In our study, we conducted a series of experiments on various splits of the dataset to demonstrate the effectiveness of our pretrained method under limited-annotation scenarios. Specifically, we performed experiments on PCOS (classification) and HEM (segmentation) data across four different data splits from 10% to 50% and visualized the results in Fig. 3 .The figures indicate that the gap between the results of UltraMAE and the from-scratch experiments is particularly pronounced when working with limited amounts of data. This emphasizes the critical importance of pre-training for ultrasound imaging tasks, where data is often scarce.  
conclusion: 
 In this work, we have introduced the first unified masking-based pre-training model for the ultrasound domain. To the best of our knowledge, this is the first study that leverages confidence-guided enhanced videos and images of ultrasound for masked pertaining. We utilized video modalities as well, as there are a lot more video ultrasound datasets but not utilized during the pre-training strategy. We train our model with a wide range of ultrasound images and videos of various anatomies obtained using different types of machines, making it the largest ultrasound pre-trained model. We evaluate our model for both classification and segmentation tasks of different human anatomy. In the future, we aim to train even larger models with other ultrasound modalities such as 3D ultrasounds. Lastly, there is also scope for improvement by making α, β and γ trainable parameters for constructing optimal CWUS images. 
 
 Appendix:  


Decision: This paper introduces a unified pre-trained model for large-scale unlabelled ultrasound datasets from diverse anatomies. Initially reviewers raise diverse comments ( three borderline and one weak accept in preliminary rating). After response period, one reviewer raised the rating to borderline accept and one decreased the rating to weak reject. Reviewers appreciate the merits of this paper on pre-training on large-scale unlabelled ultrasound dataset, reconstructing certainty weighted ultrasound, instead of the original image, to recover details and be more robust to noise and demonstrating better performance, etc. While reviewers also mention some key issues remain unsolved, including clarification is essential regarding whether the quantity of training frames is controlled, the unclear value of CWUS bringing compared to other commonly used reconstruction targets, etc. After reconciling all the comments, AC tends to suggest the decision of accept given the merits of this paper. In the meanwhile, AC strongly suggest the authors take reviewers’ comments into thorough revision for preparing the final version.

Reviews: review: paper_summary: paper_summary: This paper introduces a unified pre-trained model for large-scale unlabeled ultrasound datasets, comprising both images and videos. The contributions are threefold. Firstly, instead of reconstructing raw pixels like MAE, the paper argues that the Certainty of Weighted Ultrasound Images is a more suitable target for reconstruction, particularly due to the potential noise in ultrasound images. Secondly, the paper pioneers pre-training on a large scale of 100,000 videos and images, demonstrating enhanced generalization. Lastly, the paper validates the effectiveness of segmentation and classification tasks.
main_review: main_review: I appreciate the authors' efforts in addressing the concerns raised in the rebuttal and I would like to express my gratitude for their time and dedication.

While there are still some unresolved concerns that remain after reviewing the rebuttal, I am pleased to see this paper accepted by MIDL.
strength_weakness: strength_weakness: - In the methodology section, it appears that videos are processed similarly to images by splitting them into individual frames and feeding each frame to the backbone. This design overlooks temporal correspondence in self-supervised learning for video analysis[1,2].
- In the experiments:
    - VideoMAE[3] could serve as a strong baseline when pre-training solely with video data. The proposed method should also be compared with UniMISS[5], which deals with 2D and 3D data together.
    - Could the author evaluate some video-related tasks? Since the proposed method is pre-trained on both images and videos, it should generalize to these modalities in downstream tasks.
    - Could the author supplement the performance metrics when scaling the pre-training datasets? It would be intriguing to observe if scaling laws[4] also apply to ultrasound images.
    - Can the author provide performance metrics when varying the mask ratio?

[1] X. Wang, A. Jabri, and A. A. Efros, “Learning Correspondence From the Cycle-Consistency of Time,” in CVPR, Long Beach, CA, USA, Jun. 2019, pp. 2561–2571. [2] H. Duan, N. Zhao, K. Chen, and D. Lin, “TransRank: Self-supervised Video Representation Learning via Ranking-based Transformation Recognition,” in _CVPR_, New Orleans, LA, USA, Jun. 2022, pp. 2990–3000. 
[3] Z. Tong, Y. Song, J. Wang, and L. Wang, “VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training,” in _NeurIPS_, 2022. 
[4] J. Kaplan _et al._, “Scaling Laws for Neural Language Models.” arXiv, Jan. 22, 2020. 
[5] Y. Xie, J. Zhang, Y. Xia, and Q. Wu, “UniMiSS: Universal Medical Self-supervised Learning via Breaking Dimensionality Barrier,” in _ECCV_. 2022, pp. 558–575.
questions: questions: - For the experiments:
    - VideoMAE[3] could serve as a strong baseline when pre-training solely with video data. The proposed method should also be compared with UniMISS[5], which deals with 2D and 3D data together.
    - Could the author evaluate some video-related tasks? Since the proposed method is pre-trained on both images and videos, it should generalize to these modalities in downstream tasks.
    - Could the author supplement the performance metrics when scaling the pre-training datasets? It would be intriguing to observe if scaling laws[4] also apply to ultrasound images.
    - Can the author provide performance metrics when varying the mask ratio?
paper_summary: paper_summary: The paper proposes pretraining masked autoencoder where: i) both ultrasound images (US) and videos are utilized leveraging the transformer-based omnivore architecture, and ii) the reconstruction target for the decoder is certainty weighted ultrasound image which is obtained by weighting original US image with a confidence map generated using random walk from each pixel guided by US image formation model. Pretraining is performed using large number of images and videos, and then classification and segmentation performance is evaluated by finetuning in a set of separate small-scale datasets. The results show improvement over few other self-supervised pretraining.
main_review: main_review: The main concern persists as the rebuttal keeps this for future work: lack of proper ablation studies to identify whether the improved performance comes from the key contribution of the paper, ultrasound physics-based reconstruction target, or not.
strength_weakness: strength_weakness: - One of the major novel contributions of the paper seem to be in using CWUS as the decoder's target in masked autoencoder training. However, there is limited ablation study on its role against various baselines: keeping same training data and test sets, and same transformer-based omnivore architecture with masked autoencoding, but different decoder targets (noisy version of input, random weighting, edge-based saliency weighting that does not necessarily take US physics into account etc.)

- Some basic baselines and ablation studies could have provided better insights into the method's ability to learn right representation specific to US images (detailed comments)
questions: questions: Questions in Weakness and detailed comments.
paper_summary: paper_summary: This paper proposes a strategy for developing a pre-trained network for ultrasound images and videos. Their model is built on a Vision Transformer (ViT). For pre-training, they utilize the self-supervised technique of masked auto-encoding (MAE). They propose to reconstruct the certainty of weighted ultrasound images leveraging confidence maps to guide low-level representation learning. The model is tested for both classification and segmentation across a range of pathologies.
main_review: main_review: This paper presents a strong but complex method.  I am hesitant about the necessity of employing such complexity to achieve only a slight improvement. My main concerns are about the validation, in terms of the statistical significance of the reported improvement, and the use of a dataset (BUSI) that is missing the provenance and any documentation.
strength_weakness: strength_weakness: - As far as I understand, reported results in Table 1 are for a single model. UltraMAE outperforms all the compared methods by a small improvement. I am concerned about the lack of cross-validation and the statistical significance of the improvement.
- The citation for the BUSI dataset references a Kaggle dataset. Could you please provide the source of this dataset? There is an open discussion regarding image duplication within both the benign and malignant classes of this dataset. I'm concerned about the reliability of the results obtained from using this dataset.
- Regarding the experiments about limited data. I expect any pre-training strategy to work better than the from-scratch experiments. A fairer comparison would be against other pre-training strategies like USCL or SUSCL, or pre-training datasets like ImageNet.
- In the paper it is stated that the code and model will be made available after the review process. However, as a reviewer I cannot ensure the reproducibility and transparency of the results at this stage.
- There are some important details missing in the abstract, e.g. about the proportion of “limited data” that makes a difference for the pre-training, or the use of “efficacy” in terms of training time or accuracy?
questions: questions: 1. Concerns about validation (Table 1 and Table 2)
- Justify why there is no cross-validation in this study (Table 1) and why performance is reported for a single model (Table 1 and Table 2)
- MAE seems to be a much simpler model, is it worth it adding the extra complexity? - elaborate

2. Concerns about BUSI dataset source
- Could you specify the source of this dataset and the number of images for the benign and malignant classes?
- There's an open discussion regarding potential image duplication within both the benign and malignant classes. I'm concerned about the reliability of the results obtained from using this dataset, particularly due to ethical implications surrounding the use of research datasets with unknown sources and lack of documentation.

3. Limited data experiment
I expect any pre-training strategy to work better than the from-scratch experiments. A fairer comparison would be against other pre-training strategies like USCL or SUSCL, or pre-training datasets like ImageNet. 
- Could you justify the choice of baselines? 
- How about comparing to other natural images datasets such as ImageNet?
paper_summary: paper_summary: This paper proposes a novel method for ultrasound image analysis that utilizes an ultrasound-specific confidence map to guide low-level representation learning through masked feature acquisition. The authors demonstrated the efficacy and versatility of the framework in tackling both classification and segmentation tasks.
main_review: main_review: I am a bit disappointed by the absence of a rebuttal for my concern, as it suggests a missed opportunity for clarification. However, considering the merit of the work, I maintain my final rating as a weak accept.
strength_weakness: strength_weakness: 1. The author's assertion regarding unifying image and video within a pretraining framework appears exaggerated. Image pretraining, essentially a subset of video training, has been previously suggested in works such as USCL or VideoMAE.

2. The question remains open regarding whether a model pretrained on both modalities exhibits superior performance compared to a model pretrained on just one. Clarification is essential regarding whether the quantity of training frames is controlled, as it significantly impacts determining whether improvements stem from the quantity of training samples or solely from the temporal dependencies introduced by video. The author should address this ambiguity.
questions: questions: see weakness

Comments: title: Assessing the value of CWUS
comment: The key contribution of the paper is the approach claimed to be guided by US physics (CWUS), rather than the autoencoder architectures; thus, the main question is what value is CWUS bringing compared to other commonly used reconstruction targets. Hence, the answer that "the focus was solely on US-based reconstruction techniques" does not address the main concern I have with the paper.
Thanks to the authors for answering my questions.

1. BUSI dataset source
I appreciate the clarification. However, in the manuscript, the reference for the BUSI dataset is still a Kaggle URL. As I pointed out in my original review, the dataset hosted on Kaggle doesn't contain information about the provenance of the BUSI dataset, nor does it include any documentation. Regarding the duplicated malignant/benign images, could you clarify if you have taken any steps to identify and remove such instances?

2. Pre-training
Thanks for the explanation. Unfortunately, the changes or additions haven't been highlighted or colored, so this makes it difficult to review them. Could you clarify the changes in the manuscript about the pre-training strategies and ImageNet?

3. Comparison with MAE. Thanks for the clarification about the difference with respect to MAE. However, I am not convinced about the argument of "powerful versatile foundation model".
We believe that we have thoroughly addressed the reviewer's concerns and kindly request the reviewer to let us of any additional issues following our response. We thank the reviewer for their time.
We believe that we have thoroughly addressed the reviewer's concerns and kindly request the reviewer to let us of any additional issues following our response. We thank the reviewer for their time.
We believe that we have thoroughly addressed the reviewer's concerns and kindly request the reviewer to let us of any additional issues following our response. We thank the reviewer for their time.
Thank you for your reviews:
Q: VideoMAE[3] could serve as a strong baseline when pre-training solely with video data. The proposed method should also be compared with UniMISS[5], which deals with 2D and 3D data together.

Table 2 specifies that VideoMAE is utilized solely for video data. We will consider UniMISS in future research effort.

Q: Could the author evaluate some video-related tasks? Since the proposed method is pre-trained on both images and videos, it should generalize to these modalities in downstream tasks.

At present, we lack access to a video-related downstream dataset suitable for our experiments; however, we are keen on incorporating such datasets in our future research.

Q: Can the author provide performance metrics when varying the mask ratio?

The selection of the masking ratio was guided by the parameters used in the foundational work of OmniMAE, where a 90-95% masking ratio for both images and videos was found to yield optimal results. Therefore, we adhered to these established percentages in our study.
Thank you for your review.

Q: One of the major novel contributions of the paper seem to be in using CWUS as the decoder's target in masked autoencoder training. However, there is limited ablation study on its role against various baselines: keeping same training data and test sets, and same transformer-based omnivore architecture with masked autoencoding, but different decoder targets (noisy version of input, random weighting, edge-based saliency weighting that does not necessarily take US physics into account etc.)

A: Regarding the ablation studies, while Table 2 provides insights into the effects of different network architectures, we did not explore alternative reconstruction methods beyond those guided by ultrasound (US) physics. Although existing literature suggests that approaches such as input blurring can enhance feature learning, our focus was solely on US-based reconstruction techniques. We acknowledge the potential benefits of exploring these alternative methods and plan to incorporate them in future research efforts.



Q: A very basic baseline in Table 2 during ablation would be to finetune the same architecture and of same size with random initialization instead of the proposed pertaining; how much improvement for each test dataset separately is being added by pertaining?

A: For the initialization process, Section 3.3 and Figure 3 describe our approach, which involves randomly initializing the network while maintaining the same architecture across all experiments.

Q: While the two SOTA in US are reasonable for comparison, are they trained and tested in the same sets of data for SOTA in original version? If not, adding a couple of most popular SSLs such as SIMCLR in the current setting would provide a better idea regarding the value of MAE; this will also support the Introduction, which mentioned that Masked autoencoder architecture "shown to outperform", although no work is cited to support this where the experiments are done in settings similar to the one in this paper.

A: Regarding benchmarking against state-of-the-art (SOTA) models, we ensured that all baselines were trained and evaluated using the identical dataset as those considered SOTA. Given that one of the referenced papers already includes a comparison with SimCLR and demonstrates superior performance of the baselines, we chose not to replicate these experiments.
Thank you for your insightful review. Regarding cross-validation and statistical significance, we acknowledge the importance of specifying the p-value, which is less than 0.05. We plan to update the paper with mean and standard deviation in future versions.

Regarding the BUSI dataset source and issue: Al-Dhabyani, W., Gomaa, M., Khaled, H., & Fahmy, A. (2020) presented a dataset of breast ultrasound images in their work titled "Dataset of breast ultrasound images" published in Data in Brief, volume 28, article 104863. We appreciate you bringing to our attention the issue of data duplication across both malignant and benign classes. We will remove the duplicated instances. However, it's important to note that we used the BUSI dataset solely for tumor segmentation purposes (binary segmentation), which means the presence of duplicated malignant/benign images does not affect the outcome of the diagnosis. This is unless they appear in both the training and test sets, which would indeed constitute data leakage—a separate concern. We will take steps to identify and remove any such instances. Thank you for highlighting this issue.

Regarding the clarification on pre-training: In the context of limited data settings, we have not compared our approach with other baselines since we have already conducted comparisons using the full dataset scenario. The objective of that experiment was to demonstrate that pre-training yields better results than training from scratch, especially when the available data is extremely limited. Nonetheless, we recognize that any network benefiting from pre-training is likely to outperform a network trained from scratch. Both the USCL and SUSCL methodologies were compared with ImageNet pre-training; thus, we did not train any model explicitly using an ImageNet model. We chose USCL and SUSCL because they are ultrasound-specific pre-training models. However, for the PCOS dataset, we used a ResNet-50 model pre-trained on ImageNet and fine-tuned it to achieve a 98.12% accuracy with the full dataset. We will clear and update it in future versions.

Minor improvements include the following updates and clarifications:

1. Figure 2 will be revised to include better-quality images that more clearly highlight the differences. Additionally, other editorial oversights such as missing spaces and the need to define certain matrices will be corrected.

2. Regarding Equation 9, we confirm its correctness. To prevent any confusion, we will amend the equation in the final version to read "g_i * exp..." 

3. For the pre-training process, we utilized 4 GPUs, each with a capacity of 24 GB.

4. We plan to include segmentation results in a future version of the paper to provide a more comprehensive analysis.

5. Baselines are chosen as they are specifically designed for ultrasound modalities, as well as, already compared against other pre-training strategies. Moreover, we also only train on MAEs/VideoMAEs for fair comparison with MAE-based models.

MAE seems to be a much simpler model, is it worth it adding the extra complexity? 

We utilized video modalities as well, as there are a lot more video ultrasound datasets but not utilized during the pre-training strategy. Simply turning all the video into image frames and applying MAE will result in a lot of redundancy. Learning from both images and spatiotemporal data proved to improve [1] the overall performance, and, adding a deblurring component helps [2] while reconstructing masked tokens. We try to combine both for a powerful versatile ultrasound-based foundation model that can take both image and video input.

Girdhar, R., El-Nouby, A., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023). Omnimae: Single model masked pretraining on images and videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10406-10417).
Kang, Q., Gao, J., Li, K., & Lao, Q. (2023, October). Deblurring masked autoencoder is better recipe for ultrasound image recognition. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 352-362). Cham: Springer Nature Switzerland.