A Comprehensive Benchmark of Supervised and Self-supervised Pre-training on Multi-view Chest X-ray Classification

Authors: muhammad muneeb afzal
muhammad  osama khan
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 14:33:36

Abstract: Chest X-ray analysis in medical imaging has largely focused on single-view methods. However, recent advancements have led to the development of multi-view approaches that harness the potential of multiple views for the same patient. Although these methods have shown improvements, it is especially difficult to collect large multi-view labeled datasets owing to the prohibitive annotation costs and acquisition times. Hence, it is crucial to address the multi-view setting in the low data regime. Pre-training is a critical component to ensure efficient performance in this low data regime, as evidenced by its improvements in natural and medical imaging. However, in the multi-view setup, such pre-training strategies have received relatively little attention and ImageNet initialization remains largely the norm. We bridge this research gap by conducting an extensive benchmarking study illustrating the efficacy of 10 strong supervised and self-supervised models pre-trained on both natural and medical images for multi-view chest X-ray classification. We further examine the performance in the low data regime by training these methods on 1%, 10%, and 100% fractions of the training set. Moreover, our best models yield significant improvements compared to existing state-of-the-art multi-view approaches, outperforming them by as much as 9.9%, 8.8% and 1.6% on the 1%, 10%, and 100% data fractions respectively. We hope this benchmark will spur the development of stronger multi-view medical imaging models, similar to the role of such benchmarks in other computer vision and medical imaging domains. As open science, we make our code publicly available to aid in the development of stronger multi-view models.

Main paper: 
 introduction: 
 Medical imaging analysis has benefited significantly from improvements in deep learning and computer vision in recent years (Rajpurkar et al., 2020 (Rajpurkar et al., , 2021)) . These developments have been largely enabled by the availability of large chest X-ray datasets. However, most existing methods (Rajpurkar et al., 2017 (Rajpurkar et al., , 2018;; Li et al., 2018; Cohen et al., 2019) still largely rely on a single view (e.g. frontal) for chest X-ray analysis. In clinical practice, however, some structures and pathologies are more readily distinguishable from a lateral X-ray (Bertrand et al., 2019) . Hence, it is particularly appealing to develop methods that utilize multiple views, such as frontal (PA) and lateral (L) views. To this end, recently methods have been developed that exploit multiple views to improve chest X-ray recognition. For instance, DualNet (Rubin et al., 2018) observed an improvement of 3% mean AUC by utilizing both PA and L views. Similarly, Hashir et al. (Hashir et al., 2020) studied various methods of combining PA and L views, resulting in improvements on 32 pathologies.However, due to the challenges associated with obtaining large labeled datasets for medical imaging in general and multi-view imaging in particular, it is especially pertinent to focus on the low data regime in the multi-view setting. In such low data regimes, transfer learning via either supervised or self-supervised pre-training has shown significant improvements in both natural imaging (Xu et al., 2022; He et al., 2020; Chen et al., 2020b) and medical imaging (Azizi et al., 2022) settings. For instance, Taher et al. (Hosseinzadeh Taher et al., 2021) recently presented a thorough study illustrating the efficacy of various supervised and self-supervised learning methods for single-view medical imaging. However, in the multi-view setup, such pre-training strategies have received relatively little attention and ImageNet initialization remains largely the norm.We bridge this gap by providing an extensive study of various pre-training methods for multi-view chest X-ray classification in the low data regime. While the efficacy of supervised and self-supervised pre-training in medical imaging is known, our work addresses the research gap of its applicability in the multi-view setting. Our focus on low-data regimes, pertinent due to the high creation costs of multi-view datasets, highlights our study's practical value, offering insights that are immediately relevant to current healthcare settings. Specifically, we investigate the low data regime by training on 1%, 10%, and 100% fractions of the training set. We evaluate strong supervised and self-supervised learning methods that have demonstrated significant performance enhancements across a range of domains, including computer vision and medical imaging.Concretely, we study four transfer learning strategies (Fig. 1 ) -1) supervised learning on natural images, 2) self-supervised learning on natural images, 3) supervised learning on medical images, and 4) self-supervised learning on medical images. We fine-tune these pre-trained models for multi-view chest X-ray classification on the PadChest dataset and achieve improvements of as much as 9.9% in the low data regime compared to existing stateof-the-art methods. To summarize, we provide a timely benchmarking study investigating several strong pre-training methods for multi-view chest X-ray recognition.Briefly, our main contributions are:• We present the first extensive multi-view benchmarking study illustrating the efficacy of 10 strong supervised and self-supervised methods pre-trained on both natural as well as medical images.• We are the first to systematically study multi-view chest X-ray classification in a lowdata regime by training on various different data fractions. This data-efficient setting is of significant practical importance owing to the huge data acquisition times and annotation costs associated with collecting large multi-view labeled datasets.• Our best models yield significant improvements compared to existing state-of-the-art multi-view approaches, outperforming them by as much as 9.9%, 8.8% and 1.6% on the 1%, 10% and 100% data fractions respectively. 
related work: 
 Although chest radiography has been investigated extensively in the medical imaging community, most methods have relied on a single (i.e. the frontal) view. Many of these works (Yao et al., 2017; Rajpurkar et al., 2017; Guan et al., 2018; Kumar et al., 2018; Guan et al., 2018; Baltruschat et al., 2019) use the frontal posteroanterior (PA) view of chest x-rays to classify different diseases. Recently, however, multiple views (frontal and lateral) have also been used, emulating the usual radiology practice where multiple views are taken into account for diagnosis of chest X-rays. For instance, (Rubin et al., 2018) proposed DualNet that used two separate branches for frontal and lateral views to predict diseases. (Bertrand et al., 2019; Hashir et al., 2020) demonstrated that including lateral views with frontal views enhances performance on 32 PadChest labels. Building on DualNet, they introduced a revised architecture with auxiliary losses and curriculum learning. Recently, (Hosseinzadeh Taher et al., 2021) presented a benchmarking analysis of several strong transfer learning techniques for medical imaging. However, in contrast to (Hosseinzadeh Taher et al., 2021) who investigate single-view tasks, we study these pre-training techniques for multi-view chest X-ray analysis since it is not clear a-priori that the same trends translate to the multi-view setting owing to the unique training dynamics of multiview systems (Wu et al., 2020 (Wu et al., , 2022)) . Moreover, we focus on the data-efficient multi-view setting, which is of great clinical relevance but has not yet been explored. Hence, our work bridges these gaps by extensively evaluating various supervised and self-supervised pre-training strategies for data-efficient multi-view chest X-ray analysis. More details on related multi-view methods and pre-training methods can be found in Appendix A. 
methods: 
 Fig. 1 shows an overview of our approach. In the first stage, we pre-train our networks using one of the following three approaches: 1) Supervised learning on natural images, (2) Selfsupervised learning on natural images, or (3) Supervised learning on medical images and 4) Self-supervised learning on medical images. In the second stage, for downstream fine-tuning in the multi-view setting, we select the pre-trained backbone and use it to initialize both the frontal (PA) and lateral (L) view models. This multi-view model is then fine-tuned with the downstream data on the PadChest dataset.Implementation Details We use the pre-trained backbones from the official implementations, using the code provided by (Hosseinzadeh Taher et al., 2021) to pre-process and load the models. All our experiments use a ResNet50 backbone except for self-supervised methods in the medical domain where densenet121 is the de-facto standard. We applied a standardized set of hyperparameters across models for comparable results. All the multiview models are implemented in PyTorch and are trained for 100 epochs with an Adam optimizer, a batch size of 8, and a learning rate of 1e -4 on an NVIDIA V100 GPU.In the past, most publicly accessible chest X-ray datasets only offered one view (frontal) (Rajpurkar et al., 2017 (Rajpurkar et al., , 2018;; Li et al., 2018; Cohen et al., 2019) . However, recently, with the availability of large chest X-ray datasets (Irvin et al., 2019; Johnson et al., 2019; Bustos et al., Figure 1 : Left: Four pre-training strategies -1) Supervised learning on natural images, 2) Self-supervised learning on natural images, 3) Supervised learning on medical images and 4) Self-supervised learning on medical images. Right: Supervised fine-tuning with pre-trained encoders on paired frontal and lateral chest X-rays. 2020) with paired frontal and lateral views, it has become feasible to investigate the efficacy of paired multi-view data for chest X-ray analysis. For this study, the PadChest dataset (Bustos et al., 2020) , which contains 160,868 chest x-ray images including paired frontal (PA) and lateral views, is utilized. We selected PadChest for our benchmarking effort due to its comprehensive annotations and diverse multi-view chest X-ray images. We prepare the multi-view dataset using the same setup as that followed by (Hashir et al., 2020) . Distinct from prior multi-view models, we introduce a method to evaluate the efficacy of our models on different training data fractions. To this end, we devise the following three fractions of the train set of the PadChest multi-view dataset described above: 1%, 10%, and 100%. In creating the 1% and 10% data fractions, we aimed to maintain a distribution of conditions that closely mirrors the full dataset. This process involved random sampling of images to roughly preserve the pathology distribution for each subset. This setup helps us evaluate how decreasing dataset size affects multi-view chest X-ray classification performance. For a fair comparison with prior works (Hashir et al., 2020) , we use the same 60/20/20 split for train, validation and test respectively, and use the AUC (Area under the ROC) metric for evaluation. We conduct each experiment three times, with each run representing a different randomly selected training split to obtain more accurate results.For pre-training, we experiment with 10 different methods (see Fig. 2 ) which can be categorized into the following: 1) supervised learning on natural images, 2) self-supervised learning on natural images, 3) supervised learning on medical images, and 4) self-supervised learning on medical images. Scratch refers to the baseline method, where we train the multi-view model starting from a random initialization. For the first category, for supervised learning on natural images, we use the standard ImageNet dataset (Deng et al., 2009) . Secondly, we employ the following four strong self-supervised learning methods on natural images: MoCov2 (Chen et al., 2020c), SimCLRv2 (Chen et al., 2020b) , BYOL (Grill et al., 2020) , and Barlow Twins (Zbontar et al., 2021) . These methods are representative of major seminal works in the self-supervised domain. Thirdly, for supervised pre-training on medical images, we leverage the Chest X-ray14 (Wang et al., 2017) which is the de-facto dataset for Chest Xray classification. Lastly, for self-supervised learning on the medical domain, we experiment with MoCo-v2 (Chen et al., 2020c) and Masked Auto Encoder (MAE) (He et al., 2022) that are pre-trained on medical data in a self-supervised fashion (Xiao et al., 2023) . Furthermore, we also experiment with Robust and Efficient Medical Imaging with Self-Supervision (REMEDIS) (Azizi et al., 2022) which is the state-of-the-art method in self-supervised medical imaging that combines supervised transfer learning with self-supervised learning.For the downstream task, we use the pre-trained encoder (Fig. 1 ) and use it to initialize both the frontal and lateral backbones, which are shared during training. The representations of both these backbones are combined before finally yielding the classification prediction. This multi-view network is trained and evaluated on the PadChest dataset, containing paired frontal and lateral views, described above. 
results: 
 In this section, we present extensive results and analyses illustrating the efficacy of various pre-training methods for multi-view chest X-ray analysis. Concretely, we study supervised and self-supervised learning methods trained on both natural and medical imaging domains. Sec. 4.1 presents the benchmark of 10 different pre-training methods for data-efficient multiview learning. This is followed by a fine-grained analysis of the effect of different pre-training strategies on multi-view classification in Sec. 4.2. Lastly, Sec. 4.3 compares the performance of these pre-trained models against current SOTA multi-view methods, where it surpasses them by significant margins, especially in the low data regime. We further delve into the clinical effectiveness of the aforementioned pre-training methods, conducting a detailed analysis on the 14 commonly occuring diseases featured in the ChestXray14 dataset (see Appendix C).Here, we provide a thorough benchmark of 10 different methods, including 5 pre-trained on natural images and 4 pre-trained on medical images, as illustrated in Fig. 2 . For completeness, we compare all the methods against a random initialization baseline. Within both the natural and medical imaging pre-training settings, 1 method is pre-trained via supervised learning whereas others are pre-trained via self-supervised learning owing to the superiority of self-supervised learning witnessed in recent years. Table 1 shows the mean AUC (on 64 pathologies) across 3 labeled fractions (1%, 10% and 100%) of the multi-view PadChest dataset (Bustos et al., 2020) .Observe that 100% here corresponds to ∼19k labeled pairs of (PA+L) views which is especially difficult to collect, requiring years of acquisition time and hundreds of clinical annotation hours. This issue is exacerbated for multi-view data acquisition and annotation. Hence, it is pertinent to focus on the data-efficient fractions in the multi-view setting.Table 1 demonstrates the significant gains achieved by leveraging supervised and selfsupervised pre-training for multi-view chest X-ray diagnosis. Except for Barlow Twins and BYOL, all of the pre-training methods surpass the random initialization baseline used in prior works (Rubin et al., 2018; Hashir et al., 2020) across all 3 data fractions, highlighting the efficacy of these pre-training methods. The underperformance of Barlow Twins and BYOL suggests their optimization strategies might not be fully compatible with the intricacies of multi-view chest X-ray data. Specifically, the Barlow Twins' focus on reducing feature redundancy could remove essential, nuanced details vital for accurate X-ray classification. Similarly, BYOL relies on an asymmetrical architecture where the online network predicts the target network's representation. This setup might not effectively capture the nuanced differences between multiple views of chest X-rays. The similarity in performance between supervised pre-training with ChestXray14 and self-supervised pre-training with REMEDIS is notable and highlights the advantage of domain-specific pre-training. However, it is crucial to note that such direct, condition-specific supervision is often unfeasible in real-world applications. Hence, in cases where such large labeled datasets are not readily available, effective SSL methods like REMEDIS are pivotal.The advantages of pre-training methods become evident in the low data regime. For instance, in the data-efficient setting of 1% and 10%, the best methods, MoCov2 (medical) and REMEDIS, surpass the random baseline by big margins of 8.3% and 7.1% respectively. Hence, self-supervised and supervised pre-training is highly effective for multi-view chest X-ray diagnosis. Given these significantly improved performances, we hope practitioners will switch to using these pre-trained models as an alternative to random initialization.To decouple the effects of different pre-training strategies and datasets on downstream multi-view chest X-ray diagnosis, we compare the random initialization baseline against the best performing models from the following pre-training strategies -1) supervised learning on natural images, 2) self-supervised learning on natural images, 3) supervised learning on medical images, and 4) self-supervised learning on medical images. Supervised or Self-supervised: Overall, we observe that self-supervised pre-training tends to outperform supervised pre-training across both the natural and medical imaging domains (see Fig. 3 ). This trend aligns well with the recent works (Azizi et al., 2022; Chen et al., 2020b,c; Zbontar et al., 2021) in computer vision and medical imaging which show the superiority of self-supervised approaches over supervised pre-training. Natural or Medical: Across both supervised and self-supervised methods, in-domain pre-training on medical images yields better representations for the downstream multi-view chest X-ray classification task, as evidenced by the results in Fig. 3 . This highlights the efficacy of in-domain pre-training and emphasizes the importance of acquiring more data, especially in the multi-view setting in the medical imaging domain. Importance of Medical Pre-training: To show that the performance gains in our study are primarily due to pre-training rather than the specific method used, we conduct an ablation study with SimCLR v2 (Chen et al., 2020b) and SimCLR v2 + Medical-Pretraining (REMEDIS) (Azizi et al., 2022) . In this section, we compare our best performing method against state-of-the-art multiview chest X-ray classification algorithms. As illustrated in Fig. 3 , our best performing model REMEDIS outperforms the state-of-the-art methods by huge margins of 9.9%, 8.8% and 1.6% on the 1%, 10% and 100% data fractions. Perhaps even more strikingly, our best performing model does not leverage any of the specialized techniques employed by the other SOTA methods such as using auxiliary losses (Hashir et al., 2020) or combining multiple views by computing pixel-wise statistics of the two branches (Havaei et al., 2016) . 
conclusion: 
 Multi-view methods are more appealing compared to single-view methods since some pathologies are more readily identifiable from a particular view. However, owing to the challenges associated with collecting large multi-view labeled datasets, it is imperative to focus on the data-efficient setting when developing multi-view models. Pre-training is critical to yield good performance in such low data regimes. Hence, we present the first thorough benchmarking study illustrating the efficacy of 10 strong supervised and self-supervised models pre-trained on both natural and medical images for multi-view chest X-ray classification.Our best models yield significant improvements compared to existing state-of-the-art multiview approaches, outperforming them by as much as 9.9%, 8.8% and 1.6% on the 1%, 10% and 100% data fractions respectively. Moreover, we observe that self-supervised learning on medical images, followed by supervised fine-tuning on single-view chest X-ray classification, provides the most optimal transfer learning strategy for multi-view chest X-ray analysis.We hope this benchmark will act as a catalyst for the development of stronger multi-view medical imaging models, analogous to the role of similar benchmarks in other domains of computer vision and medical imaging.  
 
 Appendix:  


Decision: All reviewers unanimously agreed to accept the work for presentation at the 2024 MIDL meeting. The authors are requested to revise the camera-ready version in accordance with the feedback provided in the rebuttal.

Reviews: review: paper_summary: paper_summary: The authors present a benchmark for multi-view chest X-ray classification, considering the impact of model pre-training and training data size. The authors hope to popularize the development of multi-view medical imaging models, and raise awareness to the importance of available training data. Their proposed models outperform existing approaches in literature.
main_review: main_review: The authors appropriately addressed all my questions and the questions from other reviewers. They also commit to make significant adjustments in the final, revised version of their manuscript, on that condition, I finalize my rating.
strength_weakness: strength_weakness: The results on the impact of pre-training are inconclusive: Apart from the supervised and the MoCov2 approach, it is difficult to tell if a method is in itself superior or if it's the impact of pre-training.
questions: questions: - What is the authors opinion on why Balow Twins and the BYOL method performed worse than random initialization?
- Can the authors comment on why the same methods could not be used for comparing pre-training on medical and natural data? Are the differences significant when comparing the results for the Supervised and the MoCov2 methods that are in both pre-training categories? Especially for the 100% data fraction. Why do the authors assume that the good performance of REMEDIS is contributed only to it being pre-trained on medical data, and not simply the performance of that method?
paper_summary: paper_summary: This paper presents a comprehensive benchmark study for supervised and self-supervised pre-training methods for multi-view chest X-ray classification. The study evaluates 10 different pre-training models on the PadChest dataset across various data fractions (1%, 10%, and 100%) to understand their performance in multi-view chest X-ray classification. The key findings of this paper lie in the importance of pre-training for data-efficient multi-view classification models. In fact, the study reveals that leveraging larger, unlabeled datasets for pre-training can lead to substantial improvements in model performance on medical imaging tasks. Among the evaluated models, certain architectures demonstrate superior capability to generalize from limited data, suggesting the potential for these models to be deployed in clinical settings where data scarcity is common.
main_review: main_review: The authors appropriately addressed my questions pretty well. I'm looking forward to see the additional clarifications and experiments in the manuscript, specially addressing dataset focus and generalisability.
strength_weakness: strength_weakness: While this comprehensive benchmark showcases the best training strategy for multi-view chest x-ray in low data regime, the main limitation of this paper is its focus on a single dataset. In fact, the paper doesn't discuss the potential for dataset-specific biases that may not generalize across different clinical settings or populations. Additionally, while it explores the effectiveness of various pre-training methods, particularly in low-data regimes, the scalability and applicability of these methods to larger, more heterogeneous datasets remain to be fully assessed. Further research is needed to explore these models' performance across diverse medical imaging tasks and settings.
questions: questions: The main question to address would be about generalizability, the authors need to provide concrete proof of the latter in their study and discuss all the possible biases induced in their work.
paper_summary: paper_summary: This paper explores the impact of using pre-trained backbone networks on multi-view chest X-ray attribute prediction models.
main_review: main_review: My initial misunderstandings are resolved. I would have also preferred p-values be calculated to assert significance. The paper is likely relevant and interesting to readers. It also involves enough related work and baselines to be a solid contribution.
strength_weakness: strength_weakness: If I understand if correctly, in Table 1, the variance shown here is "the mean AUC (on 64 pathologies)" meaning the sum of all individual AUC scores. I do not think this provides the correct variance to compare these models. Having models trained on many different subsets of data and comparing the mean of the mean AUC values of these models would be useful to compare. Using only the mean AUC of one training invocation does not allow us to calculate the expected performance and variance of training that model. So I believe we cannot conclude if one model outperforms another with the current experiments of this paper.
questions: questions: Mainly the issue about conclusions about model performance discussed in the previous section.

When training why wasn't early stopping used? Isn't there a concern about overfitting?

Comments: title: Thanks for the response
Response
Response to the authors
Discussions
comment: I'm not sure how I missed the details about the training splits. Your responses resolve my issues.
Thank you for the adjustments, looking forward to the revised version!
I would like to extend my appreciation for your thorough response addressing the concerns raised during the review process. Your commitment to enhancing the quality and impact of your work is evident throughout your detailed explanations.

**Addressing Dataset Focus and Generalizability**

Regarding the dataset focus and generalizability, your rationale for selecting the PadChest dataset is well-founded, emphasizing its comprehensiveness and potential representation of real-world clinical scenarios. I commend your decision to incorporate additional datasets such as MIMIC-CXR to enhance the generalizability of your findings, ensuring that potential biases introduced by dataset selection are acknowledged and addressed.

**Dataset Division Clarification**

Your clarification on dataset division procedures is crucial for ensuring transparency and reproducibility, particularly when dealing with small data fractions. The detailed explanation of the sampling methodology in your revised manuscript will undoubtedly strengthen the robustness of your findings, allowing readers to better understand the distribution of conditions across subsets.

**Hyperparameter Tuning**

Regarding hyperparameter tuning, while your standardized approach aims to ensure fairness and comparability across models, it's important to acknowledge that different models may have varying sensitivities to certain hyperparameters. Thus, while applying the same set of hyperparameters is a step towards consistency, it may not fully guarantee fairness of evaluation across models. Clarifying this aspect in your methods section will provide readers with a nuanced understanding of your experimental design and the factors influencing model performance. This clarification will enhance the interpretability of your results and strengthen the credibility of your findings.
We thank the reviewer for their constructive feedback and the opportunity to further refine our manuscript. To further clarify our response, we conduct more experimental results rather than only describing the importance of pre-training. 

**REMEDIS and Pre-training Impact: Investigate Thoroughly with More Experiments**

To experimentally show that the performance of REMEDIS is largely due to pre-training on medical data, we show the following results:

| Method    | AUC (1%)       | AUC (10%)      | AUC (100%)     |
|-----------|----------------|----------------|----------------|
| Scratch   | 60.7 ± 1.02    | 67.1 ± 0.43    | 81.2 ± 0.837   |
| SimCLR v2 | 65.3 ± 1.67    | 69.4 ± 1.60    | 81.3 ± 0.381   |
| SimCLR v2 + Medical-Pretraining (REMEDIS)  | 69.0 ± 3.67    | 73.7 ± 1.26    | 84.3 ± 2.39    |

As per the REMEDIS work [1], they use SimCLR as their base method and further pretrain on various medical datasets. In other words, REMEDIS is essentially SimCLR v2 that is pretrained on medical data. Thus, the table above clearly shows that as we pretrain on medical data, the performance improves significantly. We commit to adding the aforementioned experimental results along with explanations to improve our paper. 

Furthermore, we are running additional experiments that more explicitly delineate the benefits of pre-training on medical data. This will allow us to better isolate the effects of pre-training from other variables, providing a clearer demonstration of its significance. In particular, to investigate the importance of pretraining, we experiment with MoCo-v2 [2] and MAE [3] that are first pre-trained on medical images and then finetuned on NIH Chest X-ray data. We refer to these experiments as MoCov2-FT and MAE-FT in our results. Comparing the results of MAE with MAE-FT and MoCov2 with MoCov2-FT from Table 1, our initial results indicate that additional fine-tuning on medical data is greatly beneficial and significantly improves the multi-view classification performance. Note: We commit to adding these additional methods and their experiment results to the revised manuscript. 

We are grateful for the reviewer's insights. We believe that the additional experiments and clarifications will significantly enhance our work.


**References**

[1] Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Korn- blith, Ting Chen, Patricia MacWilliams, S Sara Mahdavi, Ellery Wulczyn, et al. Robust and efficient medical imaging with self-supervision. arXiv preprint arXiv:2205.09723, 2022.

[2] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. arXiv preprint arXiv:2003.04297, 2020c

[3] He, K., Chen, X., Xie, S., Li, Y., Doll ́ar, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. pp. 16000–16009 (2022)
Thank you for thoroughly addressing my questions, and the questions of the other reviewers. Have the addressed points been corrected in a revised manuscript? It is difficult to tell how the implemented changes change the quality of the manuscript without the revised version.

Regarding the performance of REMEDIS, the authors say that they will "clarify" how the performance of REMEDIS is largely dependent on its pre-training. Will this clarification be in the main text or will it be complimented with baseline comparisons, to other architectures? Unfortunately, describing the importance of pre-training will not be sufficient to show the importance of pre-training.

The same argument can be said for the "inconclusiveness of pre-training impact", the authors promise to refine the discussion in the paper to better fit the novel application of the approach, but that does not improve the flaws of the experimental design when assessing the significance of model pre-training.
Dear Reviewers The authors have submitted their rebuttal addressing the raised questions. The paper remains open for further discussion and engagement.
We thank the reviewer for recognizing that we are tackling "an important concern" with a paper that is "well-designed and written," featuring experiments that are "elegant" and "to the point." Below, we address the detailed comments and questions raised:
 
**On the Inconclusiveness of Pre-training Impact:**
We acknowledge the reviewer's observation regarding the perceived inconclusiveness of the impact of pre-training. Our benchmarking study was designed to explore a wide array of pre-training methods, acknowledging that their efficacy might vary in nuanced ways. We emphasize that our primary contribution is the systematic comparison of these approaches in the multi-view chest X-ray classification domain, which has not been extensively studied before. To provide further clarity, we will refine our discussion in the paper, more explicitly articulating our findings and delineating which certain pre-training methods demonstrate superior performance.
 
**Code Availability:**
We recognize the critical importance of code availability for ensuring reproducibility. In the spirit of open science, we commit to publicly sharing our code.
 
**On the Performance of Barlow Twins and BYOL Methods:**
The underperformance of Barlow Twins and BYOL suggests their optimization strategies might not be fully compatible with the intricacies of multi-view chest X-ray data. Specifically, the Barlow Twins' focus on reducing feature redundancy could remove essential, nuanced details vital for accurate X-ray classification. Such details are often critical for diagnosing from medical images. We hypthesize that the Barlow Twins' aggressive decorrelation could thus detrimentally affect diagnostic performance by omitting subtle yet important features. Simlilarly, BYOL relies on an asymmetrical architecture where the online network predicts the target network's representation. This setup might not effectively capture the nuanced differences between multiple views of chest X-rays, where symmetrical learning from each view could be crucial. We will elaborate on these points and our hypothesis regarding their impact in the revised version of our paper.

**On Comparing Pre-training on Medical and Natural Data:**
The comparison between pre-training on medical versus natural data is indeed a critical aspect of our study. Our analysis did not overlook the comparative effectiveness of methods across these domains; rather, it highlighted the variability in performance enhancements attributed to the domain-specific characteristics of the pre-training data. For intance, MoCov2 which was evaluated across both domains, our findings suggest that the domain alignment of pre-training data (medical versus natural) plays a significant role in model performance, especially in the nuanced task of multi-view classification. This is observed in the 100% data fraction, where the depth and complexity of the data allow for more pronounced distinctions in method performance. We will clarify this point further, detailing the domain-specific performance insights to ensure a comprehensive understanding of these dynamics.
 
**Regarding the Assumption of REMEDIS's Performance:**
We appreciate the opportunity to clarify our position on why we attribute the strong performance of REMEDIS primarily to its pre-training on diverse medical datasets. REMEDIS, standing for "Robust and Efficient MEDical Imaging with Self-supervision," is designed as a comprehensive self-supervised learning framework specifically for medical imaging. Its core strength lies in leveraging a wide array of medical data for pre-training, thereby developing robust foundational models for medical image analysis.
This method's effectiveness in our context is not merely due to its algorithmic design but is significantly enhanced by its pre-training regimen. Pre-training on a broad spectrum of medical images enables REMEDIS to capture the nuanced variations and patterns inherent to medical data, offering a distinct advantage in tasks within the medical domain. In light of the reviewer's feedback we'll clarify how REMEDIS's extensive pre-training on diverse medical datasets is key to its superior performance, beyond just its methodological design.
 
We thank the reviewer for the valuable insights and the chance to improve our paper.
We sincerely appreciate your recognition of our study's "rigorous benchmarking" with a “particular focus on low-data regime which poses a real challenge in the clinical applicability of deep learning methods”
 
**Addressing Dataset Focus and Generalizability**
 
We understand the concerns raised regarding our study's primary focus on the PadChest dataset and potential dataset-specific biases. Our selection of PadChest in our benchmarking effort is grounded in its significant strengths: it is one of the most comprehensive datasets available, with extensive annotations and a wide variety of multi-view chest X-ray images. These characteristics make PadChest not only a robust base for our study but also potentially representative of the broader clinical landscape. Its diversity and scale offer a unique opportunity to assess the efficacy of pre-training models in a manner that, we believe, can mirror real-world clinical scenarios to a considerable extent. However, we acknowledge the critical importance of generalizability across different datasets and clinical settings. To address these concerns, we are planning to incorporate additional dataset such as MIMIC-CXR to enhance the robustness of our conclusions. 

**Dataset Division Clarification**
 
In creating the 1% and 10% data fractions, we aimed to maintain a distribution of conditions that mirrors the full dataset as closely as possible. This process involved random sampling of images, with an effort to roughly preserve the pathology distribution across each subset. We recognize that this aspect of our methodology was not sufficiently detailed in our initial manuscript and will rectify this in our revised version, ensuring clarity and transparency.
 
**Hyperparameter Tuning**
 
Our approach to hyperparameter tuning was designed to ensure fairness and comparability across all models evaluated. We applied a standardized set of hyperparameters across models to maintain direct comparability of results. This uniform approach, crucial for our study’s integrity, will be elaborated upon in the methods section of our revised manuscript to provide clear insights into our experimental design.
 
We are dedicated to enhancing our paper based on your valuable feedback, aiming to strengthen the implications and contributions of our work.
 
Thank you for your thoughtful review.
We sincerely appreciate your recognition of our work as “an interesting research question” and the potential benefit of “guiding the community to use a specific backbone model”.
 
Addressing your concerns:
 
 
**Clarifying Experimental Setup:**
 
It seems there has been a misunderstanding regarding our experimental setup described in Table 1. To clarify, as per our Section 3.1: “We conduct each experiment three times, with each run representing a different randomly selected training split to obtain more accurate results.” This approach is precisely what you recommended – “Having models trained on many different subsets of data and comparing the mean of the mean AUC values of these models.”
 
We apologize for any confusion caused by our initial presentation and agree that this critical detail was not emphasized sufficiently in our paper. In response, we commit to clearly detailing this aspect in the methods section of our revised manuscript, ensuring our experimental design's transparency and rigor.
 
**Early Stopping/Overfitting**
 
Our methodology, although not explicitly termed as "early stopping," effectively embodies its principle. We monitored the model's performance on the validation set across epochs and selected the epoch that yielded the best validation results for final evaluation. This strategy aimed to prevent overfitting by identifying the most generalizable point in the training process, akin to early stopping which halts training when validation performance no longer improves.
We realize our initial manuscript might not have clearly communicated this approach's preventive measure against overfitting. In the revised paper, we will provide a more detailed explanation of this selection process.
 
We appreciate your feedback and look forward to enhancing our paper with these clarifications.