Venue: MIDL 2023 Conference

Reviews:

final_score: rating: 4: Weak accept
review_content: summary: This paper concerns the question: For multi-modal learning, self-supervised learning and joint learning using both learning strategies, which one improves the visual representation for downstream chest radiographs classification tasks the most? And the conclusion is multi-modal learning and joint training performs better than self-supervised learning with limited supervised data, but the performance difference between all methods is largely decreased when data size scales up. 
strengths: This paper leverages CLIP, ConVIRT, MocoV2, SimCLR and VICReg method in the experiment. CLIP and ConVIRT are contrastive multi-modal learning methods, and the others are self-supervised learning methods. I am happy to see so many methods participating in the comparison. Finding the strengths of self-supervised learning method and contrastive multi-modal learning method may provide some inspiration for follow-up work.
weaknesses: I have not seen your own pipeline or framework proposed in the paper. For example, Figure 1 is just a simple representation of the experimental workflow for other methods. In the Method section, only the loss functions of other methods are introduced. 
Besides, the experimental results of the two methods on different proportions of data sets seem to be able to give further explanations, and some examples can be listed to show the classification results. From the point of view of the whole paper, this is a comparative experiment, which innovation in the model structure is not so sufficient but draws an interesting conclusion. Considering the massive image data of the existing CXR dataset, the classification results obtained by using these two methods are very different, so the reference significance on large datasets is open to question.
recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: The authors compare the performances of multi-modal learning, self-supervised learning and joint learning with both multi-modality and self-supervision. These models are initially trained with the MIMIC-CXR data. The models are then fine-tuned only at the final layer or all the layers with different amounts of labelled data from the CheXPert and NIH Chest XRay-14 data sets and their accuracies are compared. On fine-tuning with 1% labelled data, the multi-modal and joint learning models have a better performance than models trained with only self-supervision. Further, the feature representations of the different models are visualised as a t-SNE plot. 
strengths: 1. The authors have performed an exhaustive comparation of the different self-supervised learning approaches, multi-modal learning approaches and their combinations.  
2. When trained with only 1% labelled data of CheXPert data which is an external data set, the multimodal and joint learning approaches perform better than the only self-supervised learning approach.  
weaknesses: 1. With 1% or 10% labelled data, there are performance improvements of multi-modal learning over self-supervised learning. In case of NIH ChestXRay data, the improvements are only marginal. However, this is not a surprising result since multi-modal learning benefits from additional text data which is not available to the self-supervised learning approach.  In my opinion, it is not a fair comparison of the models since these models are meant for different learning scenarios.  
2.It is hard to make any conclusions from the t-SNE plots shown in Figure 2. One would expect the t-SNE plots to show more class-separations for each data set. Although there are only complete clusters for a data set.   
3. The authors claim that multi-modality improves performance. However, the evaluation results in Table 1. show no significant performance improvements of the self-supervised or multi-modal learning models over ImageNet initialized models when fine-tuned with 100% data. With 100% data, it is likely that the model is fit to the new data and the learning regimes used previously doesn’t matter anymore.  
recommendation: Poster
review_title: null

final_score: rating: 2: Weak reject
review_content: summary: The key idea of this research paper is to use multi-modal data, or data from multiple sources, to build a framework that can be used to solve tasks in the biomedical domain. The researchers conducted experiments comparing self-supervised learning and joint learning with multi-modal and self-supervised models on limited labelled datasets with 1% and 10% labelled data. They found that joint learning outperformed self-supervised learning alone, while being on par with multi-modal learning. Additionally, they found that multi modal was more robust across different environments than other methods tested.
strengths: The strengths of this paper are that it provides a novel approach to solving tasks by leveraging multi-modal data. The experiments conducted were thorough and well designed, providing evidence for their conclusions. Additionally, the code from the research project is freely available online which allows other researchers to build upon their work or replicate it if needed. CIs have been provided for the evaluation. 
weaknesses: - the authors basically compare a setup with partly enriched labels to self-supervised learning from images only. 
- what is meant by visual representation? t-SNE?
- the used datasets are multi-label, multi-class? How was this taken into account when computing the AUROC? What is meant by "mean AUROC from all pathology"?
- how does OOD play a role here? There are no known or unknowns and everything is fine-tuned on comprehensive label data, neither prediction confidence is evaluated.
detailed_comments: I found no typos but the paper is difficult to follow and it is unclear what the main motivation of this paper actually is. 
There is a lot of related work regarding multi-modal learning missing, e.g., automated report generation and recently image from text generation (google scholar will reveal a lot of references regarding multi-modal learning with CXR images; I don't provide any here to maintain anonymity) 

recommendation: []
review_title: null

Meta-review: metareview: This paper explores designing an efficient framework that adapts to multiple modalities for biomedical data and found that joint learning with multi-modal and self-supervised models outperforms self-supervised learning and is at par with multi-modal learning, and multi-modal learning is generally more robust on out-of-distribution datasets.
Two reviewers accept this paper while one rejects it. 
The authors have robustly responded to the comments and the responses solve most of the issues raised by reviewers. 
recommend: accept
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)