Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on building better models for unsupervised pretraining of deep neural networks for chest xray analysis.  This is achieved combining multi-modal information, i.e. radiological reports and chest xray images, and defining an unsupervised training procedure that predicts simultaneously masked parts of the images/reports given the non-masked part (masked record modelling). 

 

The learned representations can be used to initialize networks for downstream tasks (classification, segmentation), and allow to outperform competing methods even when using a much lower amount of labelled data. 

 
strength_and_weaknesses:  
One of the biggest issues when developing algorithms for chest xray analysis is access to labelled data, since it is expensive for radiologists to label tens of thousands of images. Most often, labels are  then generated with rule-based NLP algorithms. These labels are however very noisy, both because of the complexity of the text data to analyse, but also because in many cases radiologists might not write in the reports about diseases that are present in the image but are not clinically relevant for the patient. 

The method discussed in this paper allows to consider an alternative approach: learn transferable radiography representations using image+report data, and fine-tune the networks for downstream tasks using a low amount of labels. 

The experimental section is extensive, and shows increased performances compared to related methods. Importantly for practical purposes, thanks to the pretrained model, good classifiers can be built only using 10% of the labels. 

All the ablation studies I had in mind while reading the MRM section were done by the authors, which show that all components of the proposed model are necessary.  
It would be interesting for me to see how the model performs with different amounts of masking probabilities. In particular I am surprised that the model performs well even when removing 75% of the patches. 

The structure of the experimental section is a bit confusing the way it is split now in "Baselines" vs "Results". For example, after reading about the tasks and baselines in section 4.3.1, it would make more sense for the reader to directly see their results (section 4.5.1). 


_Typos_: 
* Second last paragraph of introduction: base->based 
* First sentence of section 3: learns -> learn 
clarity,_quality,_novelty_and_reproducibility:  

I found the paper very interesting and overall well written. To the best of my knowledge, the presented methodology is novel. 

 

 


summary_of_the_review: Overall an interesting paper with practical applications. I believe it can have an impact in the medical imaging community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel approach for self-supervised learning from radiographs and associated medical records. The proposed multimodal method learns to predict masked inputs from both record (test) and radiograph (image) data.

strength_and_weaknesses: Strengths
- Paper is well written and easy to understand 
- A thorough review of literature is provided
- Extensive experiments validate the proposed approach on the MIMIC-CXR (pre-training) and CheXPert, RSNA Pneumonia, NIH-Chest X-ray and COVID-19 Image Data Collection datasets (fine-tuning).
- Compelling results are presented showing the computational improvements in AUC with the proposed method

Weaknesses
- Not clear why masking is necessary for images, or whether the task could be super-resolution only
- Ablation studies with non-masking, non-super resolution pre-text tasks on records are missing
- Experimental comparisons to other multimodal (language+image) methods, such as M3AE (Geng et al. 2022) are missing

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is very well written and easy to understand. One part that is missing is a discussion of whether multi-modal self-supervised learning has been tackled in other methods, potentially outside of medical imaging. 
Also, what is the labelling ratio defined as? 

The approach seems like a novel combination of existing image-based and text-based methods. Code is provided and most of the implementation is well described in the manuscript. It is not clear why the weight (lambda) of the report and the image loss is equal, have the authors experimented with this weight? 

summary_of_the_review: The paper seems like a straightforward combination of existing techniques, but generates compelling results on a variety of benchmarks. Novelty of combining MAE with BERT is already proposed by some papers, e.g. M3AE, and there is no clear and demonstrated improvement over these works, aside from the novel application to medical data. However, ablation experiments, discussion, and writing is clear and complete.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses a novel method to enhance the pre-train chest x-ray image embeddings using the associated report. They train two masked auto-encoders (one for reports and the other for images). They then combine the image patch embeddings with report embeddings to decode the masked report tokens. At the same, they also decode a high-resolution image using the same image patch embeddings. They compare their method with existing methods and show that they outperform other methods by a considerable margin on multiple datasets.
strength_and_weaknesses: Strength(s):
1. The method is novel and explained well. 
2. The method outperforms other methods by a considerable margin on multiple datasets thereby signifying the method efficiently utilizes the signals from reports to learn better image embeddings.
3. The ablation experiments of different components nicely demonstrate how each component contributes to the performance.

Weakness(es)/Suggestion(s):
1. It would be nice to have confidence intervals/ SD around the metrics used to compare different methods.
2. Why GAP was used to share image information with the masked record decoder? What happens if we use other alternatives like Max pooling, or have a transfer module that takes in embedded patches and returns a global embedding for the image?
3. How is the lambda parameter in equation 3 decided? 
4. Some ablation studies to investigate how well the model learns for different masking ratios and masking patterns would be interesting for the readers.
clarity,_quality,_novelty_and_reproducibility: The method is clearly explained and novel. They have shared the code to reproduce the results.
summary_of_the_review: In summary, I found the method introduced in the paper to be interesting, novel, and elegant. The results across multiple datasets and downstream tasks demonstrate that the method is better than other methods at learning insights from reports to improve image embeddings. There are some important ablation experiments missing from the paper which would only strengthen the paper. I, therefore, give a recommendation of marginally above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new approach to learning radiograph representation through a joint masked image and language modeling. Specifically, for each pair of radiograph and radiology report pair, both the image and the report are masked for restoration tasks. The masked image patches are encoded and used for both restorations, while the masked text embedding is only used for report restoration. Besides, the input image is downsampled which combines a super-resolution task. The model is pre-trained on the large MIMIC-CXR dataset and then fine-tuned for downstream tasks: classification and segmentation on CheXpert, NIH ChestX-ray, RSNA Pneumonia, COVID-19 Image Data Collection, and SIIM-ACR Pneumonia Segmentation datasets. Experiment results show the proposed method outperforms all comparison methods, including report-supervised, self-supervised, and transfer learning methods. Ablation study experiments also provide an analysis of contributions from individual components in the proposed method.
strength_and_weaknesses: Strength

- The paper is well-written and easy to follow. The proposed method is explained well with text descriptions and figure illustrations.
- The proposed method, Masked Record Modeling (MRM), combining masked image and language modeling, is effective for learning radiograph representations. MRM outperforms state-of-the-art both report-supervised and self-supervised / transfer learning methods by large margins in some cases.
- The ablation study shows the contributions of individual components in the proposed method. Specifically, it shows masked language modeling has a significant impact on improving radiograph representation learning.

Weakness

- Some technical details need more clarification. What is the lookup table for report token embedding? What are the specific configurations of the image encoder, image decoder, and report decoder?
- In Table 2, MRM outperforms REFERS in most categories, except for “Infiltration”. Why? It is worth conducting a more in-depth analysis.
- In Section 4.4, “The cross-entropy and MSE losses are used for masked image and language modeling, respectively.” The order of two losses should be swapped.
clarity,_quality,_novelty_and_reproducibility: This paper lacks certain clarity on its own, as mentioned above. Some technical details are missing. However, the authors provided their source code. If the missing details are completed, it should be possible to reproduce the results.

The idea of combining masked image and language modeling is not entirely novel but may be so for radiograph representation learning. There are previous efforts on joint masked image and language modeling [1,2]. However, they are not published yet.
[1] MLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modeling, https://arxiv.org/abs/2109.12178
[2] Masked Vision and Language Modeling for Multi-modal Representation Learning, https://arxiv.org/abs/2208.02131

The quality of this work is high as it made a large improvement to radiograph representation learning compared to previous methods.
summary_of_the_review: This paper combined masked image and language modeling for radiograph representation learning and showed significant improvement over existing methods for downstream tasks. The weaknesses of this paper could be addressed with some more effort.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers concurred that this paper is above the bar for acceptance. Reviewers found the work to be solid overall, with a reasonable method and strong experimental validation. Demonstrating the utility of a unified framework for masked image and language modeling for radiograph+report data is a useful contribution. Although some reviewers noted that the technical novelty is a bit limited, they still appreciated the strong evaluation demonstrating the effectiveness of the method. The author rebuttal further addressed reviewer concerns and added improved ablation studies. I agree with the reviewer assessments and recommend acceptance for this paper.

justification_for_why_not_higher_score: This is a solid paper but the technical novelty is still somewhat limited.
justification_for_why_not_lower_score: I do not recommend reject, given the overall solidness of the paper contributing an effective method for a useful problem, and strong experiments. The paper was also given positive reviews by all four reviewers.
Decision: Accept: poster