Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a novel Multi-Granularity Cross-modal Alignment (MGCA) framework to seamlessly leverage the multi-granular semantic correspondences between medical images and radiology reports for generalized medical visual representation learning. Extensive experimental results on different downstream datasets show that the proposed framework achieves substantial performance with limited annotated data.
strengths_and_weaknesses: Strengths:  
(1) A multi-granularity cross-modal alignment framework is proposed for learning generalized medical visual representations from free-text radiology reports.
(2) The overall structure is clear.
(3) The experimental results show the effectiveness of the proposed model.
 
Weaknesses:
(1) The authors incorporate several well-known technologies, including instance-wise alignment, fine-grained token-wise alignment, and disease-level alignment, into contrastive learning to enhance the generalizability of learned visual representations. In this case, which parts are new? Thus, the overall novelty needs to be strengthened.
(2) In Table 1, it can be seen that the proposed model performs better than other methods. However, because the proposed model has been pre-trained on a large-scale medical image and report dataset, whether other methods have been trained on the large dataset? 
(3) The generated visual and text token embeddings are projected into normalized lower-dimensional embeddings, and the dimension d is set to 128. How to set the dimension? The parameter study should be included. 


questions: (1) The authors incorporate several well-known technologies, including instance-wise alignment, fine-grained token-wise alignment, and disease-level alignment, into contrastive learning to enhance the generalizability of learned visual representations. In this case, which parts are new? Thus, the overall novelty needs to be strengthened.
(2) In Table 1, it can be seen that the proposed model performs better than other methods. However, because the proposed model has been pre-trained on a large-scale medical image and report dataset, whether other methods have been trained on the large dataset? 
(3) The generated visual and text token embeddings are projected into normalized lower-dimensional embeddings, and the dimension d is set to 128. How to set the dimension? The parameter study should be included. 
(4) From the results in Table 1, it can be seen that the proposed model obtains large improvements over DSVE and VSE++. Please the reasons for these situations. 
(5) It is better to provide more discussions on using multi-granularity cross-modal alignment. 
limitations: See weakness.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a visual representation learning approach based on contrastive learning of correspondences between visual radiographs and textual radiology reports at multiple levels of granularity using three different learning objectives. The first objective features standard cross-modal alignment between radiographs and their corresponding radiology report pairs, by which they enforce instance level agreement. The second objective features a cross-attention approach that enforces the alignment between the visual token embeddings (local representations obtained using the Vision Transformer) and aggregated text token representations (where the visual token is used as a query for attention-based aggregation of word token representations), and likewise, the alignment between text tokens and aggregated visual tokens. This enforces alignment between mutually informative local image regions in the radiograph and parts of the textual radiology report, and the authors argue that this region-level alignment improves performance of resulting visual representations on downstream task that involve dense predicitons. The final objective features alignment on a higher-than-instance level, by enforcing consistent cross-modal clustering of representations of corresponding visual and text data. This implicitly encourages the radiographs and reports that share high-level semanics to have similar representations, regardless of instance-level pairing and modality. Authors also demonstrate that pretraining on in-domain images (medical) is important for performance on downstream tasks, and that general-domain pretraining is not as effective.
strengths_and_weaknesses: Strengths:
- The paper is clearly written and well structured.
- Implementation details and reproducibility info are clearly provided, with appendix containing additional ablation studies justifying parameter and design choices.
- Extensive experiments (both qualitative and quantitative) as well as ablation studies. The choice of downstream tasks appears to be justified given the stated focus on learning visual representations, and they seem diverse enough to put different individual aspects of their approach to the test. The comparison with the main competing approach, GLoRIA, seems fair, as the authors also evaluated a variant of their model that uses the same image encoder as GLoRIA as well as the same pre-processing step.
- The idea and implementation of the disease-level cross-modal alignment module in this setting, to the best of my knowledge, can be seen as novel.

Weaknesses:
- Lack of an ablation study involving individual learning tasks in a object detection or segmentation setting (as was done under the classification setting). Their approach does seem to more convincingly outperform GLoRIA in a object detection and segmentation setting (compared to the classification featuring the ResNet-50 encoder), but it would have been interesting to see an ablation study that explicitly shows how relevant, for instance, CTA is in a dense prediction setting.
- Lack of error bars may be a potential issue, since the differences in performance between their approach and baselines (in some settings) are sometimes small.
- This particular setup for contrastive learning based approach for visual representation learning (also in the context of paired radiographs and radiology reports), as well as the idea and motivation behind token-wise cross-modal alignment objective, at its core, seem to be inspired by ConVIRT and/or GLoRIA, so the work can, to a degree, be seen as an extension of existing work and not entirely novel.
questions: - How is CTA implemented in cases when ResNet50 is used as the image encoder? In what way are visual tokens obtained? To what degree would you say the difference in the way the tokens are obtained can account for the difference in performance between the ResNet50 and ViT variant of your approach?
- Does the use of CTA give an even bigger performance improvement in a segmentation or object detection setting (compared to classification)?
limitations: The authors list the lack of consideration of retrieval tasks as one of the limitations of their work. However, the outlined scope of their experiments is in my opinion reasonable. With regards to potentially negative societal impacts, the authors mention the possible use of sensitive data in their framework. This is a general concern when medical data is used and is therefore not particularly specific to their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a Multi-Granularity Cross-Modal Alignment method, which learns data representations from medical scans paired with the corresponding text reports. The method exploits multiple unsupervised techniques to obtain the learned representations (e.g. contrastive losses, clustering with Sinkhorn-Knopp), and each of these techniques are utilized at the appropriate level of granularity. 
The learned representations are then evaluated on a large set of image-based downstream tasks, to assess the quality of the image representations. The experimental results support the merits of the proposed method. However, there are some weaknesses and limitations, which will be listed below.
strengths_and_weaknesses: Strengths:
- The proposed method is novel, if viewed as a whole framework that employs the granularity of the features that are present in medical scans. However, if one views each level in the method independently, these levels are not novel per se. But since the proposed method aims to encapsulate these levels together, then I deem the method as novel. The only level that I would deem more novel is the CPA, which proposes to learn cross-modal disease prototypes.
- The evaluation experiments are extensive. The paper reports results on multiple dataset benchmarks, which include multiple types of tasks (classification, detection and segmentation). The work also evaluates two types of image encoders (resnet & ViT).
- The paper is clearly written, and the used language is well understandable. 

Weaknesses:
Mostly, I believe the weaknesses in this work are in some missing pieces:
1- Missing relevant references from the literature. The following are some works that perform contrastive learning on medical images:
* Taleb, Aiham, et al. "ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
* Chaitanya, Krishna, et al. "Contrastive learning of global and local features for medical image segmentation with limited annotations." Advances in Neural Information Processing Systems 33 (2020): 12546-12558.
* Han, Yan, et al. "Pneumonia detection on chest x-ray using radiomic features and contrastive learning." 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.
* Feng, Ruibin, et al. "Parts2Whole: Self-supervised contrastive learning via reconstruction." Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning. Springer, Cham, 2020. 85-95.
* Xu, Jiarui, et al. "GroupViT: Semantic Segmentation Emerges from Text Supervision." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
There are quite a lot of works that are similar along this line, would be great to cite the most relevant once from the medical imaging literature. 
Also I believe some references related to the methodological part (mainly cross-attention) are missing, such as:
Chen, Yen-Chun, et al. "Uniter: Learning universal image-text representations." (2019).
Lu, Jiasen, et al. "Hierarchical question-image co-attention for visual question answering." Advances in neural information processing systems 29 (2016).


questions: It would be interesting to have a discussion in the paper on how the ITA level behaves with the existence of CPA. These two losses have a somehow contradictory behavior. The contrastive loss in ITA pushes the different instances (negatives) apart from positive instances. However, CPA clusters instances in the feature space in order to learn disease (prototype) clusters. CPA disregards the identity of the instance and rather tries to capture high-level unifying topic models. 
How do they behave then? It seems to me in Table 4 that the addition of the CTA loss is essential for improved performance (rows 2 and 4). But adding CPA and ITA alone (in row 3), seems to perform a little worse than row 2.
Please discuss how do you imagine the losses are behaving.
limitations: The authors list the limitations adequately
review_title: null

Meta-review: metareview: A multi-granularity cross-modal alignment framework is proposed, which learns data representations from medical scans paired with the corresponding text reports. 
The reviewers find the appraoch novel and the paper well-written with an overall clear structure.
Extensive experimental results show the effectiveness of the proposed model and experimental details are provided.

After the discussion with the authors, all reviewers vote towards acceptance of the paper.
confidence: Certain
Decision: Accept