Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this work, the authors propose a new model, named ConVIRT to learn the medical visual representation from paired image and textual data in an unsupervised strategy. In ConVIRT, they mainly use a contrastive loss with two modalities (images and texts) as inputs to learn the representation. The experimental results show their proposed model achieve higher performance than other methods in image classification and zero-shot retrieval tasks.

Strength.
In this work, the authors propose a simple and straightforward method to learn the image visual representation using modified contrastive loss with two modalities as inputs. Besides, the experimental comparisons show their model outperforms other baselines and methods.

Weakness.
1. Novelty. In this work, the main contribution is the proposed modified contrastive loss with two modalities as inputs. It is an interesting and challenging problem that how to extract the visual representation in an unsupervised strategy. However, the proposed modified contrastive loss is below the standard of top-tier conference ICLR. 
2. Experiment. This work evaluates their model in many tasks and datasets. But, it might miss some baselines or other state-of-the-art methods. (1) Baselines. I suggest the authors could add the baseline using triplet loss. Both contrastive loss and triplet loss could enlarge the distances between different classes. (2) Baselines. I also suggest the authors could add the baseline only using the images as inputs, just like Chen et al. 2020a. and maybe only using the text data as inputs. 

3. Implementation details. (1) BERT encoder. In section 2.3, they use a BERT encoder to extract the textural features. I suggest the authors could give more details about it, such as how to initialize the parameters? As we known, there exists a huge gap between contexts form website and that from the medical report, since there are so many specific nouns. (2) Text transformation function. I might not agree with the text transformation function used in this work as the following reasons. There might exist a big gap and different meanings between two sentences even in one document. As the goal is to maximize the agreement between the true image-text representation pairs, there might occur a conflict between one image and two different meaning sentences sampled from the same documents.

4. Visualization. (1) I suggest the authors could give more visualization examples about the learned visual and textual representation, such as t-SNE. Such visualization might also address my above the concerns about the potential conflict during training. In these t-SNE examples, according to the proposed pipeline, the distance between the learned visual representation and the learned textual representation of the corresponding sentences in the same document should be smaller than the distance between the visual representation and some similar sentences from other documents. 
5. New type of baselines. This work aims to learn the useful visual representation from both images and texts. However, I suggest that they should report another kind of baselines, only using images to learn the visual representation in an unsurprised way, such as Chen et al. 2020a and [1] (Unsupervised Representation Learning by Predicting Image Rotations, ICLR 2018). 
The results of the proposed method should outperform this kind of baseline. 

review_title: Contrastive Learning of Medical Visual Representations from Paired Images and Text

final_score: rating: 6: Marginally above acceptance threshold
review_content: #####################   Summary   ####################

This paper presents the Contrastive VIsual Representation Learning from Text (ConVIRT) pretraining strategy to learn fine-grained medical visual representations of medical images by pretraining on large-scale image-report pairs. As a result, ConVIRT improves medical visual representations by maximizing the agreement between true image-text pairs versus random pairs via a bidirectional contrastive objective between the image and text modalities. The authors demonstrate that ConVIRT can help improve performance on down-streaming tasks like image classification and image retrieval tasks.


#####################   Strengths   #################### 

(1) This paper is really clearly written. The paper is easy to follow and understand.

(2) The paper explores an interesting direction of learning fine-grained medical visual representations for medical image understanding.

(3) The proposed ConVIRT is well motivated. The experimental results are very solid.

#####################   Weakensses   #################### 

(1) The paper is limited in its novelty borrowing ideas from some previous works: (i) contrastive learning [1][2] and (ii) image-text representations pretraining [3]. However, it is an interesting idea of applying contrastive learning to medical image and text. So, I think this paper is novel to some extent, but not that novel, since it mainly makes some incremental contribution by combining ideas from existing work.

(2) The analysis is not convinced (see below).

#####################   Questions   #################### 

I have some questions for the authors:

(1) What the medical visual representations have learned?

(2) Compared with the baseline methods (section 3.3), why the proposed ConVIRT can achieve the best results? How these results can be achieved? 

(3) Why the proposed ConVIRT can learn better medical visual representations than the SimCLR [1] and MoCo v2 [2] under the setting of image-only contrastive learning (section 5)? How they differ from representations learned by existing methods, e.g., SimCLR [1] and MoCo v2 [2]?

(4) Can the medical visual representations pretrained on the chest image be further finetuned on the bony image?

Overall, although the paper doesn't provide insights around what the representations have learned and how they differ from representations learned/used by existing methods, they have provided substantial evidence to suggest that pretraining helps in a lot of downstream tasks. In other words, the proposed ConVIRT seems to be useful for the researchers in the medical AI field.


[1] A simple framework for contrastive learning of visual representations. In ICML 2020.

[2] Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297.

[3] ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS 2019.

####################  After Rebuttal  #################### 

I thank the authors for responding to the comments and have read them carefully. The authors have addressed my concerns in the rebuttal.
review_title: An interesting work, utilizing the contrastive learning to learn medical visual representations

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

1. This paper tackles the medical image understanding problem. The aim of this paper is to learn a generic feature representation for medical image that could benefits downstream tasks like medical image classification, zero-shot classification. The main contribution of this paper is proposing a contrastive loss that the matched pair of image and text should have a higher corresponding score than the mis-matched pairs.

Pros:
1. The problem of learning a generic representation from limited training data is important.
2. Comparing with the baseline, the performance of the proposed approach looks good.
3. The paper is clearly written.

Weakness:
1. My first concern is the novelty. If I understand correctly, the main novelty of this paper is proposing the losses (Eq. 2 and Eq. 3) in the section of contrastive visual representation learning from text. However those losses are well-known in the domain of image-text retrieval. Current vision-language BERT (VLBERT) approaches use similar loss to optimize their model [Lu et. al., 2019]. The difference is that, in VLBERT papers, the model is trying to contrast the positive pair of image and text against all negative pair of image and texts that sampled from a batch: $$\ell = - \log \frac{\exp(<v_i, u_i> / \gamma)}{\sum_{k=1, k\neq i}^N\sum_{j=1, j\neq i}^N \exp(<v_j, u_k>/\gamma)}.$$ While the proposed approach uses Eq. 2 and Eq. 3 to optimize text retrieval and image retrieval, respectively. However, the motivation of separating the loss into Eq. 2 and Eq. 3 in unclear.
2. In the related work, recent advance in vision-language BERT has been mentioned (Last paragraph of Sect. 6 Related work). The paper stated that (in point 2) 'existing work has focused on visual-linguistic tasks such as visual question answering.', which might not be accurate. As current vision-language BERT models are trying to learn a generic representation for both image and text, where downstream tasks are ways to evaluate the representation. The aim of vision-language BERT is aligned with the aim of this paper.
3. The paper mentioned multiple times that the proposed approach is specific for medical images (Last paragraph of Related work). However this is unclear to me why the proposed approach is specific for medical images?

I would be happy to increase my rating, if the author could differentiate the proposed approach (losses) with the one used in current visual-language BERT. It would be the best if the author could state clearly why the proposed approach would be beneficial for medical images (Otherwise, 'medical' in the title of this paper might need to be removed). The author might also need to re-state the novelty contribution of this paper.
review_title: Good performance but limited novelty.

Meta-review: The proposed ConVIRT learns representations of medical data from paired image and text data.
While the paper addresses a relevant problem, the reviewers agree that the method has limited novelty. Two reviewers find and that the experiments are not convincing. One reviewer notes that the paper does not compare to the state-of-the-art methods for the tasks.

Decision: Reject