Contrastive Learning of Medical Visual Representations from Paired Images and Text
Authors: Yuhao Zhang
Hang Jiang
Yasuhide Miura
Christopher D Manning
Curtis Langlotz
Venue:  (2021)
Keywords: visual representation learning
contrastive learning
medical image understanding
natural language processing

Abstract: Learning visual representations of medical images is core to medical image understanding but its progress has been held back by the small size of hand-labeled datasets. Existing work commonly relies on transferring weights from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. We propose an alternative unsupervised strategy to learn medical visual representations directly from the naturally occurring pairing of images and textual data. Our method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input. We test our method by transferring our pretrained weights to 4 medical image classification tasks and 2 zero-shot retrieval tasks, and show that our method leads to image representations that considerably outperform strong baselines in most settings. Notably, in all 4 classification tasks, our method requires only 10% as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance, demonstrating superior data efficiency.

Under review as a conference paper at ICLR 2021
CONTRASTIVE LEARNING OF MEDICAL VISUAL REPRESENTATIONS FROM PAIRED IMAGES AND TEXT
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning visual representations of medical images is core to medical image un- derstanding but its progress has been held back by the small size of hand-labeled datasets, Existing work commonly relies transferring weights from ImageNet pretraining, which suboptimal due to drastically different image characteristics, rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. We propose an alternative unsupervised strategy to learn medical visual representations directly from the naturally occurring pairing of images and textual data. Our method of pretraining medical image encoders with the paired text data via bidirectional contrastive ob- jective between the two modalities is domain-agnostic, and requires no additional expert input. We test our method by transferring our pretrained weights to med- ical image classification tasks and zero-shot retrieval tasks, and show that our method leads image representations that considerably outperform strong base- lines in most settings. Notably, in all 4 classification tasks, our method requires only 10% as much labeled training data as an ImageNet initialized counterpart achieve better or comparable performance, demonstrating superior data efficiency.
INTRODUCTION
Medical image understanding has the potential to trans- form healthcare and has seen rapid progress with the use of deep neural architectures (Gulshan et al. [016)] Es- teva et al. 2017 De Fauw et al. 2018] Rajpurkar et 2018b). Yet, with expert-level performance achieved only in some specialties and under some circumstances, med- ical image understanding remains difficuult task for the majority of specialties, mainly due to its challenging na- ture and the extreme scarcity of annotated data.
Severe cardiomegaly noted the image with enlarged...
Radiograph shows pleural effusion in the right lobe...
Existing work has followed two general approaches to ob- tain annotations for medical imaging tasks. The first ap- proach has been using high-quality annotations created by medical experts (Abràmoff et al. [2016] Gulshan et al. 2016) Shih et al. 2019 Wang & Wong 2020). However, the high cost of this approach has resulted in datasets that are mostly orders of magnitude smaller than natural im- age datasets such as ImageNet (Russakovsky al.) (2155). To remedy this, existing work has relied heavily on trans- ferring model weights from ImageNet pretraining (Wang et al. 2017 Fistvva et al.) 2017 Irvin et al., 2019). This approach is suboptimal because, shown in Figure medical image understanding often requires representations of very fine- tine-grained visual features that are drastically different from those required for identifying objects in natural images. As result, Raghu et al. (2019) found that ImageNet pretraining often provides little to no benefit compared to simple random initialization.
Two example chest radio- graph images with different abnormality categories, along with sentences from their paired textual report and example views indicative of their characteristics.
Figure 1:
A second popular approach is expert-crafted rules to extract labels from the textual reports companying the medical images. This approach has led to datasets of larger scale, since the text data paired with medical images often produced naturally by medical experts in their routine work-

---

Under review as a conference paper at ICLR 2021
flow and abundant in typical hospital's IT systems. Nevertheless, this rule-based label extraction approach has two limitations: 1) the rules are often inaccurate and limited to a few major categories (Wang et al. 2017), leading to very inefficient use of the textual report data; 2) these rules are of- ten domain-specific and sensitive to the style of the text, making cross-domain and cross-institution generalization difficult (Irvin et al. 2019).
In efforts to make more efficient use of unlabeled image data, several recent studies have shown promising results from contrastive representation learning from natural images (Chen et al. [2200a] He et al. 2020 Grill et al. 2020). However, as we will show, applying these image view-based contrastive methods to medical images provides only marginal benefits compared ImageNet pre- training. a result mostly due to the high inter-class similarity of the medical images as in Figure !
In this work, we aim to improve visual representations of medical images by combining the benefits of both learning from abundant textual data and unsupervised statistical approaches. We present Contrastive VIsual Representation Learning from Text (Con (oRV) framework for learning visual representations by exploiting the naturally occurring pairing of images and textual data. CIRT improves visual representations by maximizing the agreement between true image-text pairs versus random pairs via bidirectional contrastive objective between the image and text modalities. We apply Con VIRT to the pretraining of medical image encoders, and show that it leads to higher- quality in-domain image representations that capture the subticty of visual features required for medical image understanding tasks.
Compared to existing methods, Con has the advantages of utilizing the paired text data in way agnostic to the medical specialty and requiring no additional expert input. This allows us to evaluate VIRT by transferring our pretrained weights to 4 different medical image classification tasks covering 2 different specialties. We find that the resulting models outperform all baseline initialization approaches, including the standard ImageNet pretraining and several strong baselines that also utilize the paired text data. Most notably, in all 4 tasks, ConVIRI VIRT requires only 10% as much labeled training data an ImageNet initialized counterpart performance. We further evaluate Con VIRT on two new zero-shot retrieval tasks, an image-imane and text-image retrieval task, and also find it superior to all baselines. To facilitate future research, we will make our code and the collected retrieval datasets available.
achieve better or comparable
2 METHOD
2.1 TASK DEFINITION
We start by giving formal description of our representation learning setting. We assume paired input (X,, x.) where X,, represents one or a group of images, and X., represents a text sequence which describes the imaging information in X.. Our goal is to learn a parameterized image encoder function f., which maps an image to a fixed-dimensional vector. We are then interested in transfer- ring the learned image encoder function fu into downstream tasks, such as classification or image retrieval. In this work, we model the encoder function J, as a convolutional neural network (CNN).
We note that paired image-text data (x,, x,) naturally exists for many medical domains. Medical experts such as radiologists produce textual descriptions of images as part of their routine workflow, some of which are also made publicly available DDemner-Fushman et al. 2016 Johnson et al. 2019
2.2 CONTRASTIVE VISUAL REPRESENTATION LEARNING FROM TEXT
An overview of our method, CORVIRT, for learning fe is shown in Figure 2 At a high level, our method converts each input image x, and text x., into d-dimensional vector representations v and respectively, following a similar processing pipeline. For each input image x,, our method starts by drawing a random view x, from x, with a sampled transformation function f, ~ T, where T represents a family of stochastic image transformation functions described later. Next, the encoder function fo transforms x, into a fixed-dimensional vector h,, followed by non-linear projection function go which further transforms h, into vector V:
V go (fx.)))
(1)
where v E Rdd Similarly, for each text input x,,, we obtain a span x,, from it following a sampling function l,p and then a text representation u with: gu(Su (X.)), where fu is text encoder,
2

---

Under review as a conference paper at ICLR 2021
to
ge
X
X
Image Encoder
fU
h
Hext enbrged...
tu
Iu
X i1
Clear consoliiation
Encoder
Juu
h
u
tg( vo)
No abnormalit seen
Figure 2: Overview of our ConVIRT framework. The blue and green shades represent the image and text encoding pipelines, respectively. Our method relies on maximizing the agreement between the true image-text representation pairs with bidirectional losses ((=-a)) and ((u->)))
Iu a projection, and u E Rdd. The projection functions Iu and In project representations for both modalities from their encoder space the same d-dimensional space for contrastive learning.
At training time, we sample a minibatch N imput pairs (x,, x.) from training data, and calculate their representation pairs (v, u). We (Ve, u,) denote the i-th pair. The training objective of Con VIRT involves two loss functions. The first loss function is an image-to-text contrastive loss for the i-th pair:
(( log expl(v,, u:)/r)
EK=1 exp( (V:, uz)/T)"
where (vi, u;) represents the cosine similarity, i.e., (v, u) = T u/ ||]]|||||:: and E R+ represents a temperature parameter. This loss takes the same form the InfoNCE loss (Oord et al. 2018), and minimizing it leads to encoders that maximally preserve the mutual information between the true pairs under the representation functions. Intuitively, it is the log loss of an N-way classifier that tries to predict (v,, u;) as the pair. Note that unlike previous work which loss between inputs the same modality (Chen et al.] 2020a He et al.) (020), our image-to-text contrastive loss is asymmetric for each input modality. We therefore define similar text-to-image contrastive loss as:
contrastive
exp((u; vz)/T) log
((--))
k= exp((u,, vu)/T)
Our final training loss is then computed weighted combination of the losses averaged all positive image-text pairs in each minibatch:
(Xe +1- -()e)e(--)))
L
where À E [0, 1] is a scalar weight.
2.3 REALIZATION
We note that ConVIRT framework defined above is agnostic to the specific choice of image and text encoders, transformations and projection functions. this work, following previous work (Chen et al. 2020a), we model g. and gu as separate learnable single-hidden-laye neural networks, i.e. (-)
W (-) where is a ReLU non-linearity, and similarly for gu-
For the image encoder fo, we use the ResNet50 architecture (He et al., 2016) for all experiments, as it is the architecture of choice for much medical imaging work and is shown to achieve competitive performance. For the text encoder fu, we use BERT encoder (Devlin et al. 2019) followed by max-pooling layer over all output vectors. We initialize our BERT encoder with the ClinicalBERT model (Alsentzer et al. 2019) pretrained on the MIMIC clinical notes, which achieved state-of of-te-of-the- suite of clinical NLP tasks. At training time allow the encoder to adapt
art performance to our contrastive task by freezing the embeddings and the first 6 layers of this BERT encoder and fine-tuning the last 6 layers.

---

Under review as a conference paper at ICLR 2021
For the image transformation family T where t, is sampled from, sequential applications of five random transformations: cropping, horizontal fliping, affine transformation, color jittering and Gaussian blur. Different from recent work on contrastive visual representation learning (Chen et al., 2020a|b), b), we only apply brightness and contrast adjustments in color jittering, due to the monochrome nature of the medical images. For the text transformation function t.. apply simple uniform sampling of a sentence from the input document x (i.e., X, is a randomly sampled sentence from X,, for each minibatch). We did not use more aggressive transformation mainly because sampling at the sentence level can preserve the semantic meaning of the sampled spans.
EXPERIMENTS
3.1 DATA FOR PRETRAINING
We test our CORVIRT framework by pretraining two separate image encoders covering different medical specialties using two separate paired image-text datasets:
Chest image encoder: We use version 2 of the public MIMIC-CXR database (Johnson et al. 2019), which collection of chest radiograph images paired with their textual reports, and since its release has become standard resource for studying multi-modal modeling of medical images. After preprocessing, this dataset contains a total of about 217k image-text pairs, with each pair containing an average of 1.7 images and 6.0 sentences.
Bony image encoder: We obtain a collection of musculoskeletal image-text pairs from the Rhode Island Hospital system. Following chest images, musculoskeletal images constitute the second most common type of radiograph images in a typical hospital This dataset contains a total of 48k image-text pairs, with each pair containing an average of 2.5 images and 8.0 sentences.
We include model implementation and pretraining details in Appendix A
3.2 EVALUATION TASKS & DATA
We evaluate our pretrained image encoders on three downstream medical imaging tasks: image classification, zero-shot image-image retrieval and zero-shot text-image retrieval.
Image Classification. We evaluate our pretrained image representations on four representative medical image classification tasks: 1) RSNA Pneumonia Detection (Wang et al. 2017 Shih et al.) 2019), which involves binary classification of a chest radiograph image into either a pneumonia or normal category; 2) CheXpert image classification (Irvin al. 2019), which involves multi-label binary classification of a chest image for five individual labels, i.e., atelectasis, cardiomegaly, con- solidation, edema and pleural effusion; 3) COVIDx image classification (Wang & Wong 2020), which involves multi-class classification of a chest image into one of COVID19, non-COVID pneu- monia or normal categories; and 4) MURA bony abnormality detection (Rajpurkar al.) (2088)), which involves binary classification of a musculoskeletal image into abnormal or normal. We report test accuracy for COVIDx given its balanced test set, and report the standard area under the receiver operating characteristic curve (AUC) metric for other tasks following previous work.
Following previous work (Hénaff al. 2020 Chen al. 2020a He et al. 2020), for all tasks, we evaluate each pretrained image encoder under two individual settings: a linear classification setting, where the pretrained CNN weights are frozen and only a linear classification head is trained for the task; and fine-tuning setting, where both the CNN weights and the linear head are fine-tuned. The two settings complement each other for evaluation purposes: while the linear setting directly evaluates the quality of the extracted image features with the pretrained CNN, the fine-tuning setting more closely resembles how the pretrained CNN weights are used in practical applications.
To further compare the data efficiency of different pretraining methods, for each setting we evaluate the image encoders with 1 %. 10% and all training data, respectively (except for the COVIDx dataset where omit the 1% setting due to the scarcity of data for some categories). To control the variance in results, for all settings and models, we report average results aggregated over independent training runs. We include further dataset processing and model training details in Appendix B
4

---

Under review as a conference paper at ICLR 2021
Zero-shot Image-image Retrieval. This evaluation is similar to the conventional content-based image retrieval setting in which we search for images of a particular category using a representative query image. For evaluation, group of query images and larger collection of candidate images, each with a categorical label, are given to a pretrained CNN encoder. We encode each query and candidate image with this encoder, and then for each query, rank all candidates by their cosine sim- ilarities to the query in descending order. Since a widely-used annotated benchmark for this setting not available, we create our own dataset by re-using existing annotations in the CheXpert dataset IIrvn et al. 2019) and additional expert annotations from a board-certified radiologist. The resulting dataset covers 8 different chest abnormality categories, each with 10 expert-annotatee query and 200 candidate images. We include the detailed collection and annotation procedure in Appendix C and refer to this dataset as CheXpert 8x 200 Retrieval Dataset. We focus our evaluation on retrieval precision, and evaluate our models with Precision @ k: metrics where k = 5, 10, 100.
Zero-shot Text-image Retrieval. This setting is similar to the image-image retrieval setting, but instead of using query images, we retrieve images of a particular category with textual queries. For this purpose, ask radiologist to write 5 diverse and representative textual descriptions for each of the abnormality categories for the same CheXpert 8x200 candidate images (see Appendix[Dioor encode its text with the learned text encoder fu and then
details). At test time, for each query retrieve from candidate images in a similar way. This evaluation not only evaluates the quality of the learned image representations, but also the alignment between the text representations and the image representations. We again use Precision @ k metrics where k = 5, 10, 100.
3.3 BASELINE METHOOS
We compare VIRT against the following standard or competitive initialization methods:
Random Init.: For all tasks we initialize the ResNet50 with its default random initialization.
ImageNet Init.: We use CNN weights pretrained on ImageNet (Russakovsky et al. 2015), which
remains a dominant initialization approach for medical imaging work (Raghu et al. 2019).
Caption-LSTM: We initialize the CNN weights with ImageNet, and then pretrain it with im- age captioning task using the standard CNN-LSTM with attention architecture (Xu et al. 2015). For the captioning task, we train the model to decode the paired textual report from the coded image representations. Compared to the random or ImageNet initializations, this is an "in-domain" initialization baseline which uses the paired text data for representation learning.
this initialization we replace the CNN-LSTM model in Caption-LSTM Cappion-Transfore based captioning model in Cornia et al. 2020). which recently achieves with
state-of th-the- results on the COCO image captioning benchmark (Lin et al. 2014).
Contrastive- Binary:: This bascline differs from method by contrasting the paired image and text representations with a binary classification head, as is widely done in visual-linguistic pre- training work TTn & Bansal 2019 (Su et al.) (020).. For each input pair, we first project encoder outputs h and h2 into the same dimension with linear layers, concatenate them, and use a MLP network predict binary probability of whether the input is real or "fake" pair, which train with a binary cross-entropy loss. During training, for each (X,, x.) pair in the training set, construct a "fake" pair by replacing x,, with randomly sampled one from the dataset. We expect that the binary classification task requires the encoder to learn reasonable representations of the input images, and therefore is a stronger in-domain initialization baseline.
For fair comparison, for all baselines that require paired image-text data, we use the same datasets as in our contrastive pretraining. For the captioning- based methods, we always use the model check- points that achieve the best CIDEr score (Vedantam et al. 2015) held-out validation set.
4 RESULTS
4.1 CLASSIFICATION TASKS
Linear Classification. We present all linear classification results for the medical imaging tasks in Table Ta We find that compared to random initialization, ImageNet initialization provides markedly
5

---

Under review as a conference paper at ICLR 2021
Table 1: Results for the medical image classification tasks: (a) linear classification; (b) fine-tuning setting. All results are averaged over 5 independent models. Best results for each setting are in boldface. COVIDx 1% setting is omitted due the scareity of labels in COVIDx.
RSNA (AUC)
CheXpert (AUC)
COVIDx (Accu.)
MURA (AUC)
Method
1%
10%
all
1%
10%
all
10%
all
1%
10%
all
General initialization methods
Random Init.
55.0
67.3
72.3
58.2
63.7
66.2
69.2
73.5
50.9
56.8
62.0
ImageNet Init.
82.8
85.4
86.9
75.7
79.7
81.0
83.7
88.6
63.8
74.1
79.0
In-domain initialization methods
Caption-Transformer Transformer
84.8
87.5
89.5
77.2
82.6
83.9
80.0
89.0
66.5
76.3
81.8
Caption-LSST LSMM
89.8
90.8
91.3
85.2
85.3
86.2
84.5
91.7
75.2
81.5
84.1
Contrastive- Binary
88.9
90.5
90.8
84.5
85.6
85.8
80.5
90.8
76.8
81.7
85.3
ConVIRT (Ours)
90.7
91.7
92.1
85.9
86.8
87.3
85.9
91.7
81.2
85.1
87.6
(b)
RSNA (AUC)
CheXpert (AUC)
COVIDx (Accu.) 10%
MURA A (AUC)
Method
%
10%
all
1%
10%
all
all
1%
10%
all
General initialization methods
Random Init.
71.9
82.2
88.5
70.4
81.1
85.8
75.4
87.7
56.8
61.6
79.1
ImageNet Init.
83.1
87.3
90.8
80.1
84.8
87.6
84.4
90.3
72.1
81.8
87.0
In-domain initialization methods
Caption-Transfomen Transformer
86.3
89.2
92.1
81.5
86.4
88.2
88.3
92.3
75.2
83.2
87.6
Caption-LSTY,
87.2
88.0
91.0
83.5
85.8
87.8
83.8
90.8
78.7
83.3
87.8
Contrastive-Binast
87.7
89.9
91.2
86.2
86.1
87.7
89.5
90.5
80.6
84,0
88.4
ConVIRT (Ours)
88.8
91.5
92.7
87.0
88.1
88.1
90.3
92.4
81.3
86.5
89.0
better representations, despite pretrained on very different domain of images; in-domain image ini- tialization methods that use paired image-text data further improve over ImageNet initialization in almost all settings. Among the in-domain initialization methods, our proposed Con VIRT pretrain- ing achieves best overall results in settings. Notably, we find on three out of the four tasks, with only 1% training data CORVIRT is able to achieve classification results better than the default ImageNet initialization with 100% training data, highlighting the high quality of the learned repre- sentations from VIRT
Fine-tuning. We show the fine-tuning evaluation results in Table lb Similar to the linear setting, we find that: 1) ImageNet initialization is again better than random imitialization with smaller mar- gins; 2) all in-domain initialization methods are better than the popular ImageNet initialization in most settings; and 3) our proposed CORV pretraining again achieves the best overall results in 10 out of the 11 settings, with the exception of the CheXpert dataset with all training data used, where the result of Con VIRT similar to that of the Caption-Transforee Transformer result. Most notably, on all datasets, with only 10% labeled training data ConVIRT achieves classification results that better or close to the ImageNet initialization with 100% training data results.
random initialization is different from Raghu et al.) (2019): while they showed comparable results from the two strategies, we find that using ImageNet initialization still superior than random initialization in most results, justifying its popularity. Upon closer examination, we conjecture that this is likely due to under-optimization of their models: while our ResNet50 with random initialization achieves an average AUC of 85.8 on the CheXpert dataset, their ResNet50 model only achieved 83.5 AUC on the same evaluation set.
We also notice that conclusion of using ImageNet
4.2 RETRIEVAL TASKS
We present the zero-shot image-image and text-image retrieval results in Table For the image- image retrieval setting, we present additional results from fine-tuning our pretrained model on all CheXpert training data, and them "upper bounds" of the results obtained from the of supervised labels. We find that: 1) using ImageNet pretrained CNN weights in a zero-shot image retrieval setting is only better than random guess by small margins; 2) all in-domain pretrained CNN

---

Under review as a conference paper at ICLR 2021
Table 2: Zero-shot image-image and text-image retrieval results the CheXpert 8x datasets. Random shows results from a random guess; ConVIRT + CheXpert Supervised shows results from further fine-tunin the pretrained weights with supervised training data. Text-image retrieval results are not obtained for some methods due to the lack of text encoders.
Image Image Retrieval
Text-Image Retrieval
Method
Prec@5
Prec 10
Prec@
Prec@5
Prec 10
Prec@50
Random
12.5
12.5
12.5
12.5
12.5
12.5
ImageNet
14.8
14.4
15.0
In-domain initialization methods
Aoption-Transfomeee
29.8
28.0
23.0
Caption-LSTM
34.8
32.9
Contrastive-Binan
38.8
36.6
29.7
15.5
14.5
13.7
CORVIRT (Ours)
45.0
42.9
35.7
60.0
57.5
48.8
Fine-tumed
ConVIRT CheXpert Supervised
56.8
56.3
48.9
weights achieve much better retrieval performance than ImageNet weights; and 3) our proposed Con VIRT pretraining achieves the best overall retrieval results on all metrics. We find that while Contrastive-Binanl performs notably better than other baselines in the image-image retrieval setting, its text-image retrieval results are far from VIRT pretraining. We conjecture that the lack of an explicit similarity-basee loss function the Contrastive-Binet baseline results in misaligned representations in the image and text space, leading to poor results in text-image retrieval.
To understand how well CORV pretrain- ing helps separate images from different ab- normality categories in its encoding space, in Figure 3 present t-SNE plots (Maaten & Hinton 2008) of candidate images in the
CheXpert 8x200 dataset for five selected cat- egories, from the ImageNet pretrained CNN encoder and the ConVIRT pretrained coder. It is worth noting that clustering im-
Pneumerth
Hoard
Cardiinepoyy
Aceectaats
ages in our setting is much more challenging than that in the general object classification setting due to the high inter-class similarity of the medical images. Nevertheless we find that Con VIRT pretraining achieves a better clus- tering of the images in the t-SNE plots. On the other hand, the lack of clear separations between groups suggests room for further improvement.
(a) ImageNet Pretraining
(b) VIRT Pretraining
Figure 3: t-SNE visualizations of encoded image representations from different pretraining methods.
ANALYSIS AND DISCUSSION
Comparisons to Image-only Contrastive Learning. COnVIRT shows superior results against baselines in evaluation, but an im- portant question remains as to how it com- pares against existing image-only contrastive visual representation learning methods. We study this by running two popular such meth- ods, SimCLR (Chen et al. 2020a) and MoCo v2 (Chen et al., 2020b), on the same collec- tion of images that used in our pretrain- ing. We present the results in Table 3 and in- clude model training details in Appendix E
Table 3: Comparisons of ConVIRT image- only unsupervised image representation learning ap- proaches.
RSNA Linear CheXpert Linear Image- Image (1%, AUC)
Method
(1%, AUC)
(Prec @ 10)
ImageNet
82.8
75.7
14.4
SimCLR
86.3
77.4
17.6
MoCo v2
86.6
81.3
20.6
CORVIRT
90.7
85.9
42.9
We find that compared to ImageNet initial ization, both contrastive methods lead to marginal to moderate improvements the classification

---

Under review as a conference paper at ICLR 2021
Original
ImageNet
SimCLR
MoCo v2
ConVIRT
Atelectasis
Cardiomegaly
Pleural Effusion
Figure 4: Saliency maps on sampled images for abnormality categories in the CheXpert dataset. For each image we present maps for ImageNet, SimCLR, MoCo v2 and our CORVIRT initializations. Ground truth regions that are indicative of the abnormalities are shown red boxes in the images.
and retrieval tasks. However, our training strategy substantially outperforms both methods on all tasks, demonstrating its effective of information from the paired text data.
To understand the representational difference that has led to this difference in performance, for all four initialization methods, visualize in Figure 4 the saliency maps (Simonyan al.] 2014) corresponding to the correct class on sampled images from the CheXpert dataset. Models for all ini- tialization methods trained with 1% CheXpert training data under the linear classification setting (with pretrained CNN weights frozen). We find that ImageNet pretraining has led to models that fo- cus on trivial visual features that are mostly irrelevant the task, and that the model with Con VIRT pretrained weights has focused much more relevant areas than those with SimCIR and MoCo v2 pretraining, suggesting more effective representation learning. For example, for atelectasis, while the Con VIRT model has correctly focused on the bottom of the lung regions, the SimCLR model has much more scattered focus and the MoCo model has incorrectly focused on the heart region.
Correlation Between Contrastive Loss and End Task Performance. To understand the relation between a model's performance on the Con pretraining task and its performance on the down- stream tasks, the pretrained checkpoint to the downstream tasks and evaluate its performance. The pretraining was for total of 200 epochs, and 40 points end task results. Figure presents the results of the models' validation loss on the pretraining task, the RSNA 1% data linear evaluation and the two retrieval tasks. For all three tasks, we find clear positive correlation between the pretraining performance and the end task performance. This corroborates that by learning with the VIRT objective, the im- age encoder learns gradually improved representations for the end tasks, and suggests that further improvement on the pretraining task may have positive impact on the end task performance.
analysis where for every 5 epochs during the pretraining.
transferred
obtained with varying validation loss and
and its achieved performance
Hyperparameter Analysis. We run experiments to study the impact of hyperparameters, and find that: 1) similar to previous work on image-only contrastive learning (Chen et al.) (2020a), the pre- training results most sensitive the choice of the temperature value 7; 2) unlike previous work,
8

---

Under review as a conference paper at ICLR 2021
90.5
2.6
90
35
XXX
2.4
89.5
-X
25
100
200
-2.8
-2.6
-2.4
-2.8
-2 -2.4
-2.8
-2.6
2244
(a) Pretraining Loss
(b) RSNA Linear
(c) Image-image
(d) Text-image
(1%, AUC)
(P@ 10)
(P@ 10)
Figure 5: (a) shows pretraining validation loss at different epochs; (b)-(d) shows correlation between the pretraining loss and the performance of three end tasks. For (a) the x-axis shows the training epoch number, and for (b)-(d) the x-axis shows the negative value of the pretraining loss (i.c., L) on a held-out validation set.
changing batch size does not lead substantial change in the classification results; and 3) using linear projection heads instead of non-linear ones notably hurts the retrieval results. We include our detailed comparisons Appendix F
RELATED WORK
Our work is most relevant to existing work on medical image classification, which we have covered in Section T and textual report generation from medical images (Wang et al. [0188 Jing al. 2018
Liu et al. 2019). dominant approach for initializing medical image encoders in this work has been using encoder weights pretrained on ImageNet, despite the drastic difference in image characteristics
(Raghu et al. 2019). Instead, our work proposes an alternative in-domain pretraining strategy, and compares TetgeNet pretraining with different pretraining approaches that also use the paired text data. To our knowledge our work represents the first systematic attempt in this direction.
Our work is inspired by the recent line of work on image view-based contrastive visual representa- tion learning (Hénaff et al. 22200 [Chen et al. 2020aa He et al. [2020] Grill et al.) 20220), but differs from existing studies by the contrastive learning with text modality, which as we show in Section is more effective in learning high-quality representations of medical images.
Another line of work related to ours visual-linguistic representation learning (Lu et al. 2019 Ran & Bansal 2019 Su et al., 2020). Among existing studies. [Iharco et al. (2020) and Gupta et al. (2020) used a gross-modality contrastive objective related to ours, but for purpose of probing visual-linguistic models and learning phrase grounding, respectively. Our work differs from this line of work in several crucial ways: 1) existing work in visual- lingul-in learning focused on learning visual representations from paired text via binary contrastive prediction task, whereas we showed the superior performance of the new cross- moddlltyy NCE objectives in our setting; 2) existing work has primarily used object representations extracted from image segmentation models in their prepro- cessing steps, making them less applicable to medical image understanding tasks where anatomical segmentations are extremely hard to obtain; 3) while existing work has evaluation primarily on risual-linguiss tasks such as visual question answering, we instead focus on evaluation with classification and retrieval tasks which are the center of medical image understanding research.
CONCLUSION
We presented Con VIRT, an unsupervised method for learning medical visual representations from naturally occurring pairing of images and text. Our method relies on contrasting the image repre- sentations with the paired text data via bidirectional objective between the two modalities. On medical image classification tasks and 2 image retrieval tasks, VIRT outperformed other strong in-domain initialization methods that also use the text data, and led to representations with notably higher quality. Compared ImageNet pretraining, Con VIRT is able to achieve the same level of classification accuracy with an order of magnitude less labeled data. We hope that our work can inspire future work that makes more efficient of textual data for medical image understanding.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Michael David Abråmoff, Yiyue Lou, Ali Erginay, Warren Clarida, Ryan Amelon, James C Folk, and Meindert Niemeijer. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Investigative Ophrhalmology & Visual Science, 57(13):5200-5206, 2016.
Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020b.
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed- Trans- former for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Jeffrey De Fauw, Joseph R Ledsam, Bernardino Komera-Paredee Stanislav Nikolov, Nenad Toma- sev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan ODonoghue, Daniel Visentin, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature Medicine, 2449):134-11350 2018.
Dina Demner-Fushmanl Mare Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing collection of radiol- ogy examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304-310 2016.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-eveel classification of skin cancer with deep neural networks. Nature, 542(7639):115-1188 2017.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733. 2020.
Varun Gulshan, Lily Peng, Mare Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al. Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 31 31622):2402-24100 2016.
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang. Jan Kautz, and Derek Hoiem. Con- trastive learning for weakly supervised phrase grounding. In Proceedings of the 16th European Conference on Computer Vision (ECCV), 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He. Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
10

---

Under review as a conference paper at ICLR 2021
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In International Conference Machine Learning (ICML). 2020.
Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. Probing text models for common ground with visual representations. arXiv preprint arXiv:2005.00619 2020.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, al. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference Artificial Intelligence, 2019.
Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), 2018.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lun- gren, Chih-ying Deng, Roger G Mark, and Steven Horng. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. The 2015
International Conference for Learning Representations, 2015.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willic Boag. Wei-Hung Weng, Peter Szolovits, and Marzych Ghassemi. Clinically accurate chest X-ray report generation. In Machine Learming for Healthcare Conference, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visi- olinguistic representations for lision-add-lanuage tasks. In Advances in Neural Information Processing Systems, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(Nov) 2605, 2008
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit, Association for Com- putational Linguistics (ACL) System Demonstrations, 2014.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 1807.03748, 2018
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems, 2019.
Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang, Kaylie Zhu, Dillon Laird, Robyn Ball, et al. MURA: Large dataset for abnormality detec- tion in musculoskeletal radiographs. In 1st Conference on Medical Imaging with Deep Learning (MIDL), 2018a.
Pranav Rajpurkar, Jeremy Irvin, Robyn L Ball, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, et al. Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Medicine, 15(1 1002686, 2018b
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein. et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, (5(3):211-255 2015.
11

---

Under review as a conference paper at ICLR 2021
George Shih, Carol C Wu, Safwan S Halabi, Mare D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya ing the National Institutes of Health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, (e) 80041, 2019.
et al. Augment-
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop. 2014.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre- training of generic visual-linguistic representations. In International Conference on Learning Representations (ICLR), 2020.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2019.
Ramakrishna
Vedantam, Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Linda Wang and Alexander Wong. COVID-Net: A tailored deep convolutional neural network de- sign for detection of COVID-19 cases from chest X-ray images. arXiv preprint arXiv:2003.0987/ 2020.
Xiaosong Wang. Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. ChestX-ray8; Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers. TieNet: Text-image embedding network for common thorax disease classification and reporting in chest X-rays. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rmi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State- of-the-art natural language processing. arXiv preprint arXiv: 2019.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference Machine Learning (ICML), 2015.
12

---

Under review as a conference paper at ICLR 2021
A MODEL IMPLEMENTATION AND PRETRAINING DETAILS
Dataset Preprocessing. For the MIMIC-CXR chest radiograph dataset, we use the publicly avail- able JPG version of it For both the MIMIC-CXR chest dataset and the Rhode Island Hospital bone image datasets, we resize the image files to have a size of 256 on the larger side. For the textual radiology report data, we first tokenize all reports with the default English tokenizer in version 4.0.0 of the CoreNLP library (Manning et al. 2014). Next, we keep only the Findings and Impression sections and remove all other sections. We remove all image-text pairings from the dataset where the text section is empty or has less than 3 tokens. This preprocessing procedure gives us about 21 7k total image-text pairs for pretraining our chest image encoder and 48k total pairs for pretraining our bone image encoder.
Image and Text Encoders. For the image encoder, we use the standard ResNer50 implementation provided by the torchvision library. For the text encoder, we use the BERT base encoder offered by the Transformers library (Wolf et al., 2019) and initialize with the ClinicalBERT model (Alsentzer ct al. 2019) pretrained on the MIMIC clinical notes. We also experimented with training special- ized BERT encoder on a large collection of radiology notes but found that made no substantial difference in the pretraining results. At pretraining time we freeze the embeddings and the first 6 layers of this BERT encoder, and only fine-tune the last 6 layers for our contrastive task.
Other Hyperparameters. For contrastive learning. we use projection layers with an output di- mension d = 512, a temperature value T = 0.1, loss weight À = 0.75. These hyperparameter settings are obtained by comparing the linear evaluation validation scores on the RSNA image clas- sification task with the pretrained ResNer50 weights. For the image transformation family T, we adopt the implementations offered by the torchvision library 2 We apply random cropping with a ratio sampled from [0.6, 1.0]; horizontal flipping with
0.5; affine transformation with de- gree sampled from [-20, 20). max horizontal and vertical translation fractions 0.1, and a scaling color
factor sampled from [0.95. 1.05]; color jittering with brightness and contrast adjustment ratios sam- pled from [0.6, 1.4]; and Gaussian blur with o E [001, All images are resized to 224x224 after the transformation b is applied. Limited by computational resources, we arrive at these image transformation parameters via preliminary experiments rather than a systematic search.
Pretraining Details. At pretraining time, for each dataset, we randomly sample 5k image-text pairs to form a held-out validation set. We we use the Adam optimizer (Kingma & Ba 2015) with an initial learning rate of lc-4 and weight decay of 1e-6. We initialize the image encoder with ImageNet pretrained weights at the beginning of pretraining, and use a fixed batch size of 32. We calculate the validation loss every 5000 steps, and if the validation loss does not decrease after straight evaluation runs, we anneal the learning rate by a factor of 0.5. We stop pretraining after 200 evaluation runs, and save the model checkpoint that achieves the lowest validation loss. For efficiency, we employ mixed-precisien MIMIC-CXR dataset took about 3 days on a single Titan RTX GPU card.
training, and for reference, the whole pretraining run on the
B IMAGE CLASSIFICATION EXPERIMENTS
We prepared and used the 4 image classification datasets following the procedures below:
1. RSNA Pneumonia Detection (Wang al. 2017 Shih al., 2019): used the original version of this dataset available at its Kaggle page which contains 25184/1 500/3300 annotated images in its sliming/validationneess sets, respectively.
25184/1500/3000
2. CheXpert image classification (Irvin al.) 2019):: we downloaded the original version of this dataset from its official website Since the original expert-labelee test set of this dataset is hidden and not included part of the release, instead followed |Ragu et al. (2019) and used the original expert-labeled validation set as our test set, and randomly sampled 5000 images from
13

---

Under review as a conference paper at ICLR 2021
the original training set for validation purpose. The resulting dataset contains 218414/5000/233 images in each split.
3. COVIDx image classification (Wang Wong 2020): we prepared this dataset following the scripts provided by its authors We used the version 4 of this dataset, the latest version at the time of this work. We additionally randomly sampled 300 images from the training set for validation, resulting in a dataset with 13598/300/300 images in each split.
MURA bony abnormality detection [Rajpurkar et al. 201 we downloaded the original ver- sion of this dataset from its website Similar to the CheXpert dataset, we again used the original validation set as our test set, and randomly sampled 10% images from the training set for valida- tion, resulting in dataset with 33078/3730/3197 images in each split. Different from the other 3 datasets, the MURA dataset uses patient-level evaluation, mcaning that the prediction results from different images of the same patient needs to be aggregated produce final prediction for the patient, which is then scored against the gold patient label. We therefore followed Ra- jppurkar et al. (2018a) and at test time aggregated result for patient by averaging the predicted probabilities from multiple images.
Classification Model Training Details. For all models that require ImageNet pretrained initial- ization, we use the pretrained weights from torchvision, which achieves an ImageNet top-5 error rate of 7.13%. For all datasets, we first zero-pad the input image be square, and then resize it to be 224x224. For training, we use the Adam optimizer with an initial learning rate of le-3 for the COVIDx task and le-4 for the other three tasks. We additionally apply a weight decay of 1e-6 and a dropout before the last classification layer with 0.2 in all tasks. All classification models are trained with a batch size of 64. the fine-tuning evaluation setting, we first "warmup" the classi- fication head by freezing the CNN weights and only training the classification head with learning rate of 1e-3 for 200 steps, after which we unfreeze the CNN weights and fine-tune the entire network together. Validation score is obtained after each epoch of training and we anneal the learning rate by a factor of 0.5 if the validation score is not improved after 3 epochs. The training is stopped after no validation improvement is observed for 10 straight epochs, which point the model checkpoint with the highest validation score is evaluated on the test set.
C IMAGE-IMAGE RETRIEVAL DATASET COLLECTION
We create the CheXpert 8 200 Retrieval Dataset with 8 different abnormality categories commonly found in Chest radiograph images, including atelectasis, cardiomegaly, edema, fracture, pleural effusion, pneumonia. pneumothorax and special no finding category indicating that no obvious abnormality is found in the image. We create the dataset by reusing existing rule-labeled annotations in the CheXpert dataset (Irvin et al. 2019) and additional expert annotations. To create the candidate images for a category label f, we go through all images in the CheXpert training set, and keep an image as a candidate image if only its label for is positive and all other categories negative. We only include images with this "exclusive positivity" as candidate images, mainly to avoid confounding results between categories in retrieval evaluation.
To create the query images for category E, we again first pre-select 50 exclusively positive images for this category in the CheXpert training set (with all candidate images excluded). Next, we ask board-certified radiologist to examine each of the 50 images, and exclude images that: 1) might indicate additional abnormalities other than f, 2) have uncommon color or contrast distortions in the image, 3) are not well posed during the capture of the image. This procedure is mainly to avoid including query images that have uncommon features and may therefore bias the retrieval evaluation results. At the end, we aggregate the annotation results from the radiologist and keep 10 query images for each abnormality category.
D TEXT-IMAGE RETRIEVAL DATASET COLLECTION
For the text-image retrieval dataset, we first reuse all candidate images from the CheXpert 2x image-image retrieval dataset described above, with 200 images for each of 8 categories. To create
14

---

Under review as a conference paper at ICLR 2021
Table 4: Example textual queries for each of the categories in the text-image retrieval task.
Image Category
Example Textual Query
Atelectasis
Platelike opacity likely represents atelectasis.
Cardiomegaly
The cardiac silhouette is enlarged.
Edema
The presence of hazy opacity suggests interstitial pulmonary edema.
Fracture
A cortical step off indicates the presence of a fracture.
Pleural Effusion
The pleural space is partially filled with fluid.
Pneumonia
A pulmonary opacity with ill defined borders likely represents pneumonia.
Pneumothorax A medial pneumothorax is present adjacent to the heart.
No Finding
No clinically significant radiographic abnormalities.
the textual queries for each abnormality category, we ask a board-certified radiologist to write at least 5 different sentences that he will use describe this abnormality in radiology reporting. We additionally set the following requirements: 1) the sentences must describe the category with no ambiguity and must not include other categories; 2) the sentences must be diverse from each other; and 3) the sentences should not include very specific anatomic locations or rare clinical observations. At the end, we aggregate the results and keep textual queries for each abnormality category. For reference, we present example textual queries in Table 4
E EXPERIMENTS ON IMAGE-ONLY CONTRASTIVE LEARNING METHODS
We run experiments with two popular image-only contrastive visual representation learning meth- ods: SimCLR (Chen et al. 2020a) and MoCo v2 (Chen et al.. (020b)). For fair comparison, in both experiments we use the exact same set of images from the MIMIC-CXR dataset that we use in the pretraining of our method and the baselines. Our settings for each method are:
SimCLR: We use the open PyTorch implementation available at https::///// / Siigh.tt........ hub. com/ PattaleesSSmccae For image encoder we use ResNet50. We use cosine similarity in the loss function, set the temperature value to 0.1 and set the output dimension to 128. We use the default image augmentation functions the paper except for the color jittering transformation where we set the saturation and hue adjustment to 0 due to the monochrome nature of our medical images. For training, we use the Adam optimizer with an initial learning rate of 3e-4 and weight decay of le-4. We set batch size to 128 and run training on single GPU card for 100 epochs, that increasing the batch size or number epochs does not lead to improved results. We use the default settings for all other parameters.
find
MoCo v2: We use the authors' PyTorch implementation available at tttttps:://gttiii Bacebookrresearcc/me /wcooo/cccc/ccoooo//oooo//oo For image encoder we use ResNet50. We follow the default MoCo v2 setting and use temperature value of 0.07 and an output dimension of 128. Similarly, we adopt the default image augmentation functions except for the color jittering transformation where we set the saturation and hue adjustment to O. For training, we use the SGD optimizer with a learning rate of 0.0075 and weight decay le-4. We use batch size of 64 and a queue size of 4096, and run parallel training increasing the batch size or number of epochs does not lead to improved results. During training, anneal the learning rate by a factor of 0.1 at the 60th and 80th epochs.
com/
two GPU cards for 100 epochs, as we find that further
F HYPERPARAMETER ANALYSIS
Similar to previous work on unsupervised image representation learning (Chen et al.) 2020a; He jet al. 2020). we first find that the effectiveness of ConVIRT pretraining is most sensitive to the temperature value т. As shown in Table 5 using temperature much lower than the ideal value (T = 0.01) hurts the retrieval results, and a temperature much larger (r = 1) notably hurts the performance on all tasks. Unlike previous work, we find that using a smaller or larger batch size hurts the retrieval performance, but neither setup brings substantial impact to the classification results. Lastly, find that replacing the non-linear projection heads in go and gu with linear layers hurts the
15

---

Under review as a conference paper at ICLR 2021
Table Evaluation results with different hyperparameters, for the RSNA 1% data linear evaluation, image-image retrieval and resents using linear projection layers for 9o and Ju- Our default model uses 0.1, bs 32 and non-linear projections.
text-image retrieval tasks. bs represents batch size and linear proj. rep-
RSNA Linear (1%, AUC)
Image-Image (Prec@ 10)
Text-Image (Prec@ 10)
Settings
COnVIRT (default)
90.7
42.9
57.5
0.01
90.7
40.5
21.0
r
89.6
25.0
31.0
90.3
40.0
55.8
=128
90.3
39.3
50.3
linear proj.
90.6
40.8
55.8
retrieval results moderately, suggesting worse representations. However, this is again not reflected notably in the RSNA classification results.
16


Reviews:
[{"review_id": "triDpDQjme7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#####################   Summary   ####################\n\nThis paper presents the Contrastive VIsual Representation Learning from Text (ConVIRT) pretraining strategy to learn fine-grained medical visual representations of medical images by pretraining on large-scale image-report pairs. As a result, ConVIRT improves medical visual representations by maximizing the agreement between true image-text pairs versus random pairs via a bidirectional contrastive objective between the image and text modalities. The authors demonstrate that ConVIRT can help improve performance on down-streaming tasks like image classification and image retrieval tasks.\n\n\n#####################   Strengths   #################### \n\n(1) This paper is really clearly written. The paper is easy to follow and understand.\n\n(2) The paper explores an interesting direction of learning fine-grained medical visual representations for medical image understanding.\n\n(3) The proposed ConVIRT is well motivated. The experimental results are very solid.\n\n#####################   Weakensses   #################### \n\n(1) The paper is limited in its novelty borrowing ideas from some previous works: (i) contrastive learning [1][2] and (ii) image-text representations pretraining [3]. However, it is an interesting idea of applying contrastive learning to medical image and text. So, I think this paper is novel to some extent, but not that novel, since it mainly makes some incremental contribution by combining ideas from existing work.\n\n(2) The analysis is not convinced (see below).\n\n#####################   Questions   #################### \n\nI have some questions for the authors:\n\n(1) What the medical visual representations have learned?\n\n(2) Compared with the baseline methods (section 3.3), why the proposed ConVIRT can achieve the best results? How these results can be achieved? \n\n(3) Why the proposed ConVIRT can learn better medical visual representations than the SimCLR [1] and MoCo v2 [2] under the setting of image-only contrastive learning (section 5)? How they differ from representations learned by existing methods, e.g., SimCLR [1] and MoCo v2 [2]?\n\n(4) Can the medical visual representations pretrained on the chest image be further finetuned on the bony image?\n\nOverall, although the paper doesn't provide insights around what the representations have learned and how they differ from representations learned/used by existing methods, they have provided substantial evidence to suggest that pretraining helps in a lot of downstream tasks. In other words, the proposed ConVIRT seems to be useful for the researchers in the medical AI field.\n\n\n[1] A simple framework for contrastive learning of visual representations. In ICML 2020.\n\n[2] Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297.\n\n[3] ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS 2019.\n\n####################  After Rebuttal  #################### \n\nI thank the authors for responding to the comments and have read them carefully. The authors have addressed my concerns in the rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NQlsOFVSZfR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n1. This paper tackles the medical image understanding problem. The aim of this paper is to learn a generic feature representation for medical image that could benefits downstream tasks like medical image classification, zero-shot classification. The main contribution of this paper is proposing a contrastive loss that the matched pair of image and text should have a higher corresponding score than the mis-matched pairs.\n\nPros:\n1. The problem of learning a generic representation from limited training data is important.\n2. Comparing with the baseline, the performance of the proposed approach looks good.\n3. The paper is clearly written.\n\nWeakness:\n1. My first concern is the novelty. If I understand correctly, the main novelty of this paper is proposing the losses (Eq. 2 and Eq. 3) in the section of contrastive visual representation learning from text. However those losses are well-known in the domain of image-text retrieval. Current vision-language BERT (VLBERT) approaches use similar loss to optimize their model [Lu et. al., 2019]. The difference is that, in VLBERT papers, the model is trying to contrast the positive pair of image and text against all negative pair of image and texts that sampled from a batch: $$\\ell = - \\log \\frac{\\exp(<v_i, u_i> / \\gamma)}{\\sum_{k=1, k\\neq i}^N\\sum_{j=1, j\\neq i}^N \\exp(<v_j, u_k>/\\gamma)}.$$ While the proposed approach uses Eq. 2 and Eq. 3 to optimize text retrieval and image retrieval, respectively. However, the motivation of separating the loss into Eq. 2 and Eq. 3 in unclear.\n2. In the related work, recent advance in vision-language BERT has been mentioned (Last paragraph of Sect. 6 Related work). The paper stated that (in point 2) 'existing work has focused on visual-linguistic tasks such as visual question answering.', which might not be accurate. As current vision-language BERT models are trying to learn a generic representation for both image and text, where downstream tasks are ways to evaluate the representation. The aim of vision-language BERT is aligned with the aim of this paper.\n3. The paper mentioned multiple times that the proposed approach is specific for medical images (Last paragraph of Related work). However this is unclear to me why the proposed approach is specific for medical images?\n\nI would be happy to increase my rating, if the author could differentiate the proposed approach (losses) with the one used in current visual-language BERT. It would be the best if the author could state clearly why the proposed approach would be beneficial for medical images (Otherwise, 'medical' in the title of this paper might need to be removed). The author might also need to re-state the novelty contribution of this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2THIW-CiMnP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose a new model, named ConVIRT to learn the medical visual representation from paired image and textual data in an unsupervised strategy. In ConVIRT, they mainly use a contrastive loss with two modalities (images and texts) as inputs to learn the representation. The experimental results show their proposed model achieve higher performance than other methods in image classification and zero-shot retrieval tasks.\n\nStrength.\nIn this work, the authors propose a simple and straightforward method to learn the image visual representation using modified contrastive loss with two modalities as inputs. Besides, the experimental comparisons show their model outperforms other baselines and methods.\n\nWeakness.\n1. Novelty. In this work, the main contribution is the proposed modified contrastive loss with two modalities as inputs. It is an interesting and challenging problem that how to extract the visual representation in an unsupervised strategy. However, the proposed modified contrastive loss is below the standard of top-tier conference ICLR. \n2. Experiment. This work evaluates their model in many tasks and datasets. But, it might miss some baselines or other state-of-the-art methods. (1) Baselines. I suggest the authors could add the baseline using triplet loss. Both contrastive loss and triplet loss could enlarge the distances between different classes. (2) Baselines. I also suggest the authors could add the baseline only using the images as inputs, just like Chen et al. 2020a. and maybe only using the text data as inputs. \n\n3. Implementation details. (1) BERT encoder. In section 2.3, they use a BERT encoder to extract the textural features. I suggest the authors could give more details about it, such as how to initialize the parameters? As we known, there exists a huge gap between contexts form website and that from the medical report, since there are so many specific nouns. (2) Text transformation function. I might not agree with the text transformation function used in this work as the following reasons. There might exist a big gap and different meanings between two sentences even in one document. As the goal is to maximize the agreement between the true image-text representation pairs, there might occur a conflict between one image and two different meaning sentences sampled from the same documents.\n\n4. Visualization. (1) I suggest the authors could give more visualization examples about the learned visual and textual representation, such as t-SNE. Such visualization might also address my above the concerns about the potential conflict during training. In these t-SNE examples, according to the proposed pipeline, the distance between the learned visual representation and the learned textual representation of the corresponding sentences in the same document should be smaller than the distance between the visual representation and some similar sentences from other documents. \n5. New type of baselines. This work aims to learn the useful visual representation from both images and texts. However, I suggest that they should report another kind of baselines, only using images to learn the visual representation in an unsurprised way, such as Chen et al. 2020a and [1] (Unsupervised Representation Learning by Predicting Image Rotations, ICLR 2018). \nThe results of the proposed method should outperform this kind of baseline. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed ConVIRT learns representations of medical data from paired image and text data.
While the paper addresses a relevant problem, the reviewers agree that the method has limited novelty. Two reviewers find and that the experiments are not convincing. One reviewer notes that the paper does not compare to the state-of-the-art methods for the tasks.

Decision: Reject