Contrastive Learning of Medical Visual Representations from Paired Images and Text

1 INTRODUCTION Medical image understanding has the potential to transform healthcare and has seen rapid progress with the use of deep neural architectures ( Gulshan et al. , 2016 ; Esteva et al. , 2017 ; De Fauw et al. , 2018 ; Rajpurkar et al. , 2018b ) . Yet , with expert-level performance achieved only in some specialties and under some circumstances , medical image understanding remains a difficult task for the majority of specialties , mainly due to its challenging nature and the extreme scarcity of annotated data . Existing work has followed two general approaches to obtain annotations for medical imaging tasks . The first approach has been using high-quality annotations created by medical experts ( Abràmoff et al. , 2016 ; Gulshan et al. , 2016 ; Shih et al. , 2019 ; Wang & Wong , 2020 ) . However , the high cost of this approach has resulted in datasets that are mostly orders of magnitude smaller than natural image datasets such as ImageNet ( Russakovsky et al. , 2015 ) . To remedy this , existing work has relied heavily on trans- ferring model weights from ImageNet pretraining ( Wang et al. , 2017 ; Esteva et al. , 2017 ; Irvin et al. , 2019 ) . This approach is suboptimal because , as shown in Figure 1 , medical image understanding often requires representations of very fine-grained visual features that are drastically different from those required for identifying objects in natural images . As a result , Raghu et al . ( 2019 ) found that ImageNet pretraining often provides little to no benefit compared to simple random initialization . A second popular approach is to use expert-crafted rules to extract labels from the textual reports accompanying the medical images . This approach has led to datasets of larger scale , since the text data paired with medical images are often produced naturally by medical experts in their routine work- flow and abundant in a typical hospital ’ s IT systems . Nevertheless , this rule-based label extraction approach has two limitations : 1 ) the rules are often inaccurate and limited to a few major categories ( Wang et al. , 2017 ) , leading to very inefficient use of the textual report data ; 2 ) these rules are often domain-specific and sensitive to the style of the text , making cross-domain and cross-institution generalization difficult ( Irvin et al. , 2019 ) . In efforts to make more efficient use of unlabeled image data , several recent studies have shown promising results from contrastive representation learning from natural images ( Chen et al. , 2020a ; He et al. , 2020 ; Grill et al. , 2020 ) . However , as we will show , applying these image view-based contrastive methods to medical images provides only marginal benefits compared to ImageNet pretraining , a result mostly due to the high inter-class similarity of the medical images as in Figure 1 . In this work , we aim to improve visual representations of medical images by combining the benefits of both learning from abundant textual data and unsupervised statistical approaches . We present Contrastive VIsual Representation Learning from Text ( ConVIRT ) , a framework for learning visual representations by exploiting the naturally occurring pairing of images and textual data . ConVIRT improves visual representations by maximizing the agreement between true image-text pairs versus random pairs via a bidirectional contrastive objective between the image and text modalities . We apply ConVIRT to the pretraining of medical image encoders , and show that it leads to higherquality in-domain image representations that capture the subtlety of visual features required for medical image understanding tasks . Compared to existing methods , ConVIRT has the advantages of utilizing the paired text data in a way agnostic to the medical specialty and requiring no additional expert input . This allows us to evaluate ConVIRT by transferring our pretrained weights to 4 different medical image classification tasks covering 2 different specialties . We find that the resulting models outperform all baseline initialization approaches , including the standard ImageNet pretraining and several strong baselines that also utilize the paired text data . Most notably , in all 4 tasks , ConVIRT requires only 10 % as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance . We further evaluate ConVIRT on two new zero-shot retrieval tasks , an image-image and a text-image retrieval task , and also find it superior to all baselines . To facilitate future research , we will make our code and the collected retrieval datasets available . 2 METHOD . 2.1 TASK DEFINITION . We start by giving a formal description of our representation learning setting . We assume paired input ( xv , xu ) where xv represents one or a group of images , and xu represents a text sequence which describes the imaging information in xv . Our goal is to learn a parameterized image encoder function fv , which maps an image to a fixed-dimensional vector . We are then interested in transferring the learned image encoder function fv into downstream tasks , such as classification or image retrieval . In this work , we model the encoder function fv as a convolutional neural network ( CNN ) . We note that paired image-text data ( xv , xu ) naturally exists for many medical domains . Medical experts such as radiologists produce textual descriptions of images as part of their routine workflow , some of which are also made publicly available ( Demner-Fushman et al. , 2016 ; Johnson et al. , 2019 ) . 2.2 CONTRASTIVE VISUAL REPRESENTATION LEARNING FROM TEXT . An overview of our method , ConVIRT , for learning fv is shown in Figure 2 . At a high level , our method converts each input image xv and text xu into d-dimensional vector representations v and u respectively , following a similar processing pipeline . For each input image xv , our method starts by drawing a random view x̃v from xv with a sampled transformation function tv ∼ T , where T represents a family of stochastic image transformation functions described later . Next , the encoder function fv transforms x̃v into a fixed-dimensional vector hv , followed by a non-linear projection function gv which further transforms hv into vector v : v = gv ( fv ( x̃v ) ) , ( 1 ) where v ∈ Rd . Similarly , for each text input xu , we obtain a span x̃u from it following a sampling function tu , and then a text representation u with : u = gu ( fu ( x̃u ) ) , where fu is a text encoder , gu a projection , and u ∈ Rd . The projection functions gv and gu project representations for both modalities from their encoder space to the same d-dimensional space for contrastive learning . At training time , we sample a minibatch of N input pairs ( xv , xu ) from training data , and calculate their representation pairs ( v , u ) . We use ( vi , ui ) to denote the i-th pair . The training objective of ConVIRT involves two loss functions . The first loss function is an image-to-text contrastive loss for the i-th pair : ` ( v→u ) i = − log exp ( 〈vi , ui〉/τ ) ∑N k=1 exp ( 〈vi , uk〉/τ ) , ( 2 ) where 〈vi , ui〉 represents the cosine similarity , i.e. , 〈v , u〉 = v > u/‖v‖‖u‖ ; and τ ∈ R+ represents a temperature parameter . This loss takes the same form as the InfoNCE loss ( Oord et al. , 2018 ) , and minimizing it leads to encoders that maximally preserve the mutual information between the true pairs under the representation functions . Intuitively , it is the log loss of an N -way classifier that tries to predict ( vi , ui ) as the true pair . Note that unlike previous work which use a contrastive loss between inputs of the same modality ( Chen et al. , 2020a ; He et al. , 2020 ) , our image-to-text contrastive loss is asymmetric for each input modality . We therefore define a similar text-to-image contrastive loss as : ` ( u→v ) i = − log exp ( 〈ui , vi〉/τ ) ∑N k=1 exp ( 〈ui , vk〉/τ ) . ( 3 ) Our final training loss is then computed as a weighted combination of the two losses averaged over all positive image-text pairs in each minibatch : L = 1 N N∑ i=1 ( λ ` ( v→u ) i + ( 1− λ ) ` ( u→v ) i ) , ( 4 ) where λ ∈ [ 0 , 1 ] is a scalar weight . 2.3 REALIZATION . We note that our ConVIRT framework defined above is agnostic to the specific choice of image and text encoders , transformations and projection functions . In this work , following previous work ( Chen et al. , 2020a ) , we model gv and gu as separate learnable single-hidden-layer neural networks , i.e. , gv ( · ) = W ( 2 ) σ ( W ( 1 ) ( · ) ) where σ is a ReLU non-linearity , and similarly for gu . For the image encoder fv , we use the ResNet50 architecture ( He et al. , 2016 ) for all experiments , as it is the architecture of choice for much medical imaging work and is shown to achieve competitive performance . For the text encoder fu , we use a BERT encoder ( Devlin et al. , 2019 ) followed by a max-pooling layer over all output vectors . We initialize our BERT encoder with the ClinicalBERT model ( Alsentzer et al. , 2019 ) pretrained on the MIMIC clinical notes , which achieved state-of-theart performance on a suite of clinical NLP tasks . At training time we allow the encoder to adapt to our contrastive task by freezing the embeddings and the first 6 layers of this BERT encoder and fine-tuning the last 6 layers . For the image transformation family T where tv is sampled from , we use sequential applications of five random transformations : cropping , horizontal flipping , affine transformation , color jittering and Gaussian blur . Different from recent work on contrastive visual representation learning ( Chen et al. , 2020a ; b ) , we only apply brightness and contrast adjustments in color jittering , due to the monochrome nature of the medical images . For the text transformation function tu , we apply a simple uniform sampling of a sentence from the input document xu ( i.e. , x̃u is a randomly sampled sentence from xu for each minibatch ) . We did not use a more aggressive transformation mainly because sampling at the sentence level can preserve the semantic meaning of the sampled spans . 3 EXPERIMENTS . 3.1 DATA FOR PRETRAINING . We test our ConVIRT framework by pretraining two separate image encoders covering different medical specialties using two separate paired image-text datasets : • Chest image encoder : We use version 2 of the public MIMIC-CXR database ( Johnson et al. , 2019 ) , which is a collection of chest radiograph images paired with their textual reports , and since its release has become a standard resource for studying multi-modal modeling of medical images . After preprocessing , this dataset contains a total of about 217k image-text pairs , with each pair containing an average of 1.7 images and 6.0 sentences . • Bony image encoder : We obtain a collection of musculoskeletal image-text pairs from the Rhode Island Hospital system . Following chest images , musculoskeletal images constitute the second most common type of radiograph images in a typical hospital . This dataset contains a total of 48k image-text pairs , with each pair containing an average of 2.5 images and 8.0 sentences . We include model implementation and pretraining details in Appendix A .

Summary: 1. This paper tackles the medical image understanding problem. The aim of this paper is to learn a generic feature representation for medical image that could benefits downstream tasks like medical image classification, zero-shot classification. The main contribution of this paper is proposing a contrastive loss that the matched pair of image and text should have a higher corresponding score than the mis-matched pairs.