Contrastive Learning of Medical Visual Representations from Paired Images and Text

1 INTRODUCTION Medical image understanding has the potential to transform healthcare and has seen rapid progress with the use of deep neural architectures ( Gulshan et al. , 2016 ; Esteva et al. , 2017 ; De Fauw et al. , 2018 ; Rajpurkar et al. , 2018b ) . Yet , with expert-level performance achieved only in some specialties and under some circumstances , medical image understanding remains a difficult task for the majority of specialties , mainly due to its challenging nature and the extreme scarcity of annotated data . Existing work has followed two general approaches to obtain annotations for medical imaging tasks . The first approach has been using high-quality annotations created by medical experts ( Abràmoff et al. , 2016 ; Gulshan et al. , 2016 ; Shih et al. , 2019 ; Wang & Wong , 2020 ) . However , the high cost of this approach has resulted in datasets that are mostly orders of magnitude smaller than natural image datasets such as ImageNet ( Russakovsky et al. , 2015 ) . To remedy this , existing work has relied heavily on trans- ferring model weights from ImageNet pretraining ( Wang et al. , 2017 ; Esteva et al. , 2017 ; Irvin et al. , 2019 ) . This approach is suboptimal because , as shown in Figure 1 , medical image understanding often requires representations of very fine-grained visual features that are drastically different from those required for identifying objects in natural images . As a result , Raghu et al . ( 2019 ) found that ImageNet pretraining often provides little to no benefit compared to simple random initialization . A second popular approach is to use expert-crafted rules to extract labels from the textual reports accompanying the medical images . This approach has led to datasets of larger scale , since the text data paired with medical images are often produced naturally by medical experts in their routine work- flow and abundant in a typical hospital ’ s IT systems . Nevertheless , this rule-based label extraction approach has two limitations : 1 ) the rules are often inaccurate and limited to a few major categories ( Wang et al. , 2017 ) , leading to very inefficient use of the textual report data ; 2 ) these rules are often domain-specific and sensitive to the style of the text , making cross-domain and cross-institution generalization difficult ( Irvin et al. , 2019 ) . In efforts to make more efficient use of unlabeled image data , several recent studies have shown promising results from contrastive representation learning from natural images ( Chen et al. , 2020a ; He et al. , 2020 ; Grill et al. , 2020 ) . However , as we will show , applying these image view-based contrastive methods to medical images provides only marginal benefits compared to ImageNet pretraining , a result mostly due to the high inter-class similarity of the medical images as in Figure 1 . In this work , we aim to improve visual representations of medical images by combining the benefits of both learning from abundant textual data and unsupervised statistical approaches . We present Contrastive VIsual Representation Learning from Text ( ConVIRT ) , a framework for learning visual representations by exploiting the naturally occurring pairing of images and textual data . ConVIRT improves visual representations by maximizing the agreement between true image-text pairs versus random pairs via a bidirectional contrastive objective between the image and text modalities . We apply ConVIRT to the pretraining of medical image encoders , and show that it leads to higherquality in-domain image representations that capture the subtlety of visual features required for medical image understanding tasks . Compared to existing methods , ConVIRT has the advantages of utilizing the paired text data in a way agnostic to the medical specialty and requiring no additional expert input . This allows us to evaluate ConVIRT by transferring our pretrained weights to 4 different medical image classification tasks covering 2 different specialties . We find that the resulting models outperform all baseline initialization approaches , including the standard ImageNet pretraining and several strong baselines that also utilize the paired text data . Most notably , in all 4 tasks , ConVIRT requires only 10 % as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance . We further evaluate ConVIRT on two new zero-shot retrieval tasks , an image-image and a text-image retrieval task , and also find it superior to all baselines . To facilitate future research , we will make our code and the collected retrieval datasets available . 2 METHOD . 2.1 TASK DEFINITION . We start by giving a formal description of our representation learning setting . We assume paired input ( xv , xu ) where xv represents one or a group of images , and xu represents a text sequence which describes the imaging information in xv . Our goal is to learn a parameterized image encoder function fv , which maps an image to a fixed-dimensional vector . We are then interested in transferring the learned image encoder function fv into downstream tasks , such as classification or image retrieval . In this work , we model the encoder function fv as a convolutional neural network ( CNN ) . We note that paired image-text data ( xv , xu ) naturally exists for many medical domains . Medical experts such as radiologists produce textual descriptions of images as part of their routine workflow , some of which are also made publicly available ( Demner-Fushman et al. , 2016 ; Johnson et al. , 2019 ) . 2.2 CONTRASTIVE VISUAL REPRESENTATION LEARNING FROM TEXT . An overview of our method , ConVIRT , for learning fv is shown in Figure 2 . At a high level , our method converts each input image xv and text xu into d-dimensional vector representations v and u respectively , following a similar processing pipeline . For each input image xv , our method starts by drawing a random view x̃v from xv with a sampled transformation function tv ∼ T , where T represents a family of stochastic image transformation functions described later . Next , the encoder function fv transforms x̃v into a fixed-dimensional vector hv , followed by a non-linear projection function gv which further transforms hv into vector v : v = gv ( fv ( x̃v ) ) , ( 1 ) where v ∈ Rd . Similarly , for each text input xu , we obtain a span x̃u from it following a sampling function tu , and then a text representation u with : u = gu ( fu ( x̃u ) ) , where fu is a text encoder , gu a projection , and u ∈ Rd . The projection functions gv and gu project representations for both modalities from their encoder space to the same d-dimensional space for contrastive learning . At training time , we sample a minibatch of N input pairs ( xv , xu ) from training data , and calculate their representation pairs ( v , u ) . We use ( vi , ui ) to denote the i-th pair . The training objective of ConVIRT involves two loss functions . The first loss function is an image-to-text contrastive loss for the i-th pair : ` ( v→u ) i = − log exp ( 〈vi , ui〉/τ ) ∑N k=1 exp ( 〈vi , uk〉/τ ) , ( 2 ) where 〈vi , ui〉 represents the cosine similarity , i.e. , 〈v , u〉 = v > u/‖v‖‖u‖ ; and τ ∈ R+ represents a temperature parameter . This loss takes the same form as the InfoNCE loss ( Oord et al. , 2018 ) , and minimizing it leads to encoders that maximally preserve the mutual information between the true pairs under the representation functions . Intuitively , it is the log loss of an N -way classifier that tries to predict ( vi , ui ) as the true pair . Note that unlike previous work which use a contrastive loss between inputs of the same modality ( Chen et al. , 2020a ; He et al. , 2020 ) , our image-to-text contrastive loss is asymmetric for each input modality . We therefore define a similar text-to-image contrastive loss as : ` ( u→v ) i = − log exp ( 〈ui , vi〉/τ ) ∑N k=1 exp ( 〈ui , vk〉/τ ) . ( 3 ) Our final training loss is then computed as a weighted combination of the two losses averaged over all positive image-text pairs in each minibatch : L = 1 N N∑ i=1 ( λ ` ( v→u ) i + ( 1− λ ) ` ( u→v ) i ) , ( 4 ) where λ ∈ [ 0 , 1 ] is a scalar weight . 2.3 REALIZATION . We note that our ConVIRT framework defined above is agnostic to the specific choice of image and text encoders , transformations and projection functions . In this work , following previous work ( Chen et al. , 2020a ) , we model gv and gu as separate learnable single-hidden-layer neural networks , i.e. , gv ( · ) = W ( 2 ) σ ( W ( 1 ) ( · ) ) where σ is a ReLU non-linearity , and similarly for gu . For the image encoder fv , we use the ResNet50 architecture ( He et al. , 2016 ) for all experiments , as it is the architecture of choice for much medical imaging work and is shown to achieve competitive performance . For the text encoder fu , we use a BERT encoder ( Devlin et al. , 2019 ) followed by a max-pooling layer over all output vectors . We initialize our BERT encoder with the ClinicalBERT model ( Alsentzer et al. , 2019 ) pretrained on the MIMIC clinical notes , which achieved state-of-theart performance on a suite of clinical NLP tasks . At training time we allow the encoder to adapt to our contrastive task by freezing the embeddings and the first 6 layers of this BERT encoder and fine-tuning the last 6 layers . For the image transformation family T where tv is sampled from , we use sequential applications of five random transformations : cropping , horizontal flipping , affine transformation , color jittering and Gaussian blur . Different from recent work on contrastive visual representation learning ( Chen et al. , 2020a ; b ) , we only apply brightness and contrast adjustments in color jittering , due to the monochrome nature of the medical images . For the text transformation function tu , we apply a simple uniform sampling of a sentence from the input document xu ( i.e. , x̃u is a randomly sampled sentence from xu for each minibatch ) . We did not use a more aggressive transformation mainly because sampling at the sentence level can preserve the semantic meaning of the sampled spans . 3 EXPERIMENTS . 3.1 DATA FOR PRETRAINING . We test our ConVIRT framework by pretraining two separate image encoders covering different medical specialties using two separate paired image-text datasets : • Chest image encoder : We use version 2 of the public MIMIC-CXR database ( Johnson et al. , 2019 ) , which is a collection of chest radiograph images paired with their textual reports , and since its release has become a standard resource for studying multi-modal modeling of medical images . After preprocessing , this dataset contains a total of about 217k image-text pairs , with each pair containing an average of 1.7 images and 6.0 sentences . • Bony image encoder : We obtain a collection of musculoskeletal image-text pairs from the Rhode Island Hospital system . Following chest images , musculoskeletal images constitute the second most common type of radiograph images in a typical hospital . This dataset contains a total of 48k image-text pairs , with each pair containing an average of 2.5 images and 8.0 sentences . We include model implementation and pretraining details in Appendix A .

Summary: In this work, the authors propose a new model, named ConVIRT to learn the medical visual representation from paired image and textual data in an unsupervised strategy. In ConVIRT, they mainly use a contrastive loss with two modalities (images and texts) as inputs to learn the representation. The experimental results show their proposed model achieve higher performance than other methods in image classification and zero-shot retrieval tasks.