Towards Medical Vision-Language Contrastive Pre-training via Study-Oriented Semantic Exploration

Authors: bo  liu
zexin  lu
yan  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:51:32

Abstract: Contrastive vision-language pre-training has shown great promise in representation transfer learning and cross-modality learning in the medical field. However, without fully exploiting the intrinsic properties and correlations of multimodal medical data within patient studies, current research fails to explore all the potential of available data, leading to suboptimal performance on representation learning. In this paper, we propose a novel pre-training framework for learning better medical vision-language embedding, oriented on patients' study-level data. Based on the order-agnostic property of radiology report, we adopt a two-stage feature extraction method for more representative textual characterization. Then, by leveraging momentum encoders and memory queues, study-level semantics are explored with three contrastive objectives to provide comprehensive supervision from three perspectives, i.e., cross-modal, multi-modal, and uni-modal, such that the potential information neglected by previous research can be fully exploited. The superiority of the proposed framework is demonstrated by the impressive improvements on four typical downstream tasks, including zero-shot/data-efficient image classification, image segmentation, and cross-modal retrieval.

Main paper: 
 introduction: 
 The recent success of deep learning relies heavily on the large amount of annotated data. However, acquiring a sucient amount of labeled data is particularly dicult in medical elds due to its expertise-demanding and time-consuming nature. To address this issue, many eorts [20, 28, 32, 51] have been made on vision-language contrastive pre-training to leverage the existing radiology reports.With the supervision of contrastive loss between images and their corresponding reports, the pre-trained network brings superior performance on many downstream tasks, including both multi-modal (e.g., cross-modal retrieval) and uni-modal tasks (e.g., image classication), demonstrating the eectiveness of contrastive pre-training in learning better representations on multi-modal data. However, despite the improvement they have achieved, current methods share three weaknesses, limiting their potential. (1) Less precise data structure: in current datasets, particularly regarding chest X-ray (CXR), a collection of images can correspond to a single report, which is referred to as a patient study as shown in Figure 1 (a). Most prior methods [28, 51] directly dissociate this study into several image-report pairs whose reports are the same, resulting in false negatives during cross-modal alignment; (2) Neglected semantic similarity: unlike natural image captions, patient studies of similar disease status share many semantic similarities in terms of both images and reports (cf. Figure 1(b )), which are not fully exploited by previous methods; and (3) Under-explored uni-modal information: most methods focus on leveraging the cross-modal contrast as the only source of supervision for representation learning, as shown in Figure 2 (a), neglecting the intrinsic information of each modality (Figure 2(c) ). To address the aforementioned issues, we propose a study-oriented semantic exploration (SENSE) framework to improve the medical vision-language joint embedding learning via three contrastive objectives. Firstly, unlike many previous studies [20, 28, 51] which directly apply the feature extraction methods of contrastive learning for general tasks to medical eld and neglect the intrinsic property of radiology reports, we adopt a two-step method for better textual feature extraction. As illustrated in Figure 4 , the semantics of each sentence in a radiology report is independent, and the overall semantics of the report is irrelevant to the order of sentences. Based on such observation, we propose to encode each sentence separately and fuse them with a Max-Max method to acquire better semantic representation. Then, the momentum encoders are introduced for each modality along with three studylevel supervisions (Figure 2 ), i.e., cross-modal, multi-modal, and uni-modal supervisions, to provide sucient guidance for better representation learning. In particular, (1) for conducting study-level cross-modal contrast (S-CMC) (Figure 2 (a)), we generate a single image-report pair for each patient study by randomly sampling one image (in case one study contains multiple ones) during each training iteration (shown in the left block of Figure 2 ). Then, normal instance-level contrast is performed. Moreover, with the momentum encoders and memory queues, S-CMC can sample negative pairs from the distribution of the whole dataset, eectively preventing representation collapse [16, 43] Actually, with the centroid of multi-modal embeddings as the comparison target, introduced S-MMC explicitly bridges S-CMC and S-UMC, not only further increasing the intra-modality similarities (i.e., image-to-image and report-to-report), but also encouraging the inter-modality similarities (i.e., image-to-report and report-toimage). To verify the eectiveness of the proposed method and its components, we conduct comprehensive experiments along with extensive ablation studies, covering single-modal (e.g., data-ecient image classication and segmentation) and multi-modal tasks (e.g., zero-shot image classication and cross-modal retrieval). The impressive improvements over state-of-the-art methods on all these tasks demonstrate the superiority of our framework.The contributions of this paper can be summarized as:• We propose a study-oriented contrastive learning framework to improve medical vision-language joint embeddings. By introducing three levels of contrastive supervision, i.e., cross-modal, multi-modal, and uni-modal, the semantics of patients' study-level data can be fully explored. • To better exploit the multi-modal semantic similarities between dierent studies, we propose to encode dierent semantic structures through K-means clustering and force the network to learn the similarities by pushing the representations toward their corresponding centroids. • For better characterizing the radiology reports, we propose a two-step textual feature extraction method. By rst encoding sentence-wise features separately and deriving report-level representations later, order-agnostic representations can be obtained. 
related work: 
 Medical Image-Report Joint Learning. In recent years, increasing studies focus on leveraging clinical reports as supervision signals to improve the learning of visual representations [5, 10, 19, 20, 28, 37, 39, 41, 44, 51] . Most methods can be categorized into two streams according to the architecture of the pre-training frameworks, i.e., encoder-decoder-based and dual-encoder-based. For the encoderdecoder-based methods [14, 44] a textual decoder to translate the visual features extracted by an image encoder into reports, with an objective to minimize the variation between the generated and true reports. For the dual-encoder-based paradigm [2, 20, 28, 32, 41, 51] , the pretraining objective is to align visual and textual features extracted by two separate encoders in the embedding space. Specically, ConVIRT [51] pulls the global embeddings of paired medical image and report closer while pushing those of non-paired ones away. Following ConVIRT, BioViL [2] improves the text modeling by adding clinical vocabulary and report-specic augmentation. GLoRIA [20] and LoVT [32] further propose to conduct contrastive learning on local representations of image-report pairs to capture the ne-grained alignment in addition to the global representations. Besides, SAT [28] and MGCA [41] take similarities between dierent image-report pairs into consideration. However, all these methods neglect the intrinsic properties of patients' study data. We propose a study-oriented pre-training framework to explore such information fully through three introduced objectives.Recently, large multimodal models [25, 48, 52] have achieved impressive performance, especially for medical tasks [21, 30, 50] . Our pre-trained visual extractor can be incorporated by these models to better extract visual features, improving the overall performance.Contrastive Self-supervised Learning. Self-supervised learning aims to endow deep neural networks with generic representations in an unsupervised manner, where contrastive learning [40] is one of the most representative paradigms. With the supervision of contrastive loss, better representation can be learned with less annotated data. For visual embedding learning, existing methods [6, 7, 16, 17, 29] utilize momentum encoder and various data augmentations for representation contrasting. In addition, to encode semantic relationships among images, [4, 27] propose to introduce semantics-wise contrast through clustering [3, 11] . For textual embedding learning, most eorts have been made to generate highly eective positive or negative samples, such as back-translation [13, 38] , token-level transformations [31, 42, 46] , and semantics-level augmentations [15] . Recently, multi-modal contrastive learning [23, 26, 33, 47, 49] draws increasing attention. By pulling the paired image-text closer and pushing the non-paired ones away, better performance can be achieved for not only cross-modal tasks (e.g., cross-modal retrieval), but also uni-modal ones (e.g., image classication). However, directly applying these methods to medical tasks can hardly achieve optimal performance due to their neglect of specic properties of typical medical vision-language data. In this paper, we design a radiology-specic pre-training framework that fully exploits the semantics of patients' study-level data toward better embedding for medical images and reports. 
methodology 3.1 pre-training framework overview: 
 The overview of the SENSE framework is shown in Figure 3 . Given an image-report pair (G E , G C ) sampled from a patient study (cf. Figure 2 ), we propose to use dual encoders for feature extraction with a visual encoder 5 + (•) for medical image G E and a textual encoder 5 ) (•) for radiology report G C , followed by a series of projectors to generate representations for dense semantic explorations, i.e., our proposed three contrastive objectives. Meanwhile, a momentum encoder is introduced in addition to each uni-modal encoder, and updated through a moving average strategy as in MoCo [17] . Particularly, the momentum image encoder is updated with\ 5 + U\ 5 + + (1 U)\ 5 +, where \ denotes model parameters, 5 + is momentum visual encoder, and U is a momentum coecient. There is no focal consolidation. The heart size is normal.There is no focal consolidation.No acute osseous abnormalities.s Report Encoder ! *No acute osseous abnormalities.The heart size is normal. There is no focal consolidation.Report " Figure 4 : The proposed Max-Max method for order-agnostic radiology report representation, motivated by the characteristic that the semantics of a report is usually invariant to the sentence order.Similarly, we denote momentum textual encoder as 5 ) (•). The advantage of incorporating such dual momentum encoders for medical vision-language contrastive learning will be elaborated in the following sections. After pre-training, the robust visual and textual encoders can be readily ne-tuned for downstream tasks, including uni-and multi-modal ones.For the input radiology image G E , a set of data augmentations, including resizing with random crop, horizontal ipping, and color jittering, which complies with the premise of preserving the semantic consistency within imagereport pairs, is applied to generate two semantically correlated views G 1 E and G 2 E . Following previous studies [2, 20, 51] , we utilize the ResNet-50 pre-trained on ImageNet (before logits layer) [18] as image encoder 5 + (•) (as well as momentum encoder 5 + (•)), followed by an adaptive average pooling layer outputting radiology imagefeatures v = 5 + (G 1 E ) (momentum features v = 5 + (G 2 E )).Distinct from natural image captions, the radiology report G C has specic intrinsic properties: it usually consists of several semantically independent sentences and its semantics is irrelevant to the order of sentences (Figure 4 ). Therefore, unlike prior study [2] which embraces it with a data augmentation method by randomly permuting each sentence, we propose a more powerful order-agnostic textual feature extraction method, named Max-Max. Specically, due to the presence of positional embedding in BERT-like language models, report representations gained from traditional methods [2, 20, 51] (i.e., encoding the whole report directly) are related to the order of sentences, which does not suit the radiology reports. To reduce the negative impact of sentence position feature for downstream tasks, such as cross-modal retrieval, we rst decompose the radiology report into sentences and then send each sentence into the encoder individually to extract sentence-level semantics without the aection of other sentences, keeping itself intact semantically. Here, we employ BioClinicalBERT [1] as the textual encoder 5 [20, 51] to extract the word-wise semantics. To obtain sentence-level representations ( ={B 1 , ..., B # }, where # is the number of sentences, we apply the rst max-pooling layer after 5 ) (•) to all word embeddings of each sentence. Then, we derive the report-level representation t = Max-Max(G C ) through the next max-pooling layer over (, getting salient features for better multimodal interaction. Similarly, we get the momentum textual features t by using our Max-Max method on momentum encoder 5 ) (•) with the same input G C . The detailed comparison between our Max-Max method and randomly sentence permutation proposed by [2] can be found in Section 4.2.1. After sampling one image-report pair per patient study, we can gather a total of # pairs corresponding to # studies in each iteration. Considering that paired medical image and report have the highest similarity, similar to [20, 51] , the S-CMC objective aims to maximize the alignment between the paired image and report (positive pair) while minimizing the unpaired ones (negative pairs) via contrastive loss [40] , as shown in Figure 2 (a). To address the limitation that prior studies only sample negative pairs from the current mini-batch with dozens of samples, we adopt the momentum encoder and memory queue [17, 26] to simulate the whole training dataset, which can eectively enlarge the scope of negative samples and prevent representation collapse [43] . Experimental demonstration and detailed discussion can be found in Section 4.2.1. Specically, we dene the radiology image feature v from the normal encoder as anchor and its paired textual feature t+ from the momentum encoder as positive. The & negative textual features { t j } & 9=1 are a dynamic set of momentum features maintained as a rst-in-rst-out (FIFO) memory queue. To achieve robust representation for downstream tasks, we follow [6] to conduct contrastive learning in the projection space, and formulate the image-to-report (82A ) training objective as:) (•) (momentum version 5 ) (•))EQUATIONwhere sim(E, Ĉ) = cos(? E 2A>. (E), ?C 2A>. ( Ĉ)) with two non-linear MLP projection heads ? E 2A>. (•) and ?C 2A>. (•), and cosine similarity score. g is a temperature hyper-parameter. In practice, we update the memory queue using the momentum textual features from former mini-batches [17] .Similarly, as the symmetrical term of Eq. 1, we can get the reportto-image (A 28) contrastive loss L A 28 where sim(C, Ê) = cos(? C 2A>. (C), ?E 2A>. ( Ê)) with two non-linear MLP projection heads ? C 2A>. (•) and ?E 2A>. (•) of the textual and the visual features, respectively. Thus our S-CMC loss (Figure 3 (a) ) can be formulated as:EQUATIONwhere a pair of report G C and image G E are aligned twice (i.e., 82A and A 28) in the presence of dierent augmentations, providing extra supervision.Unlike natural image-caption datasets, the variations between dierent patient studies can be rather small, especially for the radiology reports of subjects with the same disease status. However, former studies did not fully explore these semantic relationships. In this section, we propose a novel method to acquire inter-study semantics explicitly and take advantage of them for extra supervision.Semantic Structure Mining. Due to the diculty in nding direct metrics to measure similarities between study data, we propose to employ unsupervised clustering to learn the semantic structure. Specically, we rst derive informative representations of a patient study from its selected image-report pair by averaging the momentum features generated from the cross-modal projectors, i.e., m = ( ?E 2A>. ( v) + ?C 2A>. ( t))/2, which contains the intrinsic information of both the report and image. Then, we perform K-means clustering to cluster all patient studies in the training set into groups based on m (Figure 3(d) ). Note that we cluster the momentum features instead of the normal ones, as the former is expected to yield more consistent clusters during the training process for better stability. The clusters are updated once after each epoch of training.Semantic Structure Preservation. The images and reports belonging to the same cluster group share large semantic relevance, which can be exploited to learn better embeddings by forcing their representations close to the cluster centroid, as shown in Figure 2 (b) . Particularly, for each image, its feature v and the corresponding cluster centroid ĉ+ are adopted as a positive pair, while the combinations of v and the rest 1 cluster centroid {ĉ } are treated as negative pairs. For computational eciency, we only sample ! negative cluster centroid randomly for image-to-centroid (822) contrast in practice:EQUATIONwhere sim(E, 2) = cos(? E 2A>. (E), 2), and `is a cluster-specic scaling factor. If we use { <? } % ?=1 to represent the set of feature points in each cluster, and 2 + to denote its centroid, following [27] , the `of a cluster can be formulated as follows to re-balance each cluster:EQUATIONwhere n > 0. For a loose cluster, a large `will scale down the similarity between the images (reports) to its centroid (Eq. 3), pulling the samples belonging to the cluster closer. On the contrary, a small `will scale up similarity, preventing the feature embeddings from collapsing into a single point and losing semantic structure. Similarly, we have the text-to-centroid (C22) training objective L C 22 for the report feature t with sim(C, 2) = cos(? C 2A>. (C), 2). The multi-modal contrast L ( ""⇠ (Figure 3 (b) ) can be written as:L ( ""⇠ = L 822 + L C 22 .(5)Due to the limited amount of medical image-report data, eciently utilizing the available ones is critical for the quality of contrastive pre-training. To this end, we propose to introduce additional supervision signals within each modality via uni-modal contrast between the normal and the momentum encoders as shown in Figure 2 (c ).For visual modality, we follow [7] to align semantically correlated views G1 E and G 2 E , and the image-to-image (828) loss is written as:EQUATIONwhere sim(E, Ê) = cos(? E D=8. (E), ?E D=8. ( Ê)) with another two nonlinear MLP projection heads ? E D=8. (•) and ?E D=8. (•). Note that unlike [47] which directly aligns (E, Ê) from the projector heads (? E 2A>. (•), ?E 2A>. (•)) for cross-modal contrast, we introduce additional MLP projection heads for uni-modal alignment in a separate embedding space, avoiding interference with the cross-modal projection heads and preventing potential performance degradation on downstream cross-modal tasks. Naturally, we maintain another memory queue to store & unpaired visual momentum features.For textual modality, we do not directly apply augmentation to the input data like the common back translation [38] or EDA methods (random insertion, random deletion, etc.) [45] , due to the potential distortion of particular medical vocabularies, e.g., "cardiomediastinal silhouetter" is converted to "heart mediastinum silhouette" by back translation. Instead, we feed the same input G C to both normal and momentum encoders and conduct semantics-level augmentation by dropout mask [15] . Similar to Eq. 6, we can get our text-to-text (C2C) loss L C 2C with sim(C, Ĉ) = cos(? C D=8. (C), ?C D=8. ( Ĉ)). Finally, our S-UMC objective (Figure 3(c) ) can be obtained by:EQUATIONOverall, we formulate our study-oriented semantic exploration loss as:EQUATIONwhere _, V and W are loss balancing coecients. 
experiments 4.1 pre-training setup: 
 Dataset. We pretrain our framework on the training set of MIMIC-CXR (version 2) 1 [24] , the largest publicly available chest X-ray dataset to date. Following previous studies, we focus on the frontal view of MIMIC-CXR and leave out the other views in each patient study. Totally it contains about 146K studies for training, 1, 151 for validation, and 2, 210 for testing. In statistics, there are mainly 149,496 studies, 133,160 (89.07%) studies have only one front image, 15,596 studies have two front images (10.43%), and 740 (0.5%) studies have three or more front views. For each image, we resize the longer side to 256 pixels and use zero padding on the shorter side, making the nal image size of 256 ⇥ 256 pixels. For each radiology report, we only keep the Findings section which contains detailed descriptions of the corresponding image. We preprocess all the reports by dropping special characters and symbols, such as newlines, underscores, etc., and the subjects with reports of less than three words are excluded. Implementation Details. All the projection heads are 2-layer MLPs with ReLU activation, outputting 512-dimensional features. For Bio-ClinicalBERT, we freeze the parameters of the rst 6 layers during pre-training and only ne-tune the last 6 layers as [51] . We pretrain the whole framework for 100 epochs on 4 NVIDIA-A100 GPUs with a batch size of 128 and mixed-precision training. The network is warmed up in the rst 20 epochs with a linear learning rate from 2.5 ⇥ 10 6 to 5 ⇥ 10 4 , which further decays by cosine schedule in the remaining iterations. The optimizer is AdamW with a weight Table 1 : Results of cross-modal retrieval on the test set of MIMIC-CXR [24] . Recall@: (%) is used as the evaluation metrics. The best and second best results are bolded and underlined, respectively.Image-to-Report Retrieval Report-to-Image Retrieval R@1 R@5 R@10 R@1 R@5 R@10 VSE++ [12] 9 Note that in order to ensure semantic consistency between images and reports, all image transformations are mild and some popular augmentations are excluded due to the potential of causing semantic inconsistency, such as Gaussian blur and excessive ane transformations, which may obscure the appearance of lesions. In terms of the hyper-parameters, we set the momentum coecient U = 0.999 and temperature g = 0.1 following [17] , queue length & = 2048, cluster number = 10, 000, negative cluster centroid number ! = 2048, and loss balancing coecient_ = 1, V = 1,W = 0.2.For more details on implementation (even for downstream tasks), please refer to the Appendix.Comparison Methods. In this section, we compare the proposed framework with four state-of-the-art medical contrastive visionlanguage pre-training approaches, ConVIRT [51] , GLoRIA [20] , BioViL [2] , and SAT [28] , on four common downstream tasks. Con-VIRT is the rst study focusing on contrastive vision-language pre-training in the medical domain, which simply aligns global representations of paired images and reports. GLoRIA proposes to additionally align local representations of image-report pairs. Based on ConVIRT, BioViL improves text modeling by introducing radiology-specic text encoder and augmentations. Given the same dataset used for pre-training, we can directly compare our results to the ocial version. SAT removes possible falsely negative samples by calculating the cosine similarity between reports within the mini-batch. Most baseline results are cited from SAT unless otherwise specied. In addition, we also compare some other methods, which will be explained in corresponding subsections.4.2.1 Cross-modal Retrieval. This task consists of two symmetrical parts: Image-to-Report Retrieval and Report-to-Image Retrieval. Specically, given an image (report), the model attempts to identify the corresponding report (image). The pretrained normal image and report encoders with cross-projectors (? E 2A>. (E) and ? C 2A>. (C)) are directly used for cross-modal retrieval without further ne-tuning. We verify the performance on the ocial test set of Table 2 : Test AUROC score (%) of linear classication on CheXpert [22] and RSNA [35] datasets with dierent portions of training data. The baseline results are cited from SAT [28] and BioViL [2] . MIMIC-CXR [24] with a total of 2461 image-report pairs (i.e., there are multiple pairs with the same report) and use Recall@: as evaluation metrics, where : = 1, 5, 10. Besides, the result of VSE++ [12] is also presented for its outstanding performance on cross-modal retrieval task in the general domain.From the results in Table 1 , we have the following observations. (1) The rst variant of our method ('only L ( ⇠"⇠ (Max)') can be considered as an extended version of ConVIRT enhanced by the dual momentum encoders. Its superior performance to the original ConVIRT demonstrates the eectiveness of the dual momentum encoder architecture for medical contrastive vision-language pretraining. (2) The proposed Max-Max strategy can better extract textual representations for the retrieval tasks, leading to improved performance compared to single Max pooling and sequence exchange (denoted as Seq. Exc.) data augmentation [2] on all evaluations. (3) When further incorporating the semantic similarities through the clustering-based S-MMC, we achieve signicant improvements on both retrieval tasks (at least ⇠ 2.3% on all metrics), and substantially outperform all existing methods. (4) When lastly adding the uni-modal contrast to compose our full model, the optimal performance is achieved on both retrieval tasks and all metrics.4.2.2 Data-eicient Image Classification. To evaluate the data eciency of our framework, we pre-train the network with dierent amounts of data (i.e., 25%, 50%, 75%, and 100%). Following the classication protocol in previous studies, we freeze the weights of the pre-trained image encoder and ne-tune a randomly initialized linear classier with 1%, 10%, and 100% training data to evaluate the eectiveness of visual representation learning. We conduct classication on two datasets: (1) CheXpert is a multi-label chest X-ray dataset where each image is labeled based on the appearance of 14 kinds of disease symptoms. We follow the same experimental settings in [20, 28, 51] to use the validation set for evaluation because the original test set has not been made publicly available. Moreover, we randomly pick 5, 000 images from the training set for validation;(2) RSNA Pneumonia consists of two types of chest X-ray images, i.e., health and pneumonia. We use the same preprocessing procedure and experimental setting as in [20, 28] . Specically, about 30,000 front view images are split into training/validation/test sets with a ratio of 70%/15%/15%. For both datasets, we resize the image to 256 ⇥ 256 pixels The area under the receiver operating characteristic curve (AU-ROC) results are displayed in Table 2 . The results suggest that our method achieves state-of-the-art results on both datasets in most settings (only 0.1% lower than BioViL on RSNA with 1% data). Remarkably, with only 50% pre-training data, our method has already achieved comparable results with the previous best results except for BioViL on RSNA 1% data setting, demonstrating the excellent data eciency characteristics of our framework. Besides the comparison with methods in contrastive paradigm, we also compare two representative methods in masked pre-training, PTUnier [8] and M3AE [9] . It shows that our SENSE can achieve higher or comparable results. The reason why our SENSE is slightly lower than PTUnier [8] may be because PTUnier [8] used more pre-training data.Zero-shot Image Classification. Inspired by CLIP [33] , we treat the zero-shot image classication task as identifying the most similar description among a set of class-specic prompt descriptions for a given image. The pre-trained normal image and report encoders with cross-projectors (? E 2A>. (E) and ? C 2A>. (C)) are directly used for zero-shot image classication. Following the settings of BioViL [2] , which achieves the best performance among previous studies, we evaluate the model's zero-shot recognition ability on the test set of RSNA Pneumonia [35] , where class prompts for the healthy and unhealthy are set as "no evidence of pneumonia" and "ndings suggesting pneumonia", respectively. As presented in Table 3 , our method achieves the best results on both accuracy and F1 metrics, demonstrating that the learned joint embeddings are eective for zero-shot classication.Following SAT [28] , we utilize the pre-trained visual encoder as the encoder for U-Net [34] and ne-tune the whole network with the supervision of annotated pathological chest X-ray images under 1%, 10%, and 100% training data. The dataset used in this task is SIIM-ACR Pneumothorax [36] , which contains 12, 047 chest radiographs and is split into training/validation/test sets in a ratio of 70%/15%/15%.We report the Dice score for evaluation in Table 4 . The results show that the network pre-trained with our SENSE is more eective than the others in all settings, especially at the 1% and 10% training data settings, demonstrating the superiority of our pretraining framework.In this part, we conduct ablation studies and analyses to verify the eectiveness of each component of the framework. More experiments can be found in Appendix.Report Feature Extraction Methods. Because the output of the textual encoder (i.e., BioClinicalBERT [1] ) is a feature matrix consisting of the representations of all tokens, we evaluate the performance of dierent pooling methods to obtain report representations. To avoid the interference of other factors, we only use the L ( ⇠"⇠ objective here because it has the most direct impact on the cross-modal retrieval task. As displayed in Table 5 , max pooling can perform better than the others in the one-step method, and the proposed two-step method Max-Max achieves the best results on all three metrics.Objectives. As we have explored the impact of dierent pre-training objectives on cross-modal retrieval in Section 4.2.1, here we investigate the impact on the transfer learning task (e.g., image classication). As reported in Table 6 , both L ( * "⇠ and L ( ""⇠ could enhance visual representations, demonstrating that the introduced uni-modal and multi-modal supervisions can eectively supplement the cross-modal interactions. 2 When combining all of L ( ⇠"⇠ , L ( * "⇠ and L ( ""⇠ as our proposed SENSE, optimal performance can be achieved. 2 As the foundation for contrastive vision-language pre-training, the cross-modal contrast (L ( ⇠"⇠ ) cannot be ablated.To gain insight into the qualitative understanding of what our model has learned during training, particularly with the incorporation of L ( ""⇠ , we present pairs of images and corresponding reports from each patient study in MIMIC-CXR [24] , sampled from one cluster as depicted in Figure 5 . It can be seen that the images (reports) belonging to the same cluster present (describe) highly similar symptoms although they do not belong to the same patient study, demonstrating that our SENSE does capture meaningful semantics structure with the proposed study-level clustering. 
conclusion: 
 In this paper, we proposed a study-oriented pre-training framework to improve medical vision-language joint embeddings. Leveraging the order-agnostic property of radiology report, a two-step feature extraction method was adopted to obtain better texture representations. Moreover, through the momentum encoders and memory queues, three patients' study-level contrastive objectives, including uni-modal, cross-modal, and multi-modal were introduced to provide sucient guidance for semantics exploring from limited data. Extensive experiments on four downstream tasks, covering single-modal and multi-modal tasks, demonstrated the superiority of the proposed framework for better medical vision-language representation learning.Limitations and Future Work. This work mainly focuses on visual representation learning and ignores validating the eectiveness of textual ones, which can be seen as a limitation. More experiments about the performance of textual encoder on downstream tasks, such as report classication, will be conducted in future work. 
 
 Appendix: https://physionet.org/content/mimic-cxr/2.0.0/ 


Decision: All reviewers appreciate the novelty and technical contributions of this work. However, they point out that the experimental comparisons are insufficient and require further improvement. Based on the overall ratings, I suggest accepting this work. The authors are recommended to improve the manuscript based on the feedback before submitting the final version.

Reviews: review: paper_summary: paper_summary: This paper focuses on the current challenges faced by visual language models in the field of medicine. These challenges include the underutilization of the intrinsic properties and correlations present in multimodal medical data, as well as fully harnessing the potential of the available data. To overcome these challenges, the paper proposes a study-oriented contrastive learning framework. By incorporating three levels of contrastive supervision, including cross-modal, multi-modal, and uni-modal, the framework enables a comprehensive exploration of the semantics within patient research-level data. The proposed approach has been extensively validated across a wide range of downstream tasks, yielding promising results.
strength_weakness: strength_weakness: 1. The article focuses on special and specific challenges of the vision language model faced in the medical field: scarcity and unstructured data. The paper fully uses the medical data across three levels: cross-modal, multi-modal and uni-modal. 

2. Extensive experiments demonstrate that the proposed contrastive learning pre-training is effective on multiple downstream tasks, including retrieval, classification, zero-shot classification and segmentation. The experiments are detailed and sufficient.

3. The paper is well written and organized.
limitations: limitations: Related works on vision language large models in the medical field should be summarized, such as [1].

[1] @misc{yang2023enhancing,
    title={Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Models},
    author={Xiaoyu Yang and Lijian Xu and Hao Sun and Hongsheng Li and Shaoting Zhang},
    year={2023},
    eprint={2311.12327},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
paper_summary: paper_summary: This paper introduces a novel pre-training framework called SENSE, designed to enhance medical vision-language joint embeddings through study-oriented semantic exploration. It employs a two-step feature extraction method to better capture the intrinsic properties of radiology reports and leverages momentum encoders and memory queues to introduce three contrastive objectives at the study level—cross-modal, multi-modal, and uni-modal—to thoroughly exploit the semantic information within patient data.
strength_weakness: strength_weakness: This paper proposes a new method to fully explore the intrinsic properties and correlations of multimodal medical data in datasets, which to some extent solves the problem of poor representation learning performance.

This paper demonstrates the effectiveness of the proposed method through extensive and sufficient experiments on the MIMIC-CXR dataset and multiple downstream tasks.
limitations: limitations: This paper does not raise novel questions, and the contribution of the paper is not prominent enough. Moreover, the innovation of the proposed method is not strong enough, which is similar to the following works：
[1]Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning.
[2]GLoRIA- A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition.

This paper mentions one of the weaknesses of the current method in lines 90-96, which states that "in current datasets, particularly regarding chest X-ray (CXR), a collection of images can correspond to a single report". However, in chest X-ray datasets such as the MIMIC-CXR dataset, an imaging study of a patient generally includes two images: a frontal view and a lateral view, corresponding to one report. The author should explain this issue.

This paper only conducted experiments on the MIMIC-CXR dataset, and the author should conduct experiments on another common chest X-ray dataset, IU X-Ray, to verify the effectiveness and generalization of the proposed method.
paper_summary: paper_summary: The authors propose a study-oriented contrastive learning architecture to learn vision-language correspondence, which adopts multi-level contrastive learning methods to model the cross-modal alignment and boost multiple downstream tasks. Experimental results show the effectiveness of the model. However, there are some minor concerns.
strength_weakness: strength_weakness: This paper is well-written, with clear figures and descriptions that make it easy for readers to understand the ideas, methods, and results.
The authors have conducted necessary experiments to demonstrate the effectiveness of modeling medical vision-language alignment.
limitations: limitations: 1. In Section 3.2.2, using max pooling twice to learn report representations is quite confusing. Generally, pooling operation may lead to a significant loss of semantic information, such as sentence structure features. Additionally, it may also weaken the significance of key pathological words in sentences. These drawbacks seem contrary to the paper's motivation (e.g. learning intrinsic 
properties of medical data). Why not apply other methods to build report representations, such as a simple MLP?

2. Is the choice of loss coefficients in Eq.8 based on experience? It is recommended to include parameter experiments to quantitatively show how changes in the coefficients affect the final results. This would provide a more comprehensive understanding of the impact of each contrastive learning component.
paper_summary: paper_summary: The proposed paper introduces a novel pre-training framework aimed at enhancing medical vision-language embeddings by utilizing patient study-level data, addressing the limitations of current research in fully capitalizing on multimodal medical data. It employs a two-stage feature extraction process for textual data, momentum encoders, and memory queues to explore study-level semantics through three contrastive objectives: cross-modal, multi-modal, and uni-modal. Demonstrated improvements in downstream tasks like image classification, segmentation, and cross-modal retrieval, alongside comprehensive ablation studies, underscore the effectiveness of the framework, with plans to release the code and model upon acceptance.
strength_weakness: strength_weakness: Enhanced Data Utilization: The framework effectively leverages patient study-level data, which is a rich source of information often underutilized in existing models. By focusing on these comprehensive datasets, the model can capture deeper, more nuanced correlations and intrinsic properties across different modalities, leading to a more robust understanding and representation of medical data.

Innovative Feature Extraction and Learning Techniques: Utilizing a two-stage feature extraction method acknowledges and harnesses the order-agnostic nature of radiology reports for more accurate textual characterization. Additionally, the integration of momentum encoders and memory queues to explore study-level semantics through diverse contrastive objectives (cross-modal, multi-modal, and uni-modal) enables the model to capture a broad spectrum of data characteristics and relationships, enhancing the learning process.

Proven Effectiveness and Comprehensive Evaluation: The framework's superiority is demonstrated through significant performance improvements across multiple downstream tasks, including zero-shot/data-efficient image classification, image segmentation, and cross-modal retrieval. This broad array of tasks showcases the model's versatility and effectiveness in practical applications. Furthermore, the inclusion of comprehensive ablation studies provides a thorough validation of each component's contribution to the overall performance, emphasizing the thoughtfulness and rigor of the research design.
limitations: limitations: Insufficient Comparison with Relevant Literature: The paper does not sufficiently compare its approach with other significant works in the domain of medical vision-and-language pre-training. Specifically, it lacks a comparative analysis with recent and relevant methodologies such as "Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training," "Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts," and "Mapping medical image-text to a joint space via masked modeling." This omission is notable because these works also explore innovative ways to integrate and leverage multimodal data for improved representation learning.