Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning
Authors: Fuying Wang
Yuyin Zhou
Shujun Wang
Varut Vardhanabhuti
Lequan Yu
Venue: NeurIPS 2022 Accept (2022)
Keywords: medical image
medical report
cross-modal
representation learning

Abstract: Learning medical visual representations directly from paired radiology reports has become an emerging topic in representation learning. However, existing medical image-text joint learning methods are limited by instance or local supervision analysis, ignoring disease-level semantic correspondences. In this paper, we present a novel Multi-Granularity Cross-modal Alignment (MGCA) framework for generalized medical visual representation learning by harnessing the naturally exhibited semantic correspondences between medical image and radiology reports at three different levels, i.e., pathological region-level, instance-level, and disease-level. Specifically, we first incorporate the instance-wise alignment module by maximizing the agreement between image-report pairs. Further, for token-wise alignment, we introduce a bidirectional cross-attention strategy to explicitly learn the matching between fine-grained visual tokens and text tokens, followed by contrastive learning to align them. More important, to leverage the high-level inter-subject relationship semantic (e.g., disease) correspondences, we design a novel cross-modal disease-level alignment paradigm to enforce the cross-modal cluster assignment consistency. Extensive experimental results on seven downstream medical image datasets covering image classification, object detection, and semantic segmentation tasks demonstrate the stable and superior performance of our framework.

Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning
Fuying Wang', Yuyin Zhou?, Shujun Wang?, Varut Vardhanabhuti' Lequan Yu!
The University of Hong Kong UUniversity of California, Santa Cruz UUiversity of Cambridge
ffuyingwcconnect.. varvê, lqyug)-hku. hk
yzhou2840ucac..er sw9910cam ac.
Abstract
Learning medical visual representations directly from paired radiology reports has become an emerging topic in representation learning. However, existing medical image-text joint learning methods limited by instance local supervision anal- ysis, ignoring disease-level semantic correspondences. In this paper, we present a Pross-modal Alignment (MGCA) framework for general- ized medical visual representation learning by harnessing the naturally exhibited semantic correspondences between medical image and radiology reports at three different levels, i.e., pathological region-level, instance-level, and disease-level. Specifically, we first incorporate the instance-wist wise alignment module by maximiz- ing the agreement between image-report pairs. Further, for token-wise wise alignment, we introduce a bidirectional cross-attention strategy to explicitly learn the match- ing between fine-grained visual tokens and text tokens, followed by contrastive learning to align them. More important, to leverage the high-level inter-subject re- lationship semantic (e.g., disease) correspondences, disease-level alignment paradigm to enforce the cross cross-modal cluster assignment consistency. Extensive experimental results seven downstream medical image datasets covering image classification, object detection, and semantic segmentation tasks demonstrate the stable and superior performance of framework.
novel Multi Multi-Granularity
design novel cross modal
Introduction
In recent decades, deep learning techniques have significantly advanced medical image understanding when large-scale labeled datasets are available [46] 20
However, assembling such large annotated data is expensive and time-consuming. As an alternative, learning directly from radiology reports accompanied by medical images becomes mainstream without any extra manual annotation, which aims learn general medical vision representations from physicians' detailed medical records and then transfer the learned representations to downstream tasks. In the previous literature, image-tett contrastive learning has achieved huge success for wide range of medical by predicting which radiology report goes with which medical image. Considering one limitation that pathologies usually occupy small part of the whole image, Huang al. (27] proposed attention- based contrastive learning strategy to learn local representations, which capture ine-grained semanties in medical images to facilitate localized downstream tasks, e.g., medical object detection and medical semantic segmentation.
14
44
downstream tasks 61 27 62
As we can see in Figure [] medical images and radiology reports naturally exhibit multi-granularity semantic correspondence at different levels, e.g., disease-level, instance-level, and pathological region- level. However, existing image-text joint learning methods are limited to explore correspondence
*Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Discase level
Inrrase mid
MM Mitecle
Radlggaph
Pensstent
Candonsaahy was nid
Raskgroph orttonepllyy now
Positen: cadoompply
Candommpls was mild
card andiiss
cantomepply
Patholopiaa Instance-enn eeveeeee
Radingmap cedtommppty nas bycopanicalyy
Persieent cartoreeply
Feternio conssicatioon seydlped....
Small pierral effuzion
pulmorarr.... Moderate left pleural effusion
Rapidly developing
Cardooesinn wai Modorate lef pleural effision...
conseliatioon
Left lower stelectssis..
region-levee
Perestent audoamera
Persident exrtooreply
Periitere cediomeggly
Perditent cantooeggyy pleund effarion.. Rapidly ooing preas
ploutel
gleant aftaamn
Small murn ettudon
developing
Rapidly seveloping
Rapidly developing concoddarios Left lower stelectads..
Left lefilowerion lobe stelectads
lower stelectads
Figure 1: The multi gratiiiganllarttyy (disease-level, instance-level, and pathological region- -eveel) semantic correspondences across medical images and radiology reports.
supervision from only part of levels 4 medical image/report information and insufficient representation capability models. Thus, how to effectively leverage this intrinsic multi granularity cross- modsll correspondences between medical images and radiology reports from all three levels to enhance medical visual representation remains an open question.
24 61
42 leading inefficient usage of the valuable
In this paper, we present a novel Multi Guati-Granullrity Cross-modal Alignment (MGCA) framework to seamlessly leverage the multi-granula semantic correspondences between medical images and radiology reports for generalized medical visual representation learning. Specifically, we incorporate well-known instance-wise alignment, fine-grained token-wine wise alignment, and disease-level alignment via contrastive learning to enhance the generalizability of learned visual representations. Particularly, for token- wise alignment, introduce effective bidirectional cross-attention strategy explicitly cross cross-moda modal contrastive learning scheme to improve the sensitivity of local features. For discase-level alignment, we leverage the inter-subject relationship correspondence between medical images and radiology reports by enforcing the cross-modal cluster assignment consistency. By exploiting multi-granular cross-modal correspondences from three aspects, our MGCA framework has the ability to boost downstream tasks at both image and pixel levels, where only limited annotated data are required. We pre-train our MGCA framework with a large-scale medical image and report dataset, i.e., MIMIC- CXR, and then validate our learned medical visual representations with seven downstream datasets, belonging to three medical tasks: image classification, object detection, and semantic segmentation. Experimental results demonstrate that our model achieves stable superior transfer performance even training with 1% of training data, when compared with existing satte-of-the-arn medical image-text pre-training methods. Our code is in httph::///// //gi tiut....c.. com wwwp//gywagn//MCC..
learn the soft matching between local image-text representations, and then adopt
Related Work
mainstream paradigms to learn medical visual representations from report supervision. The first is to extract disease labels from radiology reports via human-designed rules 22 31 55 and then pre-train an image model for downstream tasks. However, defining such rules requires a lot of human labor and expert knowledge. Also, the trained models may be suboptimal, as the extracted labels are usually noisy 555 The second focuses on leveraging languallagguage contrastive learning to pre-train image and text encoders in an unsupervised manner [6] 27 42 62]. Supervised by naturally occurring of medical images and radiology reports, these methods demonstrate remarkable performance in various medical image downstream tasks (e.g., image classification [6]] 27), semantic segmentation (27), image-image retrieval [61]. image-text retrieval [1 27). However, these methods only utilize partial correspondence supervision of cross-modal semanties, leading to suboptimal performance for downstream tasks.
Learning Medical Visual Representations from Reports There

---

Image Encoder
Protetype
CTA
Kiy
Vilue
representation
X
embecding
Attention
Alig
Ooyst
X
prodiction
Regorts
badlar ticcuitto...
Querr
Algg
X8
Cross Attention
Simiurity matro
Incress wowiiidd.....
Valuor
let besler atolctaaii
representation
Clstering
Words
2
Nil indestads
Text Encoder
ClRc Crose allantion busad lobon-vese diggnann ITA reconci-atte Image sigrmert
Incwane Kil stelccteds
proteape diggererd
Figure 2: Illustration of our proposed multi- gratilarrtu cross cross-modal alignment frammwork. CTA. ITA, and CPA represent token-win wisee alignment, instance-wise alignment, and prototype (disease)-level alignment respectively. The green arrow represents information flow of visual features, while the purple arrow represents information flow textural features.
aims learn embedding space where positive instances stay close to each other, while negative pairs are far apart. One of the key technique is find effective positive and negative pairs. To improve the efficiency of contrastive learning, some recent works proposed to predict one view's representations from another view [4] 19]. Moreover, brought the power of contrastive learning into medical image domain and have achieved substantial performance. We refer the readers to the survey [32] 30 for more details. Recently, some prototypical contrast learning approaches were proposed exploit the prototype-
Contrastive Learning Contrastive learning [22 4 24
23 49
semantics in 54 21 proposed contrasting instance features with its paired prototype features, while (77 2 proposed clustering-basee based methods contrast prototype-protote pairs. Different from these methods working on a single modality, we propose a cross-modal prototype alignment module to exploit the cross-modal prototype-level semantic consistency.
the dataset. For example, 335
Unsupervised Learning for Dense Prediction Learning fineggrained semantic correspondence
essential for dense prediction tasks, e.g., object detection [15 38), semantic segmentation (6].
etc. Recent studies have proposed various paradigms to tackle this problem [33] 28
36
II 59]. How-
most of these methods require pre-tratee trained object detector generate proposals of interested objects. Dense contrastive learning is thus proposed to learn fine fine-graine grained visual representations without pre-trained object detector [58 57 56 3) By optimizing contrastive loss pixel-level (or region-level) between two views of input images, these methods effectively learn the correspondence between local features, which significantly benefits dense prediction tasks.
Method
3.1 Overview
We aim to learn generalized medical image representations from radiology reports to benefit various downstream medical image recognition tasks where annotated data limited. Given training set of N image-report pairs D = {(F,,I, Ir,1,, (Iv,2, 22,,),.... ..., (Iv, (r), }. we utilize an image encoder fo (e.g., ViT [2]] and text encoder Ja (e.g., BERT (T0) to map D into the latent space E = [(V1. t1), (V2, t2).... ..., WN,,t)) where = ( t, = f((),.). In detail, for the i-th image-report pair (Ie,s Zt,i), the image encoder generates sequence encoded visual tokens R; [x], (r r and global image representation v,. Similarly, the encoder generates sequence of encoded text tokens Z; [z] 2, .... 2/¹) and global report representation t,, Here, S and L denote the total number visual tokens and text tokens, respectively.
As illustrated in Figure we design novel multi-granularity cross- mooss alignment framework for representation learning by exploiting the naturally exhibited multi-granni -uati-granularity cross-modal correspon- dences, i.e., discase (ereellype)-level, instance-level, and pathology region (token)-level between

---

images and reports. Specifically, we incorporate an instance-wiss image-text alignment (ITA) module retain the cross-modal smoothness by maximizing the agreement between true image-report pairs (vi, t2) versus random pairs. Meanwhile, to harness the benefits of naturally existing fine-grained cor- respondence between visual and text tokens, we also introduce bidirectional cross-attention-bant token-wis wise alignment (CTA) module to learn the soft matching between the visual token sequence R2 and text token sequence Z, as well as align them via the contrastive learning. More important, we design novel cross-modal prototype alignment (CPA) module to benefit the high-level semantic (e.g, disease) understanding by capturing the inter-subject relationship correspondence between two modalities.
3.2 Multi-granularity Cross-modal Alignment
Instance-wise Cross-modal Alignment As the core workhorses in our framework, we incorporate an instance-wise Image-Text Alignment (ITA) module to encourage the framework to map correct image-report pairs nearby in the latent space, while mapping random pairs far apart. Specifically, following common practice 4 first use two non- non-linear projection layers (g, and gi) to transform v; and t; into normalized dower-dimensional embeddings V; E IR and t, E R2, respectively. Then, the cosine similarity of i-th image-report pair is denoted
sam 21,3) = Vi t,, where v, go (v.). = (t.)..
For i-th image-report pair ( ft,)) in minibatch, regard two modality data queries and keys alternatively to learn the correct image-report pairings. This results in two symmetric temperature- normalized InfoNCE (43) losses (image-to-text contrastive loss and text-to-image contrastive loss) to maximally preserve the mutual information between the true pairs in latent space:
exp(sim(zus:
2 = log exp(sim(zr,, To,.)
2: = log
(2)
Σ Dk=1 Ek=1esp exp(sim(Zu.t ((x)T))
Ek=1 exp(sim((x..... (21,, 2/T1)
where B is the batch size and Ti is the instance-level temperature hyperparameter. The overall objective of our instancewwise moosl alignment is the average of the two losses:
CITA
E + 122))
(3)
2N
where N is the total number of image-report pairs.
Token-wise Cross-modal Alignment Fine-grained information is more significant in medical field: pathologies only occupy small portion of the whole image and only few disease tags in report depict the crucial medical condition. Considering that these important subtle clues are likely to be ignored when optimizing the global instance-we wise representations, bidirectional cooss-attention-base based Token-wise Alignment (CTA) module to explicitly match and align the moosl local presentations between medical images and radiology reports.
introduce effective
Specifically, for the i-th image-report pair (Io,5, It,,), the generated visual and text token embeddings will first be projected into normalized liwer-dimensional embeddings, which results in R, = (F), F?,,.., wise alignment, we need find the matching between visual and text tokens. Instead of directly computing the cosine similarity of different tokens (27). we propose to calculate the soft matching
....,, F and Z, = {z] zZ,.., ) where i; E IRd, 2, E Rd, In order to conduct the token-
between generated visual and text tokens with the attsstattnnionn mechanism [52 5 41 Formally, for the j-th visual token embedding i? i-th image-report pair, we let iي attend to all text token embeddings in Z, and then calculate its corresponded cross-modal text embedding o,
O 22 (VZ))), a²2h softmax (QP)T((KE)) Vd
(4)
where Q E Rddd - K E RRdxd V E Radd are learnable matrices. After that, we adopt a Local Image-to-text Alignment (LIA) loss LLAA to pull P close to its cross-modal text embedding of but push E. away from other cross-modal text embeddings, which maximizes the lower bound on

---

local cross-modies modal mutual information within each image-report pair [43]. Considering that different visual tokens have various importance (e.g., visual tokens containing pathologies are obviously more important than those with irrelevant information), we further assign weight w the j-th visual token when calculating the LIA loss. So the LIA loss can be formulated as:
ii (log exp(sim(f,,
+ log exp(sim(o), P.)T/2) 0²; Fx) (22)
LIAA 2NS
> k= exp(sim(T) (f/ T22
> k= exp(sim(
(5)
where T2 is the token-level temperature hyperparameter. Similar with the instance-level alignment, we also employ two symmetric InfoNCE losses by taking the visual token embedding i and its cross-modal text embedding of as queries, respectively. Note that set w! as the last-layer attention weight from j-th visual token to [CLS] token averaged across multiple heads. Similarly, for j-th text token embedding z/ in i-th image-report pair, we also calculate a cross moosll image embedding of with the same manner and construct Local Text-to-imat image alignment (LTA) loss LITA by contrasting 2) with oz. The final objective of our CTA module is the combination of LJA and L7A losses:
COTA - (CLIA + LLTA).
Discussion Note that our token-wise alignment is different from the local contrastive loss in 277. Our module explicitly contrasts similarities between local tokens to maximize their mutual information and calculates per-token InfoNCE, whereas the local contrastive loss in 27 contrasts aggregated instance-level similarities and still calculates per- per-instance InfoNCE.
Disease-level Cross-modal Alignment Both ITA and CTA treat two samples as a negative pair as long as they are from different instances. Many pairs sharing the similar high-level semantics (e.g., disease) are undesirably pushed apart in the embedding space. Therefore, design a novel Cross- modal Prototype Alignment (CPA) module to harness the mooss- inter-subject correspondences between medical images and reports.
For each image-report embedding pair (V;, [2) in ( we employ the iterative Sinkhorn Knopp clustering algorithm [8] to acquire two soft cluster assignment codes qu E IR and qr E R by individually assigning v; and É, into K clusters. Meanwhile,
also pre-define K trainable cross- modal prototypes as = {C1, CK ), where C₁ E Rd. After that, we calculate the visual softmax probability Р., R of the cosine similarities between v; and all cross-modal prototypes in C, and the text softmax probability Pt.: E RRK of the cosine similarities between é; and all cross-modal prototypes in C,
(k) exp(vF Ck/TTa) Po.i Ex exp(v; (xppvi'ce/Ta) c/TTa))
(k) exp(if'cu cu/Ta) P ΣN exp(t? ck//T)/
where 73 the prototype-level temperature parameter and (k) indicates the k-th element of the vector. The cross-modal disease- (i.e., prototype) alignment is achieved by conducting cross-modal prediction and optimizing the following two cross-entropy losses:
K
K Σ
f(V.. qr.:)
Σ
(k) di, log Pr.i l(t,, qo
(k)
(h) a log Pr.
(8)
Here, the cross-modal prediction is implemented by taking the soft text cluster assignment code qr, as "pscudo-label" to train the image representation v: and taking the soft image cluster assignment code qu as "pseudo-label" to train the report representation f, Finally, the overall CPA loss the average of two prediction losses over all the image-report pairs:
LCPA 2N
1
Σ
((Vi. qu.) qus)).
5

---

3.3 Overall Objective
We train our MGCA framework with jointly optimizing the three cross-moos poss--modal alignment modules, encouraging the network to learn discriminative and generalizable medical image representation. The overall training objective can be represented as:
C = λ LITA + X2 LCTA + À3 * LCPA,
(10)
where 11, X2. and A hyperparameters to balance three-level cross- moss- alignments.
4 Experiments
We pre-train our MGCA framework on a large-scale medical image-report dataset and then evaluate the effectiveness of learned medical image representations on seven datasets from three important downstream tasks in medical imaging. In the following subsections, we first introduce the exper- imental setup of pre-training in Section 4.1 and three downstream tasks in Section [.22 Then, we compare our proposed framework with state-of-ther of-ther medical image-text pre-training methods and show the comparison results in Section 4.3 4.5 Finally, we analyze our framework in Section 4.6 More analysis results can be found in the Appendix.
4.1 Pre-Training Setup
Dataset We pre-train our MGCA framework on the JPG version of MIMIC-CXR 2.0.0 dataset B1] We follow [6]] to preprocess the dataset. We remove all lateral views from the dataset, the downstream datasets only contain frontal-view chest images. Also, we extract the impression and finding sections from free-text reports obtain detailed descriptions of medical diseases and remove reports which are empty or have less than 3 tokens, resulting in roughly 217k image-text pairs.
Implementation Details Following 22 we use BioClinicalBERT [1] as the text encoder. We choose ViT-B/16 [22 as the image encoder backbone by default for unified modal architecture design. It is worth noting that our framework is model-agnostic to the image encoder backbone and we also report the results with ResNet50 [25] as image encoder backbone for fair comparison with other methods. We train framework 50 epochs on 2 pieces of RTX 3090 GPUs with batch size of 144. The optimizer is AdamW [40] with learning rate of 2e - and weight decay of 0.05. We use linear warmup with cosine annealing scheduler (39]. We initialize learning
1e 8 and warmup epoch as 20. Following the practice in contrastive learning 44 24), the dimension = 128 and the temperature hyperparameters are T1 0.1, T2 0.07, T3 0.2. The number of prototypes is K 500. We set A = 1, A = 1, Ag = 1. More pre-training details can be found in the Appendix.
4.2 Downstream Tasks and Experimental Setup
Medical Image Classification We conduct medical image classification on three representative
datasets: (I) CheXpert (29], which contains 191, 229 frontal chest radiographs. The task to classify each image into 5 individual binary labels: atelectasis, cardiomegaly, consolidation, edema, and pleural effusion. Following 61 27), we hold out the expert-labeled validation set as test data and randomly select 5, 000 radiographs from training data for validation. (2) RSNA Pneumonia [47]. We use the stage 2 version, which contains around 29, 700 frontal view chest radiographs. The task is binary classification, i.e., classifying each chest image into normal or pneumothorax positive. Following (27), we manually split the dataset into training, validation, and test set with 10%/15%/T 915% ratio. (3) COVIDx [53], which contains over 301: CXR images from a multinational cohort of over 16, 600 patients. This dataset contains 16, 490 positive COVID- 19 images from over 2, 800 patients. We use the latest version 6 of this dataset. The task is a three-class classification, i.e., classifying each radiograph into COVID-19 199, non- con-COVID pneumonia or normal. We use the original validation dataset as test data and manually split 10% of original training set for validation.
Following the previous work 27 we use the Linear Classification setting to evaluate the transferabil- ity of our pre-trained image encoder, i.e., freezing the pre- pre-trained ViT/ResNet-50 image encoder and only training randomly initialized linear classification head for the downstream classification task. Also, we evaluate our model with 1 1% 10%, and 100% training data on each classification dataset to further verify the data efficiency of our method. We report area under the ROC curve (AUROC) on CheXpert and RSNA and acuracy (ACC) on CO as the evaluation metric following [61].

---

Table Linear classification results on CheXpert, RSNA and COVIDx with 1%, 10%. 100% training data. Area under ROC curve (AUROC [%]] are reported for CheXpert and RSNA dataset, and accuracy (ACC [%]) l) is reported for COVIDx dataset. The best and second-best results are highlighted in red and blue, respectively.
CheXpert (AUC)
RSNA (AUC)
COVIDx (ACC)
Method
1%
10%
100%
1%
10%
100%
1%
10%
100%
Random Init
56.1
62.6
65.7
58.9
69.4
74.1
50.5
60.3
70.0
ImageNet Init
74.4
79.7
81.4
74.9
74.5
76.3
64.8
78.8
86.3
pre-trained on CheXpert
DSVE
13
50.1
51.0
51.5
49.7
52.
57.8
VSE++ [6
50.3
51.2
52.4
49.4
57.2
67.9
GLoRIA (27
86.6
87.8
88.1
86.1
88.0
88.6
67.3
77.8
89.0
pre-trained on MIMIC-CN -XR
Caption-Transfone
7
77.2
82.6
83.9
Caption- LSp 660
85.2
85.3
86.2
Contrastive-Binn Binary (50 48
84.5
85.6
85.8
CORVIRT 66
85.9
86.8
87.3
77.4
80.1
81.3
72.5
82.5
92.0
GLoRIA-Mo MIMII 27
87.1
88.7
88.0
87.0
89.4
90.2
66.5
80.5
88.8
MGCA(Ourr, ResNet-50)
87.6
88.0
88.2
88.6
89.1
89.9
72.0
83.5
90.5
MGrs(Ours, ViT-B/16)
88.8
89.1
89.7
89.1
89.9
90.8
74.8
84.8
92.3
Table 2: Object detection results (mAP [%]]
RSNA and Object CXR. Each dataset is fine-tuned with 1%, 10%, 100% training data. Best results are in boldface. "_" means mAP is smaller than 1%.
RSNA
Object CXR
Method
1%
10%
100%
1%
10%
100%
Random
1.00
4.00
8.90
0.49
4.40
ImageNet
3.60
8.00
15.7
2.90
8.30
VIRT [61]
8.20
15.6
17.9
8.60
15.9
GLoRIA [27
9.80
14.8
18.8
10.6
15.6
GLoRIA-MING 27
11.6
16.1
24.8
8.90
16.6
MGCA (Ours)
12.9
16.8
24.9
12.1
19.2
Medical Object Detection We evaluate the localized performance of pre-trained image encoder on two object detection tasks: (1) RNSA Pneumonia [47 contains 29700 frontal view radiograph.
The task is to predict bounding boxes indicating evidence of pncumonia. We randomly split the original training set into 16, 010/5, 337 337/5, 337 for traiming/aaiddationnessinng (2) Object CXR [26) contains 9, 000 frontal-view chest X-rays with detection targets for foreign objects. We use the original development set as test set (1, 000) and randomly split the original training set into training (6, 400) and validation (1, 600) sets.
We evaluate the detection performance by YOLOv3 45 frozen setting, i.e., using the pre-trained ResNet-50 image encoder as frozen backbone of YOLOv3 model and only fine-tuning the non- backbone layers. Similarly, we fine-tune the model by 1%. 10% and 100% training data evaluate the data efficiency. Mean Average Precisions (mAP) are reported evaluation metric with IOU thresholds 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75.
Medical Semantic Segmentation We also evaluate the performance of our framework for medical semantic segmentation SIIM and RNSA datasets: (1) SIIM Pneumothorax (18) dataset contains 12047 chest radiographs with manually annotated segmentation mask of pneumothorax. Follow- ing [27), train/validation/tes split respectively constitutes 70%/30%\30% of original dataset. (2) RNSA Pneumonia [77 is with the same split protocol as object detection task. We convert object detection ground truths into masks for semantic segmentation.
we evaluate segmentation performance by U-Net [46 fine-tuning protocol. We use the pre-trained ResNet-50 image encoder frozen encoder backbone of U-Net and train the decoder portion using 1%, 10% and 100% training data. Dice scores are reported evaluate the segmentation performance. The other downstream experimental setup be found in the Appendix.
Following [27]

---

opaciies
pneumothorax
cardomegaly
atel eciasis
Figure Visualization of learned token cor- respondence by our MGCA. Highlighted pix- els represent higher activation weights by corre- sponding word.
Figure 4: t-SNE visualizations encoded image representations. Colors indicate the ground truth disease types and cluster assignment left and right sub-figures.
Table Semantic segmentation results (Dice [%] on SIMM and RSNA. Each dataset is fine-tuned with 1%, 10%, 100% training data. Best results of each setting in boldface.
SIIM
RSNA
Method
1%
10%
100%
1%
10%
100%
Random
9.00
28.6
54.3
6.90
10.6
18.5
ImageNet
10.2
35.5
63.5
34.8
39.9
64.0
25.0
43.2
59.9
55.0
67.4
67.5
GLORIA 27
35.8
46.9
63.4
59.3
67.5
67.8
GLORIA-MIMIC 27
37.4
57.1
64.0
60.3
68.7
68.3
MGCA (Ours)
49.7
59.3
64.2
63.0
68.3
69.8
4.3 Results
Results on Medical Image Classification Table reports the results on three classification tasks. The results of other methods CheXpert and RSNA from original paper According the pre-training dataset, we group the existing pre-training methods into two categories: : pre-truined
MIMIC-CXR. As GLoRIA only reports results with pre-training CheXpert and pre-trained on Cperpert dataset, we also reimplement their method with pre-training on MIMIC-CXP dataset ALLRIA-MIMICE for fair comparison. It is observed that our MGCA with VIT-B/16 backbone shows the best performance in all nine settings, outperforming state-of-the-an GLoRIA [27] and Con VIRT 61]. With the same Re image encoder backbone, our framework also achieves second-best performance on four settings and competitive performance on the remaining five settings, showing the effectiveness of our framework. When fine-tuning with 1% proportion of data, our MGCA with ViT-B/16 backbone outperforms GLoRIA-MING
with 1.7% AUROC on CheXpert, 2.1 111 AUROC on RSNA dataset and 8.3% ACC on COVIDx dataset, showing larger improvement than other methods and also indicating the data efficiency of our method.
Results Medical Object Detection reblerrsports the object detection performance RSNA and Object CXR datasets. All methods adopt the same SaaNet-50-YYLOv architecture. It is observed that under each setting, our MGCA outperforms Con GLoRIA, and GLORIA-MIMIC by large margin. Importantly, our model shows superior detection performance when fine-tuning on 1% training data, indicating that multi teatti-graularttyy semantic alignment benefits the image encoder to learn more discriminative localized representations.
Results Medical Semantic Segmentation Table 3 shows the semantic segmentation performance on SIIM and RSNA datasets with the same ResNet50-U-Net architecture. Compared with GLoRIA, GLoRIA-MINE -MMIC. and Con VIRT, our MGCA shows higher dice scores five six settings. When training with 1% portion of data, our MGCA achieves 12.3% and 2.7% Dice improvement than GLORIA-MIMIC on SIIM and RSNA segmentation tasks, respectively. This comparison further validates the data efficiency of our method when transferring into dense prediction tasks.
"The results of Con VIRT 61 on RSNA dataset are from the reimplemented results in dataset are updated by the organizer.
27 as the RSNA

---

Table 4: Ablation study of our framework on linear classification (CheXpert and RSNA) and semantic segmentation (SIIM) settings. We report Area under ROC curve (AUROC [%]] on CheXpert and RSNA datasets, and (Diee [%]) on SIMM dataset. Best results of each setting are in boldface.
Training tasks
CheXpert (AUC)
RSNA (AUC)
SIIM (Dice)
ITA
CTA CPA
1%
10%
100%
1%
10%
100%
1%
10%
100%
87.6
88.2
88.5
88.4
89.5
90.5
25.0
43.2
59,9
88.3
88.9
89.1
88.9
89.8
90.7
47.6
54.4
61.3
/
88.5
88.9
89.0
88.6
89.2
90.4
37.4
46.7
55.0
88.8
89.1
89.7
89.1
89.9
90.8
49.7
59.3
64.2
Table Results of natural VLP pre-trained models on linear classification setting.
CheXpert (AUC)
RSNA (AUC)
1%
10%
100%
1%
10%
100%
BLIP 34
69.1
74.9
77.7
53.7
82.0
84.1
MGCA (Ours)
88.8
89.1
89.7
89.1
89.9
90.8
4.4 Analysis of Our Framework
Visualization To better understand the behaviour of our MGCA framework, we visualize the learned local correspondence of radiographs and medical reports in Figure Our MGCA learns meaningful local correspondence between visual tokens and text tokens, which is helpful for the local discriminative feature learning. Moreover, we select 1600 medical images, each with one excluded abnormality, and present t-SNE plots [51] in Figure 4 to visualize image embeddings. The colors represent the ground truth and cluster assignment left and right sub-figures. observed that our multi-modal prototypes can learn reasonable disease-level semantic information.
Ablation Study of Component Design Table 4 shows the ablation study results on two settings: medical image classification on CheXpert and RSNA datasets with ViT-B/16 as the image encoder backbone, and medical semantic segmentation on SIIM dataset with ResNet50 as the image encoder backbone. It is observed that CTA and CPA modules can both improve the classification performance, indicating that token-level aligmment and prototype-level alignment facilitate the image encoder to learn more generalizable representations for downstream tasks. When combining CTA and CPA, we can obtain further improvement on all datasets, indicating that the benefits of CTA and CPA are complementary. According to the results on SIIM dataser, we notice that CTA and CPA can both improve semantic segmentation performance when combined with ITA. Interestingly, CTA improves larger margin the semantic segmentation performance than CPA, which further elaborates that CTA is helpful to learn fine-grained information. When we train ITA, CTA, and CPA jointly, achieves the best performance.
Results Natural Vision-Language Pre-trained Model 3 shows the results of fine mee-tunnn the oatee-o--th-aaee art natural Vision Laaguu-La Pre-trained (VLP) model BLIP (34), which is pre- pre-triined on 14M image-text pairs. Due to the large domain discrepancy between natural image-text and medical image-text, directly transferring the pre-trained BLIP model to the downstream medical image tasks leads to inferior performance. This comparison indicates that pre-training on medical image-text datasets is necessary for capturing useful medical prior knowledge.
Analysis of Error Bars Table 6 shows error bars of our method on linear classification setting with ViT-B/16 as the image encoder backbone. We re-run each downstream task three times and calculate the mean and standard deviations. It is observed that the error bars are relatively small while comparing against other methods, which shows that our proposed method performs stably in these downstream tasks.
Table 6: Error bar of our methods linear classification setting.
1%
10%
100%
CheXpert (AUC)
88.7 + 0.18
89.13 + 0.16
89.5 # 0.21
RSNA (AUC)
89.03 - 0.11
99.92 0.14
90.77 = 0.04
COVIDX (ACC)
73.9 + 0.64
92.85 = 0.50

---

Discussion and Conclusion
This work presents MGCA, a multi multi-granularity cross-modal alignment framework for learning gener- alized medical visual representations from free-text radiology reports. By harnessing the naturally exhibited multi-granularity semantic correspondences across medical images and reports, our frame- work can learn generalized and discriminative medical image presentations for versatile downstream tasks to reduce the annotation burden. Extensive experimental results on seven downstream datasets demonstrate that our framework achieves substantial performance with limited annotated data.
Limitations and Future Work As our work mainly focuses on medical visual representation learning, we did not conduct experiments can be regarded as a limitation of our work. Our current framework learns the multi-grane -uati-granularity cross-modal alignment in parallel. In future work, we would like explore how to leverage the multi-granularit correspondence in a holistic manner. Moreover, this paper mainly investigates the discrimination-base -based image-text pre-training- We also plan extend our framework as the integration of discrimination-base and generation-based pre-training methods for medical image and text learning.
image-image image-text retricval downstream tasks, which
Social Impacts Our MGCA provides a promising solution to automatically diagnose abnormality of chest X-rays with limited annotated data, which assist in reducing the workload of radiologists and promote the health in poor area. On the other hand, medical data (e.g., chest X-rays, radiology reports etc.) may contain unintended private information harmful texts, and we highly recommend users conduct a careful analysis of data before employing our model into practical applications.
Acknowledgement
We gratefully thank HK HADCL for the support of this project. The work described in this paper is supported by HKU Seed Fund for Basic Research (Project No. 202009185079 and 202111159073).
References
[1] E. Alsentzer, J. R. Murphy, W. Boag, W.HH. Weng, D. Jin, T. Naumann, and M. McDermott. Publicly available clinical bert embeddings. arXiv preprint
2019.
[2] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Boojanowki, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 13:9912-9922 2020.
[3] K. Chaitanya, E. Erdil, N. Karani, and E. Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33: 112226162555 2020.
[4] T. Chen, S. Korbliihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[5] Y.-. Chen, L., Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Learning universal image-text representations. 2019.
[6] M. Cordts, M. Omran, Ramos, Rehfeld, M. Enzweiler, R. Benenson, U. Franke, Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[7] M. Cornaa, M. Stefanini, L. Baraldi, and R. Cucchiara. Meshed membry transformer for image captioning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 10578-10587, 2020.
[8] M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
10

---

[9] J. De Fauw, J. R. Ledsam, B. Romera-Paredes S. Nikolov, N. Tomasev, S. Blackwell, H. Askham, X. Glorot, B. O'Donoghue, D. Visentin, al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, (409)1142-11350 2018.
[10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004000 2018.
[11] H. Diao, Y. Zhang. L. Ma, and H. Lu. Similarity reasoning and filtration for image-text matching. arXiv preprint arXiv:2101.00 01368, 2021.
[12] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth I 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[13] M. Engilberge, I.. Chevallier, P. Pérez, and M. Cord. Finding beans burgers: Deep semantic- visual embedding with localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3984-3993, 2018.
[14] A. Esteva, B. Kuprel, R. A. Novoa, Ko. S. M. Swetter, H. M. Blau, and S. Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature, 14257639) 2017.
[15] M. Everingham, L. Van Gool, C. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[16] Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler. Vse++: Improving sesunlccemantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612, 2017.
[17] R. Feng, Z. Zhou, M. B. Gotway, and J. Liang. Parts2whole: Self-supervised contrastive learning via reconstruction. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, pages 85-95 99. Springer, 2020.
[18] S. for Imaging Informatics in Medicine. Siim-acr pneumothorax segmentation.
[19] J.-B. Grill, F. Strub, Altché, C. Tallec, Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances Neural Information Processing Systems, 13:21271-22284, 2020.
[20] V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, et al. Development and validation of deep learning algo- rithm for detection of diabetic retinopathy in retinal fundus photographs. Jama, 316(22):2402- 2410, 2016.
[21] Y. Guo, M. J., B. Ni, X. Zhu, Z. Sun, and Y. Xu. Hese: Hierarchical contrastive selective coding. arXiv preprint arXiv:2202.00455. 2022.
[22] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), volume 2. pages 1735 1742. IEEE, 2006.
[23] Y. Han, C. Chen, A. Tewfik, Y. Ding, and Y. Peng. Pneumonia detection on chest x-ray using radiomic features and contrastive learning. In 2021 IEEE 18th International Symposium Biomedical Imaging (ISBI), pages 247-251. IEEE, 2021.
[24] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[25] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[26] J. Healthcare. Object-cxr- automatic detection of foreign objects chest x-rays.

---

[27] S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942-3951, 2021.
[28] Y. Huang. Q. Wu, C. Song, and L. Wang. Learning semantic concepts and order for image and sentence matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 0163-6171, 2018.
[29] Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-llcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence. volume 33, pages 590-597, 2019.
[30] A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F, Makedon. A survey contrastive sulf-supe learning. Technologies, 9(1):2, 2020.
[31] A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):1-8, 2019.
[32] P. H. Le-Khac, G. Healy, and A. F. Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8: 1999077-1993,,, 2020.
[33] K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pages 201-216, 2018.
[34] Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language- pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086, 2022.
[35] J. Li, P. Zhou, C. Xiong, and S. C. Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint urXiv:2005.04900 2020.
[36] K. Li, Y. Zhang, K. Li, Y. Li, and Y. Fu, Visual semantic reasoning for image-text matching. In Proceedings of the IEEE/CVF International conference on computer vision, pages 4654-4662, 2019.
[37] Y. Li, P. Hu, Z. Liu, D. Peng, J. T. Zhou, and X. Peng. Contrastive clustering. In 2021 AAAI Conference on Artificial Intelligence (AAAI), 2021.
[38] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[39] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608v:1608.03998 039983 2016.
[40] Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.51001, 2017.
[41] Lu, J. Yang. D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. Advances in neural information processing systems, 29, 2016.
[42] P. Müller, G. Kaissis, C. Zou, and D. Rückert. Joint learning of localized representations from medical images and reports. arXiv preprint (2.i::2/2.0028.. 2021.
[43] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.0378333 2018.
[44] Rajpurkar, Irvin, R. Ball, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. P. Langlotz, et al. Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists. PLoS medicine, 15(11xx1002666, 2018.
[45] Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767.02 2018.
12

---

[46] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234 241. Springer, 2015.
[47] G. Shih, C. C. Wu, S. S. Halabi, M. D. Kohli, L. M. Prevedello, T. S. Cook, A. Sharma, J. K. Amorosa, V. Arteaga, M. Galperin-Aizenber et al. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 1 x1)xee 180041 2019.
[48] W. Su, X. Zhu, Y. Cao, B. Ii, L.. Lu, F. Wei, and J. Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.000 085300 2019.
[49] A. Taleb, M. Kirchler, R. Monti, and C. Lippert. Contig: Self-supervised multimodal contrastive learning for medical imaging with genetics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20908 20921, 2022.
[50] H. Tan and M. Bansal. Lxmert: Learning mrosstmodality encoder representations from trans- formers. arXiv preprint UrXiv:190.00744 2019.
[51] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
[52] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[53] L. Wang, Z. Q. Lin, and A. Wong. Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific Reports, 10(1):1-12, 2020.
[54] X. Wang, Z. Liu, and S. X. Yu. Unsupervised feature learning by cross-level instance-group group discrimination. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12586 12586-12595, 12595, 2021.
[55] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax discases. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
[56] X. Wang, R. Zhang, C. Shen, T. Kong, and L. Li. Dense contrastive learning for sulf-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[57] E. Xie, J. Ding, W. Wang, X. Zhan, H. Xu, P. Sun, Z. Li, and P. Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8392-8401, 2021.
[58] Z. Xie, Y. Lin, Z. Zhang, Y. Cao, S. Lin, and H. Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16684-16693, 2021.
[59] J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144 2022.
[60] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[61] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00447, 2020.
[62] H.-Y. Zhou, X. Chen, Y. Zhang, R. Luo, L. Wang, and Y. Yu. Generalized radiograph represen- tation learning via cross-supervision between images and free-text radiology reports. Nature Machine Intelligence, pages 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the second paragraph in Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the last paragraph in Section S
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A] Our work mainly involves empirical contributions.
(b) Did you include complete proofs of all theoretical results? [N/A] Our work mainly involves empirical contributions.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See the Section for pre pre-training dataset and model details. More experimental details is included in the supplementary material. Our code is in the supplementary material and will be released later.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.2 for data splits. Most of dataset details are following previous work.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the second paragraph of Section
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite all used public datasets Section and 4.2
(b) Did you mention the license the assets? [Yes] All dataset are publicly available.
They under non-commercial license.
(c) Did you include any new assets either the supplemental material as URL? [Yes] Our code and pre-trained model will be released later for the assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We are using publicly available datasets for all experiments.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We are using publicly available datasets for all experiments. No personally identifiable information involved.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] No human subjects involved in our experiments.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. applicable? [N/A] No human subjects involved our experiments.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] No human subjects involved in our experi- ments.
14


Reviews:
[{"review_id": "ON7N8BsTmho", "reviewer": "Reviewer_RZTv", "summary": "This paper presents a novel Multi-Granularity Cross-modal Alignment (MGCA) framework to seamlessly leverage the multi-granular semantic correspondences between medical images and radiology reports for generalized medical visual representation learning. Extensive experimental results on different downstream datasets show that the proposed framework achieves substantial performance with limited annotated data.", "questions": "(1) The authors incorporate several well-known technologies, including instance-wise alignment, fine-grained token-wise alignment, and disease-level alignment, into contrastive learning to enhance the generalizability of learned visual representations. In this case, which parts are new? Thus, the overall novelty needs to be strengthened.\n(2) In Table 1, it can be seen that the proposed model performs better than other methods. However, because the proposed model has been pre-trained on a large-scale medical image and report dataset, whether other methods have been trained on the large dataset? \n(3) The generated visual and text token embeddings are projected into normalized lower-dimensional embeddings, and the dimension d is set to 128. How to set the dimension? The parameter study should be included. \n(4) From the results in Table 1, it can be seen that the proposed model obtains large improvements over DSVE and VSE++. Please the reasons for these situations. \n(5) It is better to provide more discussions on using multi-granularity cross-modal alignment. ", "limitations": "See weakness. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n(1) A multi-granularity cross-modal alignment framework is proposed for learning generalized medical visual representations from free-text radiology reports.\n(2) The overall structure is clear.\n(3) The experimental results show the effectiveness of the proposed model.\n \nWeaknesses:\n(1) The authors incorporate several well-known technologies, including instance-wise alignment, fine-grained token-wise alignment, and disease-level alignment, into contrastive learning to enhance the generalizability of learned visual representations. In this case, which parts are new? Thus, the overall novelty needs to be strengthened.\n(2) In Table 1, it can be seen that the proposed model performs better than other methods. However, because the proposed model has been pre-trained on a large-scale medical image and report dataset, whether other methods have been trained on the large dataset? \n(3) The generated visual and text token embeddings are projected into normalized lower-dimensional embeddings, and the dimension d is set to 128. How to set the dimension? The parameter study should be included. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "FN6dNtlrMUZ", "reviewer": "Reviewer_D1Wu", "summary": "This paper proposes a Multi-Granularity Cross-Modal Alignment method, which learns data representations from medical scans paired with the corresponding text reports. The method exploits multiple unsupervised techniques to obtain the learned representations (e.g. contrastive losses, clustering with Sinkhorn-Knopp), and each of these techniques are utilized at the appropriate level of granularity. \nThe learned representations are then evaluated on a large set of image-based downstream tasks, to assess the quality of the image representations. The experimental results support the merits of the proposed method. However, there are some weaknesses and limitations, which will be listed below.", "questions": "It would be interesting to have a discussion in the paper on how the ITA level behaves with the existence of CPA. These two losses have a somehow contradictory behavior. The contrastive loss in ITA pushes the different instances (negatives) apart from positive instances. However, CPA clusters instances in the feature space in order to learn disease (prototype) clusters. CPA disregards the identity of the instance and rather tries to capture high-level unifying topic models. \nHow do they behave then? It seems to me in Table 4 that the addition of the CTA loss is essential for improved performance (rows 2 and 4). But adding CPA and ITA alone (in row 3), seems to perform a little worse than row 2.\nPlease discuss how do you imagine the losses are behaving.", "limitations": "The authors list the limitations adequately ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method is novel, if viewed as a whole framework that employs the granularity of the features that are present in medical scans. However, if one views each level in the method independently, these levels are not novel per se. But since the proposed method aims to encapsulate these levels together, then I deem the method as novel. The only level that I would deem more novel is the CPA, which proposes to learn cross-modal disease prototypes.\n- The evaluation experiments are extensive. The paper reports results on multiple dataset benchmarks, which include multiple types of tasks (classification, detection and segmentation). The work also evaluates two types of image encoders (resnet & ViT).\n- The paper is clearly written, and the used language is well understandable. \n\nWeaknesses:\nMostly, I believe the weaknesses in this work are in some missing pieces:\n1- Missing relevant references from the literature. The following are some works that perform contrastive learning on medical images:\n* Taleb, Aiham, et al. \"ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n* Chaitanya, Krishna, et al. \"Contrastive learning of global and local features for medical image segmentation with limited annotations.\" Advances in Neural Information Processing Systems 33 (2020): 12546-12558.\n* Han, Yan, et al. \"Pneumonia detection on chest x-ray using radiomic features and contrastive learning.\" 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.\n* Feng, Ruibin, et al. \"Parts2Whole: Self-supervised contrastive learning via reconstruction.\" Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning. Springer, Cham, 2020. 85-95.\n* Xu, Jiarui, et al. \"GroupViT: Semantic Segmentation Emerges from Text Supervision.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\nThere are quite a lot of works that are similar along this line, would be great to cite the most relevant once from the medical imaging literature. \nAlso I believe some references related to the methodological part (mainly cross-attention) are missing, such as:\nChen, Yen-Chun, et al. \"Uniter: Learning universal image-text representations.\" (2019).\nLu, Jiasen, et al. \"Hierarchical question-image co-attention for visual question answering.\" Advances in neural information processing systems 29 (2016).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7gg7j4H-KsM", "reviewer": "Reviewer_ZGvT", "summary": "The authors present a visual representation learning approach based on contrastive learning of correspondences between visual radiographs and textual radiology reports at multiple levels of granularity using three different learning objectives. The first objective features standard cross-modal alignment between radiographs and their corresponding radiology report pairs, by which they enforce instance level agreement. The second objective features a cross-attention approach that enforces the alignment between the visual token embeddings (local representations obtained using the Vision Transformer) and aggregated text token representations (where the visual token is used as a query for attention-based aggregation of word token representations), and likewise, the alignment between text tokens and aggregated visual tokens. This enforces alignment between mutually informative local image regions in the radiograph and parts of the textual radiology report, and the authors argue that this region-level alignment improves performance of resulting visual representations on downstream task that involve dense predicitons. The final objective features alignment on a higher-than-instance level, by enforcing consistent cross-modal clustering of representations of corresponding visual and text data. This implicitly encourages the radiographs and reports that share high-level semanics to have similar representations, regardless of instance-level pairing and modality. Authors also demonstrate that pretraining on in-domain images (medical) is important for performance on downstream tasks, and that general-domain pretraining is not as effective.", "questions": "- How is CTA implemented in cases when ResNet50 is used as the image encoder? In what way are visual tokens obtained? To what degree would you say the difference in the way the tokens are obtained can account for the difference in performance between the ResNet50 and ViT variant of your approach?\n- Does the use of CTA give an even bigger performance improvement in a segmentation or object detection setting (compared to classification)?", "limitations": "The authors list the lack of consideration of retrieval tasks as one of the limitations of their work. However, the outlined scope of their experiments is in my opinion reasonable. With regards to potentially negative societal impacts, the authors mention the possible use of sensitive data in their framework. This is a general concern when medical data is used and is therefore not particularly specific to their work. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written and well structured.\n- Implementation details and reproducibility info are clearly provided, with appendix containing additional ablation studies justifying parameter and design choices.\n- Extensive experiments (both qualitative and quantitative) as well as ablation studies. The choice of downstream tasks appears to be justified given the stated focus on learning visual representations, and they seem diverse enough to put different individual aspects of their approach to the test. The comparison with the main competing approach, GLoRIA, seems fair, as the authors also evaluated a variant of their model that uses the same image encoder as GLoRIA as well as the same pre-processing step.\n- The idea and implementation of the disease-level cross-modal alignment module in this setting, to the best of my knowledge, can be seen as novel.\n\nWeaknesses:\n- Lack of an ablation study involving individual learning tasks in a object detection or segmentation setting (as was done under the classification setting). Their approach does seem to more convincingly outperform GLoRIA in a object detection and segmentation setting (compared to the classification featuring the ResNet-50 encoder), but it would have been interesting to see an ablation study that explicitly shows how relevant, for instance, CTA is in a dense prediction setting.\n- Lack of error bars may be a potential issue, since the differences in performance between their approach and baselines (in some settings) are sometimes small.\n- This particular setup for contrastive learning based approach for visual representation learning (also in the context of paired radiographs and radiology reports), as well as the idea and motivation behind token-wise cross-modal alignment objective, at its core, seem to be inspired by ConVIRT and/or GLoRIA, so the work can, to a degree, be seen as an extension of existing work and not entirely novel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept