Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
Authors: Zhongwei Wan
Che Liu
Mi Zhang
Jie Fu
Benyou Wang
Sibo Cheng
Lei Ma
César Quilodrán-Casas
Rossella Arcucci
Venue: NeurIPS 2023 poster (2023)
Keywords: Medical Vision Langauge Pretraining
Cross-lingual
Language bias

Abstract: The scarcity of data presents a critical obstacle to the efficacy of medical vision-language pre-training (VLP). A potential solution lies in the combination of datasets from various language communities.
Nevertheless, the main challenge stems from the complexity of integrating diverse syntax and semantics, language-specific medical terminology, and culture-specific implicit knowledge. Therefore, one crucial aspect to consider is the presence of community bias caused by different languages.
This paper presents a novel framework named Unifying Cross-Lingual Medical Vision-Language Pre-Training (\textbf{Med-UniC}), designed to integrate multi-modal medical data from the two most prevalent languages, English and Spanish. 
Specifically, we propose \textbf{C}ross-lingual \textbf{T}ext Alignment \textbf{R}egularization (\textbf{CTR}) to explicitly unify cross-lingual semantic representations of medical reports originating from diverse language communities. 
\textbf{CTR} is optimized through latent language disentanglement, rendering our optimization objective to not depend on negative samples, thereby significantly mitigating the bias from determining positive-negative sample pairs within analogous medical reports. Furthermore, it ensures that the cross-lingual representation is not biased toward any specific language community.
\textbf{Med-UniC} reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases, offering a versatile framework for unifying multi-modal medical data within diverse linguistic communities.
The experimental outcomes highlight the presence of community bias in cross-lingual VLP. Reducing this bias enhances the performance not only in vision-language tasks but also in uni-modal visual tasks.

Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
Zhongwei Wan'*,
Benyou Wang Lei Ma² César Quilodrán-Casas? Rossella Areueci
Che Liu"; Mi Zhang Jie Fust
Sibo Cheng",
'The Ohio State University IImperial College London
PPeking University "The Chinese University of Hong Kong, Shenzhen 'The Hong Kong University Science and Technology {wan.511 mizhang. 1.0osu.edii lei maêpku. edu, en, wangbenyou@cuhk. edu, cn {che, liu21, sibo.cherg c. quilodrr,,, r ararucci@@imperiil..c. uk, jiefubust. hk
Abstract
The scarcity of data presents a critical obstacle to the efficacy of medical vision- language pre-training (VI.P). A potential solution lies the combination of datasets from various language communities. Nevertheless, the main challenge stems from the complexity of integrating diverse syntax and semantics, language-specific medical terminology, and culture-specific implicit knowledge. Therefore, crucial aspect to consider is the presence of community bias caused by different languages. This paper presents a novel framework named Unifying Cross-Lingual Medical Vision- Language Pre- Training (Med-UniC), designed to integrate multi- modal medical data from the two most prevalent languages, English and Spanish. Specifically, we propose Cross-lingual Text Alignment Regularization (CTR) to explicitly unify cross-lingual semantic representations of medical reports orig- inating from diverse language communities. CTR is optimized through latent language disentanglement, rendering optimization objective not depend on negative samples, thereby significantly mitigating the bias from determining positive-negative sample pairs within analogous medical reports. Furthermore, it ensures that the cross-lingual representation is not biased toward any specific language community. Med-UniC reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases, offering a versatile framework for unifying munities. The experimental outcomes highlight the presence of community bias in cross-lingual VI.P. Reducing this bias enhances the performance not only vision-language tasks but also in uni-modal visual tasks. The source code has been released https://github. wwt.ggtthhhh...../ee Sonn/SSSSSecccBce/MMeeee
medical data within diverse linguistic com-
Introduction
English, despite not being the primary native language for vast majority of the global population, remains the dominant language in vision-language pre-training (VLP) datasets. Uni-lingual VLP models not only demonstrate restricted performance in cross-lingual tasks, but also bring the com- munity bias on English speaking populations (displayed in Fig 1), particularly in the context of medical applications.
Researchers have used machine-translate non-English corpora and techniques like masked language model (MLM) and contrastive learning to unify cross-lingual representations [1-
Equal Contribution.
'Corresponding Authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

5p
Clustert cluste22
5p
clestert
En
En
cluster2
Center dstance: 2.38
Curker dolannee D.31
(a) 3D Text Embedding from MLM.
(b) 3D Clustering from MLM.
(c) 3D Text Embedding from Med-UniC.
(d) 3D Clustering from Med-UniC.
Figure I: Graphical illustration of community bias in MLM and Med-UniC is shown with blue points representing Spanish reports and orange points indicating English reports, visualized via T-SNE. MLM denotes cross-lingual masked language modeling.
4]. However, MLM-based representations still separate languages, as shown in Fig la and lb. Additionally, as highlighted in Fig 2, the significant similarity among reports from di- verse communities suggests that the strict reliance on negative samples contrastive learn- ing could introduce more bias during text alignment. Hence, Med-UniC focuses on learning text invariants in VLP using negative-free text alignment avoid the bias mentioned above. In this work, introduce a unified framework named Med-UniC, designed to acquire language-agnostic representations from chest x- ray (CXR) scans and associated radiology reports. Med-UniC learns representation from perspectives: visual invariants, visual- textual invariants, and text invariants. Considering medical vision- language tasks, such as zero-shot image classification, are dependent semantic information and amnguage-independee
Sp
200
propose CTR (Cross-lingual Text Alignment Regularization) to explicitly minimize linguistic disparities in cross-lingual representations within the latent space, as visualized Med-UniC tackles non-English vision-language tasks without the model bias stemming from the language model (LM) pre-trained on predominantly English corpora. Additionally, we found that the unified cross-lingual representation enhances performance across a range of uni-modal visual tasks. This paper makes the following contributions:
En
Fig Ic and Id. Consequently,
Figure 2: Similarity matrix for medical reports embedding from MLM.
To the best of our knowledge, are the first empirically identify the existence of community bias originating from diverse languages in cross-lingual medical VL.P (displayed in Fig 1), and the negative impact of community bias both uni-modal and vision-language downstream tasks.
We introduce the framework Med- UniC with CTR, designed to diminish community bias in medical VLP by unifying cross-lingual latent representations. Med-UniC achieves SOTA results in medical vision-language tasks across different languages, demonstrating its efficacy and broad applicability across various language communities.
Med-UniC achieves SOTA results on all uni-modal visual tasks. This highlights the advantages of mitigating community bias and unifying cross lingull representations in medical VLP, enabling robust and comprehensive learning of visual representations.
Med-UniC effectively mitigates community bias without the requirement for manual curation or language-specific annotations. Importantly, Med-UniC enhances the accessibility of medical VLP non-English speaking populations, circumventing potential biases that may arise from predominantly English datasets in traditional VLP.
2 Related Work
Medical VLP Complex medical reports and a shortage of large-scale medical image-text datasets have limited medical VIPP research. Previous works such as CIRV [5] and GLoRIA [6] utilized contrastive objectives and global-local VLP to align image-text pairs. MGCA [7] used disease- level annotations for alignment, while MedKLIIP [8] manually extracted medically relevant entities.

---

aug
Vee
Frozen
Unfrozen
Vision-Vision Aligument
Starod weights
Lasy
EoTA
aug
Ves
Pes
+
Images
Sample-wise Alignment Identity Matrix
Vision-Language Alignment
2
Sp
+
Feature-wise Aligament
En
Eimboldings dropout
Staned
F4
Z
Cross-lingual Alignment
2'
Z
Identity Matrix
weights
Reports
Figure 3: Overview Med-UniC. CVL, SSV, CTR represent cross-lingual vision-language alignment, self-supervised vision alignment, Cross-lingual Text Alignment Regularization, respectively.
MRM [9] replaced alignment with reconstruction task involving masked visual and textual tokens. However, these methods fail to capitalize on valuable cross-lingual knowledge and do not effectively unify cross-lingual representations. As result, their performance cross-lingual tasks is consider- ably limited.
Cross-lingual VLP Most recent cross-lingual VLP research has focused on a limited of lan- guages, with a few comparing natural cross-lingual VLP to English-only VLP [10]. Studies such corpora and contrastive learning to generate language-agnosse
as [1-4] used machine-transie representations. Yet, cross-lingur VLP in the medical domain remains unexplored mainly due to two main challenges: the limitations of machine translation for medical reports and the bias introduced by hard positive-negativ pairs in contrastive learning. We propose CTR, negative-free disentangled loss method, to unify cross-lingual text representations, addressing these issues.
Large General Model Recently, there have been impressive advances in large general models for language and vision tasks, such as ChatGPT [11], SAM, and DINOv2 [11-13]. However, these models still face significant limitations in the medical domain due to their lack of domain-specific knowledge and inability to jointly process visual and textual information [14-16]. ChatGPT [11] and SAM [12], currently limited to single-modality input, are unsuitable for vision-language tasks. While SAM excelled at instance segmentation, it struggled with medical image pathology segmenta- tion [16-18]. Conversely, Med-UniC addressed these constraints by utilizing visual and textual data, integrating Noomain-specife knowledge without the high computational expenses of large general models. Surprisingly, Med-UniC outperformed the extensive vision model, DINOv2 [13], trained on 1.2B images with IB parameters, by considerable margin in multiple visual tasks. This suggests that Med-UniC is more effective and efficient solution compared to large general models.
3 Method
3.1 Overall Framework of Med-UniC
Our Med-UniC framework aims learn cross-lingual medical representation from CXR images and paired radiology reports. Given training set of N cross-lingual dataset S E V x L consisting of pairs (Ve.s, where and L are visual and text set, respectively, Ve,i is a raw image and la.; is text report, e belongs to language domain (e.g.. Spanish or English), i E N. The Med- UniC architecture mainly consists of an image encoder F. V D. to encoder the raw image into the embeddings with dimension D.. and cross-lingual text encoder Fi L + DP to encoder the text report to the embeddings with dimension Dr. Then = {(Ve.1, 1,,1) (Ve.2, where Ve,; F. (ve,s) and Le:: (lc,,))
(Ve,N, (e,N),
As depicted in Fig Med-UniC incorporates three concurrent alignment strategies: cross-lingual vision- vision-language alignment, self-supervised vision alignment, and pross-lingus text alignment regu- larization. For cross-ling lingual vision-language alignment, we extend the text encoder from uni-lingual into a cross-lingual style, which can equip the model with cross-lingual cross-modal ability by pulling the embeddings paired image-reports pairs together while pushing apart embeddings of unmatched pairs, under the loss LoVI. Meanwhile, employ self-supervised vision alignment, leveraging the

---

loss Lssv, to enhance the robustness of visual representation [19]. More importantly, we introduce a cross-lingual text alignment regularization, encompassing sample-level and feature-level approaches. to mitigate community bias stemming from different languages. This regularization is supervised by the loss LeTR. The learning objective of Med-UniC integrates the three strategies mentioned above and can be formulated as follows:
-=ver. + Lssv + LCTR
This training scheme compels Med-UniC to assimilate information from diverse perspectives, fos- tering a robust cross-modal, cross-lingual framework that concurrently learns visual invariants, visual-textual invariants, and text invariants.
3.2 Cross-lingual Vision-language Alignment
Cross-lingual Medical LM To initialize Med-UniC with the ability to process different languages and learn fundamental cross-lingual syntactic and semantic knowledge. We select CXR-BERT [20], uni-lingual LM pre-trained on large-scale biomedical corpus, as our text encoder and further adapted for cross-lingual operation. The concrete steps proceed as follows: (1) Constructing a cross-lingual vocabulary set T: we collected the radiology reports of the second language (e.g., Spanish PadChest dataset [21]), which is seldom used for medical pre-training compared to English dataset [22, 23]. Then we leverage general Spanish Spacy to build tokenizer and make use of TF-IDF tool to generate ranked new tokens Tep = { t1 Eep..... ;ta } according to their importance from multiple reports. (2) Building new wording embeddings W: we augment the original vocabulary Ten with the sorted Spanish tokens to generate T = {Ten: Tap} then expand length of M random initialized vectors W sp on CXR-BERTS word embeddings Wen. where W=Wwa Wepl. (3) Masked Cross-lingual Modeling: we following BERT [24], randomly mixed English [22] and Spanish [21] medical reports pre-train corpus. Then we randomly choose tokens in the mixed sequences, replace them with the [MASK] token and set 15% masking probability as in BERT [25]. We selectively update several high layers to alleviate catastrophic forgetting [26] during vision-language pre-training. More details will be show in Sec 4.5.
Vision-Language Alignment Following CLIP framework [27], we incorporate contrastive learning object to predict the matched pair (Ve,) from N X possible image-text pairs while mapping 2 negative pairs far apart. Specifically, two non- con-linear visual and linguistic projectors P: and P, are used to transform Ve, and le, into the same dimension d, where Ve. = Pv. L, = (l...)). and Ve,i, le, E Rd. Obtaining image feature vectors [Ve,i]/=] and text feature vectors ( I from training batch, we compute cosine similarities s.22( 4 V., and 2 Ve. representing image-text and text-image similarities, respectively. LevI. is then formulated as follows:
exp(s 22 /01)
exp(s!?//0r
C log
L. log
(2)
K ΣØ= exp( (5/( /01) N
K Σ
exp(s12y
/01)
122
LeVI 2K
( i=1
where C and E.20 are image-text and text-image InforNCE [28] contrastive loss, respectively. d is the temperature papper-prramter set to 0.07 in our experiments, K is the batch size for each step and E N. Through overall loss LevI., the model learns maximal mutual information between the matched image-text pairs containing cross-lingual attributes within a batch.
3.3 Self-supervised Vision Alignment
To obtain more exhaustive visual representation, we include visual invariant learning as a parallel objective during VLP. Drawing inspiration from [19], we initially apply random augmentations (such as random cropping and flipping) to the original images to create augmented views as positive pairs [ vo M1 while treating the rest of the images in the mini-batch as negative samples. We follow the data augmentation strategy as per [19]. Subsequently, we extract the representations of
hann:://ppayy.immmmdeeeeeeeepeee "TidiVectorizer: https//scikki-leam...r

---

the augmented views [ using the vision projector and vision encoder Fe, similar to [V] Therefore, the visual invariant learning objective becomes:
N
exp
20 /02)
SsS
log
1:2 = Ve
(4)
ex
22 /02)
where 02 is the temperature hyper-parameter also set 0.07 for overall loss objective Lssv.
3.4 Cross-lingual Text Alignment Regularization
As illustrated in Fig 1, and corroborated by research in natural language processing [29, 30], cross- lingual text representations tend to form distinct clusters based their respective languages. This trend introduces community bias within data from different language communities, even when no explicit language attribute is provided. This suggests that VLP processes medical data based on their language community, risking unfairness in clinical applications and potentially decreasing downstream task performance in both uni-modal and vision- lainunlanguage tasks [31-33]. To mitigate this bias and potential risks, we introduce Cross-linge Cross-lingual Text Alignment Regularization (CTR) to learn Linguage-indepencee text representations and neutralize the adverse effects of community bias other modalities. CTR comprises three components:
Text augmentation We first adopt the dropout strategy [34] to generate augmented the text repre- sentation I2,, from the text encoder and obtain the matched pairs [(c.. 1',== and then separate linguistic projector Pa designed for de-correlation is leveraged to generate two different view pairs [(Z e,i, B (,))];= where Z e,i = Pa(le.),, ZB P)(),))), and the new feature dimension >.
Text-feature alignment To further alleviate information redundancy [35, 36] and obtain the shared cross-lingual text representation, we first normalize the augmented embedding pairs { {ZA Z ] NDD along the batch K dimension so that each feature dimension has zero-mean and 1/vK K standard deviation distribution to generate Z.. and then compute their cross-correlation Z ZATZZ ZB The formulas of feature-dimension decorrelation can be defined as:
D'
K A 0,
2
D'
K
1 LTF = D
Σ
Bij
ZAJTZBJ i
Ž.
Ze MK (K((Ze)
VKo
J
cross-lingual invariance
cmos-lingual gap reduction
(5)
The first term's objective to learn a language-invariant representation by optimizing the diagonal elements of the correlation matrix to equal one. Simultaneously, the second term aims to shrink the cross-lingual gap and optimize information utilization in each latent dimension by driving the off-diagonal elements towards zero. Finally, We normalize the loss along with the feature dimension D'
Text-to-test alignment: Similarly, the text-to-text alignment decorrelates the cross-correlation matrix along with feature dimension D', and 2e is the normalized embeddings, Zeor BBT
LTT
1
K
1
D'
K
D
2 B.T.
2.
Ze AD (Ze)
K
(Ze)
aligament
consistency regularizer
(6) where the text instance alignment term attempts to maximize the mutual information of batch of cross-lingual text samples, and the texl consistency regularizer can also be deemed as the text in-modal consistency [37] by reducing the mismatched text pairs into 0 in batch K. Where À in Eq
5, 6. is a non-negative hyperparameter trading off two terms. We also normalize the loss with the batch dimension K. Therefore, the loss of Cross- Text Alignment Regularization LOTR r is:
LOTR LTF + LTT
5

---

4 Experiments
4.1 Pre-training Configuration
Dataset We pre-train Med-Uni framework using MIMIC-CXR [38], which contains CXR images and their corresponding radiology reports in English. Also, we involve PadChest [39], which includes CXR images and their corresponding radiology reports collected in Valencia region, Spain. Both datasets are pre- pre-processed following the approach described in [5-7], including image resizing, pixel value normalization, and text tokenization. Additionally, the dataset is filtered by excluding lateral views and reports with less than three tokens. This results in a pre-training dataset of approximately 220k image-text pairs for MIMIC-CXR [38] and 160k pairs for PadChest [39].
Implementation In the VLP stage, we employ ResNet-50 [40] and ViT [41] as the vision backbones. We report the linear classification results for these two vision encoders to illustrate the model-agnostic capabilities of Med-UniC
Meic is trained over 50 epochs using an early stop strategy on 16 V V100 GPUs with a batch size of 128 per GPU. We utilize Adam W [42] the optimizer, setting the learning rate to 4e-5 and the weight decay to 5e-2 A linear warm-up and cosine annealing scheduler also deployed in this process. Additionally, The coefficients À is set to 5.1e- following [36].
4.2 Downstream Tasks
Medical Image Linear Classification We perform this task on CheXpert [23], RSNA [43], and COVIDx [44] datasets. Following the previous work [5-7], only update the parameter of random initialized linear layer for classification and freeze the pre-trained vision backbone. We report the AUC scores (AUC) on CheXpert and RSNA and accuracy (ACC) COVIDx the evaluation metric following [6, 7].
Medical Image Zero-shot Classification We conduct this experiment the CXP500 [45] and PDC [39] datasets, which comprise CXR images annotated by clinicians from English-speaking and Spanish-speaking countries, respectively. To circumvent prompt bias, we designate English positive prompt as '(disease)' and negative prompt as 'No (disease)' Prompts in Spanish are prepared by native Spanish speakers, with the disease indicated as "(disease)' and the negative prompt represented as 'No hay (disease)' Med-UniCC is evaluated using both English and Spanish prompts across the two datasets, with additional experimental details provided in the Appendix. The results are represented as the macro average of AUC and Fl scores across all categories.
Medical Image Semantic Segmentation This task performed on the RSNA [43] and the SIIM [46] datasets, following the data preprocessing in [6. 7]. Identical to [6, 71, the U-Net [47] fine-tuning settings are adopted for segmentation. All pre-trained vision backbones are considered as frozen encoders, and only the decoders of U-Net are updated during the fine-tuning. The segmentation performance evaluated using Dice scores (Dice).
Medical Image Object Detection This task is performed on the RSNA [43] and Object-CXR [48]
datasets, following the same preprocessing of [7]. Same as [7], we utilize YOLOv3 [49] as the detection architecture, using our pre-trained vision encoder as the backbone and only updating the
detection head during fine-tuning. Mean Average Precision (mAP) with IOU thresholds 0.4~0.75, is adopted to evaluate the detection task.
For all downstream tasks, except zero-shot classification, we fine-tune with 1%, 10%, 100% of the training data. More downstream tasks' settings, including split information and train/valid/test set details, can be found in the Appendix.
4.3 Comparison to the state-of-the-art
Zero-shot Classification To assess the cross-lingual visual-textual representation learned from Med- UniC, we implement the zero-shot image classification task on two CXR datasets, which originate from distinct linguistic communities and utilize different language prompts. Tab illustrates that Med-UniC surpasses all SOTA methods on both datasets, regardless of the language setting or the linguistic community data source. Across both datasets, Med-UniC delivers an average increase of over 20% in the Fl score when using English prompts and more than 15% when using Spanish

---

Table 1: shoo-shot Image Classification results. Fl and AUC scores are reported. Best results of each setting are in boldface. 'En' and 'Sp' respectively stand for prompts English and Spanish languages. Methods with leverage discase-level annotations for pre-training.
CXP500(En)
CXP500(S))
PDC(En)
PDC(Sp)
Method
AUC
AUC
F1
AUC
F1
AUC
F1
ConVIRT|[]
59.5
19.2
60.5
15.8
45.1
26.5
49.1
12.6
GLoRIA[6]
43.2
2.4
40.2
16.1
52.3
10.1
50.3
8.2
GLoRIA-MIMIC [6]
46.2
51.5
20.3
53.1
12.1
52.2
11.3
MGCA [7]
72.1
6.5
50.4
22.3
46.4
32.5
49.8
26.1
MedKILP" [8]
70.5
14.7
55.6
21.9
50.5
28.7
51.7
25.8
MRM [9]
65.2
10.4
48.3
16.1
50.1
24.6
51,4
25.1
Ours
75.4
30.3
71.3
32.2
72.9
42.6
71.4
37.1
prompts. This showcases the effectiveness and adaptability of Med-UniC in managing cross-lingual vision-language tasks.
Interestingly, the AUC score of other SOTA methods experiences substantial drop when the prompts transition from English to Spanish on CXP500 [45]. a dataset collected from maglish-speaking communities. Similarly, all compared methods show comparably poor performance PDC [39], a dataset derived from Spanish-speaking communities. MedKLIIP [8]. despite its commendable performance the CXP500 [45] in the English setting due its supervised pre-training with disease annotation, persistently shows a drop in performance on both the CXP500 [45] and PDC [39] when Spanish prompts used, and also on the PDC [39] when using English prompts. These results highlight the significant community bias inherent in uni-lingual medical VLP models, even those utilizing annotations for pre-training. This bias adversely impacts the diagnostic quality when dealing with patients from non- noon-Engiish-o Singlis-speaking speaking communities or those who do not speak English.
The unsatisfied performance of the compared methods also suggests that these models might in- corporate linguistic community attributes during VLP, which negatively influences the learning of semantic meanings. Consequently, As result, these models have difficulties in effectively interpret- ing CXR scans or prompts from non-English communities, which substantially limits the models' transferability.. Further analysis can be found in Sec 4.4.
Table 2: Linear classification results for CheXpert, RSNA, and COVIDx datasets with 1%, 10%, and 100% training data. The best results are highlighted in bold. A standard ResNet-50 backbone is denoted as CNN-based. Methods with * leverage disease-level annotations for pre-training.
CheXpert (AUC)
RSNA (AUC)
COVIDx (ACC)
Method
1%
10%
100%
1%
10%
100%
1%
10%
100%
Random Init
56.1
62.6
65.7
58.9
69.4
74.1
50.5
60.3
70.0
ImageNet Init
74.4
79.7
81.4
74.9
74.5
76.3
64.8
78.8
86.3
CNN-bused
GLoRIA [6]
86.6
87.8
88.1
86.1
88.0
88.6
67.3
77.8
89.0
ConVIRT [5]
85.9
86.8
87.3
77.4
80.1
81.3
72.5
82.5
92.0
GLORIA-MIMIC [6]
87.1
88.7
88.0
87.0
89.4
90.2
66.5
80.5
88.8
MedKLIP* [8]
86.2
86.5
87.7
87.3
88.0
89.3
74.5
85.2
90.3
MGCA* [7]
87.6
88.0
88.2
88.6
89.9
83.5
Med-UniC (ResNet-50)
88,2
89.2
89.5
89.1
90.4
90.8
76.5
89.0
92.8
ViT-based
MRM [9]
88.5
88.5
88.7
91.3
92.7
93.3
66.9
79.3
90.8
MGCA* (ViT-B/16) [7]
88.8
89.1
89.7
89.1
89.9
90.8
74.8
84.8
92.3
Med-UniC (VIT-B/16)
89.4
89.7
90.8
91.9
93.1
93.7
80.3
89.5
94.5
Med-UniC (VIT-L/32)
89.9
90.5
91.2
92.2
93.8
94.5
81.5
91.8
95.2
Medical Image Linear Classification To assess the quality of the visual representations learned by Med-UniC we employ linear classification [50] on CheXpert [23], RSNA [43], and COVIDx [44]. As illustrated in Tab 2, Med-UniC framework consistently surpasses all uni-lingual pre-trained baselines across various settings and vision backbones. Significantly, even when MedKLIP [8] employs supervised VI.P with disease-level annotation, Med-UniC consistently surpasses MedKLII [8] across all tasks and settings. This exemplifies the adaptability and efficacy of the visual representations
7

---

Table 3: Results of semantic segmentation on SIIM and RSNA datasets and object detection on RSNA and Object-CXR datasets. The best results for each setting are highlighted in bold, and the '-' denotes mAP values smaller than I Methods with leverage disease-level annotations.
Semantic Segmentation (Dice)
Object Delection (mAP)
SIIM
RSNA
RSNA
Object CXR
Method
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
Random
9.0
28.6
54.3
6.9
10.6
1.0
4.0
8.9
0.5
4,4
ImageNet
10.2
35.5
63.5
34.8
39.9
64.0
3.6
8.0
15.7
2.9
8.3
Con CIRVIR
25.0
43.2
59.9
55.0
67.4
67.5
8.2
15.6
17.9
8.6
15.9
GLoRA[6]
35.8
46.9
63.4
59.3
67.5
67.8
9.8
14.8
18.8
10.6
15.6
GLoRIA-MIME [6]
37.4
57.1
64.0
60.3
68.7
68.3
11.6
16.1
24.8
8.90
16.6
MGCA" [7]
49.7
59.3
64.2
63.0
68.3
69.8
12.9
16.8
24.9
12.1
19.2
MedKLIP* [8]
50.2
60.8
63.9
66.2
69.4
71.9
8.9
16.3
24.5
7.1
11.6
Ours
56.7
62.2
64.4
72.6
74.4
76.7
16.6
22.3
31.1
6.6
13.3
21.6
Table 4: Ablation study of Med-UniC on linear classification, semantic segmentation and zero-shot classification. The best results of each setting are boldface.
Learning Objective
COVIDx (ACC)
RSNA (AUC)
SIMM (Dice)
CXP500 (F1)
PDC
CVL
CTR MLM
156
10%
100%
1%
10%
100%
10%
100%
En
Sp
Sp
72.8
85.5
91.8
87.7
88.5
89.4
51.9
56.5
58.7
63.5
59.8
62.2
58.5
74.5
92.2
88.1
89.3
89.9
68.5
64.6
75.0
84.3
92.5
88.2
89.6
89.7
53.8
59.6
61.5
70.3
65.9
65.4
63.7
76.5
92.8
89.1
90.4
64.4
75.4
72.9
cultivated by Med-UniC. Furthermore, this implies that unifying cross-lingual text representations via CTR can also improve the performance of uni-modal visual tasks.
Medical Image Semantic Segmentation and Object Detection In Tab 3, we further assessed the representation acquired by Med-UniC on segmentation and detection tasks. Impressively, Med-UniC outperforms all SOTA methods across every data fraction in all four tasks. Notably, Med-UniC achieves Dice score of 72.6% on RSNA segmentation with only 1% data fine-tuning, exceeding the performance of all other SOTA methods fine-tuned on 100% data. Furthermore, Med- UniC reaches a 6.6% mAP on the Object- dataset using just 1% data fine-tuning, surpassing other methods that barely achieve a 1% mAP. These findings further highlight the advantageous effects of unifying cross-lingual representations on vision-language tasks and uni-modal visual tasks.
4.4 Ablation Study and Bias Analysis
In this section, we ablate components of Med-UniC and present their performance on linear classification, zero-shot image classification, and semantic segmentation in Table 4. In all tasks, the combinations of all learning objectives achieve the highest performance, highlighting each com- ponent's s crucial role in Med-UniC. Med-UniC, when in- tegrated with CTR, significantly outperforms the version with MLM zero-shot tasks and nearly all uni-modal visual tasks.
Med-Unic with CTR
Med-Unic wio CTR
To further investigate the influence of CTR and MLM, we randomly select 1000 English and 1000 Spanish image- text pairs. We then illustrate the text and image embed- dings in Fig and Fig 4, respectively. Surprisingly, when the text encoder is pre-trained with MLM, we notice that the medical report embeddings tend to cluster by their re- spective languages, as shown in Fig lb. However, when employing CTR, the embeddings from diverse language reports draw nearer, leading to reduction in the distance between the two clusters compared the embeddings produced by MLM, as shown in Fig 1c.1d. This clear pattern illustrates CTR's 's ability to unify cross-lingual text representation within the latent space. Intriguingly, when pre-trained with CTR, the image embeddings become less distinguishable by their language community. In contrast, they form separate clusters according to their language community when pre-training does not involve CTR. This observation implies that community bias affects not only the text modality but also the
(a)
(b)
Figure 4: Visualization of image embed-
dings with or without CTR.
8

---

visual modality, shown in Fig 4. Consequently, the reduction in community bias contributes to superior performance across all tasks when Med-UniC is pre-trained with all learning objectives. More details can be found in the Appendix.
4.5 Further Analysis
Visualization of Language-agnostic Visual-textual Attention utilize Grad-CAM [51] to illustrate the areas in CXR images corresponding to various disease terminologies in both English and Spanish, as shown in Fig 5. Evidently, Med-UniC can accurately capture relevant regions that align closely with the indicated disease, exhibiting robustness across various languages. Consequently, the consistent cross- -ingull visual-textual representations cultivated by Med-Uni underscore its outstanding generalizability across multiple downstream tasks and language communities.
Ground Truth
Cardiomegaly
Cardiomegalin
Ground Truth
Edema
Edem
Ground Truth
Pleural Effusion
Derrame pleural
Ground Truth
Atelectasis
Atelectasi
Figure 5: Attention heat-maps of the visual-textual association learned by Med-UniC, compared with ground truth annotations provided by clinicians. The blue and orange words denote Spanish and English types, respectively.
Table 5: Comparison with large vision model.
Table 6: Results of Med-UniC with dif- ferent numbers of frozen layers.
CheXpert
RSNA
COVIDx
Method
10%:
100%:
10%
100%
10%
100%
COVIDx (AUC)
RSNA (Dice)
DINOv2 [52]
81.6
83.2
84.5
86.3
85.0
92.2
Frozen layers
Med-UniC(ViT--/116
89.7
90.8
93.1
93.7
89.5
94.5
0
87.7
92.5
75.0
75.8
Med-UIniC(VTTAL/9))
90.5
91.2
93.8
94.5
91.8
95.2
9
89.0
92.8
74.4
76.7
Comparison with Large Vision Model a comparison with DINOv2 [52], a large vision model trained on 1.2 billion images and comprising billion parameters, Med-UniC outperforms it in linear classification tasks across all datasets, using different data ratios and two ViT backbones, as shown in Tab 5. This demonstrates the remarkable effectiveness of Med-UniC, even in scenarios with limited domain-specife data.
Impact of Frozen Layers for LM To investigate the impact of freezing layers in LM, we experi- mented with freezing various numbers (0, 6, 9, 12) in 12-layer LM. Tab shows that updating the last three layers offers better results comparable to updating more layers, hinting that updating more might cause catastrophic forgetting [26, 53] of cross-lingual MLM-acquired semantics. Performance declined when all layers were frozen, indicating the necessity of properly updating layers.
Error Bars We conducted three separate runs of Med-UniC, using different random seeds and ResNet-50 as the vision backbone for three tasks. We then reported the average metric and its standard deviation. As indicated Tab 7, the variability between different task runs is relatively minor, demonstrating that Med- UniC consistently performs well in a variety of experiment configurations.
Table 7: Error bars analysis of linear classification, semantic segmentation, and object detection.
Ratio
COVIDx(ACC)
RSNA(Dice)
Object CXR(mAP)
1%
76.54=0.32
72.60+10.33
6.62=0.67
10%
89.01=0.16
74.43+40.19
13.34-10.27
100%
92.82=0.35
76.67:1045
21.6311.24

---

5 Conclusion
This work the first identify community bias in medical VLP stemming from diverse language communities and illustrates its negative impact on various downstream tasks. We present Med-UniC, a novel Pross-lingual medical VLP framework, along with CTR, intended to unify lonsu-l text representations and mitigate language-driver community bias. This bias impacts both text and visual modalities, thereby affecting performance across lisoonlanguuag and uni-modal visual tasks. The superior performance of Med-UniC across various tasks and data ratios underscores its efficiency and effectiveness. Through comprehensive ablation studies, we show that CTR significantly enhances performance in both vision-language and uni-modal visual tasks by effectively reducing community bias. This study provides a robust cross-lingual medical VLP framework and emphasizes the importance of inclusivity beyond English- -peaiing communities.
References
[1] M. Ni, H. Huang. L. Su, E. Cui, T. Bharti, L. Wang. D. Zhang, and N. Duan, "M3p: Learning universal representations multitask multilingual multimodal pre-tree tt-ttraniing" " in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 3977-3986.
[2] M. Zhou, L. Zhou. S. Wang. Y. Cheng. Li, Z. Yu, and Liu, "Ue2: Universal cross-lingual cross-modal lain--and-language pre-training." " Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021, pp. 4155-4165.
[3] A. Jain, M. Guo, Srimivasan, T. Chen, S. Kudugunta, C. Jia, Y. Yang. and Baldridge, "MURAL: Multimodal, multitask representations across languages," in Findings of the Association for Computational Linguistics: EMNLP 2021, Nov. 2021.
[4] Y. Zeng, W. Zhou, A. Luo, and X. Zhang, "Cross-view language modeling: Towards unified cross-lingual cross-modal pre-training." arXiv preprint 2rXiv:2206.00622 2022.
[5] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and Langlotz. "Contrastive learning of medical visual representations from paired images and text," arXiv preprint arXiv:2010.00747, 2020.
[6] S.-C. Huang, I. Shen, M. P. Lungren, and S. Yeung, "Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition," in Proceedings of the IFEF/CVF International Conference on Computer Vision, 2021, pp. 3942-3951.
[7] F. Wang. Y. Zhou, S. Wang, Vardhanabhuti, and L. Yu, "Multi "Multi granularity cross- modal alignment for generalized medical visual representation learning." arXiv preprint arXiv:2210.00444, 2022.
[8] C. Wu, X. Zhang, Zhang, Y. Wang, and W. Xie, "Medklip: Medical knowledge enhanced language-iman pre- pre-training," medRxiv, pp. 2023-00 01, 2023.
[9] H.-Y. Zhou, C. Lian, L. Wang, and Y. Yu, "Advancing radiograph representation learning with masked record modeling," " in The Eleventh International Conference on Learning Representations.
[10] S. Long, F. Cao, S. C. Han, and H. Yang. "Vision-and-languuee pretrained models: A survey."
[11] OpenAI, "Gpt-4 technical report," ArXiv, vol. abs/2303.08774, 2023.
[12] A. Kirillov, Mintun, N. Ravi, H. Mao, C. Rolland, L., Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., "Segment anything," arXiv preprint arXiv:2304.026433 2023.
[13] M. Oquab, T. Darcet, Moutakanni, Vo, M. Szafraniec, Khalidov, Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., "Dinov2: Learning robust visual features without supervision," arXiv preprint arXiv:2304.07193 2023.
[14] A. Gilson, C. Safranck, T. Huang, V. Socrates, L. Chi, R. A. Taylor, and D. Chartash, "How well does chatgpt do when taking the medical licensing exams? the implications of large language models for medical education and knowledge assessment," medRxiiv, pp. 2022 12, 2022.
[15] T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L.. De Lcon, C. Elepaño, M. Madriaga, R. Aggabao, G. Draz--andidoo. J. Maningo et al., "Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models," " PLoS digital health, vol. no. 2, p. e0000198, 2023.
[16] Y. Huang. X. Yang. L. Liu, H. Zhou, A. Chang, X. Zhou, R. Chen, J. Yu, J. Chen, C. Chen et al., "Segment anything model for medical images?" arXiv preprint 2rX.v:2304.14660, 2023.
10

---

[17] Ji, J. Q. Bi. W. Li, and L. Cheng, "Segment anything is not always perfect: An investigation of sam different real-world applications." arXiv preprini arXiv:2304.05750, 2023.
[18] S. He, R. Bao, J. Li, P. E. Grant, and Y. Ou, "Accuracy of segment-anything model (sam) in medical image segmentation tasks," " arXiv preprint arXiv:2.04..0 00..49244 2023.
[19] N. Mu, A. Kirillov, D. Wagner, and S. Xie, "Slip: Self-supervision meets language-image pre-training," in European Conference on Computer Vision. Springer, 2022, pp. 529-544.
[20] B. Boecking, N. Usuyama, S. Bannur, D. C. de Castro, A. Schwaighofer. S. L. Hyland, M. T. A. Wetscherek,
T. Naumann, A. Nori, Alvarez-Valle, H. Poon, and O. Oktay, "Making the most of text semantics to improve biomedical vision- ssiinnlanguage processing," ArXiv, ECCV.
[21] A. Bustos, A. Pertusa, J. M. Salinas, and M. de Iglesia- Vayá, "Padchest: large chest x-ray image dataset with multi-label annotated reports," Medical image analysis, vol. 66, p. 101797, 2019.
[22] A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. ying Deng, R. G. Mark, and S. Horg. "Mimic-exr, de-identified publicly available databasc of chest radiographs with free-text reports," Scientific Data, vol. 6, 2019.
[23] Irvin, Rajpurkar. M. Ko, Y. Yu, S. Ciurea-Ileus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya et al., "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison." in Proceedings of the AAAI conference artificial intelligence, vol. 33, 2019, pp. 597.
[24] J. Devlin, M.-W. Chang. K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." ArXiv, vol. abs//880004885, 2019.
[25] "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint 181.0488,, 2018.
[26] I. J. Goodfellow, M. Mirza. X. Da, A. C. Courville, and Y. Bengio, "An empirical investigation of catastrophic forgeting in gradient-based neural networks," CoRR, vol. abs/1312.6211, 2013.
[27] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark al., "Learning transferable visual models from natural language supervision." in International Conference on Machine Learning. PMLR, 2021, pp. 8748-8763.
[28] A. van den Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," " ArXiv, vol. Ahhs/180..0/ 007700378 2018.
[29] J. Singh, B. McCann, R. Socher, and C. Xiong, "Bert is not an interlingua and the bias of tokenization," in Conference on Empirical Methods in Natural Language Processing, 2019.
[30] R. Xian, Hi and H. Zhao, Cinguu-lingual transfer with wlesh-weighted language-invariant invariant representations"" in International Conference on Learning Representations, 2022.
[31] A. Lauscher, V. Ravishankar, Vulic, and G. Glavas, "From zero to hero: On the limitations of zero- shot language transfer with multilingual transformers," in Conference on Empirical Methods in Natural Language Processing, 2020.
[32] K. K. Z. Wang, S. Mayhew, and D. Roth, "Cross-linguu ability of multilingual bert: An empirical study." International Conference on Learning Representations, vol. abs/1912.07888 07.004800 2020.
[33] P. Dufter and H. Schütze, "Identifying elements essential for bert's multilinguality." in Conference on Empirical Methods in Natural Language Processing, 2020.
[34] J. Zhang and Z. Lan, "S-simese: Sampled sub-networks for contrastive learning of sentence embedding." Conference on Empirical Methods Natural Language, 2021.
[35] H. Barlow "Redundancy reduction revisited," Network: Computation in Neural Systems, vol. 12, pp. 241 -253, 2001.
[36] Zbontar, L. Jing, Misra, Y. LeCun, and S. Deny, "Barlow twins: Self-supervised learning via redundancy reduction," in International Conference on Machine Learning, 2021.
[37] S. Goel, H. Bansal, S. K. Bhatia, R. A. Rossi, V. Vinay, and A. Grover, "Cyclip: Cyclic contrastive language-image pretraining," NIPS, 2022.

---

[38] A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren. C.yy Deng. Y. Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng. "Mimic-cxt-jpg. large publicly available database of labeled chest radiographs," arXiv preprint arXiv:1901. 10000000 2019.
[39] A. Bustos, A. Pertusa, J.-M. Salinas, and M. de Iglesia- Vayá, "Padchest: A large chest x-ray image dataset with multi-label annotated reports." Medical image analysis, vol. 66, p. 101797. 2020.
[40] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[41] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, Heigold, S. Gelly et al. "An image is worth 16x16 words: Transformers image recognition at scale," arXiv preprini 2010v 19229, 2020.
[42] Loshchilov and E. Hutter, "Decoupled weight decay regularization." arXiv preprint a.1101171.005101. 2017.
[43] G. Shih, C. C. Wu, S. S. Halabi, M. D. Kohli, M. Prevedello, T. S. Cook, A. Sharma, K. Amorosa, V. Arteaga, M. Galperin-Aizenber el al., "Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia," Radiology: Artificial Intelligence, vol. 1, no. 1, p. e180041, 2019.
[44] L. Wang, Z. Q. Lin, and A. Wong, "Covid-net: A tailored deep convolutional neural network design for detection of covid-19 19 cases from chest x-ray images," Scientific reports, vol. 10, no. 1, pp. 1-12, 2020.
[45] A. Saporta, X. Gui, A. Agrawal, A. Pareek, S. Q. Truong. C. D. Nguyen, V.-D Ngo, J. Seekins. F. G. Blankenberg, A. Y. Ng et al., "Benchmarking saliency methods for chest x-ray interpretation." Nature Machine Intelligence, vol. 4, no. 10. pp. 867 878, 2022.
[46] C. Steven G. Langer, PhD and M. George Shih, MD, "Siim-aer pncumothorax segmentation." 2019.
[47] O. Ronneberger. P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmenta- tion," in Medical Image Computing and Computer-Asistedd Intervention-MICCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings. Part 18. Springer. 2015, pp. 234 234-241.
[48] J. Healthcare, (Ubject-cxr-autonee detection of foreign objects on chest x-rays," 2020.
[49] J. Redmon and A. Farhadi, "Yolov3: An incremental improvement," arXiv preprint arXiv: 2018.
[50] K. He, X. Chen, S. Xie, Li, P. Doll' ar, and R. B. Girshick, "Masked autoencoders scalable vision learners," 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15 979- 15 988, 2021.
[51] R. R. Selvaraju, A. Das, R. Vedantam, M. Cogswell, D. Parikh, and D. Batra, "Grad-cam: Visual explanations from deep networks via gradient-based localization," International Journal of Computer Vision, vol. 128, pp. 336-359, 2016.
[52] Oquab. T. Darcet, Moutakanni, H. Q. Vo, M. Szafraniec, Khalidov. P. Fernandez, D. Haziza. F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P-Y. Huang. S.-W. Li. I. Misra, M. G. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, Learning robust visual features without supervision." ArXiv, vol. abs/2304.07193, 2023.
"Dinov2:
[53] Z. Wan, Y. Yin, W. Zhang, J. Shi, L. Shang, G. Chen, X. Jiang, and Q. Liu, "G "G-map: General memory- augmented pre-trained language model for domain tasks," in Conference Empirical Methods in Natural Language Processing, 2022.
12


Reviews:
[{"review_id": "dAJP1hoxak", "reviewer": "Reviewer_YFc1", "summary": "The paper aims to address community bias caused by having data in multiple languages in medical vision-language pre-training (VLP). Specifically, it introduces the Unifying Cross-Lingual Medical Vision-Language Pre-Training framework to integrate multi-model data from English and Spanish and proposes a Criss-lingual Text Alignment Regularization (CTR) objective to unify cross-lingual semantic representations of medical reports from different languages. The paper shows superior performance compared to other methods on 5 tasks spanning 10 datasets and provides evidence of community bias in cross-lingual VLP through experimental findings. ", "questions": "-  How was the similarity in Figure 2 between medical reports obtained? What does the X-axis represent?", "limitations": "Authors have adequately addressed limitations of their work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Mitigating cultural bias in cross-lingual visual-language datasets in English/Spanish is a significant task with potential applications in the clinical field \n- The findings are novel. In particular, the separation between the learned representations in Med-Unic w/o CTR and the clear closer alignment in embedding space in Med-Unic with CTR is very interesting as it shows clearly that the representational spaces of the datasets in English and Spanish were separated without CTR and are better integrated now, which shows the alignment power of the CTR objective\n- Results - the paper obtains significant improvement over SOA techniques. \n- The CTR loss is an innovative method to address community bias in multi-lingual pre-training methodologies in a mathematically interesting way", "weaknesses": "- Looking at Figure 4, it is clear that the pre-training methodology proposed by the authors has significant improvements in bringing the learned representations from the two languages closer in the latent space. However, the two representations are still not perfectly aligned in the latent space, which means that the embeddings are still not perfectly integrated. Full integration would show Figure 4a where the two representations are fully overlapped, whereas now there is still separation between the two, with minimal overlap. While this is still a significant result, the model could probably be improved to better integrate and align the datasets.\n- I would like to understand if the authors think that with further training, these two representations can be further aligned, or if this is the maximum representational power capacity of the proposed pre-training methodology. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer YFc1\n### 1. Response for Weakness 1 and 2:\n> - Looking at Fig 4, it is clear that the pre-training methodology proposed by the authors has significant improvements in bringing the learned representations from the two languages closer in the latent space. However, the two representations are still not perfectly aligned in the latent space, which means that the embeddings are still not perfectly integrated. Full integration would show Fig 4a where the two representations are fully overlapped, whereas now there is still separation between the two, with minimal overlap. While this is still a significant result, the model could probably be improved to better integrate and align the datasets.\n> - I would like to understand if the authors think that with further training, these two representations can be further aligned, or if this is the maximum representational power capacity of the proposed pre-training methodology.\n\n\nWe appreciate your invaluable guidance regarding our research. In addressing the query about Fig 4, the sub-optimal fusion of image representations can be attributed to two fundamental factors. Initially, the two image sets, drawn from disparate communities, do not correspond to the same patient, thereby encapsulating distinct semantic interpretations. This factor could inevitably result in less than perfect integration of the representations. Following on from this, the images' origin diversity, covering a broad range from different hospitals, scanning devices, radiologists, and scanning methodologies, leads to what we identify as a `domain gap`. This inherent variability introduces additional challenges to the process of integration. However, the core thrust of our research is geared towards exploring biases introduced by language. Therefore, we do not forecast a fully congruent representation of images from two distinct communities.\n\n### 2. Response for Question 1:\n> How was the similarity in Figure 2 between medical reports obtained? What does the X-axis represent?\n\nThanks for your comment. For the similarity matrix shown in Fig 2 in the main paper, we randomly sample 250 Spanish and 250 English medical reports from both datasets. Then we utilise Cross-lingual Medical LM proposed in Sec.3.2 to obtain the [CLS] (a.k.a classification token) embedding of the texts. Therefore, X-axis represents the index of text embeddings, and the first 250 of these are Spanish samples, and the last 250 are English samples.\n"}, {"review_id": "6YmyLLJ1F5", "reviewer": "Reviewer_ZNkV", "summary": "One common challenge in performing medical vision-language pre-training (VLP) is data scarcity, especially in languages other than English. This challenge can be addressed by combining datasets from various languages to train language-agnostic models, but the authors empirically show that each language community (especially non-English ones) induces distinct linguistic biases in their data, even in language-agnostic models. The authors therefore introduce Med-UniC, which leverages cross-lingual text alignment regularization (CTR) (experimented with English and Spanish) to mitigate these biases and achieve SOTA results using chest X-ray scans and reports on many uni-modal visual tasks.", "questions": "see above.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tAppears to be novel, the idea of cross-lingual text alignment regularization (CTR) seems to effectively address a significant per-language bias problem in multilingual models based on self-vision, vision-language, and cross-lingual alignment strategies.\n•\tClear architecture explanation with hyperparameters with comprehensive experiments.\n•\tStrong results on all experiments.\n•\tGenerally well-written\n", "weaknesses": "The bias analysis section seems brief given how much attention it was given in the abstract/intro. The authors state that more analysis is in the appendix, but I would have wanted to see more in the main paper – I’m curious whether they attempted to identify the sources of the bias, or it would have been neat to see some samples with similar (language-agnostic) content but different embeddings due to this bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer ZNkV\n### 1. Response for Weaknesses:\n> The bias analysis section seems brief given how much attention it was given in the abstract/intro. The authors state that more analysis is in the appendix, but I would have wanted to see more in the main paper .\n\nThanks for your comment on our paper structure. We will modify the paper layout in the camera-ready version, and move the Sec D1, D3, D4 in the Appendix to the main paper to further explain the bias.\n\n\n> I’m curious whether they attempted to identify the sources of the bias, or it would have been neat to see some samples with similar (language-agnostic) content but different embeddings due to this bias.\n\nTo delve deeper into the origin of the bias, we randomly selected 20 reports from the English dataset, translated them into Spanish by native Spanish speakers, and created 20 English-Spanish text pairs which share the same semantic meaning. We computed the correlation coefficient of their embeddings, derived from the uni-lingual Bert, prior to our implementation of cross-lingual MLM pre-training. The correlation coefficient for each sample is depicted in $\\textbf{Fig B (left)}$ in the $\\textbf{attached PDF of authors rebuttal}$. It's noteworthy that the correlation coefficient for each English-Spanish pair (highlighted as the diagonal elements) does not approach 1.0. For enhanced visual clarity, we have plotted a histogram of the diagonal elements from the correlation coefficient matrix in $\\textbf{Fig B (right)}$ in the attached PDF of authors rebuttal, all of which fall below 0.65. This suggests that the uni-lingual Bert perceives the two language versions of the same report, despite having identical semantic meaning, as distinct texts.\n\nMoreover, following the 3D T-SNE method we used in $\\textbf{Fig A}$ in our main paper, we visualise the embeddings of Spanish and English reports generated from Uni-lingual Bert (CXR-Bert) in the $\\textbf{Fig A a1}$ and $\\textbf{Fig A a2}$ in the $\\textbf{attached PDF of authors rebuttal}$. From the $\\textbf{Fig A}$ in attached PDF, we can clearly observe that the center distance between two language cluster (language bias) is larger than Cross-lingual MLM and Med-UniC (e.g.,  $\\textbf{5.22}$(Uni-lingual bert) → $\\textbf{2.38}$(Cross-lingual Bert) → $\\textbf{0.31}$(Med-UniC)), demonstrating that language models pre-trained predominately on the English corpus produce obvious bias for other languages.\n"}, {"review_id": "06ctT5n3IL", "reviewer": "Reviewer_NLjC", "summary": "This paper presents a unified framework for Cross-Lingual Medical Vision-Language Pre-Training (Med-UniC), integrating multimodal medical data from different languages (e.g., English and Spanish). A Cross-lingual Text Alignment Regularization (CTR) is proposed to explicitly unify cross-lingual semantic representations of medical reports originating from diverse language communities. It reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases. ", "questions": "1. How are the SOTA methods implemented? What data set are they trained on? I wonder whether the comparison is fair if the SOTA methods and the proposed method are trained with different set of data. If the SOTA methods are only trained with one language, I wonder how much gain comes from the additional data.\n2. Table 1. Why for CXP500, most SOTA methods show a decrease in performance when the prompts transition from English to Spanish on CXP500, but an increase on PDC? The proposed method shows a consistent decrease in performance when the prompts transition from English to Spanish on both data sets.\n3. What does the learning objective “MLM” mean in Table 4? ", "limitations": "No discussion on limitations", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Practical and interesting problem setting, which attempts to unifying the Medical Vision-Language Pre-Training for multiple languages.\n2. The method is straight-forward and simple, making the paper easy to understand.\n3. The Cross-lingual Alignment loss seems reasonable.", "weaknesses": "1. The experimental setting is unclear. For example, the implementation and the training data of the SOTA methods are not introduced.\n2. The ablation study seems incomplete, only four settings are shown. It is hard to tell how much gain actually comes from the key design of the paper, i.e., the CTR loss. The biggest gain seems to come from the component \"MLM\", which is not introduced in the paper. I would also suggest the authors to provide ablation showing how much gain is from the increase in data quantity.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer NLjC:\n### 1.  Response for Weakness 1:\n\nWe sincerely appreciate your insightful feedback about our research. Concerning the results from other SOTA methods, these $\\textbf{were all pre-trained on MIMIC-CXR}$, the English dataset, with the exception of GLoRIA, which was pre-trained using in-house data. Results specifically pertaining to GLoRIA pre-trained on MIMIC-CXR are highlighted as GLoRIA-MIMIC. We directly adopted these results $\\textbf{from their original publications}$. Where experimental results were not documented in the original papers, we obtained their official checkpoints and used identical settings to those in our work. Thorough details of the implementation for pretrain and all downstream tasks can be found in appendix Sec B and C. The clear experimental setting with comprehensive experiments $\\textbf{are also acknowledged by}$ reviewer `ZNkV`.\n\n### 2. Response for Weakness 2:\n\n> The ablation study seems incomplete, only four settings are shown.\n\nPlease refer to $\\textbf{Tab 1 and 6 of the main paper}$ , and $\\textbf{Tab 4, 5, 6 , and 7 in the appendix}$. We have done additional ablation study for Med-UniC. \n\n- 1. We ablate three different visual backbones on the medical image classification task in Tab 1 of main paper.\n- 2. We ablate the impact on performance from the number of frozen layers in language model in Tab 6 of the main paper. \n- 3. We also further ablate three different visual backbones  on the medical image segmentation task in Tab 4 in appendix.\n- 4. We ablate  each sub-component's influence belonging to Cross-lingual Text Alignment Regularization (CTR), including the impact of text-feature alignment $L_{TF}$ and text-to-text alignment $L_{TT}$ in Tab 5 in appendix. \n- 5. We study the performance of only using the English dataset, MIMIC-CXR for pre-training without MLM but with CTR in Tab 6 of appendix.\n- 6. We analyse the dimension of the text alignment projector $\\mathcal{P}_{d}$ in Tab 7 in the appendix.\n\nHope those further ablation studies resolve your confusion.\n\n> It is hard to tell how much gain actually comes from the key design of the paper, i.e., the CTR loss.\n\nFrom $\\textbf{Tab 4}$ of main paper and $\\textbf{Tab A  in the attached PDF}$, we were able to clearly observe that CTR is a crucial component of our model. Compared to the full version of the model, without CTR causes more performance drop than without MLM (Masked language modelling). Besides, the superior performance of the full model proves that the CTR component is complementary to the MLM strategy, because we leverage MLM to initialise Med-UniC with the ability to process different languages and learn fundamental cross-lingual syntactic and semantic knowledge. But we find the model bias stemming from the language model (LM) pre-trained on predominantly English corpora. To eliminate language bias , we further optimise our model with CTR loss. Therefore, in Tab 4 of the main paper, the comparison of two rows (w/o CTR and with CTR) that the CTR loss leads to excellent performance gains.\n\n> The biggest gain seems to come from the component \"MLM\", which is not introduced in the paper.\n\nPlease refer to $\\textbf{Fig 1 or  Sec 3.2}$ in our paper, where we have described the definition of MLM (Masked language modelling).\n\n\n> I would also suggest the authors to provide ablation showing how much gain is from the increase in data quantity.\n\nPlease refer to $\\textbf{Appendix Sec D4: Med-UniC Pre-training on Uni-lingual data}$, \nThe ablation results are shown in Tab 6 in D4, which reveals that although using uni-lingual data to pre-train Med-UniC causes performance drop, compared to other baselines,our framework can also bring benefits. We attribute this to the text-to-text alignment to keep in-modal consistency and better extract text semantics.\n\n### 3.  Response for Question 1:\n\nTo fairly compare Med-UniC with other baselines without additional Spanish dataset, we have conducted an ablation study detailed in $\\textbf{Tab 6 in the Appendix}$. For this experiment, Med-UniC was solely pre-trained on MIMIC-CXR, and our model continues to significantly outperform all baselines across three disparate vision tasks. This result corroborates that the effectiveness of Med-Unic does not solely depend on the additional data, but also capitalises on the innovative CTR loss to further extract the nuanced, high-level semantics of the text.\n\n### 4.  Response for Question 2:\n\nWe are deeply appreciative of your constructive critique concerning our research. Initially, it is significant to note that despite a minor decline observed in the transition from English prompts to Spanish in the zero-shot classification task, our approach still holds superiority over all existing baselines. Additionally, referring to Table 1 in the primary manuscript, the F1 scores for most baselines consistently exhibit a downturn from English to Spanish on the PDC task. Even though there is a marginal ascent observed in the AUC scores from English prompts to Spanish, the values predominantly hover around 50. This statistic suggests a rather poor performance [1]. Hence, the minor increment observed could potentially be attributed to random fluctuations stemming from mis-tokenization, as elaborated in $\\textbf{Appendix Sec D.1}$.\n\nReference:\n\n[1] de Hond, et al.\"Interpreting area under the receiver operating characteristic curve.\" The Lancet Digital Health 4.12 (2022): e853-e855.\n\n\n### 5. Response for Question 3:\n\n> What does the learning objective “MLM” mean in Tab 4?\n\nThanks for your comments, MLM means Masked Language Modeling [1], and it is a type of language modelling task aimed to train a model to predict masked or hidden words in a sentence or text given the surrounding context to learn the semantic information. \n\nReference:\n\n[1] Kenton, et al. \"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.\" Proceedings of NAACL-HLT. 2019.\n\n\n"}, {"review_id": "KwJYYFSJpJ", "reviewer": "Reviewer_uyBA", "summary": "This paper proposes a vision language pretraining method that focusing on tackling the bias caused by different languages. The Text Alignment Regularization (CTR) is proposed to unify cross-lingual semantic representations of medical reports. The experiments show that the proposed CTR can effectively eliminate the bias between English and Spanish medical report. ", "questions": "Please refer to the concerns in the weakness section. ", "limitations": "The authors have adequately addressed the limitations", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper address an important and interesting issue of the bias caused by different languages in medical visual language pretraining tasks. \n2. The experiments show the  performance superiority of the proposed method on both normal medical recognition tasks and cross-linguistic tasks. ", "weaknesses": "1. The design of the proposed visual language model is closed to existing MLM and CLIP based VLP methods with a incremental improvement of CTR. More analysis is needed for the difference between existing methods and existing VLP methods. \n2. Seems only zero-shot classification tasks are evaluated under the cross-linguistic setting. Is it possible to evaluate other downstream task under this setting？\n3. Can you provide more analysis on why the proposed method outperform other medical VLP methods? It seems like the main difference of this method (CTR) does not really have a strong positive impact on single language recognition tasks. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer uyBA\n\n### 1. Response for Weakness 1:\n> The design of the proposed visual language model is closed to existing MLM and CLIP based VLP methods with a incremental improvement of CTR. More analysis is needed for the difference between existing methods and existing VLP methods.\n\nWe deeply appreciate your insightful critique of our research. We would like to highlight the overall novelty of our proposed approach from the following three aspects compared with existing VLP methods:\n\n- Our proposal to introduce Med-UniC marks a pioneering step in investigating $\\textbf{the bias influenced by language differences}$ within the field of medical VLP. This innovative approach $\\textbf{is acknowledged by reviewers}$  `YFc1` and `ZNkV`.\n\n- We have developed a novel CTR loss strategy aimed at improving the performance of VLP across various downstream tasks, by reducing this bias. To the best of our knowledge, Med-UniC $\\textbf{is the first initiative to identify and mitigate}$ language-driven bias in medical VLP through cross-lingual text alignment regularisation (CTR). This innovative approach to CTR design is appreciated by reviewers `ZNkV` and `YFc1`.\n\n- Mostly SOTA methods do not directly impose constraints on text embeddings, which consequently limits their ability to extract high-level semantics from text. For instance, MRM [1], despite utilising Masked Language Modeling (MLM) as a pre-training objective for the text, is mainly focused on reconstructing masked tokens rather than learning the overall embeddings of entire sentences [2]. As a solution, we propose the use of CTR to improve the learning of comprehensive sentence embeddings through contrastive learning for each text sample via $L_{TT}$. Additionally, CTR aids in disentangling the text's latent space through $L_{TF}$ to maximise the information on each latent dimension [3].\n\nReferences\n\n[1] Zhou, Hong-Yu, et al. \"Advancing Radiograph Representation Learning with Masked Record Modeling.\" The Eleventh International Conference on Learning Representations. 2023.\n\n[2] Neelakantan, Arvind (OpenAI), et al. \"Text and code embeddings by contrastive pre-training.\" arXiv preprint arXiv:2201.10005 (2022).\n\n[3] Zbontar, Jure, et al. \"Barlow twins: Self-supervised learning via redundancy reduction.\" International Conference on Machine Learning. PMLR, 2021.\n\n### 2.  Response for Weakness 2:\n> Seems only zero-shot classification tasks are evaluated under the cross-linguistic setting. Is it possible to evaluate other downstream task under this setting？\n\nWe greatly value your constructive feedback on our research. It's important to note that the other downstream tasks we have investigated - specifically, image classification, segmentation, and detection  $\\textbf{solely rely on images as input}$ , hence not necessitating any cross-lingual configurations.\n\n### 3.  Response for Weakness 3:\n\n> Can you provide more analysis on why the proposed method outperform other medical VLP methods? It seems like the main difference of this method (CTR) does not really have a strong positive impact on single language recognition tasks.\n\nWe would like to express our sincere gratitude for your thoughtful feedback regarding our research. As demonstrated in $\\textbf{Appendix Tab 6}$ , Med-UniC, equipped with the CTR loss and solely pre-trained on an English dataset, outperforms all baselines across three separate vision tasks. This is despite our configuration only utilising uni-lingual Bert and thus, $\\textbf{not implementing MLM loss}$ in this pretraining setting. This performance boost may result from the disentanglement effect that the CTR loss has on the text's latent space. This effect, in turn, allows the model to become more adept at extracting high-level semantics and representing features with greater effectiveness. Furthermore, we compare the backbone with CTR and without CTR in $\\textbf{Tab A of the attached PDF}$ in the author rebuttal. As the table shows, the CTR loss brings substantial improvement.\n"}]

Author rebuttal:
- We add the graphical explaination of the bias in $\textbf{Fig. A}$ and $\textbf{Fig. B}$  for reviewer `ZNkV`.
- We select the results from Tab 5 in the appendix D.2 and Tab 4 in the main paper to construct $\textbf{Tab. A}$ for reviewer `NLjC`. The $\textbf{Tab. A}$ shows the ablation experimental results of CTR loss.

Meta-review: This paper proposes a framework for medical vision-language pretraining for English and Spanish. The problem they attempt to solve is important and interesting, and the framework is sound. Although there are several limitations (as pointed out in the reviews) including clarity of experiments, mismatched focus on bias between abstract/intro and the main paper, these were adequately addressed in the rebuttal. I encourage the authors to revise the paper according to the rebuttal so that we can all better appreciate the important contributions of this paper.
Decision: Accept (poster)