Title: Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
Venue: NeurIPS 2023
Accepted: yes
Keywords: Medical Vision Langauge Pretraining

Cross-lingual

Language bias
Abstract: The scarcity of data presents a critical obstacle to the efficacy of medical vision-language pre-training (VLP). A potential solution lies in the combination of datasets from various language communities.
Nevertheless, the main challenge stems from the complexity of integrating diverse syntax and semantics, language-specific medical terminology, and culture-specific implicit knowledge. Therefore, one crucial aspect to consider is the presence of community bias caused by different languages.
This paper presents a novel framework named Unifying Cross-Lingual Medical Vision-Language Pre-Training (\textbf{Med-UniC}), designed to integrate multi-modal medical data from the two most prevalent languages, English and Spanish. 
Specifically, we propose \textbf{C}ross-lingual \textbf{T}ext Alignment \textbf{R}egularization (\textbf{CTR}) to explicitly unify cross-lingual semantic representations of medical reports originating from diverse language communities. 
\textbf{CTR} is optimized through latent language disentanglement, rendering our optimization objective to not depend on negative samples, thereby significantly mitigating the bias from determining positive-negative sample pairs within analogous medical reports. Furthermore, it ensures that the cross-lingual representation is not biased toward any specific language community.
\textbf{Med-UniC} reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases, offering a versatile framework for unifying multi-modal medical data within diverse linguistic communities.
The experimental outcomes highlight the presence of community bias in cross-lingual VLP. Reducing this bias enhances the performance not only in vision-language tasks but also in uni-modal visual tasks.

Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias

Abstract

The scarcity of data presents a critical obstacle to the efficacy of medical visionlanguage pre-training (VLP). A potential solution lies in the combination of datasets from various language communities. Nevertheless, the main challenge stems from the complexity of integrating diverse syntax and semantics, language-specific medical terminology, and culture-specific implicit knowledge. Therefore, one crucial aspect to consider is the presence of community bias caused by different languages. This paper presents a novel framework named Unifying Cross-Lingual Medical Vision-Language Pre-Training ( Med-UniC ), designed to integrate multimodal medical data from the two most prevalent languages, English and Spanish. Specifically, we propose C ross-lingual T ext Alignment R egularization ( CTR ) to explicitly unify cross-lingual semantic representations of medical reports originating from diverse language communities. CTR is optimized through latent language disentanglement, rendering our optimization objective to not depend on negative samples, thereby significantly mitigating the bias from determining positive-negative sample pairs within analogous medical reports. Furthermore, it ensures that the cross-lingual representation is not biased toward any specific language community. Med-UniC reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases, offering a versatile framework for unifying multi-modal medical data within diverse linguistic communities. The experimental outcomes highlight the presence of community bias in cross-lingual VLP. Reducing this bias enhances the performance not only in vision-language tasks but also in uni-modal visual tasks. The source code has been released at https://github.com/SUSTechBruce/Med-UniC.

1 Introduction

English, despite not being the primary native language for a vast majority of the global population, remains the dominant language in vision-language pre-training (VLP) datasets. Uni-lingual VLP models not only demonstrate restricted performance in cross-lingual tasks, but also bring the community bias on non-English speaking populations (displayed in Fig 1), particularly in the context of medical applications.

Researchers have used machine-translated non-English corpora and techniques like masked language model (MLM) and contrastive learning to unify cross-lingual representations [1-

4]. However, MLM-based representations still separate languages, as shown in Fig 1a and 1b. Additionally, as highlighted in Fig 2, the significant similarity among reports from diverse communities suggests that the strict reliance on negative samples in contrastive learning could introduce more bias during text alignment. Hence, Med-UniC focuses on learning text invariants in VLP using negative-free text alignment to avoid the bias mentioned above. In this work, we introduce a unified framework named Med-UniC , designed to acquire language-agnostic representations from chest xray (CXR) scans and associated radiology reports. Med-UniC learns the representation from 3 perspectives: visual invariants, visualtextual invariants, and text invariants. Considering medical visionlanguage tasks, such as zero-shot image classification, are dependent on semantic information and language-independent, we propose CTR ( C ross-lingual T ext Alignment R egularization) to explicitly minimize linguistic disparities in cross-lingual representations within the latent space, as visualized in Fig 1c and 1d. Consequently, Med-UniC tackles non-English vision-language tasks without the model bias stemming from the language model (LM) pre-trained on predominantly English corpora. Additionally, we found that the unified cross-lingual representation enhances performance across a

En

Sp

range of uni-modal visual tasks. This paper makes the following contributions:

To the best of our knowledge, we are the first to empirically identify the existence of community bias originating from diverse languages in cross-lingual medical VLP (displayed in Fig 1), and the negative impact of community bias on both uni-modal and vision-language downstream tasks.

We introduce the framework Med-UniC with CTR, designed to diminish community bias in medical VLP by unifying cross-lingual latent representations. Med-UniC achieves SOTA results in medical vision-language tasks across different languages, demonstrating its efficacy and broad applicability across various language communities.

Med-UniC achieves SOTA results on all uni-modal visual tasks. This highlights the advantages of mitigating community bias and unifying cross-lingual representations in medical VLP, enabling robust and comprehensive learning of visual representations.

Med-UniC effectively mitigates community bias without the requirement for manual curation or language-specific annotations. Importantly, Med-UniC enhances the accessibility of medical VLP to non-English speaking populations, circumventing potential biases that may arise from predominantly English datasets in traditional VLP.

2 Related Work

Medical VLP Complex medical reports and a shortage of large-scale medical image-text datasets have limited medical VLP research. Previous works such as ConVIRT [5] and GLoRIA [6] utilized contrastive objectives and global-local VLP to align image-text pairs. MGCA [7] used diseaselevel annotations for alignment, while MedKLIIP [8] manually extracted medically relevant entities.

ℒGLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

ℒGLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

′

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

′

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

aug

aug

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

′

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Shared

weights

ℒGLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

′

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Embeddings

dropout

Shared

weights

Vision-Vision

Alignment

Vision-Language

Alignment

Cross-lingual

Alignment

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

Sample-wise Alignment

Feature-wise Alignment

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc84)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc84)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc70)GLYPH(cmap:d835)GLYPH(cmap:dc85)GLYPH(cmap:d835)GLYPH(cmap:dc86)GLYPH(cmap:d835)GLYPH(cmap:dc8f) GLYPH(cmap:d835)GLYPH(cmap:dc74)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:dc86)GLYPH(cmap:dc95)

GLYPH(cmap:d835)GLYPH(cmap:dc70)GLYPH(cmap:d835)GLYPH(cmap:dc85)GLYPH(cmap:d835)GLYPH(cmap:dc86)GLYPH(cmap:d835)GLYPH(cmap:dc8f) GLYPH(cmap:d835)GLYPH(cmap:dc74)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:dc86)GLYPH(cmap:dc95)

ℒGLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

Images

Reports

+

+

... con derram

pleural bilateral

ascoi infiltr...

GLYPH(cmap:d835)

GLYPH(cmap:d835)

... persistent

bilateral pleural

effusions...

...

...

Frozen

Unfrozen

Sp

En

GLYPH(cmap:d835),GLYPH(cmap:d835)

MRM[9] replaced alignment with a reconstruction task involving masked visual and textual tokens. However, these methods fail to capitalize on valuable cross-lingual knowledge and do not effectively unify cross-lingual representations. As a result, their performance on cross-lingual tasks is considerably limited.

Cross-lingual VLP Most recent cross-lingual VLP research has focused on a limited set of languages, with a few comparing natural cross-lingual VLP to English-only VLP [10]. Studies such as [1-4] used machine-translated corpora and contrastive learning to generate language-agnostic representations. Yet, cross-lingual VLP in the medical domain remains unexplored mainly due to two main challenges: the limitations of machine translation for medical reports and the bias introduced by hard positive-negative pairs in contrastive learning. We propose CTR, a negative-free disentangled loss method, to unify cross-lingual text representations, addressing these issues.

Large General Model Recently, there have been impressive advances in large general models for language and vision tasks, such as ChatGPT [11], SAM, and DINOv2 [11-13]. However, these models still face significant limitations in the medical domain due to their lack of domain-specific knowledge and inability to jointly process visual and textual information [14-16]. ChatGPT [11] and SAM [12], currently limited to single-modality input, are unsuitable for vision-language tasks. While SAM excelled at instance segmentation, it struggled with medical image pathology segmentation [16-18]. Conversely, Med-UniC addressed these constraints by utilizing visual and textual data, integrating domain-specific knowledge without the high computational expenses of large general models. Surprisingly, Med-UniC outperformed the extensive vision model, DINOv2 [13], trained on 1.2B images with 1B parameters, by a considerable margin in multiple visual tasks. This suggests that Med-UniC is a more effective and efficient solution compared to large general models.

3 Method

3.1 Overall Framework of Med-UniC

Our Med-UniC framework aims to learn cross-lingual medical representation from CXR images and paired radiology reports. Given a training set of N cross-lingual dataset S ∈ V × L consisting of pairs ( v e,i , l e,i ) , where V and L are visual and text set, respectively, v e,i is a raw image and l e,i is a text report, e belongs to language domain (e.g., Spanish or English), i ∈ N . The Med-UniC architecture mainly consists of an image encoder F v : V ↦→ R D v to encoder the raw image into the embeddings with dimension D v , and a cross-lingual text encoder F l : L ↦→ R D l to encoder the text report to the embeddings with dimension D l . Then S = { ( v e, 1 , l e, 1 ) , ( v e, 2 , l e, 2 ) , . . . , ( v e,N , l e,N ) } , where v e,i = F v ( v e,i ) and l e,i = F l ( l e,i ) .

As depicted in Fig 3, Med-UniC incorporates three concurrent alignment strategies: cross-lingual vision-language alignment , self-supervised vision alignment , and cross-lingual text alignment regularization . For cross-lingual vision-language alignment, we extend the text encoder from uni-lingual into a cross-lingual style, which can equip the model with cross-lingual cross-modal ability by pulling the embeddings of paired image-reports pairs together while pushing apart embeddings of unmatched pairs, under the loss L CVL . Meanwhile, we employ self-supervised vision alignment, leveraging the

loss L SSV , to enhance the robustness of visual representation [19]. More importantly, we introduce a cross-lingual text alignment regularization, encompassing sample-level and feature-level approaches, to mitigate community bias stemming from different languages. This regularization is supervised by the loss L CTR . The learning objective of Med-UniC integrates the three strategies mentioned above and can be formulated as follows:

This training scheme compels Med-UniC to assimilate information from diverse perspectives, fostering a robust cross-modal, cross-lingual framework that concurrently learns visual invariants, visual-textual invariants, and text invariants.

3.2 Cross-lingual Vision-language Alignment

Cross-lingual Medical LM To initialize Med-UniC with the ability to process different languages and learn fundamental cross-lingual syntactic and semantic knowledge. We select CXR-BERT [20], a uni-lingual LM pre-trained on a large-scale biomedical corpus, as our text encoder and further adapted it for cross-lingual operation. The concrete steps proceed as follows: (1) Constructing a cross-lingual vocabulary set T : we collected the radiology reports of the second language (e.g., Spanish PadChest dataset [21]), which is seldom used for medical pre-training compared to English dataset [22, 23]. Then we leverage a general Spanish Spacy 3 to build a tokenizer and make use of TF-IDF tool 4 to generate M ranked new tokens T sp = { t 1 sp , t 1 sp , . . . , t M sp } according to their importance from multiple reports. (2) Building new wording embeddings W : we augment the original vocabulary T en with the sorted Spanish tokens to generate T = {T en , T sp } then expand a length of M random initialized vectors W sp on CXR-BERT's word embeddings W en , where W = [ W en ; W sp ] . (3) Masked Cross-lingual Modeling: we following BERT [24], randomly mixed English [22] and Spanish [21] medical reports as pre-train corpus. Then we randomly choose tokens in the mixed sequences, replace them with the [MASK] token and set 15 % masking probability as in BERT [25]. We selectively update several high layers to alleviate catastrophic forgetting [26] during vision-language pre-training. More details will be show in Sec 4.5.

Vision-Language Alignment Following CLIP framework [27], we incorporate a contrastive learning object to predict the matched pair ( v e,i , l e,i ) from N × N possible image-text pairs while mapping N 2 -N negative pairs far apart. Specifically, two non-linear visual and linguistic projectors P l and P v are used to transform v e,i and l e,i into the same dimension d , where ˆ v e,i = P v ( v e,i ) , ˆ l e,i = P l ( l e,i ) , and ˆ v e,i , ˆ l e,i ∈ R d . Obtaining image feature vectors [ ˆ v e,i ] N i =1 and text feature vectors [ ˆ l e,i ] N i =1 from a training batch, we compute cosine similarities s v 2 l i,i = ˆ v ⊤ e,i ˆ l e,i and s l 2 v i,i = ˆ l ⊤ e,i ˆ v e,i , representing image-text and text-image similarities, respectively. L CVL is then formulated as follows:

where L v 2 l v and L l 2 v l are image-text and text-image InforNCE [28] contrastive loss, respectively. σ 1 is the temperature hyper-parameter set to 0.07 in our experiments, K is the batch size for each step and K ∈ N . Through overall loss L CVL , the model learns maximal mutual information between the matched image-text pairs containing cross-lingual attributes within a batch.

3.3 Self-supervised Vision Alignment

To obtain more exhaustive visual representation, we include visual invariant learning as a parallel objective during VLP. Drawing inspiration from [19], we initially apply random augmentations (such as random cropping and flipping) to the original images to create augmented views as positive pairs [( v e,i , v ′ e,i )] N i =1 , while treating the rest of the images in the mini-batch as negative samples. We follow the data augmentation strategy as per [19]. Subsequently, we extract the representations of

the augmented views [ ˆ v ′ ] N i =1 using the vision projector p v and vision encoder F v , similar to [ ˆ v ] N i =1 . Therefore, the visual invariant learning objective becomes:

where σ 2 is the temperature hyper-parameter also set to 0.07 for overall loss objective L SSV .

3.4 Cross-lingual Text Alignment Regularization

As illustrated in Fig 1, and corroborated by research in natural language processing [29, 30], crosslingual text representations tend to form distinct clusters based on their respective languages. This trend introduces a community bias within data from different language communities, even when no explicit language attribute is provided. This suggests that VLP processes medical data based on their language community, risking unfairness in clinical applications and potentially decreasing downstream task performance in both uni-modal and vision-language tasks [31-33]. To mitigate this bias and potential risks, we introduce Cross-lingual Text Alignment Regularization (CTR) to learn language-independent text representations and neutralize the adverse effects of community bias on other modalities. CTR comprises three components:

Text augmentation We first adopt the dropout strategy [34] to generate augmented the text representation l ′ e,i from the text encoder and obtain the matched pairs [( l e,i , l ′ e,i )] N i =1 , and then a separate linguistic projector P d designed for de-correlation is leveraged to generate two different view pairs [( Z A e,i , Z B e,i )] N i =1 , where Z A e,i = P d ( l e,i ) , Z B e,i = P d ( l ′ e,i )) , and the new feature dimension D ′ > D .

Text-feature alignment To further alleviate information redundancy [35, 36] and obtain the shared cross-lingual text representation, we first normalize the augmented embedding pairs { Z A e , Z B e } ∈ R N × D ′ along the batch K dimension so that each feature dimension has a zero-mean and 1 / √ K standard deviation distribution to generate ˜ Z e , and then compute their cross-correlation ˜ Z corr e = ˜ Z AT e ˜ Z B e . The formulas of feature-dimension decorrelation can be defined as:

̸

The first term's objective is to learn a language-invariant representation by optimizing the diagonal elements of the correlation matrix to equal one. Simultaneously, the second term aims to shrink the cross-lingual gap and optimize information utilization in each latent dimension by driving the off-diagonal elements towards zero. Finally, We normalize the loss along with the feature dimension D ′ .

Text-to-text alignment : Similarly, the text-to-text alignment decorrelates the cross-correlation matrix along with feature dimension D ′ , and ˆ Z e is the normalized embeddings, ˆ Z corr e = ˆ Z A e ˆ Z BT e :

̸

where the text instance alignment term attempts to maximize the mutual information of a batch of cross-lingual text samples, and the text consistency regularizer can also be deemed as the text in-modal consistency [37] by reducing the mismatched text pairs into 0 in a batch K . Where λ in Eq 5, 6, is a non-negative hyperparameter trading off two terms. We also normalize the loss with the batch dimension K . Therefore, the loss of Cross-lingual Text Alignment Regularization L CTR is:

4 Experiments

4.1 Pre-training Configuration

Dataset We pre-train Med-UniC framework using MIMIC-CXR [38], which contains CXR images and their corresponding radiology reports in English. Also, we involve PadChest [39], which includes CXR images and their corresponding radiology reports collected in Valencia region, Spain. Both datasets are pre-processed following the approach described in [5-7], including image resizing, pixel value normalization, and text tokenization. Additionally, the dataset is filtered by excluding lateral views and reports with less than three tokens. This results in a pre-training dataset of approximately 220 k image-text pairs for MIMIC-CXR [38] and 160 k pairs for PadChest [39].

Implementation In the VLP stage, we employ ResNet-50 [40] and ViT [41] as the vision backbones. We report the linear classification results for these two vision encoders to illustrate the model-agnostic capabilities of Med-UniC. Med-UniC is trained over 50 epochs using an early stop strategy on 16 V100 GPUs with a batch size of 128 per GPU. We utilize AdamW [42] as the optimizer, setting the learning rate to 4 e -5 and the weight decay to 5 e -2 . Alinear warm-up and cosine annealing scheduler are also deployed in this process. Additionally, The coefficients λ is set to 5 . 1 e -3 following [36].

4.2 Downstream Tasks

Medical Image Linear Classification We perform this task on CheXpert [23], RSNA [43], and COVIDx [44] datasets. Following the previous work [5-7], we only update the parameter of a random initialized linear layer for classification and freeze the pre-trained vision backbone. We report the AUC scores (AUC) on CheXpert and RSNA and accuracy (ACC) on COVIDx as the evaluation metric following [6, 7].

Medical Image Zero-shot Classification We conduct this experiment on the CXP500 [45] and PDC [39] datasets, which comprise CXR images annotated by clinicians from English-speaking and Spanish-speaking countries, respectively. To circumvent prompt bias, we designate English positive prompt as '{ disease }' and negative prompt as 'No { disease }'. Prompts in Spanish are prepared by native Spanish speakers, with the disease indicated as '{ disease }' and the negative prompt represented as 'No hay { disease }'. Med-UniC is evaluated using both English and Spanish prompts across the two datasets, with additional experimental details provided in the Appendix. The results are represented as the macro average of AUC and F1 scores across all categories.

Medical Image Semantic Segmentation This task is performed on the RSNA [43] and the SIIM [46] datasets, following the data preprocessing in [6, 7]. Identical to [6, 7], the U-Net [47] fine-tuning settings are adopted for segmentation. All pre-trained vision backbones are considered as frozen encoders, and only the decoders of U-Net are updated during the fine-tuning. The segmentation performance is evaluated using Dice scores (Dice).

Medical Image Object Detection This task is performed on the RSNA [43] and Object-CXR [48] datasets, following the same preprocessing of [7]. Same as [7], we utilize YOLOv3 [49] as the detection architecture, using our pre-trained vision encoder as the backbone and only updating the detection head during fine-tuning. Mean Average Precision (mAP) with IOU thresholds 0.4 ∼ 0.75, is adopted to evaluate the detection task.

For all downstream tasks, except zero-shot classification, we fine-tune with 1% , 10% , 100% of the training data. More downstream tasks' settings, including split information and train/valid/test set details, can be found in the Appendix.

4.3 Comparison to the state-of-the-art

Zero-shot Classification To assess the cross-lingual visual-textual representation learned from MedUniC, we implement the zero-shot image classification task on two CXR datasets, which originate from distinct linguistic communities and utilize different language prompts. Tab 1 illustrates that Med-UniC surpasses all SOTA methods on both datasets, regardless of the language setting or the linguistic community data source. Across both datasets, Med-UniC delivers an average increase of over 20% in the F1 score when using English prompts and more than 15% when using Spanish

prompts. This showcases the effectiveness and adaptability of Med-UniC in managing cross-lingual vision-language tasks.

Interestingly, the AUC score of other SOTA methods experiences a substantial drop when the prompts transition from English to Spanish on CXP500 [45], a dataset collected from English-speaking communities. Similarly, all compared methods show comparably poor performance on PDC [39], a dataset derived from Spanish-speaking communities. MedKLIP [8], despite its commendable performance on the CXP500 [45] in the English setting due to its supervised pre-training with disease annotation, persistently shows a drop in performance on both the CXP500 [45] and PDC [39] when Spanish prompts are used, and also on the PDC [39] when using English prompts. These results highlight the significant community bias inherent in uni-lingual medical VLP models, even those utilizing annotations for pre-training. This bias adversely impacts the diagnostic quality when dealing with patients from non-English-speaking communities or those who do not speak English.

The unsatisfied performance of the compared methods also suggests that these models might incorporate linguistic community attributes during VLP, which negatively influences the learning of semantic meanings. Consequently, As a result, these models have difficulties in effectively interpreting CXR scans or prompts from non-English communities, which substantially limits the models' transferability. Further analysis can be found in Sec 4.4.

Medical Image Linear Classification To assess the quality of the visual representations learned by Med-UniC, we employ linear classification [50] on CheXpert [23], RSNA [43], and COVIDx [44]. As illustrated in Tab 2, Med-UniC framework consistently surpasses all uni-lingual pre-trained baselines across various settings and vision backbones. Significantly, even when MedKLIP [8] employs supervised VLP with disease-level annotation, Med-UniC consistently surpasses MedKLIP [8] across all tasks and settings. This exemplifies the adaptability and efficacy of the visual representations

cultivated by Med-UniC. Furthermore, this implies that unifying cross-lingual text representations via CTR can also improve the performance of uni-modal visual tasks.

Medical Image Semantic Segmentation and Object Detection In Tab 3, we further assessed the representation acquired by Med-UniC on segmentation and detection tasks. Impressively, Med-UniC outperforms all SOTA methods across every data fraction in all four tasks. Notably, Med-UniC achieves a Dice score of 72.6% on RSNA segmentation with only 1% data fine-tuning, exceeding the performance of all other SOTA methods fine-tuned on 100% data. Furthermore, Med-UniC reaches a 6.6% mAP on the Object-CXR dataset using just 1% data fine-tuning, surpassing other methods that barely achieve a 1% mAP. These findings further highlight the advantageous effects of unifying cross-lingual representations on vision-language tasks and uni-modal visual tasks.

4.4 Ablation Study and Bias Analysis

In this section, we ablate components of Med-UniC and present their performance on linear classification, zero-shot image classification, and semantic segmentation in Table 4.

In all tasks, the combinations of all learning objectives achieve the highest performance, highlighting each component's crucial role in Med-UniC. Med-UniC, when integrated with CTR, significantly outperforms the version with MLM in zero-shot tasks and nearly all uni-modal visual tasks.

To further investigate the influence of CTR and MLM, we randomly select 1000 English and 1000 Spanish imagetext pairs. We then illustrate the text and image embeddings in Fig 1 and Fig 4, respectively. Surprisingly, when the text encoder is pre-trained with MLM, we notice that the medical report embeddings tend to cluster by their respective languages, as shown in Fig 1a,1b. However, when employing CTR, the embeddings from diverse language

(a)

(b)

reports draw nearer, leading to a reduction in the distance between the two clusters compared to the embeddings produced by MLM, as shown in Fig 1c,1d. This clear pattern illustrates CTR's ability to unify cross-lingual text representation within the latent space. Intriguingly, when pre-trained with CTR, the image embeddings become less distinguishable by their language community. In contrast, they form separate clusters according to their language community when pre-training does not involve CTR. This observation implies that community bias affects not only the text modality but also the

visual modality, as shown in Fig 4. Consequently, the reduction in community bias contributes to superior performance across all tasks when Med-UniC is pre-trained with all learning objectives. More details can be found in the Appendix.

4.5 Further Analysis

Visualization of Language-agnostic Visual-textual Attention We utilize Grad-CAM [51] to illustrate the areas in CXR images corresponding to various disease terminologies in both English and Spanish, as shown in Fig 5. Evidently, Med-UniC can accurately capture relevant regions that align closely with the indicated disease, exhibiting robustness across various languages. Consequently, the consistent cross-lingual visual-textual representations cultivated by Med-UniC underscore its outstanding generalizability across multiple downstream tasks and language communities.

Ground Truth

Cardiomegaly        Cardiomegalia

Ground Truth

Pleural Effusion      Derrame pleural

Edema                       Edem

Ground Truth

Ground Truth

Atelectasis               Atelectasi

Comparison with Large Vision Model In a comparison with DINOv2 [52], a large vision model trained on 1.2 billion images and comprising 1 billion parameters, Med-UniC outperforms it in linear classification tasks across all datasets, using different data ratios and two ViT backbones, as shown in Tab 5. This demonstrates the remarkable effectiveness of Med-UniC, even in scenarios with limited domain-specific data.

Impact of Frozen Layers for LM To investigate the impact of freezing layers in LM, we experimented with freezing various numbers (0, 6, 9, 12) in a 12-layer LM. Tab 6 shows that updating the last three layers offers better results comparable to updating more layers, hinting that updating more might cause catastrophic forgetting [26, 53] of cross-lingual MLM-acquired semantics. Performance declined when all layers were frozen, indicating the necessity of properly updating layers.

Error Bars We conducted three separate runs of Med-UniC, using different random seeds and ResNet-50 as the vision backbone for three tasks. We then reported the average metric and its standard deviation. As indicated in Tab 7, the variability between different task runs is relatively minor, demonstrating that Med-UniC consistently performs well in a variety of experiment configurations.

5 Conclusion

This work is the first to identify community bias in medical VLP stemming from diverse language communities and illustrates its negative impact on various downstream tasks. We present Med-UniC, a novel cross-lingual medical VLP framework, along with CTR, intended to unify cross-lingual text representations and mitigate language-driven community bias. This bias impacts both text and visual modalities, thereby affecting performance across vision-language and uni-modal visual tasks. The superior performance of Med-UniC across various tasks and data ratios underscores its efficiency and effectiveness. Through comprehensive ablation studies, we show that CTR significantly enhances performance in both vision-language and uni-modal visual tasks by effectively reducing community bias. This study provides a robust cross-lingual medical VLP framework and emphasizes the importance of inclusivity beyond English-speaking communities.

References

M. Ni, H. Huang, L. Su, E. Cui, T. Bharti, L. Wang, D. Zhang, and N. Duan, 'M3p: Learning universal representations via multitask multilingual multimodal pre-training,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 3977-3986.

M. Zhou, L. Zhou, S. Wang, Y. Cheng, L. Li, Z. Yu, and J. Liu, 'Uc2: Universal cross-lingual cross-modal vision-and-language pre-training,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 4155-4165.

A. Jain, M. Guo, K. Srinivasan, T. Chen, S. Kudugunta, C. Jia, Y. Yang, and J. Baldridge, 'MURAL: Multimodal, multitask representations across languages,' in Findings of the Association for Computational Linguistics: EMNLP 2021 , Nov. 2021.

Y. Zeng, W. Zhou, A. Luo, and X. Zhang, 'Cross-view language modeling: Towards unified cross-lingual cross-modal pre-training,' arXiv preprint arXiv:2206.00621 , 2022.

Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, 'Contrastive learning of medical visual representations from paired images and text,' arXiv preprint arXiv:2010.00747 , 2020.

S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung, 'Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 3942-3951.

F. Wang, Y. Zhou, S. Wang, V. Vardhanabhuti, and L. Yu, 'Multi-granularity cross-modal alignment for generalized medical visual representation learning,' arXiv preprint arXiv:2210.06044 , 2022.

C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, 'Medklip: Medical knowledge enhanced language-image pre-training,' medRxiv , pp. 2023-01, 2023.

H.-Y. Zhou, C. Lian, L. Wang, and Y. Yu, 'Advancing radiograph representation learning with masked record modeling,' in The Eleventh International Conference on Learning Representations .

S. Long, F. Cao, S. C. Han, and H. Yang, 'Vision-and-language pretrained models: A survey.'

OpenAI, 'Gpt-4 technical report,' ArXiv , vol. abs/2303.08774, 2023.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al. , 'Segment anything,' arXiv preprint arXiv:2304.02643 , 2023.

M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al. , 'Dinov2: Learning robust visual features without supervision,' arXiv preprint arXiv:2304.07193 , 2023.

A. Gilson, C. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, and D. Chartash, 'How well does chatgpt do when taking the medical licensing exams? the implications of large language models for medical education and knowledge assessment,' medRxiv , pp. 2022-12, 2022.

T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepaño, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo et al. , 'Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models,' PLoS digital health , vol. 2, no. 2, p. e0000198, 2023.

Y. Huang, X. Yang, L. Liu, H. Zhou, A. Chang, X. Zhou, R. Chen, J. Yu, J. Chen, C. Chen et al. , 'Segment anything model for medical images?' arXiv preprint arXiv:2304.14660 , 2023.

W. Ji, J. Li, Q. Bi, W. Li, and L. Cheng, 'Segment anything is not always perfect: An investigation of sam on different real-world applications,' arXiv preprint arXiv:2304.05750 , 2023.

S. He, R. Bao, J. Li, P. E. Grant, and Y . Ou, 'Accuracy of segment-anything model (sam) in medical image segmentation tasks,' arXiv preprint arXiv:2304.09324 , 2023.

N. Mu, A. Kirillov, D. Wagner, and S. Xie, 'Slip: Self-supervision meets language-image pre-training,' in European Conference on Computer Vision . Springer, 2022, pp. 529-544.

B. Boecking, N. Usuyama, S. Bannur, D. C. de Castro, A. Schwaighofer, S. L. Hyland, M. T. A. Wetscherek, T. Naumann, A. Nori, J. Alvarez-Valle, H. Poon, and O. Oktay, 'Making the most of text semantics to improve biomedical vision-language processing,' ArXiv , ECCV.

A. Bustos, A. Pertusa, J. M. Salinas, and M. de la Iglesia-Vayá, 'Padchest: A large chest x-ray image dataset with multi-label annotated reports,' Medical image analysis , vol. 66, p. 101797, 2019.

A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. ying Deng, R. G. Mark, and S. Horng, 'Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports,' Scientific Data , vol. 6, 2019.

J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya et al. , 'Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,' in Proceedings of the AAAI conference on artificial intelligence , vol. 33, 2019, pp. 590-597.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, 'Bert: Pre-training of deep bidirectional transformers for language understanding,' ArXiv , vol. abs/1810.04805, 2019.

--,'Bert: Pre-training of deep bidirectional transformers for language understanding,' arXiv preprint arXiv:1810.04805 , 2018.

I. J. Goodfellow, M. Mirza, X. Da, A. C. Courville, and Y. Bengio, 'An empirical investigation of catastrophic forgeting in gradient-based neural networks,' CoRR , vol. abs/1312.6211, 2013.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , 'Learning transferable visual models from natural language supervision,' in International Conference on Machine Learning . PMLR, 2021, pp. 8748-8763.

A. van den Oord, Y. Li, and O. Vinyals, 'Representation learning with contrastive predictive coding,' ArXiv , vol. abs/1807.03748, 2018.

J. Singh, B. McCann, R. Socher, and C. Xiong, 'Bert is not an interlingua and the bias of tokenization,' in Conference on Empirical Methods in Natural Language Processing , 2019.

R. Xian, H. Ji, and H. Zhao, 'Cross-lingual transfer with class-weighted language-invariant representations,' in International Conference on Learning Representations , 2022.

A. Lauscher, V. Ravishankar, I. Vulic, and G. Glavas, 'From zero to hero: On the limitations of zeroshot language transfer with multilingual transformers,' in Conference on Empirical Methods in Natural Language Processing , 2020.

K. K, Z. Wang, S. Mayhew, and D. Roth, 'Cross-lingual ability of multilingual bert: An empirical study,' International Conference on Learning Representations , vol. abs/1912.07840, 2020.

P. Dufter and H. Schütze, 'Identifying elements essential for bert's multilinguality,' in Conference on Empirical Methods in Natural Language Processing , 2020.

J. Zhang and Z. Lan, 'S-simcse: Sampled sub-networks for contrastive learning of sentence embedding,' Conference on Empirical Methods in Natural Language , 2021.

H. B. Barlow, 'Redundancy reduction revisited,' Network: Computation in Neural Systems , vol. 12, pp. 241 - 253, 2001.

J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny, 'Barlow twins: Self-supervised learning via redundancy reduction,' in International Conference on Machine Learning , 2021.

S. Goel, H. Bansal, S. K. Bhatia, R. A. Rossi, V. Vinay, and A. Grover, 'Cyclip: Cyclic contrastive language-image pretraining,' NIPS , 2022.

A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y. Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng, 'Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs,' arXiv preprint arXiv:1901.07042 , 2019.

A. Bustos, A. Pertusa, J.-M. Salinas, and M. de la Iglesia-Vayá, 'Padchest: A large chest x-ray image dataset with multi-label annotated reports,' Medical image analysis , vol. 66, p. 101797, 2020.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' arXiv preprint arXiv:2010.11929 , 2020.

I. Loshchilov and F. Hutter, 'Decoupled weight decay regularization,' arXiv preprint arXiv:1711.05101 , 2017.

G. Shih, C. C. Wu, S. S. Halabi, M. D. Kohli, L. M. Prevedello, T. S. Cook, A. Sharma, J. K. Amorosa, V. Arteaga, M. Galperin-Aizenberg et al. , 'Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia,' Radiology: Artificial Intelligence , vol. 1, no. 1, p. e180041, 2019.

L. Wang, Z. Q. Lin, and A. Wong, 'Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images,' Scientific reports , vol. 10, no. 1, pp. 1-12, 2020.

A. Saporta, X. Gui, A. Agrawal, A. Pareek, S. Q. Truong, C. D. Nguyen, V.-D. Ngo, J. Seekins, F. G. Blankenberg, A. Y. Ng et al. , 'Benchmarking saliency methods for chest x-ray interpretation,' Nature Machine Intelligence , vol. 4, no. 10, pp. 867-878, 2022.

C. Steven G. Langer, PhD and M. George Shih, MD, 'Siim-acr pneumothorax segmentation,' 2019.

O. Ronneberger, P. Fischer, and T. Brox, 'U-net: Convolutional networks for biomedical image segmentation,' in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 . Springer, 2015, pp. 234-241.

J. Healthcare, 'Object-cxr-automatic detection of foreign objects on chest x-rays,' 2020.

J. Redmon and A. Farhadi, 'Yolov3: An incremental improvement,' arXiv preprint arXiv:1804.02767 , 2018.

K. He, X. Chen, S. Xie, Y. Li, P. Doll'ar, and R. B. Girshick, 'Masked autoencoders are scalable vision learners,' 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 15 97915 988, 2021.

R. R. Selvaraju, A. Das, R. Vedantam, M. Cogswell, D. Parikh, and D. Batra, 'Grad-cam: Visual explanations from deep networks via gradient-based localization,' International Journal of Computer Vision , vol. 128, pp. 336-359, 2016.

M. Oquab, T. Darcet, T. Moutakanni, H. Q. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. G. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, 'Dinov2: Learning robust visual features without supervision,' ArXiv , vol. abs/2304.07193, 2023.

Z. Wan, Y. Yin, W. Zhang, J. Shi, L. Shang, G. Chen, X. Jiang, and Q. Liu, 'G-map: General memoryaugmented pre-trained language model for domain tasks,' in Conference on Empirical Methods in Natural Language Processing , 2022.

Peer reviews:
comments: SUMMARY:
This paper proposes a vision language pretraining method that focusing on tackling the bias caused by different languages. The Text Alignment Regularization (CTR) is proposed to unify cross-lingual semantic representations of medical reports. The experiments show that the proposed CTR can effectively eliminate the bias between English and Spanish medical report.

STRENGTHS:
1. This paper address an important and interesting issue of the bias caused by different languages in medical visual language pretraining tasks. 
2. The experiments show the  performance superiority of the proposed method on both normal medical recognition tasks and cross-linguistic tasks.

WEAKNESSES:
1. The design of the proposed visual language model is closed to existing MLM and CLIP based VLP methods with a incremental improvement of CTR. More analysis is needed for the difference between existing methods and existing VLP methods. 
2. Seems only zero-shot classification tasks are evaluated under the cross-linguistic setting. Is it possible to evaluate other downstream task under this setting？
3. Can you provide more analysis on why the proposed method outperform other medical VLP methods? It seems like the main difference of this method (CTR) does not really have a strong positive impact on single language recognition tasks.

QUESTIONS:
Please refer to the concerns in the weakness section.

LIMITATIONS:
The authors have adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified framework for Cross-Lingual Medical Vision-Language Pre-Training (Med-UniC), integrating multimodal medical data from different languages (e.g., English and Spanish). A Cross-lingual Text Alignment Regularization (CTR) is proposed to explicitly unify cross-lingual semantic representations of medical reports originating from diverse language communities. It reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases.\n\nSTRENGTHS:\n1. Practical and interesting problem setting, which attempts to unifying the Medical Vision-Language Pre-Training for multiple languages.\n2. The method is straight-forward and simple, making the paper easy to understand.\n3. The Cross-lingual Alignment loss seems reasonable.\n\nWEAKNESSES:\n1. The experimental setting is unclear. For example, the implementation and the training data of the SOTA methods are not introduced.\n2. The ablation study seems incomplete, only four settings are shown. It is hard to tell how much gain actually comes from the key design of the paper, i.e., the CTR loss. The biggest gain seems to come from the component \"MLM\", which is not introduced in the paper. I would also suggest the authors to provide ablation showing how much gain is from the increase in data quantity.\n\nQUESTIONS:\n1. How are the SOTA methods implemented? What data set are they trained on? I wonder whether the comparison is fair if the SOTA methods and the proposed method are trained with different set of data. If the SOTA methods are only trained with one language, I wonder how much gain comes from the additional data.\n2. Table 1. Why for CXP500, most SOTA methods show a decrease in performance when the prompts transition from English to Spanish on CXP500, but an increase on PDC? The proposed method shows a consistent decrease in performance when the prompts transition from English to Spanish on both data sets.\n3. What does the learning objective “MLM” mean in Table 4?\n\nLIMITATIONS:\nNo discussion on limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
One common challenge in performing medical vision-language pre-training (VLP) is data scarcity, especially in languages other than English. This challenge can be addressed by combining datasets from various languages to train language-agnostic models, but the authors empirically show that each language community (especially non-English ones) induces distinct linguistic biases in their data, even in language-agnostic models. The authors therefore introduce Med-UniC, which leverages cross-lingual text alignment regularization (CTR) (experimented with English and Spanish) to mitigate these biases and achieve SOTA results using chest X-ray scans and reports on many uni-modal visual tasks.

STRENGTHS:
•	Appears to be novel, the idea of cross-lingual text alignment regularization (CTR) seems to effectively address a significant per-language bias problem in multilingual models based on self-vision, vision-language, and cross-lingual alignment strategies.
•	Clear architecture explanation with hyperparameters with comprehensive experiments.
•	Strong results on all experiments.
•	Generally well-written

WEAKNESSES:
The bias analysis section seems brief given how much attention it was given in the abstract/intro. The authors state that more analysis is in the appendix, but I would have wanted to see more in the main paper – I’m curious whether they attempted to identify the sources of the bias, or it would have been neat to see some samples with similar (language-agnostic) content but different embeddings due to this bias.

QUESTIONS:
see above.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to address community bias caused by having data in multiple languages in medical vision-language pre-training (VLP). Specifically, it introduces the Unifying Cross-Lingual Medical Vision-Language Pre-Training framework to integrate multi-model data from English and Spanish and proposes a Criss-lingual Text Alignment Regularization (CTR) objective to unify cross-lingual semantic representations of medical reports from different languages. The paper shows superior performance compared to other methods on 5 tasks spanning 10 datasets and provides evidence of community bias in cross-lingual VLP through experimental findings.\n\nSTRENGTHS:\n- Mitigating cultural bias in cross-lingual visual-language datasets in English/Spanish is a significant task with potential applications in the clinical field \n- The findings are novel. In particular, the separation between the learned representations in Med-Unic w/o CTR and the clear closer alignment in embedding space in Med-Unic with CTR is very interesting as it shows clearly that the representational spaces of the datasets in English and Spanish were separated without CTR and are better integrated now, which shows the alignment power of the CTR objective\n- Results - the paper obtains significant improvement over SOA techniques. \n- The CTR loss is an innovative method to address community bias in multi-lingual pre-training methodologies in a mathematically interesting way\n\nWEAKNESSES:\n- Looking at Figure 4, it is clear that the pre-training methodology proposed by the authors has significant improvements in bringing the learned representations from the two languages closer in the latent space. However, the two representations are still not perfectly aligned in the latent space, which means that the embeddings are still not perfectly integrated. Full integration would show Figure 4a where the two representations are fully overlapped, whereas now there is still separation between the two, with minimal overlap. While this is still a significant result, the model could probably be improved to better integrate and align the datasets.\n- I would like to understand if the authors think that with further training, these two representations can be further aligned, or if this is the maximum representational power capacity of the proposed pre-training methodology.\n\nQUESTIONS:\n-  How was the similarity in Figure 2 between medical reports obtained? What does the X-axis represent?\n\nLIMITATIONS:\nAuthors have adequately addressed limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}