SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Medical Image Object Detection This task is performed on the RSNA [43] and Object-CXR [48] datasets, following the same preprocessing of [7]. Same as [7], we utilize YOLOv3 [49] as the detection architecture, using our pre-trained vision encoder as the backbone and only updating the detection head during fine-tuning. Mean Average Precision (mAP) with IOU thresholds  $0.4\sim 0.75$  , is adopted to evaluate the detection task.
For all downstream tasks, except zero-shot classification, we fine-tune with  $1\%$ ,  $10\%$ ,  $100\%$  of the training data. More downstream tasks' settings, including split information and train-valid/test set details, can be found in the Appendix.
4.3 Comparison to the state-of-the-art
Zero-shot Classification To assess the cross-lingual visual-textual representation learned from Med-UniC, we implement the zero-shot image classification task on two CXR datasets, which originate from distinct linguistic communities and utilize different language prompts. Tab 1 illustrates that Med-UniC surpasses all SOTA methods on both datasets, regardless of the language setting or the linguistic community data source. Across both datasets, Med-UniC delivers an average increase of over  $20\%$  in the F1 score when using English prompts and more than  $15\%$  when using Spanish
Table 1: Zero-shot Image Classification results. F1 and AUC scores are reported. Best results of each setting are in boldface. 'En' and 'Sp' respectively stand for prompts in English and Spanish languages. Methods with  $\star$  leverage disease-level annotations for pre-training.
Method	CXP500(En)	CXP500(Sp)	PDC(En)	PDC(Sp)
AUC	F1	AUC	F1	AUC	F1	AUC	F1
ConVIRT[5]	59.5	19.2	60.5	15.8	45.1	26.5	49.1	12.6
GLoRIA[6]	43.2	2.4	40.2	16.1	52.3	10.1	50.3	8.2
GLoRIA-MIMIC [6]	46.2	5.5	51.5	20.3	53.1	12.1	52.2	11.3
MGCA* [7]	72.1	6.5	50.4	22.3	46.4	32.5	49.8	26.1
MedKILP* [8]	70.5	14.7	55.6	21.9	50.5	28.7	51.7	25.8
MRM [9]	65.2	10.4	48.3	16.1	50.1	24.6	51.4	25.1
Ours	75.4	30.3	71.3	32.2	72.9	42.6	71.4	37.1
prompts. This showcases the effectiveness and adaptability of Med-UniC in managing cross-lingual vision-language tasks.
Interestingly, the AUC score of other SOTA methods experiences a substantial drop when the prompts transition from English to Spanish on CXP500 [45], a dataset collected from English-speaking communities. Similarly, all compared methods show comparably poor performance on PDC [39], a dataset derived from Spanish-speaking communities. MedKLIP [8], despite its commendable performance on the CXP500 [45] in the English setting due to its supervised pre-training with disease annotation, persistently shows a drop in performance on both the CXP500 [45] and PDC [39] when Spanish prompts are used, and also on the PDC [39] when using English prompts. These results highlight the significant community bias inherent in uni-lingual medical VLP models, even those utilizing annotations for pre-training. This bias adversely impacts the diagnostic quality when dealing with patients from non-English-speaking communities or those who do not speak English.
The unsatisfied performance of the compared methods also suggests that these models might incorporate linguistic community attributes during VLP, which negatively influences the learning of semantic meanings. Consequently, As a result, these models have difficulties in effectively interpreting CXR scans or prompts from non-English communities, which substantially limits the models' transferability. Further analysis can be found in Sec 4.4.
Table 2: Linear classification results for CheXpert, RSNA, and COVIDx datasets with  $1\%$ ,  $10\%$ , and  $100\%$  training data. The best results are highlighted in bold. A standard ResNet-50 backbone is denoted as CNN-based. Methods with  $\star$  leverage disease-level annotations for pre-training.
Method	CheXpert (AUC)	RSNA (AUC)	COVIDx (ACC)
1%	10%	100%	1%	10%	100%	1%	10%	100%
Random Init	56.1	62.6	65.7	58.9	69.4	74.1	50.5	60.3	70.0
ImageNet Init	74.4	79.7	81.4	74.9	74.5	76.3	64.8	78.8	86.3
CNN-based
GLoRIA [6]	86.6	87.8	88.1	86.1	88.0	88.6	67.3	77.8	89.0
ConVIRT [5]	85.9	86.8	87.3	77.4	80.1	81.3	72.5	82.5	92.0
GLoRIA-MIMIC [6]	87.1	88.7	88.0	87.0	89.4	90.2	66.5	80.5	88.8
MedKLIP* [8]	86.2	86.5	87.7	87.3	88.0	89.3	74.5	85.2	90.3
MGCA* [7]	87.6	88.0	88.2	88.6	89.1	89.9	72.0	83.5	90.5
Med-UniC (ResNet-50)	88.2	89.2	89.5	89.1	90.4	90.8	76.5	89.0	92.8
ViT-based
MRM [9]	88.5	88.5	88.7	91.3	92.7	93.3	66.9	79.3	90.8
MGCA* (ViT-B/16) [7]	88.8	89.1	89.7	89.1	89.9	90.8	74.8	84.8	92.3
Med-UniC (ViT-B/16)	89.4	89.7	90.8	91.9	93.1	93.7	80.3	89.5	94.5
Med-UniC (ViT-L/32)	89.9	90.5	91.2	92.2	93.8	94.5	81.5	91.8	95.2
Medical Image Linear Classification To assess the quality of the visual representations learned by Med-UniC, we employ linear classification [50] on CheXpert [23], RSNA [43], and COVIDx [44]. As illustrated in Tab 2, Med-UniC framework consistently surpasses all uni-lingual pre-trained baselines across various settings and vision backbones. Significantly, even when MedKLIP [8] employs supervised VLP with disease-level annotation, Med-UniC consistently surpasses MedKLIP [8] across all tasks and settings. This exemplifies the adaptability and efficacy of the visual representations
Table 3: Results of semantic segmentation on SIIM and RSNA datasets and object detection on RSNA and Object-CXR datasets. The best results for each setting are highlighted in bold, and the ’-’ denotes mAP values smaller than  $1\%$ . Methods with  $\star$  leverage disease-level annotations.
Semantic Segmentation (Dice)	Object Detection (mAP)
Method	1%	SIIM 10%	100%	RSNA 1%	10%	100%	RSNA 10%	100%	100%	Object CXR 1%	10%	100%
Random	9.0	28.6	54.3	6.9	10.6	18.5	1.0	4.0	8.9	-	0.5	4.4
ImageNet	10.2	35.5	63.5	34.8	39.9	64.0	3.6	8.0	15.7	-	2.9	8.3
ConVIRT[5]	25.0	43.2	59.9	55.0	67.4	67.5	8.2	15.6	17.9	-	8.6	15.9
GLoRA[6]	35.8	46.9	63.4	59.3	67.5	67.8	9.8	14.8	18.8	-	10.6	15.6
GLoRIA-MIMIC [6]	37.4	57.1	64.0	60.3	68.7	68.3	11.6	16.1	24.8	-	8.90	16.6
MGCA* [7]	49.7	59.3	64.2	63.0	68.3	69.8	12.9	16.8	24.9	-	12.1	19.2
MedKLIP* [8]	50.2	60.8	63.9	66.2	69.4	71.9	8.9	16.3	24.5	-	7.1	11.6
Ours	56.7	62.2	64.4	72.6	74.4	76.7	16.6	22.3	31.1	6.6	13.3	21.6
Table 4: Ablation study of Med-UniC on linear classification, semantic segmentation and zero-shot classification. The best results of each setting are in boldface.
Learning Objective	COVIDx (ACC)	RSNA (AUC)	SIIM (Dice)	CXP500 (F1)	PDC (F1)
SSV	CVL	CTR	MLM	1%	10%	100%	1%	10%	100%	1%	10%	100%	En	Sp	En	Sp
✓	✓	72.8	85.5	91.8	87.7	88.5	89.4	51.9	56.5	58.7	63.5	59.8	62.2	58.5
✓	✓	✓	74.5	85.8	92.2	88.1	89.3	89.9	53.4	58.7	60.1	68.5	62.1	64.6	61.7
✓	✓	✓	75.0	84.3	92.5	88.2	89.6	89.7	53.8	59.6	61.5	70.3	65.9	65.4	63.7
✓	✓	✓	✓	76.5	89.0	92.8	89.1	90.4	90.8	56.7	62.2	64.4	75.4	71.3	72.9	71.4
cultivated by Med-UniC. Furthermore, this implies that unifying cross-lingual text representations via CTR can also improve the performance of uni-modal visual tasks.
Medical Image Semantic Segmentation and Object Detection In Tab 3, we further assessed the representation acquired by Med-UniC on segmentation and detection tasks. Impressively, Med-UniC outperforms all SOTA methods across every data fraction in all four tasks. Notably, Med-UniC achieves a Dice score of  $72.6\%$  on RSNA segmentation with only  $1\%$  data fine-tuning, exceeding the performance of all other SOTA methods fine-tuned on  $100\%$  data. Furthermore, Med-UniC reaches a  $6.6\%$  mAP on the Object-CXR dataset using just  $1\%$  data fine-tuning, surpassing other methods that barely achieve a  $1\%$  mAP. These findings further highlight the advantageous effects of unifying cross-lingual representations on vision-language tasks and uni-modal visual tasks.
4.4 Ablation Study and Bias Analysis
In this section, we ablate components of Med-UniC and present their performance on linear classification, zero-shot image classification, and semantic segmentation in Table 4.
In all tasks, the combinations of all learning objectives achieve the highest performance, highlighting each component's crucial role in Med-UniC. Med-UniC, when integrated with CTR, significantly outperforms the version with MLM in zero-shot tasks and nearly all uni-modal visual tasks.
To further investigate the influence of CTR and MLM, we randomly select 1000 English and 1000 Spanish image-text pairs. We then illustrate the text and image embeddings in Fig 1 and Fig 4, respectively. Surprisingly, when the text encoder is pre-trained with MLM, we notice that the medical report embeddings tend to cluster by their respective languages, as shown in Fig 1a,1b. However, when employing CTR, the embeddings from diverse language
reports draw nearer, leading to a reduction in the distance between the two clusters compared to the embeddings produced by MLM, as shown in Fig 1c,1d. This clear pattern illustrates CTR's ability to unify cross-lingual text representation within the latent space. Intriguingly, when pre-trained with CTR, the image embeddings become less distinguishable by their language community. In contrast, they form separate clusters according to their language community when pre-training does not involve CTR. This observation implies that community bias affects not only the text modality but also the
(b)
Figure 4: Visualization of image embeddings with or without CTR.
visual modality, as shown in Fig 4. Consequently, the reduction in community bias contributes to superior performance across all tasks when Med-UniC is pre-trained with all learning objectives. More details can be found in the Appendix.
Visualization of Language-agnostic Visual-textual Attention We utilize Grad-CAM [51] to illustrate the areas in CXR images corresponding to various disease terminologies in both English and Spanish, as shown in Fig 5. Evidently, Med-UniC can accurately capture relevant regions that align closely with the indicated disease, exhibiting robustness across various languages. Consequently, the consistent cross-lingual visual-textual representations cultivated by Med-UniC underscore its outstanding generalizability across multiple downstream tasks and language communities.
Cardiomegaly
Pleural Effusion
Cardiomegalia
Derrame pleural
Ground Truth
Ground Truth
Figure 5: Attention heat-maps of the visual-textual association learned by Med-UniC, compared with ground truth annotations provided by clinicians. The blue and orange words denote Spanish and English types, respectively.
Edem
Atelectasi
Table 5: Comparison with large vision model.
Method	CheXpert	RSNA	COVIDx
10%	100%	10%	100%	10%	100%
DINOv2 [52]	81.6	83.2	84.5	86.3	85.0	92.2
Med-UniC(ViT-B/16)	89.7	90.8	93.1	93.7	89.5	94.5
Med-UniC(ViT-L/32)	90.5	91.2	93.8	94.5	91.8	95.2
Table 6: Results of Med-UniC with different numbers of frozen layers.
Frozen layers	COVIDx (AUC)	RSNA (Dice)
10%	100%	10%	100%
0	87.7	92.5	75.0	75.8
6	88.0	92.3	74.3	76.3
9	89.0	92.8	74.4	76.7
12	87.1	90.5	72.3	74.4
Comparison with Large Vision Model In a comparison with DINOv2 [52], a large vision model trained on 1.2 billion images and comprising 1 billion parameters, Med-UniC outperforms it in linear classification tasks across all datasets, using different data ratios and two ViT backbones, as shown in Tab 5. This demonstrates the remarkable effectiveness of Med-UniC, even in scenarios with limited domain-specific data.
Impact of Frozen Layers for LM To investigate the impact of freezing layers in LM, we experimented with freezing various numbers (0, 6, 9, 12) in a 12-layer LM. Tab 6 shows that updating the last three layers offers better results comparable to updating more layers, hinting that updating more might cause catastrophic forgetting [26, 53] of cross-lingual MLM-acquired semantics. Performance declined when all layers were frozen, indicating the necessity of properly updating layers.
Error Bars We conducted three separate runs of Med-UniC, using different random seeds and ResNet-50 as the vision backbone for three tasks. We then reported the average metric and its standard deviation. As indicated in Tab 7, the variability between different task runs is relatively minor, demonstrating that Med-UniC consistently performs well in a variety of experiment configurations.
Table 7: Error bars analysis of linear classification, semantic segmentation, and object detection.
Ratio	COVIDx(ACC)	RSNA(Dice)	Object CXR(mAP)
1%	76.54±0.32	72.60±0.33	6.62±0.67
10%	89.01±0.16	74.43±0.19	13.34±0.27
100%	92.82±0.35	76.67±0.45	21.63±0.24
This work is the first to identify community bias in medical VLP stemming from diverse language communities and illustrates its negative impact on various downstream tasks. We present Med-UniC, a novel cross-lingual medical VLP framework, along with CTR, intended to unify cross-lingual text representations and mitigate language-driven community bias. This bias impacts both text and visual modalities, thereby affecting performance across vision-language and uni-modal visual tasks. The superior performance of Med-UniC across various tasks and data ratios underscores its efficiency and effectiveness. Through comprehensive ablation studies, we show that CTR significantly enhances performance in both vision-language and uni-modal visual tasks by effectively reducing community bias. This study provides a robust cross-lingual medical VLP framework and emphasizes the importance of inclusivity beyond English-speaking communities.