SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhongwei Wan $^{1*}$ , Che Liu $^{2*}$ , Mi Zhang $^{1\dagger}$ , Jie Fu $^{5\dagger}$ , Benyou Wang $^{4}$ , Sibo Cheng $^{2}$ , Lei Ma $^{3}$ , César Quilodrán-Casas $^{2}$ , Rossella Arcucci $^{2}$
<sup>1</sup>The Ohio State University <sup>2</sup>Imperial College London
3Peking University 4The Chinese University of Hong Kong, Shenzhen
<sup>5</sup>The Hong Kong University of Science and Technology
{wan.512, mizhang.1}@osu.edu, lei.ma@pku.edu.cn, wangbenyou@cuhk.edu.cn
{che.liu21, sibo.cheng, c.quilodran, r.arcucci}@imperial.ac.uk, jiefu@ust.hk
The scarcity of data presents a critical obstacle to the efficacy of medical vision-language pre-training (VLP). A potential solution lies in the combination of datasets from various language communities. Nevertheless, the main challenge stems from the complexity of integrating diverse syntax and semantics, language-specific medical terminology, and culture-specific implicit knowledge. Therefore, one crucial aspect to consider is the presence of community bias caused by different languages. This paper presents a novel framework named Unifying Cross-Linguual Medical Vision-Language Pre-Training (Med-UniC), designed to integrate multimodal medical data from the two most prevalent languages, English and Spanish. Specifically, we propose Cross-lingual Text Alignment Regularization (CTR) to explicitly unify cross-lingual semantic representations of medical reports originating from diverse language communities. CTR is optimized through latent language disentanglement, rendering our optimization objective to not depend on negative samples, thereby significantly mitigating the bias from determining positive-negative sample pairs within analogous medical reports. Furthermore, it ensures that the cross-lingual representation is not biased toward any specific language community. Med-UniC reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases, offering a versatile framework for unifying multi-modal medical data within diverse linguistic communities. The experimental outcomes highlight the presence of community bias in cross-lingual VLP. Reducing this bias enhances the performance not only in vision-language tasks but also in uni-modal visual tasks. The source code has been released at https://github.com/SUSTechBruce/Med-UniC.
English, despite not being the primary native language for a vast majority of the global population, remains the dominant language in vision-language pre-training (VLP) datasets. Uni-lingual VLP models not only demonstrate restricted performance in cross-lingual tasks, but also bring the community bias on non-English speaking populations (displayed in Fig 1), particularly in the context of medical applications.
Researchers have used machine-translated non-English corpora and techniques like masked language model (MLM) and contrastive learning to unify cross-lingual representations [1-
(a) 3D Text Embedding from MLM.
(b) 3D Clustering from MLM.
(c) 3D Text Embedding from Med-UniC.
Figure 1: Graphical illustration of community bias in MLM and Med-UniC is shown with blue points representing Spanish reports and orange points indicating English reports, visualized via T-SNE. MLM denotes cross-lingual masked language modeling.
(d) 3D Clustering from Med-UniC.
4]. However, MLM-based representations still separate languages, as shown in Fig 1a and 1b. Additionally, as highlighted in Fig 2, the significant similarity among reports from diverse communities suggests that the strict reliance on negative samples in contrastive learning could introduce more bias during text alignment. Hence, Med-UniC focuses on learning text invariants in VLP using negative-free text alignment to avoid the bias mentioned above.
In this work, we introduce a unified framework named Med-UniC, designed to acquire language-agnostic representations from chest x-ray (CXR) scans and associated radiology reports. Med-UniC learns the representation from 3 perspectives: visual invariants, visual-textual invariants, and text invariants. Considering medical vision-language tasks, such as zero-shot image classification, are dependent on semantic information and language-independent, we propose CTR (Cross-lingual Text Alignment Regularization) to explicitly minimize linguistic disparities in cross-lingual representations within the latent space, as visualized in Fig 1c and 1d. Consequently, Med-UniC tackles non-English vision-language tasks without the model bias stemming from the language model (LM) pre-trained on predominantly English corpora. Additionally, we found that the unified cross-lingual representation enhances performance across a range of uni-modal visual tasks. This paper makes the following co
Figure 2: Similarity matrix for medical reports embedding from MLM.
- To the best of our knowledge, we are the first to empirically identify the existence of community bias originating from diverse languages in cross-lingual medical VLP (displayed in Fig 1), and the negative impact of community bias on both uni-modal and vision-language downstream tasks.
- We introduce the framework Med-UniC with CTR, designed to diminish community bias in medical VLP by unifying cross-lingual latent representations. Med-UniC achieves SOTA results in medical vision-language tasks across different languages, demonstrating its efficacy and broad applicability across various language communities.
- Med-UniC achieves SOTA results on all uni-modal visual tasks. This highlights the advantages of mitigating community bias and unifying cross-lingual representations in medical VLP, enabling robust and comprehensive learning of visual representations.
- Med-UniC effectively mitigates community bias without the requirement for manual curation or language-specific annotations. Importantly, Med-UniC enhances the accessibility of medical VLP to non-English speaking populations, circumventing potential biases that may arise from predominantly English datasets in traditional VLP.
Medical VLP Complex medical reports and a shortage of large-scale medical image-text datasets have limited medical VLP research. Previous works such as ConVIRT [5] and GLoRIA [6] utilized contrastive objectives and global-local VLP to align image-text pairs. MGCA [7] used disease-level annotations for alignment, while MedKLIIP [8] manually extracted medically relevant entities.
Figure 3: Overview Med-UniC. CVL, SSV, CTR represent cross-lingual vision-language alignment, self-supervised vision alignment, Cross-lingual Text Alignment Regularization, respectively.
MRM [9] replaced alignment with a reconstruction task involving masked visual and textual tokens. However, these methods fail to capitalize on valuable cross-lingual knowledge and do not effectively unify cross-lingual representations. As a result, their performance on cross-lingual tasks is considerably limited.
Cross-lingual VLP Most recent cross-lingual VLP research has focused on a limited set of languages, with a few comparing natural cross-lingual VLP to English-only VLP [10]. Studies such as [1-4] used machine-translated corpora and contrastive learning to generate language-agnostic representations. Yet, cross-lingual VLP in the medical domain remains unexplored mainly due to two main challenges: the limitations of machine translation for medical reports and the bias introduced by hard positive-negative pairs in contrastive learning. We propose CTR, a negative-free disentangled loss method, to unify cross-lingual text representations, addressing these issues.
Large General Model Recently, there have been impressive advances in large general models for language and vision tasks, such as ChatGPT [11], SAM, and DINOv2 [11-13]. However, these models still face significant limitations in the medical domain due to their lack of domain-specific knowledge and inability to jointly process visual and textual information [14-16]. ChatGPT [11] and SAM [12], currently limited to single-modality input, are unsuitable for vision-language tasks. While SAM excelled at instance segmentation, it struggled with medical image pathology segmentation [16-18]. Conversely, Med-UniC addressed these constraints by utilizing visual and textual data, integrating domain-specific knowledge without the high computational expenses of large general models. Surprisingly, Med-UniC outperformed the extensive vision model, DINOv2 [13], trained on 1.2B images with 1B parameters, by a considerable margin in multiple visual tasks. This suggests that Med-UniC is a more effective and efficient solution compared to large general models.
3.1 Overall Framework of Med-UniC
Our Med-UniC framework aims to learn cross-lingual medical representation from CXR images and paired radiology reports. Given a training set of  $N$  cross-lingual dataset  $S \in \mathcal{V} \times \mathcal{L}$  consisting of pairs  $(v_{e,i}, l_{e,i})$ , where  $\mathcal{V}$  and  $\mathcal{L}$  are visual and text set, respectively,  $v_{e,i}$  is a raw image and  $l_{e,i}$  is a text report,  $e$  belongs to language domain (e.g., Spanish or English),  $i \in N$ . The Med-UniC architecture mainly consists of an image encoder  $\mathcal{F}_v: \mathcal{V} \mapsto \mathbb{R}^{D_v}$  to encoder the raw image into the embeddings with dimension  $D_v$ , and a cross-lingual text encoder  $\mathcal{F}_l: \mathcal{L} \mapsto \mathbb{R}^{D_l}$  to encoder the text report to the embeddings with dimension  $D_l$ . Then  $\mathbf{S} = \left\{\left(\mathbf{v}_{e,1}, \mathbf{l}_{e,1}\right), \left(\mathbf{v}_{e,2}, \mathbf{l}_{e,2}\right), \ldots, \left(\mathbf{v}_{e,N}, \mathbf{l}_{e,N}\right)\right\}$ , where  $\mathbf{v}_{e,i} = \mathcal{F}_v(v_{e,i})$  and  $\mathbf{l}_{e,i} = \mathcal{F}_l(l_{e,i})$ .
As depicted in Fig 3, Med-UniC incorporates three concurrent alignment strategies: cross-lingual vision-language alignment, self-supervised vision alignment, and cross-lingual text alignment regularization. For cross-lingual vision-language alignment, we extend the text encoder from uni-lingual into a cross-lingual style, which can equip the model with cross-lingual cross-modal ability by pulling the embeddings of paired image-reports pairs together while pushing apart embeddings of unmatched pairs, under the loss  $\mathcal{L}_{CVL}$ . Meanwhile, we employ self-supervised vision alignment, leveraging the
loss  $\mathcal{L}_{SSV}$ , to enhance the robustness of visual representation [19]. More importantly, we introduce a cross-lingual text alignment regularization, encompassing sample-level and feature-level approaches, to mitigate community bias stemming from different languages. This regularization is supervised by the loss  $\mathcal{L}_{CTR}$ . The learning objective of Med-UniC integrates the three strategies mentioned above and can be formulated as follows:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {C V L}} + \mathcal {L} _ {\mathrm {S S V}} + \mathcal {L} _ {\mathrm {C T R}} \tag {1}
$$
This training scheme compels Med-UniC to assimilate information from diverse perspectives, fostering a robust cross-modal, cross-lingual framework that concurrently learns visual invariants, visual-textual invariants, and text invariants.
3.2 Cross-lingual Vision-language Alignment
Cross-lingual Medical LM To initialize Med-UniC with the ability to process different languages and learn fundamental cross-lingual syntactic and semantic knowledge. We select CXR-BERT [20], a uni-lingual LM pre-trained on a large-scale biomedical corpus, as our text encoder and further adapted it for cross-lingual operation. The concrete steps proceed as follows: (1) Constructing a cross-lingual vocabulary set  $\mathcal{T}$ : we collected the radiology reports of the second language (e.g., Spanish PadChest dataset [21]), which is seldom used for medical pre-training compared to English dataset [22, 23]. Then we leverage a general Spanish Spacy  $^3$  to build a tokenizer and make use of TF-IDF tool  $^4$  to generate  $M$  ranked new tokens  $\mathcal{T}_{sp} = \{t_{sp}^{1}, t_{sp}^{1}, \ldots, t_{sp}^{M}\}$  according to their importance from multiple reports. (2) Building new wording embeddings  $\mathbf{W}$ : we augment the original vocabulary  $\mathcal{T}_{en}$  with the sorted Spanish tokens to generate  $\mathcal{T} = \{\mathcal{T}_{en}, \mathcal{T}_{sp}\}$  then expand a length of  $M$  random initialized vectors  $\mathbf{W}_{sp}$  on CXR-BERT's word embeddings  $\mathbf{W}_{en}$ , where  $\mathbf{W} = [\mathbf{W}_{en}; \mathbf{W}_{sp}]$ . (3) Masked Cross-lingual Modeling: we following BERT [24], randomly mixed English [22] and Spanish [21] medical reports as pre-train corpus. Then we randomly choose tokens in the mixed sequences, replace them with the [MASK] token and set  $15\%$  masking probability as in BERT [25]. We selectively update several high layers to alleviate catastrophic forgetting [26] during vision-language pre-training. More details will be show in Sec 4.5.
Vision-Language Alignment Following CLIP framework [27], we incorporate a contrastive learning object to predict the matched pair  $(v_{e,i},l_{e,i})$  from  $N\times N$  possible image-text pairs while mapping  $N^2 -N$  negative pairs far apart. Specifically, two non-linear visual and linguistic projectors  $\mathcal{P}_l$  and  $\mathcal{P}_v$  are used to transform  $\mathbf{v}_{e,i}$  and  $\mathbf{l}_{e,i}$  into the same dimension  $d$ , where  $\hat{\mathbf{v}}_{e,i} = \mathcal{P}_v(\mathbf{v}_{e,i}),\hat{\mathbf{l}}_{e,i} = \mathcal{P}_l(\mathbf{l}_{e,i})$  and  $\hat{\mathbf{v}}_{e,i},\hat{\mathbf{l}}_{e,i}\in \mathbb{R}^{d}$ . Obtaining image feature vectors  $[\hat{\mathbf{v}}_{e,i}]_{i = 1}^{N}$  and text feature vectors  $[\hat{l}_{e,i}]_{i = 1}^{N}$  from a training batch, we compute cosine similarities  $s_{i,i}^{2l} = \hat{\mathbf{v}}_{e,i}^{\top}\hat{\mathbf{l}}_{e,i}$  and  $s_{i,i}^{l2v} = \hat{\mathbf{l}}_{e,i}^{\top}\hat{\mathbf{v}}_{e,i}$ , representing image-text and text-image similarities, respectively.  $\mathcal{L}_{\mathrm{CVL}}$  is then formulated as follows:
$$
\mathcal {L} _ {v} ^ {v 2 l} = - \log \frac {\exp \left(s _ {i , i} ^ {v 2 l} / \sigma_ {1}\right)}{\sum_ {j = 1} ^ {K} \exp \left(s _ {i , j} ^ {v 2 l} / \sigma_ {1}\right)}, \mathcal {L} _ {i} ^ {l 2 v} = - \log \frac {\exp \left(s _ {i , i} ^ {l 2 v} / \sigma_ {1}\right)}{\sum_ {j = 1} ^ {K} \exp \left(s _ {i , j} ^ {l 2 v} / \sigma_ {1}\right)} \tag {2}
$$
$$
\mathcal {L} _ {\mathrm {C V L}} = \frac {1}{2 K} \sum_ {i = 1} ^ {N} \left(\mathcal {L} _ {v} ^ {v 2 l} + \mathcal {L} _ {l} ^ {l 2 v}\right), \tag {3}
$$
where  $\mathcal{L}_v^{v2l}$  and  $\mathcal{L}_l^{l2v}$  are image-text and text-image InforNCE [28] contrastive loss, respectively.  $\sigma_1$  is the temperature hyper-parameter set to 0.07 in our experiments,  $K$  is the batch size for each step and  $K\in N$ . Through overall loss  $\mathcal{L}_{\mathrm{CVL}}$ , the model learns maximal mutual information between the matched image-text pairs containing cross-lingual attributes within a batch.
3.3 Self-supervised Vision Alignment
To obtain more exhaustive visual representation, we include visual invariant learning as a parallel objective during VLP. Drawing inspiration from [19], we initially apply random augmentations (such as random cropping and flipping) to the original images to create augmented views as positive pairs  $[(v_{e,i},v_{e,i}^{\prime})]_{i = 1}^{N}$ , while treating the rest of the images in the mini-batch as negative samples. We follow the data augmentation strategy as per [19]. Subsequently, we extract the representations of
<sup>3</sup>https://spacy.io/models/es-dep-news-trf
$^{4}$ TidfVectorizer: https://scikit-learn.org/
the augmented views  $[\hat{\mathbf{v}}^{\prime}]_{i=1}^{N}$  using the vision projector  $p_v$  and vision encoder  $\mathcal{F}_v$ , similar to  $[\hat{\mathbf{v}}]_{i=1}^{N}$ . Therefore, the visual invariant learning objective becomes:
$$
\mathcal {L} _ {S S V} = - \frac {1}{K} \sum_ {j = 1} ^ {N} \log \frac {\exp \left(s _ {i , i} ^ {v 2 v ^ {\prime}} / \sigma_ {2}\right)}{\sum_ {j = 1} ^ {N} \exp \left(s _ {i , j} ^ {v 2 v ^ {\prime}} / \sigma_ {2}\right)}, s _ {i, i} ^ {v 2 v ^ {\prime}} = \hat {\mathbf {v}} _ {e, i} ^ {\top} \hat {\mathbf {v}} _ {e, i} ^ {\prime} \tag {4}
$$
where  $\sigma_{2}$  is the temperature hyper-parameter also set to 0.07 for overall loss objective  $\mathcal{L}_{SSV}$ .
3.4 Cross-lingual Text Alignment Regularization
As illustrated in Fig 1, and corroborated by research in natural language processing [29, 30], cross-lingual text representations tend to form distinct clusters based on their respective languages. This trend introduces a community bias within data from different language communities, even when no explicit language attribute is provided. This suggests that VLP processes medical data based on their language community, risking unfairness in clinical applications and potentially decreasing downstream task performance in both uni-modal and vision-language tasks [31-33]. To mitigate this bias and potential risks, we introduce Cross-lingual Text Alignment Regularization (CTR) to learn language-independent text representations and neutralize the adverse effects of community bias on other modalities. CTR comprises three components:
Text augmentation We first adopt the dropout strategy [34] to generate augmented the text representation  $\mathbf{l}_{e,i}^{\prime}$  from the text encoder and obtain the matched pairs  $[( \mathbf{l}_{e,i},\mathbf{l}_{e,i}^{\prime})]_{i = 1}^{N}$ , and then a separate linguistic projector  $\mathcal{P}_d$  designed for de-correlation is leveraged to generate two different view pairs  $[( \mathbf{Z}_{e,i}^{\mathbf{A}},\mathbf{Z}_{e,i}^{\mathbf{B}})]_{i = 1}^{N}$ , where  $\mathbf{Z}_{e,i}^{\mathbf{A}} = \mathcal{P}_d(\mathbf{l}_{e,i})$ ,  $\mathbf{Z}_{e,i}^{\mathbf{B}} = \mathcal{P}_d(\mathbf{l}_{e,i}^{\prime})$ , and the new feature dimension  $D^{\prime} > D$ .
Text-feature alignment To further alleviate information redundancy [35, 36] and obtain the shared cross-lingual text representation, we first normalize the augmented embedding pairs  $\{\mathbf{Z}_{\mathrm{e}}^{\mathbf{A}}, \mathbf{Z}_{\mathrm{e}}^{\mathbf{B}}\} \in \mathbb{R}^{N \times D'}$  along the batch  $K$  dimension so that each feature dimension has a zero-mean and  $1 / \sqrt{K}$  standard deviation distribution to generate  $\tilde{\mathbf{Z}}_{\mathrm{e}}$ , and then compute their cross-correlation  $\tilde{\mathbf{Z}}_{e}^{corr} = \tilde{\mathbf{Z}}_{e}^{\mathrm{AT}} \tilde{\mathbf{Z}}_{e}^{\mathrm{B}}$ . The formulas of feature-dimension decorrelation can be defined as:
$$
\mathcal {L} _ {T F} = \frac {1}{D _ {\prime}} \left\{\underbrace {\sum_ {j} ^ {D ^ {\prime}} \left(1 - \sum_ {i} ^ {K} \tilde {\mathbf {Z}} _ {e , i} ^ {A , j \mathbf {T}} \tilde {\mathbf {Z}} _ {e , i} ^ {B , j}\right) ^ {2}} _ {\text {c r o s s - l i n g u a l i n v a r i a n c e}} + \underbrace {\lambda \sum_ {j} ^ {D ^ {\prime}} \sum_ {i \neq j} ^ {K} \tilde {\mathbf {Z}} _ {e , i} ^ {A , j \mathbf {T}} \tilde {\mathbf {Z}} _ {e , i} ^ {B , j}} _ {\text {c r o s s - l i n g u a l g a p r e d u c t i o n}} \right\}, \quad \tilde {\mathbf {Z}} _ {e} = \frac {\mathbf {Z} _ {\mathrm {e}} - \mu_ {K} (\mathbf {Z} _ {e})}{\sqrt {K} \sigma (\mathbf {Z} _ {\mathrm {e}})} \tag {5}
$$
The first term's objective is to learn a language-invariant representation by optimizing the diagonal elements of the correlation matrix to equal one. Simultaneously, the second term aims to shrink the cross-lingual gap and optimize information utilization in each latent dimension by driving the off-diagonal elements towards zero. Finally, We normalize the loss along with the feature dimension  $D'$ .
Text-to-text alignment: Similarly, the text-to-text alignment decorrelates the cross-correlation matrix along with feature dimension  $D^{\prime}$ , and  $\hat{\mathbf{Z}}_e$  is the normalized embeddings,  $\hat{\mathbf{Z}}_e^{corr} = \hat{\mathbf{Z}}_e^{\mathbf{A}}\hat{\mathbf{Z}}_e^{\mathbf{B}\mathbf{T}}$ :
$$
\mathcal {L} _ {T T} = \frac {1}{K} \left\{\underbrace {\sum_ {j} ^ {K} \binom {D ^ {\prime}} {i} \hat {\mathbf {Z}} _ {e , i} ^ {A , j} \hat {\mathbf {Z}} _ {e , i} ^ {B , j , \mathbf {T}}} _ {\text {t e x t i n s t a n c e a l i g n m e n t}} + \underbrace {\lambda \sum_ {j} ^ {K} \sum_ {i \neq j} ^ {D ^ {\prime}} \hat {\mathbf {Z}} _ {e , i} ^ {A , j} \hat {\mathbf {Z}} _ {e , i} ^ {B , j , \mathbf {T}}} _ {\text {t e x t c o n s i s t e n c y r e g u l a r i z e r}} \right\}, \quad \hat {\mathbf {Z}} _ {e} = \frac {\mathbf {Z} _ {\mathbf {e}} - \mu_ {D ^ {\prime}} (\mathbf {Z} _ {e})}{\sqrt {D ^ {\prime}} \sigma (\mathbf {Z} _ {\mathbf {e}})} \tag {6}
$$
where the text instance alignment term attempts to maximize the mutual information of a batch of cross-lingual text samples, and the text consistency regularizer can also be deemed as the text in-modal consistency [37] by reducing the mismatched text pairs into 0 in a batch  $K$ . Where  $\lambda$  in Eq 5, 6, is a non-negative hyperparameter trading off two terms. We also normalize the loss with the batch dimension  $K$ . Therefore, the loss of Cross-lingual Text Alignment Regularization  $\mathcal{L}_{CTR}$  is:
$$
\mathcal {L} _ {C T R} = \mathcal {L} _ {T F} + \mathcal {L} _ {T T} \tag {7}
$$
4.1 Pre-training Configuration
Dataset We pre-train Med-UniC framework using MIMIC-CXR [38], which contains CXR images and their corresponding radiology reports in English. Also, we involve PadChest [39], which includes CXR images and their corresponding radiology reports collected in Valencia region, Spain. Both datasets are pre-processed following the approach described in [5-7], including image resizing, pixel value normalization, and text tokenization. Additionally, the dataset is filtered by excluding lateral views and reports with less than three tokens. This results in a pre-training dataset of approximately  $220k$  image-text pairs for MIMIC-CXR [38] and  $160k$  pairs for PadChest [39].
Implementation In the VLP stage, we employ ResNet-50 [40] and ViT [41] as the vision backbones. We report the linear classification results for these two vision encoders to illustrate the model-agnostic capabilities of Med-UniC. Med-UniC is trained over 50 epochs using an early stop strategy on 16 V100 GPUs with a batch size of 128 per GPU. We utilize AdamW [42] as the optimizer, setting the learning rate to  $4e^{-5}$  and the weight decay to  $5e^{-2}$ . A linear warm-up and cosine annealing scheduler are also deployed in this process. Additionally, the coefficients  $\lambda$  is set to  $5.1e^{-3}$  following [36].
4.2 Downstream Tasks
Medical Image Linear Classification We perform this task on CheXpert [23], RSNA [43], and COVIDx [44] datasets. Following the previous work [5-7], we only update the parameter of a random initialized linear layer for classification and freeze the pre-trained vision backbone. We report the AUC scores (AUC) on CheXpert and RSNA and accuracy (ACC) on COVIDx as the evaluation metric following [6, 7].
Medical Image Zero-shot Classification We conduct this experiment on the CXP500 [45] and PDC [39] datasets, which comprise CXR images annotated by clinicians from English-speaking and Spanish-speaking countries, respectively. To circumvent prompt bias, we designate English positive prompt as ‘{disease}’ and negative prompt as ‘No {disease}’. Prompts in Spanish are prepared by native Spanish speakers, with the disease indicated as ‘{disease}’ and the negative prompt represented as ‘No hay {disease}’. Med-UniC is evaluated using both English and Spanish prompts across the two datasets, with additional experimental details provided in the Appendix. The results are represented as the macro average of AUC and F1 scores across all categories.
Medical Image Semantic Segmentation This task is performed on the RSNA [43] and the SIIM [46] datasets, following the data preprocessing in [6, 7]. Identical to [6, 7], the U-Net [47] fine-tuning settings are adopted for segmentation. All pre-trained vision backbones are considered as frozen encoders, and only the decoders of U-Net are updated during the fine-tuning. The segmentation performance is evaluated using Dice scores (Dice).