KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model
Authors: Lei Geng
Xu Yan
Ziqiang Cao
Juntao Li
Wenjie Li
Sujian Li
Xinjie Zhou
Yang Yang
Jun Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Biomedical
Cross-lingual
Multi-lingual
Pretrained Language Model

Abstract: Most biomedical pretrained language models are monolingual and cannot handle the growing cross-lingual requirements. The scarcity of non-English domain corpora, not to mention parallel data, poses a significant hurdle in training multilingual biomedical models. Since knowledge forms the core of domain-specific corpora and can be translated into various languages accurately, we propose a model called KBioXLM, which transforms the multilingual pretrained model XLM-R into the biomedical domain using a knowledge-anchored approach. We achieve a biomedical multilingual corpus by incorporating three granularity knowledge alignments (entity, fact, and passage levels) into monolingual corpora. Then we design three corresponding training tasks (entity masking, relation masking, and passage relation prediction) and continue training on top of the XLM-R model to enhance its domain cross-lingual ability. To validate the effectiveness of our model, we translate the English benchmarks of multiple tasks into Chinese. Experimental results demonstrate that our model significantly outperforms monolingual and multilingual pretrained models in cross-lingual zero-shot and few-shot scenarios, achieving improvements of up to 10+ points.

KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained
Language Model
Lei Geng'
Xu Yan' Ziqiang Cao Juntao Li
Wenjie Li
Sujian Li
Xinjie Zhou' Yang Yang' Jun Zhang
*Peking University The Hong Kong Polytechnic University *Pharmcube Changping Laboratory suda. euda.edu.cm cn, (zqcao, 1jt]@suda... kjt]@suda.edu. cn
Innttitute of Artificial Intelligence, Soochow University
{20215227001, xyannlpJ@stu. petssstu.uud.eeee
cswjli@comp. polyu. polyu.edu.hir edu.hk, (lisujian, justinzhang edu. cn (zhouxinjie, Jongyangg)@harmmccule com
Abstract
WHIPEDIA
Most biomedical pretrained language models are monolingual and cannot handle the grow- ing cross-lingual requirements. The scarcity non- non-English domain corpora, not to men- tion parallel data, poses significant hurdle in training multilingual biomedical models. Since knowledge forms the core domain--pecific corpora and can be translated into various lan- guages accurately, we propose a model called KBioXLM, which transforms the multilingual pretrained model XLM-R into the biomedical domain using a knowledge-anchoree approach. We achieve a biomedical multilingual corpus by incorporating three granularity knowledge alignments (entity, fact, and passage levels) into monolingual corpora. Then we design three corresponding training tasks (entity mask- ing, relation masking, and passage relation pre- diction) and continue training top of the XLM-R model to enhance its domain cross- lingual ability. To validate the effectiveness of our model, we translate the English bench- marks of multiple tasks into Chinese. Experi- mental results demonstrate that our model sig- nificantly outperforms monolingual and mul- tilingual pretrained models in cross- looss-lingual zero-shot and few-shot scenarios, achieving improvements of up to 10+ points. Our code is publicly available at https:/ thub. com/ ngwlh- gaalh-gg/KBioXLM.
Luadamer continne agamereneet 100 aptzgpuggg pigzs ee 1%% sarphian
EN-12000
CN-9649907
Eatlise
Loaix aroodtu Mepl a Open papp Mi
UMLS
Rebbtleen:
(eptix popp. axcdud vtb. ww #12 (311
Enity Karwledge Aignneett St
[CLS] Lendanes occlars apprecizandy 10% apean squindeat in 1% mucphase....
Fact Kuowlddge Aiquncat
soliio comsdes sppcteeatty popps... (SEP)) (.2E779...
Pawage Kaswledge Alkysment:
Figure 1: The construction process of aligned biomed- ical data relies on three types of granular knowledge We begin with an English passage identified as 42030. which serves as the primary source. From this passage, we extract entities and relations using UMLS and we search for the corresponding aligned Chinese pas- sage 6640907 from Wikipedia. Combining these three granular knowledge sources, we construct aligned cor- pora. We will predict the relationship between the two passages using the "[CLS]" token.
ing tasks highlight the pressing need for cross- lingual capability. However, most biomedical PLMs focus on monolingual and cannot address cross-lingual requirements, while the performance of existing multilingual models in general domain fall far behind expectations (Devlin et al., 2018; Conneau et al., 2019; Chi et al., 2021). Multi- lingual biomedical models can effectively tackle cross-lingual tasks and monolingual tasks. There- fore, the development of a multilingual biomedical pretrained model is urgently needed!.
1 Introduction
In recent years, biomedical pretrained language models (PLMs) (Lee et al., 2020; Gu et al., 2020; Yasunaga et al., 2022) have made remarkable progress in various natural language processing (NLP) tasks. While the biomedical annotation data (Li et al., 2016; Dogan et al., 2014; Du al., 2019; Collier and Kim, 2004; Gurulingappa et al., 2012) are predominantly in English. Therefore, non- non-English biomedical natural language process-
Unlike general domains, there a scarcity of non-English biomedical corpora and even fewer parallel corpora in the biomedical domain, which presents significant challenge for training mul-
'Due to the lack of expertise in other languages, we have only conducted experiments in Chinese and English.
"Corresponding author.

---

tilingual biomedical models. In general domains, back translation (BT) (Sennrich et al., 2015) is commonly used for data augmentation. However, our experiments (refer to "XLM- Table 5) reveal that due to the quality issues of domain translation, back translation does not sig- nificantly improve multilingual biomedical mod- els' cross-lingual understanding ability. Unlike the entire text, translating entities and relations con- stituting domain knowledge is unique. Since do- main knowledge is considered the most crucial con- tent (Michalopoulos et al., 2020; He et al., 2020), we propose a novel model called KBioXLM to bridge multilingual PLMs like XLM-R (Conneau et al., 2019) into the biomedical domain by lever- aging a knowledge-anchored approach. Concretely, we incorporate three levels of gramularity in knowl- edge alignments: entity, fact, and passage levels, to create a text-aligned biomedical multilingual cor- pus. We first translate the UMLS knowledge base? into Chinese to obtain bilingual aligned entities and relations named UMLStrons- At the entity level, we employ code-switching (Yang et al., 2020) to replace entities in sentences with expressions in another language according to UMLStrane At the fact level, we transform entity pairs with relation- ships into another language using UMLStrans and concatenate them after the original monolingual sentence. At the passage level, we collect paired biomedical articles in English and Chinese from Wikipedia' to form a coarse-grained aligned corpus. An example of the construction process can be seen in Figure 1. Furthermore, we design three train- ing tasks specifically tailored for the knowledge- aligned data: entity masking, relation masking, and passage relation prediction. It is worth not- ing that in order to equip the model with prelimi- nary biomedical comprehension ability, we initially pretrain XLM-R on monolingual medical corpora in both Chinese and English. Then, continuously training on top of the model using these three tasks, our approach has the ability to handle tross-lingual biomedical tasks effectively.
strate that our model consistently outperforms both monolingual and other multilinguul pretrained mod- els cross-lingual zero-shot and few-shot scenar- ios. On average, our model achieves an impressive improvement of approximately 10 points than gen- eral multilingual models. Mcanwhile, our method maintains a comparable monolingual ability by comparing common benchmarks in both Chinese and English biomedical domains.
listed in
To summarize, our contributions can be outlined as follows:
We innovatively propose a knowledge- anchored method for the multi-lingual biomedical scenario: achieving text alignment through knowledge alignment.
We design corresponding tasks for multi- granularity knowledge alignment texts and develop the first multilingual biomedical pre- trained language model to our knowledge.
We translate and proofread four biomedical datasets fill the evaluation gap in cross- lingual settings.
Related Work
2.1 Biomedical Pretrained Language Models
In recent years, the advent of pre-trained language models like BERT (Devlin et al., 2018) has brought about revolution in various downstream NLP tasks, including biomedical research. Researchers such Lee et al. (2020); Huang et al. (2019); Gu et al. (2020) have focused on training domain- specific PLMs using specialized corpora. For example, PubMedBERT (Gu et al., 2020) con- structs a domain-specific vocabulary by leverag- ing articles from PubMed and is trained on this data. BioLLikBERTT (Yasunaga et al., 2022) in- troduces document-link relation prediction a pre-training task, achieving oftte-of-the-art (SOTA) performance on question-answering and document classification tasks. BERT-MK (He et al., 2019) and KeBioLM (Yuan et al., 2021) incorporate UMLS knowledge into the training process. In Chinese, Zhang et al. (2020, 2021b); Cai et al. (2021) have further enhanced BERT with biomedi- cal knowledge augmentation techniques.
To compensate for the lack of cross- cross-lingual eval- uation datasets, we translate and proofread four English biomedical datasets into Chinese, involv- ing three different tasks: named entity recognition (NER), relation extraction (RE), and document clas- sification (DC). The experimental results demon-
While these models have demonstrated impres- sive results in monolingual settings, many of them lack the ability to handle multilingual tasks effec- tively. Thus, there is still a need for models that can tackle the challenges presented by multilingual and cross cross-lingual tasks.
Phttps://ww/ nlm. nih. gov/ressearchuuns Phttps: //wwww wik wakiipddia.oo org/

---

Tokens
tems) is widely acknowledged the largest and most comprehensive knowledge base in the biomedical domain, encompassing a vast collec- tion of biomedical entities and their intricate re- lationships. While UMLS offers broader range of entity types in English and adheres to rigorous classification criteria, it lacks annotated data in Chi- nese. Recognizing that knowledge often possesses distinct descriptions across different languages, we undertake a meticulous manual translation process. A total of 982 relationship types are manually trans- lated, and we leverage both Google Translator' and ChatGPT to convert 880k English entities into their corresponding Chinese counterparts. Man- ual verification is conducted on divergent trans- lation results to ensure accuracy and consistency. This Chinese-translater version of UMLS is called UMLStrans providing seamless cross-lingual ac- cess to biomedical knowledge.
Eetity-level
31.3M
Fact-level
19.6M
Passage-level 14M
Table 1: Tokens of the three-level biomedical multilin- gual corpus by incorporating three granularity knowl- edge alignments.
2.2 Multi-lingual Pretrained Language Models
Multilingual pre-trained models represent multiple languages in a shared semantic vector space and enable effective cross-lingual processing. Notable examples include mBERT (Devlin et al.. 2018), which utilizes Wikipedia data and employs Multi- lingual Masked Language Modeling (MMLM) dur- ing training. XLM (Lample and Conneau, 2019) focuses on learning cross-lingual understanding ca- pability from parallel corpora. ALM (Yang et al., 2020) adopts a code-switching approach for sen- tences in different languages instead of simple con- catenation. XLM-R (Conneau et al., 2019), based on RoBERTa (Liu et al., 2019), significantly ex- pands the training data and covers one hundred lan- guages. To further enhance the cross-lingual trans- ferability of pre-trained models, InfoXLMM (Chi et al., 2021) introduces a new pretraining task based on contrastive learning. However, as now, there is no specialized multilingual model specifically tailored for the biomedical domain.
Wikipedia. Wikipedia contains vast knowledge across various disciplines and is available in mul- tiple languages. The website offers complete data downloads providing detailed information about each page, category membership links, and interlan- guage links. Initially, we collect relevant items in medicine, pharmaceuticals, and biology by follow- ing the category membership links. We carefully filter out irrelevant Chinese articles to focus on the desired content by using a downstream biomedi- cal NER model trained on CM&EE- V2 (Zan et al., 2021) dataset. The dataset includes nine major cat- egories of medical entities, including 504 common pediatric diseases, 7085 body parts, 12907 clini- cal manifestations, and 4354 medical procedures.
3 Method
Then, using the interlanguage links, we associate the textual contents of the corresponding Chinese and English pages, generating aligned biomedical bilingual text.
This section presents the proposed knowledge- anchored multi-lingual biomedical PLM called KBIOXLM. Considering the scarcity of biomed- ical parallel corpora, we utilize language-agnosti knowledge to facilitate text alignment at three lev- els of granularity: entity, fact, and passage. Subse- quently, we train KBioXLM by incorporating these knowledge-ane anowledgeannchored tion of the multilingual XLM-R model. Figure 2 provides an overview of the training details.
3.1.2 Entity-level Knowledge
Entities play crucial role in understanding textual information and instrumental in semantic un- derstanding and relation prediction tasks. Drawing inspiration from ALM (Yang et al., 2020), we adopt entity-level code-switching and devise the entity masking pretraining objective. The process con- structing entity-level pseudo-bilingual corpora is illustrated in orange in Figure 1. We extract enti- ties from UMLSSrans that appear in monolingual
anchored aligned data the founda-
3.1 Three Granularities of Knowledge
3.1.1 Knowledge Base
We obtain entity and fact-level knowledge from UMLS and retrieve aligned biomedical articles from Wikipedia.
httpps://taasslate. gow..e...olll.ooooooo
hhttps://openai http:///pnaii..mmmmooo///acaaaap hhttps://dumps. www.wediiaiiai .rg/
UMLS. UMLS (Unified Medical Language Sys-

---

Entity knowledge aligned
[CLS] Laudanum contains approximately 10%
equivalent 1% [MASK] [MASK]
Fact knowledge aligned data:
[CLS] Laudanum... 10% opium poppy |SEP]
TE [MASK]MASKKJMAASK] S
Passage knowledge aligned data:
[CLS] Laudanum contains approximately SEP
English segment
Chinese segment
input
KBioXLM
output
Entity knowledge target output:
[CLS] Laudanum contains approximately 10% D 527E equivalent to 1%
Fact knowledge target output:
[CLS] Laudanum contains upproximately 10% opium poppy ISEP
x B
Passage knowledge target output:
[CL.S] Laudanum contains approximately ...SPP
positive
English segment
Chinese segment
Figure 2: Overview of KBioXLM's training details. Masked entity prediction, masked relation token prediction, and contextual relation prediction tasks are shown in this figure. The color orange represents entity-level pretraining task, green represents fact-level, and blue represents passage-level. Given that the two passages are aligned, our model predicts "positive" relationship between them.
sentences and their counterparts in the other lan- guage. To ensure balance, we randomly substitute 10 biomedical entities with their respective coun- terparts in each sample, keeping an equal number of replaced entities in both languages.
constructing bilingual corpora at the fact level in- volves the following steps:
Retrieve potential relationships between paired entities from UMLS in monolin- gual corpus.
We design specific pretraining tasks to facili- the exchange of entity-level information be- tween Chinese and English. Given sentence 22, ,.,,,,,,,))))))))). containing Chinese and En- glish entities, we randomly mask 15% of the tokens representing entities in the sentence. The objective of KBioXLM's task is to reconstruct these masked entities. The loss function is defined as follows:
Organize these facts in another language and concatenate them with the original monolin- gual sentence.
X (x1,
An example depicted in green color in Figure 1. Given the input text "Laudanum contains approxi- mately 10% opium poppy ...", we extract the fact "(opium poppy, associated with, morphine)" and its corresponding Chinese translation "( * #t, #8 p7 The final text would be "Lau- danum contains approximately 10% opium poppy [SEP] 03 * relationship "
L Σ
log X(e,
(1)
Here, 6 represents the masked Chinese or English entity.
We will mask the
The fact-level task to reconstruct the masked relationships. The loss function for relation mask- ing is defined as follows:
3.1.3 Fact-level Knowledge
Fact refers to relationship between a subject, pred- icate, and object in the knowledge base. Our as- sumption is that if both entities mentioned in the fact are present together in text, then the text should contain this fact. We employ fact matching to create bilingual corpora and develop a pretrain- ing task called relation masking. The process of
L, - - log P(f(X),
(2)
where fa represents the masked relationship, which can be either a Chinese or an English representa- tion.

---

3.1.4 Passage-level Knowledge
capability across various downstream tasks. To tai- lor XLM-R to the biomedical domain, we conduct additional pretraining using a substantial amount of biomedical monolingual data from CNKI (2.15 billion tokens) and PubMed (2.92 billion tokens). The pre-training strategy includes whole word masking (Cui et al., 2021) and biomedical entity masking. We match the biomedical Chinese enti- ties and English entities contained in UMISStrans with the monolingual corpora in both Chinese and English for the second pretraining task. Clearly, we have already incorporated entity-level knowledge at this pretraining stage to enhance performance. For specific details regarding the pretraining pro- cess, please refer to Section A. For convenience, we refer to this model as XLMMRR+
Some biomedical NLP tasks are performed the document level, we broaden the scope of cross- lingual knowledge encompass the passage level. This expansion is illustrated in blue in Figure 1. Specifically, we employ paired biomedical English and Chinese Wikipedia articles to create an aligned corpus at the passage level. This corpus serves as the foundation for designing pretraining task fo- cused on predicting passage relationships. Inspired by Yasunaga et al. (2022), the strategies employed construct the passage-level corpus are as follows:
Randomly selecting one Chinese segment and one English segment, we label them as "pos- itive" if they belong to paired articles and as "random" otherwise.
We pair consecutive segments in the same lan- guage to create contextualized data pairs and label them as "context".
Biomedical Dataset Construction
Google Translator
Input: After taking SlMMetoisgaadded/e. she developed <l <l=oycknedas//>> and period #2uunepprnteeeescce
Ultimately, we gather collection of 30k segment pairs, with approximately equal quantities for each of the three types of segment pairs. The pretrain- ing task employed to incorporate bilingual passage knowledge into the model is passage relationship prediction. The loss function for this task is as follows:
Prompt:
Let's think stop by stop. will provide you with a marked English taxt and its translated derect and correct missing marls snum- strum- and Chinese You postive esample:
Input:
EN: Sovere <1> bleomysin </1> +Q lung condckt +/0+ reversal with high dose con coricozzrrids..
1 </1> paptt: moce B8MT
Output:
- 18 Wi <D> -its Hi (Sinput SOutput trandartor).
Ip 1-=-opp log X(c)ipparr).
Steasw return the result
Output:
Acconding to your requent have obtaiedd the follooiing conection realte
where C E {positive, random, conlezt), Xpar is the hidden state with global contextual information. The tokens present in the three-level biomedi- cal multilingual corpus are documented in Table 1. KBioXLM is trained using equal proportion of monolingual data and the previously construeted three-level bilingual corpora ensure the model's proficiency in monolingual understanding. The overall pretraining loss function for KBioXLM is defined as follows:
Human Revilion
Figure 3: This picture illustrates the sentence, "After taking Metoclopramide, she developed dyskinesia and period unresponsiveness." which is initially marked for translation and subsequently revised through col- laborative effort involving ChatGPT and manual editing.
Due to the lack of biomedical cross- mooss-lingual un- derstanding benchmarks, we translate several well- known biomedical datasets from English into Chi- nese by combining translation tools, ChatGPT, and human intervention. As shown in Table 2, these datasets BC5CDR (Li et al., 2016) and ADE (Gurulingappa al., 2012) for NER, GAD for RE, and HoC (Hanahan and Weinberg. 2000) for DC. ADE and HoC belong to the BLURB bench- mark². Please refer to Section A.2 for details about
L Le Lip
By integrating these three multi-task learning objec- tives, KBioXLM exhibits improved cross-lingual understanding capability.
3.2 Backbone Multilingual Model
Our flexible approach can be applied to any multi- lingual pre-trained language model. In this study, we adopt XLM-R as our foundational framework, leveraging its strong cross-lingual understanding
https::/kkns anki.net/knss " "nttps://huggingface datadatsss//////e ppbbmed "https://microoofft witpoffttttttt.. thub io/BLURB/

---

Dataset
Task
Train
Dev
Test
Language
Biomedical
BCSCDR
NER
500
500
500
eHealth
CN
ADE
NER
3845
427
SMedBERT
CN
GADÉ
RE
4261
535
534
BioBERT
HoCt
DC
1295
186
371
PubMedBERT
EN
BioLinkBERT
EN
mBERT
Multi
Table 2: Statistics of four biomedical cross-lingual datasets. 1.Task types; 2. Number of samples in train/dew/test dataset. 3. The dataset with special mark belongs to BLURB benchmark.
InfoXLM
Multi
XLM-R
XLM-R-BT
Multi
XLM-R-tthree KL.
Multi
KBIXLLM
Multi
these four tasks.
Table 3: Characteristics our baselines. indicates
that the model has this feature while "X" means the op- posite. "CN" means Chinese, "EN" represents English and "Multi" represents Multilingual.
The process, as depicted in Figure can be sum- marized as follows: we conduct simple transla- tion using Google Translator for the document clas- sification dataset. To preserve the alignment of En- glish entities and their relationships in the NER and RE datasets after translation, we modify them by re- placing them with markers "<num>>anttt-/nuns based on the NER golden labels in the original sen- tences. Here, "num" indicates the entity's 's order in the original sentence. During post post-procsssing, we match the translated entities and their relation- ships back to their original counterparts using the numerical information in the markers. Despite the proficiency of Google Translator, it has some limitations, such as missing entity words, incom- plete translation of English text, and semantic gaps. To address these concerns, we design a prompt to leverage ChatGPT in identifying inconsistencies in meaning or incomplete translation between the original sentences and their translations. Subse- quently, professional annotators manually proof- read the sentences with identified defects while sample of error-free sentences randomly checked. This rigorous process guarantees accurate and con- sistent translation results, ensuring proper align- ment of entities and relationships. Ultimately, we obtain high-quality Chinese biomedical datascts with accurately aligned entities and relationships through meticulous data processing.
5.1.2 Finetuning
For monolingual tasks and cross-lingual under- standing downstream tasks listed in Table 2, the backbone of Named Entity Recognition is the en- coder part of the language model plus conditional random fields (Lafferty et al., 2001). The simplest sequence classification is employed for relation ex- traction and document classification tasks. Fl is used as the evaluation indicator for these tasks.
5.2 Baselines
To compare the performance of our model in mono- lingual comprehension tasks, select SOTA mod- els in English and Chinese biomedical domains. Similarly, assess our model's cross-lingual com- prehension ability, we conduct comparative experi- ments with models that possess strong cross-lingual understanding capability in general domains, as there is currently a lack of multilingual PLMs specifically tailored for the biomedical domain. Monolingual Biomedical PLMs. For En- glish PLMs, select BioBERT (Lee et al., 2020). PubMedBERT (Gu al., 2020), and Bi- oLinkBERT (Yasunaga al., 2022) for compari- son, while for Chinese, we choose eHealth (Wang et al., 2021) and SMedBERT (Zhang et al., 2021b). Multilingual PLMs. XLM-R baseline model and other SOTA multilingual PLMs, including mBERT (Devlin et al., 2018), InfoXLM (Chi et al.,
5 Experiments
5.1
Dataset and Settings
5.1.1 Pretraining
2021) are used our baselines. We also compare the results of two large language models (LLMs) that currently perform well in generation tasks on these four tasks, namely ChatGPT and ChatGLM- 6B 10
The pretraining process of KBioXLM employs learning rate of 5e-5, batch size of 128, and total of 50,000 training steps. 5,000 warm-up steps are applied the beginning of the training. The model is pretrained on NVIDIA RTX A5000 GPUs for 14 hours.
"httpss//chattglnncn/bbl

---

LLMs
Multilingual PLMs InfoXLM
Multilingual Biomedical PLMs
Datasets
ChatGPT
ChatGLM
mBERT
XLM-R
XIMM-++T
XIMM--three KI. KBioXIM
ADE
64.50
27.30
59.43
64.23
57.62
64.78
65.47
70.88
EN-1o-CN BCSCDR
63,00
40.70
54.59
66.39
57.43
63.83
70.64
73.02
GAD
48.30
48.30
59.52
67.03
68.79
70.65
75.29
78.91
HoC
35.10
28.00
14.29
44.83
37.58
60.45
68.09
78.83
AVG
52.73
36.08
46.96
60.62
55.36
64.93
69.87
75.41
ADE
71.79
42,90
71.31
79.52
77.38
77.92
78.07
85.61
CN-to-EN
BC5CDR
54.80
38.80
64.12
75.36
72.40
76.29
78.92
84.52
GAD
51.20
52.20
64.16
69.69
69.96
74.74
76.36
81.16
HoC
41.30
31.70
37.08
48.34
37.67
57.65
61.09
73.99
41,40
59.17
71.65
73.61
Table 4: Cross lingual zero shot results. "EN-to-CN" and "CN-to-EN" indicate training in English and testing on Chinese datasets, and vice versa. AVG represents the average Fl score across four cross-lingual tasks.
10-shot
100-shot
ADE
BC5CDR
GAD
HoC
AVG
ADE
BC5CDR
GAD
HoC
AVG
cHealth
64.57
67.10
67.57
62.08
65.33
78.39
79.09
72.83
74.66
76.24
CN Bio PLMS
SMedBERT
54.61
61.11
67.61
33.17
54.13
75.18
75.25
69.36
66.24
71.51
mBERT
$7.88
55.15
66.44
44.61
56.02
69.88
74.11
73.65
67.08
71.18
Multi PLMs
InfoXLM
66.92
62.47
69.48
57.47
64.09
76.35
75.62
76.19
67.70
73.97
XLM--
61,02
56.66
73.85
43.74
58.82
73.06
72.93
76.18
64.12
71.57
ChatGPT
66.20
60.40
49.40
50.20
56.55
LLMs
ChatGLM
26.20
27.10
52.60
23.50
32.35
XL--RRthreee KI.
71.50
71.01
76.36
73.45
73.08
76.
77.19
77.86
76.89
77.01
Multi PLMS
KBioXLM
75.49
74.98
80.76
79.41
77.66
79.45
80.63
81.98
83,20
81.32
Table 5: Cross lingual CN-to-CN few shot results. "Bio" represents Biomedical and "Multi" represents Multilingual. Due to the limited number of input tokens, we only conduct 10-shot experiments for LLMs.
Multilingual Biomedical PLMs. To our knowl- edge, there currently multilingual pretraining model in biomedical. Therefore, build two base- line models on our own. Considering the effective- ness of nack-translation (BT) as a data augmenta- tion strategy in general multilingual pretraining, we train baseline XLM-R+BTT using back-translated biomedical data. Additionally, in order to assess the impact of additional pretraining in KBioXL.M, we directly incorporate the three levels of knowl- edge mentioned earlier into the XLM-R architec- ture, forming thre KL. Please refer to Table for basic information about our baselines.
and few-shot. As shown in 4 and 5, KBioXLM achieves SOTA performance in both cases. Our model has a strong cross-lingual understanding ability in the zero-shot scenario under both "EN-to- CN" and "CN-to-EN" settings. worth noting that the performance of LLMs language under- standing tasks is not ideal. Moreover, compared to ChatGPT, the generation results of ChatGLM are unstable when the input sequence is long. Similarly, general-domain multilingual PLMs also exhibit performance difference of over 10 points compared to our model. The poor performance of LLMs and multilingual PLMs underscores the importance of domain adaptation. XLM-R+three KL is pretrained with just 65M tokens, and it already outperforms XLM-R by 14 and 9 points under these two set- tings. And compared to XLM-R+BT, there is also an improvement of 5 and 2 points, highlighting the importance of knowledge alignment. Compared to KBioXLM, XLM-R+three KL performs 5 or more points lower. This indicates that excluding the pre- training step significantly affects the performance of biomedical cross lingual understanding tasks,
5.3 Main Results
This section explores our model's cross-lingual and monolingual understanding ability. Please refer to Appendix A.3 for the model's monolingual under- standing performance on more tasks.
5.3.1 Cross-lingual Understanding Ability
We test our model's cross-lingual understanding ability on four tasks in two scenarios: zero-shot

---

Dataset
ADE
BCSCDR
GAD
HoC
AVG
Pubmedhert
90.22
89.37
80.50
83.97
86.02
EN Bio PLMS
BioBERT
89.85
87.14
83,95
82.44
85.85
BioLinkBERT
90.12
89.76
85,90
84.53
87.58
mBERT
89.61
84,44
79.67
80.04
83.44
EN-to-EN
Multi PLMs
InfoXLM
89.91
86.50
77.33
78.70
83.11
XLM-R
90.07
85.92
80.45
80.12
84,14
XLM- KL
89.83
86.50
82.35
82.08
85.19
Multi Bio PLMs
BBioLMM
90.56
88.87
83.04
83.66
86.53
elfealth
85.20
72.22
77.53
78.67
78 .1
CN Bio PLMMs
SMedBERT
84.30
41.30
74.34
79.27
69.80
mBERT
83.27
67.83
75.69
76.35
75.79
CN-to-CN Multi PLMs
InfoXLM
83.48
72.00
71.71
78.53
76.43
XLM-R
82.67
70.57
78.13
78.52
77.47
XL. R+three KI.
83.26
78.09
79.61
80.73
80.42
Multil Bio PLMs
BiiooLM
83.02
78.94
82.76
83.47
82.05
Table The performance of M and our baselines on English and Chinese monolingual comprehension
tasks.
Datasets
ADE
BCSCDR
GAD
HoC
improvement of up to points. BioLinkBERT per- forms slightly better than ours on English compre- hension tasks because incorporates more knowl- edge from Wikipedia. KBioXLM's focus, however, lies in cross-lingual scenarios, and we only utilize small amount of aligned Wikipedia articles.
KBioXLM
70.88
73.02
78.91
78.83
Pas
69.99
71.55
78.08
77.28
w/o Pas+Fact
70.20
69.78
76.87
76.92
w/o Pas+Fact+Ent
66.61
67.03
75.93
76.74
Table 7: KBioXLM's cross-lingus understanding abla- tion experiments in the zero-shot scenario.
5,4 Ablation Study
This section verifies the effectiveness of different parts the used datasets. Table presents the re- sults of ablation experiments in zero-shot scenarios. Removing the bilingual aligned data at the passage level results in 1 1-point decrease in model perfor- mance across all four tasks. Further removing the fact-level data leads to a continued decline. When all granularity bilingual knowledge data is removed, our model's performance drops by approximately points. These experiments demonstrate the effec- tiveness of constructing aligned corpora with three different granularities of knowledge. Due to the utilization of entity knowledge in the underlying XLM-R L.MM-Rpretraining. it is difficult to accurately as- sess the performance when none of the three types of knowledge used.
highlighting the importance of initially pretraining XLM-R to enhance its biomedical understanding capability.
In the "EN-to-CN" few-shot scenario, we test models' cross-lingual understanding ability under two settings: 10 training samples and 100 training samples. also can be observed that XtM-R+ KL and KBioXLM perform the best among these four types of PLMs. Multilingual PLMs and Chi- nese biomedical PLMs have similar performance. However, compared to our method, there is dif- ference of over 10 points in the 10-shot scenario and over 5 points in the 100-shot scenario. This indicates the importance of both domain-specific knowledge and multilingual capability, which our model satisfies.
Conclusion
5.3.2 Monolingual Understanding Ability
Although the focus of our model is on cross-lingual scenarios, also test its monolingual comprehen- sion ability these four datasets. Table shows the specific experimental results. It can be seen that KBioXLM can defeat most other PLMs in these tasks, especially in the "CN-to-CN" scenario. Compared to XLM-R, KBioXLM has an average
This paper proposes KBioXLM, model that transforms the general multi-lingual PLM into the biomedical domain using knowledge-anchored approach. We first obtain biomedical multilingual corpora by incorporating three levels of knowl- edge alignment (entity, fact, and passage) into the monolingual corpus. Then we design three train-

---

ing tasks, namely entity masking, relation mask- ing, and passage relation prediction, to enhance the model's s cross-lingual ability. KBioXLM achieves SOTA performance in cross-lingual zero-shot and few-shot scenarios. In the future, we will explore biomedical PLMs in more languages and also ven- ture into multilingual PLMs for other domains.
the Association for Computational Linguistics: Hu- man Language Technologies, pages 3576-3580 On- line. Association for Computational Linguistics.
Fenia Christopoulou, Makoto Miwa, and Sophia Ana- niadou. 2019. Connecting the dots: Document- neural relation extraction with edge-oriented graphs. arXiv preprint 1eXiv:/99.00020
Nigel Collier and Jin-Dong Kim. 2004. Introduction to the bio-entity recognition task at julpba. In Proceed- ings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Appli- cations (NLPBA/BiONLP), pages 73 78.78.
Limitations
Due to the lack of proficient personnel in other less widely spoken languages, our experiments were limited to Chinese and English only. However, Our method can be applied to various other languages, which is highly significant for addressing cross- lingual understanding tasks in less-resourced lan- guages. Due to device and time limitations, we did not explore our method on models with larger pa- rameter sizes or investigate cross-lingual learning performance generative models. These aspects are worth exploring in future research endeavors.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott. Luke Zettle- moyer, and Veselin Stoyanov. 2019, Unsupervised cross-lingua representation learning at scale. arXiv preprint
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert. IEEE/ACM Transac- rions on Audio, Speech, and Language Processing, 29:3504-3514.
Acknowledgements
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint
We thank all reviewers for their valuable comments. This work was supported by the Young Scientists Fund of the National Natural Science Foundation of China (No. 62106165).
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Nebi disease corpus: a resource for dis- ease name recognition and concept normalization. Journal of biomedical informatics, 47:1-10.
References
Ålex Bravo, Janet Piñero, Núria Queralt-Rosinachh Michael Rautschka, and Laura I Furlong. 2015. Ex- traction of relations between genes and diseases from text and large-scale data analysis: implications for translational research. BMC bioinformatics. 16(1):1- 17.
Jingcheng Du, Qingyu Chen, Yifan Peng, Yang Xiang, Cui Tao, and Zhiyong Lu. 2019. Ml-net: multi-label classification of biomedical texts with deep neural networks. Journal of the American Medical Infor- matics Association, 26(11):1279-1285
Markus Eberts and Adrian Ulges. 2019. Span-based joint entity and relation extraction with transformer pre- pre-training. arXiv preprint (tt.:ii:99.00775000
Razvan Bunescu, Ruifang Ge, Rohit J Kate, Edward M Marcotte, Raymond J Mooney, Arun K Ramani, and Yuk Wah Wong. 2005. Comparative experiments on learning information extractors for proteins and their interactions. Artificial intelligence in medicine, 33(2):139-155.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing.
Zerui Cai, Taolin Zhang, Chengyu Wang, and Xiaofeng He. 2021. Embert: A pre-trained language model for chinese medical text mining. In Web and Big Data: 5th International Joint Conference, APWeb- WAIM 2021, Guangzhou, China, August 23-25, 2021, Proceedings, Part 5, pages 242-257. Springer.
T. Guan, H. Zan, X. Zhou, H. Xu, and K Zhang 2020. CMelE: Construction and Evaluation of Chi- nese Medical Information Extraction Dataset. Nat- ural Language Processing and Chinese Computing, 9th CCF International Conference, NLPCC 2020. Zhengzhou, China, October 14-18, 2020, Proceed- ings, Part I.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal. Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-tan -teeorrtit framework for lingull language model pre-training. Proceedings of the 2021 Conference of the North American Chapter of
Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development of a benchmark

---

corpus to support the automatic extraction of drug- related adverse effects from medical case reports. Journal of biomedical informatics, 45(5):885-892.
Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022. Biored: a rich biomedical relation extraction dataset. Briefings in Bioinformatics, 23
Douglas Hanahan and Robert A Weinberg. 2000. The hallmarks cancer. cell, 100(1):57-70.
George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen H Chen, and Alexander Wong, 2020. Umls- bert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical lan- guage system metathesaurus. In North American Chapter of the Association for Computational Lin- guistics.
Bin He, Di Zhou, Jinghui Xiao, Qun Liu, Nicholas Jing Yuan, Tong Xu, al. 2019. Integrating graph contex- tualized knowledge into pre-trained language models. arXiv preprint arXiv: 1912.00177.
Yun He, Ziwei Zhu, Yin Zhang, Qin Chen, and James Caverlee. 2020. Infusing disease knowledge into bert for health question answering, medical infer- ence and disease name recognition. arXiv preprint arXiv:2010.03746.
Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Anamiadou. 2019. Inter-sentence relation extraction with document-leve. graph convolutional neural network. arXiv preprint arXiv: 096.04684.
María Herrero-Zazo, Isabel Segura-Bedmat Bedmar, Paloma Martínez, and Thierry Declerck. 2013. The ddi corpus: An annotated corpus with pharmacological substances and drug- drug interactions. Journal of biomedical informatics, 46(5):914-920.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709
Quan Wang. Songtai Dai, Benfeng Xu, Yajuan Lyu, Yong Zhu, Hua Wu, and Haifeng Wang. 2021. Building chinese biomedical language models via multi-level text discrimination. arXiv preprint arXiv:21
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint 13X4::9904,03442.
Martin Krallinger, Obdulia Rabal, Saber A Akhondi, Martın Pérez Pérez, Jesús Santamaría, Gael Pérez Ro- driguez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio López, Umesh Nandal, et al. 2017. inter-
Jian Yang. Shuming Ma. Dongdong Zhang, Shuangzhi Wu, Zhoujun Li, and Ming Zhou. 2020. Alternating language modeling for cross- cross-inguall pre-training, In Proceedings the AAAI Conference Artificial Intelligence, volume 34, pages 9386-9393.
Overview of the biocreative vi chemical- action track. Proceedings of the sixth BioCreative challenge evaluation workshop, volume 1, pages 141- 146.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang, 2022. Linkbert: Pretraining language models with document links. arXiv preprint arXiv:2203. 15827.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data.
Zheng Yuan, Yijia Liu, Chuanqi Tan, Songfang Huang, and Fei Huang. 2021. Improving biomedical pre- trained language models with knowledge. arXiv preprint 104.44.00..
Guillaume Lample and Alexis Conncau. 2019. Cross- lingual language model pretraining. arXiv preprint 1r1.v91901.07291..
H. Zan, W. Li, K. Zhang, Y. Ye, and Z. Sui. 2021. Build- ing a Pediatric Medical Corpux: Word Segmentation and Named Entity Annotation. Chinese Lexical Se- manties.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghycon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234 1240.
Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, al. 2021a. Cblue: A chi- nese biomedical language understanding evaluation benchmark. arXiv preprint arXiv:2106.08083
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci- aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v edr task corpus: a resource for chemical disease relation extraction. Database, 2016.
Ningyu Zhang. Qianghuai Jia, Kangping Yin, Liang Dong, Feng Gao, and Nengwei Hua. 2020. Conceptu- alized representation learning for chinese biomedical text mining, arXiv preprint arXiv:2008. 10813.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint urXiv:19077 11692.
Taolin Zhang, Zerui Cai, Chengyu Wang. Minghui Qiu. Bite Yang, and Xiaofeng He. 202 Smedhert: A knowledge-enhanced pre-trained language model with structured semanties for medical text mining. arXiv preprint arXiv:2108.0880

---

Hui Zong, Jinxuan Yang. Zeyu Zhang, Zuofeng Li, and Xiaoyan Zhang. 2021. Semantic categorization of chinese eligibility criteria in clinical trials using ma- chine learning methods. BMC Medical Informatics Decis. Mak., 21(1):
Dataset
Task
CMeÉE (Zan et 2021)
NBR
15009
5000
5000
CMelE/Gunn
4482
Disgsosis Nonuulicatio
2000
10192
CUPsSIS
Sentence Simiantl
16000
4000
100000
CHIPCCTC (Zong et al, 2021)
Sentence Chreafizztion
22062
7682
10000
KUAKE-QIC
Inierr Classification
6931
1955
1994
KUAKE-QIR
Quary
24174
2913
5465
A Appendix
KUAKE-QOR
Quary-Quae Relevanes
Table 8: Statistics of Chinese biomedical datasets.
A.1 XLM-R Pretraining Settings
1.Task types; 2. Number samples train/dev/test dataset.
Building upon XLM-R, we train XLM-R using medical data from CNKI of 2.15B tokens and data from PubMed of 2.92B tokens. During the training process, we initialize the model parameters with XLM-R. It is important to note that in order to speed up the training process, we first calculate the distribution of tokens from enki and pubmed in the XL.M- vocabulary. Then, we utilize one- hot matrix to reduce the original MLM head of XLM-R from 250002 X 768 to 37030 x 768, and use it as the new MLM head. The pre-training strategy includes whole word masking (Cui et al., 2021) and biomedical entity masking. We match the biomedical Chinese entities and English enti- ties contained in UMLStrans with the monolingual corpora in both Chinese and English for the second pretraining task. The proportion of masked tokens in the sentence is the same as XLM-R, and both strategies masked tokens at 1:1 ratio. We limit the masked biomedical entities to a maximum length of 3 to accelerate the model's learning process. The peak learning rate for this training process is set to le-4, with batch size of 1280 and total of 150,000 training steps. In the first 10,000 steps, the learning rate linearly increases. The model was pretrained on NVIDIA RTX A5000 GPUs for two weeks.
Dutuset
Task
Train
Dev
Test
BCS-cheem (Li 2016)
4560
45%1
4797
BCS-discase! et 2016)
4581
NCBI-discsee' (Dojan al. 2014) 2014)
5424
923
940
BC3GM (Du et al., 2019)
12500
2500
5000
JNLPBA' (Collier and Kim, 2004)
NER
16807
1739
3856
BCSCDR (Li et al,, 2016)
500
500
500
(Gurulingappu et 2012)
CHR (Sahu et al., 2019)
7248
1182
3614
BioRED (L.nn et al., 2022)
400
100
100
ChemProf (Krallinger DDI' (Herrero-Zaze et al., 2013)
2017)
18035
11268
25296
2496
5716
GAD' (Bravo ot al., 2015)
RB
4261
535
534
ATMed (Bunescu et al., 2005)
525
583
HoC! (Hanahan and Weinberg, 2000)
DCC
1295
186
371
Table 9: Statistics of English biomedical datasets. 1.Task types; 2. Number of samples in train/dev/test dataset; 3. The dataset with special mark belongs to BLURB benchmark.
Dataset
ehealth
SMedBERT
XLM-R
KBioXLM
CMcEE-V2
59.56
59.86
57.94
59.48
CMelE-V2
49.74
48.72
50.32
50.22
CHIP-CDN
59.32
55.46
51.04
$7.89
CHIP-STS
85.48
84,98
81.71
83.20
CHIP-CTC
63.15
68.05
58.39
66.30
KUAKE-QIC
85,66
85.71
83.10
82.85
KUAKE-QTR
64.56
61.59
58.23
59.49
KUAKE-QQR
85.65
82,83
80,52
81.89
AVG
69.14
68.40
65.16
67.67
Table 10: Monolingual Chinese results.
A.2 Four Downstream Tasks
BC5CDR, BC5CDR comprises a collection of 1500 PubMed abstracts and has been prepro- cessed by Christopoulou et al. (2019). The objec- tive of the model is to identify two distinct entity types in the text: chemical and discase.
markers, namely " "@GENES" and "@ DISEASES"" respectively.
HoC. The Hallmarks of Cancer (HoC) corpus com- prises PubMed abstracts with binary labels indi- cating specific cancer hallmarks. It contains 37 detailed hallmarks grouped into ten top-level cat- egories. Models are required to predict these 10 top-level categories.
ADE. ADE is another NER dataset sourced from PubMed documents, primarily focused on identify- ing drugs and adverse effects entities. We leverage the dataset provided in SpERT (Eberts and Ulges, 2019).
Please refer Table 2 for quantity statistics.
GAD. GAD serves the purpose of detecting the association between gene entities and disease en- tities in a given sentence, The gene and disease entities within the sentences denoted by special
A.3 Monolingual Biomedical Tasks
The Chinese Biomedical Language Understanding Evaluation (CBLUE) (Zhang et al., 2021a) bench- 12 12 in the Chinese medical domain includes
ncbi. nih.
Pht / thub. comp///lLFEeeencmarr//B...

---

Dataset
PubMedBEKT
BioLinkBERT
XLM-R
KBRXLLM
BCS-chem
93.08
92.97
$8.74
91.84
Nchi-diseuse
86.50
86.69
$6.85
86.67
BC2GM
83,74
84.33
81.91
83,05
JNLPRA
78
79.42
BCSCDR
89.37
89.76
85.92
8X.86
CHR
91.61
91.16
91.08
91.82
BioRED
91.14
88.
ChemProt
76.76
68.49
76.68
DDI
81.02
79.57
74,30
79. 43
AlMed
88.41
85.96
70.37
84.26
HoC
83.97
84.53
80.12
83.66
AVG
85.85
85.97
81.46
85.27
Table 11: Monolingual English results.
tasks such NER, RE, sentence classification,, and more. Similarly, the English medical domain also encompasses these tasks. The quantity statistics for the Chinese and English datascts are shown in Table 8 and Table 9, respectively.
Here, we compare KBioXLM with two SOTA Chinese biomedical models, eHealth and SMed- BERT CBLUE benchmark and the English SOTA models, PubMedBERT and BioLinkBERT on the corresponding English biomedical tasks.
Table 10 and Table represent the results of evaluating the model's monolingual comprehen- sion ability on Chinese and English biomedical benchmarks, respectively. Our model KBioXLM shows significant improvements of two points and four points compared to XLM-R on average, re- spectively. This indicates that compared to gen- eral multilingual PLMs. model has stronger biomedical comprehension capability. However, as our model primarily focuses on addressing cross- lingual understanding tasks, falls slightly behind the current SOTA monolingual biomedical models.


Reviews:
[{"review_id": "MMtzUXM4bZ", "reviewer": "Reviewer_VPvR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to you for your insightful and comprehensive feedback on our manuscript. your thoughtful comments and suggestions have significantly contributed to the enhancement of our work. We are truly appreciative of your time and effort in reviewing our paper.\n \n1.\"Limited evaluation scope by only testing on Chinese, additional languages should be considered for a more comprehensive evaluation.\"\n\nThank you for pointing out this issue. \nOur base model XML-R is multilingual, giving our model the potential to support multiple languages. Additionally, the approach we've employed for incorporating knowledge also caters to multiple languages. However, as stated in the \"Limitations\" section, due to the lack of expertise in other languages, we have only conducted experiments in Chinese and English. We will acknowledge this limitation using a footnote in the introduction. In the future, we plan to explore the cross-lingual capabilities of testing our model in other language datasets that align with English dataset.\n \n2.\"Investigating the performance of the model when there are fewer tokens at each granularity level could provide additional insights.\"\n\nI appreciate you bringing this matter to my attention.\nWe attempt to reduce the number of tokens at each granularity to 1/5 of the original number of tokens, and once again verify the cross language understanding performance of the model. The results are presented in the table \"less tokens\" below. When we reduce the number of training tokens to the original 20%, the performance of the model significantly decreases, but it still outperforms the baseline model. From the experiment, it is not difficult to see that the knowledge granularity aligned corpus and its training method proposed in this article are effective, and the training token is positively correlated with the performance of the model.\n\n|       |  BC5CDR   | GAD  |\n| ---- |  ----  | ----  |\n| KBioXLM | 73.02 | 78.91 |\n| w/o Pas+Fact+Ent  | 67.03 | 75.93 |\n| Loss tokens | 70.22 | 76.52 |\n\n3.\"Ablation tests considering situations with only one or two granularity levels (entity, fact, or passage) would further validate the proposed approach. (An extension of Table 7)\"\n\nYour observation of this matter is duly noted and appreciated.\nWe validated the effectiveness of each granularity separately, and the results are presented in the table below. It can be seen that compared to the baseline model, adding any granularity of knowledge alignment corpus can help improve the cross language understanding performance of the model.\n\n|       |  BC5CDR   | GAD  |\n| ---- |  ----  | ----  |\n| KBioXLM | 73.02 | 78.91 |\n| w/o Pas+Fact+Ent  | 67.03 | 75.93 |\n| w only Ent | 69.78 | 76.87 |\n| w only Fact | 70.17 | 77.68 |\n| w only Pas | 69.73 | 77.19 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well-motivated paper shedding light on the biomedical applications for minor languages.\n- Demonstrates impressive performance improvement (up to 10+ points) in cross-lingual zero-shot and few-shot scenarios.", "reasons_to_reject": "- Limited evaluation scope by only testing on Chinese, additional languages should be considered for a more comprehensive evaluation.\n- Investigating the performance of the model when there are fewer tokens at each granularity level could provide additional insights.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces KBioXLM, a knowledge-anchored approach to transform the multilingual pretrained model XLM-R for handling biomedical domain tasks across languages. The main contributions are:\n\n- Addressing the scarcity of non-English domain corpora and the absence of parallel data in the biomedical domain.\n- Validating the model through translated English benchmarks into Chinese", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Ablation tests considering situations with only one or two granularity levels (entity, fact, or passage) would further validate the proposed approach. (An extension of Table 7)"}, "extra_text": {}}, {"review_id": "GGwuZ4hwSd", "reviewer": "Reviewer_yywL", "summary": "", "questions": "See weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt appreciation to you for your meticulous evaluation of our manuscript. Your constructive criticisms and valuable insights have immensely aided in refining the quality of our research. We are thankful for your dedication to providing thorough feedback.\n \n1.\"As authors use both English and Chinese Wikipedia to build the knowledge-enhanced dataset, it is natural to be curious about how a multilingual pertaining language model directly trained by this bilingual corpus  will perform in benchmarks mentioned in the article.\"  \n\nI'm grateful for you highlighting this concern.\nThis dataset only contains 14M tokens, which is insufficient for pretraining a language model.  Taking into account the English biomedical pretrained language model \"PubMedBERT\", which was trained on a medical corpus containing 3B tokens. \n\n2.\"It’s hard to understand why the fact knowledge task works. Actually, the improvement brought by the Fact Knowledge task seems to be minor compared with Pas & Ent based on Table 7. It would be appreciated if the authors can provide more discussion about this task.\"\n\nIt's great that you've flagged up this issue.\nIn biomedical texts, entities and their relationships are particularly important for understanding tasks, and the biomedical knowledge base records the rich relationships between entity pairs. Therefore, we came up with the idea of constructing fact granularity aligned corpora and designing relationship token prediction tasks to learn the internal relationships between entities, which is particularly helpful for downstream relation extraction(RE) tasks. Kindly review the performance of the GAD dataset as displayed in the table below. (GAD is a RE task).\n\n|       |  BC5CDR   | GAD  |\n|  ----  | ----  | ---- |\n| KBioXLM | 73.02 | 78.91 |\n| w/o Pas+Fact+Ent  | 67.03 | 75.93 |\n| w only Fact | 70.17 | 77.68 |\n \n\n3.\"Adding English-only and Chinese-only biomedical PLM in results along with multilingual PLMs may provide more insights.\" \n\nI'm thankful for your recognition of this matter.\nIn a fully fine-tuning monolingual scenario, we also conducted experimental comparisons on many different types of tasks. Please refer to Tables 10 and 11 in the appendix. Our model maintains excellent monolingual comprehension performance in English compared with English biomedical SOTA models. Our model performs slightly less effectively in Chinese. However, this is partly due to the fact that training data for other Chinese medical pretrained language models is proprietary and closely aligned with the testing tasks, such as medical consultation data.\n\nWe learn the model in English and test it in Chinese in a 10-shot scenario. And compared the performance of Chinese only biomedical PLM in this scenario. Please refer to Table 5 for relevant experiments. \"CN Bio PLMs\" represent Chinese only biomedical PLMs, including eHealth and SMedBERT. It can be observed that our model performs much better than the current Chinese biomedical monolingual SOTA model in the few shot scenario.\n\nIn conclusion, in the domain of medicine, the majority of literature is in English, and benchmarks in other languages are scarce. Hence, the model's cross-lingual capability is of greater significance. Therefore, in the main experiments, we have contrasted the model's cross-lingual comprehension capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Providing a valuable insight that knowledge (which is usually multilingual in the KB) is essential for bridging multilingual abilities.\n2. Performance of KBioXLM significantly higher than other multilingual biomedical PLMs on biomedical benchmarks.\n3. Proposing a multi-level knowledge-enhanced pretraining method.", "reasons_to_reject": "  1.  As authors use both English and Chinese Wikipedia to build the knowledge-enhanced dataset, it is natural to be curious about how a multilingual pertaining language model directly trained by this bilingual corpus will perform in benchmarks mentioned in the article.\n  2.  It’s hard to understand why the fact knowledge task works. Actually, the improvement brought by the Fact Knowledge task seems to be minor compared with Pas & Ent based on Table 7. It would be appreciated if the authors can provide more discussion about this task.\n  3.  Adding English-only and Chinese-only biomedical PLM in results along with multilingual PLMs may provide more insights.", "questions_to_authors": "See weakness.", "paper_topic_and_main_contributions": "This work proposes a knowledge-centric pretraining strategy specifically aims to improve data efficient in multilingual pertaining on the biomedical domain. Comprehensive experiments on downstream bilingual benchmarks show certain method does reduce dependency on parallel datasets while obtain good performance on biomedical benchmarks.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tajVcrWFxs", "reviewer": "Reviewer_eLNp", "summary": "", "questions": "\n1. The model is currently only for English and Chinese, so it is rather bilingual than multilingual.\n\n2. The introduction is confusing, there is a lot about methods and results, and not much about why building a biliangual model, instead of relying on a cross-ligual, and advantages/disadvantages of each of them.\n\n3. Methods: I miss a schema/figure of the approach and more details about the manual verification for divergences (Page 3, lines 203-205). Further, in section 3.2, I wonder why th authors did no rely on bilingual documents from PubMed, but just the monolinhual ones. \n\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your valuable and comprehensive feedback on our paper.  Your thoughtful observations and suggestions have played a pivotal role in shaping the direction of our research. We value your expertise and commitment to helping us improve our work.\n \n1.\"The model is currently only for English and Chinese, so it is rather bilingual than multilingual.\"\n\nI want to express my gratitude for your notice of this issue.\nOur base model XML-R is multilingual, giving our model the potential to support multiple languages. Additionally, the approach we've employed for incorporating knowledge also caters to multiple languages. However, as stated in the \"Limitations\" section, due to the lack of expertise in other languages, we have only conducted experiments in Chinese and English. We will acknowledge this limitation using a footnote in the introduction. In the future, we plan to explore the cross-lingual capabilities of testing our model in other language datasets that align with English dataset.\n \n2.\"The introduction is confusing, there is a lot about methods and results, and not much about why building a biliangual model, instead of relying on a cross-ligual , and advantages/disadvantages of each of them.\"\n\nYour mention of this problem is valued and acknowledged.\nFirstly, bilingual models can effectively tackle cross-lingual tasks. Furthermore, Chinese-only and English-only tasks are also essential, and we have conducted experiments in the appendix to validate their effectiveness. However, it's evident that cross-lingual capabilities hold more practical value, and we intend to revise the introduction to include additional relevant descriptions.\n \n3.\"Methods: I miss a schema/figure of the approach and more details about the manual verification for divergences (Page 3, lines 203-205). Further, in section 3.2, I wonder why th authors did no rely on bilingual documents from PubMed , but just the monolinhual ones.\"\n\nI'm obliged to you for pointing out this issue.\nFirstly, regarding the issue of manual verification, we invite experts with medical knowledge to assist us in our translation work. \n\nSecondly, thank you very much for providing us with the source of the bilingual PubMed data. Currently, our paper has only utilized parallel Wikipedia data for passage-level knowledge. In the future, we plan to retrieve bilingual PubMed articles to incorporate into our work. The main purpose of section 3.2 is to transition XLM-R from the general field to the medical field. We found that PubMed mainly collects English medical literature and organizes it in a unified data format on huggingface, while CNKI has a more comprehensive collection in Chinese medical literature. Due to our previous data preparation work, we have also accumulated rich CNKI corpus, so we chose CNKI.\n \n4.\"Typos Grammar Style And Presentation Improvements\"\n\nThank you very much for providing these suggestions.\nWe will address the issues you have pointed out in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n- A new bilingual model for en/zh that outperformed previous works for various NLP tasks and datasets.", "reasons_to_reject": "\n- They refer to bilingual model, but it is rather only multilingual.\n- I miss more motivation on building the bilingual model instead of relying on a cross-lingual model", "questions_to_authors": "\n1. The model is currently only for English and Chinese, so it is rather bilingual than multilingual.\n\n2. The introduction is confusing, there is a lot about methods and results, and not much about why building a biliangual model, instead of relying on a cross-ligual, and advantages/disadvantages of each of them.\n\n3. Methods: I miss a schema/figure of the approach and more details about the manual verification for divergences (Page 3, lines 203-205). Further, in section 3.2, I wonder why th authors did no rely on bilingual documents from PubMed, but just the monolinhual ones. \n\n", "paper_topic_and_main_contributions": "\nThe authors present the KBioXLM, a bilingual model (English and Chinese) for the biomedical domain. It is based on the multilingual XLM-R model and incorporates three levels of knowledge (entity, fact, passage). ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\n- Page 1, line 34: \"While, \" -> remove comma\n\n- Page 1, lines 43-45: Maybe put the references at the end of the sentence, not in the middle.\n\n- Page 1, lines 54-59: this passage describes results in the motivation section, it's better left for later.\n\n- Page 2, line 65: Since the proposed system is based on XLM-R, it coud have been introduced with some details before.\n\n- Page 3, line 200: does expertly translated means manually?\n\n- Page 3, lines 219-220: please add more details about the dataset.\n\n- Table 3: throughout the paper, please use either \"cn\" or \"zh\" for Chinese.\n\n\n\n"}}]

Author rebuttal:

Meta-review: Reviewers have found paper to be :
"Well-motivated paper shedding light on the biomedical applications for minor languages."
Demonstrates impressive performance improvement (up to 10+ points) in cross-lingual zero-shot and few-shot scenarios.
Providing a valuable insight that knowledge (which is usually multilingual in the KB) is essential for bridging multilingual abilities."

Furthermore, the authors have been very proactive with sharing additional analysis requested by the reviewers. We suggest the authors to include the same in the revised draft, e.g.
- Performance of the model when there are fewer tokens at each granularity level
- Effectiveness of each granularity separately,
Decision: Accept-Findings