Multilingual Clinical NER: Translation or Cross-lingual Transfer?
Year: 2023

Abstract:
Natural language tasks like Named Entity Recognition (NER) in the clinical domain on non-English texts can be very time-consuming and expensive due to the lack of annotated data. Cross-lingual transfer (CLT) is a way to circumvent this issue thanks to the ability of multilingual large language models to be fine-tuned on a specific task in one language and to provide high accuracy for the same task in another language. However, other methods leveraging translation models can be used to perform NER without annotated data in the target language, by either translating the training set or test set. This paper compares cross-lingual transfer with these two alternative methods, to perform clinical NER in French and in German without any training data in those languages. To this end, we release MedNERF a medical NER test set extracted from French drug prescriptions and annotated with the same guidelines as an English dataset. Through extensive experiments on this dataset and on a German medical dataset (Frei and Kramer, 2021), we show that translation-based methods can achieve similar performance to CLT but require more care in their design. And while they can take advantage of monolingual clinical language models, those do not guarantee better results than large general-purpose multilingual models, whether with cross-lingual transfer or translation.

heading: 1 Introduction
text: In recent years, pre-trained language models based on the Transformer architecture (Vaswani et al., 2017) have demonstrated high performance on many natural language tasks such as Named Entity Recognition (NER), Natural Language Inference or Question-Answering (Devlin et al., 2018; Liu et al., 2019). These models, which are generally pre-trained on general domain data, can be finetuned on downstream tasks to achieve state-of-theart results. Such models can also be adapted to a
∗Equal contribution.
specific domain such as the legal (Chalkidis et al., 2020) or the biomedical fields (Gu et al., 2021) and can then outperform the general-domain models on domain-specific tasks.
Extracting medical entities from unstructured texts has become an essential tool to structure medical reports, and pre-trained language models have been naturally used to perform this task (Khan et al., 2020; Yang et al., 2020). These models use training data for fine-tuning that come from biomedical NER datasets like NCBI-disease (Islamaj Doğan and Lu, 2012) or n2c2 (Henry et al., 2019). However most of these datasets are only available in English and consequently the majority of such medical NER algorithms are developed for the English language, while medical reports or drug prescriptions are rather written in the country’s language.
In the clinical domain, non-English datasets to fine-tune a model are even rarer than in the general domain. Even large domain-specific unlabeled corpora are mostly found in English. For example, the biomedical scientific literature is mostly written and available in English. Moreover, gathering medical texts and annotating them using expert knowledge is very expensive for low-resource languages and more generally for any non-English language. This restricts the development of medical NER models for non-English languages.
Fortunately, Cross-lingual Transfer (CLT) can work around the absence of training data in the target language, by making use of language models pre-trained on multilingual data. CLT consists in applying on a specific task a multilingual large language model (MLLM) fine-tuned in another language for the same task. For example, models like mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) can be fine-tuned on a generaldomain NER task in English and provide competitive results when evaluated in other languages on this task (Pires et al., 2019; Wu and Dredze, 2019).
Before MLLMs, cross-lingual adaptation
289
was generally tackled using translation methods (Yarowsky and Ngai, 2001). The lack of annotated data can be overcome by translating an English training set into the target language and by projecting the annotations with an alignment algorithm. Another approach consists in translating a test set into English to apply an English NER on it and in projecting the predicted labels in the original language.
Recently, translation models based on the Transformer architectures (Ott et al., 2019; Kocmi et al., 2022) have demonstrated huge improvements over other machine translation algorithms. Such models can therefore be leveraged to perform crosslingual learning. However, comparing CLT with translation-based methods has attracted little attention and only comparisons in the general domain (Yarmohammadi et al., 2021) has been done.
Our proposed contribution is three-fold: (1) we perform extensive experiments with CLT and translation-based methods on a medical NER task in French and in German without any training data in those languages, (2) we release MedNERF, a French clinical NER dataset based on drug prescriptions, which serves as a test set for our experiments in French, (3) we demonstrate that CLT and translation provide comparable results for clinical NER, that the choice of the technique depends on the model’s size and that using domain-specific models does not necessarily improve the results over using multilingual models.

heading: 2 Related Work
text: Multilingual large language models (MLLMs). MLLMs are the logical multilingual extension of Large Language Models, trained on multilingual corpora. There exist several versions of MLLMs among which the most popular are mBERT (Devlin et al., 2018), which is BERT multilingual version pre-trained on Wikipedia in a hundred different languages instead on only English Wikipedia (and BookCorpus), and XLM-R (Conneau et al., 2020) which is a multilingual Transformer encoder using the same pre-training principles as RoBERTa (Liu et al., 2019), trained in 100 languages on a larger and more diverse corpus than Wikipedia.
XLM-R, mBERT and its distilled version distilmBERT (Sanh et al., 2019) are not trained on any parallel data and show nevertheless strong CLT abilities, for example on NER tasks, even between languages which use different sets of characters
(Pires et al., 2019; Wu and Dredze, 2019).
Translation-based cross-lingual learning. Cross-lingual learning can also be achieved by using a translation model for either training an algorithm on translated data or using a NER model on translated texts at inference. These techniques (Yarowsky and Ngai, 2001; Yarowsky et al., 2001) make use of translation models and alignment algorithms and have been compared with CLT on general domain tasks by Yarmohammadi et al. (2021) who have shown that using translated and aligned training data improved over zero-shot learning for tasks and languages with weak CLT performances. However this comparison has not been done for domain-specific tasks that often require specific language models like in the medical field. These techniques seem nevertheless to provide good results since they have been leveraged to propose the GERNERMED (Frei and Kramer, 2021) and GERNERMED++ (Frei et al., 2022) models which are medical German NER trained on a German automatic translation of the English dataset n2c2 (Henry et al., 2019).
Performing cross-lingual adaptation in a domainspecific setting raises other questions: such as whether domain-specific language models could be leveraged in translation-based methods, or if translation and alignment models fine-tuned on indomain data can benefit those methods. To the best of our knowledge, those questions have not been addressed in the literature.
Neural machine translation models. Current state-of-the-art machine translation algorithms are based on the Transformer architecture (Kocmi et al., 2022) and can be either encoder-decoder models (Tiedemann and Thottingal, 2020; Ng et al., 2020) or decoder-only models (Gao et al., 2022). The quality of the translation can be assessed by different metrics such as the traditional BLEU score (Papineni et al., 2002), which is a statistical algorithm based on matching n-grams between a proposed translation and a reference one. It has been used for years but new scoring algorithms such as COMET (Rei et al., 2020) which leverage large multilingual models seem to provide more accurate evaluations.
Word alignment algorithms. Word alignment algorithms are designed to provide a mapping between the words of a sentence and those of its translation. They originally relied on statistical fea-
tures like the fast_align algorithm (Dyer et al., 2013) and have been outperformed by models using contextualized embeddings (Jalili Sabet et al., 2020; Dou and Neubig, 2021).
Evaluation corpora for CLT. The CLT abilities of MLLMs can be assessed on several tasks in many languages thanks to multilingual benchmarks like XTREME (Hu et al., 2020) which cannot be used to evaluate medical NER models since they contain only general-domain tasks. Despite the existence of non-English medical NER datasets like QUAERO in French (Névéol et al., 2014) or GGPONC in German (Borchert et al., 2022), CLT cannot be evaluated on these datasets as there is no English counterpart annotated with the same guidelines, which is a pre-requisite for CLT evaluation. In order to tackle this issue, Frei et al. (2022) have introduced a small test dataset of 30 German medical sentences from Electronic Health Records (EHR) annotated in the same way as the n2c2 dataset and have used it to assess the performances of their GERNERMED++ model. Following their path we propose to release a medical NER dataset in French based on drug prescriptions.

heading: 3 Method and models
text: We now describe the different methods we compare to perform NER without any annotations in the target language: cross-lingual transfer and two translation-based methods, where either the train set or the test set is translated. All of them only use the English annotations from the n2c2 dataset (Track 2, Adverse Drug Events and Medication Extraction) (Henry et al., 2019), which is an English dataset of medical entities extracted from EHR.

heading: 3.1 Cross-lingual transfer (CLT)
text: The most intuitive way to perform NER without annotations in the target language is CLT which has shown impressive cross-lingual performances (Wu and Dredze, 2019). In our setting, we fine-tuned a multilingual large language model to perform NER on the English n2c2 dataset and evaluate on French and German test sets. In our experiments, XLM-R Base is preferred over mBERT as it has the same number of layers but outperforms it on multilingual benchmarks. XLM-R Large is also used to evaluate the impact of model size and distilmBERT, a smaller MLLM obtained by distillation of mBERT, is used to give insights about what is possible with less resources.
With CLT, these models will only see English NER labels during training and will be evaluated on a German and a French NER test set.
3.2 translate-train
The translate-train approach consists in constructing a translated version of the n2c2 dataset and in training a NER algorithm in French or German on the translated dataset.
The creation of the synthetic translated dataset is done in two steps. First the whole dataset is translated to the target language using a machine translation algorithm (Tiedemann and Thottingal, 2020; Ng et al., 2020). Then the labels must be aligned, which means identifying in the translated sentences the spans of text corresponding to the original English annotations. This task is tackled using either a statistical model fast_align (Dyer et al., 2013) or the neural algorithm awesome-align (Dou and Neubig, 2021). These alignment tools are applied to the original English sentence and its translation. They provide a mapping between the words of both sentences which is used to transfer the English annotations to the translated sentences. There are cases, namely in German, where the order of the words in a sentence differ from English, when an English annotation corresponds to several disjoint groups of words in the target language. For example, the sentence "She received an additional three units PRBC overnight"1 is translated to "Sie erhielt über Nacht drei weitere PRBC-Einheiten" and the entity "three units" which is translated by "drei Einheiten" is split into two disjoint words. In those cases, all parts of the split entities have been labeled as the original entity.
This method is similar to the one proposed by Frei et al. (2022) and we improve over it by fine-tuning the translation and alignment models on a corpus of parallel medical texts. The influence of using fine-tuned models for translation and alignment is studied in Section 6.2.
3.3 translate-test
The translate-test method consists in translating the data into English at inference time and in applying an English NER model on it. The labels obtained with the NER models can be used to recover the entities in the original text with an alignment algorithm. The major drawback of this method is that it requires to translate the
1PRBC stands for "Packed Red Blood Cells".
text at inference time while the translation in the translate-train method occurs only once during training.

heading: 4 Evaluation data
text: While the model is trained on the English dataset n2c2 or a translation of it, it must be evaluated on French and German data, annotated similarly to n2c2 to assess its cross-lingual adaptation abilities.

heading: 4.1 The MedNERF dataset
text: We release MedNERF2, a Medical NER dataset in the French language. It has been built using a sample of French medical prescriptions annotated with the same guidelines as the n2c2 dataset.
Sentences containing dosage instructions were obtained from a private set of scanned typewritten drug prescriptions. After anonymization of the drug prescriptions we used a state-of-the-art Optical Character Recognition (OCR) software3. We then discarded low quality sentences from the output of the OCR and we manually identified the sentences containing dosage instructions. For the purpose of this paper only 100 sentences have been randomly sampled and made public through the MedNERF dataset, which is intended to be a test and not a training dataset.
The annotations of the medical sentences use the n2c2 labels DRUG, STRENGTH, FREQUENCY, DURATION, DOSAGE and FORM. We did not use the ADE (Adverse Drug Event) label since it is very rare that such entities are present in drug prescriptions. We also discarded the ROUTE and REASON labels as in (Frei et al., 2022) because of either their ambiguous definition or the lack of diversity of the matching samples. A total of 406 entities were annotated in 100 sentences (cf. Table 1)4.
2The dataset is available at https://huggingface. co/datasets/Posos/MedNERF.
3https://cloud.google.com/vision 4Randomly sampled examples in Appendix G.

heading: 4.2 The GERNERMED test dataset
text: The evaluation of the different cross-lingual adaptation techniques in German is done using the GERNERMED test set released by Frei et al. (2022), which consists of 30 sentences from physicians annotated with the same guidelines as n2c2. Table 2 provides statistics about the different datasets used in this paper.

heading: 5 Pre-selecting translation and alignment
text: The translation-based methods require a translation and an alignment models. We present in this section how we fine-tuned translation and alignment algorithms and how we chose which algorithms to use in our experiments. The choice of these algorithms can be seen as an hyper-parameter and for fair comparison with CLT, the selection should not be based on downstream cross-lingual abilities as this would mean cross-lingual supervision.

heading: 5.1 Translation models
text: We perform the automated translation of the n2c2 dataset from English to French and German with the following transformer-based machine translation algorithms: Opus-MT (Tiedemann and Thottingal, 2020) and FAIR (Ng et al., 2020) which we fine-tuned on a corpus of bilingual medical texts proposed in the BioWMT19 challenge 5 (Bawden et al., 2019). We used the UFAL dataset, which is a collection of medical and general domain parallel corpora in 8 languages paired with English, and Medline which is a dataset containing the titles and abstracts of scientific publications from Pubmed in English and a foreign language.
Since the UFAL dataset is orders of magnitude larger than Medline, we downsampled it to have equal proportions of sentences coming from Medline, from the medical part of UFAL and from UFAL general data. This resulted in approximately 90k sentences for the German translation models and 164k sentences for translation into French.
5The links of the datasets of the BioWMT19 challenge are available on its page https://www.statmt.org/ wmt19/biomedical-translation-task.html
We fine-tuned the Opus-MT model (Tiedemann and Thottingal, 2020) for translation to French and German and the FAIR model (Ng et al., 2020) only for translation to German as no version of it is available in French.
The quality of the different translation models is measured on the Medline test set of the BioWMT 19 challenge and on the Khresmoi dataset (Dušek et al., 2017). Results are presented in Tables 3 and 4.
The analysis of these results lead us to choose the FAIR fine-tuned model as the best translation model for English to German translation and the Opus-MT fine-tuned model as the best translation model for English to French translation.
The same pre-selection evaluation was performed for the translate-test approach, with translation models from German and French to English. The models were fine-tuned on the same parallel dataset and similar results led to the same choice of best translation models (cf. Appendix A).

heading: 5.2 Alignment models
text: fast_align and awesome-align are two popular choices for word alignment (Yarmohammadi et al., 2021). Since awesome-align can be fine-tuned on parallel data we use the data used for translation to fine-tune the alignment model.
Choosing the right alignment models for the task can be tricky. While parallel corpora for fine-tuning awesome-align might be available in several languages and domains, annotated word alignment
on parallel data is more scarce. In our case, annotated word alignment test data is not available in the clinical domain. The best alignment models can thus only be selected based on performance on a general-domain dataset. awesome-align pre-trained on general-domain data is preferred in French, and the same model with further finetuning on biomedical data is selected for German.
Table 6 summarizes the choices of the best translation and alignment methods.

heading: 6 Results
text: Having selected the best translation and alignment model for each language based on intrinsic evaluation, translate-train and translate-test approaches can be compared to Cross-lingual Transfer (CLT). The impact of the translation and alignment models can also be analyzed, as well as using the translation fine-tuning data to improve directly the models used in CLT. Since translate-train can leverage monolingual domain-specific model, we evaluate the outcome of such a strategy. Five different random seeds were used for each model and results presented in this section show the average performance along with the standard deviation (more implementation details in Appendix B).

heading: 6.1 Comparison of the different methods
text: For a fair comparison between the three methods detailed in Section 3, we use the pre-selected translation and alignment models of Table 6 for the translate-train and translate-test methods. We report the F1-scores of the different methods in Table 7. The translation and alignment models providing the best test scores are also
provided for comparison, revealing what can be missed with the pre-selection.
CLT with a sufficiently large MLLM provides the best results. When compared with translationbased methods with pre-selected translation and alignment models, CLT with XLM-R models gives higher scores, except for XLM-R Base in German.
On the other hand, it seems that using an English NER on a translated version of the test set provides the best results with a small model like distilmBERT. DistilmBERT might be better as a monolingual model than a multilingual one. In the same vein, XLM-R Base struggles in German, while its large version does not. Small language models underperform in CLT and their generalization ability is not sufficient compared to translationbased methods. A first take-away is consequently that translation-based methods should be favored with small language models.
The translate-testmethod is consistently outperformed by translate-train for large models. Even using a specific biomedical model like (Gu et al., 2021) for translate-test does not improve the results (results in Appendix F). Indeed translation and alignment errors only harm the training set of translate-train, which does
not prevent a large model from generalizing despite some errors in the training data, while errors of translation or alignment in translate-test are directly reflected in the test score. In the rest of this analysis we will consequently compare CLT only with translate-train.
Providing a large-enough MLLM, CLT outperforms pre-selected translate-train and translate-test. However, choosing the translation and the alignment model beforehand does not lead to the best results. To the exception of XLM-R Base in French, there always exists a pair of translation and alignment models that leads to a better score for the translate-train method over CLT. This agrees with Yarmohammadi et al. (2021) and encourages practitioners to explore different methods to perform cross-lingual adaptation.

heading: 6.2 Influence of the translation model
text: The choice of the translation and alignment models can have an important impact on the final NER performances as shown on Table 7. This section studies their impact in details. A German NER model was trained using the Opus model instead of the FAIR model for translating the training set into German. Using a worse model (see Table 3) for translation leads to lower NER scores as shown in Table 8: the NER model based on the FAIR translation beats by more than 2 points the one using the Opus translation, whatever aligner is used.
While choosing between different base translation models (like Opus or FAIR) based on their translation scores on in-domain data seems to provide the best results, deciding between the finetuned version of a translation model and the base one by comparing the BLEU or COMET scores on biomedical data does not guarantee the best downstream F1 score as Table 9 shows. The translation model was fine-tuned on biomedical data, which improved intrinsic results on the BioWMT19 translation dataset. But this dataset belongs to a specific biomedical sub-domain (PubMed abstracts), and fine-tuning might not improve translation for the clinical sub-domain of the NER dataset.
The takeaway is that, while a small gain in translation accuracy (obtained with further fine-tuning) might not necessarily improve the result of the translate-train approach, a completely different model (like FAIR with respect to Opus) has more chance to improve cross-lingual adaptation.

heading: 6.3 Influence of the alignment model
text: While choosing a translation model based solely on intrinsic performance should not harm downstream cross-lingual adaptation performances, the choice of the alignment model seems more tricky. Based on intrinsic performances like Error Rate on annotated alignment (Table 5), awesome-align seems to be the right aligner for the task. However, while it provides better downstream results than fast_align in German (Table 8), it does not hold for French (Table 9).
Table 10 shows that using different aligners leads to different levels of accuracy according to the types of entity we want to retrieve. While the global F1 score suggests that fast_align is better suited for the cross-lingual adaptation, looking at the detailed results for each entity type shows that the gap is mainly due to the FREQUENCY class on which awesome-align performs poorly. But this is not the case on other classes. FREQUENCY entities are usually more verbose than drugs or dosages. Table 11 shows that fast_align make obvious errors like aligning "240 mg" to "in morning and night", but
awesome-align can miss the preposition when aligning "daily" with "jour" instead of "par jour", leading eventually to a consequent score drop.
The choice of the alignment model must thus be made more carefully than the translation one. Intrinsic performances of alignment models are not sufficient information. Some additional postprocessing might be needed, as in Yarmohammadi et al. (2021), where awesome-align gives better results, but entities that are split by the aligner like "au moment du coucher" in Table 11 are merged by including all words in between. This would work in that particular case, but could cause problems in others, particularly for languages where the word order is different.

heading: 6.4 Using parallel data to realign models
text: With the right translation and alignment model, it seems that CLT can be outperformed by the translate-train method. However the latter relies on additional resources: a translation and an alignment models, trained on parallel data. This parallel data could also be used to re-align the representations of the multilingual models used in CLT.
To improve a multilingual language model with parallel data, it is trained for a contrastive alignment objective following Wu and Dredze (2020). Words aligned with awesome-align are trained to have more similar representations than random in-batch pairs of words (details in Appendix E). After this realignment step, CLT can be applied.
Results in Table 12 show that while realignment does not systematically provide improvement over CLT as observed by Wu and Dredze (2020), it does significantly boost results in some cases, allowing to outperform the best translate-train baseline in German for XLM-R Base and in French for XLM-R Large. This, yet again, encourages practitioners to explore different methods, including realignment to perform cross-lingual adaptation.

heading: 6.5 Using domain-specific language models
text: We evaluate now the relevance of using languagespecific models like CamemBERT (Martin et al., 2020) or GottBERT (Scheible et al., 2020) on the translated version of the training dataset or language and domain-specific models like DrBERT (Labrak et al., 2023) or medBERT.de (Bressem et al., 2023) which are BERT models fine-tuned on medical corpora in respectively French and German. We report in Table 13 and 14 the results of the translate-train method for the best translation/alignment algorithms pair and for the pre-selected one, compared to using XLM-R Base.
The translate-train approach allows to rely on models that are specific to the language and domain of the target evaluation. However, Table 13 and 14 show that their use does not always bring significant improvement over XLM-R Base. The
performances of these models can be explained by the quantity of training data used. XLM-R models are indeed trained on 2.5 TB data while DrBERT and medBERT.de use less than 10GB data, which can explain their low score. Besides, the language-specific models CamemBERT and GottBERT are trained with more data (138 GB and 145 GB) and achieve better performances, even beating XLM-R in German. Finally, it must be noted that the best translate-train model in French, DrBERT Pubmed, is actually pre-trained on the English PubMed dataset and then on French clinical texts, which suggests that multilingual models should be preferred, even with a translation-based cross-lingual adaptation.

heading: 6.6 Computing times
text: To conclude the analysis of the different crosslingual adaptation methods studied in this paper we finally compare their computing times. Table 15 gathers the training and inference times of the three methods using the XLM-R base model and the awesome-align alignment model in French.
This comparison shows a longer training time for the translate-train method, which is due to the translation of the whole training set before the training of the NER model. On a single GPU with 8GB of RAM this translation step is even longer than the NER training. However, once training is done the translate-train method has the same inference time as the CLT method, while the translate-test method now suffers from the need of translation at inference time.

heading: 7 Conclusion
text: This paper shows that cross-lingual transfer with general-domain MLLMs is efficient for a domainspecific task like clinical NER, giving comparable results with translating the training set. But CLT has the advantage of working off-the-shelf, while translation-based methods require choosing
the translation and alignment models carefully. Selecting these models based on intrinsic domainspecific values, like fine-tuning scores on clinical parallel data, or using a domain-specific language model does not provide significantly better downstream results in the target language. The selection of the alignment model was shown to be particularly crucial, and results of translate-train could probably be improved by post-processing the alignment. CLT also has a margin of progression as realigning the representations of MLLMs can increase the results dramatically in some cases.
It is also worth noting that training on translated data provide better results than translating at inference time. The translate-test approach should then be used only when large multilingual models cannot be used. While training on translated data allows to leverage domain-specific monolingual language models, those latter models can give better results over multilingual models like XLM-R only if pre-trained with sufficient data.
Pre-training a MLLM with only clinical data is a good lead for further improvements in clinical cross-lingual transfer. While the results show that using a domain-specific monolingual model in translate-train or translate-test is not on par with general-purpose multilingual models, they also show that the French clinical model DrBERT provides the best results for translate-train when it uses the English biomedical model PubmedBERT as initialization.
We finally advocate for the release of more non-English clinical datasets annotated with similar guidelines as English (or other) ones. Even a relatively small dataset like MedNERF or the GERNERMED test set are crucial to evaluate crosslingual adaptation in the clinical domain.
Limitations
This paper is limited to the study of clinical NER models using an encoder-only architecture. The use of generative models with a zero-shot learning approach (Hu et al., 2023) is another promising approach for low-resource languages that could be compared with CLT and translation-based approaches in a future work. However such methods require a careful prompt selection strategy and cannot be directly compared to supervised models.
This paper is also limited to cross-lingual transfer to French and German. Ideally, this work could have included experiments with other target lan-
guages and also other source languages than English, as Yarmohammadi et al. (2021) do in their general-domain comparison of strategies for crosslingual transfer. However evaluation datasets are lacking for that purpose in the clinical domain. Similarly, more general conclusions about cross-lingual adaptation methods in the clinical domains could be drawn with further studies on various clinical NLP tasks such as relation extraction or sentence classification. However, the lack of evaluation datasets in the clinical domain prevented us from extending the experiments to such other clinical NLP tasks. Finally, the authors assume that the findings will be task-specific and encourage practitioners to explore all methods when facing a new NLP task.
The authors also want to point out that MedNERF is drawn from drug prescriptions while the n2c2 and GERNERMED datasets use clinical reports. This domain difference could have made the cross-lingual generalization more challenging, but in practice we found that the different models used were not really affected by the possible domain-shift, showing similar French and German F1 scores. Moreover, when comparing randomly sampled examples from all three datasets, we do not find any critical differences (see Appendix G). Sentences drawn from MedNERF are shorter and less written, but they contain similar annotated entities as the n2c2 sentences, and the n2c2 dataset also contains some short examples that resemble the MedNERF ones.

heading: Acknowledgments
text: We would like to thank the anonymous reviewers for their comments which helped enrich the discussion around the results.
Some experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations6.

heading: B Training and implementation details
text: All models were written in Pytorch using the Huggingface librairies (Wolf et al., 2020) and were finetuned using the AdamW optimizer (Loshchilov and Hutter, 2019) and a learning rate of 6 · 10−6 with linear decay. We used 4 epochs for the translation models and 8 epochs for the NER models. The NER models were trained on a single GPU (Nvidia GeForce RTX 2070 with 8GB of RAM) for approximately one hour.

heading: C Alignment methods used
text: fast_align was applied asymmetrically, by mapping words from source language (English) to target language. Although it might increase alignment score, symmetrization was not used, because it might remove important links during the labels projection step.
awesome-align was used with softmax (instead of the alternative α-entmax function) and without the optional consistency optimization objective. For completeness we added the results with the consistency optimization objective (w/ co lines) in the tables of Appendix F and we observed that they did not improve the NER scores. The base model used is mBERT as in the original paper (Dou and Neubig, 2021). The pre-trained version of awesome-align used is the one provided by the authors, fine-tuned on general-domain parallel data. Throughout the paper, in tables, "AWESOME" designates this latter pre-trained version. "AWESOME ft" is awesome-align with the raw mBERT model fine-tuned on the clinical parallel data only and "AWESOME pt+ft" is the pretrained model, fine-tuned again on the clinical parallel data.

heading: D Transformer-base models used
text: We report in Table 18 the number of parameters and the quantity of training data of the different large language models used in this paper.
Although distilmBERT has more parameters than CamemBERT, it must be noted that it has also more words in its vocabulary, due to its multilingual nature. Hence most of its parameters are embeddings weights that are not necessarily used in our experiments as they might be embeddings of words from other languages. So in our setting, distilmBERT can be considered a smaller model than CamemBERT and GottBERT despite the higher number of parameters.

heading: E Realignment method
text: The reader might refer to Wu and Dredze (2020) for the realignment method itself. The representations of the last layer of the pre-trained model to be realigned were used in a contrastive loss where pairs of words aligned with awesome-align are encouraged to be more similar than the other possible pairs of words in the batch. The strong alignment objective was used, meaning that pair of same-language words were also used as negative examples for the contrastive loss. The version of awesome-align was the one pre-trained on general-domain data, released by Dou and Neubig (2021), with softmax and without the optional consistency optimization objective, the same used by the authors of the realignment method used.
The parallel data used for realignment was the same as for fine-tuning the translation and alignment models (Section 5.1). The two datasets (English-German and English-French) were used together to realign a given model, which can then be used either for generalization to French or German. For each base model, five realigned models were obtained for the five random seeds, each of them used in the corresponding fine-tuning by seed.
The realignment was done for 20,000 steps of batches of size 16, with Adam optimizer, a learning rate of 2× 10−4, and with linear warm-up for 10% of the total steps. This means that the whole dataset was repeated approximately 1.25 times.

heading: F Additional results
text: Detailed results are shown in the following tables:
• Summary of results for cross-lingual adaptation, with pre-selected and best pairs of translation and alignment models: Table 19 for French and 20 for German;
• CLT and translate-trainwith multilingual models: Table 21 (fr) and 22 (de);
• translate-train with language- and domain-specific models: Table 23 and 24;
• translate-test with multilingual language models: Table 25 and 26;
• translate-test with PubmedBERT: Table 27 and 28;
• Breakdown of the results class-by-class in French for multilingual models: Table 29.
∗results reported twice as there is no pre-selection process

heading: G Examples from the different datasets
text: EXAMPLES FROM N2C2 The patient’s agitation was managed with nightlyFrequency haldolDrug with as neededFrequency haldolDrug as well. Improvement in clinical status was noted overnight and his morphineDrug drip was discontinued. - hold all antihypertensivesDrug ; plan to add back slowly at reduced doses and varying schedule - rule out MI - bolusDosage NSDrug to maintain MAP > 60 with caution given ESRD and oliguric. folic acidDrug 1 mgStrength TabletForm Sig: One (1)Dosage TabletForm PO DAILY (Daily)Frequency. IronDrug 50 mgStrength TabletForm Sustained Release Sig: One (1)Dosage Tablet Sustained ReleaseForm PO once a dayFrequency.
EXAMPLES FROM GERNERMED TEST SET
Das EplerenonDrug ist wegen Ihrer Herzinsuffizienz. Da können wir jetzt auf 50 mgStrength p.o. 1-0-0 augmentieren. Eplerenon is for your heart failure. We can now augment to 50mg p.o. 1-0-0. Wegen der COPD-Exazerbation wurde TerbutalinDrug 0,25 mgStrength dem Patienten appliziert. Hierfür wurde der subkutane Weg gewählt.
Because of the COPD exacerbation, terbutaline 0.25 mg was administered to the patient. The subcutaneous route was chosen for this.
Bei Vorhofflimmern ist neben BetablockernDrug auch die Gabe von MagnesiumDrug p.o. sinnvoll. Hierfür würden wir mit 300 mgStrength einmal täglichFrequency starten. Sofern möglich, ist eine Einnahme mittagsFrequency (ca. 12 Uhr) zu bevorzugen.
In atrial fibrillation, in addition to beta-blocks, the administration of magnesium p.o. is also useful. For this we would start with 300 mg once a day. If possible, it is preferable to take it at noon (around 12 o’clock).
Zur Optimierung der Herzinsuffizienztherapie wurde die Dosis von Sacubitril / ValsartanDrug auf 97 / 103 mgStrength in TablettenformForm mit Einnahme am Morgen und am AbendFrequency erweitert.
To optimize heart failure therapy, the dose of sacubitril / valsartan was extended to 97 / 103 mg in tablet form with intake in the morning and evening.
Bei bekannter koronarer Herzerkrankung sollte lebenslangDuration AcetylsalicylsäureDrug 100 mgStrength morgens täglichFrequency in oraler Applikation eingenommen werden.
In cases of known coronary artery disease, acetylsalicylic acid 100mg should be taken orally daily in the morning as a lifelong treatment.
EXAMPLES FROM MEDNERF
TRAMADOL / PARACETAMOLDrug 37,5mg / 325mgStrength TRAMADO/PARACETAMOL 37,5mg/325mg AMLODIPINEDrug 5 mgStrength ; cprForm 1Dosage compriméForm matinFrequency 1Dosage compriméForm soirFrequency AMLODIPINE 5mg; tab 1 tablet in the mording 1 tablet in the evening DOLIPRANETABSDrug 1000 MGStrength CPR PELLForm PLQ / 8 (ParacétamolDrug 1.000 mgStrength compriméForm) DOLIPRANETABS 1000mg TAB PLQ / 8 (Paracetamol 1,000mg tablet) ACIDE ACETYLSALICYLIQUEDrug (sel de lysineDrug) 75 mgStrength pdre p sol buv sachForm (KARDEGICDrug)
ACETYLSALICYLIC ACID (lysine salts) 75mg oral powder for suspension (KARDEGIC)
1Dosage sachetForm matin midi et soirFrequency si besoin 1 packet in the morning, at noon, and in the evening, if needed

heading: Future Work
text: , 2023) is another promising approach for low-resource languages that could be compared with CLT and translation-based approaches in a future work. However such methods require a careful prompt selection strategy and cannot be directly compared to supervised models. This paper is also limited to cross-lingual transfer to French and German. Ideally, this work could have included experiments with other target lan-
guages and also other source languages than English, as Yarmohammadi et al. (2021) do in their general-domain comparison of strategies for crosslingual transfer. However evaluation datasets are lacking for that purpose in the clinical domain