# Multilingual Clinical NER: Translation or Cross-lingual Transfer?

Félix Gaschi, Xavier Fontaine, Parisa Rastin, Yannick Toussaint

► To cite this version:

Félix Gaschi, Xavier Fontaine, Parisa Rastin, Yannick Toussaint. Multilingual Clinical NER: Translation or Cross-lingual Transfer?. 5th Clinical Natural Language Processing Workshop, Jul 2023, Toronto, Canada. pp.289-311, 10.18653/v1/2023.clinicalnlp-1.34. hal-04193182

HAL Id: hal-04193182

https://hal.science/hal-04193182v1

Submitted on 1 Sep 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Multilingual Clinical NER: Translation or Cross-lingual Transfer?

Félix Gaschi¹,²* Xavier Fontaine¹* Parisa Rastin² Yannick Toussaint²

¹SAS Posos, France   ²LORIA, France

{xavier,felix}@posos.fr

{felix.gaschi,parisa.rastin,yannick.toussaint}@loria.fr

## Abstract

Natural language tasks like Named Entity Recognition (NER) in the clinical domain on non-English texts can be very time-consuming and expensive due to the lack of annotated data. Cross-lingual transfer (CLT) is a way to circumvent this issue thanks to the ability of multilingual large language models to be fine-tuned on a specific task in one language and to provide high accuracy for the same task in another language. However, other methods leveraging translation models can be used to perform NER without annotated data in the target language, by either translating the training set or test set. This paper compares cross-lingual transfer with these two alternative methods, to perform clinical NER in French and in German without any training data in those languages. To this end, we release MedNERF a medical NER test set extracted from French drug prescriptions and annotated with the same guidelines as an English dataset. Through extensive experiments on this dataset and on a German medical dataset (Frei and Kramer, 2021), we show that translation-based methods can achieve similar performance to CLT but require more care in their design. And while they can take advantage of monolingual clinical language models, those do not guarantee better results than large general-purpose multilingual models, whether with cross-lingual transfer or translation.

## 1 Introduction

In recent years, pre-trained language models based on the Transformer architecture (Vaswani et al., 2017) have demonstrated high performance on many natural language tasks such as Named Entity Recognition (NER), Natural Language Inference or Question-Answering (Devlin et al., 2018; Liu et al., 2019). These models, which are generally pre-trained on general domain data, can be fine-tuned on downstream tasks to achieve state-of-the-art results. Such models can also be adapted to a

specific domain such as the legal (Chalkidis et al., 2020) or the biomedical fields (Gu et al., 2021) and can then outperform the general-domain models on domain-specific tasks.

Extracting medical entities from unstructured texts has become an essential tool to structure medical reports, and pre-trained language models have been naturally used to perform this task (Khan et al., 2020; Yang et al., 2020). These models use training data for fine-tuning that come from biomedical NER datasets like NCBI-disease (Islamaj Doğan and Lu, 2012) or n2c2 (Henry et al., 2019). However most of these datasets are only available in English and consequently the majority of such medical NER algorithms are developed for the English language, while medical reports or drug prescriptions are rather written in the country's language.

In the clinical domain, non-English datasets to fine-tune a model are even rarer than in the general domain. Even large domain-specific unlabeled corpora are mostly found in English. For example, the biomedical scientific literature is mostly written and available in English. Moreover, gathering medical texts and annotating them using expert knowledge is very expensive for low-resource languages and more generally for any non-English language. This restricts the development of medical NER models for non-English languages.

Fortunately, Cross-lingual Transfer (CLT) can work around the absence of training data in the target language, by making use of language models pre-trained on multilingual data. CLT consists in applying on a specific task a multilingual large language model (MLLM) fine-tuned in another language for the same task. For example, models like mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) can be fine-tuned on a general-domain NER task in English and provide competitive results when evaluated in other languages on this task (Pires et al., 2019; Wu and Dredze, 2019).

Before MLLMs, cross-lingual adaptation

<footnote>*Equal contribution.</footnote>
was generally tackled using translation methods (Yarowsky and Ngai, 2001). The lack of annotated data can be overcome by translating an English training set into the target language and by projecting the annotations with an alignment algorithm. Another approach consists in translating a test set into English to apply an English NER on it and in projecting the predicted labels in the original language.

Recently, translation models based on the Transformer architectures (Ott et al., 2019; Kocmi et al., 2022) have demonstrated huge improvements over other machine translation algorithms. Such models can therefore be leveraged to perform cross-lingual learning. However, comparing CLT with translation-based methods has attracted little attention and only comparisons in the general domain (Yarmohammadi et al., 2021) has been done.

Our proposed contribution is three-fold: (1) we perform extensive experiments with CLT and translation-based methods on a medical NER task in French and in German without any training data in those languages, (2) we release MedNERF, a French clinical NER dataset based on drug prescriptions, which serves as a test set for our experiments in French, (3) we demonstrate that CLT and translation provide comparable results for clinical NER, that the choice of the technique depends on the model's size and that using domain-specific models does not necessarily improve the results over using multilingual models.

## 2 Related Work

**Multilingual large language models (MLLMs).** MLLMs are the logical multilingual extension of Large Language Models, trained on multilingual corpora. There exist several versions of MLLMs among which the most popular are mBERT (Devlin et al., 2018), which is BERT multilingual version pre-trained on Wikipedia in a hundred different languages instead on only English Wikipedia (and BookCorpus), and XLM-R (Conneau et al., 2020) which is a multilingual Transformer encoder using the same pre-training principles as RoBERTa (Liu et al., 2019), trained in 100 languages on a larger and more diverse corpus than Wikipedia.

XLM-R, mBERT and its distilled version distilmBERT (Sanh et al., 2019) are not trained on any parallel data and show nevertheless strong CLT abilities, for example on NER tasks, even between languages which use different sets of characters

(Pires et al., 2019; Wu and Dredze, 2019).

**Translation-based cross-lingual learning.** Cross-lingual learning can also be achieved by using a translation model for either training an algorithm on translated data or using a NER model on translated texts at inference. These techniques (Yarowsky and Ngai, 2001; Yarowsky et al., 2001) make use of translation models and alignment algorithms and have been compared with CLT on general domain tasks by Yarmohammadi et al. (2021) who have shown that using translated and aligned training data improved over zero-shot learning for tasks and languages with weak CLT performances. However this comparison has not been done for domain-specific tasks that often require specific language models like in the medical field. These techniques seem nevertheless to provide good results since they have been leveraged to propose the GERNERMED (Frei and Kramer, 2021) and GERNERMED++ (Frei et al., 2022) models which are medical German NER trained on a German automatic translation of the English dataset n2c2 (Henry et al., 2019).

Performing cross-lingual adaptation in a domain-specific setting raises other questions: such as whether domain-specific language models could be leveraged in translation-based methods, or if translation and alignment models fine-tuned on in-domain data can benefit those methods. To the best of our knowledge, those questions have not been addressed in the literature.

**Neural machine translation models.** Current state-of-the-art machine translation algorithms are based on the Transformer architecture (Kocmi et al., 2022) and can be either encoder-decoder models (Tiedemann and Thottingal, 2020; Ng et al., 2020) or decoder-only models (Gao et al., 2022). The quality of the translation can be assessed by different metrics such as the traditional BLEU score (Papineni et al., 2002), which is a statistical algorithm based on matching n-grams between a proposed translation and a reference one. It has been used for years but new scoring algorithms such as COMET (Rei et al., 2020) which leverage large multilingual models seem to provide more accurate evaluations.

**Word alignment algorithms.** Word alignment algorithms are designed to provide a mapping between the words of a sentence and those of its translation. They originally relied on statistical fea-
tures like the *fastAlign* algorithm (Dyer et al., 2013) and have been outperformed by models using contextualized embeddings (Jalili Sabet et al., 2020; Dou and Neubig, 2021).

**Evaluation corpora for CLT.** The CLT abilities of MLLMs can be assessed on several tasks in many languages thanks to multilingual benchmarks like XTREME (Hu et al., 2020) which cannot be used to evaluate medical NER models since they contain only general-domain tasks. Despite the existence of non-English medical NER datasets like QUAERO in French (Névéol et al., 2014) or GGPONC in German (Borchert et al., 2022), CLT cannot be evaluated on these datasets as there is no English counterpart annotated with the same guidelines, which is a pre-requisite for CLT evaluation. In order to tackle this issue, Frei et al. (2022) have introduced a small test dataset of 30 German medical sentences from Electronic Health Records (EHR) annotated in the same way as the n2c2 dataset and have used it to assess the performances of their GERNERMED++ model. Following their path we propose to release a medical NER dataset in French based on drug prescriptions.

# 3 Method and models

We now describe the different methods we compare to perform NER without any annotations in the target language: cross-lingual transfer and two translation-based methods, where either the train set or the test set is translated. All of them only use the English annotations from the n2c2 dataset (Track 2, Adverse Drug Events and Medication Extraction) (Henry et al., 2019), which is an English dataset of medical entities extracted from EHR.

## 3.1 Cross-lingual transfer (CLT)

The most intuitive way to perform NER without annotations in the target language is CLT which has shown impressive cross-lingual performances (Wu and Dredze, 2019). In our setting, we fine-tuned a multilingual large language model to perform NER on the English n2c2 dataset and evaluate on French and German test sets. In our experiments, XLM-R Base is preferred over mBERT as it has the same number of layers but outperforms it on multilingual benchmarks. XLM-R Large is also used to evaluate the impact of model size and distilmBERT, a smaller MLLM obtained by distillation of mBERT, is used to give insights about what is possible with less resources.

With CLT, these models will only see English NER labels during training and will be evaluated on a German and a French NER test set.

## 3.2 translate-train

The *translate-train* approach consists in constructing a translated version of the n2c2 dataset and in training a NER algorithm in French or German on the translated dataset.

The creation of the synthetic translated dataset is done in two steps. First the whole dataset is translated to the target language using a machine translation algorithm (Tiedemann and Thottingal, 2020; Ng et al., 2020). Then the labels must be aligned, which means identifying in the translated sentences the spans of text corresponding to the original English annotations. This task is tackled using either a statistical model *fastAlign* (Dyer et al., 2013) or the neural algorithm *awesome-align* (Dou and Neubig, 2021). These alignment tools are applied to the original English sentence and its translation. They provide a mapping between the words of both sentences which is used to transfer the English annotations to the translated sentences. There are cases, namely in German, where the order of the words in a sentence differ from English, when an English annotation corresponds to several disjoint groups of words in the target language. For example, the sentence "She received an additional three units PRBC overnight"<fnref n="1" /> is translated to "Sie erhielt über Nacht drei weitere PRBC-Einheiten" and the entity "three units" which is translated by "drei Einheiten" is split into two disjoint words. In those cases, all parts of the split entities have been labeled as the original entity.

This method is similar to the one proposed by Frei et al. (2022) and we improve over it by fine-tuning the translation and alignment models on a corpus of parallel medical texts. The influence of using fine-tuned models for translation and alignment is studied in Section 6.2.

## 3.3 translate-test

The *translate-test* method consists in translating the data into English at inference time and in applying an English NER model on it. The labels obtained with the NER models can be used to recover the entities in the original text with an alignment algorithm. The major drawback of this method is that it requires to translate the

<footnote n="1">¹PRBC stands for "Packed Red Blood Cells".</footnote>
text at inference time while the translation in the
`translate-train` method occurs only once
during training.

# 4 Evaluation data

While the model is trained on the English dataset n2c2 or a translation of it, it must be evaluated on French and German data, annotated similarly to n2c2 to assess its cross-lingual adaptation abilities.

## 4.1 The MedNERF dataset

We release MedNERF<fnref n="2" />, a Medical NER dataset in the French language. It has been built using a sample of French medical prescriptions annotated with the same guidelines as the n2c2 dataset.

Sentences containing dosage instructions were obtained from a private set of scanned typewritten drug prescriptions. After anonymization of the drug prescriptions we used a state-of-the-art Optical Character Recognition (OCR) software<fnref n="3" />. We then discarded low quality sentences from the output of the OCR and we manually identified the sentences containing dosage instructions. For the purpose of this paper only 100 sentences have been randomly sampled and made public through the MedNERF dataset, which is intended to be a test and not a training dataset.

The annotations of the medical sentences use the n2c2 labels DRUG, STRENGTH, FREQUENCY, DURATION, DOSAGE and FORM. We did not use the ADE (Adverse Drug Event) label since it is very rare that such entities are present in drug prescriptions. We also discarded the ROUTE and REASON labels as in (Frei et al., 2022) because of either their ambiguous definition or the lack of diversity of the matching samples. A total of 406 entities were annotated in 100 sentences (cf. Table 1)<fnref n="4" />.

<table><caption>Table 1: Distribution of labels in MedNERF.</caption><thead><tr><td>NER Tag</td><td>Count</td></tr></thead><tbody><tr><td>DRUG</td><td>67</td></tr><tr><td>STRENGTH</td><td>51</td></tr><tr><td>FREQUENCY</td><td>76</td></tr><tr><td>DURATION</td><td>43</td></tr><tr><td>DOSAGE</td><td>76</td></tr><tr><td>FORM</td><td>93</td></tr><tr><td>Total</td><td>406</td></tr></tbody></table>

## 4.2 The GERNERMED test dataset

The evaluation of the different cross-lingual adaptation techniques in German is done using the GERNERMED test set released by Frei et al. (2022), which consists of 30 sentences from physicians annotated with the same guidelines as n2c2. Table 2 provides statistics about the different datasets used in this paper.

<table><caption>Table 2: Statistics about the datasets.</caption><thead><tr><td>dataset</td><td>lang.</td><td>sent.</td><td>entities</td></tr></thead><tbody><tr><td>n2c2</td><td>en</td><td>16,656</td><td>65,495</td></tr><tr><td>GERNERMED-test</td><td>de</td><td>30</td><td>119</td></tr><tr><td>MedNERF</td><td>fr</td><td>100</td><td>406</td></tr></tbody></table>

# 5 Pre-selecting translation and alignment

The translation-based methods require a translation and an alignment models. We present in this section how we fine-tuned translation and alignment algorithms and how we chose which algorithms to use in our experiments. The choice of these algorithms can be seen as an hyper-parameter and for fair comparison with CLT, the selection should not be based on downstream cross-lingual abilities as this would mean cross-lingual supervision.

## 5.1 Translation models

We perform the automated translation of the n2c2 dataset from English to French and German with the following transformer-based machine translation algorithms: Opus-MT (Tiedemann and Thottingal, 2020) and FAIR (Ng et al., 2020) which we fine-tuned on a corpus of bilingual medical texts proposed in the BioWMT19 challenge <fnref n="5" /> (Bawden et al., 2019). We used the UFAL dataset, which is a collection of medical and general domain parallel corpora in 8 languages paired with English, and Medline which is a dataset containing the titles and abstracts of scientific publications from Pubmed in English and a foreign language.

Since the UFAL dataset is orders of magnitude larger than Medline, we downsampled it to have equal proportions of sentences coming from Medline, from the medical part of UFAL and from UFAL general data. This resulted in approximately 90k sentences for the German translation models and 164k sentences for translation into French.

<footnote n="2">²The dataset is available at https://huggingface.co/datasets/Posos/MedNERF.</footnote>

<footnote n="3">³https://cloud.google.com/vision</footnote>

<footnote n="4">⁴Randomly sampled examples in Appendix G.</footnote>

<footnote n="5">⁵The links of the datasets of the BioWMT19 challenge are available on its page https://www.statmt.org/wmt19/biomedical-translation-task.html</footnote>
We fine-tuned the Opus-MT model (Tiedemann and Thottingal, 2020) for translation to French and German and the FAIR model (Ng et al., 2020) only for translation to German as no version of it is available in French.

The quality of the different translation models is measured on the Medline test set of the BioWMT 19 challenge and on the Khresmoi dataset (Dušek et al., 2017). Results are presented in Tables 3 and 4.

<table><caption>Table 3: Evaluation of the translation models from English to German. Best model in bold and second underlined. ft for finetuned.</caption><thead><tr><th rowspan="2">model</th><th colspan="2">BioWMT19</th><th colspan="2">Khresmoi</th></tr><tr><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th></tr></thead><tbody><tr><td>FAIR</td><td><u>32.8</u></td><td>0.628</td><td><strong>33.7</strong></td><td><strong>0.667</strong></td></tr><tr><td>+ ft</td><td><strong>34.2</strong></td><td><strong>0.734</strong></td><td><u>32.4</u></td><td><u>0.666</u></td></tr><tr><td>Opus</td><td>32.2</td><td>0.651</td><td><u>32.4</u></td><td>0.608</td></tr><tr><td>+ ft</td><td>32.5</td><td><u>0.700</u></td><td>30.5</td><td>0.619</td></tr></tbody></table>

<table><caption>Table 4: Evaluation of the translation models from English to French.</caption><thead><tr><th rowspan="2">model</th><th colspan="2">BioWMT19</th><th colspan="2">Khresmoi</th></tr><tr><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th></tr></thead><tbody><tr><td>Opus</td><td>35.9</td><td>0.672</td><td><strong>48.0</strong></td><td><strong>0.791</strong></td></tr><tr><td>+ ft</td><td><strong>36.7</strong></td><td><strong>0.786</strong></td><td>46.5</td><td><strong>0.791</strong></td></tr></tbody></table>

The analysis of these results lead us to choose the FAIR fine-tuned model as the best translation model for English to German translation and the Opus-MT fine-tuned model as the best translation model for English to French translation.

The same pre-selection evaluation was performed for the *translate-test* approach, with translation models from German and French to English. The models were fine-tuned on the same parallel dataset and similar results led to the same choice of best translation models (cf. Appendix A).

## 5.2 Alignment models

`fast align` and `awesome-align` are two popular choices for word alignment (Yarmoham-madi et al., 2021). Since `awesome-align` can be fine-tuned on parallel data we use the data used for translation to fine-tune the alignment model.

Choosing the right alignment models for the task can be tricky. While parallel corpora for fine-tuning `awesome-align` might be available in several languages and domains, annotated word alignment

on parallel data is more scarce. In our case, annotated word alignment test data is not available in the clinical domain. The best alignment models can thus only be selected based on performance on a general-domain dataset. `awesome-align` pre-trained on general-domain data is preferred in French, and the same model with further fine-tuning on biomedical data is selected for German.

<table><caption>Table 5: Average Error Rate (AER) for various aligners.</caption><thead><tr><th>model</th><th>fr</th><th>de</th></tr></thead><tbody><tr><td>FastAlign</td><td>10.5</td><td>27.0</td></tr><tr><td>AWESOME from scratch</td><td>5.6</td><td>17.4</td></tr><tr><td>+ ft on clinical</td><td>4.7</td><td>15.4</td></tr><tr><td>AWESOME pre-trained</td><td><strong>4.1</strong></td><td>15.2</td></tr><tr><td>+ ft on clinical</td><td>4.8</td><td><strong>15.0</strong></td></tr></tbody></table>

Table 6 summarizes the choices of the best translation and alignment methods.

<table><caption>Table 6: Pre-selected translation and alignment models.</caption><thead><tr><th>lang</th><th>translation</th><th>alignment</th></tr></thead><tbody><tr><td>fr</td><td>Opus ft</td><td>AWESOME</td></tr><tr><td>de</td><td>FAIR ft</td><td>AWESOME pt+ft</td></tr></tbody></table>

# 6 Results

Having selected the best translation and alignment model for each language based on intrinsic evaluation, `translate-train` and `translate-test` approaches can be compared to Cross-lingual Transfer (CLT). The impact of the translation and alignment models can also be analyzed, as well as using the translation fine-tuning data to improve directly the models used in CLT. Since `translate-train` can leverage monolingual domain-specific model, we evaluate the outcome of such a strategy. Five different random seeds were used for each model and results presented in this section show the average performance along with the standard deviation (more implementation details in Appendix B).

## 6.1 Comparison of the different methods

For a fair comparison between the three methods detailed in Section 3, we use the pre-selected translation and alignment models of Table 6 for the `translate-train` and `translate-test` methods. We report the F1-scores of the different methods in Table 7. The translation and alignment models providing the best test scores are also
<table><caption>Table 7: Comparing the three methods with pre-selected translation and alignment models (select.). Best performing pairs are provided for comparison and are underlined when better than CLT.</caption><thead><tr><th>method</th><th>fr</th><th>de</th></tr></thead><tbody><tr><td colspan="3">distilmBERT</td></tr><tr><td>CLT</td><td>65.9<sub>±3.3</sub></td><td>64.6<sub>±2.4</sub></td></tr><tr><td>translate-train select.</td><td>66.5<sub>±1.9</sub></td><td><b>68.3</b><sub>±1.3</sub></td></tr><tr><td>translate-test select.</td><td><b>69.2</b><sub>±1.4</sub></td><td><b>68.3</b><sub>±1.8</sub></td></tr><tr><td>translate-train best</td><td><u>69.2</u><sub>±1.2</sub></td><td><u>69.2</u><sub>±1.2</sub></td></tr><tr><td>translate-test best</td><td><u>69.7</u><sub>±1.5</sub></td><td><u>68.3</u><sub>±1.8</sub></td></tr><tr><td colspan="3">XLM-R Base</td></tr><tr><td>CLT</td><td><b>79.1</b><sub>±0.8</sub></td><td>72.2<sub>±0.7</sub></td></tr><tr><td>translate-train select.</td><td>74.6<sub>±0.9</sub></td><td><b>73.7</b><sub>±0.9</sub></td></tr><tr><td>translate-test select.</td><td>74.2<sub>±1.6</sub></td><td>72.7<sub>±0.8</sub></td></tr><tr><td>translate-train best</td><td>78.6<sub>±0.5</sub></td><td><u>74.8</u><sub>±1.0</sub></td></tr><tr><td>translate-test best</td><td>74.4<sub>±1.3</sub></td><td>72.7<sub>±0.8</sub></td></tr><tr><td colspan="3">XLM-R Large</td></tr><tr><td>CLT</td><td><b>77.9</b><sub>±1.7</sub></td><td><b>78.5</b><sub>±0.4</sub></td></tr><tr><td>translate-train select.</td><td>76.5<sub>±0.7</sub></td><td>77.4<sub>±1.3</sub></td></tr><tr><td>translate-test select.</td><td>75.3<sub>±0.9</sub></td><td>76.1<sub>±2.8</sub></td></tr><tr><td>translate-train best</td><td><u>78.0</u><sub>±0.5</sub></td><td><u>79.4</u><sub>±1.3</sub></td></tr><tr><td>translate-test best</td><td>75.3<sub>±0.9</sub></td><td>76.1<sub>±2.8</sub></td></tr></tbody></table>

provided for comparison, revealing what can be missed with the pre-selection.

CLT with a sufficiently large MLLM provides the best results. When compared with translation-based methods with pre-selected translation and alignment models, CLT with XLM-R models gives higher scores, except for XLM-R Base in German.

On the other hand, it seems that using an English NER on a translated version of the test set provides the best results with a small model like distilmBERT. DistilmBERT might be better as a monolingual model than a multilingual one. In the same vein, XLM-R Base struggles in German, while its large version does not. Small language models underperform in CLT and their generalization ability is not sufficient compared to translation-based methods. A first take-away is consequently that translation-based methods should be favored with small language models.

The translate-test method is consistently outperformed by translate-train for large models. Even using a specific biomedical model like (Gu et al., 2021) for translate-test does not improve the results (results in Appendix F). Indeed translation and alignment errors only harm the training set of translate-train, which does

not prevent a large model from generalizing despite some errors in the training data, while errors of translation or alignment in translate-test are directly reflected in the test score. In the rest of this analysis we will consequently compare CLT only with translate-train.

Providing a large-enough MLLM, CLT outperforms pre-selected translate-train and translate-test. However, choosing the translation and the alignment model beforehand does not lead to the best results. To the exception of XLM-R Base in French, there always exists a pair of translation and alignment models that leads to a better score for the translate-train method over CLT. This agrees with Yarmohammadi et al. (2021) and encourages practitioners to explore different methods to perform cross-lingual adaptation.

## 6.2 Influence of the translation model

The choice of the translation and alignment models can have an important impact on the final NER performances as shown on Table 7. This section studies their impact in details. A German NER model was trained using the Opus model instead of the FAIR model for translating the training set into German. Using a worse model (see Table 3) for translation leads to lower NER scores as shown in Table 8: the NER model based on the FAIR translation beats by more than 2 points the one using the Opus translation, whatever aligner is used.

While choosing between different base translation models (like Opus or FAIR) based on their translation scores on in-domain data seems to provide the best results, deciding between the fine-tuned version of a translation model and the base one by comparing the BLEU or COMET scores on biomedical data does not guarantee the best downstream F1 score as Table 9 shows. The translation model was fine-tuned on biomedical data, which improved intrinsic results on the BioWMT19 translation dataset. But this dataset belongs to a specific biomedical sub-domain (PubMed abstracts), and fine-tuning might not improve translation for the clinical sub-domain of the NER dataset.

The takeaway is that, while a small gain in translation accuracy (obtained with further fine-tuning) might not necessarily improve the result of the translate-train approach, a completely different model (like FAIR with respect to Opus) has more chance to improve cross-lingual adaptation.
<table><caption>Table 8: translate-train in German with XLM-R Base using either fine-tuned or base Opus model.</caption><thead><tr><th>aligner</th><th>Opus f1</th><th>FAIR f1</th></tr></thead><tbody><tr><td>FastAlign</td><td>70.9<sub>±1.8</sub></td><td><b>72.8</b><sub>±1.6</sub></td></tr><tr><td>AWESOME</td><td>72.2<sub>±1.7</sub></td><td><b>73.1</b><sub>±1.3</sub></td></tr><tr><td>AWESOME ft</td><td>71.1<sub>±1.2</sub></td><td><b>74.1</b><sub>±1.1</sub></td></tr><tr><td>AWESOME pt+ft</td><td>71.2<sub>±1.1</sub></td><td><b>74.1</b><sub>±1.3</sub></td></tr></tbody></table>

<table><caption>Table 9: translate-train in French with XLM-R Base using either fine-tuned or base Opus model.</caption><thead><tr><th>aligner</th><th>base</th><th>fine-tuned</th></tr></thead><tbody><tr><td>FastAlign</td><td>78.2<sub>±0.8</sub></td><td><b>78.6</b><sub>±0.5</sub></td></tr><tr><td>AWESOME</td><td><b>76.4</b><sub>±2.0</sub></td><td>74.2<sub>±0.9</sub></td></tr><tr><td>AWESOME ft</td><td><b>74.6</b><sub>±1.6</sub></td><td>74.5<sub>±1.6</sub></td></tr><tr><td>AWESOME pt+ft</td><td>75.8<sub>±1.7</sub></td><td><b>76.3</b><sub>±1.0</sub></td></tr></tbody></table>

## 6.3 Influence of the alignment model

While choosing a translation model based solely on intrinsic performance should not harm downstream cross-lingual adaptation performances, the choice of the alignment model seems more tricky. Based on intrinsic performances like Error Rate on annotated alignment (Table 5), *awesome-align* seems to be the right aligner for the task. However, while it provides better downstream results than *fastAlign* in German (Table 8), it does not hold for French (Table 9).

Table 10 shows that using different aligners leads to different levels of accuracy according to the types of entity we want to retrieve. While the global F1 score suggests that *fastalign* is better suited for the cross-lingual adaptation, looking at the detailed results for each entity type shows that the gap is mainly due to the *FREQUENCY* class on which *awesome-align* performs poorly. But this is not the case on other classes.

*FREQUENCY* entities are usually more verbose than drugs or dosages. Table 11 shows that *fastalign* make obvious errors like aligning "240 mg" to "in morning and night", but

<table><caption>Table 10: Comparison of *fastalign* and *awesome-align* (pre-trained only) for three different entity types (F1-score), for *translate-train* with XLM-R Base on MedNERF with Opus fine-tuned. (Full results in Appendix F).</caption><thead><tr><th>aligner</th><th>Freq.</th><th>Strength</th><th>Drug</th><th>f1</th></tr></thead><tbody><tr><td>FastAlign</td><td><b>72.0</b></td><td>89.7</td><td><b>83.2</b></td><td><b>78.6</b></td></tr><tr><td>AWESOME</td><td>50.2</td><td><b>92.5</b></td><td>82.2</td><td>74.2</td></tr></tbody></table>

<table><caption>Table 11: Examples of frequencies transformed with translation and alignment. Bold indicates the right annotation and // indicates that the entity has been split.</caption><thead><tr><th>original</th><th>FastAlign</th><th>AWESOME</th></tr></thead><tbody><tr><td>in morning and night</td><td>le matin et la nuit et 240 mg</td><td><b>le matin et la nuit</b></td></tr><tr><td>daily</td><td><b>par jour</b></td><td>jour</td></tr><tr><td>once a day</td><td><b>une fois par jour</b></td><td>fois par jour</td></tr><tr><td>at bedtime</td><td><b>au moment du coucher</b></td><td>au // coucher</td></tr></tbody></table>

*awesome-align* can miss the preposition when aligning "daily" with "jour" instead of "par jour", leading eventually to a consequent score drop.

The choice of the alignment model must thus be made more carefully than the translation one. Intrinsic performances of alignment models are not sufficient information. Some additional post-processing might be needed, as in Yarmoham-madi et al. (2021), where *awesome-align* gives better results, but entities that are split by the aligner like "au moment du coucher" in Table 11 are merged by including all words in between. This would work in that particular case, but could cause problems in others, particularly for languages where the word order is different.

## 6.4 Using parallel data to realign models

With the right translation and alignment model, it seems that CLT can be outperformed by the *translate-train* method. However the latter relies on additional resources: a translation and an alignment models, trained on parallel data. This parallel data could also be used to re-align the representations of the multilingual models used in CLT.

To improve a multilingual language model with parallel data, it is trained for a contrastive alignment objective following Wu and Dredze (2020). Words aligned with *awesome-align* are trained to have more similar representations than random in-batch pairs of words (details in Appendix E). After this realignment step, CLT can be applied.

Results in Table 12 show that while realignment does not systematically provide improvement over CLT as observed by Wu and Dredze (2020), it does significantly boost results in some cases, allowing to outperform the best *translate-train* baseline in German for XLM-R Base and in French for XLM-R Large. This, yet again, encourages practitioners to explore different methods, including realignment to perform cross-lingual adaptation.
<table><caption>Table 12: F1 scores for CLT from scratch and CLT with realignment. Best F1-score in bold. Results underlined show improvement of realignment over CLT.</caption><thead><tr><th>model</th><th>fr</th><th>de</th></tr></thead><tbody><tr><td>distilmBERT</td><td>65.9<sub>±3.3</sub></td><td>64.6<sub>±2.4</sub></td></tr><tr><td>+ realign</td><td><u>66.4</u><sub>±1.4</sub></td><td><u>67.9</u><sub>±1.5</sub></td></tr><tr><td>translate-train best</td><td><b>69.2</b><sub>±1.2</sub></td><td><b>69.2</b><sub>±1.2</sub></td></tr><tr><td>XLM-R Base</td><td><b>79.1</b><sub>±0.8</sub></td><td>72.2<sub>±0.7</sub></td></tr><tr><td>+ realign</td><td>76.7<sub>±0.7</sub></td><td><b>75.8</b><sub>±1.3</sub></td></tr><tr><td>translate-train best</td><td>78.6<sub>±0.5</sub></td><td>74.8<sub>±1.0</sub></td></tr><tr><td>XLM-R Large</td><td>77.9<sub>±1.7</sub></td><td>78.5<sub>±0.4</sub></td></tr><tr><td>+ realign</td><td><b>78.8</b><sub>±1.6</sub></td><td>78.3<sub>±1.6</sub></td></tr><tr><td>translate-train best</td><td>78.0<sub>±0.5</sub></td><td><b>79.4</b><sub>±1.3</sub></td></tr></tbody></table>

## 6.5 Using domain-specific language models

We evaluate now the relevance of using language-specific models like CamemBERT (Martin et al., 2020) or GottBERT (Scheible et al., 2020) on the translated version of the training dataset or language and domain-specific models like DrBERT (Labrak et al., 2023) or medBERT.de (Bressem et al., 2023) which are BERT models fine-tuned on medical corpora in respectively French and German. We report in Table 13 and 14 the results of the translate-train method for the best translation/alignment algorithms pair and for the pre-selected one, compared to using XLM-R Base.

<table><caption>Table 13: Comparison of domain and language specific models for translate-train in French.</caption><thead><tr><th>model</th><th>pre-selected</th><th>best</th></tr></thead><tbody><tr><td>CamemBERT Base</td><td>73.5<sub>±1.5</sub></td><td>76.7<sub>±0.9</sub></td></tr><tr><td>DrBERT 7GB</td><td>70.7<sub>±1.3</sub></td><td>73.5<sub>±1.4</sub></td></tr><tr><td>DrBERT Pubmed</td><td><b>76.1</b><sub>±1.3</sub></td><td><b>78.8</b><sub>±1.4</sub></td></tr><tr><td>XLM-R Base</td><td>74.6<sub>±1.9</sub></td><td>78.6<sub>±0.5</sub></td></tr></tbody></table>

<table><caption>Table 14: Comparison of domain and language specific models for translate-train in German.</caption><thead><tr><th>model</th><th>pre-selected</th><th>best</th></tr></thead><tbody><tr><td>GottBERT</td><td><b>75.5</b><sub>±1.4</sub></td><td><b>76.6</b><sub>±0.8</sub></td></tr><tr><td>medBERT</td><td>72.7<sub>±0.5</sub></td><td>75.0<sub>±1.6</sub></td></tr><tr><td>XLM-R Base</td><td>73.7<sub>±0.9</sub></td><td>74.8<sub>±1.0</sub></td></tr></tbody></table>

The translate-train approach allows to rely on models that are specific to the language and domain of the target evaluation. However, Table 13 and 14 show that their use does not always bring significant improvement over XLM-R Base. The

performances of these models can be explained by the quantity of training data used. XLM-R models are indeed trained on 2.5 TB data while DrBERT and medBERT.de use less than 10GB data, which can explain their low score. Besides, the language-specific models CamemBERT and GottBERT are trained with more data (138 GB and 145 GB) and achieve better performances, even beating XLM-R in German. Finally, it must be noted that the best translate-train model in French, DrBERT Pubmed, is actually pre-trained on the English PubMed dataset and then on French clinical texts, which suggests that multilingual models should be preferred, even with a translation-based cross-lingual adaptation.

## 6.6 Computing times

To conclude the analysis of the different cross-lingual adaptation methods studied in this paper we finally compare their computing times. Table 15 gathers the training and inference times of the three methods using the XLM-R base model and the awesome-align alignment model in French.

<table><caption>Table 15: Training and inference times for the different methods, with the XLM-R Base model in French, on a single GPU.</caption><thead><tr><th>method</th><th>training time (total)</th><th>inference time (per sample)</th></tr></thead><tbody><tr><td>CLT</td><td>1.2h</td><td>0.04s</td></tr><tr><td>translate-train</td><td>2.7h</td><td>0.04s</td></tr><tr><td>translate-test</td><td>1.2h</td><td>0.32s</td></tr></tbody></table>

This comparison shows a longer training time for the translate-train method, which is due to the translation of the whole training set before the training of the NER model. On a single GPU with 8GB of RAM this translation step is even longer than the NER training. However, once training is done the translate-train method has the same inference time as the CLT method, while the translate-test method now suffers from the need of translation at inference time.

# 7 Conclusion

This paper shows that cross-lingual transfer with general-domain MLLMs is efficient for a domain-specific task like clinical NER, giving comparable results with translating the training set. But CLT has the advantage of working off-the-shelf, while translation-based methods require choosing
the translation and alignment models carefully. Selecting these models based on intrinsic domain-specific values, like fine-tuning scores on clinical parallel data, or using a domain-specific language model does not provide significantly better downstream results in the target language. The selection of the alignment model was shown to be particularly crucial, and results of *translate-train* could probably be improved by post-processing the alignment. CLT also has a margin of progression as realigning the representations of MLLMs can increase the results dramatically in some cases.

It is also worth noting that training on translated data provide better results than translating at inference time. The *translate-test* approach should then be used only when large multilingual models cannot be used. While training on translated data allows to leverage domain-specific monolingual language models, those latter models can give better results over multilingual models like XLM-R only if pre-trained with sufficient data.

Pre-training a MLLM with only clinical data is a good lead for further improvements in clinical cross-lingual transfer. While the results show that using a domain-specific monolingual model in *translate-train* or *translate-test* is not on par with general-purpose multilingual models, they also show that the French clinical model DrBERT provides the best results for *translate-train* when it uses the English biomedical model PubmedBERT as initialization.

We finally advocate for the release of more non-English clinical datasets annotated with similar guidelines as English (or other) ones. Even a relatively small dataset like MedNERF or the GERNERMED test set are crucial to evaluate cross-lingual adaptation in the clinical domain.

## Limitations

This paper is limited to the study of clinical NER models using an encoder-only architecture. The use of generative models with a zero-shot learning approach (Hu et al., 2023) is another promising approach for low-resource languages that could be compared with CLT and translation-based approaches in a future work. However such methods require a careful prompt selection strategy and cannot be directly compared to supervised models.

This paper is also limited to cross-lingual transfer to French and German. Ideally, this work could have included experiments with other target languages

and also other source languages than English, as Yarmohammadi et al. (2021) do in their general-domain comparison of strategies for cross-lingual transfer. However evaluation datasets are lacking for that purpose in the clinical domain. Similarly, more general conclusions about cross-lingual adaptation methods in the clinical domains could be drawn with further studies on various clinical NLP tasks such as relation extraction or sentence classification. However, the lack of evaluation datasets in the clinical domain prevented us from extending the experiments to such other clinical NLP tasks. Finally, the authors assume that the findings will be task-specific and encourage practitioners to explore all methods when facing a new NLP task.

The authors also want to point out that MedNERF is drawn from drug prescriptions while the n2c2 and GERNERMED datasets use clinical reports. This domain difference could have made the cross-lingual generalization more challenging, but in practice we found that the different models used were not really affected by the possible domain-shift, showing similar French and German F1 scores. Moreover, when comparing randomly sampled examples from all three datasets, we do not find any critical differences (see Appendix G). Sentences drawn from MedNERF are shorter and less written, but they contain similar annotated entities as the n2c2 sentences, and the n2c2 dataset also contains some short examples that resemble the MedNERF ones.

## Acknowledgments

We would like to thank the anonymous reviewers for their comments which helped enrich the discussion around the results.

Some experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations<fnref n="6" />.

## References

Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, Antonio Jimeno Yepes, Madeleine Kittner, Martin Krallinger, Nancy Mah, Aurelie Neveol, Maria Neves, Felipe Soares, Amy Siu, Karin Verspoor, and Maika Vicente Navarro. 2019. Findings of the WMT 2019 Biomedical Translation Shared Task: Evaluation for MEDLINE Abstracts and Biomedical

<footnote n="6">⁶see https://www.grid5000.fr.</footnote>
Terminologies. In Proceedings of the Fourth Con-ference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 29–53, Florence, Italy. Association for Computational Linguistics.

Florian Borchert, Christina Lohr, Luise Modersohn, Jonas Witt, Thomas Langer, Markus Follmann, Matthias Gietzelt, Bert Arnrich, Udo Hahn, and Matthieu-P. Schapranow. 2022. GGPONC 2.0 - the German clinical guideline corpus for oncology: Cur-ration workflow, annotation policy, baseline NER taggers. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3650–3660, Marseille, France. European Language ResOURCES ASSOCIATION.

Keno K Bressem, Jens-Michalis Papaioannou, Paul Grundmann, Florian Borchert, Lisa C Adams, Leon-hard Liu, Felix Busch, Lina Xu, Jan P Loyen, Ste-fan M Niehues, et al. 2023. Medbert. de: A compre-hensive german bert model for the medical domain. arXiv preprint arXiv:2303.08179.

Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka-siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Conference of the European Chapter of the Association for Computational Linguistics (EACL).

Ondřej Dušek, Jan Hajič, Jaroslava Hlaváčová, Jindřich Libovický, Pavel Pecina, Aleš Tamchyna, and Zdeňka Urešová. 2017. Khresmoi summary translation test data 2.0. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.

Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648.

Johann Frei, Ludwig Frei-Stuber, and Frank Kramer. 2022. GERNERMED++: Transfer Learning in German Medical NLP.

Johann Frei and Frank Kramer. 2021. GERNERMED – An Open German Medical NER Model.

Yingbo Gao, Christian Herold, Zijian Yang, and Hermann Ney. 2022. Is encoder-decoder redundant for neural machine translation? In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 562–574, Online only. Association for Computational Linguistics.

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. 3(1).

Sam Henry, Kevin Buchan, Michele Filannino, Amber Stubbs, and Ozlem Uzuner. 2019. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. Journal of the American Medical Informatics Association, 27(1):3–12.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org.

Yan Hu, Iqra Ameer, Xu Zuo, Xueqing Peng, Yujia Zhou, Zehan Li, Yiming Li, Jianfu Li, Xiaoqian Jiang, and Hua Xu. 2023. Zero-shot clinical entity recognition using chatgpt. arXiv preprint arXiv:2303.16416.

Rezarta Islamaj Doğan and Zhiyong Lu. 2012. An improved corpus of disease mentions in PubMed citations. In BioNLP: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing, pages 91–99, Montréal, Canada. Association for Computational Linguistics.

Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings. In Find-ings of the Association for Computational Linguistics: EMNLP 2020, pages 1627–1643, Online. Association for Computational Linguistics.

Muhammad Raza Khan, Morteza Ziyadi, and Mohamed AbdelHady. 2020. Mt-bioner: Multi-task
learning for biomedical named entity recognition us-
ing deep bidirectional transformers. *arXiv preprint*
*arXiv:2001.08904*.

Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton
Dvorkovich, Christian Federmann, Mark Fishel,
Thamme Gowda, Yvette Graham, Roman Grund-
kiewicz, Barry Haddow, Rebecca Knowles, Philipp
Koehn, Christof Monz, Makoto Morishita, Masaaki
Nagata, Toshiaki Nakazawa, Michal Novák, Martin
Popel, and Maja Popović. 2022. Findings of the 2022
conference on machine translation (WMT22). In
*Proceedings of the Seventh Conference on Machine
Translation (WMT)*, pages 1–45, Abu Dhabi, United
Arab Emirates (Hybrid). Association for Computa-
tional Linguistics.

Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. 2023. Drbert: A robust pre-trained model in french for biomedical and clinical domains. *bioRxiv*, pages 2023-04.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. *arXiv preprint arXiv:1907.11692*.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In *International Conference on Learning Representations*.

Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7203–7219, Online. Association for Computational Linguistics.

Aurélie Névéol, Cyril Grouin, Jérémy Leixa, Sophie Rosset, and Pierre Zweigenbaum. 2014. The quaero french medical corpus : A ressource for medical entity recognition and normalization.

Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2020. Facebook FAIR's WMT19 News Translation Task Submission. In *Proc. of WMT*.

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-
Jing Zhu. 2002. Bleu: a method for automatic evalu-
ation of machine translation. In *Proceedings of the
40th annual meeting of the Association for Computa-
tional Linguistics*, pages 311–318.

Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon
Lavie. 2020. COMET: A Neural Framework for MT
Evaluation. In *Proceedings of the 2020 Conference
on Empirical Methods in Natural Language Processing (EMNLP)*, pages 2685–2702, Online. Association
for Computational Linguistics.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. *CoRR*, abs/1910.01108.

Raphael Scheible, Fabian Thomczyk, Patric Tippmann, Victor Jaravine, and Martin Boeker. 2020. Gotbert: a pure german language model. *arXiv preprint arXiv:2012.02110*.

Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In *Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT)*, Lisbon, Portugal.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 38–45, Online. Association for Computational Linguistics.

Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 833–844, Hong Kong, China. Association for Computational Linguistics.

Shijie Wu and Mark Dredze. 2020. Do explicit alignments robustly improve multilingual encoders? In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 4471–4482, Online. Association for Computational Linguistics.
Xi Yang, Jiang Bian, William R Hogan, and Yonghui Wu. 2020. Clinical concept extraction using transformers. *Journal of the American Medical Informatics Association*. Ocaa189.

Mahsa Yarmohammadi, Shijie Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, and Benjamin Van Durme. 2021. Everything is all it takes: A multi-pronged strategy for zero-shot cross-lingual information extraction. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 1950–1967, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

David Yarowsky and Grace Ngai. 2001. Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In *Second Meeting of the North American Chapter of the Association for Computational Linguistics*.

David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In *Proceedings of the First International Conference on Human Language Technology Research*.
## A Pre-selection of the translation model for translate-test

The translate-test method needs translation algorithms from German and French to English. Similarly to Section 5.1 we fine-tuned the Opus-MT and FAIR algorithms on the same medical datasets and obtained the COMET and BLEU scores presented in Tables 16 and 17. These scores are used to select the best translation model for the translate-test approach and they lead to the same model choices as for the translate-train method.

<table><caption>Table 16: Evaluation of the translation models from German to English. Best model bold and second underlined. ft for finetuned.</caption><thead><tr><th rowspan="2">model</th><th colspan="2">BioWMT19</th><th colspan="2">Khresmoi</th></tr><tr><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th></tr></thead><tbody><tr><td>FAIR</td><td><u>38.2</u></td><td>0.538</td><td><strong>47.1</strong></td><td><strong>0.764</strong></td></tr><tr><td>+ ft</td><td><strong>38.5</strong></td><td><strong>0.675</strong></td><td><u>46.8</u></td><td><strong>0.764</strong></td></tr><tr><td>Opus</td><td>35.3</td><td>0.587</td><td>43.6</td><td>0.723</td></tr><tr><td>+ ft</td><td>38.1</td><td><u>0.640</u></td><td>44.3</td><td>0.729</td></tr></tbody></table>

<table><caption>Table 17: Evaluation of the translation models from French to English. Best model bold. ft for finetuned.</caption><thead><tr><th rowspan="2">model</th><th colspan="2">BioWMT19</th><th colspan="2">Khresmoi</th></tr><tr><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th></tr></thead><tbody><tr><td>Opus</td><td>33.9</td><td>0.721</td><td><strong>48.3</strong></td><td>0.798</td></tr><tr><td>+ ft</td><td><strong>36.3</strong></td><td><strong>0.749</strong></td><td>48.0</td><td><strong>0.799</strong></td></tr></tbody></table>

## B Training and implementation details

All models were written in Pytorch using the Huggingface libraries (Wolf et al., 2020) and were fine-tuned using the AdamW optimizer (Loshchilov and Hutter, 2019) and a learning rate of $6 \cdot 10^{-6}$ with linear decay. We used 4 epochs for the translation models and 8 epochs for the NER models. The NER models were trained on a single GPU (Nvidia GeForce RTX 2070 with 8GB of RAM) for approximately one hour.

## C Alignment methods used

`fastAlign` was applied asymmetrically, by mapping words from source language (English) to target language. Although it might increase alignment score, symmetrization was not used, because it might remove important links during the labels projection step.

`awesome-align` was used with softmax (instead of the alternative $\alpha$-entmax function) and without the optional consistency optimization objective. For completeness we added the results with the consistency optimization objective (w/ co lines) in the tables of Appendix F and we observed that they did not improve the NER scores. The base model used is mBERT as in the original paper (Dou and Neubig, 2021). The pre-trained version of `awesome-align` used is the one provided by the authors, fine-tuned on general-domain parallel data. Throughout the paper, in tables, "AWESOME" designates this latter pre-trained version. "AWESOME ft" is `awesome-align` with the raw mBERT model fine-tuned on the clinical parallel data only and "AWESOME pt+ft" is the pre-trained model, fine-tuned again on the clinical parallel data.

## D Transformer-base models used

We report in Table 18 the number of parameters and the quantity of training data of the different large language models used in this paper.

<table><caption>Table 18: Size of the different base models.</caption><thead><tr><th>model</th><th>params (M)</th><th>emb. (M)</th><th>train (GB)</th></tr></thead><tbody><tr><td colspan="4">Multilingual models</td></tr><tr><td>distilmBERT</td><td>135</td><td>92</td><td>42</td></tr><tr><td>XLM-R Base</td><td>278</td><td>192</td><td>2.5k</td></tr><tr><td>XLM-R Large</td><td>560</td><td>256</td><td>2.5k</td></tr><tr><td colspan="4">Language-specific models</td></tr><tr><td>CamemBERT (fr)</td><td>111</td><td>25</td><td>138</td></tr><tr><td>GottBERT (de)</td><td>126</td><td>40</td><td>145</td></tr><tr><td colspan="4">Clinical models</td></tr><tr><td>medBERT (de)</td><td>109</td><td>23</td><td>10</td></tr><tr><td>DrBERT 7GB (fr)</td><td>111</td><td>25</td><td>7.4</td></tr><tr><td>DrBERT PubMed (fr)</td><td>109</td><td>23</td><td>28</td></tr></tbody></table>

Although `distilmBERT` has more parameters than `CamemBERT`, it must be noted that it has also more words in its vocabulary, due to its multilingual nature. Hence most of its parameters are embeddings weights that are not necessarily used in our experiments as they might be embeddings of words from other languages. So in our setting, `distilmBERT` can be considered a smaller model than `CamemBERT` and `GottBERT` despite the higher number of parameters.
## E Realignment method

The reader might refer to Wu and Dredze (2020) for the realignment method itself. The representations of the last layer of the pre-trained model to be realigned were used in a contrastive loss where pairs of words aligned with *awesome-align* are encouraged to be more similar than the other possible pairs of words in the batch. The strong alignment objective was used, meaning that pair of same-language words were also used as negative examples for the contrastive loss. The version of *awesome-align* was the one pre-trained on general-domain data, released by Dou and Neubig (2021), with softmax and without the optional consistency optimization objective, the same used by the authors of the realignment method used.

The parallel data used for realignment was the same as for fine-tuning the translation and alignment models (Section 5.1). The two datasets (English-German and English-French) were used together to realign a given model, which can then be used either for generalization to French or German. For each base model, five realigned models were obtained for the five random seeds, each of them used in the corresponding fine-tuning by seed.

The realignment was done for 20,000 steps of batches of size 16, with Adam optimizer, a learning rate of $2 \times 10^{-4}$, and with linear warm-up for 10% of the total steps. This means that the whole dataset was repeated approximately 1.25 times.

## F Additional results

Detailed results are shown in the following tables:

* Summary of results for cross-lingual adaptation, with pre-selected and best pairs of translation and alignment models: Table 19 for French and 20 for German;

* CLT and translate-train with multilingual models: Table 21 (fr) and 22 (de);

* translate-train with language- and domain-specific models: Table 23 and 24;

* translate-test with multilingual language models: Table 25 and 26;

* translate-test with PubmedBERT: Table 27 and 28;

* Breakdown of the results class-by-class in French for multilingual models: Table 29.

<table><caption>Table 19: Summary of results for cross-lingual adaptation to French.</caption><thead><tr><th>model</th><th>pre-selected</th><th>best</th></tr></thead><tbody><tr><td colspan="3">translate-train</td></tr><tr><td>distilmBERT</td><td>66.5±1.9</td><td>69.2±1.2</td></tr><tr><td>XLM-R Base</td><td>74.2±0.9</td><td>78.6±0.5</td></tr><tr><td>XLM-R Large</td><td>76.5±0.7</td><td>78.0±0.5</td></tr><tr><td>CamemBERT</td><td>73.5±1.5</td><td>76.7±0.9</td></tr><tr><td>DrBERT</td><td>70.7±1.3</td><td>73.5±1.4</td></tr><tr><td>DrBERT Pubmed</td><td>76.1±1.3</td><td>78.8±1.4</td></tr><tr><td colspan="3">translate-test</td></tr><tr><td>distilmBERT</td><td>69.2±1.4</td><td>69.7±1.5</td></tr><tr><td>XLM-R Base</td><td>74.2±1.6</td><td>74.4±1.3</td></tr><tr><td>XLM-R Large</td><td>75.3±0.9</td><td>75.3±0.9</td></tr><tr><td>PubmedBERT</td><td>73.3±1.3</td><td>73.5±1.2</td></tr><tr><td colspan="3">CTL*</td></tr><tr><td>distilmBERT</td><td>65.9±3.3</td><td>65.9±3.3</td></tr><tr><td>+ realigned</td><td>66.4±1.4</td><td>66.4±1.4</td></tr><tr><td>XLM-R Base</td><td><b>79.1</b>±0.8</td><td><b>79.1</b>±0.8</td></tr><tr><td>+ realigned</td><td>76.7±0.7</td><td>76.7±0.7</td></tr><tr><td>XLM-R Large</td><td>77.9±1.7</td><td>77.9±1.7</td></tr><tr><td>+ realigned</td><td>78.8±1.6</td><td>78.8±1.6</td></tr></tbody></table>

<table><caption>Table 20: Summary of results for cross-lingual adaptation to German.</caption><thead><tr><th>model</th><th>pre-selected</th><th>best</th></tr></thead><tbody><tr><td colspan="3">translate-train</td></tr><tr><td>distilmBERT</td><td>68.3±1.3</td><td>69.2±1.2</td></tr><tr><td>XLM-R Base</td><td>73.7±0.9</td><td>74.8±1.0</td></tr><tr><td>XLM-R Large</td><td>77.4±1.3</td><td><b>79.4</b>±1.3</td></tr><tr><td>GottBERT</td><td>75.5±1.4</td><td>76.6±0.8</td></tr><tr><td>MedBERT.de</td><td>72.7±0.5</td><td>75.0±1.6</td></tr><tr><td colspan="3">translate-test</td></tr><tr><td>distilmBERT</td><td>68.3±1.8</td><td>68.3±1.8</td></tr><tr><td>XLM-R Base</td><td>72.7±0.8</td><td>72.7±0.8</td></tr><tr><td>XLM-R Large</td><td>76.1±2.8</td><td>76.1±2.8</td></tr><tr><td>PubmedBERT</td><td>72.6±1.5</td><td>73.3±1.7</td></tr><tr><td colspan="3">CTL*</td></tr><tr><td>distilmBERT</td><td>64.6±2.4</td><td>64.6±2.4</td></tr><tr><td>+ realigned</td><td>67.9±1.5</td><td>67.9±1.5</td></tr><tr><td>XLM-R Base</td><td>72.2±0.7</td><td>72.2±0.7</td></tr><tr><td>+ realigned</td><td>75.8±1.3</td><td>75.8±1.3</td></tr><tr><td>XLM-R Large</td><td><b>78.5</b>±0.4</td><td>78.5±0.4</td></tr><tr><td>+ realigned</td><td>78.3±1.6</td><td>78.3±1.6</td></tr></tbody></table>

<footnote>* results are reported twice as there is no pre-selection process</footnote>

<footnote>* results reported twice as there is no pre-selection process</footnote>
<table><caption>Table 21: Cross-lingual Transfer and translate-train results in French for multilingual base models.</caption><thead><tr><th>translation</th><th>aligner</th><th>precision</th><th>recall</th><th>micro-f1</th><th>macro-f1</th></tr></thead><tbody><tr><td colspan="6" style="text-align:center; font-weight:bold;">distilmBERT</td></tr><tr><td>Opus</td><td>FastAlign</td><td>67.8±2.1</td><td>68.7±1.5</td><td>68.3±1.6</td><td>70.7±1.4</td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>67.1±1.0</td><td>68.7±1.3</td><td>67.9±0.3</td><td>70.4±0.3</td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td><strong>68.2</strong>±1.6</td><td>70.1±1.1</td><td><strong>69.2</strong>±1.2</td><td><strong>71.7</strong>±1.1</td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>65.0±0.8</td><td>67.4±1.8</td><td>66.2±1.0</td><td>68.8±0.8</td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>65.2±1.2</td><td>68.4±1.8</td><td>66.8±1.3</td><td>69.4±1.1</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>66.5±1.3</td><td>69.1±1.2</td><td>67.8±1.0</td><td>70.2±0.9</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>66.6±1.2</td><td><strong>70.2</strong>±1.8</td><td>68.3±1.1</td><td>70.8±1.1</td></tr><tr><td>Opus ft</td><td>FastAlign</td><td>66.0±1.1</td><td>69.5±0.4</td><td>67.7±0.5</td><td>70.0±0.5</td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td>65.2±1.9</td><td>67.8±1.9</td><td>66.5±1.9</td><td>69.1±1.7</td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>65.7±1.2</td><td>68.3±1.9</td><td>66.9±1.4</td><td>69.9±1.2</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>64.6±0.8</td><td>67.4±1.0</td><td>65.9±0.4</td><td>68.6±0.5</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>64.9±0.7</td><td>68.8±1.5</td><td>66.8±1.1</td><td>69.4±1.0</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>64.5±1.1</td><td>68.1±1.4</td><td>66.2±1.0</td><td>69.0±1.0</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>63.5±1.0</td><td>68.3±1.1</td><td>65.8±0.9</td><td>68.7±1.0</td></tr><tr><td colspan="2">Cross-lingual Transfer</td><td>64.9±2.5</td><td>66.8±4.1</td><td>65.9±3.3</td><td>68.2±3.2</td></tr><tr><td colspan="2">Cross-lingual Transfer with realignment</td><td>68.1±2.2</td><td>64.9±1.0</td><td>66.4±1.4</td><td>67.4±1.6</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">XLM-R Base</td></tr><tr><td>Opus</td><td>FastAlign</td><td>77.4±0.9</td><td>79.1±0.7</td><td>78.2±0.8</td><td>79.8±0.6</td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>75.4±2.4</td><td>77.3±1.6</td><td>76.4±2.0</td><td>78.6±1.6</td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td>76.0±1.0</td><td>77.6±1.6</td><td>76.8±1.2</td><td>78.8±1.2</td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>73.8±1.7</td><td>75.4±1.6</td><td>74.6±1.6</td><td>76.8±1.4</td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>75.0±0.8</td><td>76.9±0.6</td><td>76.0±0.5</td><td>78.2±0.6</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>74.8±1.5</td><td>76.8±2.0</td><td>75.8±1.7</td><td>78.0±1.7</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>75.1±1.3</td><td>76.8±1.6</td><td>76.0±1.4</td><td>78.2±1.2</td></tr><tr><td>Opus ft</td><td>FastAlign</td><td>77.9±0.2</td><td>79.4±1.0</td><td>78.6±0.5</td><td>80.3±0.3</td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td>72.9±1.4</td><td>75.6±0.8</td><td>74.2±0.9</td><td>76.7±0.7</td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>74.6±1.0</td><td>76.6±0.8</td><td>75.6±0.9</td><td>77.9±0.7</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>73.4±1.7</td><td>75.6±1.6</td><td>74.5±1.6</td><td>77.0±1.4</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>74.2±1.9</td><td>76.7±2.2</td><td>75.5±2.0</td><td>78.0±1.7</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>75.1±1.0</td><td>77.5±1.1</td><td>76.3±1.0</td><td>78.5±0.9</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>75.1±1.9</td><td>77.6±1.7</td><td>76.3±1.8</td><td>78.7±1.5</td></tr><tr><td colspan="2">Cross-lingual Transfer</td><td><strong>78.7</strong>±1.8</td><td><strong>79.6</strong>±0.5</td><td><strong>79.1</strong>±0.8</td><td><strong>80.9</strong>±0.9</td></tr><tr><td colspan="2">Cross-lingual Transfer with realignment</td><td>76.9±1.4</td><td>76.6±0.2</td><td>76.7±0.7</td><td>78.9±0.8</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">XLM-R Large</td></tr><tr><td>Opus</td><td>FastAlign</td><td>78.8±0.7</td><td>77.2±0.6</td><td>78.0±0.5</td><td>79.8±0.5</td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>76.9±1.1</td><td>76.1±1.8</td><td>76.5±1.2</td><td>78.7±1.1</td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td>76.5±1.2</td><td>75.3±1.5</td><td>75.9±1.3</td><td>78.2±1.1</td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>74.6±1.0</td><td>73.9±0.4</td><td>74.2±0.6</td><td>76.8±0.5</td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>74.9±0.3</td><td>75.7±0.8</td><td>75.3±0.3</td><td>78.1±0.5</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>75.8±1.4</td><td>75.0±1.0</td><td>75.4±1.0</td><td>78.0±0.7</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>75.7±2.2</td><td>76.2±1.3</td><td>75.9±1.7</td><td>78.3±1.8</td></tr><tr><td>Opus ft</td><td>FastAlign</td><td>76.2±2.0</td><td>76.9±2.5</td><td>76.6±2.1</td><td>78.3±2.0</td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td>76.1±1.0</td><td>76.9±0.8</td><td>76.5±0.7</td><td>78.9±0.5</td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>74.6±1.5</td><td>76.0±1.3</td><td>75.3±0.9</td><td>77.9±0.6</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>74.1±1.4</td><td>75.5±0.4</td><td>74.8±0.8</td><td>77.4±0.8</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>75.5±1.9</td><td>75.6±1.2</td><td>75.5±1.4</td><td>78.1±1.2</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>75.1±0.5</td><td>76.0±1.2</td><td>75.5±0.6</td><td>78.1±0.5</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>75.0±1.8</td><td>75.8±0.8</td><td>75.4±1.1</td><td>77.7±1.0</td></tr><tr><td colspan="2">Cross-lingual Transfer</td><td>78.2±2.5</td><td>77.6±1.2</td><td>77.9±1.7</td><td>80.0±1.4</td></tr><tr><td colspan="2">Cross-lingual Transfer with realignment</td><td><strong>79.7</strong>±1.6</td><td><strong>77.9</strong>±1.8</td><td><strong>78.8</strong>±1.6</td><td><strong>80.8</strong>±1.4</td></tr></tbody></table>
<table><caption>Table 22: Cross-lingual Transfer and translate-train results in German for multilingual base models.</caption><thead><tr><th>translation</th><th>aligner</th><th>precision</th><th>recall</th><th>micro-f1</th><th>macro-f1</th></tr></thead><tbody><tr><td colspan="6" style="text-align: center; font-weight: bold;">distilmBERT</td></tr><tr><td>FAIR</td><td>FastAlign</td><td>69.0±1.2</td><td>63.9±0.5</td><td>66.3±0.8</td><td>44.2±2.4</td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>68.1±1.6</td><td>66.1±1.3</td><td>67.1±1.3</td><td>50.5±3.0</td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td>68.0±2.0</td><td><b>67.4</b>±1.4</td><td>67.7±1.0</td><td>50.6±3.7</td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>68.5±3.6</td><td>65.0±2.2</td><td>66.6±1.2</td><td>48.3±4.0</td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td>67.6±2.4</td><td>66.1±3.5</td><td>66.7±1.4</td><td>46.4±5.1</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>69.6±2.7</td><td>65.9±2.8</td><td>67.6±1.1</td><td>46.8±6.4</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>68.8±2.6</td><td>66.6±3.7</td><td>67.5±1.4</td><td>49.2±6.0</td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>70.5±1.9</td><td>65.2±1.4</td><td>67.7±1.3</td><td><b>52.2</b>±1.5</td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td>69.3±2.0</td><td>66.9±1.6</td><td>68.0±1.1</td><td>48.0±3.6</td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>68.6±2.8</td><td>66.4±2.0</td><td>67.4±1.5</td><td>47.4±4.0</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>70.9±2.7</td><td>67.2±2.1</td><td>69.0±1.9</td><td>49.8±3.7</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td>71.4±2.0</td><td>67.2±1.8</td><td><b>69.2</b>±1.2</td><td>49.0±4.3</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td>69.3±1.8</td><td><b>67.4</b>±1.9</td><td>68.3±1.3</td><td>49.2±3.2</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>70.4±1.9</td><td>67.1±1.6</td><td>68.7±1.2</td><td>49.5±3.1</td></tr><tr><td colspan="2">Cross-lingual Transfer</td><td>62.8±2.9</td><td>66.4±2.0</td><td>64.6±2.4</td><td>46.3±3.1</td></tr><tr><td colspan="2">Cross-lingual Transfer with realignment</td><td><b>72.0</b>±2.3</td><td>64.2±1.4</td><td>67.9±1.5</td><td>46.5±5.2</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">XLM-R Base</td></tr><tr><td>FAIR</td><td>FastAlign</td><td>74.0±2.6</td><td>71.6±1.4</td><td>72.8±1.6</td><td>55.5±5.7</td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>73.3±1.8</td><td>72.9±0.8</td><td>73.1±1.3</td><td>53.2±3.8</td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td>73.9±2.8</td><td>72.8±2.2</td><td>73.3±2.3</td><td>53.1±4.4</td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>74.1±1.1</td><td>74.1±1.1</td><td>74.1±1.1</td><td>57.5±4.3</td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td>75.4±2.3</td><td>74.1±1.4</td><td>74.8±1.8</td><td>58.0±4.2</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>74.6±1.8</td><td>73.6±1.1</td><td>74.1±1.3</td><td>56.3±2.2</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>74.9±1.9</td><td>73.8±1.0</td><td>74.4±1.4</td><td>57.5±3.3</td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>74.7±2.2</td><td>72.1±0.6</td><td>73.3±1.3</td><td>54.2±4.6</td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td>76.8±1.6</td><td>72.8±1.0</td><td>74.7±1.0</td><td>53.0±2.7</td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>75.9±1.7</td><td><b>73.8</b>±0.6</td><td>74.8±1.0</td><td>57.5±4.5</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>77.0±1.5</td><td>72.1±0.6</td><td>74.5±0.8</td><td>51.0±0.9</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td>76.2±1.2</td><td>72.1±1.3</td><td>74.1±1.1</td><td>50.9±1.0</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td>75.5±1.2</td><td>71.9±1.1</td><td>73.7±0.9</td><td>52.1±3.5</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>75.5±0.9</td><td>72.6±1.1</td><td>74.0±1.0</td><td>52.6±3.8</td></tr><tr><td colspan="2">Cross-lingual Transfer</td><td>71.1±1.1</td><td>73.3±1.0</td><td>72.2±0.7</td><td>55.1±5.7</td></tr><tr><td colspan="2">Cross-lingual Transfer with realignment</td><td><b>78.2</b>±1.8</td><td>73.5±1.9</td><td><b>75.8</b>±1.3</td><td><b>58.1</b>±4.9</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">XLM-R Large</td></tr><tr><td>FAIR</td><td>FastAlign</td><td>77.7±3.6</td><td>75.1±2.3</td><td>76.4±2.7</td><td>65.8±2.8</td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>80.1±1.1</td><td>77.5±0.6</td><td>78.7±0.5</td><td>65.0±3.1</td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td>79.9±1.3</td><td>77.0±2.4</td><td>78.4±1.7</td><td>64.4±2.2</td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>80.7±1.6</td><td>77.3±0.9</td><td>79.0±0.6</td><td>65.8±0.6</td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td>80.0±1.7</td><td><b>78.5</b>±1.1</td><td>79.2±1.1</td><td>66.3±1.1</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>79.3±0.8</td><td>77.3±1.1</td><td>78.3±0.7</td><td>64.5±2.0</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>79.1±1.8</td><td>75.6±1.2</td><td>77.3±0.8</td><td>63.7±1.6</td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>78.5±2.5</td><td>75.1±1.6</td><td>76.7±1.7</td><td>61.7±1.9</td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td><b>83.2</b>±2.8</td><td>76.0±0.9</td><td><b>79.4</b>±1.6</td><td>64.3±2.2</td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>83.0±1.0</td><td>76.1±1.7</td><td><b>79.4</b>±1.3</td><td>64.7±1.3</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>80.0±1.5</td><td>76.0±1.5</td><td>77.9±1.2</td><td>64.9±0.9</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td>81.7±2.0</td><td>75.1±1.7</td><td>78.3±1.8</td><td>64.8±1.6</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td>80.3±1.9</td><td>74.6±1.6</td><td>77.4±1.3</td><td>62.9±3.2</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>80.2±0.9</td><td>75.3±2.0</td><td>77.6±1.1</td><td>63.0±2.9</td></tr><tr><td colspan="2">Cross-lingual Transfer</td><td>81.2±1.2</td><td>76.0±0.9</td><td>78.5±0.4</td><td>64.9±2.5</td></tr><tr><td colspan="2">Cross-lingual Transfer with realignment</td><td>80.7±2.1</td><td>76.0±1.4</td><td>78.3±1.6</td><td><b>66.8</b>±1.8</td></tr></tbody></table>
<table><caption>Table 23: translate-train results in French for domain and language-specific base models.</caption><thead><tr><th>translation</th><th>aligner</th><th>precision</th><th>recall</th><th>mirco-f1</th><th>macro-f1</th></tr></thead><tbody><tr><td colspan="6" style="text-align: center; font-weight: bold;">CamBERT Base</td></tr><tr><td>Opus</td><td>FastAlign</td><td><strong>74.9</strong><sub>±1.1</sub></td><td><strong>78.5</strong><sub>±0.9</sub></td><td><strong>76.7</strong><sub>±0.9</sub></td><td><strong>78.7</strong><sub>±1.0</sub></td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>73.2<sub>±1.2</sub></td><td>77.7<sub>±1.3</sub></td><td>75.4<sub>±1.2</sub></td><td>77.9<sub>±1.2</sub></td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td>74.4<sub>±0.8</sub></td><td>77.5<sub>±0.8</sub></td><td>75.9<sub>±0.8</sub></td><td>78.1<sub>±1.0</sub></td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>71.9<sub>±1.1</sub></td><td>76.5<sub>±1.1</sub></td><td>74.1<sub>±1.1</sub></td><td>76.7<sub>±1.0</sub></td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>72.3<sub>±2.0</sub></td><td>77.4<sub>±1.3</sub></td><td>74.8<sub>±1.7</sub></td><td>77.3<sub>±1.3</sub></td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>74.0<sub>±1.4</sub></td><td>77.8<sub>±1.5</sub></td><td>75.9<sub>±1.4</sub></td><td>78.2<sub>±1.3</sub></td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>73.3<sub>±1.0</sub></td><td>77.9<sub>±1.0</sub></td><td>75.5<sub>±0.9</sub></td><td>77.8<sub>±1.0</sub></td></tr><tr><td>Opus ft</td><td>FastAlign</td><td>74.2<sub>±2.1</sub></td><td>78.4<sub>±1.3</sub></td><td>76.2<sub>±1.7</sub></td><td>78.3<sub>±1.5</sub></td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td>71.0<sub>±1.6</sub></td><td>76.2<sub>±1.5</sub></td><td>73.5<sub>±1.5</sub></td><td>76.1<sub>±1.3</sub></td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>72.0<sub>±1.8</sub></td><td>77.4<sub>±1.7</sub></td><td>74.6<sub>±1.7</sub></td><td>77.2<sub>±1.6</sub></td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>70.7<sub>±1.8</sub></td><td>75.0<sub>±1.4</sub></td><td>72.8<sub>±1.6</sub></td><td>75.4<sub>±1.5</sub></td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>72.3<sub>±1.6</sub></td><td>76.6<sub>±1.6</sub></td><td>74.4<sub>±1.6</sub></td><td>76.8<sub>±1.5</sub></td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>72.2<sub>±2.4</sub></td><td>77.1<sub>±1.5</sub></td><td>74.6<sub>±1.9</sub></td><td>76.9<sub>±1.7</sub></td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>71.6<sub>±1.2</sub></td><td>77.0<sub>±0.4</sub></td><td>74.2<sub>±0.8</sub></td><td>76.7<sub>±0.4</sub></td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">DrBERT 7GB</td></tr><tr><td>Opus</td><td>FastAlign</td><td>70.9<sub>±2.4</sub></td><td>72.4<sub>±2.1</sub></td><td>71.7<sub>±2.1</sub></td><td>73.2<sub>±2.1</sub></td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>69.5<sub>±1.4</sub></td><td>71.3<sub>±1.5</sub></td><td>70.4<sub>±1.3</sub></td><td>72.3<sub>±1.6</sub></td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td>69.5<sub>±1.7</sub></td><td>71.7<sub>±0.7</sub></td><td>70.6<sub>±0.8</sub></td><td>72.6<sub>±0.7</sub></td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>69.3<sub>±1.1</sub></td><td>71.7<sub>±0.7</sub></td><td>70.4<sub>±0.8</sub></td><td>72.7<sub>±0.7</sub></td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>68.1<sub>±0.8</sub></td><td>70.3<sub>±1.5</sub></td><td>69.2<sub>±0.7</sub></td><td>71.3<sub>±0.8</sub></td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>69.4<sub>±1.4</sub></td><td>71.7<sub>±1.3</sub></td><td>70.5<sub>±1.2</sub></td><td>72.6<sub>±1.1</sub></td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>70.0<sub>±1.4</sub></td><td>71.4<sub>±1.0</sub></td><td>70.7<sub>±0.6</sub></td><td>72.7<sub>±0.5</sub></td></tr><tr><td>Opus ft</td><td>FastAlign</td><td><strong>73.2</strong><sub>±1.9</sub></td><td><strong>73.7</strong><sub>±1.5</sub></td><td><strong>73.5</strong><sub>±1.4</sub></td><td><strong>74.9</strong><sub>±1.4</sub></td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td>69.6<sub>±1.6</sub></td><td>71.7<sub>±1.1</sub></td><td>70.7<sub>±1.3</sub></td><td>72.7<sub>±1.1</sub></td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>70.5<sub>±1.6</sub></td><td>71.9<sub>±1.2</sub></td><td>71.2<sub>±1.2</sub></td><td>73.4<sub>±1.1</sub></td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>69.1<sub>±1.5</sub></td><td>70.6<sub>±1.1</sub></td><td>69.8<sub>±0.5</sub></td><td>71.8<sub>±0.3</sub></td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>70.8<sub>±1.5</sub></td><td>72.6<sub>±1.3</sub></td><td>71.6<sub>±1.0</sub></td><td>73.7<sub>±0.8</sub></td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>70.7<sub>±1.0</sub></td><td>71.7<sub>±2.1</sub></td><td>71.2<sub>±1.4</sub></td><td>73.2<sub>±1.2</sub></td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>70.1<sub>±0.8</sub></td><td>70.6<sub>±1.7</sub></td><td>70.4<sub>±1.1</sub></td><td>72.4<sub>±1.2</sub></td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">DrBERT-PubMedBERT</td></tr><tr><td>Opus</td><td>FastAlign</td><td><strong>76.2</strong><sub>±1.4</sub></td><td>79.4<sub>±0.8</sub></td><td>77.8<sub>±1.1</sub></td><td>79.7<sub>±0.8</sub></td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>75.2<sub>±1.2</sub></td><td>77.5<sub>±0.7</sub></td><td>76.4<sub>±0.9</sub></td><td>78.4<sub>±0.7</sub></td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td>76.0<sub>±1.0</sub></td><td>79.2<sub>±1.3</sub></td><td>77.6<sub>±1.1</sub></td><td>79.6<sub>±0.9</sub></td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>74.3<sub>±1.4</sub></td><td>78.9<sub>±1.3</sub></td><td>76.5<sub>±1.2</sub></td><td>78.7<sub>±1.1</sub></td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>74.0<sub>±1.2</sub></td><td>77.8<sub>±1.1</sub></td><td>75.9<sub>±1.1</sub></td><td>78.2<sub>±1.0</sub></td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>73.9<sub>±1.1</sub></td><td>77.7<sub>±0.6</sub></td><td>75.8<sub>±0.9</sub></td><td>78.1<sub>±0.7</sub></td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>75.2<sub>±0.4</sub></td><td>79.1<sub>±0.8</sub></td><td>77.1<sub>±0.5</sub></td><td>79.0<sub>±0.4</sub></td></tr><tr><td>Opus ft</td><td>FastAlign</td><td><strong>76.2</strong><sub>±1.8</sub></td><td><strong>81.5</strong><sub>±1.0</sub></td><td><strong>78.8</strong><sub>±1.4</sub></td><td><strong>80.4</strong><sub>±1.3</sub></td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td>73.7<sub>±1.4</sub></td><td>78.7<sub>±1.2</sub></td><td>76.1<sub>±1.3</sub></td><td>78.4<sub>±1.0</sub></td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>75.4<sub>±1.2</sub></td><td>81.2<sub>±0.8</sub></td><td>78.2<sub>±0.8</sub></td><td>80.3<sub>±0.7</sub></td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>74.9<sub>±0.9</sub></td><td>80.7<sub>±0.7</sub></td><td>77.7<sub>±0.8</sub></td><td>79.7<sub>±0.5</sub></td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>74.8<sub>±1.3</sub></td><td>79.7<sub>±0.8</sub></td><td>77.2<sub>±1.1</sub></td><td>79.2<sub>±0.8</sub></td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>75.6<sub>±1.1</sub></td><td>79.3<sub>±1.5</sub></td><td>77.4<sub>±1.2</sub></td><td>79.4<sub>±1.1</sub></td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>75.5<sub>±1.2</sub></td><td>80.3<sub>±1.3</sub></td><td>77.8<sub>±1.2</sub></td><td>79.8<sub>±1.0</sub></td></tr></tbody></table>
<table><caption>Table 24: translate-train results in German for domain and language-specific base models.</caption><thead><tr><th>translation</th><th>aligner</th><th>precision</th><th>recall</th><th>micro-f1</th><th>macro-f1</th></tr></thead><tbody><tr><td colspan="6" style="text-align: center; font-weight: bold;">GottBERT</td></tr><tr><td>FAIR</td><td>FastAlign</td><td>75.9±3.2</td><td>70.3±2.2</td><td>73.0±2.6</td><td>54.8±4.8</td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>79.5±1.6</td><td>73.4±2.0</td><td>76.3±1.7</td><td>60.5±4.6</td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td>77.9±1.6</td><td>72.9±1.4</td><td>75.3±1.4</td><td>57.2±5.2</td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>78.4±2.8</td><td>73.1±2.4</td><td>75.7±2.4</td><td>57.9±5.4</td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td>77.9±1.8</td><td>72.6±2.3</td><td>75.1±1.8</td><td>53.4±2.7</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>79.0±1.8</td><td><strong>74.1</strong>±1.6</td><td>76.5±1.6</td><td><strong>60.8</strong>±3.5</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>77.9±2.8</td><td>73.6±2.5</td><td>75.7±2.6</td><td>57.6±4.4</td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>76.6±3.2</td><td>70.1±1.9</td><td>73.2±2.4</td><td>53.7±4.5</td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td><strong>80.2</strong>±1.1</td><td>73.3±1.0</td><td><strong>76.6</strong>±0.8</td><td>58.7±6.1</td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>79.2±0.8</td><td>73.1±1.1</td><td>76.0±0.7</td><td>58.8±2.4</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>78.5±0.7</td><td>72.4±1.0</td><td>75.3±0.8</td><td>56.1±6.6</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td>78.8±2.3</td><td>72.3±1.9</td><td>75.4±1.8</td><td>55.2±6.4</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td>78.6±1.6</td><td>72.6±1.6</td><td>75.5±1.4</td><td>55.5±6.4</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>79.4±1.8</td><td>72.3±0.8</td><td>75.6±1.1</td><td>56.5±3.2</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">medBERT.de</td></tr><tr><td>FAIR</td><td>FastAlign</td><td>75.1±2.8</td><td>69.2±1.6</td><td>72.0±2.0</td><td>56.7±5.3</td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>75.4±1.6</td><td>71.9±1.8</td><td>73.6±1.2</td><td>58.1±5.0</td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td><strong>77.0</strong>±3.5</td><td>72.9±1.6</td><td>74.9±2.4</td><td>59.5±5.9</td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>76.4±4.5</td><td>70.9±1.6</td><td>73.5±2.7</td><td>56.2±5.4</td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td>75.8±4.0</td><td>72.4±2.5</td><td>74.1±3.1</td><td>57.2±6.4</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>74.9±3.9</td><td>71.3±1.6</td><td>73.0±2.4</td><td>56.7±5.5</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>76.1±4.1</td><td>71.6±1.9</td><td>73.7±2.6</td><td>57.2±5.7</td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>72.6±2.0</td><td>68.9±0.8</td><td>70.7±1.2</td><td>60.7±1.2</td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td>75.2±2.0</td><td>72.6±0.9</td><td>73.9±1.0</td><td>61.1±1.9</td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>76.4±2.4</td><td><strong>73.6</strong>±0.9</td><td><strong>75.0</strong>±1.6</td><td>62.2±3.3</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>75.1±3.6</td><td>71.8±2.8</td><td>73.4±3.1</td><td>60.5±3.5</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td>75.3±3.2</td><td>71.9±2.2</td><td>73.6±2.4</td><td>58.6±6.1</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td>74.1±1.1</td><td>71.4±0.8</td><td>72.7±0.5</td><td>57.8±4.9</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>75.5±1.3</td><td>72.3±1.0</td><td>73.8±1.0</td><td><strong>62.6</strong>±1.2</td></tr></tbody></table>
<table><caption>Table 25: Full results for the translate-test approach in French with multilingual language models.</caption><thead><tr><th>translation</th><th>aligner</th><th>precision</th><th>recall</th><th>micro-f1</th><th>macro-f1</th></tr></thead><tbody><tr><td colspan="6" style="text-align:center; font-weight:bold;">distilmBERT</td></tr><tr><td>Opus</td><td>FastAlign</td><td>65.9±1.8</td><td>67.0±1.5</td><td>66.4±1.5</td><td>68.5±1.3</td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td><b>70.5</b>±1.7</td><td>68.8±1.4</td><td>69.6±1.4</td><td>71.8±1.3</td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td><b>70.5</b>±1.7</td><td><b>69.0</b>±1.4</td><td><b>69.7</b>±1.5</td><td><b>71.9</b>±1.3</td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>70.0±1.8</td><td>68.8±1.6</td><td>69.4±1.6</td><td>71.6±1.4</td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>69.7±1.8</td><td>68.8±1.6</td><td>69.2±1.6</td><td>71.5±1.4</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>70.1±1.7</td><td>68.9±1.5</td><td>69.5±1.5</td><td>71.7±1.4</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>69.0±1.8</td><td>68.8±1.6</td><td>68.9±1.6</td><td>71.3±1.4</td></tr><tr><td>Opus ft</td><td>FastAlign</td><td>63.2±1.7</td><td>66.6±1.6</td><td>64.8±1.5</td><td>66.7±1.2</td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td>69.9±1.6</td><td>68.4±1.3</td><td>69.2±1.4</td><td>71.4±1.2</td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>69.2±1.7</td><td>68.7±1.4</td><td>68.9±1.5</td><td>71.3±1.3</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>69.4±1.8</td><td>68.6±1.6</td><td>69.0±1.6</td><td>71.2±1.4</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>69.1±1.7</td><td>68.6±1.5</td><td>68.9±1.5</td><td>71.1±1.3</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>69.1±1.7</td><td>68.6±1.5</td><td>68.8±1.5</td><td>71.1±1.3</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>67.3±1.7</td><td>68.6±1.5</td><td>67.9±1.5</td><td>70.5±1.3</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">XLM-R Base</td></tr><tr><td>Opus</td><td>FastAlign</td><td>68.7±1.7</td><td>71.8±1.2</td><td>70.2±1.4</td><td>72.4±1.2</td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>74.9±1.5</td><td>73.8±1.2</td><td>74.3±1.3</td><td>76.5±1.1</td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td>74.8±1.6</td><td><b>74.1</b>±1.3</td><td><b>74.4</b>±1.3</td><td><b>76.6</b>±1.2</td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>74.6±1.6</td><td>73.8±1.2</td><td>74.2±1.2</td><td>76.3±1.1</td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>74.1±1.5</td><td>73.8±1.2</td><td>74.0±1.3</td><td>76.1±1.1</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>74.6±1.6</td><td>73.8±1.2</td><td>74.2±1.2</td><td>76.3±1.1</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>73.4±1.5</td><td>73.8±1.2</td><td>73.6±1.2</td><td>75.9±1.1</td></tr><tr><td>Opus ft</td><td>FastAlign</td><td>67.7±1.6</td><td>71.7±1.7</td><td>69.6±1.4</td><td>71.4±1.3</td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td><b>75.4</b>±1.7</td><td>73.1±1.7</td><td>74.2±1.6</td><td>76.3±1.5</td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>74.2±1.7</td><td>73.3±1.7</td><td>73.8±1.6</td><td>76.0±1.6</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>74.6±1.8</td><td>73.1±1.7</td><td>73.8±1.6</td><td>75.8±1.5</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>74.1±1.8</td><td>73.1±1.7</td><td>73.6±1.6</td><td>75.6±1.5</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>74.3±1.8</td><td>73.2±1.8</td><td>73.7±1.7</td><td>75.7±1.6</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>72.3±1.7</td><td>73.1±1.7</td><td>72.7±1.6</td><td>75.0±1.5</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">XLM-R Large</td></tr><tr><td>Opus</td><td>FastAlign</td><td>69.6±1.3</td><td>71.0±0.9</td><td>70.3±1.1</td><td>72.7±0.9</td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>75.9±1.0</td><td>73.4±0.8</td><td>74.7±0.9</td><td>77.0±0.8</td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td>76.0±1.0</td><td><b>73.7</b>±0.7</td><td>74.8±0.8</td><td>77.2±0.8</td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>75.5±1.0</td><td>73.5±0.8</td><td>74.5±0.9</td><td>76.8±0.8</td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>75.2±1.1</td><td>73.5±0.8</td><td>74.3±0.9</td><td>76.7±0.8</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>75.6±1.1</td><td>73.5±0.8</td><td>74.5±0.9</td><td>76.8±0.8</td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>74.5±1.1</td><td>73.5±0.8</td><td>74.0±0.9</td><td>76.5±0.8</td></tr><tr><td>Opus ft</td><td>FastAlign</td><td>70.8±1.4</td><td>72.2±0.5</td><td>71.5±1.0</td><td>73.2±0.9</td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td><b>77.3</b>±1.3</td><td>73.3±0.6</td><td><b>75.3</b>±0.9</td><td><b>77.6</b>±0.7</td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>76.3±1.1</td><td>73.6±0.6</td><td>75.0±0.9</td><td>77.4±0.6</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>76.4±1.3</td><td>73.4±0.6</td><td>74.9±0.9</td><td>77.1±0.7</td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>76.1±1.2</td><td>73.4±0.6</td><td>74.7±0.9</td><td>77.0±0.7</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>76.1±1.2</td><td>73.5±0.6</td><td>74.8±0.8</td><td>77.0±0.7</td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>74.1±1.1</td><td>73.4±0.6</td><td>73.8±0.8</td><td>76.3±0.6</td></tr></tbody></table>
<table><caption>Table 26: Full results for the translate-test approach in German with multilingual language models.</caption><thead><tr><th>translation</th><th>aligner</th><th>precision</th><th>recall</th><th>micro-f1</th><th>macro-f1</th></tr></thead><tbody><tr><td colspan="6" style="text-align:center; font-weight:bold;">distilmBERT</td></tr><tr><td>FAIR</td><td>FastAlign</td><td>37.6±2.2</td><td>36.5±1.1</td><td>37.0±1.5</td><td>24.4±1.2</td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>66.8±2.7</td><td>63.7±1.4</td><td>65.2±1.9</td><td>49.1±1.6</td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td>66.9±2.7</td><td>63.9±1.6</td><td>65.3±1.9</td><td>49.3±1.7</td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>68.6±2.9</td><td>63.7±1.4</td><td>66.0±1.9</td><td>49.6±1.6</td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td>68.8±2.8</td><td>64.5±1.4</td><td>66.6±1.9</td><td>50.2±1.6</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>67.2±2.8</td><td>62.9±1.4</td><td>64.9±1.9</td><td>48.6±1.6</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>67.3±2.9</td><td>63.9±1.6</td><td>65.5±2.0</td><td>49.4±1.7</td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>29.1±0.6</td><td>31.1±0.8</td><td>30.1±0.6</td><td>18.3±0.6</td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td>69.8±2.6</td><td>65.4±1.4</td><td>67.5±1.8</td><td>50.9±3.8</td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>68.9±2.5</td><td>66.4±1.4</td><td>67.6±1.7</td><td>51.4±3.8</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>69.5±2.3</td><td>65.4±1.4</td><td>67.4±1.7</td><td>50.8±3.8</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td>69.7±2.3</td><td>65.5±1.5</td><td>67.5±1.6</td><td>51.0±3.8</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td><b>69.9</b>±2.4</td><td><b>66.7</b>±1.6</td><td><b>68.3</b>±1.8</td><td><b>51.9</b>±3.7</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>69.2±2.1</td><td>66.2±1.4</td><td>67.6±1.6</td><td>51.3±3.8</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">XLM-R Base</td></tr><tr><td>FAIR</td><td>FastAlign</td><td>37.0±1.5</td><td>41.2±0.8</td><td>39.0±0.8</td><td>28.4±0.5</td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>71.1±0.7</td><td>72.3±0.8</td><td>71.7±0.3</td><td>56.4±0.6</td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td>71.1±0.7</td><td>72.3±0.8</td><td>71.7±0.3</td><td>56.4±0.6</td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>72.0±0.7</td><td>71.4±0.8</td><td>71.7±0.3</td><td>56.1±0.7</td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td><b>72.3</b>±0.7</td><td>72.3±0.8</td><td>72.3±0.3</td><td><b>56.7</b>±0.7</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>70.6±0.7</td><td>70.6±0.8</td><td>70.6±0.3</td><td>55.2±0.6</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>70.8±0.7</td><td>71.4±0.8</td><td>71.1±0.3</td><td>55.8±0.6</td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>29.7±0.7</td><td>35.1±1.1</td><td>32.2±0.7</td><td>22.0±0.8</td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td>71.0±1.2</td><td>71.9±2.0</td><td>71.4±1.2</td><td>54.3±5.2</td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>70.4±0.8</td><td>72.9±1.6</td><td>71.6±0.6</td><td>54.8±5.0</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>71.7±1.2</td><td>72.4±1.6</td><td>72.1±0.8</td><td>55.1±5.3</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td>72.1±1.2</td><td>72.4±1.6</td><td>72.3±0.7</td><td>55.2±5.2</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td>72.0±1.0</td><td><b>73.4</b>±1.7</td><td><b>72.7</b>±0.8</td><td>55.5±5.0</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>71.7±1.1</td><td>72.8±2.0</td><td>72.2±1.0</td><td>55.3±5.5</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">XLM-R Large</td></tr><tr><td>FAIR</td><td>FastAlign</td><td>39.9±1.7</td><td>41.4±0.7</td><td>40.6±1.0</td><td>28.7±1.2</td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>76.4±1.6</td><td>72.5±0.9</td><td>74.4±1.0</td><td>49.2±0.4</td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td>76.4±1.6</td><td>72.5±0.9</td><td>74.4±1.0</td><td>49.2±0.4</td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>77.5±1.7</td><td>71.6±0.9</td><td>74.5±1.0</td><td>49.0±0.5</td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td>77.7±1.7</td><td>72.5±0.9</td><td>75.0±1.0</td><td>49.6±0.5</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>75.9±1.7</td><td>70.8±0.9</td><td>73.3±1.0</td><td>47.9±0.4</td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>76.1±1.6</td><td>71.6±0.9</td><td>73.8±1.0</td><td>48.6±0.4</td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>30.6±1.5</td><td>34.0±0.9</td><td>32.2±1.2</td><td>20.9±0.9</td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td>76.6±3.5</td><td>72.7±2.1</td><td>74.6±2.7</td><td>49.4±1.8</td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>76.5±3.7</td><td><b>74.4</b>±2.1</td><td>75.4±2.8</td><td>50.7±1.9</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>77.8±3.6</td><td>73.3±2.2</td><td>75.5±2.7</td><td>50.1±1.9</td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td><b>78.3</b>±3.7</td><td>73.3±2.2</td><td>75.7±2.8</td><td>50.2±1.9</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td>77.9±3.8</td><td><b>74.4</b>±2.1</td><td><b>76.1</b>±2.8</td><td><b>51.0</b>±1.9</td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>77.5±3.8</td><td>73.5±2.1</td><td>75.5±2.8</td><td>50.2±1.9</td></tr></tbody></table>
<table><caption>Table 27: Results of translate-test in French with PubMedBERT.</caption><thead><tr><th>translation</th><th>aligner</th><th>precision</th><th>recall</th><th>micro-f1</th><th>macro-f1</th></tr></thead><tbody><tr><td>Opus</td><td>FastAlign</td><td>68.5<sub>±1.2</sub></td><td>69.1<sub>±1.2</sub></td><td>68.8<sub>±1.0</sub></td><td>71.2<sub>±1.0</sub></td></tr><tr><td>Opus</td><td>AWESOME w/o co</td><td>75.2<sub>±1.4</sub></td><td><b>71.8</b><sub>±1.3</sub></td><td><b>73.5</b><sub>±1.2</sub></td><td><b>75.8</b><sub>±1.2</sub></td></tr><tr><td>Opus</td><td>AWESOME w/ co</td><td>74.8<sub>±1.5</sub></td><td><b>71.8</b><sub>±1.4</sub></td><td>73.3<sub>±1.3</sub></td><td>75.6<sub>±1.2</sub></td></tr><tr><td>Opus</td><td>AWESOME ft w/o co</td><td>74.7<sub>±1.4</sub></td><td><b>71.8</b><sub>±1.3</sub></td><td>73.2<sub>±1.2</sub></td><td>75.5<sub>±1.2</sub></td></tr><tr><td>Opus</td><td>AWESOME ft w/ co</td><td>74.1<sub>±1.4</sub></td><td>71.5<sub>±1.3</sub></td><td>72.8<sub>±1.2</sub></td><td>75.1<sub>±1.2</sub></td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/o co</td><td>74.7<sub>±1.4</sub></td><td><b>71.8</b><sub>±1.3</sub></td><td>73.2<sub>±1.2</sub></td><td>75.5<sub>±1.2</sub></td></tr><tr><td>Opus</td><td>AWESOME pt+ft w/ co</td><td>73.3<sub>±1.4</sub></td><td>71.5<sub>±1.3</sub></td><td>72.4<sub>±1.2</sub></td><td>74.9<sub>±1.2</sub></td></tr><tr><td>Opus ft</td><td>FastAlign</td><td>67.9<sub>±1.3</sub></td><td>69.3<sub>±1.5</sub></td><td>68.6<sub>±1.2</sub></td><td>70.6<sub>±1.2</sub></td></tr><tr><td>Opus ft</td><td>AWESOME w/o co</td><td><b>75.4</b><sub>±1.3</sub></td><td>71.4<sub>±1.6</sub></td><td>73.3<sub>±1.3</sub></td><td><b>75.8</b><sub>±1.3</sub></td></tr><tr><td>Opus ft</td><td>AWESOME w/ co</td><td>74.1<sub>±1.4</sub></td><td>71.4<sub>±1.7</sub></td><td>72.7<sub>±1.4</sub></td><td>75.4<sub>±1.4</sub></td></tr><tr><td>Opus ft</td><td>AWESOME ft w/o co</td><td>74.7<sub>±1.4</sub></td><td>71.4<sub>±1.5</sub></td><td>73.0<sub>±1.3</sub></td><td>75.4<sub>±1.3</sub></td></tr><tr><td>Opus ft</td><td>AWESOME ft w/ co</td><td>74.3<sub>±1.4</sub></td><td>71.5<sub>±1.6</sub></td><td>72.9<sub>±1.4</sub></td><td>75.3<sub>±1.4</sub></td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/o co</td><td>74.3<sub>±1.3</sub></td><td>71.5<sub>±1.6</sub></td><td>72.9<sub>±1.3</sub></td><td>75.4<sub>±1.3</sub></td></tr><tr><td>Opus ft</td><td>AWESOME pt+ft w/ co</td><td>72.3<sub>±1.3</sub></td><td>71.5<sub>±1.6</sub></td><td>71.9<sub>±1.3</sub></td><td>74.7<sub>±1.3</sub></td></tr></tbody></table>

<table><caption>Table 28: Results of translate-test in German with PubMedBERT.</caption><thead><tr><th>translation</th><th>aligner</th><th>precision</th><th>recall</th><th>micro-f1</th><th>macro-f1</th></tr></thead><tbody><tr><td>FAIR</td><td>FastAlign</td><td>40.5<sub>±2.0</sub></td><td>41.8<sub>±1.0</sub></td><td>41.1<sub>±1.4</sub></td><td>29.2<sub>±1.2</sub></td></tr><tr><td>FAIR</td><td>AWESOME w/o co</td><td>73.6<sub>±2.7</sub></td><td>71.9<sub>±1.1</sub></td><td>72.7<sub>±1.6</sub></td><td>55.1<sub>±3.9</sub></td></tr><tr><td>FAIR</td><td>AWESOME w/ co</td><td>73.6<sub>±2.7</sub></td><td>71.9<sub>±1.1</sub></td><td>72.7<sub>±1.6</sub></td><td>55.1<sub>±3.9</sub></td></tr><tr><td>FAIR</td><td>AWESOME ft w/o co</td><td>74.7<sub>±2.7</sub></td><td>71.1<sub>±1.1</sub></td><td>72.8<sub>±1.6</sub></td><td>54.9<sub>±3.8</sub></td></tr><tr><td>FAIR</td><td>AWESOME ft w/ co</td><td><b>74.8</b><sub>±2.9</sub></td><td>71.9<sub>±1.1</sub></td><td><b>73.3</b><sub>±1.7</sub></td><td><b>55.4</b><sub>±3.9</sub></td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/o co</td><td>73.2<sub>±2.8</sub></td><td>70.3<sub>±1.1</sub></td><td>71.7<sub>±1.6</sub></td><td>53.9<sub>±3.8</sub></td></tr><tr><td>FAIR</td><td>AWESOME pt+ft w/ co</td><td>73.4<sub>±2.8</sub></td><td>71.1<sub>±1.1</sub></td><td>72.2<sub>±1.6</sub></td><td>54.5<sub>±3.9</sub></td></tr><tr><td>FAIR ft</td><td>FastAlign</td><td>29.7<sub>±1.8</sub></td><td>34.3<sub>±1.7</sub></td><td>31.8<sub>±1.6</sub></td><td>21.7<sub>±2.6</sub></td></tr><tr><td>FAIR ft</td><td>AWESOME w/o co</td><td>72.6<sub>±0.7</sub></td><td>70.8<sub>±2.3</sub></td><td>71.6<sub>±1.5</sub></td><td>51.9<sub>±4.0</sub></td></tr><tr><td>FAIR ft</td><td>AWESOME w/ co</td><td>71.3<sub>±0.9</sub></td><td>71.9<sub>±2.0</sub></td><td>71.6<sub>±1.3</sub></td><td>52.4<sub>±3.6</sub></td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/o co</td><td>73.4<sub>±1.0</sub></td><td>71.1<sub>±2.5</sub></td><td>72.2<sub>±1.7</sub></td><td>52.7<sub>±4.5</sub></td></tr><tr><td>FAIR ft</td><td>AWESOME ft w/ co</td><td>73.4<sub>±1.0</sub></td><td>71.1<sub>±2.5</sub></td><td>72.2<sub>±1.7</sub></td><td>52.7<sub>±4.5</sub></td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/o co</td><td>72.9<sub>±0.9</sub></td><td><b>72.3</b><sub>±2.3</sub></td><td>72.6<sub>±1.5</sub></td><td>53.1<sub>±3.9</sub></td></tr><tr><td>FAIR ft</td><td>AWESOME pt+ft w/ co</td><td>72.9<sub>±0.8</sub></td><td>71.8<sub>±2.5</sub></td><td>72.3<sub>±1.6</sub></td><td>53.0<sub>±4.5</sub></td></tr></tbody></table>
<table><caption>Table 29: Comparison class by class for translate-train and CLT on MedNERF with multilingual language models.</caption><thead><tr><th>model</th><th></th><th>Drug</th><th>Strength</th><th>Frequency</th><th>Duration</th><th>Dosage</th><th>Form</th></tr><tr><th></th><th></th><th>p</th><th>r</th><th>p</th><th>r</th><th>p</th><th>r</th></tr><tr><th></th><th></th><th>f1</th><th>f1</th><th>f1</th><th>f1</th><th>f1</th><th>f1</th></tr></thead><tbody><tr><td colspan="8" style="text-align: left; font-weight: bold;">distilmBERT</td></tr><tr><td>CLT</td><td></td><td>52.2</td><td>82.4</td><td>63.8</td><td><strong>79.1</strong></td><td><strong>92.2</strong></td><td><strong>85.1</strong></td><td><strong>63.1</strong></td><td><strong>58.2</strong></td><td><strong>60.5</strong></td><td>87.6</td><td>88.4</td><td>88.0</td><td>67.6</td><td>67.4</td><td>67.5</td><td>55.3</td><td>38.5</td><td>44.4</td></tr><tr><td>+ realigned</td><td></td><td>67.8</td><td>80.9</td><td>73.8</td><td>78.5</td><td>85.9</td><td>82.0</td><td>50.3</td><td>37.9</td><td>43.2</td><td>74.7</td><td>73.5</td><td>74.1</td><td>67.9</td><td>63.2</td><td>65.4</td><td>70.9</td><td>61.3</td><td>65.8</td></tr><tr><td>Opus - FastAlign</td><td></td><td><strong>69.9</strong></td><td>77.0</td><td>73.2</td><td>76.3</td><td>84.3</td><td>80.1</td><td>48.7</td><td>44.7</td><td>46.6</td><td><strong>94.2</strong></td><td><strong>90.7</strong></td><td><strong>92.4</strong></td><td>64.1</td><td>70.5</td><td>67.1</td><td>67.4</td><td>62.2</td><td>64.6</td></tr><tr><td>Opus - AWESOME pt</td><td></td><td>68.1</td><td>82.7</td><td>74.7</td><td>77.1</td><td>89.0</td><td>82.5</td><td>40.5</td><td>39.2</td><td>39.8</td><td>93.8</td><td><strong>90.7</strong></td><td>92.2</td><td>67.0</td><td>70.8</td><td>68.8</td><td>69.7</td><td>59.8</td><td>64.2</td></tr><tr><td>Opus - AWESOME ft</td><td></td><td>68.0</td><td><strong>83.3</strong></td><td><strong>74.8</strong></td><td>74.4</td><td>87.1</td><td>80.1</td><td>33.7</td><td>33.2</td><td>33.4</td><td>93.8</td><td><strong>90.7</strong></td><td>92.2</td><td>66.6</td><td>70.5</td><td>68.5</td><td>68.2</td><td>60.0</td><td>63.7</td></tr><tr><td>Opus - AWESOME pt+fr</td><td></td><td>65.9</td><td>82.4</td><td>73.2</td><td>76.4</td><td>89.4</td><td>82.3</td><td>39.4</td><td>38.9</td><td>39.1</td><td>92.9</td><td><strong>90.7</strong></td><td>91.8</td><td><strong>68.4</strong></td><td><strong>72.1</strong></td><td><strong>70.2</strong></td><td>70.1</td><td>60.4</td><td>64.8</td></tr><tr><td>Opus ft - FastAlign</td><td></td><td>68.4</td><td>80.6</td><td>74.0</td><td>74.3</td><td>85.1</td><td>79.3</td><td>45.0</td><td>44.7</td><td>44.8</td><td>91.2</td><td><strong>90.7</strong></td><td>90.9</td><td>59.7</td><td>68.2</td><td>63.6</td><td>71.0</td><td><strong>64.3</strong></td><td><strong>67.4</strong></td></tr><tr><td>Opus ft - AWESOME pt</td><td></td><td>66.3</td><td>81.8</td><td>73.2</td><td>77.0</td><td>91.8</td><td>83.7</td><td>37.1</td><td>36.3</td><td>36.7</td><td>92.9</td><td><strong>90.7</strong></td><td>91.8</td><td>62.4</td><td>68.4</td><td>65.3</td><td>69.8</td><td>59.1</td><td>64.0</td></tr><tr><td>Opus ft - AWESOME ft</td><td></td><td>65.5</td><td>83.0</td><td>73.2</td><td>78.5</td><td>90.6</td><td>84.0</td><td>34.2</td><td>32.9</td><td>33.5</td><td>93.3</td><td><strong>90.7</strong></td><td>92.0</td><td>61.0</td><td>69.5</td><td>65.0</td><td>70.4</td><td>59.1</td><td>64.1</td></tr><tr><td>Opus ft - AWESOME pt+fr</td><td></td><td>65.0</td><td>82.7</td><td>72.8</td><td>77.6</td><td>90.2</td><td>83.4</td><td>37.3</td><td>37.9</td><td>37.6</td><td>93.3</td><td><strong>90.7</strong></td><td>92.0</td><td>59.8</td><td>69.5</td><td>64.3</td><td><strong>71.1</strong></td><td>58.5</td><td>64.1</td></tr><tr><td colspan="8" style="text-align: left; font-weight: bold;">XLM-R Base</td></tr><tr><td>CLT</td><td></td><td>83.8</td><td>83.0</td><td>83.4</td><td>87.7</td><td>97.6</td><td>92.4</td><td><strong>73.0</strong></td><td><strong>75.3</strong></td><td><strong>74.0</strong></td><td>88.8</td><td>87.9</td><td>88.3</td><td>77.4</td><td>76.6</td><td>77.0</td><td>70.9</td><td>69.5</td><td>70.2</td></tr><tr><td>+ realigned</td><td></td><td><strong>83.9</strong></td><td>82.8</td><td><strong>83.3</strong></td><td>87.2</td><td>98.0</td><td>92.3</td><td>66.9</td><td>69.7</td><td>68.2</td><td>90.7</td><td>90.1</td><td>90.4</td><td>71.7</td><td>67.8</td><td>69.7</td><td>71.8</td><td>66.9</td><td>69.3</td></tr><tr><td>Opus - FastAlign</td><td></td><td>81.2</td><td>82.7</td><td>82.0</td><td>83.2</td><td>94.9</td><td>88.6</td><td>66.9</td><td>72.1</td><td>69.4</td><td>89.6</td><td>88.4</td><td>89.0</td><td>78.5</td><td>77.6</td><td>78.1</td><td>73.9</td><td>70.3</td><td>72.0</td></tr><tr><td>Opus - AWESOME pt</td><td></td><td>80.5</td><td>82.4</td><td>81.4</td><td>85.3</td><td>95.3</td><td>90.0</td><td>55.3</td><td>61.6</td><td>58.1</td><td><strong>92.0</strong></td><td><strong>90.2</strong></td><td><strong>91.1</strong></td><td>78.3</td><td><strong>78.7</strong></td><td>78.4</td><td>75.2</td><td>69.7</td><td>72.2</td></tr><tr><td>Opus - AWESOME ft</td><td></td><td>80.5</td><td>82.4</td><td>81.4</td><td>84.9</td><td>94.9</td><td>89.6</td><td>51.0</td><td>56.1</td><td>53.4</td><td>91.5</td><td>89.8</td><td>90.6</td><td>73.1</td><td>74.2</td><td>73.6</td><td><strong>75.3</strong></td><td>69.7</td><td>72.4</td></tr><tr><td>Opus - AWESOME pt+fr</td><td></td><td>81.1</td><td>81.8</td><td>81.4</td><td>85.1</td><td>95.7</td><td>90.1</td><td>52.1</td><td>57.9</td><td>54.8</td><td>91.4</td><td>89.3</td><td>90.4</td><td><strong>79.9</strong></td><td><strong>78.7</strong></td><td><strong>79.3</strong></td><td>73.2</td><td>71.2</td><td>72.2</td></tr><tr><td>Opus ft - FastAlign</td><td></td><td>83.7</td><td>82.7</td><td>83.2</td><td>84.2</td><td>96.1</td><td>89.7</td><td>70.7</td><td>73.4</td><td>72.0</td><td>90.5</td><td>87.9</td><td>89.2</td><td>75.0</td><td>75.0</td><td>75.0</td><td>72.7</td><td><strong>72.3</strong></td><td><strong>72.5</strong></td></tr><tr><td>Opus ft - AWESOME pt</td><td></td><td>81.2</td><td><strong>83.3</strong></td><td>82.2</td><td>87.3</td><td><strong>98.4</strong></td><td>92.5</td><td>47.4</td><td>53.4</td><td>50.2</td><td>90.2</td><td>89.3</td><td>89.7</td><td>74.0</td><td>74.7</td><td>74.3</td><td>73.0</td><td>69.9</td><td>71.4</td></tr><tr><td>Opus ft - AWESOME ft</td><td></td><td>81.0</td><td>82.4</td><td>81.7</td><td><strong>89.0</strong></td><td>98.0</td><td><strong>93.3</strong></td><td>46.3</td><td>52.9</td><td>49.4</td><td>90.2</td><td>89.8</td><td>90.0</td><td>74.8</td><td>76.6</td><td>75.7</td><td>74.8</td><td>69.7</td><td>72.1</td></tr><tr><td>Opus ft - AWESOME pt+fr</td><td></td><td>80.7</td><td>82.4</td><td>81.5</td><td>87.0</td><td>97.3</td><td>91.9</td><td>54.9</td><td>61.6</td><td>58.0</td><td>89.8</td><td>89.3</td><td>89.5</td><td>78.9</td><td><strong>78.7</strong></td><td>78.8</td><td>72.5</td><td>69.9</td><td>71.2</td></tr><tr><td colspan="8" style="text-align: left; font-weight: bold;">XLM-R Large</td></tr><tr><td>CLT</td><td></td><td>80.3</td><td>81.5</td><td>80.9</td><td><strong>90.5</strong></td><td>97.3</td><td><strong>93.8</strong></td><td>66.0</td><td>68.9</td><td>67.3</td><td>93.2</td><td><strong>89.3</strong></td><td>91.2</td><td>76.1</td><td>75.3</td><td>75.7</td><td>75.8</td><td>67.5</td><td>71.3</td></tr><tr><td>+ realigned</td><td></td><td><strong>85.5</strong></td><td>82.1</td><td>83.7</td><td>90.2</td><td>97.3</td><td>93.6</td><td><strong>66.6</strong></td><td>67.6</td><td>67.1</td><td>92.7</td><td>88.8</td><td>90.7</td><td><strong>80.2</strong></td><td>77.4</td><td><strong>78.7</strong></td><td>74.2</td><td>68.0</td><td>70.9</td></tr><tr><td>Opus - FastAlign</td><td></td><td>82.0</td><td>80.3</td><td>81.2</td><td>87.6</td><td>96.5</td><td>91.8</td><td>65.7</td><td>69.7</td><td><strong>67.6</strong></td><td>92.3</td><td>88.4</td><td>90.3</td><td>76.5</td><td>76.1</td><td>76.3</td><td>78.7</td><td>66.2</td><td>71.9</td></tr><tr><td>Opus - AWESOME pt</td><td></td><td>80.8</td><td>82.7</td><td>81.7</td><td>89.8</td><td>96.1</td><td>92.8</td><td>54.8</td><td>60.0</td><td>57.3</td><td>93.2</td><td>88.8</td><td>91.0</td><td>76.7</td><td>75.5</td><td>76.0</td><td>80.3</td><td>68.0</td><td>73.5</td></tr><tr><td>Opus - AWESOME ft</td><td></td><td>82.7</td><td>81.2</td><td>81.9</td><td>88.1</td><td>95.7</td><td>91.7</td><td>43.9</td><td>49.5</td><td>46.5</td><td>92.7</td><td>88.4</td><td>90.5</td><td>76.8</td><td>76.8</td><td>76.8</td><td><strong>80.4</strong></td><td>67.5</td><td>73.4</td></tr><tr><td>Opus - AWESOME pt+fr</td><td></td><td>83.8</td><td>81.2</td><td>82.4</td><td>88.8</td><td>96.5</td><td>92.5</td><td>49.0</td><td>56.6</td><td>52.5</td><td>93.2</td><td>88.8</td><td>91.0</td><td>78.3</td><td><strong>77.6</strong></td><td>78.0</td><td>79.2</td><td>65.4</td><td>71.6</td></tr><tr><td>Opus ft - FastAlign</td><td></td><td>84.6</td><td>80.6</td><td>82.5</td><td>79.0</td><td>88.2</td><td>83.3</td><td>61.5</td><td><strong>72.1</strong></td><td>66.3</td><td>91.8</td><td>88.4</td><td>90.1</td><td>77.3</td><td>74.7</td><td>76.0</td><td>75.4</td><td>68.6</td><td>71.8</td></tr><tr><td>Opus ft - AWESOME pt</td><td></td><td>83.2</td><td><strong>85.1</strong></td><td><strong>84.1</strong></td><td>89.9</td><td>96.9</td><td>93.2</td><td>51.5</td><td>58.2</td><td>54.6</td><td>94.2</td><td>88.8</td><td>91.4</td><td>74.8</td><td>76.3</td><td>75.5</td><td>79.4</td><td>70.3</td><td><strong>74.6</strong></td></tr><tr><td>Opus ft - AWESOME ft</td><td></td><td>84.4</td><td>81.8</td><td>83.0</td><td>89.9</td><td><strong>97.6</strong></td><td>93.6</td><td>45.4</td><td>51.1</td><td>48.1</td><td>93.2</td><td><strong>89.3</strong></td><td>91.2</td><td>74.3</td><td>76.6</td><td>75.4</td><td>75.6</td><td><strong>71.4</strong></td><td>73.4</td></tr><tr><td>Opus ft - AWESOME pt+fr</td><td></td><td>82.4</td><td>80.9</td><td>81.6</td><td>89.6</td><td><strong>97.6</strong></td><td>93.4</td><td>51.4</td><td>56.1</td><td>53.6</td><td><strong>94.6</strong></td><td><strong>89.3</strong></td><td><strong>91.9</strong></td><td>74.0</td><td>76.3</td><td>75.1</td><td>75.1</td><td>70.5</td><td>72.6</td></tr></tbody></table>
## G Examples from the different datasets

### EXAMPLES FROM N2C2

The patient's agitation was managed with nightly<sub>Drug</sub> **haldol**<sub>Drug</sub> with as needed<sub>Frequency</sub> **haldol**<sub>Drug</sub> as well.

Improvement in clinical status was noted overnight and his morphine<sub>Drug</sub> drip was discontinued.

- hold all antihypertensives<sub>Drug</sub>; plan to add back slowly at reduced doses and varying schedule - rule out MI - bolus<sub>Dosage</sub> NS<sub>Drug</sub> to maintain MAP > 60 with caution given ESRD and oliguric.

folic acid<sub>Drug</sub> 1 mg<sub>Strength</sub> Tablet<sub>Form</sub> Sig: One (1)<sub>Dosage</sub> Tablet<sub>Form</sub> PO DAILY (Daily)<sub>Frequency</sub>.

Iron<sub>Drug</sub> 50 mg<sub>Strength</sub> Tablet<sub>Form</sub> Sustained Release Sig: One (1)<sub>Dosage</sub> Tablet Sustained Release<sub>Form</sub> PO once a day<sub>Frequency</sub>.

### EXAMPLES FROM GERNERMED TEST SET

Das Eplerenon<sub>Drug</sub> ist wegen Ihrer Herzinsuffizienz. Da können wir jetzt auf 50 mg<sub>Strength</sub> p.o. 1-0-0 augmentieren.

*Eplerenon is for your heart failure. We can now augment to 50mg p.o. 1-0-0.*

Wegen der COPD-Exazerbation wurde Terbutalin<sub>Drug</sub> 0,25 mg<sub>Strength</sub> dem Patienten appliziert. Hierfür wurde der subkutane Weg gewählt.

*Because of the COPD exacerbation, terbutaline 0.25 mg was administered to the patient. The subcutaneous route was chosen for this.*

Bei Vorhofflimmern ist neben Betablockern<sub>Drug</sub> auch die Gabe von Magnesium<sub>Drug</sub> p.o. sinnvoll. Hierfür würden wir mit 300 mg<sub>Strength</sub> einmal täglich<sub>Frequency</sub> starten. Sofern möglich, ist eine Einnahme mittags<sub>Frequency</sub> (ca. 12 Uhr) zu bevorzugen.

*In atrial fibrillation, in addition to beta-blocks, the administration of magnesium p.o. is also useful. For this we would start with 300 mg once a day. If possible, it is preferable to take it at noon (around 12 o'clock).*

Zur Optimierung der Herzinsuffizienztherapie wurde die Dosis von Sacubitril / Valsartan<sub>Drug</sub> auf 97 / 103 mg<sub>Strength</sub> in Tablettenform<sub>Form</sub> mit Einnahme am Morgen und am Abend<sub>Frequency</sub> erweitert.

*To optimize heart failure therapy, the dose of sacubitril / valsartan was extended to 97 / 103 mg in tablet form with intake in the morning and evening.*

Bei bekannter koronarer Herzerkrankung sollte lebenslang<sub>Duration</sub> Acetylsalicylsäure<sub>Drug</sub> 100 mg<sub>Strength</sub> morgens täglich<sub>Frequency</sub> in oraler Applikation eingenommen werden.

*In cases of known coronary artery disease, acetylsalicylic acid 100mg should be taken orally daily in the morning as a lifelong treatment.*

### EXAMPLES FROM MEDNERF

TRAMADOL / PARACETAMOL<sub>Drug</sub> 37,5mg / 325mg<sub>Strength</sub>

TRAMADO/PARACETAMOL 37,5mg/325mg

AMLODIPINE<sub>Drug</sub> 5 mg<sub>Strength</sub> ; cpr<sub>Form</sub> 1<sub>Dosage</sub> comprimé<sub>Form</sub> matin<sub>Frequency</sub> 1<sub>Dosage</sub> comprimé<sub>Form</sub> soir<sub>Frequency</sub>

AMLODIPINE 5mg; tab 1 tablet in the mording 1 tablet in the evening

DOLIPRANETABS<sub>Drug</sub> 1000 MG<sub>Strength</sub> CPR PELL<sub>Form</sub> PLQ / 8 (Paracétamol<sub>Drug</sub> 1.000 mg<sub>Strength</sub> comprimé<sub>Form</sub>)

DOLIPRANETABS 1000mg TAB PLQ / 8 (Paracetamol 1,000mg tablet)

ACIDE ACETYLSALICYLIQUE<sub>Drug</sub> (sel de lysine<sub>Drug</sub>) 75 mg<sub>Strength</sub> **pdre p sol buv sach**<sub>Form</sub> (KARDEGIC<sub>Drug</sub>)

ACETYLSALICYLIC ACID (lysine salts) 75mg oral powder for suspension (KARDEGIC)

1<sub>Dosage</sub> sachet<sub>Form</sub> matin midi et soir<sub>Frequency</sub> si besoin

*1 packet in the morning, at noon, and in the evening, if needed*