EuroBERT: Scaling Multilingual Encoders for European Languages
Authors: Nicolas Boizard
Hippolyte Gisserot-Boukhlef
Duarte Miguel Alves
Andre Martins
Ayoub Hammal
Caio Corro
CELINE HUDELOT
Emmanuel Malherbe
Etienne Malaboeuf
Fanny Jourdan
Gabriel Hautreux
João Alves
Kevin El Haddad
Manuel Faysse
Maxime Peyrard
Nuno M Guerreiro
Patrick Fernandes
Ricardo Rei
Pierre Colombo
Venue: COLM 2025 (2025)
Keywords: Encoder
Multilingual
EuroBERT
Training
Vector Representations
Bidirectional
European
TL;DR: We introduce EuroBERT, a family of multilingual encoders leveraging recent architectural advances, achieving state-of-the-art performance across diverse tasks with support for sequences up to 8,192 tokens.

Abstract: General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

Published as a conference paper at COLM 2025
EuroBERT: Scaling Multilingual Encoders for European Languages
Nicolas Boizard 1,3 Hippolyte Gisserot-Boukhlef 2,3 Duarte M. Alves* 4,5
André Marting Ayoub Hammal Caio Corro Céline Hudelot Emmanuel Malherbe Etienne Malaboeuf Fanny Jourdan Gabriel Hautreux João Alves Kevin El-Haddad Manuel Faysse Maxime Peyrard Nuno M. Guerreiro Patrick Fermandes 4,5188 Ricardo Rei", Pierre Colombo
Diabolocom, 2 Artefact Research Center, MICC, CentraleSupélec, Université Paris-Saclay, *Instituto Superior Técnico & Universidade de Lisboa (Lisbon ELLIS Unit), 'Instituto de Telecomunicações, Unbabel, 7 Université Paris-Saclay, $CNRS, LISN, INSS Rennes, IRISA, "CINES, IRT Saint Exupéry, Illuii Technology Université Grenoble Alpes, Grenoble INP, LIG, Equul,, 17ISIA Lab, Carrnege Mellon University
unicolas.boizard, bwwiarddidddioo hipoold bobpaach.ppppoeeeeeerrooooooeeee gisserooo-buuheeeeeeeeerrrssssss fr, duarteemalveséteeniic. ulisboa.pp
Abstract
General-purpose multilingual vector representations, used in retrieval, regression, and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively support sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.
1 Introduction
Many important tasks in Natural Language Processing (NLP), including information re- trieval, classification, or regression, are built upon general-purpose vector representations. These representations are traditionally obtained from bidirectional encoder models, which aggregate information from the left and right contexts of each token (Devlin et al., 2019; Conneau et al., 2020; He et al., 2023). In contrast, recent advances in generative modeling have shifted the research community's attention towards uridirrectooal architectures (Bai et al., 2023; Llama Team, 2024; OLMo et al., 2025). Notably, these efforts have identified several key performance drivers that span architectural advances, data improvements, and increased scale. Yet, despite apparent barrier to transferring these insights to bidirectional architectures, little effort has been devoted towards this objective, forcing practitioners to depend on outdated models.
"Equal contribution. lhttps://huggiinffac.comm htt.://ccc..c//o/eEEEeeRe

---

Published as a conference paper at COLM 2025
MIRACL
XNLI
94
EuroBERT-610M EuroBERT-2.18
85
EuroBERT-2.1I
BUrOBBRRT610M
nDCG@10
92
mGTE-305M 305M
XLM-R-3.5B
A
Accuracy
XLM-R-3.5B
90
BUToBERT-210M
80
XLM-R-560M
XI.M-R-560M
mDeBERRa-220
Euro BERT--0M
88
mGTB-305M
86
75
X1MM-R-80M
XLM-R-280M
0.2
0.5
1
3
0.2
0.5
2
Model Size (Billions)
Model Size (Billions)
No
MathShepherd
80
EuroBERT-610M EuroBERT-2.1B
90
EuroBERT-610M EuroBERT-2.1B
EuroBERT-210M
DDCG@10
60
BUROBERT-
ModernBHKK--955M
Accuracy
80
Modern E. XLM-R-3.5B
ModemBEERT-3RM
ModemBERR-150M
XLM-R-3.5B
mGTH-305M
40
XLM-R-560M
mDeBERTT-280M
MGTE-305M
XLM-R-280M
70
XLM-R-280M
! XLMMRR560M
20
0.2
0.5
2
0.2
0.5
3
Model Size (Billions)
Model Size (Billions)
Figure 1: Pareto plots for multilingual tasks (top), showing retrieval performance on MIR- ACL and sentence classification on XNLI, and for math and code tasks (bottom), featuring NodeSearchNet and MathShepherd. The shaded regions indicate the Pareto frontiers.
In this paper, we introduce a refreshed recipe for training general-purpose multilingual encoders, resulting in the EuroBERT family. Drawing inspiration from recent progress in decoder models, our models feature an updated architecture ($2.1), and are trained on 5T-token multilingual dataset, covering widely spoken European and global languages, along with mathematics and code (62.2). We adopt a masked language modeling objective, and employ two-phase training pipeline, adjusting the data distribution in the second training phase to improve downstream performance (52.3).
We extensively evaluate the EuroBERT models, comparing with similarly sized alternatives across a suite of tasks representative of real-world encoder applications ($3). Our models match exceed the performance of alternative models, such XIMMRRoERTa (Conneau et al., 2020), mGTE-MLM (Zhang et al., 2024) and ModernBERT (Warner et al., 2024), on multilingual retrieval, classification and regression tasks, and outperform them on code and mathematics tasks (Figure 1).
In order to provide further insights into the methodologies involved in large-scale encoder training, we also examine the impact of our design choices through systematic ablations on several components of our annealing recipe (S4). We explore the choice of masking ratio, showing that while higher masking ratios benefit retrieval tasks, lower ratios improve sentence classification. Additionally, we highlight that including data for code and mathe- matics improves multilingual retrieval, but degrades classification accuracy. Contrary to expectations, we also observe that only selecting documents with high educational value degrades performance, which improves decoder LLMs (Penedo et al., 2024), and instead find that encoders benefit from a broader range of data sources.
Accompanying this work, release the EuroBERT family, comprising three models with 210M, 610M and 2.1B parameters. To facilitate future research, we also release intermediate training checkpoints, well our training framework.
2

---

Published as a conference paper at COLM 2025
2 EuroBERT: A Refreshed Multilingual Encoder
The EuroBERT models incorporate design choices similar to the Llama 3 architec- ture (Llama Team, 2024) (52.1). They are trained a large multilingual corpus, which also includes code and mathematics (62.2). Their training pipeline has two stages, pre-training and annealing, and employs the masked language modeling (MLM) objective ($2.3).
2.1 Architecture
The EuroBERT models are based on standard dense transformer (Vaswani et al., 2017), with several architectural changes. Similarly to Llama 2 (Touvron et al., 2023), we remove all biases. Additionally, we incorporate grouped query attention (Ainslie et al., 2023), swish gated linear units (Shazeer, 2020), root mean square layer normalization (Zhang & Sennrich, 2019), and rotary position embeddings (Su et al., 2024). However, unlike decoder models, we do not apply causal masking?
2.2 Dataset
To train EuroBERT, we construct a multilingual 5T-token corpus - 4.8T tokens for pre- training and 200B for annealing - which includes 15 languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, and Hindi. Following prior work on curriculum learning (Hu et al., 2024), we adjust the data distribution to emphasize higher-quality datasets during annealing.
Pre-training mixture. We use FineWeb (Penedo et al., 2024) for English, and Cul- turaX (Nguyen et al., 2024) for multilingual data. We also incorporate EuroLLM parallel data (Martins et al., 2024), which can improve cross-lingual transfer (Conneau & Lample, 2019; Reid & Artetxe, 2022; 2023), by concatenating to- to-English and trom-English translation pairs, separated by special <lparallel_sep) training on code for decoder models (Aryabumi et al., 2024), we add 38 programming languages from The Stack v2 (Lozhkov et al., 2024), and Proof-Pile- (Azerbayev et al., 2024) for mathematics, both of which we find improve multilingual retrieval (54).
token. Finally, inspired by the benefits of
Annealing mixture. We classified data not seen during pre-training into four quality levels according to educational value using the multilingual classifier from Martins et al. (2024).* 4 We then kept the documents above the third threshold which, contrary to our expectations, improved performance on downstream tasks. Additionally, we adjusted the data distribution based on multiple ablations. Specifically, we decreased English while proportionally increasing the remaining languages. We also decreased the amount of code and math while increasing parallel data (54).
2.3 Training Recipe
Masked language modeling. We pre-train EuroBERT models with a 50% masking ratio, following the insights from Wettig et al. (2023), who find that masking 15% and 30% of tokens is sub-optimal, and that larger models benefit from higher masking ratios. For the subsequent annealing, however, we lower the masking ratio to 10% based on downstream evaluations (54), aligning with the findings from Yang et al. (2023) and Ankner al. (2024).
We employed the Sarmup-Stable-Deccyy (WSD) scheduler (Shen et al., 2024), with linear warm-up phase of 2,000 steps, constant learning rate of 1 x 10-4
Hyperparameters.
WW provide architecture details in Appendix A.
'These languages were selected to balance European and widely spoken global languages, and ensure representation across diverse alphabets and language families. 4Similar to Penedo et al. (2024), the classifier groups documents into buckets based on their educational value, with higher numbers indicating higher quality.
We provide further details our pretraining and annealing datasets in Appendix C.
3

---

Published as a conference paper at COLM 2025
during pre-training, and cosine scheduler decaying to 0 during the annealing phase. During pre-training, we packed sentences to 2,048 tokens and used a Rotary Position Embedding (RoPE) value of 10,000. In the annealing phase, increased the RoPE theta to 250,000 and randomly cropped our training documents to lengths between 12 and 8,192 tokens. We adopted this approach because, due to pre-processing constraints, our training data had already been segmented into fixed-length documents, making standard variable-length training infeasible. Therefore, we introduced random cropping of these fixed-length
seqquee sequences as an approximation of variable-length training. We found that this approach outperforms training only on fixed lengths ($4), further highlighting the necessity for variable length documents during long context training (Gao et al., 2024).
Infrastructure. We trained EuroBERT using 92 MI250X GPUs for EUR 384 MI250X GPUs for EuroBERT-610M, and 96 MΣ300А GPUs for EuroBERT-2.11 for total of 200k GPU hours. Our training framework incorporates FlashAttention (Dao, 2023), fused cross-entropy from LigerKernel (Hsu et al., 2024), co (Ansel et al., 2024), and hybrid sharding with Fully Sharded Data Parallel (Zhao et al., 2023).
Evaluation
3.1 Evaluation Setup
Datasets and tasks. We select suite of tasks to cover various real-world use cases for encoders. For multilingual tasks, we evaluate retrieval using MIRACL (Zhang al., 2023), MLDR (Chen et al., 2024), WikipeeiaaRerievaa!5 and CC-News (de Gibert et al., 2024). We assess sentence classification with XNLI (Conneau et al., 2018), PAWS-X (Yang et al., 2019), AmazonReviews (Keung et al., 2020) and Massivelntent (Keung et al., 2020). Additionally, we evaluate token classification using the NER task from the XGLUE benchmark (Liang et al., 2020). We evaluate sequence regression on the WMT quality estimation task (Bojar et al., 2017; 2018; Barrault et al., 2019; 2020; Akhbardeh et al., 2021; Kocmi et al., 2022), and on summary evaluation using SeaHorse (Clark et al., 2023). For code-related tasks, we evaluate retrieval on NodeSearchNet (Husain et al., 2019) and DupStackMath (Hoogeveen et al., 2015), and classification on CodeDefect (Zhou et al., 2019) and CodeComplexity (Jeon et al., 2023). Finally, in the mathematical domain, we test retrieval on the MathFormula (Drechsel et al., 2025) task, and classification on MathShepherd (Wang et al., 2024b).?
Baselines. We compare with the multilingual XLM-RoBERTa (Conneau et al., 2020; Goyal et al., 2021), MGTE-MIM (Zhang et al., 2024)$ and mDeBERTa-v3 (He et al., 2023). For code and mathematics, we also compare with the English-only ModernBERT (Warner et al., 2024).
Fine-tuning. For each task, models are trained for 10,000 steps (unless otherwise specified) on the corresponding training split using a batch size of 32, 10% warm-up ratio, and linear learning rate decay. For small datasets requiring multiple epochs, we apply early stopping with a patience of one epoch based on validation performance. To account for model specificities, we fine-tune using 10 logarithmically spaced learning rates (1 x 10-5 to 1 x 10 +), selecting the one that achieves the highest validation metric.? 9 For sequence classification, we use the cross-entropy loss during training, while for sequence regression, we substitute it with mean squared error." 10 For token classification tasks, we use the token- level cross-entropy loss, assigning each sub-token in an entity to the corresponding entity
'https: htttt:://huggingfaaee fcce wtteeeeeeeettaaeeeeeeeeeeee http:///moeeed/////aeeeeee www.teeeeealleieeeeee Mul We detail the evaluation setup for each task in Appendix D.
8Since the EuroBERT models are ceneral-purpose encoders, we evaluate them against the pre- trained mGTE-MLM variant, which, similarly, was not optimized for retrieval tasks. "For the baselines, we set additional fine-tuning hyperparameters according the original paper. For EuroBERT models, we maintain the values from pre-training and annealing.
10On summarization (SeaHorse), we train for 5,000 steps reduce computational costs.
4

---

Published as a conference paper at COLM 2025
mDeBERTa 280M
mGTE 305M
XIM-RoBERTa
EuroBERT
Benchmark
280M
560M
3.5B
210M 610M
2.1B
Retrieval
(nDCG@10)
MIRACL
37.5
91.2 2
85.4
39.4
91.4
90.8
92.6 2
92.9
43.7
93.8
89.5
92.6
95.1
94.8
MLDR
18.3
67.8
54.6
60.8 4
65.9
65.4
68.6
66.1
20.0
73.2
58.7
4
70.0
73.4
72.9
CC-News
7 15.8
71.3
61.6
72.8 3
80.9
67.2
75.6
75.9
71.5
60.4
72.1 4
80.9
69.0
76.6
76.9
Wikipedia
57.6 5
94.1 2
91.0
93.1
3
96.3
94.4
95.9
95.8
58.9 5
94.6 3
91.7
93.6 3
96.7
95.6
96.6
96.6
Sequence Classification (Accuracy)
XNLI
79.5
75.8 5
74.1
81.7 2
83.7
76.6
81.9 3
84.1
78.4
76.6
86.1
79.9
86.8
PAWS-X
91.9
89.8 3
88.9
92.4 2
92.9
89.9
92.2
93.0
91.9
89.8
88.9
92.4 2
92.9
89.9
92.2
93.0
Aeazon
62.1
61.5 3
61.1
63.1
63.6
61.7
62.6
63.2
63.7
62.7
62.7
64.5
64.7
63.0
64.0 2
64.5
Massivelntent
86.5
86.9 2
86.3
88.2
87.9
86.5
87.2
87.5
87. 2
87.5
87.2
88.8
88.5
87.2 2
87.8
88.2
Token Classification (F1 Score)
NER
95.5
96.1
2
96.3
95.9 4
96.2
95.2
95.5
96.1
96.3
96.7
95.9 4
95.2
Sequence Regression (Spearman)
WMT (Ref-based)
45.7
43.9 5
43.0
45.3 4
47.7
45.2
46.0 3
47.3
46.5
44.0
43.1
48.5
45.1
48.5
WMT (Ref-free)
42.0
38.5 5
36.5
40.8 4
44.5
41.0
41.5 3
38.7
34.2
44.4
40.5
38.8
SeaHorse
61.1
65.5 4
67.5
66.0
3
60.3
59.2
56.9
61.4
63.3
60.1
62.7
64.0
Table 1: Results for multilingual tasks, with scores aggregating all languages shown above and scores aggregating European languages in italic below. Models are grouped into statistically significant clusters, with best ranked models highlighted in bold.
label." 11 For all retrieval tasks, we finetune for 1,000 steps on MS-MARCO (Bajaj et al., 2016) using the InfoNCE loss (Oord et al., 2018) with in-batch negatives and cosine similarity.
Evaluation metrics. We report accuracy for sequence classification, Spearman rank corre- lation for regression, F1 score for token classification, and nDCG@10 for retrieval tasks. We also follow Freitag et al. (2023), and group systems into language-spectine clusters based on statistically significant performance gaps at 95% confidence thresholds. We then compute system-level rankings using a normalized Borda count (Colombo et al., 2022), defined as the average over the obtained per-language clusters. Note that a first cluster will only exist if model significantly outperforms all others on a majority of languages.
3.2 Results
Table presents the aggregated results across multilingual tasks, and Table summarizes performance on code and mathematics benchmarks.. 13
11 At inference time, the final label of an entity is determined by majority vote over the sub-tokens. 12Since many retrieval datasets lack dedicated training splits, we use MS-MARCO, an English-only dataset. This choice also allows us to assess cross-lingual generalization.
We provide per-language results in Appendix F.
5

---

Published as a conference paper at COLM 2025
ModernBERT 150M 395M
mDeBERTa
mGTE 305M
XLM-RoBERTa
EuroBERT
Benchmark
280M
280M 560M
3.5B
210M 610M
2.1B
Code Retrieval (mDCG@10)
NodeSearchNet
53.9
65.8 3
2.8
34.07
23.0 8 40.8 6 54.1 5
58.9 4
69.9 2
72.6 1
DupStackMath
39.7 4
4
10.2 7
37.54
29.3 6 36.9 42.9
41.7 3
46.0
48.3
Code Classification (Accuracy)
CodeComplexity
86.1 3
88.6 3
73.9 5
74.5 5
74.1 5 83.6 4 84.3 4
91.9 2
94.2
1
95.2
CodeDefecc
65.8 3
67.0
64.7 3
63.5 4
61.9 4 54.3 5
65.8 3
69.5
69.0
67.7 2
Math Retrieval (nDCG@10)
MathFormula
89.65 91.9
85.27
83.4 8 83.1 8 81.4 89.1 6
91.53 92.6 1 91.0
Math Classification (Accuracy)
MathShepherd
77.7 4 83.6
75.1 5
77.2 4 71.9 6 67.6 7 82.5 3
84.0 2
87.3
86.8
Table 2: Results for code and mathematical tasks. Models are grouped into statistically significant clusters, with best ranked models highlighted in bold.
XLM-ReBERTa Fertility
XLM-RoBEKTa Fertility
XLM-RoBERTa Fertility
4-
2
4-
EuroBERR Fertility
14
5.9
13.7
21.6
1.4
5,3
11.6
18.5
1.1
4.8
11.0
17.7
EuroBERT
0.6
XLM RoBERTa
2
-5.9
-1.5
6.3
14.3
-5.2
-1.3
5.0
12.0
-5.5
-18
4.3
11.1
0.4
-8.6
3.6
11.6
-6.7
-2.8
3.5
10.4
-7.7
-.11
2.1
8.8
0,2
-17.6
-5.3
-10.9
-3.1
4+
BuroRERT-210M
EuroBERT-610M
BuroBERT-2.1B vs
Fertility Bucket
XLM-RRBBKTT--2II
XLM-RRBERTA-56OM
XLMM-RBBEKTe-3558
Figure 2: Difference in Fl Score between EuroBERT and XLM-RoBERTa by tokenizer fertil- ity (left plot) and fertility distribution for both tokenizers (right plot) on the NER dataset.
The EuroBERT family delivers strong performance across diverse domains and tasks. Our largest model, EuroBERT-2.19 1BB1B ranks first on 10 out of 18 tasks, competing closely with the larger KLM-RoBERTa--355 EuroBERT-610M is also on par with XLMM-RBERRia33.5 across several multilingual tasks while being five times smaller, and outperforms it on code and mathematics benchmarks. Likewise, EuroBERT-210M matches JLM-ROBERTa-56OM performance while having less than half the parameters, and consistently outperforms other models of similar size, showing especially strong results on European languages.
EuroBERT is effective at document ranking. Across domains, EuroBERT consistently ranks high for retrieval tasks. Notably, the 210M and 610M models outperform all alterna- tives of comparable sizes, and are competitive with the larger XLM--RBEERTa-35B..
EuroBERT models are on par with similarly sized models for sequence classification. On sequence classification, no model sigmificantly outperforms all others. During the development of EuroBERT, we found that several design decisions lead to trade-off between retrieval and classification capabilities (54). We highlight, however, that EuroBERT. 2.1B still among the highest ranking systems, and that the smaller models in the family are competitive with models of comparable size.
14For retrieval, increasing model size did not always lead to better results. Further analysis, in Appendix F, revealed that FuroBERT--211 benefits significantly from thorough grid search.

---

Published as a conference paper at COLM 2025
MLDR
Hease
80
74
70
70
nDCC
Spearman
60
66
50
62
40
58
91
92
93
94
95
Document I length Quantile (g)
Document Length Quantile (4)
XLM-RoBERIa-280M
XLM- BLM-RRBBRTT-560M
XLM-RoBERTa-335B
mGTE-305M
EuroBERT-1OM
EuroBERT-610M
EuroBERT-2.1B
Figure 3: Results by length of the positive documents for retrieval (MLDR) and input documents for summarization (SeaHorse).
There is room for improvement token classification. On the NER task, EuroBERT lags behind XLM-RRBERTa. However, we observed that models achieve comparable F1 scores when splitting entities into similar number of tokens, as shown in Figure In contrast, models perform significantly worse when segmenting entities into a larger number of tokens. We posit that token classification tasks may benefit from larger vocabularies, with lower fertility, such as the one used in the XLM-RoBERTa family. We also note, however, that increasing the vocabulary size also increases the number of parameters, particularly for smaller models." 15
EuroBERT can function an evaluation metric. EuroBERT models match or exceed the performance similarly sized systems in reference-based translation evaluation. For reference-free evaluation, while EuroBERT-2.11 lags behind the larger XLM-RoBERTa, the 210M and 610M variants are competitive with other baselines. In the future, we will explore other training signals to further enhance EuroBERT's cross- loss-lingual capabilities. For summary evaluation, EuroBERT models consistently outperform similarly sized alternatives.
EuroBERT maintains performance at longer context lengths. Figure 3 compares the long context performance of EuroBERT and XLM-RoBERTa. On both retrieval and summary evaluation, EuroBERT maintains performance at longer contexts, whereas XLM-RoBERTa suffers notable degradation.
The EuroBERT family performs strongly in tasks related to code and mathematics. On these tasks in the code and math domain, all EuroBERT models consistently surpass other systems. Notably, EuroBERT-210M reflects most of the performance of the larger models in the family, and ranks above all baselines, highlighting its capabilities at a smaller scale.
Training Recipe Analysis
We measure the impact of various design decisions made during the development of EuroBERT with extensive ablations. Following Blakeney et al. (2024) and Llama Team (2024), we perform multiple annealing runs on 40B tokens, each varying a different component of our recipe, and measure the performance on the XNLI and MIRACL validation sets, the former representing multilingual classification and the latter multilingual retrieval." 16
For example, doubling the vocabulary size of FuroBERT-210M would add 100M parameters to the model embeddings. 16We follow the procedure from $3, but instead evaluate on the validation splits considering only European languages.
7

---

Published as a conference paper at COLM 2025
1.0
1.0
1.0
1.0
1.0
Metric Delta
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
-0.5
-0.5
-0.5
-0.5
-0.5
-1.0
-1.0
-1.0
-1.0
-1.0
41%
26%
17%
8%
4%
2%
876
6%
2%
5%
8%
1%
0%
English
Math
Code
Parallel
IFT
XNLI
MIRACL
Figure 4: Impact of changing data subset ratios during annealing. The first vertical axis in each subplot denotes the reference data mix from Table 6.
2
2
Metric Delta
3
1
-6
-2
-2
2k
[12,
2k]
[12, 8k]
50%
30%
10%
3
3+4
Sentence Length
Mask Ratio
Quality Bucket
XNLI
MIRACL
Figure Impact of hyperparameter choices during annealing. The first vertical axis in each subplot denotes the reference data mix from Table 6.
Balancing the language distribution enhances performance. The left-most plot in Figure reports retrieval and classification performance as the proportion of English is reduced and re-distributed between other languages. We observe that more balanced distribution improves overall results. However, when the language distribution becomes too close to uniform, there is a degradation in performance.
Including math and code improves multilingual retrieval, but degrades multilingual clas- sification. The second and third plots Figure show MIRACL performance dropping and XNLI accuracy rising as the quantity of math and code data decreases. In future work, we will investigate how to better balance downstream task performance during pre-training.
Increasing parallel data improves multilingual classification and retrieval. The forth plot in Figure 4 presents XNLI and MIRACL performance when increasing parallel data. In line with recent work showing the benefits of pre-training with parallel data (Anil al., 2023; Briakou et al., 2023; Alves et al., 2024), we find it improves both benchmarks.
Adding instruction fine-tuning data degrades model performance. The right-most plot in Figure 4 analyses the impact of adding instructions during annealing, which can improve performance for decoder language models (Wei et al., 2022; Chung et al., 2024). In contrast to decoders, it leads to worse performance when training an encoder model.
Varying sentence length improves performance. The first plot in Figure 5 examines the impact of variable sentence lengths during annealing. Compared to the fixed packed sentence lengths employed in pretraining, variable sentence lengths significantly boosts XNLI and moderately MIRACL performance! 17 This improvement remains stable, without degradation when the maximum context length extended 8,192 tokens.
We hypothesize this gap stems from the prevalence of shorter sequences in the XNLI dataset.
8

---

Published as a conference paper at COLM 2025
XNLI
MIRACL
Data Quantity (%)
50
25
25
2
4
2
Quality Bucket
Quality Bucket
Premise
Hypothesis
Anchor
Positive
Figure 6: Quality buckets for XNLI and MIRACL English training subset.
A reduced masking ratio during annealing enhances classification accuracy. Similarly to previous research advocating lower masking ratio later training (Yang et al., 2023; Ankner et al., 2024), we also find that reducing it to 10% during the annealing phase improves EuroBERT's accuracy on XNL.,, though it leads decline in MIRACL scores.
Filtering data based on educational value can degrade results. Contrary to initial expec- tations, using the highest quality data bucket during annealing does not result in better performance XNLI and MIRACL. Instead, illustrated in the right-most plot of Figure 5, mixing the buckets with quality levels 3 and 4 leads to the best overall results. Curiously, inspecting the evaluation splits of these datasets revealed that our quality filter would discard nearly all examples in XNLI, and many in MIRACL, as shown Figure 6. This result highlights a potential domain mismatch, wherein our training data deviates from the distribution of downstream tasks. Indeed, while educational value fits assistant-like tasks typically delegated LLMs, broader coverage of textual data, reflected in mixing both quality buckets, may be more inline with general-purpose vector representations. In future work, we would like to explore quality filters that are better tailored to encoders.
Final annealing configuration. The previous results revealed several design choices that trade off classification and retrieval performance. the final data mix, we aimed balance these two tasks. Based on the previous analysis, we created our final annealing dataset by selecting data above the third threshold. We reduced the proportion of English to 26% while proportionally increasing the share of the remaining languages. We allocated 6% and 4% of the data mix to math and code, respectively. Additionally, we increased the proportion of parallel data to 6%, and removed instructions. We finally lowered the masking ratio to 10% and performed annealing with random sentence lengths of up to 8,192 tokens.
5 Related Work
Encoder models have shown strong performance in non-generative tasks, such as classifica- tion and retrieval (Devlin et al., 2019; Liu et al., 2019; He et al., 2023; Acheampong et al., 2021; Ma et al., 2019; Karpukhin et al., 2020; Wang et al., 2024a). Variants of these models have also extended support to multiple languages and cross-lingual tasks (Conneau et al., 2020). However, scaling to many languages introduces the "curse of multilinguality" (Conneau et al., 2020; Chang et al., 2024), where interference across languages degrades performance. Notably, increasing model capacity has been shown to mitigate this effect (Conneau al., 2020), motivating our focus on scale.
Encoders are typically trained with masked language modeling (MLM) (Devlin et al., 2019). While alternatives like replaced token detection (He et al., 2023) exist, we adopt the MLM objective because initial evaluations of existing models showed more balanced results across tasks. Building upon the effectiveness of higher masking ratios (Wettig et al., 2023), mask 50% of the training tokens during pre-training. Prior work has also shown the benefits of decreasing the masking ratio in later phases of training (Yang et al., 2023; Ankner et al., 2024),

---

Published as a conference paper at COLM 2025
also lower masking ratio to 10% during annealing. Interestingly, demonstrate that this choice improves classification accuracy, but reduces retrieval quality.
Recent concurrent work, such as ModernBERT (Warner et al., 2024) and mGTE (Zhang et al., 2024), also revisits encoders light of advances in decoder models. Similar to our approach, they incorporate grouped query attention (Ainslie et al., 2023), rotary positional embeddings (Su et al., 2024), gated normalization (Zhang & Sennrich, 2019), and support for longer context windows. However, we additionally draw inspiration from Llama Team (2024); Yang et al. (2024) by including code and mathematical data during pre-training, which we show improves retrieval quality.
units (Shazeer, 2020), root mean square layer
Conclusion
We propose a recipe for training general-purposee multilingual encoders, creating the Eu- roBERT family. We incorporate recent architectural advances from decoder models, and train on a multilingual dataset containing European and globally spoken languages, to- gether with code and mathematics. Our models outperform existing alternatives comprehensive suite of tasks covering multilingual capabilities, mathematics and code. We also extensively analyze the design decisions behind EuroBERT's dataset and training pipeline. Alongside this paper, we release all models in the EuroBERT family, including intermediate training checkpoints, and our training framework to facilitate future research.
Acknowledgments
We sincerely thank the ADASTRA supercomputer (CINES) for its technical support and high--peformance computing (HPC) resources, provided through grants C1615122 and GDA2401. We also appreciate the support of the French government through the France 2030 program as part of the ArGiMi project. This work was also supported by the EU's Hori- zon Europe Research and Innovation Actions (UTTER, contract 101070631), by the project DECOLLAGE (ERC-2022-CoG 101088763) by the Portuguese Recovery and Resilience Plan through project C645008882-0000000 (Center for Responsible AI), and by FCT/ /EECT through national funds and when applicable co-funded EU funds under UID/ /10550008 In- stituto de Telecomunicações. Duarte was also partially supported by the DatalA Institute, whose contributions facilitated the completion of this work.
10

---

Published as a conference paper at COLM 2025
References
Francisca Adoma Acheampong, Henry Nunoo-Mensah, and Wenyu Chen. Transformer models for text-based emotion detection: a review of bert-based approaches. Artificial Intelligence Review, 54(8), 2021. URL https: :/dl acm. wwwg//o//bb///////oooo 10071/00402000000 09958-2.
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints, In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, December 2023. Association for Computational Linguistics. URL https://aclanthollogy wrg..023..o./
2023. .mnll-mminn 298/.
Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Ra- jen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonge An- gela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Samponsah-Kaayyiinn Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 confer- ence on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, Online, November 2021. Association for Computational Linguistics. URL nttps://aclantholt
org/2021 1.1-1.1/.
Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peteers Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. Tower: An open multilingual large language model for translation-rrelated tasks. In First Conference on Language Modeling, 2024. URL
wttp.//.rggbbbb2220... 17733.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexan- dre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Jbbreg,,Junhher, Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report. Xiv preprint ar Xit:23005110433 2023. URL https://arxiv www.aiiww..ogaasseee.oee wtw.:/gg////ba/ee/oo 10403.
Jacob
Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, and Matthew Leavitt. Dynamic masking rate schedules for MLM pretraining. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL https:/ pttps://aclanthol/ org/2024. eacl wwcl-sshrt.... short. 42/.
11

---

Published as a conference paper at COLM 2025
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voz- nesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph com- pilation. In Proceedings of the 29th ACM International Conference on Architectural Sup- port for Programmning Languages and Operating Systems, Volume 2, 2024. URL https: //dl. org//oi/abs//1 / 1100..115///0 1145/3620665.1... 3640366.
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training. CoRR, abs/2408.10914, 2024. URL https://doi.or 10. 10.48550/arXiv 2408. 10914.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2024. URL
net/forummid net//oorummddddiiiRR995155
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL
wrtp.////.///......///////// 2389. 16609.
Payal Bajai, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint wrXi::611.00266. 2016. ÜRL httptttts////ii..ooo http:/wwww.w.pb///////////////// 092688.
Loic Barrault, Ondłej Bojar, Marta R. Costa-jussä, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, August 2019, Association for Computational Linguistics. URL httts::/aclaatthtt
1rrg/W19-53011
Loic Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jusså, Christian Fed- ermann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2020 conference on machine translation (WMT20). In Proceed- ings of the Fifth Conference on Machine Translation, Online, November 2020. Association for Computational Linguistics. URL httpp:///aclanthologyy hwt.tthhwwlgg.ooggo.oooooo 2020. 1.t-1.1/.
Cody Blakeney, Mansheej Paul, Brett W Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training. In First Conference on Language Modeling, 2024.
Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shu- jian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. URL https: htatt::s//aclaahhooogg.. htwpc//nnnntoooooooog/1111111111 org/N17-47177/
Ondfej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Belgium, Brussels, October 2018. Association for Computational Linguistics. org/ /W1 /t////111111/..1
URL
12

---

Published as a conference paper at COLM 2025
Eleftheria Briakou, Colin Cherry, and George Foster. Searching for needles in haystack: On the role of incidental bilingualism in PaLM's translation capability. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, July 2023. Association for Computational Linguistics. URL httpps://ccee http:////aaaattooogyyy.. org/2023 acl-lcng lww..on...... 524/
Tyler A. Chang, Catherine Amett, Zhuowen Tu, and Ben Bergen. When multilingual- ity curse? language modeling for 250 high- and low-resource languages. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Mi- ami, Florida, USA, November 2024. Association for Computational Linguistics. URL https://aclanthol..gg org/2024, emnlp-main 236/.
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3- embedding: Multi-linguality, nulti-functionality, -uaaiaggry text embeddings through self-knowledge distillation. In Findings of the Association for Computational Lin- guistics: ACL 2024, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL htttp:://aclantiiitt orggggy..or//// www.orgg///2222 2024. findings-acl. acl. 137/.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Zuuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetueed language models. J. Mach. Learn. Res., 25:70:1 2024. URL https://jmlr...
org/ httg//appes//////// -870. html.
Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur Parikh. SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, Singapore, December 2023. Association for Computational Linguistics. URL https: //aclanthology org org wtwp:/222.eem.ppmm..m wwn.p-mminn....... 584.
Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stephan Clémençon. What are the best systems? new perspectives on nlp benchmarking. In Ad- vances in Neural Information Processing Systems, volume 35. Curran Associates, Inc., 2022. URL http:://proccedinss.eerrte newrips.e.ppp.pp wwtps/pccppppppeeeeeeeeeeee ac4920f 0r13:66666133311119999999-----00 coneeeeeeeeeee-eeeeeneneeeeeee pdf.
Alexis Conneau and Guillaume Lample.
Cross-lingual language model pretrain- ing. In Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL htpps///rroco wwwpeeeiiiinnnnnnniineeeeee neurips. ccc/ppper.illes/ppprr///// htte::occ.2cc2277700f57747722222 Paper. pdf.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, Brussels, Belgium, October-Novembee 2018. Association for Computational Linguistics. URL https:tt:/// httph:ttp///clllae htcpasshooocllaooooooooooo wrw.eay..rrr1111// org/
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross- ronss-inggul representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics. URL httphttss//acaaathlllleeeoo main. 747n.
org/2020.00 acl-
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Tweifth International Conference on Learning Representations, 2023. URL https:// arxiv. /bg/2 08691.
13

---

Published as a conference paper at COLM 2025
Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaox- iong Ji, Jaume Zaragoza-Bemaaeu, Mikko Aulamo, Gema Ramírez- Saniree-Sänchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. A new massive mul- tilingual dataset for high-performance language technologies. In Proceedings of the 2024 Joint International Conference Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia, May 2024, ELRA and ICCL. URL https://aclantholooyy org/ 1rec-main. 100/.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. URL / httt::://accanttollooo org/ N19-1423/.
Jonathan Drechsel, Anja Reusch, and Steffen Herbold. MAMUT: A novel framework for modifying mathematical formulas for the generation of specialized datasets for language model training, 2025. URL https: org/ httgg/bb/25550 20855.
Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. Results of wmn23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Confer- ence on Machine Translation, Singapore, December 2023, Association for Computational Linguistics. URL https://aclanthhl.ogr org/2023. wm1-1.51.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660, 2024. URL https:// arxiv.orggabss/41100 02660000000
Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. Larger-scale transformers for multilingual masked language modeling. In Proceedings of the 6th Work- shop Representation Learning for NLP (Re2L4NLP-2021), Online, August 2021. Association for Computational Linguistics. URL
ottp://aaaaatoooooooo///21111111 rep14n1p111
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentanglee embedding sharing. In The Eleventh International Conference on Learning Representations, 2023. URL https://arxe org/ /
19543.
Doris Hoogeveen, Karin M Verspoor, and Timothy Baldwin. Cqadup- stack: A benchmark data set for community question-answering re- search. In Proceedings of the 20th Australasian document computing sympo- sium, 2015. URL https: //ll//dl//// acm. org/doi/ www.--ttkhh-JalAAAA... AAA.://AAAAAAAAAASSOOOOOOOAOAE
htrg/doiaabs/1// abs/10/ss1 1145/2838931 28189347casa ZXWp://WWWSSSSYVW//////PP/PMnnM 00t.://0000000..0..22eeaaa000000
Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for Ilm training, 2024. URL / http://axii..oor////////////
org/ 10989.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference Language Modeling, 2024. URL https://
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arv preprint ar XXK::1999.09436, 2019. URL https: /ttp///axxvvv..v wwt.vorr/abs///// 909. 09436.
14

---

Published as a conference paper at COLM 2025
Mingi Jeon, Seung-yeop Baik, Joonghyuk Hahn, Yo-Sub Han, and Sang-Ki Ko. Deep learning-based source code complexity prediction. openreview, 2023. URL https:// openreview. wtt.:///oormmmmmdiirrk/////
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question an- swering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. org/ 2020. emnlp-main. 550/.
URL https://aclantthllll
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The multilingual Amazon reviews corpus. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. URL https:
emnlp-main. 369//
Tom Kocmi, Rachel Bawden, Ondlej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. Findings of the 2022 confer- ence on machine translation (WMT22)). In Proceedings of the Seventh Conference on Machine Translation (WMT), Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Associa- tion for Computational Linguistics. httpst
/ htttt:::///cllathoooogo.o www.///g.....e//eg// wmt-1. 1.1.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. URL https://aclantholoog.coo org./// smnlp-maing 484/.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https:///rxiii http://////xxivoo/aa//////// 11691
AI @ Meta Llama Team. The llama 3 herd of models, 2024. URL https:: https:////xxvvoraoee 2407.21783.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint arXio:2402.19113, 2024. URL https://arxia www.oogg/////.ooo ab/p//bs2222.... 19173.
Xiaofei Ma, Peng Xu, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. Domain adap- tation with BERT-based domain classification and data selection. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), Hong Kong, China, November 2019. Association for Computational Linguistics. URL https://aclanttoooogy.yo http://aaaanttooooooor///------
Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse Mateusz Kli- maszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. Eurollm: Multilingual language models for europe, 2024. URL https://arxiv. wwtprrxiv.orggbb/.../e/ org. 16235.
Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia, May 2024. ELRA and ICCL. URL https://aaclanthologyy.o
Irec-main 377/.
15

---

Published as a conference paper at COLM 2025
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Ji- acheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitzz Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2025. URL https://arxiv. https///arxivorrg/bb/225555 20055.001..6....
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arviv preprint arXio:1807.03748 2018. URL https://arxiv org/abs/ 1807 03748.
Guilherme Penedo, Hynek Kydlfček, Loubna Ben allal, Anton Lozhkov, Mar- garet Mitchell, Colin A Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Ad- vances in Neural Information Processing Systems, volume 37. Curran Associates, Inc., 2024. URL /tttoeed//proccedinggss www.nne.riisi....oooo...... hwtppi/ssssnrriiipeppppreerrreee httprrpppp..pppppppeel/////////p fitpr//l/l//ppe///////iill 370df (atodde1f8ff922999 51bda- Sattt--aaaa---aaatssaao htt.aaasts...aoeeeenncrssss...a. pdf.
Machel Reid and Mikel Artetxe. PARADISE: Exploiting parallel data for multilingual sequence-to-sequencc pretraining In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, Seattle,, United States, July 2022. Association for Computational Linguistics. URL https: // aclanthology org/ 2022. nwa.l-aallmnnnnmm 58/.
Machel Reid and Mikel Artetxe. On the role of parallel data in cross-lingual transfer learning, In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 2023, Association for Computational Linguistics. URL / htttps:/acclatthlloo org/ 2023. findings-ache acl. 372/.
Noam Shazeer. Glu variants improve transformer, 2020. URL https://arxiv.org//att
Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D. Cox, and Rameswar Panda. Power scheduler: A batch size and token number agnostic learning rate scheduler, 2024. URL https: //arxiv.org/ abss//0s/222..... 2ro.v...rg/ab//2408.. 13359.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Ro- former: Enhanced transformer with rotary position embedding. Neurocomputingg 568, 2024. ISSN 0925-2312. URL
5092523122300
Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-inddependen named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002), 2002. URL https: / www..ccwwet. org wrgganthologg//W2.. 2024.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CONLLL-2033 shared task: Language-independen named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL https://ww..r aclweb. org/ //tthologyy/003-04199
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew
16

---

net/ Published as a conference paper at COLM 2025
Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL
09288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural In- formation Processing Systems, 2017. URL https://proceedingss www.igs..nniss... ccccppppr_fllsss / hat.se/eeeeeee10oo55ccccccchaaaa
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by supervid contrastive pre-training, 03533.
2024a. URL https: htttpp://axiv..../
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, August 2024b. Association for Computational Linguistics. URL ptttps://aclanthhon orr.//goooooo//oeooeeee 2024. acl- long 510/.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Jermyy Howard, and lacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 13663.
2024. URL https: www.arwwwww../abaooooaaooe
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL forum?id -rrum?dd-gE2rGGozdqr
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. URL https/////caanttooooooorrr wttp://///2...al-meeeee. eacl- 2177.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), New Orleans, Louisiana, June 2018. Association for Computational Linguistics. URL htttt:://acaaathollog.oooo hwwphhooogg.ooooo//Moo org/N1 8 1101/.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https: / /ttpx//xxvvvv////a//eee///aa org/abs/2407 10671.
Dongjie Yang, Zhuosheng Zhang, and Hai Zhao. Learning better masking for better lan- guage model pre-training. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, July 2023, Association for Computational Linguistics. URL https: / wwwpaaooonnntoooo..coooooooo.aae or org/2023..o. acl- 400/.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversar- ial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical
17

---

Published as a conference paper at COLM 2025
Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, November 2019. Association for Computational Linguistics. URL httpss///cdaanttooooooor///////
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Ad- vaices in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedins..errr 2ea19498822203333344717775570002
newpeeeidddnnnnnnnissss..e.. newrips. wwtp:ccppppppppeeeeee fiapeerrlllessppp///211//////
pdf.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang, mGTE: Generalized long-conters text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing: Industry Track, Miami, Florida, US, November 2024. Association for Computa- tional Linguistics. URL nttts::/acllatthol/l wwwp:g...org////0 2024. emnlp-industry 103/.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11, 2023. URL https: http:ttt////cllathhhoooyy wtw.aanntoooooooooo//22222/.. www.oorg..........o tacl-1 63/.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: Experiences scaling fully sharded data parallel. Proceedings of the VLDB Endowment, 16(12), 2023. URL https://dl.ae acm. org/ /o/o/i/// /esi/abs//ae /10. 14778/ 361 1540. 3611569.
acm.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural information Processing Systems, volume 32. Curran Associates, Inc., 2019, URL httpss: inu.rrcrediinnnsnuiii........... httprppsccppppefffeeeeeeeeeeeeee files/paper/2019/ file//0656--cccreee3335444443333 .ddf.
18

---

Published as a conference paper at COLM 2025
A
EuroBERT Model Architecture
Table reports the architectural details of the EuroBERT model family.
Model Size
210M
610M
2.1B
Layers
12
26
32
Embedding Dimension
768
1,152
2,304
FFN Dimension
3,072
4,096
6,144
Attention Heads
12
18
18
/ey/Value Heads
12
6
Layer Normalization
RMSNorm
RMSNorm
x 10-5
Activation Function
SwiGLU
Vocabulary Size
128,000
Positional Embeddings
RoPE
RoPE 0
250,000
Tokenizer
LLaMA
Table 3: Summary of architectural hyperparameters for EuroBERT models of different sizes.
B Training Details
We trained the EuroBERT family utilizing 92 MI250X GPUs for EuroBERT-210M over 15k hours, 384 MI250X GPUs for EuroBERT-610M over 92k hours, and 96 MI300A GPUs for EuroBERT-221B over 106k hours, hyperparameter selections are detailed Table 4. We find this training recipe highly stable, with no loss spikes or need for intervention to address model training divergence (Figure 7).
10
Train Loss (Cross Entropy)
3 10
10
10a
100000
200000
300000
400000
500000
Global Step
EuroBERT-2100
EuroBERT 610M
EuroBERT-2.18
Figure 7: Pre-training Loss for all EuroBERT models on a logarithmic scale.
19

---

Published as a conference paper at COLM 2025
Parameter
210M
610M
2.1B
Pre-training
LR
1c-4
LR Scheduler
WSD
Warmup Steps
2,000
Context Length
2,048
Weight Initialisation
N(H 0, 02 0.2)
Annealing
LR
1e-4 to Cosine 8,192
Scheduler
Context Length
Optimizer
Optimizer
AdamW
Betal
0.9
Beta2
0.95
Epsilon (eps)
1e-5
Weight Decay
0.1
Clip Grad Norm
1.0
Training Setup
Per-GPU Batch Size
24
12
10
Gradient Accumulation Steps
1
GPUs
192
384
96
Tokens/Step
9,437,184
9,437,184
9,830,400
Table 4: Training hyperparameters for EuroBERT models (210M, 610M, 2.1B). The optimizer and Tokens/Step remain consistent across both pre-training and annealing phases.
20

---

Published as a conference paper at COLM 2025
C
Data Mix
Table details the data sources used throughout training, as well as the number of tokens used from each of the data source during pre-training. Table specifies the different annealing mixes used for ablations.
Source
Tokens (M)
(%)
Source
Subset
Tokens (M)
(%)
FineWeb
English
2. 002, 327
41.34
The-Stack v2
JavaScript
38, 440
1.21
CulturaX
French
295,
6.09
The-Stack v2
PHP
25, 620
CulturaX
German
291,514
6.02
The-Stack v2
C#
24, 842
0.51
CulturaX
Spanish
290,
6.00
The-Stack v2
Python
21,521
CulturaX
Chinese
238, 467
4,92
The-Stack v2
Java
20, 950
0.43
CulturaX
Italian
120,
2,48
The-Stack v2
766
CulturaX
Russian
116, 797
2.41
The-Stack v2
TypeSeript
11,307
0.23
CulturaX
Portuguese
112,321 321
2.32
The-Stack v2
HTML
7, 7,962
CulturaX
Japanese
112, 242
2.32
The-Stack v2
Lua
7,733
0.16
CulturaX
Polish
111,659
The-Stack
Ruby
5,524
CulturaX
Turkish
53, 126
1.10
The-Stack v2
Vue
5,411
0.11
CulturaX
The-Stack R
5,267
CulturaX
Vietnamese
50,661
1.05
The-Stack v2
Shell
4,793
0.10
CulturaX
Dutch
1.05
The-Stack
Swift
3,766
CulturaX
Hindi
25,544
0,53
The-Stack v2
reStructuredTTet
3,761
0,08
M Parallel
es
1.05
The-Stack
JSON
3,586
EL
Parallel
fr 6+ en
44,
0.93
The-Stack v2
Rust
3,152
0.07
EuroLLM
Parallel
en it ++ en
30,541
The-Stack
YAML
EuroLLM
Parallel
18, 702
0.39
The-Stack v2
Dart
2,678
0.06
EuroLLM
Parallel
13,808 808
0.29
The-Stack
RMarkdown
2,058
Eu
Parallel al ++ en
12, 666
0.26
The-Stack v2
HCL
1,423
0,03
EuroLLM
Parallel pl
7,280
0.13
The-Stack
PowerShell
1,027
Eu Parallel at en
6,41 414
0.13
The-Stack v2
VBA
1,027
0,02
EuroLMM Parallel zh
6,20
0.13
The-Stack
AsciiDoc
970
M Parallel cs en
5,458
0.11
The-Stack v2
Groovy
540
0.01
EuroLMM Parallel
4,599
0.09
The-Stack v2
CUDA
406
M Parallel
vi 69 en
3,395
0.07
The-Stack v2
Dockerfile
281
0.01
EuroLMM Parallel
2,975
0.06
The-Stack v2
Cython
103
E Parallel
ja 49 en
2,687
0.06
The-Stack v2
COBOL
96
0.01
E Parallel
hi
1,136
0.02
The-Stack v2
GraphQI.
0.01
Proof-pile-2
Arxiv
121,503
2.51
The-Stack v2
HTTP
0.01
Proof-pile-2
Matt-Web-Math
54,
1.12
The-Stack v2
ABAP
71
0.01
Proof-pile-2
Aigebraic-stack
38,985
0.74
The-Stack v2
RDoc
0.01
The The-Stack v2
C++
085
2.48
The-Stack v2
Metal
8
0.01
The-Stack v2
SQL
75,348
1.56
The-Stack v2
AppleScript
0.01
The-Stack v2
C
59,404
1.23
Total
4, 4,843,357
100
Table 5: Pre-training data, with total of 4.8 trillion tokens (as measured by EuroBERT's tokenizer). We report the list of all dataset names and subsets used, including the number of tokens selected and their proportion in the final data mix.
Mix
cn
de
it
ja
pl
m
es
ar
zh
Code
Math
Parallel
IFT
Reference
5.7
0.3
1.5
1.0
1.0
0.4
4.7
8.7
1.2
English 26%
26.0
6.0
6.0
4.0
4.0
4.0
4.0
4.0
4,0
4.0
6.0
4.0
6.0
4.0
4.0
4.0
5.0
1.0
English
5.0
Math 4%
46.3
5.8
5.7
1.0
0.3
1.5
0.8
1.0
1.4
1.0
3.7
0.4
4.7
1.0
8.7
4.0
5.2
1.2
Math 2%
5.7
0.3
1.5
1.0
1.0
0.4
4.7
8.7
2.0
1.2
Code 8%
46.3
5.8
5.7
1.0
0.3
1.5
0.8
1.0
1.0
5.7
0.4
4.7
6.0
8.2
1.2
Code 4%
46.3
5,8
5.7
1.0
0.3
1.5
0.8
1.0
1.4
1.0
3.7
0.4
4.7
1.0
4.0
8.2
5.2
1.2
Code 2%
46.3
5.8
5.7
1.0
0.3
1.5
1.0
1.0
5.7
0.4
4.7
2.0
8.2
5.2
1.2
Parallel 8%
46.3
5.8
5.7
1.0
0.3
1.5
0.8
1.0
1.4
1.0
3.7
0.4
4.7
1.0
8.7
8.2
8.0
1.2
IFT 0%
46.3
5.7
0.3
1.5
0.8
1.
1.0
0.4
4.7
8.7
8.2
0.0
Table 6: Data mix employed in the ablation study measuring the importance of different data subsets in the EuroBERT annealing phase.
21

---

Information Retrievul Published as a conference paper at COLM 2025
D Details of Evaluation Datasets
This appendix offers additional details on the datasets used for evaluation. Table 7 presents the language coverage of all evaluation datasets, and below are additional specifications the evaluated tasks.
European Languages
Extra-European Languages es it nl pl pt ar hi ja ru vi
Code
Math
Task
en
de
MIRACI
/
MLDR
/
l
Wikipedia
CC-News
CodeSearchNet
DupStackMath
MathFormula
Sequence Classification
XNLI
PAWS-X
Revzon
Massivelntent
Dodelefect
CodeComplexity
MathShepherd
Sequence Regression
WMT
SeaHorse
/
Token Classification
NER
Table 7: Language coverage across evaluation datasets.
Retrieval datasets:
MS-MARCO (Bajaj et al., 2016) - English-only retrieval dataset used for fine-tuning, where each anchor-positive pair includes mined hard negative, forming triplet structure.
MIRACL (Zhang et al., 2023) - Multilingual retrieval dataset. We use the semi- supervised version with labeled positive pairs provided by CentenceTransformmet as the primary data source. Anchors serve as queries, and the corpus consists of all positive documents in the dataset. Since only single data split is available, we create validation and test sets by partitioning 50% the original split for each, using queries the split key to ensure no data leakage.
MLDR (Chen et al., 2024) - Long-conted multilingual retrieval dataset. As with the triplet version provided by Sentence Transfommers and apply the same validation-test split strategy.
MIRACL,
Wikipedia 19 - Multilingual information retrieval dataset. Since only single data split is available, we partition 50% of the queries into validation and test sets.
CC-News (de Gibert et al., 2024) - Highly multilingual retrieval dataset. As with MIRACL, we the Sentence Transformers dataset version as the primary data source and apply the same test-validation split method.
NodeSearchNet (Husain et al., 2019) Code retrieval dataset with comment-code query-positive pairs (SentenceTransformmess verssio) processed similarly to the previous datasets.
http:///wwg.eoooooooooo//o////ee 844467286//3551114447755
22

---

Published as a conference paper at COLM 2025
DupStackMath (Hoogeveen et al., 2013) Code retrieval dataset with queries, corpus, and relevant documents, processed the same way as the above catasets.
MathFormula (Drechsel et al., 2025) - Mathematical retrieval dataset consisting of pairs of equivalent formulas. The original dataset contains formula pairs labeled as true or false based on their equivalence, spanning 71 well-known mathematical formulas. To construct the retrieval dataset, extract only equivalent formula pairs, retaining positive instances. Due to the dataset's large size, we sample 100 positive pairs per formula type for both validation and test sets. The final dataset is processed following the same methodology as other pair-based datasets.
Sequence classification datasets:
XNLI (Conneau et al., 2018) - Natural language inference task extending MNLI (Williams et al., 2018) non-English languages, consisting in classifying sentence pairs into entailment, contradiction, or neutral.
PAWS-X (Yang et al., 2019) - Paraphrase identification task aimed at determining whether two sentences convey the same meaning. Fine-tuning performed cross- lingually, with training on the English subset and evaluation across all available lan- guages.
AmazonReviews (Keung et al., 2020) Sentiment analysis task consisting the satisfaction level of multilingual Amazon product reviews on 1-to-5 scale. Fine- tuning is performed on all available languages.
in estimating
MassiveIntent (Keung al., 2020) Multilingual classification task consisting in assigning sentences to one of 60 topic categories. Fine-tuning is performed on all available languages.
CodeDefect (Zhou et al., 2019) Binary classification task aimed identifying whether a given code snippet contains a defect.
CodeComplexity (Jeon et al., 2023) - Computational analysis task consisting in estimat- ing the order of complexity of a code-formulated computer science problem.
MathShepherd (Wang et al., 2024b) Binary classification task aimed at determining whether a step-by-step math rationale is correct given a problem prompt. We limited the dataset to rationales with 3 steps to mitigate the class imbalance observed in longer rationales, where incorrect solutions become more frequent. As the dataset lacks validation split, allocate half of the test set for validation.
Sequence regression datasets:
WMT (Bojar et al., 2017; 2018; Barrault et al., 2019; 2020; Akhbardeh et al., 2021; Koemi et al., 2022) - Regression task consisting in estimating translation quality given source sentence, and possibly a reference translation. As the original test set covers only three language pairs, we construct validation and test sets by sampling 5% of the training set for each, ensuring broader language coverage in evaluation. We report results under both the reference-free and reference-based evaluation settings.
SeaHorse (Clark et al., 2023) - Multilingual summarization evaluation task, where each text-summary pair is annotated across binary evaluation dimensions. The final score is obtained by averaging these labels, yielding a continuous value between 0 and 1. To avoid penalizing models with limited context lengths, the summary is placed first in the input, followed by the main text, ensuring the model can attend to the full summary.
Token classification datasets:
NER (Liang et al., 2020) Named entity recognition task from the XGLUE bench- mark, combining subsets CoNI.12002 (Tjong Kim Sang, 2002) and CONILL203 (Tjong Kim Sang & De Meulder, 2003), adapted for cross-lingual evaluation with English-only training. It spans four languages (English, German, Spanish, and Dutch) and targets four entity types (Person, Location, Organization, and Miscellaneous).
23

---

Published as a conference paper at COLM 2025
E Results on larger hyper-parameter search
In this appendix, we investigate why the largest EuroBERT model does not significantly outperform its 610M counterpart on retrieval tasks. We hypothesize the larger model may benefit from a more comprehensive hyper-parameter search. To test this hypothesis, we perform a more extensive grid search for fine-tuning on the MS-MARCO dataset across the EuroBERT family. Rather than varying only the learning rate with fixed optimization settings, we systematically explore broader set of hyperparameters: Adam's B2 (0.95 [default], 0.98, 0.999), Adam's (10-5 [default], 10-8), and the number of training steps (1,000 [default), 2,000).
MIRACL
MLDR
70
93
92
91
66
Default
Extensive
Default
Extensive
Wikipedia
CC-News
96
75
95
70
Default
Extensive
Default
Extensive
EuroBERT-210M
EuroBERT-610M
+ EuroBERT-2.1B
Figure Retrieval performance EuroBERT models under the default fine-tuning config- uration (Default) compared to the more extensive hyperparameter grid search (Extensive). Results are reported as average nDCG@10 across supported languages.
Figure demonstrates that, while all models benefit from denser hyperparameter grid search, the largest EuroBERT model exhibits the most substantial improvements, particularly on the MIRACL and MLDR datasets. Additionally, as shown in Table 8, increasing the number of training steps from 1,000 to 2,000 consistently enhances performance across all model sizes. Also, we observe that models generally benefit form increasing f2 and reducing the and learning rate.
24

---

Published as a conference paper at COLM 2025
MIRACL
Learning Rate
Adam
Adam
Steps
Model
Default
Extensive
Default
Extensive
Default
Extensive
Default
Extensive
EuroBERT-210M
4.6e-05
2.8e-05
0.95
0.98
1e-05
1e-05
1,000
2,000
EuroBERT-610M
3.6e-05
2.8e-05
0.95
1e-05
1e-08
1,000
1,000
EuroBERT-2.1B
3.6e-05
1.7e-05
0.95
0.98
1e-05
1e-08
1,000
2,000
MLDR
Learning Rate
Adam f2
Adam c
Steps
Model
Default
Extensive
Default
Extensive
Default
Extensive
Default
Extensive
EuroBERT-210X
2.8e-05
2.8e-05
0.95
0.98
1c-05
1c-05
1,000
2,000
EuroBERT-610M
2.2e-05
2.8e-05
0.95
0.95
1e-05
1e-05
1,000
2,000
EuroBERT-2.1B
4.6e-05
1.3e-05
0.95
0.98
1e-05
1e-08
1,000
2,000
Wikipedia
Learning Rate
Adam f
Adam
Steps
Model
Default
Extensive
Default
Extensive
Default
Extensive
Default
Extensive
EuroBERT-211 210M
2.8c-05
2.2c-05
0.95
0.98
1c-05
1e-08
1,000
2,000
EuroBERT-610M
3.6e-05
2.2e-05
0.95
0.95
1e:05
1e:08
1,000
2,000
FuroBERT-2.1B
2.8-05
2.8e-05
0.95
0.95
le-05
le-05
1,000
2,000
CC-News
Learning Rate
Adam f2
Adam
Steps
Model
Default
Extensive
Default
Extensive
Default
Extensive
Default
Extensive
EuroBERT-210M
4.6e-05
3.6e-05
0.95
0.98
le-05
le-05
1,000
2,000
EuroBERT-110M
4.6e-05
2.8e-05
0.95
0.95
1e-05
1e-05
1,000
2,000
EuroBERT-2.1B
3.6e-05
3.6e-05
0.95
0.95
1e-05
1e-05
1,000
2,000
Table 8: Overview of the optimal hyperparameters selected on the validation set for both the Default and Extensive fine-tuning configurations.
25

---

Published as a conference paper at COLM 2025
F
Detailed Results
Table 9 and Table 10 present per-language results for the retrieval and sequence classification tasks, respectively. Table 11 and Table report detailed performance on sequence regression. Finally, Table 13 provides per-language results on token classification (NER task).
MIRACL
European Languages
Extra-European Languages
Average
Model
de
fr
it
nl
pl
pt
ar
hi
ja
tu
vi
2h
Euro
World
LLM-RoBERTa--
560M
89.7
91.6
93.5
89.0
81.1
91.1
88.6
90.3
91.5
89.4
XLM-RoBEKIa+3.58
mDeeEiRTa--3-222 23-288M
45.3
39.6
46.2
34.4
36.0
33.7
29.6
35.3
43.7
37.5
mGGTE-MMM305MM
ModemBBERT-ISM
93.4
71.2
81.2
8.6
3.0
36,0
21.3
45.6
81.9
45.0
ModemBBER--99MM
Euro BuroBERT-210M
94.1
95.4
95.8
90.0
83.2
90.8
85.7
90.9
95.1
90.8
BuRoBERT-610M
92.4
EuroBERT-2.18
94.2
95.0
95.3
93.0
87.1
93.4
91.5
94.1
94.8
92.9
European Languages
Extra-European Languages
Average
Model
pl
pt
hi
ja
Euro
World
XLM RoBERTa-2R00
59.4
56.7
58.0
64.5
53.4
60.3
44.4
56.4
50.3
43.6
53.8
58.7
54.6
XLM-KoBERRla-3.8
68.9
66.1
72.7
73.5
67.5
71.0
56.5
6188
62.6
60.8
63.5
70.0
65.9
mDD+BERTa-3-2HIM
mCTE-MLLM-U5M
63.5
68.7
79.5
78.2
71.4
78.1
55.7
66.2
624
60.8
60.9
73.2
67.8
ModemBERR-1S0N
ModemnBERI-3955
68.4
25.3
42.6
58.0
21.2
37.1
0.5
2.0
3.5
3.7
0.9
42.1
23.9
BuroBERT-210M
52.1
59.1
73.4
EuroBERT-610M
72.5
69.5
80.3
79.8
73.9
79.0
55.5
60.9
61.6
62.5
59.0
75.8
68.6
EuroBER1-2.11
77.6
59.3
Wikipedia
European Languages
Extra-Buropean Languages
Average
Model
en
de
nl
pl
pt
ar
hi
8a
m
vi
2h
Furo
World
XLM-RRBERII-28IX
RLM--RBBERI-560M
95.4
93.5
93.4
93.4
92.5
90.7
93.6
95.1
96.3
94.5
96.7
ADDeBERTa-3-2880
66.1
60.4
53.6
57.6
56.5
51.3
58.9
57.6
MCTF-MI M-300M
96.7
94.7
92.
94.1
ModemBBRR-110M
97.3
56.5
60.5
572
67.0
5.7
67.7
57
ModemBBRRT-3WNN
67.4
83.2
120
77.7
97.7
94.7
94.5
95.6
95.7
88.6
93.6
94.4
EuroBERT-610M
92.6
EaroBERT-2.18
99.0
96.1
96.0
96.0
93.9
92.0
96.6
95.8
European Languages
Extra-European Languages
Average
Model
pt
hi
ja
Euro
World
KLM-RoBERTI-22Um
69.9
57.8
55.8
57.5
57.2
66.8
55.1
63.1
72.2
31.1
73.7
76.2
47.8
61.5
77.1
60.4
61.6
R.M-ROBBRT--SG
75.3
82.8
51.0
82.3
83.3
61.1
73.7
72.1
72.8
XLM-RoBERII~3.58
84.4
77.4
79.7
79.1
79.6
83.8
79.5
84.0
88.1
59.5
87.4
88.8
72.1
82.9
86.8
80,9
809
mDDeBFRRT-<3-2HI
20.4
13.8
4.7
33.6
34.8
13.8
mCTE MITTE-MMM.R 305M
76.1
68.7
72.8
70.1
68.4
76.5
65.1
74.3
79.6
32.5
85.1
83.3
56.7
72.3
88.2
71.5
71.3
MedeemBERI-150M
75.6
10.9
2.3
6.7
2.5
8.5
5.2
23.0
ModemBBRR 395M
84.9
21.5
33.4
41.3
20.0
36.2
4.2
27.8
2.2
2.0
9.5
4.1
9.6
10.0
9.9
33.6
21.1
BuroBERT-210M
17.0
79.7
FaroBERT-6100
84.0
72.9
76.4
75.5
73.8
79.6
70.
74.9
84.0
49.7
84.9
85.1
62.4
66.7
88.1
76.6
75.6
EuroBERT-2.1B
85.8
77.1
76.9
73.8
79.0
70.3
79.7
84.2
49.9
88.2
86.5
60.7
63.0
89.9
76.9
75.9
Table 9: Detailed results on multilingual retrieval tasks (nDCG@10, in %).
26

---

Published as a conference paper at COLM 2025
XNLI
European Languages
Extra-European Languages
Average
Model
CT1
pl
pt
hi
ja
m
zh.
Euro
World
NLM-RoBEKTT-28IN
80.0
74,8
76.4
75.1
71.5
69.1
75.4
72.4
71.0
73.2
76.5
71.1
KLM-ReBERIT-SoA
87.1
83.0
82.9
80.4
77.8
81.2
80.6
80.2
80.6
84.1
81.7
XLM-ReBERRI-3358
89.0
85.5
85.3
84.5
81.5
80.5
83.1
82.4
65.3
62.3
86.1
65.7
mDDeHEKTa-33-28IM
84.9
81.1
77.6
761
79.1
77.7
78.1
78.4
87.0
79.5
mGTE-MLM-33MM
81.1
76.9
78.5
77.2
73.6
71.
75.4
72.9
75.9
75.5
78.4
75,8
ModemnBEKI-150M
82,8
65.7
70.6
56,9
540
63.1
53.9
58.3
68.1
72.3
64.3
ModeemBERI-PWM
89.4
75.6
79.1
39.6
557
70.9
60.6
65.0
76.1
80,8
70.9
BuroBERT-200M
83.5
77.8
79.4
78.9
74.3
70.6
76.6
74.2
75.1
75.3
79.9
76.6
EaroBERT- 1-61-6
87.8
82.9
84.6
83.6
79.5
76.7
82.0
80.3
801
80.7
84.7
81.9
EuroBERR-2.18
89.6
86.4
85.8
82.8
79.9
83.3
83.0
82.3
82.3
86.8
84.1
PAWS-X
European Languages
Extra-European Languages
Average
Model
de
hi
zh
Euro
World
HLM-RoBEKTa-28I1
93.8
88.0
889
88.9
KLM-RoBERTa-566U
95.5
91.0
91.8
92.4
92.4
XLM-RoBEKTla3.58
95.8
91.9
91.7
92.3
92.9
92.9
mDeBERTa-33226MM
95.7
90.2
91.3
91.9
91.9
mGTE-MLLM305M
94.7
87.5
89.8
ModernBERT- 15MM
94.7
72.0
74.0
77.7
79,6
79.6
ModemBBEKI-3955
75.4
84.3
84.3
EaroBERT --100M
95.6
86.5
88.7
88.9
89,9
89.9
BuroBERT-610M
95.6
90.0
91.3
92.0
92.2
92.2
EaroBERT-2.18
96.2
91.6
91.8
92.5
93,0
45.0
AmazonReviews
European I Languages
Extra-European Languages
Average
pl
pt
ar
Ja
X. RoBERTT--2000
64.9
65.0
60.6
60.0
39.0
56.8
62.7
61.1
LLM-RRBBETT-S6IM
66.9
62.1
63.1
XI 3.5-BRTT-3355
67.1
67.8
62.4
61.5
63.7
59.2
64.7
63.6
mDDeBERTa-33-200
66.1
60.6
604
63.7
62.1
ACCTE-MIM-005
63.0
65.3
609
99.5
61.2
57.4
62.7
61.5
ModemBERTT
ModemBBERT-3940
67.6
64.9
608
60.0
58.2
57.8
63.3
61.5
FuroRERT-210M
EaroBERT-610M
66.7
66.4
61.6
61.2
61.7
58.1
64.0
62.6
FuroBERT-2.18
Massivelntent
European Languages
European Languages
Average
Model
de
fr
it
m1
pl
ar
hi
ja
m
vi
zh
Euro
World
87.3
XI
90.3
87.7
88.0
89.0
88.5
89.1
88.8
88.8
83.5
88.1
88.9
89.0
87.8
88.9
87.1
88.8
88.2
88.1
86.4
86.9
87.3
87.6
88.0
87.0
86.9
79.8
86.0
87.3
87.3
85.9
86.5
8%.9
87.3
86.5
ModemBBRRT-190
85.5
74.8
76.6
74.4
25.1
74.2
71.1
78.2
57.6
56.1
76.6
74.4
62.3
66.3
79.9
76.9
72.6
ModemBERT
BuroBERT-210M
89.0
86.0
86.9
86.9
87.0
87.1
86.8
87.9
81.2
86.9
87.4
87.2
85.8
85.0
86.0
87.2
86.5
87.8
87.3
88.3
88.2
87.8
EuroBERT:2.11
88.9
87.2
88.0
88.7
87.9
88.2
88.1
88.2
83.2
87.6
89.0
88.1
87.1
85.4
87.0
88.2
87.5
Table 10: Detailed results on multilingual sequence classification tasks (accuracy, in %).
27

---

Published as a conference paper at COLM 2025
Ref-free
European Pairs
Extra- European Pairs
Average
Other
Euro
World
en-pl
pl-en
en-ja
un-ru
jeren
xh-en
NLM-RoBERT--28UM
45.3
53.6
26.7
16.3
32.0
$7.0
56.5
42.5
10.4
21.8
+0.7
25.3
34.2
36.5
XLM RoBERIa-6ill
50.7
66.0
30.7
15.8
41.1
29.8
52.1
61.2
66.2
472
11.0
24.8
45.4
29.0
39,0
40.8
XLM RobERRT--
65.1
66.9
35.9
18.7
43.3
56.3
64.9
52.2
13.1
47.7
30.6
44.5
52.8
61.6
33.1
18.
39.6
516
61.2
67.
46.7
11.2
24-2
47.3
28.9
42.0
MCTE-MLLM-15MM
48.6
55.2
30.4
18.5
37.5
35.9
18.3
57.2
59.7
15.5
10.6
23.4
11.5
27.3
37.7
38.5
ModernBEKTT1500
47.4
29,8
18.2
21.9
41.6
57.8
10.7
21.5
41.3
ModemBBEKT-39N%
51.7
32.4
20.0
27.6
44.7
41.5
12.3
42.7
BumOBERT-210M
52.9
58.4
33.2
17.5
40.6
40.3
51.1
57.9
57.5
48.3
14.3
26.7
44.3
30.8
40.5
41.0
EuroBERT-610M
52.9
61.1
324
18.2
12.6
09.2
51.
59.1
6233
18.6
12.2
26.6
11.1
29.7
11.1
11.5
EuroBER1-2.11
29.8
19.3
38.5
10.7
27.5
Ref-based
European Pairs
Extra European Pains
Average
Other
cn-pl
de-en
pl-en
en-ja
0n-ru
en-tr
en-2hh
ja-en
ru-en
zhyen
XLM. RaBERTa-280N
49.6
56.0
34.4
29.2
47.8
41.9
49.8
60.5
63.1
50.1
14.2
25.3
45.7
31.1
43.1
43.0
M.M-RoBERT--600
52.5
63.9
37.3
26.8
51.1
42.3
53.6
63.3
70.9
53.0
12.6
25.2
49.2
32.3
45.6
45.5
XLM-RoBERTa-3.58
mD -3-288M
53.0
65.6
37.6
27.9
43.7
52.7
63.5
69.1
33.0
13.1
27.8
49.9
32.5
46.5
45.7
mCTE-MI M-M1MM
51.0
57.4
36.6
29.9
49.6
39.3
51.5
61.8
63.5
52.4
13.8
26.0
48.7
32.8
44.0
43.9
ModemBBERT-393
47.4
33.9
39.2
31.6
43.5
34.8
47.1
52.3
60.0
48.6
14.9
28.9
53.1
34.0
41.7
42:1
EuroBEKI-ZIUM
52.5
99.8
38.8
30.5
49.7
39.6
522
61.5
63.9
33.1
15.2
28.7
90.5
34.7
45.1
45.2
EuroRERT-510M
37.8
29.4
42.5
16.2
EuroBERT-2.18
56.9
65.7
39.
31.0
45.0
55.0
61.3
68.
515
14.
28.3
52.0
34.3
68.5
47.3
Table 11: Detailed results the WMT tasks (Spearman rank correlation, in %).
European Languages
EnttraEEuropenn Languages
Average
Model
es
pl
pt
ja
World
XIMMROBERTa
280M
55.6
62.0
67.3
59.3
56.9
61.1
M.M-RRHFRTT-50OM
57.1
60.1
66.9
73.6
72.6
62.7
61.4
65.5
XLM-RoBERTS-3.5U
58.1
62.1
69.7
75.6
75.5
64.0
63.3
67.5
mDeBERTa-v3-200
56.2
58.5
66.3
72.6
71.9
59.6
60.3
64.2
ICTTE-MLM-305M
52.7
58.6
66.3
69.7
69.6
614
63.0
ModemBBERT110M
44.6
47.9
528
59.6
53.8
52.1
51.8
McdemBERT-335M
56.9
56.8
64.0
66.7
65.6
59.0
59.3
61.5
BuroBERT-ZIOM
54.4
58.7
672
70.0
71.2
61.3
60.1
63.8
EuroBERT-610M
57.2
60.6
70.3
72.6
73.5
61.5
62.7
66.0
FuroBERT-2.11
58.8
62.1
71.1
74.4
75.7
628
64.0
67.5
Table 12: Detailed results on the SeaHorse summary evaluation task (Spearman rank correlation, in %).
European Languages
Extra- European Languages
Average
Model
en
de
es
ft
pl
pt
ar
hi
ja
ru
tr
vi.
Euro
World
XIM-ROBERTa-280M
97.6
94.5
93.6
96.1
95.5
95.5
XIM-RR8FFRI3-60M
97.7
93.9
XIM-RoBERIL-3.5B
98.3
96.4
94.5
96.3
96.3
MD4BERTT-+33280M
98.1
96.4
94.2
96.2
96.2
MCTE-MLM-305M
97.9
94.0
93.0
95.7
95.2
95.2
ModemBB-KI-150
84.0
89.4
89.8
92,8
89.0
89.0
ModemBERT-339M
97.8
81.5
92.5
94.6
91.6
91.6
EuroBERT-210M
97.7
91.8
93.8
95.5
94.7
94.7
EuroBERT-610M
97.6
96.0
94.0
95.8
95.9
95.9
FuroBERT-2.1B
97.6
94.2
93.8
95.0
95.2
95.2
Table Detailed results on the NER task (F1 score, in %).
28


Reviews:
[{"review_id": "kESl9gkHFf", "reviewer": "Reviewer_aExR", "summary": "EuroBERT is a family of multilingual encoders covering English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, and Hindi using a dataset of 5T tokens. Model capabilities span across multilingual retrieval, sequence classification, token classification, sequence regression as well as code and math retrieval and classification tasks.\n\nMethodological insights include:\n- Higher masking ratios benefit retrieval tasks, lower ratios improve sentence classification\n- Inclusion of data for code and math improves multilingual retrieval but degrades classification accuracy\n- Only selecting documents with high educational value degrades performance", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is comprehensively and rigorously written, with a specification of training procedure that is sufficiently detailed that one may replicate this work with relative informational ease. \n- The paper has a thorough recipe analysis with several useful methodological insights.", "reasons_to_reject": "- EuroBERT only focuses on a select few (relatively high-resource) European languages: English, German, Spanish, French, Italian, Dutch, Polish and Portugese, but misses out on the following (out of a list of the 24 official languages of the EU): Bulgarian, Croatian, Czech, Danish, Estonian, Finnish, Greek, Hungarian, Irish, Latvian, Lithuanian, Maltese, Romanian, Slovak, Slovenian, Swedish. This significantly diminishes the claims of this family of encoders being EuroBERT, and acceptance of this paper does a disservice to the multilingual and low-resource languages literature.", "questions_to_authors": "- In Tables 9 and 10, I would suggest rephrasing \"Extra-European Languages\" to \"Non-European Languages\" to avoid confusion and reduce Eurocentricity. \n- Given the large hyperparameter sweep for the larger EuroBERT, did the authors investigate applying MuP?", "extra_scores": {}}, {"review_id": "jB2fyXBFQ4", "reviewer": "Reviewer_pgGu", "summary": "This paper presents a large-scale multilingual encoder, trained on European languages as well as a few non European major languages with a masked language modeling objective. The model is evaluated on several multilingual tasks, including code and math related.\nResults show that the proposed model is competitive on various tasks and state-of-the-art on code/math tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* the experimental results are thorough and strong\n* the model will be publicly released (although I cannot find a hyperlink or supplementary material with the models available yet), which could benefit the community\n* the paper is well written and detail oriented", "reasons_to_reject": "* even though the results are strong, the fact that XLM-Roberta is ranked 1st 7 times while EuroBERT is ranked first 6 times (see Table 1) could prevent adoption. EuroBERT outperforms all other baselines on math/code but this is likely because related data was added to training.\n* a comparison on decoder architecture would be informative", "questions_to_authors": "* 98: \"due to pre-processing constraints\": more details on those constraints would be welcome.\n* on non code/math tasks, XLM-Roberta still seems to outperform EuroBERT (7 wins vs 6 wins) but has 3.5B parameters compared to 2.1B parameters for EuroBERT. Could increasing capacity bridge the gap?\n* 180-181: \"whereas XLM-RoBERTa suffers notable degradation.\" this seems accurate on MLDR but not on SeaHorse", "extra_scores": {}}, {"review_id": "gArST5HeBL", "reviewer": "Reviewer_iq5e", "summary": "This paper introduces EuroBERT, a family of multilingual encoder models designed to advance the state-of-the-art in multilingual NLP tasks. The authors address the growing need for robust multilingual encoders by leveraging recent architectural and training advancements from decoder models, while focusing on European languages and global languages. EuroBERT is trained on a 5T-token corpus spanning 15 languages, including code and mathematics, and demonstrates superior performance across retrieval, classification, regression, and token classification tasks compared to existing models like XLM-RoBERTa and mGTE-MLM. The paper also provides an in-depth analysis of design choices, including dataset composition and training strategies, and releases the models, checkpoints, and framework to facilitate future research.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors conduct extensive experiments across various tasks, including retrieval (MIRACL, MLDR), classification (XNLI, PAWS-X), regression (WMT, SeaHorse), and token classification (NER). Results demonstrate that EuroBERT consistently outperforms or matches baselines (XLM-RoBERTa, mGTE-MLM) while requiring fewer parameters.\n\nThe release of models, checkpoints, and training code is commendable and aligns with reproducibility standards. This transparency will accelerate follow-up work on multilingual encoders.", "reasons_to_reject": "My major concern of this paper is the innovation seems limited.  The basic idea is to transfer to the training recipe  from decoder models to encoder models, I don't see enough innovation in the proposed method. So maybe the author could elaborate the key new findings of this paper.\nAnd the results seems a little bit empirical and lack in depth explanations, such as in Sec4, Why does instruction tuning degrade encoder performance, contrary to decoders? Is this tied to task objectives or architecture?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F2xiKtVi6r", "reviewer": "Reviewer_6A47", "summary": "The paper trains new multilingual encoder models of three different sizes (200M, 600M, 2B parameters), adopting various improvements from recent work on decoder-only models, and evaluates them on a range of retrieval, classification and regression benchmarks. They show that the model outperforms or is on par with alternatives across a wide range of tasks and languages. They promise to release the models, intermediate checkpoints and the training code base.\n\nI think this paper is a valuable contribution to the community. The encoders will be useful for a wide range of research on downstream tasks and non-English languages that so far had to rely on outdated encoders. The reported ablation experiments (and promised intermediate checkpoints and training code) will also be very informative for future encoder training. The paper is well-written, easy to follow, and I don't see any major flaws in the experimental setup.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Valuable resource that enables future research. Fills existing gaps (outdated encoders, language coverage). Well documented and released along with code to reproduce.", "reasons_to_reject": "Minor criticism: The biggest model performs a bit worse than expected, as the authors also acknowledge and investigate in the appendix. A reason could be that all sizes were trained with the same learning rate, batch size and other hyper parameters. Typically, hyper parameters need to scale along with the model size. Existing work even shows how to avoid tuning hyper parameters at larger scale by transferring from small models (see e.g. [1] and [2]). Despite this potential limitation, the trained models are still valuable.\n\n[1] Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer\n[2] Scaling Exponents Across Parameterizations and Optimizers", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a set of multilingual encoders covering European languages and beyond. Reviewers praised the clarity of the writing and the rigor of the experimental evaluation, and overall considered the work a valuable contribution to the community. While not a major issue, I agree with reviewer aExR that the name EuroBERT may be misleading, I recommend considering an alternative name.
Decision: Accept