mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
Authors: name: Marc Marone; name: Orion Weller; name: William Fleshman; name: Eugene Yang; name: Dawn Lawrie; name: Benjamin Van Durme
Published: 2025-09-08 17:08:42

Abstract: Encoder-only languages models are frequently used for a variety of standard
machine learning tasks, including classification and retrieval. However, there
has been a lack of recent research for encoder models, especially with respect
to multilingual models. We introduce mmBERT, an encoder-only language model
pretrained on 3T tokens of multilingual text in over 1800 languages. To build
mmBERT we introduce several novel elements, including an inverse mask ratio
schedule and an inverse temperature sampling ratio. We add over 1700
low-resource languages to the data mix only during the decay phase, showing
that it boosts performance dramatically and maximizes the gains from the
relatively small amount of training data. Despite only including these
low-resource languages in the short decay phase we achieve similar
classification performance to models like OpenAI's o3 and Google's Gemini 2.5
Pro. Overall, we show that mmBERT significantly outperforms the previous
generation of models on classification and retrieval tasks -- on both high and
low-resource languages.