The Russian-focused embedders' exploration: ruMTEB benchmark and Russian
  embedding model design
Authors: name: Artem Snegirev; name: Maria Tikhonova; name: Anna Maksimova; name: Alena Fenogenova; name: Alexander Abramov
Published: 2024-08-22 15:53:23

Abstract: Embedding models play a crucial role in Natural Language Processing (NLP) by
creating text embeddings used in various tasks such as information retrieval
and assessing semantic text similarity. This paper focuses on research related
to embedding models in the Russian language. It introduces a new
Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark,
the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our
benchmark includes seven categories of tasks, such as semantic textual
similarity, text classification, reranking, and retrieval. The research also
assesses a representative set of Russian and multilingual models on the
proposed benchmark. The findings indicate that the new model achieves results
that are on par with state-of-the-art models in Russian. We release the model
ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code,
integration into the original framework and a public leaderboard.