RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
Text embeddings play an important role in many Natural Language Processing (NLP) tasks, from clustering to semantic textual similarity (STS) and information retrieval (IR).
The community has addressed this demand by releasing several powerful text embedding models (or embedders) Wang et al. ( 2024 , 2023a ); Chen et al. ( 2024 ) . However, there is still a lack of such embedders developed specifically for the Russian language. The most popular Russian-oriented models, such as rubert-tiny2 2 2 2 https://huggingface.co/cointegrated/rubert-tiny2 , SBERT {}_{\textnormal{large-nlu-ru}} 3 3 3 https://huggingface.co/ai-forever/sbert_large_nlu_ru , and SBERT {}_{\textnormal{large-mt-nlu-ru}} 4 4 4 https://huggingface.co/ai-forever/sbert_large_mt_nlu_ru , have been released several years ago and thus do not include modern data in their training corpora. The latest models are based on an outdated version of the ruBERT Zmitrovich et al. ( 2023 ) model as a backbone. Moreover, being monolingual, they can not profit from knowledge transfer between languages.

Given the usability of such text embeddings, evaluating their quality and the corresponding embedders is also important. One general approach is to evaluate text embeddings on a set of standard text embedding tasks (classification, clustering, etc.) For English, Massive Text Embedding Benchmark, or MTEB Muennighoff et al. ( 2023 ) , is considered to be a standard for such an evaluation. For Russian, however, there are significantly fewer evaluation resources. Only few tasks from MTEB contain Russian subsets, while until recently, the only embedding benchmark for Russian was enkodechka 5 5 5 https://github.com/avidale/encodechka , which appeared several years ago and is still actively used. However, it has significantly fewer tasks than MTEB and has no tasks to evaluate the retrieval abilities of the model.

This paper addresses both problems by presenting a novel Russian-focused embedding model, also adapted for the English language, allowing knowledge transfer from this high resource language, and a new benchmark for text embedding evaluation in Russian called ruMTEB (see Figure 1 for its general structure), which contains 23 text embedding tasks in MTEB format. Among them, 17 tasks are new, and the other 6 are formed on the multilingual MTEB datasets. Thus, the contributions of our work are the following:

we publicly release a Russian-focused text embedding model 6 6 6 https://huggingface.co/ai-forever/ru-en-RoSBERTa adapted for the English language;

we present the Russian version of MTEB and release 17 new Russian datasets for text embedding evaluation 7 7 7 https://huggingface.co/collections/ai-forever/ru-mteb-6650a6a6708dc5107a9e0ba3 , which form the benchmark backbone;

we explore model cross-lingual transfer knowledge abilities and various model training hypotheses, which define our final training pipeline;

we evaluate the presented model on ruMTEB and compare its performance with a set of open-source baseline solutions.
Text Embedding Models
General text embedding models are widely used in various applications such as retrieval-augmented generation (RAG) Lewis et al. ( 2020 ) , STS, as well as multimodal scenarios Radford et al. ( 2021 ) . One of the first approaches for training such models was to fine-tune a pre-trained language model on the collection of labeled text pairs, such as SNLI Bowman et al. ( 2015 ) . Natural Language Inference (NLI) has been shown Reimers and Gurevych ( 2019 ) to help such models learn useful representations of texts for STS and other downstream applications. Recent approaches for model training utilize labeled datasets, which can be divided into symmetric (NLI, STS) and asymmetric (Retrieval) tasks. Hence, the training objective takes the form of multitask learning over one or multiple objectives, and the specialized instructions are applied for each task Su et al. ( 2022 ) .

Instead of training on limited labeled datasets, in Wang et al. ( 2022a ) , it has been proposed to split fine-tuning into two stages: contrastive pre-training uses a large-scale pair dataset of noisy (or weakly-supervised) text examples, and contrastive fine-tuning utilize a smaller number of high-quality examples. The authors of the E5 {}_{\textnormal{mistral-7b-instruct}} Wang et al. ( 2023a ) utilize an approach for model training which does not include expensive contrastive pre-training that has been shown to be useful for smaller encoder-only model XLM-R Conneau et al. ( 2019 ) . While their quality remains comparable, encoder-only models are more cost-effective during inference.

Examples of modern English-focused models include E5 Wang et al. ( 2022a ) , BGE Xiao et al. ( 2023a ) , GTE Li et al. ( 2023 ) , Nomic Nussbaum et al. ( 2024 ) and Arctic Embed Merrick et al. ( 2024 ) . Scaling the number of languages supported (including Russian) has been demonstrated in mE5 Wang et al. ( 2024 ) models and BGE-M3 Chen et al. ( 2024 ) , thereby extending their applicability in multilingual contexts. The Russian-oriented models are mainly represented by SBERT models and rubert-tiny2 and their modifications.

Models mentioned above are often used for additional fine-tuning on a specific task. To preserve the general ability of the embedding model, it has been proposed Xiao et al. ( 2023b ) to merge the fine-tuned model with its base model.

To address this lack of contemporary Russian-focused embedding models performing on par with their multilingual counterparts, we present ru-en-RoSBERTa .
Text Embedding Benchmarks
Model evaluation has always played an inevitable role in NLP progress. Starting from GLUE Wang et al. ( 2018 ) and SuperGLUE Wang et al. ( 2019 ) benchmarks have been a standard model evaluation method. As for text embedding representation evaluation, it has been in focus for many years and was commonly evaluated on a STS, for which yearly released SemEval Agirre et al. ( 2016 ); Cer et al. ( 2017 ); Chen et al. ( 2022 ) 8 8 8 https://semeval.github.io/ datasets were commonly used. Being a single dataset inevitably limits the SemEval expressivity. Following the same approach, SentEval Conneau and Kiela ( 2018 ) , which focuses on classifier models on top of embedding, overcomes this limitation by aggregating multiple STS datasets. Still, it lacks the evaluation instruments for the suitability of embedding for retrieval or clustering tasks. Due to the inefficiency of a single STS evaluation, USEB Wang et al. ( 2021 ) , focusing on reranking tasks, and BEIR Thakur et al. ( 2021 ) , aimed at zero-shot information retrieval evaluation, were created. SciRepEval Singh et al. ( 2023 ) , a multi-format benchmark for scientific document representations, includes 24 realistic tasks across four formats: classification, regression, ranking, and search. Finally, uniting and unifying all main classes of the embedding tasks MTEB Muennighoff et al. ( 2023 ) has been proposed and is now considered a multilingual text embedding evaluation standard. Moreover, its approach was also adopted and recreated for the Scandinavian languages in the SEB Enevoldsen et al. ( 2024 ) and Chinese in the C-MTEB Xiao et al. ( 2023a ) 9 9 9 https://huggingface.co/C-MTEB benchmarks.

Most of the benchmarks mentioned above are English-focused. Even MTEB, despite being multilingual, lacks Russian-language tasks. Only few of the datasets contain Russian subsets, which is not enough for a proper embedding evaluation in this language. Apart from these few MTEB subsets, the only Russian embedding benchmark remains enkodechka, which has significantly fewer tasks than MTEB and no tasks to evaluate the model’s retrieval abilities.

Still, there is a need to evaluate text embedding in Russian. To address this demand, we propose ruMTEB comprising a set of text embedding tasks in MTEB format.
Benchmark Tasks
ruMTEB comprises 23 datasets divided into 7 task types mentioned above: six datasets based on the Russian subsets from the original multilingual MTEB set (MassiveIntendClassification, MassiveScenarioClassification, MIRACLReranking, MIRACLRetrieval, RuParaphraserSTS, STS22) and 17 new datasets we release within the research.The latter are based on popular Russian time-tested and community-tested datasets.

We took the datasets based on the original MTEB without any changes. For the datasets adapted from other Russian sets, we performed data cleaning and automatic filtering, where necessary, removed duplicates, and formatted them in the MTEB format. The main dataset information and their statics are given in Table 1 , and the detailed task descriptions and preprocessing for the new sets are in Appendix A.3 .
Text Embedding Model for Russian
This section is devoted to the text embedding model ru-en-RoSBERTa released within the research. We describe the training data, the base model, and the final training pipeline, motivated by the experiments described in Section 5 .
Training Data
Following previous work Wang et al. ( 2022a ); Li et al. ( 2023 ); Nussbaum et al. ( 2024 ) , we use publicly available data, high-quality and synthetic datasets to create training pairs (see Appendix A.1.1 for the full training list) 10 10 10 To avoid potential data leakage we use only the training parts of all the sets. , which, for experiment purpose (see Section 5 ), we divide into four groups described below.

Basic Russian Datasets . This group consists of 17 tasks. It includes pairs from SberQuAD Efimov et al. ( 2020 ) , XNLI Conneau et al. ( 2018 ) , parallel translations Bañón et al. ( 2020 ); Tiedemann ( 2012 ); Zhang et al. ( 2020 ) , and publicly available data from various domains, such as news, blogs, QA platforms, and other Internet resources. We filter this data mostly with manual rules (see Appendix A.1.2 for the details).

Basic English Datasets . The group is formed from MEDI Su et al. ( 2022 ) corpus without provided instructions. We also exclude instructional datasets from Super-NI Wang et al. ( 2022b ) and thus retain 30 datasets representing different domains and tasks. We do not apply any additional preprocessing steps.

Additional Synthetic Datasets . The group includes Query2doc MS-MARCO Wang et al. ( 2023b ) , DINO-STS-x1x2 Schick and Schütze ( 2021 ) , RuHNP Malashenko et al. ( 2024a ) , entailment and contradiction pairs from RuWANLI Malashenko et al. ( 2024b ) , and a sample of generated pairs by ruT5-base 11 11 11 https://huggingface.co/ai-forever/ruT5-base model from WikiOmnia Pisarevskaya and Shavrina ( 2022 ) . We do not change the data content and use the datasets as is.

Additional Retrieval Datasets . We use Russian and English parts of Mr. Tydi Zhang et al. ( 2021 ) and MIRACL Zhang et al. ( 2022 ) from BGE-M3 fine-tuning data. These datasets provide high-quality examples and are designed for the same retrieval tasks included in our benchmark.

We mine negatives similar to Xiao et al. ( 2023a ) using the mE5 {}_{\textnormal{small}} Wang et al. ( 2024 ) and sample documents by rank in the range of 20-100. For all datasets, the provided hard negatives are also used. For additional synthetic and retrieval datasets, the provided negatives are used (if available), and the rest are randomly sampled from the same dataset.
Base Model and English Language Adaptation
Since we focus on the Russian language, we use ruRoBERTa 12 12 12 https://huggingface.co/ai-forever/ruRoberta-large Zmitrovich et al. ( 2023 ) , which has the highest scores on the classic Russian SuperGLUE Shavrina et al. ( 2020 ) benchmark among the models of its size. In addition, we adapt it to the English language, allowing knowledge transfer from this high-resource language (see Section 5 for the corresponding experiments).

We extend the original ruRoBERTa tokenizer with tokens from RoBERTa 13 13 13 https://huggingface.co/FacebookAI/roberta-large Liu et al. ( 2019 ) . To learn new token embeddings, we train the model using Masked Language Modeling (MLM) objective Devlin et al. ( 2018 ) . We use the same hyperparameters as in the ruRoBERTa and the batch size of 1024. We use unique training texts from Section 4.1 and train for one epoch (~11k steps).

The whole process takes one day on two A100 80GB cards. To reduce the effect of catastrophic forgetting Kirkpatrick et al. ( 2017 ) , we merge encoder layers using spherical linear interpolation (SLERP) algorithm 14 14 14 https://gist.github.com/dvschultz/3af50c40df002da3b751efab1daddf2c with the factor of 0.25 to the original model.

In our work, we refer to the obtained model version with the extended vocabulary as ru-en-RoBERTa.
Contrastive Fine-tuning
Following Su et al. ( 2022 ) , we perform contrastive fine-tuning for ru-en-RoBERTa on a mix of supervised and unsupervised data (from the Section 4.1 ).
We use prefix strategy from Reimers et al. ( 2023 ) applying prefixes for each pair to avoid a conflicting reward signal (see Appendix A.1.1 for the prefix rules and the full prefix list).

We employ the standard InfoNCE contrastive loss Oord et al. ( 2018 ) , keep a fixed temperature value of 0.02, and obtain normalized text embedding using CLS pooling. The batch is filled with pairs of the same dataset (stratified sampling), and proportional batch sampling is applied. Negative examples are formed from 7 hard negatives per query, and the remaining negatives are taken from a batch of the same device (in-batch negatives). After fine-tuning, the SLERP merging is applied to the base model with a factor of 0.1. See A.2 for the training details. We report the computational, energy, and carbon costs in Section 10 .
Training Procedure Analysis
This section describes experiments we conducted to determine the final training pipeline. We used basic Russian, English, and additional synthetic datasets, the training approach described in Section 4 unless otherwise specified, and the full ruMTEB version for evaluation. Further findings are given in Appendix A.6 .
Cross-lingual Knowledge Transfer and Data Sources
We explored five training data configurations to study whether the model can profit from knowledge transfer between languages and various data sources.
For this, we trained embedding models based on ru-en-RoBERTa: on basic English datasets only, basic Russian datasets only, and their mixture, simple or augmented with additional synthetic/synthetic+retrieval datasets. Each model is trained for 1500 steps.

Results presented in Table 2 indicate that the embedding model gets better results when trained on data in Russian and English simultaneously.
Additional synthetic datasets and high-quality retrieval datasets further improve the embedding model quality despite the tasks these datasets solve already being well represented in the basic datasets. Given that in all scenarios, the number of steps is fixed, the change in the results could not account for longer training.

The model especially benefits from synthetic datasets on STS-related tasks, while quality degradation in clustering tasks remains unclear. Note that the model trained on almost all data (except the additional retrieval dataset is better by only 0.6 points. The results obtained on data in English may be due to the better quality of the tasks presented in MEDI.
English Language Adaptation
Having shown that the model can profit from cross-lingual knowledge transfer, we turned to selecting the optimal language adaptation strategy. Namely, we compared:

ruRoBERTa and XLM-R used as baselines;

ru-en-RoBERTa from subsection Section 4.2 ;

ru-en-RoBERTa w/ RetroMAE same approach as previous where we substituted MLM with RetroMAE Shitao et al. ( 2022 ) , which proved beneficial for BGE-M3 in Chen et al. ( 2024 ) . For this configuration, set the masking ratio of decoder input tokens to 30%;

ru-en-RoBERTa w/o SLERP same as ru-en-RoBERTa without SLERP after English language adaptation.

We perform contrastive fine-tuning for each model and then evaluate them on ruMTEB. Results (see Table 3 ) show that ru-en-RoBERTa outperforms both baselines by a significant margin. Additionally, the fact that XLM-R slightly outperforms ruRoBERTa may indicate that XLM-R copes better with knowledge transfer from basic English datasets, which provide diverse examples of high quality. Merging the encoder layers after language adaptation with the original model improves the model quality while using RetroMAE leads to decreased results.
Training Examples
In this series of experiments (see Table 3 ), we show the effects of prefixes, stratified sampling, and the number of hard negatives.

Remove prefixes . Fine-tuning the model on symmetric and asymmetric tasks simultaneously can hurt performance without instructions but improve it when instructions are used Su et al. ( 2022 ) . We found that removing prefixes consistently worsens the results, but STS-related tasks were not as affected.

Disable stratified sampling . To explore whether stratified sampling is beneficial Merrick et al. ( 2024 ) in our case, we disabled it, used prefixes only for queries, and negatives were exchanged across devices. The latter increases the number of negatives per query to 8k. We found that the stratified version works better.

Hard-negatives . To study whether adding more hard-negatives Ren et al. ( 2021 ) is beneficial, we increased their number to 15 and reduced per device batch size to 64, maintaining the total number of negative examples. To keep the same number of steps, we apply gradient accumulation. Similarly to Nussbaum et al. ( 2024 ) , we found that despite processing almost twice as many texts, the results did not improve.
Training Objective
In this experiment (see Table 3 ), we examine four modifications of the training objective described in Section 4.3 .

Additive margin . Following Yang et al. ( 2019 ) , we applied an additive margin with the value of 0.01, and larger values caused convergence problems. We do not use additive margin in our final model and found that datasets are sensitive to margin values.

Document penalty . The authors of GTE Li et al. ( 2023 ) add a penalty for query-query, document-document, and document-query matching in the denominator of InfoNCE loss that improves model performance on MTEB. We applied a similar approach to Yang et al. ( 2019 ); Su et al. ( 2022 ) , adding a penalty for document-query matching as an additional loss. The penalty significantly improved model performance on STS and worsened on Retrieval.

AnglE similarity . Normalized dot-product is usually used to score a pair of texts. AnglE similarity was proposed in Li and Li ( 2023 ) to optimize the angle difference of pairs in complex space. We replaced cosine similarity with AnglE. It is worth noting that in the original work AnglE was used in slightly different scenario, therefore, not finding any improvement, we left this for future work.

Mean pooling . Without an experiment, the choice of pooling strategy remains unclear. Mean pooling is used in E5, GTE and Nomic Li et al. ( 2023 ); Nussbaum et al. ( 2024 ) , BGE and Arctic Embed Xiao et al. ( 2023a ); Merrick et al. ( 2024 ) are apply CLS pooling. We observed consistent improvement of the latter, compared to mean pooling.
Evaluation
We evaluate ru-en-RoSBERTa and 9 publicly available embedding models for Russian, including the multilingual ones and the two instruct models, on the ruMTEB benchmark. See Table 4 for the baseline information and Appendix A.4 for other details.

We evaluate all models in the same environments
and scenarios by the procedure described in 3.1 . We use MTEB framework 15 15 15 https://github.com/embeddings-benchmark/mteb/tree/1.12.75 for evaluation where we integrated evaluation on the new ruMTEB tasks 16 16 16 https://github.com/embeddings-benchmark/mteb/pull/815 17 17 17 https://github.com/embeddings-benchmark/mteb/pull/881 .
Results
Table 5 shows model scores averaged within the task category, and detailed results of the task-wise model evaluation are in Appendix A.5 .

Results analysis reveals that there is a gap between instruct and non-instruct models, mE5 {}_{\textnormal{large-instruct}} and E5 {}_{\textnormal{mistral-7b-instruct}} are better than their non-instruct competitors in all task categories except for Retrieval where BGE-M3 is heading the list. Moreover, while the instruct/non-instruct difference is not that significant for STS and retrieval, the advantage of the instruct models becomes obvious for other task categories.

As for the non-instruct model analysis, it can be seen that BGE-M3, ru-en-RoSBERTa, and mE5 {}_{\textnormal{large}} perform practically on par. Moreover, ru-en-RoSBERTa performs better than its non-instruct competitors on all task categories except for 2 (Retrieval and Reranking), probably due to the absence of the contrastive pre-train. For Retrieval and Reranking, BGE-M3 and mE5 {}_{\textnormal{large}} receive much better scores, resulting in ru-en-RoSBERTa being in the second place. The evaluation results show that ru-en-RoSBERTa is a robust embedding model suitable for various textual tasks. Additionally, unlike monolingual Russian models, the bilingual nature of ru-en-RoSBERTa allows it to be further trained or fine-tuned using the much more considerable amount of English data available.

The evaluation results positively characterize the benchmark as being complex enough for modern embedding models, allowing researchers to evaluate text embedding at a high level.
Conclusion
This paper introduces a new Russian-focused embedding model, which also supports English, and a new benchmark for text embedding evaluation, comprising 23 datasets divided into 7 task types. Among the benchmark datasets, 17 datasets are new and were created within this research.

We report the new embedding model architecture design, pre-training corpus, and training procedure details. We describe the datasets comprising the benchmark and propose the methodology for the text embedding evaluation on it inspired by the MTEB benchmark. We evaluate the presented encoding model and several baselines, thus verifying the ruMTEB complexity and performing the comparative analysis of our model results with the results of standard encoders.