RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Limitation
Model limitations. The training data for ru-en-RoSBERTa includes large segments from the Internet domain. Consequently, it contains various stereotypes and biases from English and Russian sources. Therefore, a proper model evaluation is still needed to explore their possible vulnerabilities in generalizing to out-of-domain data. The model’s context is limited to a length of 512.
One of the model’s limitations is that due to limited computational resources, we skip the contrastive pre-training stage, leaving it for future work, although it was found Wang et al. ( 2022a , 2023a ) to improve the results on the retrieval-related tasks.

Lack of evaluation in English. As this study focuses on Russian, we do not perform a comprehensive evaluation of ru-en-RoSBERTa in English.

Speed and optimization. The ruMTEB benchmark comprises 23 tasks, including 6 tasks from the multilingual version. As the project is collaborative and we plan to expand the benchmark with new representative tasks, this may lead to resource-intensive and time-consuming runs. Additionally, continuously updating the benchmark makes previous model results obsolete. Due to the potential expansion of ruMTEB with new tasks and the general trend toward using larger models, there is a need to optimize the benchmark evaluation procedure.

Datasets. The collaborative nature and aggregation of the existing sets in the benchmark make it challenging to ensure uniformly high data quality across all tasks.

For all benchmark datasets we checked the licenses and filtered the datasets. Unfortunately, despite the joint effort, some tasks still possess errors (e.g., incorrect labels for some examples, grammatical errors, surplus technical symbols, etc.). Moreover, there may still be biases in the data across different domains and sources, and there is still a need to extend tasks in some categories. We encourage researchers to collaborate further to fill the gaps and ensure a more comprehensive and balanced language and task representation in the benchmark.

Data leakage. All benchmark datasets are either publicly available or created using data found on the Web. This can lead to data leakage when some models trained on parts of the dataset may produce inflated scores on the benchmark. In the future, it’s crucial to develop methods for automatically identifying data leakage in the task.
Ethical Considerations
Inference Costs. Evaluating embedding models on ruMTEB depends on its architecture and size and can be optimized with distributed inference libraries. For example, one run of ru-en-RoSBERTa of the complete evaluation experiment on a single A100 GPU 80GB takes approximately 19 hours.

Energy Efficiency and Usage. We compute the CO_{2} emissions from pre-training and fine-tuning ru-en-RoSBERTa as Equation 1 Strubell et al. ( 2019 ) :

CO_{2}=\frac{PUE*kWh*I^{CO2}}{1000} (1)

The power usage effectiveness ( PUE ) of our data centers is 1.3 . The resulting CO2 emission is 3.66k kg. Model compression techniques can reduce the computational costs associated with model inference.

Potential Misuse. The ruMTEB can be used as training data for acceptability classifiers, potentially improving the quality of generated texts. We acknowledge that these improvements in text generation might lead to the misuse of LLMs for harmful purposes. The intended use of ruMTEB is for research and development purposes , and we are aware of the potential negative uses.

AI-assistants Help. We improve and proofread the text of this paper using Grammarly 18 18 18 https://app.grammarly.com/ to correct grammatical, spelling, and style errors and paraphrasing sentences. Thus, some segments of our publication can be potentially detected as AI-generated, AI-edited, or human-AI-generated.
A.1.1 Training Data Information
The list of datasets included in ru-en-RoSBERTa training data and the corresponding prefix used for them are given in Table 6 .

We use the following basic rules to choose a prefix:

search_query and search_document prefixes are for answer or relevant paragraph retrieval

clustering prefix is for asymmetric retrieval of title or summary and relevant document

classification prefix is for symmetric paraphrasing related tasks (STS, NLI, bitext mining)
A.1.2 Data Filtration Details
We apply the following steps to the basic Russian datasets. First, texts longer than 500 tokens (ruRoBERTa-large 20 20 20 https://huggingface.co/ai-forever/ruRoberta-large tokenizer is used) are filtered out. A small number of tokens is reserved for instructions or prefixes. Pairs from YandexQ 21 21 21 https://huggingface.co/datasets/IlyaGusev/yandex_q_full , Pikabu 22 22 22 https://huggingface.co/datasets/IlyaGusev/pikabu , StackOverflow 23 23 23 https://huggingface.co/datasets/IlyaGusev/ru_stackoverflow , Habr 24 24 24 https://huggingface.co/datasets/IlyaGusev/habr and Habr QnA 25 25 25 https://huggingface.co/datasets/its5Q/habr_qna are filtered by content popularity (e.g. views, ratings, votes). Cosine similarity obtained from LaBSE Feng et al. ( 2022 ) is applied to filter NewsCommentary and MultiParaCrawl. We filter pairs from paraphrase-NMT-Leipzig 26 26 26 https://huggingface.co/datasets/cointegrated/ru-paraphrase-NMT-Leipzig by p_good score (equivalent meaning). The XNLI is formed from entailment (relevant document) and contradiction (irrelevant negative) examples. For MIRACL, we use the title as the query and the first paragraphs (until we reach the token limit) as the document. We form pairs for Paraphrases 27 27 27 https://huggingface.co/datasets/inkoziev/paraphrases from paraphrases field, taking one as a query and the others as positive documents. The content of RuNews 28 28 28 https://huggingface.co/datasets/IlyaGusev/ru_news is not changed. After exact match deduplication, the final training pairs for all datasets are randomly sampled from the remaining pairs.
A.2 Model Training Details
We train the model in bf16 dtype with gradient checkpointing and use AdamW Loshchilov and Hutter ( 2017 ) with a learning rate of 1e-5 and weight decay of 0.01 for exactly one epoch, which is approximately 3700 steps, of which linear warmup is 200 steps. After fine-tuning, the SLERP merging is applied to the base model with a factor of 0.1.

We apply stratified sampling per device batch (mini-batch). Therefore, the global batch includes mini-batches consisting of pairs of different datasets. On the one hand, it becomes impossible to exchange negatives between devices and thus scale the number of in-batch negatives. On the other hand, this increases the diversity of sources in the global batch. Therefore, we do not apply the DisCo Chen et al. ( 2023 ) trick to exchange negative examples across devices. The batch size is 128 per device, giving 1024 documents per query. Context length is set to 512 for queries and documents Merrick et al. ( 2024 ) .

Training is conducted on a single H100 node. We utilize codebase from BGE 29 29 29 https://github.com/FlagOpen/FlagEmbedding and adapt it for our experiments. PyTorch’s expandable_segments helps us to mitigate fragmentation issues due to variable sequence length.
A.3 ruMTEB Dataset Description
This section describes new tasks we present with the research and data preparation details.
A.3.2 Pair Classification
TERRa . The dataset was presented as one of the Russian SuperGlue tasks Shavrina et al. ( 2020 ) and related to the Textual Entailment Recognition task. Given two texts, the task is to determine whether the meaning of one text entailed from the another text. Since the test split is hidden, we took the dev split without changes. A total of 307 examples are available.
A.3.3 Multi-Label Classification
CEDRClassification . The dataset is a task of classifying comments into five emotions (joy, sadness, surprise, fear, and anger). A total of 9,410 comments were presented from the following sources: social networks, news, and blogs. The dataset was used as is, without any modifications Sboev et al. ( 2021 ) . We took the original test split, which includes 1882 examples.

SensitiveTopicsClassification . The dataset contains sentences that can be classified into one or more sensitive topics 36 36 36 https://github.com/s-nlp/inappropriate-sensitive-topics/blob/main/Version3/sensitive_topics.csv Babakov et al. ( 2021 ) . The original dataset includes 18 classes, all classes are used. Since part of the dataset is not only manually labeled, we first formed a test split from manually labeled examples, and the remaining examples were combined with semi-automatically labeled examples. We have selected the most reliable examples based on the confidence scores indicated in the examples. The final test split consists of 2048 examples and preserves the original class distribution.
A.3.4 Clustering
GeoReviewClustering A clustering dataset based on the Yandex Maps 37 37 37 https://yandex.ru/maps reviews 38 38 38 https://github.com/yandex/geo-reviews-dataset-2023 , where given a review text one has to cluster the samples according to their rubrics or review categories (e.g., Bank, Supermarket, Pharmacy, etc.). The original dataset was balanced and split into three parts (train, validation, and test). For each review, we took its first rubric as the main label, leaving only samples corresponding to the top 100 most popular labels. This threshold limited the categories exceeding 10,000 examples. The final dataset was converted into the MTEB format.

RuSciBenchGRTNI/OECDClustering .
This is a dataset for the clustering of scientific text headings. Each article has its OECD and GRNTI headings, and there are 29 OECD headings and 28 GRNTI headings in the dataset (e.g., Mathematics, Biological Sciences, Economics and Business, etc.). The data was sourced from the original dataset RuSciBench 39 39 39 https://huggingface.co/datasets/mlsa-iai-msu-lab/ru_sci_bench . During preprocessing, duplicates were removed, the title and abstract were combined, and the set was balanced, leaving only the same number of samples for each class. The resulting dataset was then divided into test and training parts.
A.3.5 Semantic Textual Similarity (STS)
RuSTSBenchmarkSTS . The dataset used for the STS task is derived from the original multilingual STS Benchmark 40 40 40 https://github.com/PhilipMay/stsb-multi-mt . This multilingual set comprises various translations of the original English version of the STSbenchmark dataset, with the translations completed using deepl.com 41 41 41 https://www.deepl.com/ru/translator . The Russian segment of the dataset was extracted and refined using the RuCoLa Mikhailov et al. ( 2022 ) classifier 42 42 42 https://huggingface.co/RussianNLP/ruRoBERTa-large-rucola . In all parts of the sets (train/dev/test), instances categorized as not linguistically acceptable were excluded. Additionally, any duplicate entries were eliminated.
A.3.6 Reranking
RuBQReranking . The dataset is based on RuBQ version 2.0 Rybin et al. ( 2021 ) . The dataset contains examples of questions and paragraphs from Wikipedia. Paragraphs that answer the question are considered relevant. Paragraphs that contain the answer are used as positive documents. Negative documents are paragraphs relevant to the question’s topic but do not contain an answer. We only used questions from the test split with at least nine negative documents. The final test split contains 1551 examples.
A.3.7 Retrieval
RuBQRetrieval . Unique paragraphs from the dataset are used for the document bank, resulting in 56,826 documents. Documents were deduplicated while links to relevant documents were maintained. The original test split was taken without changes and has 2845 examples.

RiaNewsRetrieval . The original dataset RussiaSegodnya 43 43 43 https://github.com/RossiyaSegodnya/ria_news_dataset (also known as RiaNews) consists of news articles and their headlines Gavrilov et al. ( 2019 ) . Texts are presented in lowercase format, and the capitalization of individual characters has not been changed. Since the article texts are available in HTML, we used the BeautifulSoup 44 44 44 https://www.crummy.com/software/BeautifulSoup library to clean them of markup. Additionally, texts were normalized, and extra spaces were removed. We also removed, if possible, the first sentence in each article text since it does not relate to the article’s content and is a kind of meta information (“Moscow, 1 Dec — ria news.”). We filtered out the texts of articles with more than 2000 characters so that models limited to a context of 512 tokens could handle the entire text. All examples were deduplicated based on the headline and text of the article. Our final dataset consists of 10,000 randomly sampled headlines as queries, and article texts (724344) are used as documents.
A.4 Experimental Setup Details
This section describes the prompt and embedding configuration we used in our experiments. Namely, we use normalized embeddings for evaluation on all ruMTEB tasks. We use pooling and instruction strategies required by the corresponding model we evaluate. Table 7 presents all the prefixes and instructions used. Specifically:

we do not utilize any special prompts for rubert-tiny2, BGE-M3, SBERT {}_{\textnormal{large-ru}} , and SBERT {}_{\textnormal{large-mt-ru}} ;

we use special prefixes for ru-en-RoSBERTa;

mE5 {}_{\textnormal{small/medium/large}} models share the same set of prefixes;

mE5 {}_{\textnormal{large-instruct}} and E5 {}_{\textnormal{mistral-7b-instruct}} models share the same set of instructions.
A.5 Detailed Results
Table 8 shows results on individual ruMTEB datasets.
A.6 Additional Experimental Findings
In this part, we describe early-stage experiments that were conducted on different data subsets and different base models.

Prefixes . We found that E5 prefixes Wang et al. ( 2022a ) performed slightly worse and assume that the variant we use helps to better separate tasks during training. The clustering prefix is more suitable for tasks where thematic identification is required, so in many classification problems, we use it instead of classification , despite the name. We tried adding prefixes with some probability; this improved the results without using prefixes and also worsened the results with them. In addition to stratified sampling, we implemented a sampling strategy that takes pairs with the same prefix but saw no improvement.

Losses . It was shown that Sigmoid Loss Zhai et al. ( 2023 ) performed better at smaller batch sizes. We found that SigLIP is more sensitive to selecting the initial values of the bias and temperature parameters to achieve convergence. CoSENT loss Li and Li ( 2023 ) shows better results for STS-like tasks; we adapted the loss for the case with many negatives. In both cases, we were unable to achieve comparable results and left this for further work.

Augmentations . Although the model trained on 1500 steps shows comparable results to full training, we tried to apply text level and embedding level augmentations but found no meaningful performance improvement. For the text level, we used character-level augmentation from the Augmentex 45 45 45 https://github.com/ai-forever/augmentex library Martynov et al. ( 2024 ) for both languages. In another experiment, we applied the NEFTune Jain et al. ( 2023 ) with 3, 5, and 10 alpha parameters.
Figure 1: The scheme of the ruMTEB benchmark presenting all benchmark tasks divided into 7 task categories.; The scheme of the ruMTEB benchmark presenting all benchmark tasks divided into 7 task categories.
The ruMTEB task outline. The Train , Val , and Test columns show the sizes of the dataset splits (“–” means the absence of the split). Datasets from the original MTEB benchmark are in Italic; for them, the sizes of the Russian subsets are reported.
Different data sources impact. Model performance is measured on ruMTEB. Avg. stands for the average score and is computed as the mean of the category scores. The best score is put in bold, the second best is underlined.
Results of the model, method, and data variation. † The reference results for the training objective and training examples sections is model based on ru-en-RoBERTa. Each experiment changes a single component (e.g., use AnglE similarity instead of cosine). Model performance is evaluated on ruMTEB. The best score across all experiments is bold, the second best is underlined.
The evaluated mode description. Instruct models are marked with the corresponding suffix.
Average model results on ruMTEB task categories. The result for each category represents the mean model score on the tasks from the corresponding task types. Avg. stands for the average score and is computed as the mean of the category scores. The best score is put in bold, the second best is underlined.
The full training corpus with corresponding prefixes. We report the number of pairs in thousands. For the tasks with two different prompts, query and document, they are written with a slash.
Prompts used for ruMTEB evaluation. For the tasks with two different prompts, query and document, they are written with a slash. E5 prefix shows prefixes used for mE5 {}_{\textnormal{small/medium/large}} models. E5 instruction shows instructions used for E5 {}_{\textnormal{mistral-7b-instruct}} and mE5 {}_{\textnormal{large-instruct}} models.
The full results of model evaluation on the ruMTEB benchmark. The aggregated score for each task category is reported in Section 7 . The best
score is put in bold, the second best is underlined.