MMTEB: Massive Multilingual Text Embedding Benchmark
Authors: Kenneth Enevoldsen
Isaac Chung
Imene Kerboua
Márton Kardos
Ashwin Mathur
David Stap
Jay Gala
Wissam Siblini
Dominik Krzemiński
Genta Indra Winata
Saba Sturua
Saiteja Utpala
Mathieu Ciancone
Marion Schaeffer
Diganta Misra
Shreeya Dhakal
Jonathan Rystrøm
Roman Solomatin
Ömer Veysel Çağatan
Akash Kundu
Martin Bernstorff
Shitao Xiao
Akshita Sukhlecha
Bhavish Pahwa
Rafał Poświata
Kranthi Kiran GV
Shawon Ashraf
Daniel Auras
Björn Plüster
Jan Philipp Harries
Loïc Magne
Isabelle Mohr
Dawei Zhu
Hippolyte Gisserot-Boukhlef
Tom Aarsen
Jan Kostkan
Konrad Wojtasik
Taemin Lee
Marek Suppa
Crystina Zhang
Roberta Rocca
Mohammed Hamdy
Andrianos Michail
John Yang
Manuel Faysse
Aleksei Vatolin
Nandan Thakur
Manan Dey
Dipam Vasani
Pranjal A Chitale
Simone Tedeschi
Nguyen Tai
Artem Snegirev
Mariya Hendriksen
Michael Günther
Mengzhou Xia
Weijia Shi
Xing Han Lù
Jordan Clive
Gayatri K
Maksimova Anna
Silvan Wehrli
Maria Tikhonova
Henil Shalin Panchal
Aleksandr Abramov
Malte Ostendorff
Zheng Liu
Simon Clematide
Lester James Validad Miranda
Alena Fenogenova
Guangyu Song
Ruqiya Bin Safi
Wen-Ding Li
Alessia Borghini
Federico Cassano
Lasse Hansen
Sara Hooker
Chenghao Xiao
Vaibhav Adlakha
Orion Weller
Siva Reddy
Niklas Muennighoff
Venue: ICLR 2025 Poster (2025)
Keywords: natural language processing
benchmark
sentence embeddings
multilingual
Primary area: datasets and benchmarks
TL;DR: We introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) including 500+ tasks across 1,000+ languages, greatly expanding multilingual evaluation for embeddings.

Abstract: Text embeddings are typically evaluated on a narrow set of tasks, limited in terms of languages, domains, and task types. To circumvent this limitation and to provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) -- a large-scale community-driven initiative expanding MTEB to over 500 quality-controlled evaluation tasks across 1,000+ languages. MMTEB includes a wide range of challenging novel tasks such as instruction following, long-document retrieval, and code retrieval, and represents the largest multilingual collection of evaluation tasks for embedding models to date. We use this collection to construct multiple highly multilingual benchmarks. We evaluate a representative set of models on these benchmarks.
Our findings indicate that, while LLM-based models can achieve state-of-the-art performance on a subset of languages, the best-performing publicly available model across languages is the notably smaller, multilingual-e5-large-instruct.

Massive benchmarks often impose high computational demands, limiting accessibility, particularly for low-resource communities. To address this, we downsample tasks based on inter-task correlation (i.e., selecting only a diverse set of tasks) while preserving relative rankings.
We further optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks at a significantly lower computational cost. For instance, we introduce a new zero-shot English benchmark that maintains a similar ordering at a fraction of the cost.

Published as a conference paper at ICLR 2025
MMTEB: MASSIVE MULTILINGUAL TEXT EMBED- DING BENCHMARK
Kenneth Enevoldsen 3 Isaac Chung Imene Kerboua Márton Kardos
Ashwin Mathur", David Stap", Jay Gala', Wissam Siblini? Dominik Krzeminski*,
Genta Indra Winata², Saba Sturua', Saiteja Utpala". Mathieu Ciancone', , Marion Schaeffer'.
Gabriel Sequeira', Diganta Misra5657 Shreeya Dhakal². Jonathan Rystrøm", Roman Solomatin 124 Ömer Çagatan Akash Kundu Martin Bernstorff', Shitao Xiao Akshita Sukhlecha',
Bhavish Pahwa?, Rafal Poswiata Kranthi Kiran GV Shawon Ashraf", Daniel Auras
Björn Plüster Jan Philipp Harries Loic Magne', Isabelle Mohr?, Mariya Hendriksen
Dawei Zhu Hippolyte Gisserot-Boukllee222 Tom Aarsen Jan Kostkan', Konrad Wojtasik
Taemin Lee Marek Suppa Crystina Zhang? Roberta Rocca', Mohammed Hamdy Andrianos Michail John Yang", Manuel Faysse Aleksei Vatolin Nandan Thakur
Manan Dey Dipam Vasani² Pranjal Chitale Simone Tedeschi Nguyen Tai
Artem Snegirev 38 Michael Günther', Mengzhou Xi Weijia Shi Xing Han Li Jordan Clive"!
Jordan Gayatri Krishnakumar Anna Maksimova" Silvan Wehrli Maria Tikhonova
Henil Panchal Aleksandr Abramov Malte Ostendorff Zheng Liu Simon Clematide"! Lester James Miranda" Alena Fenogenova Guangyu Song Ruqiya Bin Safi Wen-Ding Li Alessia Borghini Federico Cassano Hongjin Su Jimmy Lin Howard Yen Lasse Hansen! Sara Hooker Chenghao $3, 63, Vaibhav Adlakha 10, 54, Orion Weller 55,4 Niklas Mucmnighofff2.7..833
Síva Reddy10.54.;
Aarhus University. PIndividual Contributor, EEsker, *INSA Lyon, LIRIS, University of Amsterdam, MBZUAI, 'Jina AI, "Microsoft Research, Wikit, McGGil University, "University of Oxford, 12IMMO University, "oc University, "Heritage Institute of Technology, 15 Apart Research, BAAII
National Information Processing Institute, New York University, Elllamind,
Peking University, #CCentraleSupéle,, 22 Artefact Research Center, Hugging Face, Wroclaw University 25 Korea University, 26lluiin Technology.
Comenius University Bratislava, Cisco Systems, University of Waterloo,
Cohere For AI, University of Zurich, Stanford University, FRC CSC RAS, 34 Salesforce, 35 IT Madras, Sapienza University of Rome,
University Pennsylvania, 38 "SaluteDevicess Princeton University, University Washington, "Imppeiia College London, 42R. V. College of Engineering,
Robert Koch Institute, *HSE University, NSiima University, *OOceiglot
47 Allen Institute for AI, 48nono Labs, 49The London Institute of Banking and Finance,
50 Cornell University. 51 Noottheaattern University, Hong Kong University
Durham University, 54 S'ServiceNow Research, 55 Johns Hopkins University,
56LLIS Institute Tübingen MPPI- Tübingen Coontextual AI
$ Managing Team
Correspondence:

---

Published as a conference paper at ICLR 2025
ABSTRACT
Text embeddings are typically evaluated on a limited set of tasks, which are con- strained by language, domain, and task diversity. To address these limitations and provide more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) a large-scale, community-drive expan- sion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes diverse set of challenging, novel tasks such instruction following, Long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models date. Using this collection, we develop several highly multilingual benchmarks, which
evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-ane performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e--lan ---large-instruce with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, benchmark maintains a similar ranking order as the full-scale version but only requires 2% of the original documents vastly reducing the computational cost
newly introduced zero-shot English
INTRODUCTION
Text embeddings are used in many applications, such as semantic search (Reimers & Gurevych, 2019 Muennighoff] 2022, Hendriksen et al., 2023 Winata et al. 2023a 2024b) and classification tasks (Wang et al. 2018: 2019). Additionally, text embeddings play a crucial role in serrival--uumentee
generation (RAG, Borgeaud al. 2022 Lewis et al. (222T) and often provide significant gains in performance on low- to mid-resource languages, enabling the incorporation of previously inaccessible information. Despite the wide range of applications. there's lack of benchmarks that evaluate text embeddings across multiple domains, languages, and tasks. Existing benchmarks tend to focus on specific domains, demarcated by subject (c.g., medical, legal, fiction (Thorne al.! 2018b)) particular tasks (c.g., retrieval (Thakur et al.) 202I)). literary type (c.g., fiction, and non-fiction) or form (e.g., spoken and written). Embeddings also tend focus on a subset of languages (Nørregaard & Derczynski] 2021).
While recent efforts (Thakur et al.) 2021| Muennighoff al. 2023b Zhang et al.) 2022) have aimed to broaden the scope by encompassing more tasks, domains, or languages (Cohan et al. 2020a Wrzalik & Krechel (20T)) a large gap in language coverage remains. This work bridges this gap by creating benchmark that includes a much broader range of low- to mid-resource languages, along with broader coverage domains and task categories. To create such an expansive benchmark, initiated large-scale, open collaboration. Contributors include native speakers from diverse linguistic backgrounds, NLP practitioners, academic and industry researchers, and enthusiasts. To ensure high-quality submissions, each dataset required systematic tests, detailed metadata, and review.
The result of this extensive collaborative effort is MMTEB, the Massive Multilingual Text Embedding Benchmark, which comprises more than 500 distinet tasks across 10 task categories, covering over 250 languages, and spans a wide array of domains such as fiction, social media, medical texts, and technical programming documentation. It also integrates recent, high-quality benchmarks that test model's 's capabilities following instructions (Winata al. 2021: Weller et 2024), embedding long documents (Zhu et al., 2024), solving reasoning tasks (Xiao et al. 2024a Su et al. 2024), and
cross-lingual retrieval (Franco- Salvado et al. 2014). For an overview see Figure
MMTEB comes with open-source code and a public lcaderboard available at https///wnggnniiieeeccccssseeeee
2

---

Published as a conference paper at ICLR 2025
Semantic Text Similarity
MMTEB
Retrieval
1 342
@
C
Massive Multilingual Text Embedding Benchmark
Millligg WIN
Coloiittiod
Muhilingual
Mutti-task
Mutti-domain
Clan Ventoon
Andektted
Classification
222
324
14
languages
550 Tasks
Domains
Lang Docamane
Coms Engual
Aicras lygi bestival John
Disourne Colerr
Dullec Castitiinn
Summariaation
Clustering
Instruction Rebrieval
W
e
X
8:0
[ a
C 1
Mutti tabel-taellle Classiiccatoon 6
Pair Cassiiiiittnn
Bibest Minnng
Reranking @ to
670
X 20
C
@ 1043
12
Tryis Casaeaatio
Creas Engua
Dupiccaa Danttors
Tork waw.uag Ditection
Diphiaa Liteciio
Sclentii Swariing
An overview of MMTEB. The boxes represent the overall task categories with a sample of task categories represented within each. Blue borders represent closely-related task categories.
Figure 1:
Given the known co-occurrence of limited computational resources and low-resource languages, often referred to as the "low-resource double bind" (Ahia et al. 202T). we made it our goal to make the MMTEB benchmark accessible to low- lowrresource communities. Evaluating models extensively is often resource-intensive. For example, evaluating a single 7B large language model (LLM) on the HELM benchmark consumes over 4,000 GPU hours (Liang et al. 2022). Similarly, the English MTEB (henceforth referred to as MTEB(eng, v1)) benchmark requires up to two days of processing on single A 100 GPU even for moderately sized LLMs (Muennighofi et al. 2023b [hhhnamGhader et al. 2024). These high resource demands pose a challenge for low-resource language communities that often lack access to powerful computing resources. MMTEB addresses these challenges by expanding its coverage and optimizing the evaluation process. It significantly reduces computational cost (3.11 hours on an H100 GPU for 7B model) by using only 2% of the original documents (6% of the original number of characters) while maintaining sensitivity as a benchmark to rank models accurately.
2
MMTEB CONSTRUCTION
2.1 OPEN SCIENCE EFFORT
To ensure the broad applicability of MMTEB across various domains, we recruited diverse group of contributors. We actively encouraged participation from industry professionals, low-resource language communities, and academic researchers. To clarify authorship assignment and recognize desired contributions, we implemented a point- system, similar to Lovenia et al. (2024). To facil- itate transparency, coordination managed through GitHub. A detailed breakdown of contributors and the point system can be found in Appendix A
2.2 ENSURING TASK QUALITY
To guarantee the quality of the added tasks each contributors. In addition, we required task submissions to include metadata fields. These fields included details such annotation source, dataset source, license, dialects, and citation information. Appendix B.4 provides a comprehensive description of each field.
each task was reviewed by at least one of the main
Furthermore, we ensured that the performance on submitted tasks fell within reasonable range to avoid trivially low or unrealistically high performance. Therefore, we required two multilingual models to be run on the task; multiinngual-e5-sman & Gurevych 2019). A task was examined further if the models obtained scores close to a random baseline (within a 2% margin), near-perfect score, if both models obtained roughly similar scores.
(Wang et al. 2022) and MiniLM-L12 (Reimers)
"A task includes dataset and implementation for model evaluation.
3

---

Published as a conference paper at ICLR 2025
These tasks were examined for flawed implementation poor data quality. Afterwards, decision was made to either exclude or include the task. We consulted with contributors who are familiar with the target language whenever possible before the final decision. A task could be included despite failing these checks. For example, scores close to the random baseline might be due to the task's inherent difficulty rather than poor data quality.
2.3 ACCESSIBILITY AND BENCHMARK OPTIMIZATION
As detailed in /Section 1 extensive benchmark evaluations often require significant computational
resources. This trend is also observed in MTEE (eng, v1) [Muennighoff et al. 2023b), where running moderately sized LLMs can take up two days a single A100 GPU. Accessibility for low- resource communities is particularly important for MMTEB, considering the common co- oo-ocuurrence of computational constraints (Ahia et al., 202Tb.
Below, we discuss three main strategies implemented to make our benchmark more efficient. We additionally elaborate further code optimization in Appendix C.2
2.3.1 DOWNSAMPLING AND CACHING EMBEDDINGS
The first strategy involves optimizing the evaluation process by downsampling datasets and caching embeddings. Encoding large volume of documents for tasks such as retrieval and clustering be a significant bottleneck in evaluation. Downsampling involves selecting a representative subset the dataset and reducing the number of documents that require processing. Caching embeddings prevents redundant encoding by using already processed documents.
Clustering. In MTEB, clustering is evaluated by computing the v-measure score (Rosenberg & Hirschberg. 2007) on text embeddings clustered using k-means. This process repeated over multiple distinct sets, inevitably resulting in large number of documents being encoded. To reduce this encoding burden, we propose a bootstrapping approach that reuses encoded documents across sets. We first encode 4% subsample of the corpus and sample 10 sets without replacement. Each set undergoes k-means clustering. and we record performance estimates. For certain tasks, this approach reduces the number of documents encoded by 100x. In Appendix B.2 compare both approaches and find an average speedup of 16. I 1x across tasks, while preserving the relative ranking of models (Average Spearman correlation: 0.96).
Retrieval. A key challenge in retrieval tasks is encoding large document collections, which can contain millions of entries Nguyen et al. (2024). To maintain performance comparable to the original datasets while reducing the collection size, we adopted the TREC pooling strategy (Buckley) et al. 2007: (Soooroff & Robertson 2003). which aggregates scores from multiple models to select representative documents F For each dataset, we retained the top 250 ranked documents per query, threshold determined through initial tests that showed negligible differences in absolute scores and no changes in relative rankings across representative models (see Appendix [C.1.2 for details on downsampling effects). These documents are merged to form a smaller representative collection. For datasets exceeding 1,000 queries, we randomly sampled 1 1,000 queries, reducing the largest datasets from over million documents to a maximum of 250,000. This approach accelerated evaluation while preserving ranking performance.
Bitext Mining. We apply similar optimization to bitext mining tasks. Some datasets, such as Flores (Costa-jussà et al. (022) share the same sentences across several language pairs (e.g., English sentences are the same in the English-Hing embeddings, we reduce the number of embedding computations, making it linear in the number of languages instead of quadratic. For the English documents within Flores this results reduction of documents needed to be embedded from 410,000 in MTEE(eng v1) to just 1,012 in our benchmark.
pair and the Hangiish-Bosnian pair). By caching the
We utiliz stllized a range of models: BM25 for lexical hard negatives, c5-multilingual-lan top-performing BERT-large multilingual model, and E.-Mistra--Intrraa 7B, the largest model leveraging instruction-base data.
4

---

Published as a conference paper at ICLR 2025
2.3.2 ENCOURAGING SMALLER DATASET SUBMISSIONS
The second strategy focused on encouraging contributors to downsample datasets before submission. To achieve this, we used a stratified split based on target categories. This helped us to ensure that the downsampled datasets could effectively differentiate between candidate models. To validate the process, we compared scores before and after downsampling. For details, we refer to Appendix C.1
2.3.3 TASK SELECTION
To further reduce the computation overhead we seek to construct task subset that reliably predict task scores outside the subset.
For task selection, we followed an approach inspired by Xia et al. (2020). We seek to estimate the model my E M scores st,m; on an unobserved task based on scores on observed tasks $j.m, SS I. This allows to consider the performance of tasks features within a prediction problem. Thus we can treat task selection as feature reduction, well-formulated task within machine learning. Note that this formulation allows us to keep the unobserved task arbitrary, representing generalization to unseen tasks (Chollet (019)). We used a backward selection method, where one task left out be predicted, an estimator is fitted on the performance of all models except one, and the score of the held-out model is predicted. This process is repeated until predicted scores are generated for all models on all tasks. The most predictable task is then removed, leaving the estimators in the task subset group. Optionally, we can add additional criteria to ensure task diversity and language representation. Spearman's 's rank correlation was chosen as the similarity score, as it best preserved the relative ranking when applicd to the MTEB (eng, v1).
2.4 BENCHMARK CONSTRUCTION
From the extensive collection of tasks in MMTEB, we developed several representative benchmarks, including a highly multilingual benchmark, MMEB Minguall, as well as regional geopolitical benchmarks, MEurooe)) and MTEB Indic). Additionally, (eng, v1) (Muennighoff et al. 2023b), which we refer to as MTEB (eng, v2). MMTEB also integrates domain-specifte benchmarks like CoIR for code retrieval (Li al. 2024 and LongEmbed for long document retrieval (Zhu et al. 2024). MMTEB also introduces sooguage-speciffee benchmarks, extending the existing suite that includes Scandinavian (Enevoldsen et al. 2024), Chinese (Xiao et al.)
introduce faster version of
2024b). Polish (Poswiata et al.) 2024), and French (Ciancone et al. 2024). For an overview of the benchmarks, we refer to Appendix H.
In the following section, we detail methodology that we designed to create more targeted and concise benchmarks. This methodology includes: 1) clearly defining the initial scope of the benchmark (Initial Scope). 2) reducing the number of tasks by iterative task selection tasks based on intertask correlation (Refined Scope), and 3) performing a thorough manual review (Task Selection and Review). We provide overview in Table
In addition to these benchmarks, we provide accompanying code to facilitate the creation of new benchmarks, to allow communities and companies to create tailored benchmarks. In the following, we present MTEB (Multilingual) and MTEB (eng, v2) as two example cases. For a comprehensive overview of benchmark construction and the tasks included in each benchmark, refer to Appendix H.2
MTEB(Multilingual)) We select all available languages within MMTEB as the initial scope of the benchmark. This results in 550 tasks. We reduce this selection by removing machine-translated datasets, datasets with under-specified licenses, and highly domain-specific datasets such as code-retrieval datasets. This results in 343 tasks covering >250 languages. Following this selection, we evaluate this subset using representative selection of models (See Section [3 and apply task selection remove the most predictable tasks. To ensure language diversity and representation across task categories, we avoid removing task that would eliminate language from the respective task category. Additionally, we did not remove a task if the mean squared error between predicted
'We use the term "estimator" to differentiate between the evaluated embedding model. For our estimator, we linear regression.
5

---

Published as a conference paper at ICLR 2025
Benchmark
Initial Scope
Refined Scope
Task Selection and Review
MTEB (Mul MMMuliilingua iilingal)
>500
343
132
MTEB(Eur)pe)
420
228
74
MTEB (IEB(Indic)
55
44
23
MTEB(eng, v2)
56
54
41
Table I: Number of tasks in each benchmark after each filtering step. The initial scope includes tasks relevant to the benchmark goal, notably language of interest. The refined scope further reduced the scope, e.g. removing datasets with underspecified licenses.
and observed scores exceeded 0.5 standard deviations. This is to avoid inadvertantly overindexing to easier tasks. The process of iterative task removal (Section 2.3.3 is repeated until the most predictable held-out task obtained a Spearman correlation of less than 0.8 between predicted and observed scores, if no tasks were available for filtering. This results in a final selection of 131 diverse tasks. Finally, the selected tasks were reviewed, if possible, by contributors who spoke the target language. If needed, the selection criteria were updated, and some tasks were manually replaced with higher-quality alternatives.
MTEBBeen,, v2): Unlike the multilingual benchmarks which target a language group, this benchmark
designed match MTEB v1), incorporating computational efficiencies (see Section 2.3 and reducing the intertask correlation using task selection. To prevent overfitting, we intend it as zero-shot benchmark, excluding tasks like MS MARCO (Nguyen al., 2016) and Natural Questions (Kwiatkowski et al. 2019), which are frequently used in fine-tuning.
We start the construction by replacing each task with its optimized variant. This updated set obtains a Spearman correlation of 0.97, < .0001 (Pearson 0.99, p < .0001) with MTEB (eng, v1) using mean aggregation for the selected models (see (Subsection 3.T). The task selection process then proceeds similarly to MTEB (Multilingual)) ensuring task diversity by retaining task if its removal would eliminate a task category. Tasks, where the mean squared error between predicted and observed performance exceeds 0.2 standard deviations, are also retained. This process continues until the most predictable held-out task yields a Spearman correlation below 0.9 between predicted and observed scores. The final selection consists of 41 tasks. We compare this with (TEB v1) (Muennighofii cct al. 2023b) Section
3
EXPERIMENTAL SETTINGS
3.1 MODELS
select representative set of models, focusing on multilingual models across various size categories. We benchmark the multilingual LaBSE (Feng et al.] (022). trained on paraphrase corpora, English and multilingual versions of MPNet (Song et al. 2020). and MiniLM (Wang et al. 2021b) model, trained on diverse datasets. We also evaluate the multilingual e5 series models (Wang ct al. 2024 2022) trained using a two-step approach utilizing weak supervision. Additionally, to understand the role of scale well as instruction finetuning, benchmark M-itLM-7B (Muennighoff
jet al.] 2024) and 3--muuiiinguul-7---iie
(Wang et al. 2023), which are both based on the Mistral
7B model (Jiang al. 2023).
Revision IDs, model implementation. and prompts used are available in Appendix G We ran the models all the implemented tasks encourage further analysis of the model results. Results, including multiple performance metrics, runtime, CO2 emissions, model metadata, etc., are publicly available in the versioned results repository
3.2 EVALUATION SCORES
For our performance metrics, report average scores across all tasks, scores per task category, and weighted by task category. We compute model ranks using the Borda count method (Colombo et al.

---

Published as a conference paper at ICLR 2025
Max Tokens 100K
multiinguale----rp---needd
GrttMM778
60
LaBSE
Highly Multilingual pre- training compensates for differences model size
Mean (Task)
Bl-MHniLM-LL--2
30
100
10M
100M
18
108
Number of Parameters
Figure 2: Mean performance across tasks MTEB(Multilingual)) according to the number of parameters. The circle size denotes the embedding size, while the color denotes the maximum sequence length of the model. To improve readability, only certain labels are shown. We refer to the public leaderboard for interactive visualization. We see that the notably smaller model obtains comparable performance to Mistral 7B and GritLM-7B, note that these overlap in the figure due to the similarity of the two models.
2022), derived from social choice theory. This method, which is also employed in election systems based preference ranking, has been shown to be more robust for comparing NLP systems. To compute this score, we consider each task as preference voter voting for each model, and scores are aggregated according the Borda Count method. In the case of ties, the tournament Borda count method.
3.3 MULTILINGUAL PERFORMANCE
While MMTEB includes multiple benchmarks (see Appendix H.I], we select three multilingual benchmarks to showcase. These constitute fully multilingual benchmark MTEB (Multiling) ingual) and two targeting languages with varying levels of resources: MTEB(Europe) and MTEB (Indic). The performance of our selected models on these tasks be seen in Table 2 For performance metrics per task, across domains, etc., we refer to AAppeddix E.
ANALYSIS AND DISCUSSION
Table 2 shows the performance across the three presented multilingual benchmarks. Two trends are clearly observable;
Models trained with Intruction-tuning perform significantly better compared to those without This is especially clear when comparing the multilingual-e5-lae to its instruction-tuned counterpart Instruction tuning increases performance most drastically on bitext mining and clustering, though the effect remains pronounced across all task categories. Notably, this happens despite many tasks using generic prompts for the task category and no model-specific tuning of prompts per task. Surprisingly, multiinnual-e--laagef-innntnn Large (Conneau et al. 2019) generally outperform the considerably larger <5-mistral-7b-instruee and GriLMM7B, both of which are based Mistral-7B (Jiang al., (023). This effect is notably pronounced for mid-to-low resource languages (<300M speaker, see Appendix E.11 and likely emerges due differences in pre- mraiirain with Mistral being predominantly pre- trainedddd on English, while XLM-R targets 100 languages. All three models utilize similarly multilingual datasets for fine-tuning. However, GritL.M still remains best in class for retrieval on MTTE (ultiiinll))) has higher maximum sequence length (see Figure 2) and outperforms the cinilinguul-e---arg-nnnee on MTEB(Code) and MTEB(eng, v2).
models, based XLM-R
Discrepancies in Multilingual benchmarks ranking seem to stem from discrepancies in pre- training. While the multilingual benchmarks obtain seemingly similar performance rankings, we see to mainly stem from narrow multilingual few notable discrepancies. These discrepancies coetual-mpee--basee base) during training, resulting in focus (GritLM-77 7B, e5 e5-mistral-77 -bbitrl----instrucct multilingual-mae disproportionally higher performance the targeted languages (typically mid-high resource Euro-
7

---

Published as a conference paper at ICLR 2025
Rank (4)
Average Across
Average per Category
Model (4)
Borda Count
All
Category Bixt Pr Clf Clf STS Rirvl M. CIf Clust Rrnk
MTEB (Mul tilingual)
Number of datasets (-+)
(132)
(132)
(132)
(13) (11) (43) (16) (18)
(5)
(17)
(6)
(1375)
63.2
62.1
80.1 80.9 64.9 76.8 57.1
22.9
51.5
62.6
GritLM-7B
(1258)
70.5 79.9 73.3
22.8
50.5
c5-mistral-7b-insttu
(1233)
60.3
59.9
70.6 81.1 60.3 74.0 55.8
22.2
51.4
63.8
multilingualle5-arrg
4 (1109)
58.6
58.2
71.7 79.0 59.9 54.1
21.3
42.9
62.8
multilingual-e--base
(944)
57.0
56.5
69.4 77.2 58.271.4 52.7
20.2
42.7
60.2
mulilinggallmppett-boee
(830)
$2.0
52.1 81.2 69.7 39.8
16.4
mulillnguul-e55-mmn
(784)
55.5
55.2
67.5 76.3 56.5
49.3
19.1
41.7
60.4
LaBSE
(719)
52.1
51.9
76.4 76.0 54.6 33.2
20.1
39.2
50.2
multllingual-Mini..... Minigguuall-iiiLM-LLLLLL M--L12
(603)
48.8
48.0
44.6 79.0
$1.7 66.6
36.6
14.9
39.3
51.0
all-mpnet-base
(526)
42.5
21.2 70.9
57.0 32.8
16.3
40.8
all-Minil.M--II
11 (490)
42.2
40.9
22.9
71.7
46.8
57.2
32.5
14.6
36.8
44.3
all-Minil.M-I
12 (418)
41.4
39.9
20.1
71.2
46.2
56.1
32.5
15.1
38.0
40.3
MTEB (Europe)
Number of datasets (++)
(74)
(74)
(74)
(7) (6) (21) (9)
(15)
(2)
(6)
Grit
(757)
63.0
62.7
90.4 89.9 64.7
76.1
57.1
17.6
45.3
60.3
mulilingual----lare--nnttuu
(732)
62.2
62.3
90.4 90.0 63.2 77.4
54.8
17.3
46.9
58.4
c5-mistral-7h-instriit
(725)
61.7
61.9
89.6 91.2 62.9 76.5
53.6
15.5
46.5
mulilingual-e5--arge
(586)
58.5
58.7
84.5 88.8 60.4 75.8
50.8
15.0
38.2
55.9
Sultilingualle--buue
(499)
57.2
57.5
84.1 87.4 57.9 73.7 50.2
14.9
38.2
53.9
mulilingual-mpnet-boee
(463)
54.4
54.7
79.5 90.7 56.6 74.3
41.2
6.9
35.8
52.3
multilingual-e5
(399)
55.0
55.7
80.9 86.4 56.1
46.1
14.0
36.5
54.1
LaBSE
(358)
51.8
53.5
88.8 85.2 55.1 65.7
34.4
16.3
34.3
48.7
multilingual Mittrnuuaa--MMiiMMM- MML11
(328)
51.7
52.4
77.0 88.9 72.5 37.6
5.7
34.4
50.2
all-mpnet-base
(310)
44.7
44.7
29.8 80.5 49.2 63.9
37.3
10.9
36.2
49.6
all all-MiniLMM-U
11 (292)
44.4
44.1
32.1
81.5
49.2
64.2
36.2
7.6
32.5
49.2
all-MiniLM--LA
12 (237)
43.4
43.2
27.2 80.2
47.8
62.7
37.3
8.8
33.6
47.7
MTEB (Indic)
Number of datasets (-+)
(23)
(23)
(23)
(4)
(1)
(13)
(1)
(2)
(0)
(1)
(1)
multilingual -e-ggual-------e c5-large-instruct
(209)
70.2
71.6
80.4
76.3
67.0
53.7 84.9
51.7 87.5
moltilingual-c5-Beee
(188)
66.4
65.1
77.7
75.1
43.9 82.6
25.6 86.0
multilingual-e' base
(173)
64.6
74.2
72.8
41.8
24.6 83.8
multilingual-cS -mmall
(164)
64.7
63.2
73.7 73.8
63.8 40.8 76.8
29.1 84.4
GritLM-7B
5 (151)
60.2
58.0
58.4 67.8
27.0 79.5
28.0 84.7
e5 65-mistral-70
(144)
60.0
58.4
59.1
73.0
59.6 23.0 77.3
32.7 84.4
LaBSE
(139)
74.1
64.6
61.9 52.8
21.1 79.0
multilingual-mpal mpnet-base
(137)
58.5
55.2
44.2
82.0
61.9
34.1 57.9
32.1
74.3
multilingual- Miiiaaal--ii--aa- MiniLM-L12
9 (98)
49.7
42.2
15.3
77.8
57.6
19.8 48.8
16.7
59.3
all-mpnet-base all-MiniLM-LLI LMniiMM-L12 all-Minil M-166
10 (68)
33.6
22.6
3.7
52.6
45.2
2.5
12.9
4.0
42.6
(49)
3.5
55.0
13.9
3.7
47.6
12 (40)
31.8
20.4
2.5
53.7
44.1
-6.3
6.2
3.1
39.2
Table 2: The results for three multilingual benchmarks ranked using Borda count. We provide averages across all tasks, per task category. and weighted by task category. The task categories are shortened follows: Bitext Mining (Btxt), Pair Classification (Pr CIf), Classification (CIf), Semantic text similarity (STS), Retrieval (Rtrvl), Multilabel Classification (M. Clf), Clustering and Hierarchical Clustering (Clust) and Reranking (Rmnk). We highlight the best score in bold. Note that while Instruction retrieval (Weller et al. 2024) is included in MTEB (Europe) and MTEB (Multilingus but excluded from the average by task category due to limited model support. For broader model evaluation, refer to the public leaderboard.
pean ones). These are typically outperformed by the multilingually pre trained mutilingual----laag--innnee lower-resource languages in (TErooe)) and all languages in
8

---

Published as a conference paper at ICLR 2025
MTTEB((uropean)
MTEB(Inndic)
muhlinngall-s-log--mmmm
GeIdLM
Borda Hank
es mistra) 7b insiinttcctt
5
Su0
100M
10M
1M
100M
10M
No. Speakers
No. Speakers
Figure 3: Performance rank of top 3 multilingual models languages in MTEB(Europe) and MIEB(Indic) and by the number of native speakers. We see that Mistral-based models are outper- formed by multilingual-e---age--mnttttt fewer parameters.
lower-resource languages, despite it having substantially
Model Rank
10
12
LaBSE
mutiliinuul Mingguuul-Miiiaaaa MiMii 12
mutilingual-ed small
all-MiniL M-L6
all Ul-MiniLAd M-LML12
mulilingual wawuuuul-ppetuuuuu -uase
all (l-mpner--aa chuss
mutilinguul-e base
mutilingual waa--a----l- large
mulilingual large-------e instruct
GritL
e5-misttral 7b-instruct
Figure 4: Performance difference MTEB(eng, v1) (flag) and MTEB (MMllitMllllng (globe).
MIEB(Indic) (see Figure 3), despite being substantially smaller than Mistral models, the performance of which steadily decreases and becomes more volatile for languages with increasingly lower number of native speakers and this trade-off is well-known (Xue al. 2020).
Besides these, we observe the expected detrimental performance of English models (all-MiniLM L12, all-MiniLM-L6, all-mpnet-base) applied non-English languages and relatively high bitext
performance of LaBSE (see
Figure 4.
compare the performance of MTEB(eng, v1) and MTEB MTEB(eng, v2) in Figure 5 SJobtaining Spearman correlation of 0.90, p < 0.0001 (Pearson 0.96. p < 0.0001). For the precise scores, we refer to (Subsecttion H.3 This includes a reduction from 56 to 40 tasks along with optimized task runtime speeding up the runtime on the benchmark (3.11 hours for GritLM-7B and 0.81 I hours for Al-MMiniLM-L12 on an H100). We see that notably, the smaller English models (all-MiniLMM-12. M-1-M, all-mpnet-base perform worse the new benchmark. This is likely because they were trained on MS MARCO and Natural questions, which were removed part of the benchmark conversion zero-shot benchmark.
MTEB(eng, v1) vs. zero-shot MTEB v2)
5 RELATED WORK
Text Embedding Benchmarks. BEIR (Thakur et al. [022] pioneered the use of publicly available datasets from diverse information retrieval (IR) tasks and domains and evaluated 10 various retrieval

---

Published as a conference paper at ICLR 2025
Number of Samples (Millions)
0.75
Maintains rank while using 2% of the original documents
0.7
GritLM77B
30
0.65
C mistrallaa
MTEB(eng, V1)
Multiinggal----lnn
(instruct)
0.6
meltilingull----lgg
all-MiniLM--11
malrlinguul-e---aa
20
mukilingul----mal
all-MiniL.M-L
multlingual mpnee base
multllngual MinigMM-1i
0.45
LaBSE
Pearson r: 0.96 (p=0.00)
Spearman r: 0.90 (p=0.00)
0.55
0.5
0.65
0,7
Previous
Current
MTEB(eng, v2)
Figure 5: Performance on MTEB v1) and MTEB v2)
systems. MTEB (Muennighoff et al. 2023b) introduced a comprehensive text embedding benchmark that spans only IR but also other task categories, including clustering and re-ranking. MTEB covers a total of 58 tasks and 112 languages, though this multilinguality is mainly derived from machine-translated tasks bitext mining. Its leaderboard has grown popularity and evolved into the de facto embedding model benchmark that supports over 300 models. MIRACL (Zhang al. 2022) supports 18 languages from different language families for monolingual retrieval. MINERS (Winata et al. 2024b) is designed to evaluate the ability of multilingual LMs in semantic retrieval tasks including classification and bitext mining tasks in more than 200 languages, including code-switching. Our work extends the number of languages to over 1000 (250 excluding bitext- mining tasks), particularly to cover more low-resource languages. We also expand the MTEB's 8 embedding tasks to 10 and the 58 datasets to over 400, significantly broadening the scope of multilingual benchmarking.
Massive Collaborative Projects. Open research initiatives and participatory approaches to science have been shown to stimulate innovation (Park et al. 2003),, reduce negative biases (Gudowsky) 2021
2022), and increase the diversity of data sources (Hanley et al.) 2020 [ingh al. 2024b Gomez al.
Winata et al.
2024a). By involving diverse stakeholders, these practices enhance ethical, robust, and
reproducible research (Hagerty & Rubinov 2019). Recently, the field of natural language processing has seen growing number of community driven collaborative projects. These can be grouped into several categories. (a) Model creation, such as BLOOM (BigScience Workshop et al. 2023
-urivnn
Muennighoff et al. 2023c), StarCoder (Li et al. 2023a Lozhkov et al. 2024), Aya model (Ustin
ctt al. 2024), and Cendol (Cahyawijaya et al. 2024); (b) Datasel creation, such as NusaX (Winata et al. 2023b), OpenAssistant (Köpf et al. 2023), Nusa NusaWrites Writes (Cahyawijaya al., (223c)) and Aya dataset (Singh et al. 2024b); (c) Benchmark creation, such as BIG-BBech (Srivastava al. (202)), NusaCrowd (Cahyawijaya et al. 2023a). WorldCuisines (Winata et al., (024a). HLE (Phan et al. 2025). SEACrowd (Lovenia et al. [204]) and Eval-Hamessses (Gao et al. 2021 Ben Allal ct al.
2022 Biderman al. 2024); and (d) Other artifacts, such as NL-Augmenter (Dhole al. (021)
the Data Provenance Initiative (Longpre et al.] [2033 2024a b) or the Wikibench annotation tool (Kuo al. 2024). MMTEB expands upon earlier work within the Benchmark creation category. Our
effort significantly differs from prior collaborative benchmarks as we focus on text embeddings, use
custom point system incentivize contributions, and handle all communication openly via GitHub.
CONCLUSION
This work introduced the Massive Multilingual Text Embedding Benchmark (MMTEB), large- scale open collaboration resulting benchmark with more than 500 tasks covering more than 1000 languages. From these, we constructed three multilingual benchmarks: one fully multi-
10

---

Published as a conference paper at ICLR 2025
lingual (MTEB((ultilinguaa) and two targeting Indic (MTEB(Indic)) and European languages (MTEB(Europp)) respectively. Acknowledging that multiple additional benchmarks can be con- structed from the MMTEB additions, propose simple approach constructing benchmarks. To make these benchmarks accessible to low-resource communities, we introduced several opti- mizations by downsampling retrieval tasks using hard negative mining and bootstrapping clustering evaluation to re-use encoded documents across sets. This leads to a notable reduction in the number of text samples that need be embedded.
Our findings indicate that while large (7B) LLM-based embedding models obtain state-of-the-ar performance on the English benchmark, they are still outperformed in highly multilingual or low- resource settings by smaller models based on XLM-R Large, even when accounting for notable improvements like prompt-based embeddings.
LIMITATIONS
English Leakage. While MMTEB filters out machine-translated datasets, permits (human) translations. This inclusion leads to tasks like 318200ClusteringSSS, where labels from English samples are transferred their translations, potentially introducing bias towards English or models trained on translated content. Consequently, the benchmark may inadvertently encourage model developers to favor English or translated content by increasing their proportion in pre-training data.
Credit Assignment for Large-scale Collaborations. One of MMTEB's s goals was highlight the benefits of collaboration. The managing group believes the point system successfully defined contribution terms but acknowledges it isn't perfect. For instance, equal points were awarded for dataset submissions regardless of effort-some datasets were readily available, while others needed significant work like reformulation, HTML parsing, and multiple review rounds.
Languages Representation. While the benchmark includes over 250 languages and 500 tasks, the distribution is skewed toward high-resource languages (see Figure 6), with low-resource languages being better represented in specific task categories like bitext-mining and classification. We encourage future collaborations to fill these gaps and enhance language diversity in the collection.
Number Tasks
Language
Figure 6: Number of tasks per language. For readability, we remove English (290 tasks) and only plot the 100 languages with the most tasks.
ETHICAL CONSIDERATIONS
We acknowledge the environmental impact of the benchmark that stems from the compute needed across tasks. As such, emissions tracking is added using codecarbon (Courty al. (024) to measure kilograms of CO2-equivalents (CO2eg) and estimate the carbon footprint per task. The benchmark is collaborative project and contains datasets of different data quality and origin. Thus, additional efforts are still required to identify and minimize biases in the benchmark datasets.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lec. Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. arXiv preprint M.XXv:23099074455 2023a.
David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nix- dorf, Chris Chinenye Emezue, sana al azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsuddeen Hassan Muhammad, Teshome Mulugeta Ababu, Saheed Abdullahi Salahudeen, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Abdulmumin, Mahlet Taye, Oluwabusayo Awoyomi, Iyanuoluwa Shode, Tolulope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adeeko, Abeeb Afolabi, Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wangari Kimotho, Onyekachi Ogbu, Chinedu Mbonu, Chiamaka Chuk- wuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola Awosan, Tadesse Kebede, Toadoum Sari Sakayo, Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf, Mardiyyah Oduwole, Tshinu Tshinu, Ussen Kimanuka, Thina Diko, Siyanda Nxakama, Sinodos Nigusse, Abdulmejid Johar, Shafie Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, Jules Jules, Ivan Ssenkungu, and Pontus Stenetorp. Masakhanews: News topic classification for african languages, 2023b.
Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. Semeval-2012 task 6: pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics Volume Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, SemEval '12, pp. 385-393, USA, 2012. Association for Computational Linguistics.
Eneko Agirre, Daniel Matthew Cer, Mona T. Diab, Aitor Monnzalez-Agirre, and Weiwei Guo. *sem 2013 shared task: Semantic textual similarity. In International Workshop on Semantic Evaluation, 2013. URL htths:ts:://ptp httt:://app..smantiicccooo ar warg/CorrpuII... 10241043
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, al. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pp. 252-263, 2015.
Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. Lince: A centralized benchmark for linguistic code-switching evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 1803-1813, 2020.
Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. The low-resource double bind: An empirical study of pruning for low-resource machine translation. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3316-3333, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v 1/2021 wwt.222...iiingseemeeee findings-emmlp.222. URL https::/acllntthoooy.ooot 2rt.:///nnnooooooooo///22/11 www./iii...iiifssss... Londings--enll... 282
Vesa Akerman, David Baines, Damien Daspit, Ulf Hermjakob, Tacho Jang, Colin Leong, Michael Martin, Joel Mathew, Jonathan Robie, and Marcus Schwarting. The ebible corpus: Data and model benchmarks for bible translation for low-resource languages. arXiv preprint arXiv:2304.090 2023.
Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados, Leonardo Neves, Vitor Silva, and Francesco Barbieri. Twitter Topic Classification. In Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju. Republic of Korea, oct 2022. International Committee on Computational Linguistics.
Gaurav Arora. iNLTK:
Natural language toolkit for indic languages. In Eunjeong L. Park, Masato Hagiwara, Dmitrijs Milajevs, Nelson F. Liu, Geeticka Chauhan, and Liling Tan (eds.). Proceedings of Second Workshop for NLP Open Source Software (NLP-OSE) WOLP- pp. 66-71, Online, nov 2020. Association for Computational Linguistics. doi: 18.18653/v 1/2855//112022mmlossss 1.10. URL https
//acl //acllatthlooyy.or//2222 01t.2ss11ppsss1111101
12

---

Published as a conference paper at ICLR 2025
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross- coossllngual transferability of monolin- gual representations. CoRR, 0tss/191..11856, 2019.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprini arXiv:2108.0775 077332. 2021.
Soran Badawi, Arefeh Kazemi, and Vali Rezaie. Kurdisent: a corpus for kurdish sentiment analysis. Language Resources and Evaluation, pp. 1 1-20, 01 2024. doi: 10.10077/10579-023-000
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. arXiv preprini arXiv:2308. 16884, 2023.
Anil Bandhakavi, Nirmalie Wiratunga, Deepak P, and Stewart Massie. Generating word-emotion lexicon from #emotional tweets. In Johan Bos, Anette Frank, and Roberto Navigli (eds.), Pro- ceedings of the Third Joint Conference on Lexical and Compulational Semanticy (*SEM 2014), pp. 12-21, Dublin, Ireland, aug 2014. Association for Computational Linguistics and Dublin City University. doi: 10.3115/v1/S14-10000 1002. URL http::///aacaaattooooooorr//////
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Colladon XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirfeenth Language Resources and Evaluation Conference, pp. 258-266 Marseille, France, jun 2022. European Language Resources Association. URL httpst http///aaaaatttca http///aaaaannnoo..aoeoo wwtp:y/..rr///222.../// 1rec-1..eee1122
Parishad Behnam Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint XXiv:2444.00561 2024.
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models, 2022. URL hnttps://githi hrtp://eerrroeeett/oiooedeeeeaaa
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In International Conference on Learning Representations, 2020.
Paheli Bhattacharya, Kripabandhu Ghosh, Saptarshi Ghosh, Arindam Pal, Parth Mehta, Arnab Bhattacharya, and Prasenjit Majumder. Aila 2019 precedent & statute retrieval task, oct 2020.
URL https:///oi..... org/10. 5281/zenodo. 4063986
Ergun Biçici. RTM-DCU: Predicting semantic similarity with referential translation machines. In Preslav Nakov, Torsten Zesch, Daniel Cer, and David Jurgens (eds.). Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pp. 56-63, Denver, Colorado, jun 2015, Association for Computational Linguistics. doi: 18853/v1/S11-20000 URL https wwwnthhlogg..org/S1551
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.1477 14782, 2024.
Teven Le Scao BigScience Workshop, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammana- manchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, al. Bloom: A 176b-parameter open-access multilingual language model, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
13

---

Published as a conference paper at ICLR 2025
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean- Saan-Bptiste Lespiau. Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens, 2022.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. A full-text learning rank dataset for medical information retrieval. 2016. URL http://www. cl. uni-hhidee Iberg .e/
Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen Voorhees. Bias and the limits of pooling for large collections. Information retrieval, 10:491-508, 2007.
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji. Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, et al. Nusacrowd: Open source initiative for indonesian nlp resources. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13745-125 13818, 2023a.
Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Dea Adhista, Emmanuel Dave, Sarah Oktavianti, Salsabil Akbar, Jhonson Lee, Nuur Shadieq, Tjeng Wawan Cenggoro, Hanung Linuwih, Bryan Wilie, Galih Muridan, Genta Winata, David Moeljadi, Alham Fikri Aji, Ayu Purwarianti, and Pascale Fung. Nusawrites: Constructing high-quality corpora for underrepresented and extremely low-resource languages. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 921-9445 Nusa Dua, Bali, nov 2023b. Association for Computational Linguistics. URL https:tt://// https:://clannthologg org org/2023.1jjnlpp 1iip2:iiiinll-mii... 60)
Chupter
Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Dea Adhista, Emmanuel Dave, Sarah Oktavianti, Salsabil Akbar, Jhonson Lee. Nuur Shadieq, Tjeng Wawan Cenggoro, al. Nusawrites: Con- structing high-quality corpora for underrepresented and extremely low-resource languages. In Proceedings of the 13th International Joint Conference on Natural Language Processing and Chapter of the Association for Computational Linguistics
the 3rd Conference of the ( (Volume I: Long Papers). pp. 921 9455,945,225 2023c.
Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Rifki Afina Putri, Emmanuel Dave, Jhonson Lee, Nuur Shadieq, Wawan Cenggoro, Salsabil Maulana Akbar, Muhammad Ihza Mahendra, al. Cendol: Open instruction-tunee generative large language models for indonesian languages. arXiv preprint ar arXiv:2404.061 2024.
lñigo Casanueva, Tadas Temeinas, Daniela Gerz, Matthew Henderson, and Ivan Vuli'c. Efficient intent detection with dual sentence encoders. In Tsung-Hsien Wen, Asli Celikyilmaz, Zhou Yu, Alexandros Papangelis. Mihail Eric, Anuj Kumar, Iñigo Casanueva, and Rushin Shah (eds.), Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pp. 38 45, Online, jul 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. nlp4convai-1.5 1.5. URL
wtgg//20...oo.o orgg2022.nnp44oovaii111 nlp.conviiiii.55
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens (eds.), Proceedings of the 1 1ith International Workshop on Semantic Evaluation (SemEval- 2017), pp. 1-14, Vancouver, Canada, aug 2017. Association for Computational Linguistics. doi: http:////lllaanttooo.oorr.. wrwpey/-gg///1121200
10.18653/v1/S17-2000
Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. Multieurlex a multi- lingual and multi-label legal document classification dataset for zero-shot cross- -ingull transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2021. URI. httttttts///:xiiv wwt.://rg///bb///eee//o
00904.
Amit Kumar Chaudhary. Kurt Micallef, and Claudia Borg. Topic classification and headline generation for Maltese using public news corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. Association for Computational Linguisties, may 2024.
14

---

Published as a conference paper at ICLR 2025
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, et al. Evaluating large language models trained on code, 2021.
Xi Chen, Ali Zeynali, Chico Camargo, Fabian Fl'ock, Devin Gaffney, Przemyslaw Grabowicz. Scott Hale, David Jurgens, and Mattia Samory. SemEval-2022 task 8: Multilingual news article similarity. In Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, and Shyam Ratan (eds.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) pp. 1094-1106, Seattle, United States, jul URL
2022. Association for Computational Linguistics. doi: 10. httpo:://ac.antiooooo..oog
1e
François Chollet. On the Measure of Intelligence. arXiv:1911.015477 [es), November 2019. URL arXiv: 1911.01547.
http:///rrxiv.rrg/abs//1111 01547
Mathieu Ciancone, Imene Kerboua, Marion Schaeffer, and Wissam Siblini. Extending the massive text embedding benchmark to french, 2024.
cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and nithum. Jig- saw unintended bias in toxicity classification, 2019. URL httpps://kgglle...... http://aaglle.../cooooo///////// coom/competiiiic tions// Digssw-unintended-buee in- http://aaaai.otooooooooaaaaaaaaa
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1 2018.
Benjamin Clavié. Jacolbert and hard negatives, towards better japanese- first embeddings for retrieval: Early technical report, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Arman Cohan, Sergey Feldman, Beltagy, Doug Downey, and Daniel Weld. Specter: Document- level representation learning using citation-informel transformers, 2020a.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. Specter: Document-
level representation learning using citation-informed transformers. In ACL, 2020b.
Pierr Colombo, Nathan Noiry, Ekhine Irurozki, and Stephan Clémençon. What are the best systems? new perspectives on nlp benchmarking. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 26915-26932. Curran Associates, Inc., 2022. URL httppss://procceedigg...nt neuripss wwwpeewwwws...ppeeeeeeeeeeeeeeee cc/paper filess/paerr/002//f55 http:///2222fi1lee///////// 0at.://0000000201111110115300000
pdf
Tatoeba community. Tatoeba: Collection of sentences and translations, 2021.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv: /91102/911.0//////011/ 16, 2019.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672 2022.
Benoit Courty, Victor Schmidt, Goyal-Kamal, MarionCoutarel. Boris Feld, Jérémy Lecourt, LiamCon- nell, SabAmine, inimaz, supatomic, Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Amine Saboni, Hugues de Lavoreille, Niko Laskaris, Edoardo Abati, Douglas Blank, Ziyao Wang, Armin Catovic, alencon, Michat Stęchly, Christian Bauer, Lucas-Otavies JPW. and MinervaBooks, mlco2/codeccrbon:: 12.4.1, May 2024. URL httpp:://oii.om org/ w2wp://00022////.1.101010.
15

---

Published as a conference paper at ICLR 2025
Mathias Creutz. Open subtitles paraphrase corpus for six languages, 2018.
Slawomir Dadas, Michat Perelkiewicz, and Rafat Poświata. Evaluation of sentence representa- tions in Polish. In Nicoletta Calzolari, Fr' ed eric B echet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, H'elène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceed- ings of the Twelfth Language Resources and Evaluation Conference, pp. 1684-1680, Marseille, France, may 2020. European Language Resources Association. ISBN 979- 10-95546--444-4 URL httpss//aclaattholoyy.rrrg orgph///ooo...o./22200/0 1rec-1.200
Slawomir Dadas. Training effective neural sentence encoders from automatically mined paraphrases, 2022.
David Davis. Swahili: News classiffction dataset (0.2). Zenodo, 2020. doi: URL https:://oi..rg//////// 5201/////o/2/eeeoo525530 5514420
Nisansa de Silva. Sinhala text classification: Observations from the perspective of a resource poor language. Year of Publication, 2015.
Leon Derczynski and Alex Speed Kjeldsen. Bornholmsk natural language processing: Resources and tools. In Proceedings of the Nordic Conference of Computational Linguistics (2019), pp. 344. Linköping University Electronic Press. URL hhttps://pure http::///uur............ itu. wtw..///w///1eee//5551111001
6138. .138.pdf
Ameet Deshpande, Carlos Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Rajpurohit, Ashwin Kalyan, Danqi Chen, and Karthik Narasimhan. Csts: Conditional semantic textual similarity. arXiv preprint arXiv:2305.15000 15093, 2023.
Swapnil Dhanwal, Hritwik Dutta, Hitesh Nankami, Nilay Shrivastava, Yaman Kumar, Junyi Jessy Li, Debanjan Mahata, Rakesh Gosangi, Haimin Zhang, Rajiv Ratn Shah, and Amanda Stent. annotated dataset of discourse modes in Hindi stories. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, may 2020. European Language Re- sources Association. ISBN 979-10-95546-34-4 TRL/https: wwww.aclwebb.or/aattho..oo 2020. 1re.-1.14111999
Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, al. NI-augmenter: framework for task-sensitive natural language augmentation. arXiv preprint arXiv:2112.0277 2021.
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. Climate-fever: A dataset for verification of real-world climate claims, 2021.
Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff, and Kristoffer Laigaard Nielbo. The scandinavian embedding benchmarks: Comprehensive assessment of multilingual and monolingual text embedding. arXiv preprint arXivv2406.02396, 2024.
Alexander R Fabbri, Wojeiech Kry'sci'nsse nski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626, 2020.
Christian Federmann, Tom Kocmi, and Ying Xin. NTREX- 128 news test references for MT evaluation of 128 languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pp. 21 24,24, Online, nov 2022. Association for Computational Linguistics. URL https://aclanttoloog) www.oo/....2...ooo sumeval-1.4
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic BERT sentence embedding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Dublin, Ireland, May 2022. Association for Computational Linguis- tics. doi: 10.18653/v1l /2022.2222222 acl-long.66 URL https: ftttpss://cclaathoooogooor///
1: Long Papers), pp.
acl-long
62
16

---

Published as a conference paper at ICLR 2025
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist. Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. Massive: A Im-example multilingual natural language understanding dataset with 51 typologically-diverse languages, 2022.
Wikimedia Foundation. Wikimedia downloads. URL htttps://dlmps..wkiiimt .rg
Marc Franco-Salvador, Paolo Rosso, and Roberto Navigli. A knowledge-based representation for cross-language document retrieval and categorization. In Shuly Wintner, Sharon Goldwa- ter, and Stefan Riezler (eds.), Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 414 Gothenburg, Sweden, April 2014, Association for Computational Linguistics. doi: 10.3115/v1/E14-144 URL https waathology.orr/E111.
Jay Gala, Pranjal A Chitale, A K Raghavan, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar M, Janki Atul Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M Khapra, Raj Dabre, and Anoop Kunchukuttan, Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. Trunsactions on Machine Learning Research, 2023. ISSN 2835-8856. URL /ttps://openrevieeet net/forum?
id=vfT4YuZAYA
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL. https: / //doi. org/ / 5281/zenodo 5371628
Gregor Geigle, Nils Reimers, Andreas Ruuckl'e, and Iryna Gurevych. Tweac: Transformer with extendable qa agent classifiers. arXiv preprint, abs/2104.07081, 2021. hRtp:ptp:/arxxiv.rr 0t5/21104.07881
Tsvetanka Georgieva-Trifonoe Milena Stefanova, and Stefan Kalchev. Dataset for "Customer Feedback Text Analysis for Online Stores Reviews in Bulgarian", 2018. URL /ttttts:///iiorr// 10. 0w..790/DDV//TXXSS TXIK9P
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1-9, Prague, jun 2007. Association for Computational Linguistics. URL https:://claatthol.oo org/
Hippolyte Gisserot-Boukhle Manuel Faysse, Emmanuel Malherbe, Céline Hudelot, and Pierre Colombo. Towards trustworthy reranking: simple yet effective abstention mechanism. arXiv preprint arXiv:2402.129977 2024.
Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. Building large monolingual dictionaries the leipzig corpora collection: From 100 to 200 languages. In Proceedings of the Eighth International Conference Language Resources and Evaluation 12), 2012.
Charles J Gomez, Andrew C Herman, and Paolo Parigi. Leading countries in global science increasingly receive more citations than other countries doing similar research. Nature Human Behaviour, 6(7):919-929, 2022.
Matilde González, Clara García, and Lucía Sánchez. Diabla: A corpus of bilingual spontaneous written dialogues for machine translation. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4192 4198, 2019.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc' Aurelio Ranzato, and Francisco Guzm an. The flores- 101 evaluation benchmark for low- fow-resource and multilingual machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 19-35, 2022.
Niklas Gudowsky. Limits and benefits of participatory agenda setting for research and innovation. European Journal of Futures Research, 9(1):8, 2021.
17

---

Published as a conference paper at ICLR 2025
Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas- Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore. Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay. Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. Legalbench: collaboratively built benchmark for measuring legal reasoning in large language models, 2023.
Ren'e Haas and Leon Derczynski. Discriminating between similar Nordic languages. In Marcos Zampieri, Preslav Nakov, Nikola L.uubsii' c, Jrorg Tiedemann, Yves Scherrer, and Tommi Jauhi- ainen (eds.), Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pp. 67-75, Kiyv, Ukraine, apr 2021. Association for Computational Linguistics. URL https::/aclanthiie
orgoorg 2021. vardial-1.8
Ivan Habernal, Tom's as Pt' acek, and Josef Steinberger. Sentiment analysis in Czech social media using supervised machine learning. In Alexandra Balahur, Erik van der Goot, and Andres Montoyo (eds.), Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 65-74, Atlanta, Georgia, jun 2013. Association for Computational Linguisties. URL httpt:ttttcccanttttttt
org/w13. 3-
Alexa Hagerty and Igor Rubinov. Global ai ethics: review of the social impacts and ethical implications of artificial intelligence, 2019.
Margot Hanley, Apoorv Khandelwal, Hadar Averbuch-Elor, Noah Snavely, and Helen Nissenbaum. ethical highlighter for people-centric dataset creation. arXiv preprint arXiv:2011.1358, 2020.
Mariya Hendriksen, Svitlana Vakulenko, Ernst Kuiper, and Maarten de Rijke. Scene-centric vs. object-centric image-text cross- retrieval: reproducibility study. In European Conference on Information Retrieval, pp. 68-85. Springer, 2023.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurlPS, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fisth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
Søren Vejlgaard Holm. Are gllms danoliterate? benchmarking generative nlp in danish. 2024.
Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. Cqadupstack: A benchmark data set research. In Proceedings of the 20th Australasian Document Compuring Symposium (ADCS), ADCS 15, pp. 3:1-3:8, New York, NY, USA, 2015. ACM. ISBN 978-1-4553-4440-3 doi: 10. (0.1145/2838931 2838934. URI. http:://oi. acm. wrw.10.gg/10o..... / 1145/
for community
2838931 2838934
Christoph Hoppe, David Pelkmann, Nico Migenda, Daniel Hötte, and Wolfram Schenck. Towards intelligent legal advisors for document retrieval and question-answering in german legal docu- ments. In 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pp. 29-32, 2021. doi: 10.1109/AIKE55661 1.22:/22221122220021 20201000111.
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. Cosqa: 20,000+ web queries for code search and question answering, 2021. URL https
//arxiv.org/abset 23235
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. CodeScarchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.094433 090909909446.... 2019.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint 22.02///2.091 18, 2021.
18

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b, 2023.
Dame Jovanoski, Veno Pachovski, and Preslav Nakov. Sentiment analysis in Twitter for Macedonian. In Ruslan Mitkov, Galia Angelova, and Kalina Bontcheva (eds.), Proceedings of the International Conference Recent Advances in Natural Language Processing, pp. 249-257, Hissar, Bulgaria, sep 2015. INCOMA Ltd. Shoumen, BULGARIA. URL/Ittpss//ccLanttolooy/ org/R15- 1034
Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. HAGRID: A human-Ilm collaborative dataset for generative information-seeking with attribution. arXiv:2307. 16883, 2023,
Jenna Kanerva, Filip Ginter, Li-Hsin Chang, liro Rastas, Valtteri Skantsi, Jemina Kilpel" ainen, Hanna- Mari Kupari, Jenna Saarni, Maija Sev' on, and Otto Tarkka. Finnish paraphrase corpus. In Simon Dobnik and Lilja Øvrelid (eds.). Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 288 298, Reykjavik, Iceland (Online)) 2021. Link "oping University Electronic Press, Sweden. URL
org/ 2021. www.noolllooooo iida moddalid-mmain222 main. 29
Jiwon Kim and Won Ik Cho. Kocasm: Korean automatic https:/gittaa..co//pelllnn Yourkoo You/korean-sarcasm, 2019.
sarcasm detection.
Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. Ai4bharat-indicnlp corpus: Monolingual corpora and word embed- dings for indic languages. arXiv preprint arXiv:2005, 00085, 2020.
Tzu-Sheng Kuo, Aaron Lee Halfaker, Zirui Cheng, Jiwoo Kim, Meng- Hsin Wu, Tongshuang Wu, Kenneth Holstein, and Haiyi Zhu. Wikibench: Community-drivee data curation for ai evaluation on wikipedia. Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI '24, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400003300. doi:
URL https: /tw.or//////// 1...11166 0115/361390// 3642278
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Kenton Lee, et al. Natural ques- tions: a benchmark for question answering research, 2019. /tRllhttss//aacaanthoooo org/ Q19-
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment, 2023.
Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. A continuously growing dataset of sentential paraphrases. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 1234, Copenhagen, Denmark. sep 2017. Association for Computational Linguistics. doi: 10. 10.116533/1/D1171111 URL https aoo.lannhooogy.org//1111--
Ken Lang. Newsweeder: Learning to filter netnews. In Armand Prieditis and Stuart Russell (eds.). Machine Learning Proceedings 1995, pp. 331-339. Morgan Kaufmann, San Francisco (CA), 1995. doi: https://doi http:////ooorr//1/.////// wtt.:/...//1111///1.... 1016//978- 1-55860-3377-6550044888 URL. / http://wwww..eeeeeeeeeeceooo/oca 7tt.
ISBN 978-1-55860-37776
Chankyu Lee, Rajarshi Roy. Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Ny-embed: Improved techniques for training I1ms as generalist embedding models, 2024.
Jean Lee, Taejun Lim, Heejun Lee, Bogeun Jo, Yangsok Kim, Heegeun Yoon, and Soyeon Caren Han. K-MHaS: A multi-label hate speech detection dataset in Korean online news comment, In Proceedings of the 29th International Conference on Computational Linguistics. pp. 3530-3538, Gyeongju, Republic of Korea, oct 2022. International Committee on Computational Linguistics. col col.og2...lnnn-.1.1.
URL
19

---

Published as a conference paper at ICLR 2025
Antoine Seffbvrre-Brossatc Stephane Gazaille, and Michel C. Desmarais. Alloprof: new french question-answer education dataset and its use in an information retrieval case study, 2023. URL hhtts:://axxi..rr//bbp
07738
Joao Augusto Leite, Diego F. Silva, Kalina Bontcheva, and Carolina Scarton. Toxic language detection in social media for brazilian portuguese: New dataset and multilingual analysis. CoRR, abs/2010.04543, 2020. URL httpp:::/aaxiv.oog///aaaa//
04543
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evalu- ating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475, art. arXiv: 1910.07475, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeened generation for knowledge-intensive nlp tasks, 2021. URL https //axxiv.org/bbs2200. 11401.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Maiil
Philipp Schmid, Sylvain Gugger, Clement Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, and Thomas Wolf. Datasets: A community library for natural language processing, CoRR, 02422009002446 2021. URL httpss://axxi.... org/ /
2109. 02846)
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2950-2962, Online, apr 2021, Association for Computational Linguistics. doi: 10. 15665//112001..cc-mmaa.2233 URL
org.:///0oooo..222221111 eacl- 257
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you!, 2023a.
Xiangyang Li, Kuicai Dong. Yi Quan Lee, Wei Xia, Yichun Yin, Hao Zhang, Yong Liu, Yasheng Wang, and Ruiming Tang. Coir: A comprehensive benchmark for code information retrieval models, 2024. URL
http://arx.v..orr////////0 24888
Yudong Li, Yuqing Zhang, Zhe Zhao, Linlin Shen, Weijie Liu, Weiquan Mao, and Hui Zhang. Csl: A large-scale chinese scientific literature dataset, 2022.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang, Towards general text embeddings with multi-stage contrastive learning, 2023b. /ttpss//:///axxv abs/2308 03281
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint 1:X.::2211.091111 2022.
Daniele Licari, Praveen Bushipaka, Gabriele Marino, Giovanni Comand'e e, and Tommaso Cucinotta. Legal holding extraction from italian case documents using italian-legal- bert text summarization. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL '23, pp. 148-156, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701979. doi: 10.1145/559436335551777 URL httpp:://ooi: wrw.oorg///1/. B3951777
Zi Lin. Zihan Wang. Yongqi Tong, Yangkun Wang. Yuxin Guo, Yujia Wang. and Jingbo Shang. Toxiechat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023.
20

---

Published as a conference paper at ICLR 2025
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Maruu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, Deb Roy. and Sara Hooker. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai, 2023.
Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Mare Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep. Ahmad Anis, An Dinh, Caroline Chitongo, Da Yin, Damien Silco, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Sandy Pentland. Consent in crisis: The rapid decline of the data commons. 2024a. URL https://arxiv wwwg/bs/////oooo.oo...oo.g..grr 2407. 14933
Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Manan Dey, Mohammed Hamdy, Nayan Saxena, Ahmad Mustafa Anis, Emad A. Alghamdi, Vu Minh Chien, Naana Obeng-Marnu, Da Yin, Kun Qian, Yizhi Li, Minnie Liang, An Dinh, Shrestha Mohanty, Deividas Mataciunas, Tobin South, Jianguo Zhang, Ariel N. Lee, Campbell S. Lund. Christopher Klamm, Damien Silco, Diganta Misra, Enrico Shippole, Kevin Klyman, Lester JV Miranda, Niklas Muennighoff, Seonghyeon Ye, Scungone Kim, Vipul Gupta, Vivek Sharma, Xuhui Zhou, Caiming Xiong, Luis Villa, Stella Biderman, Alex Pentland, Sara Hooker, and Jad Kabbara. Bridging the data provenance gap across text, speech and video, 2024b. URL
Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno Kampman, et al. Seacrowd: A multilingual multimodal data hub and benchmark suite for southeast asian languages. arXiv preprint arXiv:2406. 10118, 2024.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain. Yixuan Su, Xuanli Hc, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff. Xiangru Tang. Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024. URL hhtp:::tt///att //arxiv wwteriivorrgg org/ abs/ abs/2402. 2402. 19173
Xing Han Lu, Siva Reddy, and Harm de Vries. The StatCan dialogue dataset: Retrieving data tables through conversations with genuine intents. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2799-2829, Dubrovnik, Croatia, may 2023. Association for Computational Linguistics. http:/ttps///xxxi.rr//////a///
01412
Xing Han LU, Zdeněk Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue, 2024.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, jun 2011. Association for Computational Linguistics. URL htLLhtpss:/ httpss:///claatthooogoo ort/
Yash Madhani, Mitesh M. Khapra. and Anoop Kunchukuttan. Bhasa- Chass-abbijnaanam. sctivt and romanized language identification for 22 Indic languages. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the s1 Annual Meeting of the Association
21

---

Published as a conference paper at ICLR 2025
for Computational Linguistics (Volume 2: Short Papers), pp. Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: 10.18653/w1/2023... 10.1655/v112022.clshhhhoo/ acl-short.71. URL https: https://acllamtooooyy.orr///////
acl shl-short.71
Andani Madodonga, Vukosi Marivate, and Matthew Adendorff. lzindaba- Machine learning
news categorisation for long and short text for isizulu and siswati. 4, Jan. 2023. doi: 10.55492/dhasa.
URL BRRLhttps:/ https::/upjournalll
ww. ac wwo..aa/innee.. httpa////.nnc.p.oo///////////a/a
Wei Chen Maggie, Phil Culliton. Tweet sentiment extraction, 2020. URL https:///agglee.co
commetitions/ttweer sentt-eeniii ientt-meetteee sentimeeteextaaattiii
Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan, Fahrurrozi Rahman, and Clara Vania. IndoNLI:
A natural language inference dataset for Indonesian. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 10511-10527, Online and Punta Cana, Dominican Republic, nov 2021. Association for Computational Linguistics. URLhtttss eem.ie-maalllll 821
//aclanthology.orga
Arthur Malajyan, Karen Avetisyan, and Tsolak Ghukasyan. Arpa: Armenian paraphrase detection corpus and models, 2020.
Malo, A. Sinha, P. Korhonen, Wallenius, and Takala. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (IREC' 14), pp. 216- 223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http://wwwy http:/wwwweeccooom wtt.:/oooooo.ooceeiiiiiiiee wwtp:///oooeeeeeeeee///e/11111// http://e/e////3/PPpPPaaeee
Vukosi Marivate, Moseli Mots' Oehli, Valencia Wagner, Richard Lastrucci, and Isheanesu Dzingirai. Puoberta: Training and evaluation of a curated language model for setswana. In SACAIR 2023 (To Appear), 2023.
Philip May. Machine translated multilingual sts benchmark dataset. 2021. URL https:///ithhub hoom/PPiliiphayyyttb-mllll----i
Philip May, Brooke Fujita, and Tom Aarsen. stsb-multi-mt, 2021. URL https://github.. PhilipMay/stsb-mulzz GitHub repository.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr- embeedingg-mistral.ennaama mesarele text retrieval with transfer learning. Salesforce Al Research Blog, http://f--emmeedddddiittterre
2024. URL hhttts:://llll wwwesfooreaiieseeeeeeeeeeeeeeeee
Yev Meyer, Marjan Emadi, Dhruv Nathawani, Lipika Ramaswamy, Kendrick Boyd, Maarten Van Seg- broeck, Matthew Grossman, Piotr Mlocek, and Drew Newberry. Synthetic Text-To- To-SL: A syn- thetic dataset for training language models to generate sql queries from natural language prompts, April 2024. http::
ace. co.
Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. Spartqa: A textual question answering benchmark for spatial reasoning. In Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pp. 4582-4598, 2021.
Julius Monsen and Arne onsoso... A method for building non-english corpora for abstractive text summarization. In Proceedings of CLARIN Annual Conference, 2021.
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search, 2022.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07 07 2023a.
22

---

Published as a conference paper at ICLR 2025
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text em- bedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014-2037, Dubrovnik, Croatia, May 2023b. Association for Computational Linguistics. doi: wttp:////1/223.aalmmmma 148. URL httpps://aclaanthooogg.../ www.ogyy.org//....o/./// aee23eeecl-miin 148
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf. Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning, 2023c.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning, 2024.
Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ifeoluwa Adelani, Seid Muhie Yimam, Ibrahim Sa id Ahmad, Meriem Beloucif, Saif Mo- hammad, Sebastian Ruder, Oumaima Hourrane, Pavel Brazdil, Felermino D' D'ario M' ario Ant Ant'onio onio Ali, Davis Davis, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Belay, Wendimu Baye Messelle, Hailu Beshada Balcha, Sisay Adugna Chala, Hagos Tesfahun Gebremichael, Bernard Opoku, and Steven Arthur. Afrisenti:a twitter sentiment analysis benchmark for african languages. 2023.
Timo Möller, Julian Risch, and Malte Pietsch. Germanquad and germandpr: Improving non-english question answering and passage retrieval, 2021.
Jørgen Johnsen Navjord and Jon-Mikkel Ryen Korsvik. Beyond extractive: advancing abstractive au- tomatic text summarization in norwegian with transformers. Master's thesis, Norwegian University of Life Sciences, As, 2023.
Thong Nguyen, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. Multimodal learned sparse retrieval with probabilistic expansion control. In European Conference on Information Retrieval, pp. 448 464. Springer, 2024.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng, MS MARCO: A human generated machine reading comprehension dataset. CoRR, 0bs//611.00926. 2016. URL htttp://arxiv.oog//ab///ba 09268
Dan Nielsen. ScandEval: A benchmark for Scandinavian natural language processing. In Tanel Alum"ae ae and Mark Fishel (eds.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 185-201, T orshavn, Faroe Islands, may 2023. University of Tartu Library. URL httppsss//alaaattooooyy//// wwwp:yy..rrr.o..o..oee modaalioddlida--a 1.20
Joel Niklaus, Matthias Stürmer, and Ilias Chalkidis. An empirical study cross-x transfer for legal judgment prediction, 2022.
Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, and Natalia Silveira. Universal dependen- cies vl: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (1REC'16), pp. 1659-1666, 2016.
Jeppe Nørregaard and Leon Derezynski. DanFEVER:
claim verification dataset for Danish. In Simon Dobnik and Lilja Øvrelid (eds.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 422 428, Reykjavik, Iceland (Online), 2021. oping University Electronic Press, Sweden. httphhttt////lnttttii
wwwp:rgg/2000 2021 moda lidaaladdmminnn 47
Maciej Ogrodniczuk and Mateusz Kope'c. The Polish summaries corpus. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.). Proceedings of the Ninth International Conference Language Resources and Evaluation (LREC'I pp. 3712-377 37155 Reykjavik, Iceland, may 2014. European Language Resources Association (ELRA). URL http: : www. Irec- conf .ogg/ ppoceddiigg//ree2211//////////2/
pdf
23

---

Published as a conference paper at ICLR 2025
Maciej Ogrodniczuk and Lukasz Kobyliński (eds.). Proceedings of the PolEval 2019 Workshop. Warsaw, Poland, 2019. Institute of Computer Science, Polish Academy of Sciences. ISBN 978-83- 63159-28-3, URL ptttp:/22011 woweaol...lllllleooooal.l....... piteeeaa.l//illlss///eeaaaa///
James O'Neill, Polina Rozenshtein, Ryuichi Kiryo. Motoko Kubota, and Danushka Bollegala. I wish I would have loved this one. but I didn't - a multilingual dataset for counterfactual detection in product review. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing. pp. 7092-7106 Online and Punta Cana, Dominican Republic, nov 2021. Association for Computational Linguistics. doi: 10.186553 http:////eeeemmmmaaaeeemmm Lemnlp-main.566. URL https //aclanthology.orr org/ www.e2111111..11mmmmmmmmm ewn.l-mmaiiii.. 568
Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji. Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, et al. Semrel2024: A collection of semantic textual relatedness datasets for 14 languages. arXiv preprint arXiv:2402.0866 2024.
Hille Pajupuu, Jaan Pajupuu, Rene Altrov, and Kairi Tamuri. Estonian Valence Corpus / Eesti valentsikorpus. 11 2023. doi: 10.6084/m9 htt4///9..igsaaee2425555 11:e222517754.. v054. URL https:://fgghhareeeeer com/ Valence_Corpus wwt.eeiii.ialeettsoooorsr/s/111.
articles/dataast/Esttonn
Christos Papaloukas. Ilias Chalkidis, Konstantinos Athinaios, Despina Apppans-Athaassia Pantazi, and Mano- lis Koubarakis. Multi-granulat legal topic classification on greek legislation. In Proceedings of the Natural Legal Language Processing Workshop 202/, pp- 63-75, Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 00.48550/rrxiw22199..5 URL http::////xxivvvv.. http:s///arrll.oorraa//////// 15298
Shantipriya Parida, Sambit Sekhar, Soumendra Kumar Sahoo, Swateek Jena, Abhijeet Parida, Satya Ranjan Dash, and Guneet Singh Kohli. Odiagenai: Generative and Ilm initiative for 2023.
the odia language. https//huuginggtee.o//oCCCCCCCo
Michael Park, Erin Leahey, and Russell Funk. Papers and patents are becoming less disruptive over time. Nature, 613: 13444,,, 2023. URL https: www.stiicccoollaaat. wtgg/orrgssiilTCC
255466666
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnay Chopra, Adam Khoja, Ryan Kim, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Daron Anderson, Tung Nguyen, Mobeen Mahmood, Fiona Feng, Steven Y. Feng, Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Jessica P. Wang, Pawan Kumar, Oleksandr Pokutnyi, Robert Gerbicz, Serguei Popov, John- -lark Levin, Mstyslav Kazakov, Johannes Schmitt, Geoff Galgon, Alvaro Sanchez, Yongki Lee, Will Yeadon, Scott Sauers, Mare Roth, Chidozie Agu, Søren Riis, Fabian Giska, Saiteja Utpala, Zachary Giboney, Gashaw M. Goshu, Joan of Are Xavier, Sarah-Jane Crowson, Mohinder Maheshbhai Naiya, Noah Burns, Lennart Finke, Zerui Cheng. Hyunwoo Park, Francesco Fournier-Facio, John Wydallis, Mark Nandor, Ankit Singh, Tim Gehrunger, Jiaqi Cai, Ben McCarty, Darling Duclosel, Jungbac Nam, Jennifer Zampese, Ryan G. Hoerr, Aras Bacho, Gautier Abou Loume, Abdallah Galal, Hangrui Cao, Alexis C Garretson, Damien Sileo, Qiuyu Ren, Doru Cojoc, Pavel Arkhipov, Usman Qazi, Lianghui Li, Sumeet Motwani, Christian Schroeder de Witt, Edwin Taylor, Johannes Veith, Eric Singer, Taylor D. Hartman, Paolo Rissone, Jaehyeok Jin, Jack Wei Lun Shi, Chris G. Willcocks, Joshua Robinson, Aleksandar Mikov, Ameya Prabhu, Longke Tang, Xavier Alapont, Justine Leon Uro, Kevin Zhou, Emily de Oliveira Santos, Andrey Pupasov Maksimov, Edward Vendrow, Kengo Zenitani, Julien Guillod, Yuqi Li, Joshua Vendrow, Vladyslav Kuchkin, Ng Ze- An, Pierre Marion, Denis Efremov, Jayson Lynch, Kaiqu Liang, Andrew Gritsevskiy, Dakotah Martinez, Ben Pageler, Nick Crispino, Dimitri Zvonkine, Natanael Wildner Fraga, Saeed Soori, Ori Press, Henry Tang, Julian Salazar, Sean R. Green, Lina Brüssel, Moon Twayana, Aymeric Dieuleveut, T. Ryan Rogers, Wenjin Zhang, Bikun Li, Jinzhou Yang. Arun Rao, Gabriel Loiseau, Mikhail Kalinin, Marco Lukas, Ciprian Manolescu, Subrata Mishra, Ariel Ghislain Kemogne Kamdoum, Tobias Kreiman, Tad Hogg, Alvin Jin, Carlo Bosio, Gongbo Sun, Brian P Coppola, Tim Tarver, Haline Heidinger, Rafacl Sayous, Stefan Ivanov. Joseph M Cavanagh, Jiawei Shen, Joseph Marvin Imperial, Philippe Schwaller, Shaipranesh Senthilumma, Andres M Bran, Ali Dehghan, Andres Algaba, Brecht Verbeken, David Noever, Ragavendran P V, Lisa Schut, Ilia
24

---

Published as a conference paper at ICLR 2025
Sucholutsky, Evgenii Zheltonozhskii, Derek Lim, Richard Stanley, Shankar Sivarajan, Tong Yang, John Maar, Julian Wykowski, Martí Oller, Jennifer Sandlin, Anmol Sahu, Yuzheng Hu, Sara Fish, Nasser Heydari, Archimedes Apronti, Kaivalya Rawal, Tobias Garcia Vilchis, Yuexuan Zu, Martin Lackner, James Koppel, Jeremy Nguyen, Daniil S. Antonenko, Steffi Chemn Bingchen Zhao, Pierrot Arsene, Alan Goldfarb, Sergey Ivanov, Rafal Poswiata, Chenguang Wang, Daofeng Li, Donato Crisostomi, Andrea Achilleos, Benjamin Myklebust, Archan Sen, David Perrella, Nurdin Kaparov, Mark H Inlow, Allen Zang, Elliott Thornley, Daniil Orel, Vladislav Poritski, Shalev Ben-David, Zachary Berger, Parker Whitfill, Michael Foster, Daniel Munro, Linh Ho, Dan Bar Hava, Aleksey Kuchkin, Robert Lauff. David Holmes, Frank Sommerhage, Keith Schneider, Zakayo Kazibwe, Nate Stambaugh, Mukhwinder Singh, Ilias Magoulas, Don Clarke, Dae Hyun Kim, Felipe Meneguitti Dias, Veit Elser, Kanu Priya Agarwal, Victor Efren Guadarrama Vilchis, Immo Klose, Christoph Demian, Ujjwala Anantheswaran, Adam Zweiger, Guglielmo Albani, Jeffery Li, Nicolas Daans, Maksim Radionov, Václav Rozhoñ, Ziqiao Ma. Christian Stump, Mohammed Berkani, Jacob Platnick, Volodymyr Nevirkovets, Luke Basler, Marco Piccardo, Ferenc Jeanplong, Niv Cohen, Josef Tkadlec, Paul Rosu, Piotr Padlewski, Stanislaw Barzowski, Kyle Montgomery, Aline Menezes, Arkil Patel, Zixuan Wang, Jamie Tucker-Foltz, Jack Stade, Tom Goertzen, Fereshteh Kazemi, Jeremiah Milbauer, John Arnold Ambay, Abhishek Shukla, Yan Carlos Leyva Labrador, Alan Givré, Hew Wolff, Vivien Rossbach, Muhammad Fayez Aziz, Younesse Kaddar, Yanxu Chen, Robin Zhang, Jiayi Pan, Antonio Terpin, Niklas Muennighoff, Hailey Schoelkopf, Eric Zheng, Avishy Carmi, Adam Jones, Jainam Shah, Ethan D. L. Brown, Kelin Zhu, Max Bartolo. Richard Wheeler, Andrew Ho, Shaul Barkan, Jiaqi Wang. Martin Stehberger, Egor Kretov, Kaustubh Sridhar, Zienab EL- Wasif, Anji Zhang, Daniel Pyda, Joanna Tam, David M. Cunningham, Vladimir Goryachev, Demosthenes Patramanis, Michael Krause, Andrew Redenti, Daniel Bugas, David Aldous, Jesyin Lai, Shannon Coleman, Mohsen Bahaloo, Jiangnan Xu, Sangwon Lee, Sandy Zhao, Ning Tang, Michael K. Cohen, Micah Carroll, Orr Paradise, Jan Hendrik Kirchner, Stefan Steinerberger. Maksym Ovchynnikov, Jason O. Matos, Adithya Shenoy, Benedito Alves de Oliveira Junior, Michael Wang. Yuzhou Nie, Paolo Giordano, Philipp Petersen, Anna Sztyber-Betley Priti Shukla, Jonathan Crozier, Antonella Pinto, Shreyas Verma, Prashant Joshi, Zheng-Xin Yong, Allison Tee, Jérémy Andréoletti, Orion Weller, Raghav Singhal, Gang Zhang, Alexander Ivanov, Seri Khoury, Hamid Mostaghimi, Kunvar Thaman, Qijia Chen, Tran Quoe Khánh, Jacob Loader, Stefano Cavalleri, Hannah Szlyk, Zachary Brown, Jonathan Roberts, William Alley, Kunyang Sun, Ryan Stendall, Max Lamparth, Anka Reuel, Ting Wang. Hanmeng Xu, Sreenivas Goud Raparthi, Pablo Dmitry Malishev, Thomas Preu, Tomek Korbak, Marcus Abramovitch, Dominic Williamson, Ziye Chen, Biró Bálint, M Saiful Bari, Peyman Kassani, Zihao Wang, Behzad Ansarinejad, Laxman Prasad Goswami, Yewen Sun, Hossam Elgnainy, Daniel Tordera, George Balabanian, Earth Anderson, Lynna kvistad, Alejandro José Moyano, Rajat Maheshwari, Ahmad Sakor, Murat Eron, Isaac C. McAlister, Javier Gimenez, Innocent Enyekwe, Andrew Favre D. O., Shailesh Shah, Xiaoxiang Zhou, Firuz Kamalov, Ronald Clark, Sherwin Abdoli, Tim Santens, Khalida Meer, Harrison K Wang, Kalyan Ramakrishnan, Evan Chen, Alessandro Tomasiello, G. Bruno De Luca, Shi-Zhuo Looi, Vinh-Kha Le, Noam Kolt, Niels Mündler, Avi Semler, Emma Rodman, Jacob Drori, Carl Fossum, Milind Jagota, Ronak Pradeep, Honglu Fan, Tej Shah, Jonathan Eicher, Michael Chen, Kushal Thaman, William Merrill, Carter Harris, Jason Gross, llya Gusev, Asankhaya Sharma, Shashank Agnihotri, Pavel Zhelnov, Siranut Usawasutsakom, Mohammadreza Mofayezi, Sergei Bogdanov, Alexander Piperski. Mare Carauleanu, David K. Zhang, Dylan Ler, Roman Leventov, Ignat Soroko. Thorben Jansen, Pascal Lauer, Joshua Duersch, Vage Taamazyan, Wiktor Morak, Wenjie Ma, William Held, Tran Duc Huy. Ruicheng Xian, Armel Randy Zebaze, Mohanad Mohamed, Julian Noah Leser, Michelle X Yuan, Laila Yacar, Johannes Lengler, Hossein Shahrtash, Edson Oliveira, Joseph W. Jackson, Daniel Espinosa Gonzalez, Andy Zou, Muthu Chidambaram, Timothy Manik, Hector Haffenden, Dashiell Stander, Ali Dasouqi, Alexander Shen, Emilien Duc. Bita Golshani, David Stap, Mikalai Uzhou, Alina Borisovna Zhidkovskaya, Lukas Lewark, Mátyás Vincze, Dustin Wehr, Colin Tang. Zaki Hossain, Shaun Phillips, Jiang Muzhen, Fredrik Ekström, Angela Hammon, Oam Patel, Nicolas Remy, Faraz Farhidi, George Medley, Forough Mohammadzadeh, Madellene Peñaflor, Haile Kassahun, Alena Friedrich, Claire Sparrow, Taom Sakal, Omkar Dhamane, Ali Khajegili Mirabadi, Eric Hallman, Mike Battaglia, Mohammad Maghsoudimehrabami,, Hieu Hoang, Alon Amit, Dave Hulbert, Roberto Pereira, Simon Weber, Stephen Mensah, Nathan Andre, Anton Peristyy, Chris Harjadi, Himanshu Gupta, Stephen Malina, Samuel Albanie, Will Cai, Mustafa Mehkary, Frank Reidegeld, Anna- Katharina Dick, Cary Friday, Jasdeep Sidhu, Wanyoung Kim, Mariana Costa, Hubeyb Gurdogan, Brian Weber,
Freddie Martin,
25

---

Published as a conference paper at ICLR 2025
Harsh Kumar, Tong Jiang, Arunim Agarwal, Chiara Ceconello, Warren S. Vaz, Chao Zhuang, Haon Park, Andrew R. Tawfeek, Daattavya Aggarwal, Michael Kirchhof, Linjie Dai, Evan Kim, Johan Ferret, Yuzhou Wang. Minghao Yan, Krzysztof Burdzy, Lixin Zhang, Antonio Franca, Diana T. Pham, Kang Yong Loh, Joshua Robinson, Shreen Gul, Gunjan Chhablani, Zhehang Du, Adrian Cosma, Colin White, Robin Riblet, Prajvi Saxena, Jacob Votava, Vladimir Vinnikov, Ethan Delaney, Shiv Halasyamani, Syed M. Shahid, Jean-Christophe Mourrat, Lavr Vetoshkin, Renas Bacho, Vincent Ginis, Aleksandr Maksapetyan, Florencia de Rosa, Xiuyu Li, Guillaume Malod, Leon Lang. Julien Laurendeau. Fatimah Adesanya, Julien Portier, Lawrence Hollom, Victor Souza, Yuchen Anna Zhou, Yigit Yalı,, Gbenga Daniel Obikoya, Luca Amaboldi, Rai, Filippo Bigi, Kaniuar Bacho, Pierre Clavier, Gabriel Recchia, Mara Popescu, Nikita Shulga, Ngefor Mildred Tanwie, Thomas C. H. Lux, Ben Rank, Colin Ni, Alesia Yakimchyk, Huanxu, Liu, Olle Häggström, Emil Verkama, Himanshu Narayan, Hans Gundlach, Leonor Brito-Santana, Brian Amaro, Vivek Vajipey. Rynaa Grover, Yiyang Fan, Gabriel Poesia Reis e Silva, Linwei Xin, Yosi Kratish, Jakub Lucki, Wen-Ding Li, Justin Xu, Kevin Joseph Scaria, Freddic Vargus, Farzad Habibi, Long. Lian, Emanuele Rodolà, Jules Robins, Vincent Cheng, Declan Grabb, Ida Bosio, Tony Fruhauff, Ido Akov, Eve J. Y. Lo. Hao Qi, Xi Jiang, Ben Segev, Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Michael P. Brenner, Mao Mao, Yibo Jiang, Xinyu Zhang, David Avagian, Eshawn Jessica Scipio, Muhammad Rehan Siddiqi, Alon Ragoler, Justin Tan, Deepakkumar Patil, Rebeka Plecnik, Aaron Kirtland, Roselynn Grace Montecillo, Stephane Durand, Omer Faruk Bodur. Zahra Adoul, Mohamed Zekry, Guillaume Douville, Ali Karakoe, Tania C. B. Santos, Samir Shamseldeen, Loukmane Karim, Anna Liakhovitskaia, Nate Resman, Nicholas Farina, Juan Carlos Gonzalez, Gabe Maayan, Sarah Hoback, Rodrigo De Oliveira Pena, Glen Sherman, Hodjat Mariji, Rasoul Pouriamanesh, Wentao Wu, Gözdenur Demir, Sandra Mendoza, Ismail Alarab, Joshua Cole, Danyelle Ferreira, Bryan Johnson, Hsiaoyun Milliron, Mohammad Safdari, Liangti Dai, Siriphan Arthornthurasuk, Alexey Pronin, Jing Fan, Angel Ramirez- -riniezd,TTiiddd,, Ashley Cartwright, Daphiny Pottmaier, Omid Taheri, David Outevsky, Stanley Stepanic. Samuel Perry, Luke Askew, Raúl Adrián Huerta Rodríguez, Abdelkader Dendane, Sam Ali. Ricardo Lorena, Krishnamurthy Iyer, Sk Md Salauddin, Murat Islam, Juan Gonzalez, Josh Ducey, Russell Campbell, Maja Somrak, Vasilios Mavroudis, Eric Vergo, Juehang Qin, Benjámin Borbás, Eric Chu, Jack Lindsey, Anil Radhakrishnan, Antoine Jallon, I. J. McInnis, Alex Hoover, Sören Möller, Song Bian, John Lai, Tejal Patwardhan, Summer Yue, Alexandr Wang, and Dan Hendrycks, Humanity's last exam, 2025. URL https://arxiv http:://arriv.oog//a///551.1 14249
Lidia Pivovarova, Ekaterina Pronoza, Elena Yagunova, and Anton Pronoza. Paraphraser: Russian paraphrase corpus and shared task. In Conference on artificial intelligence and natural language, pp. 211-225. Springer, 2017.
Rafat Poáwiata, Stawomir Dadas, and Michat Perelkiewicz. PL-MTEB: Polish Massive Text Embed- ding Benchmark. arXiv preprint arXiv:2405. 2024.
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages. Trunsactions of the Association for Compu- tational Linguistics, 10: 162, 02 2022. ISSN 2307-387X. doi: 10. 10.11622 1,2/t6c1ccc__.. a 00452. URL https://doi. orw.eii..rg///..
Nils Reimers and Iryna Gurevych. Sentence-hert Sentence embeddings using siamese bert-networks, 2019.
Nils Reimers, Philip Beyer, and Iryna Gurevych. Task-oriented intrinsic evaluation of semantic textual similarity. In Yuji Matsumoto and Rashmi Prasad (eds.), Proceedings of COLING 2016, the 26th International Conference on Computationall Linguisties: Technical Papers, pp. 87 96, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https www.aantoooogoorrg/cceeeo 1009.
Neil Christian R. Riego, Danny Bell Villarba, Ariel Antwaun Rolando C. Sison, Fernandez C. Pineda, and Herminiño C. Lagunzad. Enhancement to low low-resource text classification via sequential transfer learning. United International Journal for Research and Technology, 04:72-82,, 2023.
26

---

Published as a conference paper at ICLR 2025
Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushmant Kyle Lo. Ian Soboroff, Ellen Voorhees, Lucy Lu Wang, and William R Hersh. Searching for scientific evidence in a pandemic: An overview of tree-covid, 2021.
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer ai complete question answering: A set of prerequisite real tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5722-8731, 2020.
Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In Jason Eisner (ed.), Proceedings of the 2007 Joint Conference on Empir- ical Methods in Natural Language Processing and Computational Natural Language Learning SEMNLP-CONLLA pp. 410--200 Prague, Czech Republic, June 2007. Association for Computa- tional Linguistics. URL https: pottps::/acclla //apa////o/l .og.llloooooooooooooooooooo wawpeyooog///000000 1044
Paul R Rottterr Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pier- rehumbert. HateCheck: Functional tests for hate speech detection models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Con- ference on Natural Language Processing (Volume I: Long Papers), pp. 41-58, Online, aug 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 22233/11/221...looonog....4 URL
https://aclanthoooo...r////// wttp:/orr/2221.lll11111 acl- 1ong-
Ivan Rybin, Vladislav Korablinov, Pavel Efimov, and Pavel Braslavski. Ruby 2.0: An innovated russian question answering dataset. In ASWC, pp. 532-547, 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Common- sense reasoning about social interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural 2019.
Language Processing (EMNLP-IJCNLP), pp.
Salim Sazzed. Cross-lingual sentiment classification in low-resource bengali language. In Proceedings of the Sixth Workshop on Noisy User-generated Tex (W-NUT 2020), pp. 50-60, 2020.
Alexander Shoev, Aleksandr Naumov, and Roman Rybka. Data-driven model for emotion detection in russian texts. Procedia Computer Science, 190:637-642, 2021.
Konstantinos Sechidis, Grigorios Tsoumakas, and Ioannis Vlahavas. On the stratification of multi- label data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, pp. 158. Springer, 2011.
Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov. Adversarial domain adaptation for duplicate question detection. Ellen Riloff, David Chiang, Julia Hockenmaier. and Jun ichi Tsuji (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1056-1063, Brussels, Belgium, 2018 Association for Computational Linguistics. doi: 10. 1tt.1166////111--- 131. URL (ttps:: /etttp::/aaclanth..... ogo.oo///////111111111
Zareen Sharf. Roman Urdu Data Set. http:://doo.or/11.24433//
UCI Machine Learning Repository, 2018. DOI:
Eva Sharma, Chen Li, and Lu Wang. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. CoRR, 2e5/19060037411 2019. UKL/itpp//arrrl.or///////9900 03741
Tatiana Shavrina, Alena Fenogenova, Anton Fmelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Eylampiev. Russiansuperglue: A russian language understanding evaluation benchmark. arXiv preprint arXiv:2010.15925, 2020.
Emily Sheng and David Uthus. Investigating societal biases in poetry composition system, 2020.
27

---

Published as a conference paper at ICLR 2025
Iyanuoluwa Shode, David Ifeoluwa Adelani, Jing Peng, and Anna Feldman. Nollysemti: Leveraging transfer learning and machine translation for nigerian movie sentiment classification. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 986-998, 2023.
Harman Singh, Nitish Gupta, Shikhar Bharadwaj. Dinesh Tewari, and Partha Talukdar. Indicgenbench: A multilingual benchmark evaluate generation capabilities of Ilms indic languages, 2024a.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemińísi,, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzich Fadace, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024b.
Artem Snegirev, Maria Tikhonova, Anna Maksimova, Alena Fenogenova, and Alexander Abramov. The russian embedders' exploration: rumteb benchmark and russian embedding model design, 2024. URL htttp:://arxxw.org//bbsss
12503]
Vésteinn Snæebaannason, Annika Simonsen, Goran Glavaš, and Ivan Vulić. Transfer to low-resource language via close relatives: The case study faroese. In Proceedings of the 24th Nordic Conference on Computational Linguisties (NoDaLiDa), Tórshavn, Faroe Islands, may 22-24-22222222222 2023, Link oping University Electronic Press, Sweden.
Ian Soboroff and Stephen Robertson. Building a filtering test collection for trec 2002. In Proceed- ings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pp. 243-250, 2003.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Van Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33: 16857-16867, 2020.
Gizem Sogancioğlu, Hakime Özüürk, and Arzucan Özgür. BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, 33(14))449--58 07 2017. ISSN 1367- 4803. doi: 10. 1.l199/bioinfommacisttt222 URL http://///oo...or//1.111//////// btx238
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Carriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
Michal Stef"anik, Marek Kadlc'ık, ik, Piotr Gramacki, and Petr Sojka. Resources and few-shot learners for in-context learning in slavic languages. arXiv preprint arXiv:2304.01922, 2023.
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, and Tao Yu. Bright: A realistie and challenging benchmark for reasoning-intensive retrieval, 2024. URL htttt::/// http:/aaww.....ggbbbaaa/a/a/aaee wttp/b////////////eoo 2407. 12883
Piotr Szymański and Tomasz Kajdanowicz. A network perspective on stratification of multi- label data. In Paula Branco Luis Torgo and Nuno Moniz (eds.), Proceedings of the First In- ternational Workshop on Learning with Imbalanced Domains: Theory and Applications, vol- ume 74 of Proceedings of Machine Learning Research, pp. 22-35. PMLR, 22 Sep 2017. URL https://proceedings. mwr.mmrrr.. wtt.....ee/////////s/s..5...... 17a.lsiikki1a..... html
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. Towards benchmarking and improving the temporal reasoning capability of large language models. arXiv preprint arXiv:2306. 08955, 2023.
Nandan Thakur, Nils Reimers, Andreas R d'uckl'e. e, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. htRl//htts::////m
28

---

Published as a conference paper at ICLR 2025
"Nandan Thakur, Luiz Bonifacio, Maik Fr"obe, Alexander Bondarenko, Ehsan Kamalloo, Martin Potthast, Matthias Hagen, and Jimmy Lin". "systematic evaluation neural retrieval models on the Touch's e 2020 argument retrieval subset of BEIR". In "Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval", 2024.
James Thome, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 809-819, New Orleans, Louisiana, jun 2018a. Association for Computational Linguistics. doi: (0.18653/v1/N18. 1074. URL
James Thome, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification, 2018b.
J'org Tiedemann and Santhosh Thottingal. -p building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT), 2020.
Herbert Ultrich, Jan Drehal, Martin Rýpar, Hana Vincourov'a, a, and V'aclav Moravec. Csfever and ctkfacts: acquiring czech data for fact verification. Language Resources and Evaluation, 57(4): 1571-1605, 2023.
Elena Volodina, Yousuf Ali Mohammed, and Julia Klezl. Dalaj a dataset for linguistic acceptability judgments for swedish: Format, baseline, sharing, 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.18655/11/1885446 URL
httts:://ac.attioooog.ooor/// wrwp:/hooooogoo.//////o/ 5444
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. CoRR, 00s/1995.00537, 2019. URL http:::/aax....rg/////// wwtp:/iiii.oo//bbbba. /
0905.00537
Kexin Wang, Nils Reimers, and Iryna Gurevych. Tsdae: Using transformer-basee sequential denoising auto-encoderfor unsupervised sentence embedding learning. arXiv preprint arXiv:2104.06979,
2021a. URL /ttps://:///r abgp/////////////// 06979
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2022.
Liang Wang, Nan Yang, Xiaolong Huang. Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00366 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilin- gual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672, 2024.
Wenhui Wang. Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. MiniLMv2: Multi- head self-attention relation distillation for compressing pretrained transformers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Com- putational Lingnistics: ACL-IJCNLP 2021, pp. 2140-21511 Online, August 2021b. Associ- URL https
ation for Computational Linguistics. doi: 10. http:///11/2211..nnngn-------/ TTaclanthology.orgg//// www.fiiinnnaaaeee 188
Silvan Wehrli, Bert Amrich, and Christopher Irrgang. German text embedding clustering benchmark, 2024. httpht//s///xxi http://.wggg/bbb/////eo 02700
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching information retrieval models to follow instructions, 2024.
29

---

Published as a conference paper at ICLR 2025
Orion Weller, Benjamin Chang, Eugene Yang, Mahsa Yarmohammadi, Sam Barham, Sean MacA- vaney, Arman Cohan, Luca Soldaini, Benjamin Van Durme, and Dawn Lawrie mfollowir: a multilingual benchmark for instruction following in retrieval, 2025. URL https://arxiv.orr abs/2501. 19264
Andika William and Yunita Sari. Click-id: A novel dataset for indonesian clickbait headlines. Data in Brief, 32:106231, 2020. ISSN 2352-3409. doi: http:://bi..org//1.006/......i
106231, URL
com/ http://ccc...cccoooc.ccooooooaaa
Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Yifan Gao, and Daniel Preotiuc-Pietre Efficient zero-shot cross- Pross-lingual inference via retrieval. In Proceedings of the 13th International Joint Conference Natural Language Processing and the 3rd Conference of the Asia-Pucijte Chupter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 93-104, 2023a.
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. Language models are few-shot multilingual learners. In Proceedings of the 1st Workshop Multilingual Representation Learning, pp. 1-15, 2021.
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. Nusax: Multilingual parallel sentiment dataset for 10 indonesian local languages, 2022.
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, et al. Nusax: Multilingual parallel sentiment dataset for 10 indonesian local languages. In Proceedings of the 17th Conference of the European Chupter of the Association for Computational Linguislics, pp. 815-834, 2023b.
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, et al. Worldcuisines: A massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines. arXiv preprinl arXiv:2410. 220.222050 2024a.
Genta Indra Winata, Ruochen Zhang, and David Ifeoluwa Adelani. Miners: Multilingual language models as semantic retrievers. arXiv preprint arXiv:2406.07424, 2024b.
Marco Wrzalik and Dirk Krechel. GerDaLIR: A German dataset for legal information retrieval.
In Proceedings of the Natural Legal Language Processing Workshop 2021, pp. 123-128, Punta Cana, Dominican Republic, nov 2021. Association for Computational Linguistics. URLlhttps
//aclanthology.co wrgg/222....nnn
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. MIND: A large-scale dataset for news recommen- dation. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3597-3606, Online, jul 2020a. Association for Computational Linguistics. doi: 10. 00.18653/v1/2020..o acl-main.333 URL https://aclantholoogyo wttp./rrr.......mmmaa.a...... 331
2020
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. Mind: A large-scale dataset for news recommendation. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 3597-3606, 2020b.
Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. Pre- dicting performance for natural language processing tasks. CoRR, abs/2005. 00870, 2020. URL https://arxiv. org. abs/ 2005. 00870
Chenghao Xiao, G Thomas Hudson, and Noura Al Moubayed. Rar-b: Reasoning as retrieval benchmark. arXiv preprint wwXiv:2404.06344 2024a.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
30

---

Published as a conference paper at ICLR 2025
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 641-649, 2024b.
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiang- sheng Li, Haitao Li, Yiqun Liu, and Jin Ma. T2ranking: A large-scale chinese benchmark for passage ranking, 2023.
Wei Xu, Chris Callison- Callison-Burch, and Bill Dolan. SemEval-200 2015 task 1: Paraphrase and semantic similarity Twitter (PIT). In Preslav Nakov, Torsten Zesch, Daniel Cer, and David Jurgens (eds.), Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pp. 1-11, Denver, Colorado, jun 2015. Association for Computational Linguisties. doi: 10.18653/v1/ S15-2001. httpthtttss//laatttttntnn
org www.oorgg/////// S102
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1 /194, 2020.
Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. Codetransocean: A comprehen- sive multilingual benchmark for code translation, 2023. URL httpssss//arrivv.og////////33 04951
Hitomi Yanaka and Koji Mineshima. Compositional evaluation on japanese textual entailment and similarity. Transactions of the Association for Computational Linguistics, 10: 10:1266-1284 2022.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. Paws-x: A cross-lingual adversarial dataset for paraphrase identification, 2019.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning- HotpotQA: dataset for diverse, explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369,2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 1/D////////1///1/1111 URL httptttss//aaaanthllgggooo/ http://cclanttooooo.rrr///////// www/ 1259
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 2e..22399.2222,,, 12284, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. Making miracl: Multilingual information retrieval across continuum of languages, 2022.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso Hermelo, Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Compula- lional Linguisticss 11:1114-1133 09 2023. ISSN 2307-387X. doi: 10. /0. 010.:..112666611 wt//////acclee a05995
URL
https://doi.. org/10.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterppeeee:: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. /4658, 2024.
31

---

Published as a conference paper at ICLR 2025
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Longembed: Extending embedding models for long context retrieval. arXiv preprint arXiv:2404. 12000 2024.
Elena Zotova, Rodrigo Agerri, Manuel Nuñez, and German Rigau. Multilingual stance detection in tweets: The Catalonia independence corpus. Nicoletta Calzolari, Fr'ed' eric B B'echet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, H' elène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 1368-1375. European Language Resources Association, may 2020. ISBN 979-10-95546-34-4
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. In Serge Sharoff, Pierre Zweigenbaum, and Reinhard Rapp (eds.), Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pp. 60-67, Vancouver, Canada, aug 2017. Association for Computational Linguistics. doi: 10.18653/v1/WW11-5555 URL
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei Vin Ko, Daniel D souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model, 2024.
Lukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymczak, Marcin Watroba, Arkadiusz Janz, Piotr Szymański. Mikolaj Morzy, Tomasz Kajdanow- iez, and Maciej Piasecki. This is the way: designing and compiling lepiszcze, a comprehensive nlp benchmark for polish, 2022. UtLhttss://arxx www.//orgaaaooee orgg/bs//25 abs/2211.13112
Michal Štefánik, Marek Kadlčík, Piotr Gramacki, and Petr Sojka. Resources and few-shot learners for in-context learning in slavic languages, 2023.
32

---

Published as a conference paper at ICLR 2025
APPENDIX TABLE OF CONTENTS
Contributions
33
Overview and Construction of Tasks
35
Introduction benchmark tasks
37
2
Task construction
38
Novel datasets
39
B 4
Task Metadata
40
B.4.J Domains
40
Benchmark Optimizations
42
Speeding Up Tasks
42
C.1.1 Clustering
42
Retrieval
44
C.2 Code Optimizations
45
D Task Overview
46
DJ Tasks
46
D.2 Languages
46
D.3 Examples
48
Full results
49
Performance per Number Speakers
50
New Metrics
50
F.1
Abstention for retrieval and reranking tasks
50
Models
51
Benchmark Construction and Overview
Benchmark creation
51
2
Benchmark task overview
52
H.3
Performance on Meng, v2)
53
Performance on MTEB(Code)
A
CONTRIBUTIONS
We list the contributions of every author in Table The possible types of contributions and their associated points
New dataset: A new dataset includes creating implementation (subclass) of task using new dataset. points were awarded for implementing the task and points for each new language introduced by the task.
33

---

Published as a conference paper at ICLR 2025
New task: An implementation of a new task category such multi-label classification instruction retrieval. 2 points were given for a new task, as well as points following adding a dataset.
Annotations: Many existing datasets were not annotated with proper metadata. To encourage high-quality annotations we awarded point for each full dataset annotation.
Fixes: These included bug fixes, usability fixes, speed improvements and more. For these, we typically awarded 2-10 points depending on the size of the contribution.
Running Models: This includes both running and implementing models for MMTEB. We typically awarded point per model
full set of relevant tasks. Relevant tasks for specific model are limited to those pertinent to its language. For instance, a Russian model does not need be run on French tasks.
Review PR: A large part of ensuring good dataset quality comes from the dataset review. We award points for a review. a PR had multiple reviewers, points were awarded to each. Often reviewers finalized dataset additions, helped with data formatting, and resolving bugs. In many cases, adding 2 points for review was considered either too low (a perfect PR with little no corrections) or high (lengthy discussion examining dataset quality, debugging implementations and more), however average we believe it was appropriate.
Writing: At this point many of the authors writing the paper already qualified for co- authorship and thus had reasonable experience with the MMTEB point system. Thus, it was generally possible to discuss a reasonable amount of points based on the efforts made in earlier stages.
Coordination: Included Coordination of contributors and initial idcation were given points at the end of the project based on relative effort. These points were given, similar to paper writing, based relative effort.
A total of 10 points had to be obtained to be invited as a co-author. To see each contribution mapped to specific PRs, see ht /gi thtb. thub. com/ embeddings /arcm//an///tteeree////////////a points, where the name of JSON files corresponds to the PR id.
Table 3: Contributions by GitHub users. See Table for the mapping between authors and GitHub handles
Total
Bug fixes
New datasse
Paper writing
Cooedination
Running Models
KemnethFeevvideen
Isoac-chung
453
50
194
120
12
54
imenelydisker
358
24
144
120
0
70
swinml
s-tahdeveloping
32
davidstag
176
0
D
176
B
jaygala24
149
0
0
149
0
Muennighoff
orioow
125
20
20
11
75
dokato
112
12
94
0
jopyterjuzz
108
102
0
D
182
8
vaibhavad
4
75
MuhheeCCineeone
schmarion
CahrielSoqueira
0
D
88
8
digantamisnat5
0
0
71
0
shnoay--lhakal
Rysiss
Samoed
22
2
18
8
gowithellu--1 1998
0
50
0
dvareddyy
impuriuss
14
34
0
Alzsh
19 100104
0
0
46
81
MartinBensterff
13
8
2
20
siaodao
sukhlehu
rafolpowniatn
36
0
0
36
8
bp high
0
36
14
bjuempl
0
28
0
on next page
34

---

Published as a conference paper at ICLR 2025
Table 3: (Continued) Contributions by GitHub users. See Table 4 for the mapping between authors and GutHub handles
Githal [andlee
Total
Reviow PR
Paper wriking
Cocediaation
Running Models
lines
dataset
annotations
lack
loicmagne
28
0
johne
0)
28
D
ShuwonAshhaff
maiychendrikses
24
0
D
dwzho-pte pku
0
0
24
0
hgissbii
13
jalkounchnined
neminlee
22
0
lomtarsen
20
crystinse
0
D
21
ManuelFay
15
Alesey Vatolin
fbooc
20
john-b-yang
0
20
mmhandy
manandey
hokur-nandan
18
0
18
PranjalChinile
16
Sakshamrzz
16
dipum7
16
14
antemeneginv
12
midgguya
jordielive
guenthrmi
12
0
12
Ovnutul
12
Ar3mmis77
xhlaen
21polmak2005
12
192o
MariyaTik
be
simon-elemaiine
10
10
0
jimmy lin
10
10
urahooker
sej0419
10
sismengshon
10
10
0
ABorghini
0
10
xulkev
limiirand9921
10
10
howard-ynn
10
guangyssogg
10
Alenoch
10
6
cosssnof
ZhengTiu00
Rugyai
10
0
3
munikernemo
NouamaneTaat Tazi
0
2
Mesicanl amonade
2
2
May
2
schibb
0
antoniolanzal9ee
hanhinebola
B OVERVIEW AND CONSTRUCTION OF TASKS
In this appendix, we first provide an overview of existing tasks in MTEB benchmark and newly introduced tasks in benchmark (Section B.I. We proceed by explaining how the tasks were constructed (Section B.2) from existing datasets. Lastly, we introduce newly constructed datasets specifically designed for MMTEB (Section B.3
35

---

Published as a conference paper at ICLR 2025
Giillub
Fins nume
Last name
Atfliations
KenoohEnevlbenn
Kenneth
Enevolbsen
Asehus University
t-labdevellopig
Mínva
Kandrs
Asshos University
unnnelydisker
Kerhoun
LIRIS
wissam-sit sib
Wissam
Sihlini
Individual Contritutor
CiabriiSequciin
Gibbiia
Sequeira
Individual Contriluitee
schmarion
Marion
Schaeller
Wiki
MathicCCiancone
Mathieu
Clancone
Bernstofff
Martia
Remstorff
Asehos University
stauxãoo
Shitao
Xiao
Beijing Acudemy of Artificial Inaelligence
ZhengL.in i0
Zheng
Liu
Beijing Acaaemy of Artificial Innelligence Individual Contritutee
achibb
Chihb
cossamoo
Federioo
Casano
Nonbcaster Univensity and Cumor
taidggsyen
Nguyen
Tii
University of Pennoyyvame Ivamia
xu3kev
Wen Ding
Li
Comeell University
Rysiss
Rystrem
University Oxford
tacminles
Univeritty Human-Inspieed Research
izhs
Xin
Zhang
Harbin Instituee of Technology
orioms
Orion
Weller
Johns Hopkins Univiiity
skawwee
Webrli
Roben Koch Institute
manandey
Manan
Day
Salesforce
sanc-chung
Tasanc
Chung
Individual Contributor
asparias
Omer
Cazutan
Kag UnaversityTuriee
willposwiata
Information Processing Institute
rbroe
Koberta
Rocca
Aushus University
uwinml
Ashwin
Mathur
Individual Contributor
guangy usong
Guangyu
Song
Tano Labs
davidstap
Stup
University Amsterdam
Hlaasse
Lasse
Hansen
Aachus University
juygalaz4
Jay
Gala
MEZUAE
dignatamisra8g
Max. Plankk Institute for Intelligent
ELLIS Institute Tübingen
PranjulChitale
Pranjal
Chitale
Indian
Tochnology
14sh190104
Akash
Kundu
Hesitage lastitte of Techaology und Aport Research
dwzhu plu
Dawei
Zha
Peking University
1mmirunda921
Lester James
Miranda
Allen Institute for
Andriants
Michail
University
sinoon <lematide
Simon
Clematide
University of Zunch
Saiteja
Saiteja
Uipula
Microsofi Research
munhaddy
Mohuameed
Hamdy
Cohere Comuunity
upyteejuzz
Rugyai
Rugjyn
Bin Soft
NaN
Kranthitiv
Kranthi Käran
GV
New Yoek University
streeyu dhakal
Streeya
Diakal
Individual Contributor
dipan?
Dipas
Contrilud
Arthaastry
Geyatn
R. College of Enganeecing
ankounchained
Jan
Kostkan
Aarhus University
bp-high
Bhavish
Pahwaa
Microsoft Reseach
rasdani
Daniel
ellamind. Germany
AtuwenAshhraf
Shawon
Ashrnf
ellamind, Germany
bjoempl
Bjöm
Pilister
ellamind, Germany
Philipp
Harnis
ellamind, Germany
maltins
Maltc
Ossiglat
ManuelFay
Mancel
Faysse
CentraleSupéles and Illoin Technology
hgiisbihh
Hippolyte
Gässeror Houkhlef
CentraleSupéles and Artefact Research Center
sted97
Todeschi
Sapienza University
gestaiscool
Winaza
Contributoe
homipp105
Henil
Panchul
Nirma Univerity
ABrgghnni
Alossia
Borghini
Sapienza University of Rome
jordictive
Jordan
Clive
College London
gow gowithelows-11 1998
Chenghao
Xiao
Durham University
staryaaenndiissn
Mariyn
Hendrkksnn
University of Amsterdam
dokato
Daminik
Krzeminiski
Cohere For Al Community
Samoed
Solomatin
ITMO University
Alesa
Fenogenowa
SaluteDevies
1b1092
Aleksandr
Abramow
SaluteDevices
untomenegiee
Ariam
Sneginev
SaluteDevices
anpalmak2003
Maksimova
SaluteDevics
MeriyoTTkhhnowa
Maria
Tikhonova
SaluneDeveess and HSE Univenity
vaibiind
Vaihiu
Adfakha
Mila, McGill University and ServiceNow Research
sivaredity
Siva
Reddy
Mila, McGill University and ServiceNow Research
guentherni
Cürther
violenil
Akohin
Sukhlecha
Individual Cuntributor
Muemnighoff"
Niklas
Moennighoff
Stanford Univenity and Contextaal AT
Alexey Varolin
Alcksei
Vanlii
FRC:
xhluca
Xing
La
Mila, McGill Univenity
crysinn-2
Xinyn
Zhang
University of Wateroo
tommursen
Town
Aarien
Hugging Face
Marek
Suppa
University Bratislava Systems
swjB419
Werjia
University Washington
siamengshon
Mongzhoa
Xia
Prinection Univenity
sshn-b-ynn
John
Yang
Stanford Univenity
ttakur-tandaa
Nandon
Thakur
University Watefloo
leiemagne
Loic
Magne
Individual Contribator
smbbokerr
Sara
Hooker
Coberc For Al
kwojusi
Konral
Wejtssik
Wrechow Unisersity of Science and Technology
University
hongjin-ss
Hongin
Su
University Hong Kong Princction Univenity
howarl-yen
Howord
You
Table 4: Author overview, along with their affiliations and GitHub handles.
36

---

Published as a conference paper at ICLR 2025
B.1 INTRODUCTION TO BENCHMARK TASKS
Classification First, a train set is constructed by sampling (8-16) samples for each label. If only test set available, a section is split off as a training set. Both sets are then embedded and used to train a logistic regression using maximum of 100 iterations. Afterwards, performance metrics are calculated. For robustness, this process is repeated 10 times.
Pair classification For two paired texts, the goal is to predict the label. Examples of such tasks include paraphrase detection or duplicate detection. The task is solved by embedding all documents and then computing the distance either using model or Manhattan. Using the best binary threshold, performance metrics are computed.
metric, cosine, euclidean, dot product,
Bitext mining The dataset consists of matching pairs of sentences, and the goal is to find the match All matching pairs of sentences are embedded, and the closest match is found using cosine similarity, and metrics are reported.
Clustering and hierarchical clustering Clustering starts with a set of documents and an associated set of labels. First embed all documents, then take subsets of the data of size k for each of 10 consecutive experiments. All the documents are embedded, and a set of size k is sampled from the embedded documents. The embeddings are then clustered using K-means clustering, and performance metrics are calculated between the estimated clusters and labels. If the clustering problem is hierarchical, this procedure is repeated for each level of the hierarchy separately. Hierarchical tasks were formerly either split into multiple tasks, or later levels of the cluster hierarchy were ignored.
Note that this formulation differs from that of MTEB in that the sets are randomly sampled from the embedded documents instead of being specified a-priori. This drastically reduced runtime as one document can be used in multiple subsets without the need to embed it multiple times. The new formulation also allows us to gain robust estimate of performance with a lower number of documents.
Retrieval Retrieval tasks consist of a corpus, queries, and mapping between the queries and their relevant documents. The goal is to retrieve these relevant documents. Both queries and documents are embedded using the model. We allow these to be embedded differently depending on the model. For each query, the corpus documents are ranked using a similarity score, and performance metrics are calculated based on the reference mapping.
Multi-label classification Classification tasks in MTEB were previously limited to utilizing only
one label per document. As such, some, otherwise useful multi-label classification tasks had to be dropped or reformulated. We addressed this by introducing a multi-label classification task type Similarly to our novel clustering task, we down sample training sets for 10 experiments. We limit the training sets to include instances of each unique label, and train Neighott classifier. Every classifier is then evaluated on the same test set. We opted for Accuracy, F1 and Label Ranking Average Precision (LRAP) as evaluation metrics.
Instruction retrieval Instruction retrieval builds on the traditional retrieval task by incorporating
detailed instructions alongside the queries. Unlike standard retrieval, where queries are usually brief keywords, instruction retrieval pairs each query with comprehensive instruction that outlines the criteria for document relevance. These instructions are specific to each query and not generic to the entire dataset. Therefore, the task involves using both the query and its associated instruction to retrieve relevant documents from the corpus. For the main metric, we use Robustness 10.
Reranking Similar to the retrieval task, reranking includes corpus, query, and list of relevant and irrelevant reference texts. The aim is to rank the results according to their relevance to the query. References and queries are embedded and references are compared to the query using cosine similarity. The resulting ranking is scored for each query and averaged across all queries, and performance metries computed. For the main metric, we use MAP@ 1000.
Semantic text similarity Semantic text similarity (STS) tasks consist of sentence pairs, where the goal is to determine their similarity. Labels are continuous scores, with higher numbers indicating more similar sentences. All sentences are embedded using the model, and the similarity of the pair is computed using various distance metries, allowing for model-specified similarity metrics. Distances
37

---

Published as a conference paper at ICLR 2025
benchmarked with ground truth similarities using Pearson and Spearman correlations. Spearman correlation based on highest similarity serves as the main metric (Reimers et al. 2016)
B.2 TASK CONSTRUCTION
This section outlines approach to constructing tasks, primarily from pre-existing data. For details on the newly introduced dataset in MMTEB, we refer to Section B.3
Task construction from existing datasets consisted of number of steps ensure that the task is compatible with formulations in the benchmark and matches our standards: 1. Dataset preprocessing: start by applying minimal additional processing ensure the data is in the required format. 2. Dataset size reduction: to maintain manageable evaluation times, we proceed by reducing dataset size whenever applicable. 3. Relevance filtering: To ensure the datasets are relevant for the types of tasks being evaluated, we apply relevance-basee dataset filtering. 4. Differentiation testing: we assess the task's s ability differentiate between the performance of two candidate models.
For further details on dataset transformations for specific tasks, we refer to the dataset_transfore method implementation for each task.
Classification and pair classification For both classification tasks, we used existing datasets with minimal adjustments, primarily trimming them down to more manageable sizes. For performance evaluation. we rely on such metrics as Fi score, accuracy, or average precision. Whenever feasible, we align our choice of the primary metric with those used in related publications. If specific guidance exists, we default to accuracy for general classification tasks and average precision for pairwise classification. In scenarios with significant class imbalance, the F1 score prioritized.
Bitext mining Bitext mining tasks were constructed using established paired datasets. Similar to the classification tasks, the primary focus was on adjusting the dataset sizes to maintain the same model rank while reducing computational load. F1 scores were chosen to be the primary metric, unless specified otherwise.
Clustering and hierarchical clustering Clustering tasks were derived from existing corpora, such as news articles or encyclopedic entries. The source datasets typically included categories or labels assigned by their original authors or publishers. In some cases, like the SNL and VG datasets (Navjord & Korsvik, (202), which featured hierarchical labels, we reformulated the tasks from flat to hierarchical clustering.
Retrieval A variety of tasks were integrated as retrieval tasks, including existing retrieval, question- answer, and news datasets. For question-answer datasets, the questions were used as queries, and the answers formed the corpus, with correct answers identified as properly retrieved documents. In news datasets, headlines were treated as queries, and both the full articles were considered part of the corpus, with matched summaries and articles serving as relevant documents. For the primary metric, we use nDCG@1 0, unless otherwise specified by the dataset publication.
Multi-label classification For multi-label classification, we used existing datasets that required minimal adjustments. A critical aspect of these tasks was maintaining the balance of label distributions across the training and evaluation splits. To achieve this, we employed advanced stratification techniques (Szymanski & Kajdanowicz 2017 Sechidis et al. 2011) that consider higher-order relationships between labels, ensuring balanced samples and improved classification quality. For the main metric, use accuracy.
Instruction Retrieval For instruction retrieval tasks, we incorporated datasets like FollowIR (Weller et al.) 2024 2025). which consist of comprehensive narratives created by professional assessors. These datasets were initially developed for TREC shared tasks and included rich, context-heavy queries to evaluate retrieval systems' performance more intricate retrieval problems.
For reranking tasks, we adapted datasets covering a range of topics and languages, including academic paper ranking, news articles (Wu et al.] 2020b), QA pair relevance from online platforms, and passage ranking (Xie et al. 2023). For the primary metric, we use MAP unless otherwise specified by the dataset publication.
Reranking
38

---

Published as a conference paper at ICLR 2025
Semantic text similarity For STS tasks, adapted well-known benchmarks like STSbenchmark
(May et al. 2021) and cross-lingual STS datasets from SemEval (Agirre et al. [2115. We also adapted paraphrase datasets in various languages, such the Russian ParaPhraser (Pivovarova et al. 2017)
and the Finnish Paraphrase Corpus (Kanerva et al. 2021). As the main metric, we use Spearman
correlation based on the highest similarity (Reimers et al. 2016).
B.3 NOVEL DATASETS
This section introduces task specifically created as a part of the MMTEB contributions. For informa- tion on how existing datasets were adapted to MTEB we refer to Appendix B
PublicHealthQA:
This retrieval task is built on top of novel dataset containing question-and answer pairs in Public Health, specifically related to the COVID-19 disease. They are sourced from Q&A pages and Frequently Asked Questions (FAQ) sections of the Centers for Disease Control and Prevention (CDC) and World Health Organization (WHO) websites. They were produced and collected between 2019-12 12 and 2020-04.
WebLINXReranking: This is novel HTML reranking task derived from WebLINX, benchmark for training and evaluating web agents with conversational capabilities (Lu et al. 2024). Whereas the original work introduces retrieval task with the goal of retrieving HTML elements using conversational context, we propose the first task with the goal of reranking HTML elements based on their relevance for actions executed in web environments, including clicks, hovers, and text insertions.
WikiClustering: is a multilingual clustering benchmark hased on Wikipedia's main topic classifica- tions. The goal is create clustering benchmark that works for multiple languages.
To construct a WikiClustering dataset for a given language, we apply the following steps. First, download the wiki dump of the categories, the articles, and the category links. Second, we find the main topic classifications for all articles. The main topic classifications can be found by looking the category page for the language We only use the first paragraph of each article construct paragraph- techh-oo-paaaee paragraph (P2P) task similar to other P2P tasks within MTEB. Third, we filter out articles with more than one main topic and remove any topic with only one article associated with This step avoids ambiguity in the clustering task. Finally, we sample 2048 articles with associated main topics.
While the WikiClustering benchmark can be extended to any language with main topic classifications, currently implemented for the following: Bosnian, Catalan, Czech, Danish, Basque, Manx, Ilokano, Kurdish, Latvian, Minangkabau, Maltese, Scots, Albanian, and Walloon. All code is available on GitHub.
This is multilingual retrieval and reranking dataset based on succinct queries generated by a strong multilingual LLM grounded in Wikipedia articles. The dataset was made to resemble SQuAD. Sampled Wikipedia articles of a target language were chunked and passed to GPT4-o using the following prompt:
and RiliipediaRRrannningMillii
Your task is to anticipate possible search queries by users in the form of question for given document.
The question must be written in {{ language }}
The question should be formulated concretely and precisely and relate to the
information from the given document
The question must be coherent and should make sense without knowing the document
The question must be answerable by the document
The question should focus one aspect and avoid using subclauses connected wi th 'and'
The question should be overly specific and should mimic request of a user who is just starting to research the given topic not draw on your prior knowledge
for details, we classifiiationsffornngisss
refer
to
39

---

Published as a conference paper at ICLR 2025
2.00
030
Phoripergto
Inaat embedd
Dxg NMh Speneer
0.85
pcraphrase
budl
atteachnult Engluaie55 acfaacmult fingual lorge
0.75
0.70
Figure 7: Comparison MRR on synthetic retrieval and gold (GermanQuAD). The synthetic dataset generated using GPT4-turbo.
Generate question in {{ language }} for the following document: <document>
{{ document }}
</document>
Search query:
We filtered articles with less than paragraphs and sampled 1500 articles from the top 100k viewed articles. We then selected random window of consecutive paragraphs per article and chose the middle one to be the positive context and generated query paragraphs act as hard negatives. The 9 paragraphs per article are used for the reranking task with one positive and negatives. The one positive, 8 hard negatives, and the remaining corpus as negatives used in the retrieval task.
with gpt-4o. The surrounding
These datasets where constructed fro the following languages: "bul-Cyrl", "ben-Beng", "ces-Latn", "dan-Latn", "deu-Latn", "eng-Latn" "fas-Arab". "fin-Latn", "hin-Deva", "ita-Latn", "nld-Latn", "por-Latn" "ron-Latn", "srp-Cyrrl", "dan-Latn", "nob-Latn", "swe-Latn".
To estimate the quality of these samples compare it to the GermanQuAD (Möller et al. 2021 in Figure We obtain a Spearman rank correlation 0.93 with a 95% CI of [0.69;
B.4
TASK METADATA
Table 5 shows the required metadata to fill before adding a task to the benchmark. We provide detailed description of each field, along with examples and possible values.
B.4.1 DOMAINS
For our domains, we include the following:
Scholarly writing and research publications typically found in journals, theses, and dissertations.
Academic:
Blog:
Informal or conversational posts often found on websites or personal pages, covering wide range of topies.
Constructed: Text speech that is deliberately invented constructed, often used for experimental purposes to target specific abilities.
Encyclopaedic: Structured, reference-based texts that provide comprehensive and factual information wide range of subjects.
40

---

Published as a conference paper at ICLR 2025
Field
Description
Name
A concise name the task.
Description
A brief explanation
task's goals and objectives..
Type
The primary task category (e.g. classification, summarization. retrieval).
Category
The general data structure format of the task. This can be specified using a combinution single- letter codes (e.g.. "s" for sentence, "p" for paragraph. "d" for document). For example. "s2s" indicates a contence-to-sentene task, "s2p" indicates a semeenceto-puragrran paragraph paragmahh task.
tusk, and "pžp" indicates
Task Subtype
A more specifii subcategory within the primury task type. This can be used to further refine the task and provide additional context. For example, "Summarizatiin" might have subtypes like "Extractive Summarization"
"Abstractive Summarization"
Reference
A URL or citation to the original source material (e-g- paper, dataset repository).
Evaluation Splits
The specific subscts of the data used for training, validation. and testing. A list of ISO 639-3 language codes (e.g.. "eng" "fra") followed by ISO 15924 script codes (e.g.. "Lato", : "Cyrl") for cach language used in the evaluation. For example: [["eng". "Latn") " ("fra", "Latn") ") multiple scripts are used within single language, we specify them as list (e.g. [C'eng" ["Lat"", "Grek"DI..
Evaluation Languages
Date
The time period when the data was gathered. Specified as tuple of two dates.
Main
The prinury metric used evaluate task performance.
Form
The format of the data (e.g., "spoken", "written")
License
The licensing terms for the dataset (e.g.. CC BY-SA, MIT).
Domains
The subject areas fields covered the data (e-g., medical, logal, news). One dataset helong multiple domains.
Annotation Creators
annotated" (annotated e.g. by mtuukees), "derived" (derived from structure in the data). "LM-generraeed" (gener- ated using a languuge model) and "LM-generated and reviewed" (generated using a language model and reviewed by humans experts).
The type
annotators. Includes "expert annotated" (annotated by experts),
Dialect
The specific dialect regional variation of the language. How the text was generated. Includes "found", "created", "human-translated and localized", "human- translated", "muchine-trine "muchine-translatte""" "machine-transllted and verified", imachine-translate and local- ized". "LM--enerte generated and verified".
Text Creation
Bibtex Citation
The BibTeX format citation for dataset.
Number of samples Avg. Number of characters
The total number of data points in the dataset.
The average character length of the samples in the dataset.
Table 5: Required metadata for adding a new task to MMTEB.
Fiction:
Narrative writing based on imaginative content, including novels, short stories, and other forms of storytelling.
Government: Official documents, reports, and publications produced by governmental bodies.
Legal: Documents and texts relating to laws, legal proceedings, contracts, and legal theory.
Medical: Scientific and clinical literature related healthcare, treatments, medical research, and patient care.
News:
Joumalistic content that covers current events, politics, economy, and other topical issues.
Non-fiction: Writing based on factual accounts and real-world subjects, such as biographies, essays, and documentaries.
Poetry:
Literary form focused expressive language, often structured with meter, rhyme, or free verse.
Religious:
Texts related to religious teachings, doctrines, sacred scriptures, and spiritual discussions.
Reviews: Critical evaluations of works such as books, movies, music, products, or services.
Social: Written or spoken communication on social media platforms, forums, and other digital environments.
Spoken: Oral communication, including speeches, dialogues, interviews, and recorded conversations.
Subtitles: Textual transcriptions or translations of spoken language in films, videos, or multimedia presentations.
Web: Text content found on websites, covering a wide range of subjects, often hyperlinked and multimedia-enriched..
41

---

Published as a conference paper at ICLR 2025
Written: General term for any form of text-based communication, whether printed or digital.
Programming: Text written in programming languages instruct computers, often for software development.
Our definition of domain aligns with that of the Universal Dependencies project (Nivre et al. (016). We do not claim that our definition neither precise nor comprehensive. However, and include subject fields such as "medical", "legal", and "news" and literary type such as "fiction", "non-fiction". They are not mutually exclusive.
C BENCHMARK OPTIMIZATIONS
C.I SPEEDING UP TASKS
We aim to reduce the total amount of time needed to run the complete of MTEB task. In particular, we investigate how to drastically reduce runtime on clustering and retrieval tasks while maintaining relative model rankings. This appendix provides full details of the approach described in Section 2.3.2
C.1.1
CLUSTERING
Task
Spearman
Speedup
Biorxiv P2P
0.9505
31.50x
Biorxiv $2S
0.9890
14.31x
Medrxiv P2P
0.9615
21.48x
Medrxiv $2S
0.9560
8.39x
Reddit S2S
0.9670
11.72x
Reddit P2P
0,9670
22.77x
StackExchange S2S
0.9121
9.55x
StackExchange P2P
0.9670
20.20x
Newsgrouussssu
1.0000
5.02x
Average
0.9634
16.11x
Table 6: Agreement on model rankings on selection of English clustering tasks using Spearman's 's correlation across the scores of 13 models of various sizes.
In the main paper, we present a down-sampled and bootstrapped version the clustering task. We highlight the main results in Table 6) but refer to. We observe an average speedup across tasks of 16.1 Ix while maintaining the relative ordering of models on the evaluated tasks. The largest average speed-up was seen for e5-large (16.93x), but we expect this effect be even more pronounced among 7b or larger models.
9 single-level English clustering tasks are evaluated on 13 models across various sizes. A fraction of the documents are sampled and stratified by their target categories. At the same time, we wish to the fast approach should be able determine highly similar
maintain robustness the evaluation, model ranking to that from the original approach. As such, we investigate the extent of agreement between the original clustering task and ours in each task on the model rankings.
The model ranking is determined from the mean of V-measure scores from evaluations, where higher mean gives a higher model rank. Spearman's 's rank correlation score is then calculated based on the ranks from ours and the original approach. We additionally calculate the significant model rank which is determined by computing the significance of the given model's V-measure bootstrapped distribution based its mean of V-measure scores using our approach against that of the original approach. Significant S is then calculated based on the significant ranks from our and the original approach.
To find a balance between speedup and the robustness of the approach, 4% of the dataset chosen the fraction to down-sample to, with the exception of RedditS2S and StackExchange where
42

---

Published as a conference paper at ICLR 2025
Deeth 5ec
Organa
(b)
(c)
2
300
200
wacc www.eraweee...oooo LONA
Modd
Piat
1000
Drttagon SaaE HELLEEE
(h)
(i)
Figure 8: Distribution of scores per task across models.
43

---

Published as a conference paper at ICLR 2025
Task
Sig.
Biorxiv P2P
0,9390
Biorxiv S2S
0.9679
Medrxiv P2P
0.8200
Medrxiv S2S
0.9510
Reddit S2S
0,9790
Reddit P2P
0.7370
StackExchange S2S
0.9486
StackExchange P2P
0,9497
TwentyNewsgroups
0.9832
Average
0.9195
Table 7: Agreement on model rankings on English clustering tasks using significant Spearman's 's rank correlation with selected models of various sizes.
Natural Questions
TREC-COVID
3
Bank
5
20
100
250
300 Al
10
50
100
250
300 AI
Number Documents
Number of Documents
654brge-v2
one- contiora.msmarce
Ige-targe contreeaa-msmano
SFR-Embedding
g1o 2000 dnv225
e5.arge-v2
Ranking different models on subsampled versions of the datasets using hard negatives. We see that NQ can be reduced to just two documents per query (relevant hard negative) while still maintaining the rank while TREC-COVID less stable.
Figure 9:
sammlllsss = 32768. 7 that all evaluated datasets have very high significant Spearman's rank scores between our and the original approach. Figure Bigure&rrports reports the distribution of V-measure scores obtained from evaluation per model in each dataset for the ClusteringFas and the original approach. There is generally strong agreement between the rankings from both approaches. We also observe that the FlusteringFast approach often (5 out of 9 datasets) produces smaller spread (i.e. smaller variance) in its V-measure distributions. Reddit P2P has the lowest significant Spearman score among this set. also has the lowest average character length for its documents.
C.1.2 RETRIEVAL
In this section we provide details about the method used to downsample retrieval datasets.
To ensure the downsampling kept the efficacy of the evaluation we aimed to examine several axes: (1) wide range of models be sure that the evaluation task could still properly rank the models just as if it were not downsampled (2) that this method works for retrieval datasets that are sparsely judged and densely judged and (3) seeing it was possible to use hard negatives from a smaller set of models due to the computational expense to gather these hard negatives on the full datasets 7
To meet these goals we chose NQ (for sparse relevance annotations, one per query) and TRBC- COVID (for dense judgements. > 500 query). To test using a small set of hard negatives, we gather the hard negatives with la-gerge-v2 only. We evaluate wide range of models for this analysis. including the current -fate-of-the-ar and some of the previous state-of-the-art. NV-Embed-vl Lee
We also tested whether ensuring that the ground truth relevant document is present in these hard negatives made difference- we found that it did not, most models ranked the ground truth in the top N, manually including it little help already included.
44

---

Published as a conference paper at ICLR 2025
Natural Questions
TREC-COVID
0.0
10C0990
0.7
0.7
0.5
2.5
5.4
50
500
Number of Documents
Number Documents
Hodel Name
Modil Name uto 9252 c1-115
Erribed vi.
larg
agelorge en c2-5
NV +1
agolorge en
5F540mbedsing Mistral
5F7-Cmbedsingg-a
#S-lergee?
Figure 10: Absolute scores of different models on subsampled versions of the datasets using hard negatives. NQ has relevant document per query while TREC-COVID has 500+ relevant documents per query which is why we see NQ scores gradually increasing whereas TREC-COVID scores vary.
ct al. 2024), BER-Embedding-- Misttsla Mistral (Meng et al. 2024). e5
[Wang et al. 2023 J. c5 c5-large-v2 (Wang et al., 2022), gle-base-en-v/ cn-v 1.5 (Li et al. 2023b), lare-e-ee-----... (Xiao et al. 2023). and untriever-msmaree
(Izacard et al.] 2021). We then evaluated the models on versions of the datasets with N hard negatives documents per query where N ( 5, 10, 50, 100, 500, all). We then compared the absolute scores and the relative rank positions to see what settings best retain the difficulty of the original task.
Ability to rank models correctly For good evaluation, it must be able to rank models correctly and determine the best model. For this we examine how the ranking of the models change when we lower the number of hard negatives. For NQ the rank remains stable even with just one hard negatives (Figure 9). For TREC-COVID the ranking becomes unstable starting at 100 hard negatives, continuing to change the number gets smaller.
Keeping the absolute score similar In an ideal case the scores for the task should remain similar and not trend towards perfect scores, remaining useful. We see that scores go very high when there are only few hard negatives for NQ (Figure (0). For TREC-COVID it is more stable, but we see some wider swings with smaller documents. Overall, the scores are relatively similar at 100+ hard negatives.
Summary Overall, we see that staying above 100 hard negatives gives similar absolute scores while maintaining the ranking ability. Thus we opted for conservative 250 documents per query to keep these characteristics.
C.2 CODE OPTIMIZATIONS
We here document the major code optimizations within MTEB not related to dataset scores, task reformulation
Dataset loading One important issue identified was about loading multilingual and cross-lingual datasets composed of numerous small files in their repositories. Even for total dataset sizes under 10MB, loading could take hours due significant overhead from managing high number of network requests and the improper opening and closing of gzipped files. collaboration with the datasets team (Lhoest et al. 2021), addressed these problems with two-side implementation improvements: the datasets library optimized the loading a large number of requested files, and we restructured the datasets and codebase to leverage the benefits of the newer implementation. This ultimately reduced loading times by almost a factor of 100, bringing the largely cross-lingual dataset bitext-mining loading under minute.
Deduplication Upon in-depth scrutiny of all datasets, cases with repeated samples were identified and deduplicated (e.g. MindSmalllRernninng) As this led change in scores, second version
45

---

Published as a conference paper at ICLR 2025
of the task was introduced to maintain compatible scores with existing benchmarks. To move the optimizations to existing MTEB tasks we implement local cache to avoid encoding a sample twice.
D TASK OVERVIEW
D.1 TASKS
To get overview of the all the tasks implemented in MMTEB refer to the automatically updated tables the documentation which include the available metadata for all of the task, including license, task category, domains, etc.
D.2 LANGUAGES
Additionally, the top 100 out of the total 1051 languages in ISO 639-3 language codes and their respective task counts are in Table
Multilaeelllassi cissiiicctiion
InstructionRetrieval
PairClassilication
Summarization
Classification
BilextMining
ISO Code
Language ge
Clustering
Reranking
Retrieval
Family
Speed
STS
Sum
eng
English
Indo-Europear
16
143
16
3
8
8
92
13
2
1
303
deu
German
Indo-European
14
7
0
6
2
18
0
0
58
fra
French
hulo-European
13
8
0
5
3
15
4
0
1
57
rus
Russian
Indo- European
13
6
0
4
2
0
0
52
pol
Polish
Indo-European
11
4
0
4
0
18
4
0
0
46
emn Mandarin Chinese
Sino-Tibetan
10
4
0
3
4
0
0
44
spa
Spanish
Indo-European
13
4
O
2
2
13
4
0
0
43
hin
Hindi
Indo-Eune Europenn
12
2
0
1
2
10
0
0
38
code
unknown
Programming
O
0
0
0
0
37
0
0
0
37
jpn
Japanese
Japonic
8
3
0
1
3
13
2
0
0
35
kor
Korean
Koreanie
8
0
2
1
9
3
0
0
29
ara
Arabic
Afro-Asiatic
12
0
0
2
1
9
2
0
0
28
ben
Bengali
Indo- Eulo-en
2
0
1
2
0
0
28
ita
Italian
Indo-European
9
1
0
2
1
5
3
0
0
27
por
Portuguese
Indo-Euroo European
0
2
0
0
27
tel
Telugu
Dravidian
7
2
0
0
0
1
5
2
0
0
24
dan
Danish
Indo- Fur--an
2
0
0
0
23
swe
Swedish
Indo- -uroppnn
8
3
0
1
4
0
0
0
22
ind
Indonesian
Austronesian
0
I
0
0
21
tam
Tamil
Dravidian
7
7
2
0
1
O
3
0
0
21
tha
Thai
Tad-Kadai
0
6
0
0
21
Marathi
Indo- Eurooean
6
2
0
1
0
2
2
0
0
20
zho
Chinese
Tino-Tibetan
1
0
1
13
0
0
20
fin
Finnish
Uralic
1
0
2
5
0
0
19
kan
Kannada
Dravidian
7
2
0
0
2
0
0
19
mal
Malayalam
Dravidian
2
0
0
0
2
0
0
19
nld Dutch
Indo-Europeaat
6
1
0
0
1
2
2
0
0
19
nob
Norwegian Bokmål
Unclassified
5
0
0
0
0
0
19
tur Turkish
Turkic
7
1
0
2
0
3
2
0
0
19
urd Urdu
Indo-European
2
0
0
0
0
0
19
For the latest version see http://tt.tt.ccc.eieiciaieeeeeee
tasks.m
46

---

Published as a conference paper at ICLR 2025
gu pan
Gujarati
Mado-European
6
2
0
1
0
2
0
0
18
Panjabi
Indo- Eurooen
6
2
0
0
0
0
18
Romanian
Indo- Mudo-Eur
6
1
0
0
1
3
0
0
18
vie
Vietnamese
Austroasiatic
6
0
1
0
5
0
0
0
18
fas
Persian
Indo- Indo-European
4
0
0
2
9
0
0
0
17
ces
Czech
Indo-European
5
2
0
1
1
2
0
0
0
16
ell
Modern Greek
Indo-European
0
2
0
0
16
yor
Yoruba
Atlantic-Congo
5
3
0
0
1
3
0
0
0
16
ory
Odia
Indo- Europen
2
0
0
2
0
15
swa
Swahili
Atlantic-Congo
7
2
0
3
0
0
15
amh
Amharic
Afro-Asiatic
3
0
0
0
14
asm
Assamese
Indo-European
3
2
0
0
2
0
0
14
hau
Hausa
Afro-Asiatic
3
0
0
0
0
0
14
bul
Bulgarian
Indo- Europpnn
4
1
0
1
2
0
0
13
jav
Javanese
Austronesian
0
0
0
0
0
13
hun
Hungarian
Uralic
3
0
0
0
2
0
0
12
ibo
Igbo
Atlantic-Congo
5
3
0
0
0
0
0
12
slk
Slovak
Indo-European
0
3
0
12
heb
Hebrew
Afro-Asiatic
5
0
0
0
0
0
11
afr
Afrikaans
Indo- Europeann
10
hrv
Croatian
Indo-European
3
0
0
0
10
kat
Georgian
Kartvelian
10
san
Sanskrit
3
0
0
0
0
10
slv
Slovenian
Indo- Paro-eur
0
0
0
10
xho
Xhosa
Atlantic-Congo
3
3
0
0
0
0
10
hye
Armenian
Indo- Euro-En
0
1
0
isl
Icelandic
Indo- European
4
1
0
0
min
Minangkabau
Austronesian
4
2
0
mlt
Maltese
Afro-Asiatic
2
2
0
mya
Burmese
Sino-Tibetan
4
1
0
0
0
som
Somali
Afro-Asiatic
3
0
srp
Serbian
Indo-European
1
0
0
1
2
sun
Sundanese
Austronesian
0
arb
Standard Arabic
Afro-Asiatic
1
1
0
0
0
2
cat
Catalan
Indo- auto-European
2
0
cym
Welsh
Indo- European
4
0
0
0
est
Estonian
Uralic
0
Basque
Unclassified
2
2
0
0
0
kaz
Kazakh
Turkic
0
0
0
khm
Khmer
Austroasiatic
0
0
kin
Kinyarwanda
Atlantic-Congo
0
0
0
0
lin
Lingala
Atlantic-Congo
3
lit
Lithuanian
Indo-European
1 1
0
0
lug
Ganda
Atlantic-Cong
3
nno Norwegian Nynorsk
Unclassified
3
npi
Nepali
Indo-Euroo European
sna
Shona
Atlantic-Congo
2
3
snd
Sindhi
Indo- -uropen
tgl
Tagalog
Austronesian
0
0
0
Tigrinya
Afro-Asiatic
3
0
0
0
ukr
Ukrainian
Indo-European
2
1
0
0
0
0
0
8
ary
Moroccan Arabic
A Afro-Asiatic
0
0
0
0
7
bug
Buginese
Austronesian
4
1
0
0
0
0
7
fao
Faroese
Indo-European
2
1
0
0
0
0
0
7
kir
Kirghiz
Turkic
3
1
0
0
0
0
0
7
mai
Maithili
Indo-European
2
1
0
0
0
0
0
7
mkd
Macedonian
Indo-Europeat
1
0
0
0
0
mni
Manipuri
Sino-Tibetan
2
1
0
0
0
7
pem
Nigerian Pidgin
Indo-Eut aaoo-European
2
0
47

---

Published as a conference paper at ICLR 2025
sat sin
Santali
Austroasiatic
2
1
0
0
0
7
Sinhala
European
0
0
0
Swati
Atlantic-Come Congo
3
0
0
0
0
7
tsn
Tswana
Atlantic-Congo
1
0
0
0
0
7
Tsonga
Atlantic-Congo
4
0
0
0
0
7
uig
Uighur
Turkic
2
1
0
0
0
0
7
zul
Zalu
Atlantic-Congo
0
awa
Awadhi
Audo-European
2
1
0
0
bak
Bashkir
Turkic
0
bel
Belarusian
Indo-Europes
4
1
0
0
0
bho
Bhojpuri
Indo- Pudo-European
0
bod
Tibetan
Sino-Tibetan
1
1
0
0
0
bos
Bosnian
Indo-European
1
2
0
0
0
0
0
ceb
Cebuano
Austronesian
1
1
0
0
0
0
0
6
ckb
Central Kurdish
Indo-Europpe Europeu
1
0
0
0
0
ilo
Iloko
Austronesian
1
2
0
0
0
Table The top 100 languages across MMTEB tasks in ISO 639-3 language codes and their respective task counts.
D.3 EXAMPLES
Table 9 and Table 10 provide examples for each new task type introduced in MMTEB. For exam- ples of bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, and summarization datasets, we refer to the MTEB paper Muennighoft et al. (202hb).
Dataset
Query
OG Instructions
Short query
Relevant Document
Robust04
Who is Relevant documents will contain any information
Find doc- Schengen Space Concern- uments ing the mission traditionally the about the actions of sig- that an- performed by overoserggi
involved
Schengen natories of the Schengen swer this border traffic-the new direc- agree- agreement such as: mea- ques- ment to sures eliminate border tion
torate must fit into a Europe of immigration. The inte- rior minister is therefore ask- ing DICILC to step up its control of crossborder traf- fic, "particularly
eliminate border con-
controls (removal traf- Schengen fic obstacles, lifting of agree-
traffic restrictions); im- ment trols in plementation of the in- actions.
the fu- ture external borders of the Schengen space." " Originally scheduled in February 1994 but constantly postponed, the implementation of the agree- ments signed Schengen by nine European countries (the Twelve, minus Great Britain, Ireland, and Denmark), pro- vides for the free circulation of nationals within the space common to the territories of their nine countries...
formation system data Western Europe bank that contains unified and what visa issuance procedures; do they or strengthening of bor- hope accom- plish?
to der controls at the exter- nal borders of the treaty area in exchange for free movement at the inter- nal borders. Discussions of border crossovers for business purposes not relevant.
Table 9: Instruction Retrieval examples.
48

---

Published as a conference paper at ICLR 2025
Dataset
Text
Label
Maltese
Hi kellha 82 sena Id-dinja mužikali fi-Italja tinsab Fluttu wara I-mewt tal-attriči kantanta popolari Milva, li fis-snin 70 kienet meqjusa "ikona" fost it-Taljani. Milva kienet kisbet suččess kbir, fl-istess epoka ta" Mina Ornella Vanoni. Milva harget numru kbir ta' albums tul il-karriera taghiha u fiadet sehem f" f'Sanremo ghal xejn anqas minn 15-il darba; ižda qatt ma rebliet il-festival. Hi kellha 82 sena, telqet mix-xens xena tal-ispettaklu ežatt snin ilu.
culture(2), inter-
News Cate
national(10)
gories
Table 10: Multilabel Classification examples.
imporrt mteb
from mteb. task_selection import results_ data
tasks = tleb.gee_taskes
bask_tpps==""Retrivall
languages=("eng", "fra"l, "mains=="legal""
model
Lodel_names = [
"inffloa//mlliiingguullle
"intfloat/mmltiiinguulll -es-bass""
iinfloat/mmltiiingul----
g
models [mteb: get_model_metae (name) for in model_names,
results = mteb boaadresulttt (mo (models=models, tasks=tasks)
of
(results)
Figure 11: Simple example of how to obtain all scores on English (eng) and French (fra) retrieval tasks within the Legal domain for of models.
E FULL RESULTS
During this work, multiple models were evaluated on more than >500 tasks, with multiple tasks containing multiple language subsets covering more than 1000 languages, This makes a comprehen- sive overview unreasonable. While we have supplied scores aggregated across task categories, we realize that readers might be interested examining scores for their specific language, domain of interest, and task. To ensure that such aggregation is available and easily accessible,
make all results available on the public and versioned results repository 9 These results include time run, evaluation time, and wide set of performance metries pr. language subset, CO2 emission, version number, and more.
To make these detailed results subject to easy analysis, have added functionality for loading and aggregating these results within the mteb package. It is, for instance, possible to retrieve the scores for specific models all English (eng) and French (fra) retrieval tasks within the Legal domain using the code snippet in Figure 11
We refer the documentation for the latest version of this code.
cwt.////eeeeddnngeeeeeeinrrrrree htttpo///ttttto.ccm///adddeeeees the specific version repository used for this work see commit id hhtp::/0acaaaaaaaaoooooooooaaaaa whtpts::
49

---

Published as a conference paper at ICLR 2025
E.1 PERFORMANCE PER NUMBER OF SPEAKERS
Spp
Specker
Figure 12: Models' rank on the MTEB (MULlMuiillnggull) by the total number of speakers of a language. Trendlines represent moving average with window size of 10
F NEW METRICS
F.1 ABSTENTION FOR RETRIEVAL AND RERANKING TASKS
addition to the existing ranking metrics used for Retrieval and Reranking tasks (Muennighofi et al. 2023b), we propose to assess score calibration through the evaluation of model abstention ability, using the implementation Gisserot- Boukhlef et al. (0024).
Intuitively, model abstains given instance (q, ... dx) (one query and le candidate docu- ments) if c (g. di. dx) < т, where e is confidence function and t is threshold regulating abstention likelihood. Therefore, to evaluate abstention capacity given test set S, an approach consists of making I vary to achieve several abstention rates. In the case of effective abstention, the metric score increases with the abstention rate.
More formally, models' ability to abstain is evaluated by computing the normalized area under the metric-absteenton curve (nAUC). Given confidence function c, metric function 17(12 and labeled test dataset S, AAUC is computed as follows:
1. Multi-thresholding Given model f and dataset D, define a set of abstention thresholds T1 ....... Tu. such that Ti < .... < Ty- For each threshold Ti, we construct a corresponding sub-dataset S, C D by applying the abstention criterion. We then evaluate the model on each sub-dataset S; using the metric function m. To quantify the model's performance across these thresholds, we compute the area under the metric-abstentionn curve, denoted as AUCmoode
2. Compute lower-bound: Since AUCmadel depends on the model's raw performance without abstention, compute the effective lower bound AUC This corresponds the under the curve when the metric remains constant as abstention increases, representing the baseline where abstention does not improve the metric.
3. Compute bound: To establish the upper bound, AUC+,
we evaluate an oracle model that has access to the true labels. The oracle can selectively retain the best instances at each abstention rate, yielding the theoretical maximum area under the sectric-abstention curve. This represents the optimal model performance under abstention.
4. Compute normalized AUC: Finally, we compute the normalized area under the curve, denoted nAUC model, by scaling AUC between the lower and upper bounds:
AUCmodel - AUC
nAUCmmodet
AUC AUC
"In our implementation, we rely on three simple confidence functions all taking the instance's query-document cosine similarity scores as input: the maximum score, the standard deviation of scores and the difference between the highest and second highest scores.
We utilize the metrics initially implemented the evaluation of Retrieval and Reranking MTEB tasks MMunnighofff et al. 2023b).
50

---

Published as a conference paper at ICLR 2025
Name in Paper
HF Name
Revision ID
M-7B
Gritl M/GrM// MM7B
e5-mistral-7b-innte
antflow/e5-mistee 7b-imstruct
71658722114411222208777655552223
multilingual
intfloat/mltilingual
113311222h6611000777000000004040
muliiiingualleS-hang
intBoat/mulllinggg
mulillingulle55- -e-ruul----------------
li-turg--nnstcct instruct
mulilinguulleS--e -S-mmalll
intiou/mmttilngguul-S---
LaBSE
s-V/LaBSE
all-MiniLM-L11
Walll-MimiLM-L122-2
01580777647777770444444444544413
M--66
--Vall-MiniLM-LL
all-mpnct-base base
s-t/all-mpoct-
basc-v2
multilingual Min1L
s-t/paraphrase wattanaaassmlluilinnuuu mww.ilinggalll---aaa- www.tiinngglllMiiiiMMMM------
Table 11: Model name appears in the paper, its name on Huggingface Hub, and their associated
revision IDs. Note: s-t stands for sentence-transformerr.
G MODELS
Models used for task selection along with their revision IDs can be found in Table 11 Code for running the models, including prompts, available within MTEB's model registry avail- able at /https://gi //tp::///ithub...../ com/ http:/cccccceobebbbbniigggggggg.
Unless
otherwise specified within the model implementation, the prompt available in the file tps //github. wwt.cooo/emmeee
wwwneeesssinsstutionsssmmm py As some debugging happened during the running of the models, multiple versions MTEB were used. Due to the computational cost of rumning these large models the vast amount of datasets, it was deemed unfeasible to run all the models using the exact same version. However, for each task, all mod- els were
the same version of the specific task. Model results be found in JSON format in the results repository; these include additional performance metrics, model metadata, CO2 emission, time of run, and exact version of MTEB used: /tttps://gi com/ http://hhhw..cccm///eeeeee//n//e wttp//edddnnnsbbennneeeee
H BENCHMARK CONSTRUCTION AND OVERVIEW
H.1 BENCHMARK CREATION
The following section introduces benchmarks created as a part of the MMTEB open contribution, which aren't introduced within the main article. MTEB additionally includes a variety of benchmark including the language-speciffc, notably the original English MTEB, MTEE( v2) (Muennighoff al. 2023b). the Scandinavian embedding benchmark MTEB (Scandinavian) (Enevoldsen al. 2024. the French benchmark MTEB(fra) (Ciancone et al. 2024), the German benchmark MTEB (deu) (Wehrli et al., (0224), the Korean benchmark Mkor) ) the Chinese benchmark (Xiao al.) 2024b). the Polish benchmark MTEB (pol) (Poswiata et al. 2024). Along with these MTEB also include an instruction based retrieval based benchmark MTEB (FollowII) (Weller et al., 2024), , a benchmark for law (Law) the bitext section of the MINER benchmark MINERSBi textMi Meiininnnnnn target at low resource languages (Winata et al.) 2024b), and the CoIR benchmark for code retrieval CoIR (Li et al. 2024 For this benchmark, refer to their associated paper and pull requests.
For an up-to-date overview maintained benchmarks please see the benchmark registry
MTEB(rus) (Snegirev et al. 2024): Although Russian has approximately 258 million speakers world-wide, it was almost completely absent from the original benchmark and represented only in few multilingual datasets (e.g.. MnssiveeneenCllasiife number of Russian datasets in the new multilingual benchmark. For this, we selected popular Russian time-tested and community-teste -tested datasets representing the main MMTEB tasks. Additionally, performed data cleaning and automatic filtering, where necessary, and formatted datasets in the
To address this problem, included
51

---

Published as a conference paper at ICLR 2025
MMTEB format. The final Russian part includes 18 datasets covering 7 main tasks: Classification (7 datasets), Clustering (3 datasets), auuulaaabblllssffccttin (2 tasks), PairClassffication (1 task), Reranking (1 task), Retrieval (2 tasks), and STS (2 tasks). This dataset manually constructed.
The Reasoning as Retrieval Benchmark (RAR-b) (Xiao et al. 2024a) evaluates reasoning- level understanding abilities stored in embedding models, and assesses whether correct answers to reasoning questions can be retrieved as top similar to queries, under w/ and w/o instruction settings. The benchmark provides insights into whether representations of nuanced expressions are aligned and well-encoded by current embedding models, going beyond the established reliance on evaluating with STS or traditional topical-level IR tasks.
RAR-b:
The benchmark puts together 17 tasks made from 15 datasets (with reasoning questions from 12 datasets and 3 extra datasets to enlarge the corpus), covering commonsense reasoning: WinoGrande. PIQA, SIQA, oNLL. HellaSwag, ARC-Challenge. Quail, CSTS (Sakaguchi et al., 2022 Bisk et al.
2020: Sap et al., 2019; Bhagavatula
al. 2020; Zellers et al. 2019 Clark et al., 2018 Rogers al. 2020 Deshpande et al. 2023). 2) temporal reasoning (Tan et al., 2023). 3) spatial reasoning:
SpartQA (Mirzaee et al. 202 12 4) numerical reasoning: GSM8K, MATH (Hendrycks et al.) 202
Cobbe et al. 2021 Yu et al. 2023), and 5) symbolic reasoning: HumanEvall Pack and MBPP (Husain
ct al. 2019 Austin et al. 202n Chen et al., 202T; Muennighoft et al. 2023a). The comprehensive assessment provides an carly checkpoint for abilities envisioned to be necessary for next-generation emhedding models (Xiao et al. 2024a).
MTEB(Europe): We begin by selecting 56 official languages of the European Union, along with languages recognized by Schengen-ares countries, such as Norwegian Bokmäl, Icelandic, Romani, and Basque. This initial selection results in 420 tasks. We then reduce this selection by filtering out machine-translated datasets, datasets with unclear licenses, and highly specialized datasets (e.g.. code retrieval datasets). Additionally, we remove tasks such as AfriSentiClassifican ion, which, while containing European languages, primarily target African or Indic languages. After these exclusions, 228 tasks remain. Next, we run representative selection of models (see Section [3. and iteratively filter out the most predictable tasks (see Section (2..3) To preserve language diversity and ensure fair representation across task categories, we avoid removing any task if it would eliminate language from particular task category. Furthermore, we retain tasks where the mean squared error between predicted and observed performance exceeds 0.5 standard deviations. This process continues until the most predictable tasks yield Spearman correlation of less than 0.8 between predicted and observed scores, or until no further tasks can be removed. Ultimately, this results a final selection of 96 tasks. Finally, contributors proficient in the target languages review the selected tasks, replacing some manually with higher-quality alternatives if necessary.
(IEBB This benchmark is constructed similarly the previous European benchmark but focuses on a set of Indic languages [1| Initially, we selected 55 tasks. After manual filtering, 44 tasks remain, and following task selection and review, the final benchmark contains 23 tasks.
H.2 BENCHMARK TASK OVERVIEW
The following tables give an overview of the tasks available within constructed benchmarks. For more information about the specific tasks, we refer to the task metadata available through the mteb package. 15
Table 12 and Table 13
Gives an overview of the 'MTEEMMulilingual)" benchmark
Table 14
Gives overview of the 'MuroeEurope) benchmark
Table 15
Gives an overview of the 'MTEE(Indic)) benchmark
Table 16
Gives an overview of the 'MTEB(eng, v2) benchmark
Table 17
Gives an overview of the 'Moee( benchmark
Whee following iso639-3 codes: asm, awa, ben, bge, bho, doi, gbm, gom, guj, hin, hne, kan, kas, mai, mal, mar, mni, mup, mwr, nep, npi, ori, ory, pan, raj, san, snd, tam, tel, urd httpp:://giii
52

---

Published as a conference paper at ICLR 2025
Tye
Xum
Lageage
Donaan
Acecsdow drcaee 26 uaglli
Ricobloig B.CL Sorgghagga p01
Witnes]
30299
craad
413452
"Sacll Ticket
Dallahadtte (Chetaace
41033044
Tagae
raadaad localed
Winsell
Taxal:
[W]]]
3601
3401
Chikken Tees
Steal: Winke]]
Cyntle Eaccccn lunbalr
3423
Teackeads Wrika]]
3433
3434
Ghullaa Ceceeaallteeee Chiiii
194
3434
5443
Rodel. "Witner3
30465
793
(Witee))
1687
NCAAE book
Tel.
Shastaa booksas Pomorr
11000
1107) HT
3401
30.00
Beaawe Wes Soadd...
342d
40:00
FAG Likko Aigro Maher
[34
Liga: Wiskal
5455
[34
WWisor Socid)
4014
[Raket Maaee New Notdected...)
craed
0182
1467
Se.d
2045
m1 3013
Sead
2011
Snich Inidhagnon Choc
1961 T5 TW/
Wickwil
Swad
-000
Tooe book
Wickee
-5.1
Clivarng
3601
2555
53787
Daad
HALE
17590
trsilied enperianne unebiie
107700
1989
deried
Makhaachamm hamm mosser
deriod
Table 12: The tasks included in (TEB(Multilingual) (part 1).
H.3 PERFORMANCE ON MTEg v2)
Table 18 show the performance of our representative set of model on (TEB v2).
H.4 PERFORMANCE ON (Code)
Table 19 show the performance of our representative set of model on (TEBe))
53

---

Published as a conference paper at ICLR 2025
Naw
Lagagg
Strumglee
(962)
[Noms' Wisker
1red
CHERClook aaukkaaa Dharet
1Baa "Scial "kkg"..
Lanc carirad
1882
Maraherof bardheooooaaaeea bede (1a
expert anorated
25p7
1002
(Spdie "paker
TTacker Nonciea Webr
Spea 2 Reckasanns Cor (ich
[wg]
(Hanc)) Wiadt
XNL
30220
201
Nome
code
expect meorated
300
(Maa; "Nowe "iike))
Cugast a01
11348 335575
Coddlated
Now
(4090)
TTicker Weaic]
100
940 433
regist
173255
1992- 1054
23424 0435
Thaydoaecks Wesker
12304 4000
ww worlee
LM granaddd resirand
71020) 1:000
201
CoroaTt hea.
New
2079
2011 (jnna 2.3
1rand
1963
S151
120
2105
SESL QU Fa0R
TWilg Wol
1753
S151 SeEeEE 2ill
ceard
1065
2019
2455
Table 13:
The tasks included in MTEB (Mul (Multilingual) (part 2). *For the number of samples, given the total number of samples all languages included, for Retricval tasks are given the (number of queries- number of documents).
54

---

Published as a conference paper at ICLR 2025
Cont hodelleelooooii
2014
D0pul bndook (ort
[ogg
Fand
Actrod
lokc Cadcaaiinnide
1164
11000
Facud
[4-]
2045
of derd
[1g
242-
[eg
0154- Line
610
2010
12234
1994
1971
THt
Fardl
bouu brack Bute. HARDE
[0g
Eacal
Actroo
Ma BLEOHEL
Tud "ce" Vae
TTagi "Geccaace Witror
11:300
Opagiidls Land
1431
1971
unt
24000
Ter. Mer. my.
"Wisoo
madd
Table 14: The tasks included in MTEB (Eurooe)) The language column shows all the languages of the task. When running the tasks we limit it to the languages specified in the benchmark. For the number of samples, are given the total number of samples all languages included, for Retrieval tasks are given the (number of queries number of documents).
Newe
Doraes
-1]
Lega: Lanmenee"
515144
New:
Sacal: WWika!
lesc
5897
2045
CagraPlon Fiiddll
ledelag sottttt
Partert WWrme]
repec
eng
1361
CEE" del
deasd
13200
Towwe "Winue 1
frnd
[hai]]
TRniiss "Witnw]]
frnd
dutied
2041
Ang'....
Swal:
Scao, Tixrer commens
Tal
Vei 11
Wea WWaamy
Table 15: The tasks included in MTEB(In The language column shows all the languages of the task. When running the tasks we limit it to the Indic languages specified in the benchmark. For the number of samples, are given the total number of samples all languages included, for Retrieval tasks are given the (number of queries- number of documents).
55

---

Published as a conference paper at ICLR 2025
Name
Langango
Donmine
Seeple
Assoidiics crader Nb varple
Choidoaiis
1p61
[Wetnen]]
hurea
inBiclonikaatoo Maeh 2.7001
Tog
[Rexinos (iihho
Frand
detinal
25000
Spekes)
benunttrratted beakned
235337
[Splier]
wer.sttaanllllloo beateal
laasee anaraadd
355357
Llakiing
15g
233
detivel
52797
[ag
Avedlaass Molkal Wilttol)
datindl
37530
(uddera Am
"Wob: WWenes
detivel
2045
[ag
WWl. Wiikes
Fraed
detindl
74814
log
(Sxaal Weitee
Foadd
LS77)
el i. in 00 p0006
lag
(Nows Wikker]
feadd
336779)
['Webr "Wmmas
45320 1155
[ag
Wiikkes WWl Prognessingg
foad
Aatieal
47382
("aaydopassi"
[mm]]
Name
182254 1000
ROAM (Takkrrtt ct.77
20N1
[ag
Wiskes Finanial]
foad
laacna anaraadl
57638 648
[mm]]
l'Acndeme: "Wasenn NNt Actser]
25837
TXBOOOM
[agg]
TMaaica Aundesie "Witbar)
515
SIXX Ee eit
[ag
WWA., WWikee
lassea anaraad
0927
STSI Ciiinessi
[ang]
[WWck, 'Mooo' Nansiiiiin
lasaua anarade
1550
STSI 17317 2515 00141
[ag
[Bleg "Web) Speder)
Jainal
3150
STSI
der long....
(Nows) "Wic" "Walkar)
lasss asaraaa
5345
CAAAA aaa M0R
conn Mes ...
(Noos "Waskor
Foand
lasse asavaad
3555
Table 16: The tasks included in MTEB v2). The language column shows all the languages of the task. When running the tasks we limit it to the languages specified in the benchmark. For the number of samples, are given the total number of samples all languages included, for Retrieval tasks are given the (number of queries number of documents).
Type Name
Lasgaagei
Donsin
Sample cecatio Aunctaiios cecaless Nà Sumples
Ratricad AppaResizsal/ MaaBBoiieulHHnllscbbbao (H ctal
['nng' pptlonn]]
[Paynamming
'Wiiten"
dasinel
'asa "yrsssrppe pptp)
Progrmmiig
ColdElliSerrdRRtzaaaa Machiigludrer
"Whnee fondd
13000 13000
Progaamiin
'Wonee" found
derived
13277 66383
CaldFerdarkS1 (irra
('eng]
PPayamming
'Witlen foand
deried
11906 156526
1034
[pp". jauz 'ymourrpp'
Pmagmmming
"Witten found
derined
53561 1005474
CodeStanciNeteeetieee ( (1s5in 2019
go:. jasa "snascrpp
Progmammigg
"Winen found
derned
6000 5000
DodehhanDeeannoonne TI 7013
[ce+: pothon
Progamming
[wnnen found
derned
221 1001
Tal-ThnAssmii Yoek 2015
yyllon
Paynanming
"Raite"" found
desived
150 916
CosQA Houng dl 7014
cng'. pothon']
Pragaannnn
"Waine"" found
derived
500 20604
[mn]
[Pagammingg
"Witten) foand
deried
19931
[mng (uf))
[PPagommiing
"Wiiten]]
foand
derived
105851
Table 17: The tasks included in MTEB (Code). For the number of samples, are given the total number of samples all languages included, for Retrieval tasks are given the (number of queries number of documents).
Rank Borda Count
Average Across Calegory
Average by Category STS Retrieval Clustering Reranking
All
Pair CIf. CIí.
model
eS-mistral--h-instuee GritLM-7B
1 (393)
67.0
67.2
88.4
75.2
83.6
54.8
51.4
49,8
2 (384)
66.4
66.7
87.3
77.0
82.5
53.2
50.8
49.6
(357)
65.2
65.6
86.2
73.2 84,3
51.0
49.9
48.7
mulliinguulle--lrrge
(270)
62.1
62,4
84.7
72.8
80.6
49.0
42.8
44.7
all-mpnet-buse-22
5 (211)
56.0
58.1
83,0
56.6
72.2
41.9
46.6
48.4
multilingualcc--ass
6 (211)
60.2
60.9
83.6
70.0
79.1
46.1
42.2
44.3
(188)
57.3
58.8
68.6
79.8
34.1
43.5
45.2
all-MimiLM-LL2--22
(172)
54.7
57.0
82.5
55.8
70.7
40.7
44.6
47.5
all-Minil M-
9 (149)
54.4
56.7
82,4
55.4
70.4
39.8
44,9
47.1
muliiinguul-e--small
10 (147)
58.4
59.3
82.7
67.7
77.6
43.7
40.8
43.2
paruphrase Trughrae-mmutiingullMiiiiLLLL
(109)
55.1
57.0
80.0
64.4
77.5
32.8
41.7
45.4
LaBSE
(49)
48.6
51.7
78.9
66.8
70.2
16.8
36.1
41.3
Table 18: Performance on MTEB(eng, v2) across task categories.
56

---

Published as a conference paper at ICLR 2025
Rank Borda Count
Average Across All
Average by Language
C++
Go
Java JavaScript PHP Python Ruby
Model
GritLM-7B
(88)
73.6
73.1
84.9
81.7
77.8
86.4
83.8
c5-
(74)
69.2
68.3
83.0
80.9
79.4
75.6
83.6
81.1
multilingual-e5-lare
large-instruct
(65)
65.0
56.4
74.7
74.7
71.7
71.6
79.1
74,9
multilingual-e5- large
(63)
61.7
46.8
73.4
72.2
66.6
69.1
75.7
73.4
mulillnggua-ee--baee
(55)
57.5
48.9 73.2
71.0
66.1
67.8
75.2
72.7
multllnggulleS--small
(53)
58.4
70.6
67.9
65.2
66.6
73.6
68.1
all-mppettbase-v2
(44)
56.4
46.3 67.4
62.2
63.1
61.7
69.0
65.7
Mll-MiniLM--L--22
8 (34)
52.7
48.1 64.4
57.4
62.2
60.4
68.1
66.6
M-L-
(27)
50.2
46.8 68.1
57.3
63.6
62.7
68.7
67.8
LaBSE
101
28.8
27.6 40.6
36.6
42.3
34.8
43.9
42.2
Table 19: Performance on MTEB(Code) across task categories. Because all code-related tasks are for retrieval, metrics by category are omilted.
57


Reviews:
[{"review_id": "KASzsizMtw", "reviewer": "Reviewer_PWdB", "summary": "This paper introduces a new set of benchmarks called \"massive multilingual text embedding benchmark\". This benchmark includes more than 500 tasks and covers a lot of low resource languages as well. They also introduce downsampling technique such that the resources required for the evaluation is minimized.", "questions": "- How do the top models on MTEB leaderboard do on this new dataset and whether this new dataset changes the ranking of the leaderboard?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is well written and the main points are clearly communicated\n- The dataset is a great extension to the MTEB and would be a good resource to research community towards building largescale multilingual embedding models\n - The coverage of the dataset is great", "weaknesses": "Based on table9, one limitation is that most of the crowd submissions are already based on existing public datasets from multiple language domains and not particularly for this dataset construction effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3b80d4K0vT", "reviewer": "Reviewer_CXhF", "summary": "MMTEB addresses the limitations of traditional text embedding evaluations to extend the current popular MTEB benchmark to over 500 quality-controlled tasks across thousand languages, making it the largest multilingual collection for embedding model evaluation. MMTEB is a large-scale, open collaboration benchmark where the contributors have diverse backgrounds and introduce diverse and challenging tasks, such as instruction following, long-document retrieval, and code retrieval.", "questions": "- How does the evaluation score compare with other embedding benchmarks? For instance, since the AIR-Benchmark doesn’t disclose its evaluation sets, does this mean there might be no overlap with MMTEB?\n- Why do smaller models perform better in multilingual contexts while larger models excel on English datasets? Is this pattern unique to comparisons involving only the e5 models?\n- Could you provide more MMTEB benchmark results using LLM embedding models from literature, such as SFR-Embeddings, NV-Embed, bge, and Qwen models? Since some of these models are English-based, please include their results in Table 15.\n- In Table 9, could you provide the sample counts (queries and documents) for each task? Additionally, please list the 1000 languages and 500 quality-controlled evaluation tasks with examples.\n- What is the sample count for each language, and is there an imbalance in sample numbers between languages? Why is it necessary to collect samples exhaustively from native speakers? Could machine translation help address sample imbalance?\n- Could you provide contributor statistics, such as distribution across countries, native speakers, domains, and similar tasks?\n- Since MMTEB appears to cover most of existing public embedding evaluation datasets, has there been any further data collection, annotation, or synthetic dataset creation for MMTEB? If so, please provide details.\n- Paper does not properly explain the code and long-document benchmarks. Could you provide details on these benchmarks and the performance numbers for models from the literature?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Given that recent embedding models often shows the trends for optimized for MTEB benchmark tasks, it would be valuable to develop larger-scale benchmarks that include a broader range of tasks.\n- Additionally, MMTEB downscales datasets and caches embeddings to help alleviate computational bottlenecks during evaluation.", "weaknesses": "In general, dataset information (such as sample numbers, multilingual types, etc) and relevant model benchmark numbers are missing. Find more details in questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lJnD2MW8Fm", "reviewer": "Reviewer_jHr7", "summary": "This paper introduces MMTEB, a massive multilingual text embedding benchmark that covers over 500 tasks in more than 1,000 languages. Compared to previous benchmarks, MMTEB considers the “low-resource double bind” during its construction and significantly reduces the computational resources needed for evaluation through various strategies while preserving the relative ranking of models.", "questions": "1. In Lines 86 to 93, could you provide a more intuitive metric for comparing computational resources of different benchmarks, such as the time required to complete evaluations on a single A100 GPU?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. I believe the efforts to reduce the computational resources required for evaluation are very meaningful, as they will encourage more researchers from low-resource language regions to use this benchmark. If MMTEB had simply expanded the scale of MTEB, it could be expected that most strong baseline models would originate from commercial companies with high computational resources, which could hinder the rapid development of text embedding research.\n2. Each computational resource optimization strategy is described in detail, and the methods are easy to implement, which facilitates the adaptation of custom datasets.", "weaknesses": "1. The depth of analysis across different datasets seems inconsistent. For instance, the “Clustering” section in 2.3.1 provides an average Spearman correlation, but the “Retrieval” and “Bitext Mining” sections lack similar metrics. Moreover, as seen from the results in Appendix C.1.2, the selection of “Retrieval” strategy is based on analyses from only the NQ and TREC-COVID datasets, which may lead to biased hyperparameter selection. Although the current level of detail is already quite high, given MMTEB’s potential impact, I believe further detail would only be beneficial.\n2. The abstract mentions “a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval,” but I saw little content related to these datasets in the paper. I think the authors should clearly explain:\n-  **Why were these tasks included in MMTEB?** (This is a benchmark for multilingual text embeddings, yet instruction retrieval is currently available only in one language.)\n- **How were these new tasks integrated into the benchmark?** (I believe directly including long-document retrieval under the “retrieval” category should be done with caution, as it would require researchers to consider incorporating long-document-related datasets in their training data, which to some extent runs counter to the goal of addressing the “low-resource double bind.”)\n- **How will these new tasks impact model performance?** (The context length limitation of models such as multilingual-e5-large-instruct could hinder their performance on tasks like long-document retrieval. In the LongEmbed evaluations [1], it performs worse than the Mistral-based version. Additionally, the results in Table 16 show that the Mistral-based models perform better on MTEB (code). Thus, claiming the exceptional performance of multilingual-e5-large-instruct in the Introduction without further clarification may mislead readers.)\n\n[1] LongEmbed: Extending Embedding Models for Long Context Retrieval. arXiv:2404.12096", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3lAK43294u", "reviewer": "Reviewer_RchW", "summary": "The paper introduces MMTEB, an extensive evaluation suite designed to assess text embedding models across over 1,000 languages and 500 tasks, serving as a multilingual extension to previous benchmarks like MTEB. MMTEB includes novel task categories, such as instruction following, long-document retrieval, and code retrieval. A significant contribution of MMTEB is its introduction of computational optimizations, including downsampling and hard negative sampling, which reduce compute requirements and enhance accessibility. The authors' findings reveal that smaller, instruction-tuned multilingual models outperform larger monolingual models in low-resource language settings.", "questions": "1. Should Section 3 be titled “Experimental Settings” instead of “Results” to better reflect its content?\n2. The evaluation metrics and the main metrics for certain tasks are not described, e.g. instruction retrieval, reranking, multi-label classification.\n3. Should bitext mining and STS be considered closely-related task categories in Figure 1?\n4. Summarization showed minimal correlation with embedding performance in MTEB. If it is still included in MMTEB, what justifies its inclusion?\n5. Does MMTEB include programming language benchmarks, such as CoIR? Additionally, what criteria of multilingualism are used to determine inclusion in the study?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(+++) MMTEB exemplifies a remarkable community-driven effort, engaging diverse contributors and fostering inclusivity.\n\n(+++) Introduces computational optimizations like downsampling and hard negative sampling, reducing evaluation costs to 3.11 hours on a 7B model (H100 GPU), making it accessible to low-resource settings.\n\n(++) Covers over 500 tasks across 10 categories in more than 1,000 languages, with a strong focus on low-resource languages and domains. But it lacks enough justification demonstrating the quality and value of each dataset. Provides an open-source, public leaderboard that encourages continuous contributions to advancing multilingual embedding research.\n\n(+) Expands traditional benchmarks by including new task types like instruction following, long-document retrieval.", "weaknesses": "1. The study lacks a clear articulation of the specific knowledge gap that MMTEB addresses beyond what MTEB has already achieved in evaluating multi-task capabilities of embedding models. The results section suggests that multilingual scores are closely aligned with English, making it difficult to discern the unique value that MMTEB offers. Additional analysis could better exploit the benchmark's value and clarify its unique contributions.\n\n2. While MMTEB aims to include as many relevant datasets as possible, it is unclear how these datasets were constructed or validated. Details on dataset quality, annotation methods (e.g., human vs. model-generated), and statistics (e.g., query-document ratios) would enhance transparency and reliability, especially given some datasets may be model-generated, such as FollowIR.\n\n3. The paper mentions retaining the top 250 ranked documents per query for each dataset and model but does not specify which model(s) were used to select these hard negatives. Clarifying this would help assess the robustness of the benchmark's retrieval tasks.\n\n4. The combination of 132 tasks makes it challenging to interpret a model's performance on specific languages or language families. While geopolitical categorization is helpful, further segmentation by language, domain, or specific capabilities could provide a more systematic and granular view of model performance. Expanding on the existing MTEB language families in Appendix H could offer researchers a clearer understanding of model weaknesses by language or domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)