HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks
Authors: Adnan El Assadi
Isaac Chung
Roman Solomatin
Niklas Muennighoff
Kenneth Enevoldsen
Venue: ICLR 2026 Poster (2026)
Keywords: Human Evaluation
Embeddings
MTEB
Benchmarking
NLP
Multilingual
Primary area: datasets and benchmarks
TL;DR: We introduce HUME, a framework for measuring human performance on embedding tasks, providing baselines across 16 datasets to interpret model scores.

Abstract: Comparing human and model performance offers a valuable perspective for understanding the strengths and limitations of embedding models, highlighting where they succeed and where they fail to capture meaning and nuance. However, such comparisons are rarely made, as human performance on embedding tasks is difficult to measure. To fill this gap, we introduce HUME: Human Evaluation Framework for Text Embeddings. While frameworks like MTEB provide broad model evaluation, they lack reliable estimates of human performance, limiting the interpretability of model scores. We measure human performance across 16 MTEB datasets spanning reranking, classification, clustering, and semantic textual similarity across linguistically diverse high- and low-resource languages. Humans achieve an average performance of 77.6% compared to 80.1% for the best embedding model, though with substantial variation: models reach high performance on some datasets while struggling on notably low-resource languages. Our human annotations also reveal multiple dataset issues. We additionally benchmark nine LLMs as annotators on reranking, classification, and STS tasks, finding that they fall short of human performance (76.1% vs. 81.2%) despite offering scalability advantages. We provide human performance baselines, insights into task difficulty patterns, and an extensible evaluation framework that enables a more meaningful interpretation of results and informs the development of both models and benchmarks. Our code, dataset, and leaderboard are publicly available at https://github.com/embeddings-benchmark/mteb.

HUME: MEASURING THE HUMAN-MODEL PERFORMANCE GAP IN TEXT EMBEDDING TASKS
Adnan EI Assadi Carleton University
Isaac Chung Zendesk
Roman Solomatin SaluteDevices, MIRAI
Niklas Muennighoff Stanford University
Kenneth Enevoldsen Aarhus University
ABSTRACT
Comparing human and model performance offers valuable perspective for under- standing the strengths and limitations of embedding models, highlighting where they succeed and where they fail to capture meaning and nuance. However, such comparisons are rarely made, as human performance on embedding tasks is dif- ficult to measure. To fill this gap, introduce HUME: Human Evaluation Framework for Text Embeddings. While frameworks like MTEB provide broad model evaluation, they lack reliable estimates of human performance, limiting the interpretability of model scores. We measure human performance across 16 MTEB datasets spanning reranking, classification, clustering, and semantic textual similarity across linguistically diverse high- and low-resource languages. Humans achieve an average performance of 77.6% compared 80. 1 for the best embed- ding model, though with substantial variation: models reach high performance on some datasets while struggling on notably low-resource languages. Our hu- man annotations also reveal multiple dataset issues. We additionally benchmark nine LLMs as annotators on reranking. classification, and STS tasks, finding that they fall short of human performance (76.1% vs. 81.2%) despite offering scala- bility advantages. We provide human performance baselines, insights into task difficulty patterns, and extensible evaluation framework that enables more meaningful interpretation of results and informs the development of both models and benchmarks. Our code, dataset, and leaderboard are publicly available https: tht... com www.ddiieenneee.ee. ngs ben hma r / mteb
S Given
Ower
80
78.3 78.2
77.6
77.5
76.9
76.8
70.4
69.0
68.2
66.6
64.2
61.9
Avenge score (%)
60
40
lasper on visiee Iaggugee of
SFR Embantinnn-aae Minitaiin--Mistaall Mistral
minnttll Th-instract
Be ge-Owwnn-1L. 1.5 Si-instrost
Homm
sella, on 1.50 v5
Qwent Embaldag--o oll
multinguul- --ggge
multllngual le samall
muttlligual lel--eaa
mulati antaaf-entedddoo largev)
enfsedinngeems 300m
all-mpnet-hhan 22
all all-Mint M.16222
Figure I: Human performance versus 13 embedding models across 16 tasks. Humans rank 4th (77.6). showing competitive but not dominant performance. Darker shades indicate larger models.

---

INTRODUCTION
Embedding models are central to modern NLP systems, powering applications such as search, recom- mendation, semantic analysis, and information retrieval. Many benchmarks test the performance of embedding models, with the most comprehensive offering a diverse suite of tasks that test their gener- ality and robustness (Muennighoff et al., 2022; Xiao al. 2025 Enevoldsen et al. 2025). Despite these advances, the interpretation and quality of these scores are often unclear as it is absent of human performance references. Current metrics define performance in terms of theoretical maxima (e.g., MAP= ..0) that assume perfect consensus on task outcomes. However, many NLP tasks inherently involve ambiguity and disagreement (Plank) 2022), making model's score difficult meaningfully interpret without reasonable references. This interpretability gap has serious consequences. When benchmarks reward models for fitting noisy labels, labels where even human annotators disagree, the field risks "blind optimization". expending R&D resources to replicate annotation artifacts rather than achieve semantic progress. For instance, model achieves 0.85 MAP in reranking, it is unclear whether this should be considered strong. mediocre, or beyond what annotators typically achieve. This disconnect highlights the need for human-centered evaluation that contextualizes benchmark results. Importantly, human performance should not be treated as upper bound but as diagnostic signal: a way to understand where tasks are inherently noisy, where models may surpass typical annotator agreement, and where model behaviour diverges from human judgment.
To address this, we introduce HUME: a Human Evaluation framework for text embedding tasks. HUME evaluates annotator performance across four task categories: reranking, classification, clus- tering, and semantic textual similarity (STS), using 16 diverse datasets from the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al. 2022), adapted for human annotation feasibility. Through
experiments, we analyze task difficulty, quantify variation across humans, and compare results directly against state-of-the-an embedding models.
Our contributions are threefold: (1) a generalizable framework for human evaluation of embedding tasks, (2) empirical evidence of how humans perform across diverse datasets and task types, and (3) comparative analysis of models and humans that highlights strengths, weaknesses, and ambiguities both benchmarks and models that yield actionable insights. Together, these contributions establish a foundation for human-aligned evaluation of embedding models and guide future benchmark design.
Beyond human evaluation, we also investigate whether Large Language Models (LLMs) can serve as scalable proxies for human judgment. The promise of L.LM-as-amottator approaches is compelling: if LLMS can reliably replicate human judgments, they could enable large-scale benchmark development at lower cost. However, this assumes LLMs capture the same semantic distinctions humans make, rather than exhibiting systematic biases. We evaluate nine state-of-ie state-of-the-an the-art LLMs (GPT-5, GPT-4.1, Gemini, Mistral, and Qwen3 variants) on identical annotation tasks to assess their viability human proxies and identify task-specific limitations.
2 RELATED WORK
2.1 TEXT EMBEDDING MODELS AND MTEB
Text embedding models map natural language into dense vectors that capture semantic information.
They have progressed from static embeddings (Mikolov et al. 2013 Pennington et al. 2014) to
contextual encoders (Devlin et al. 2019 Liu al. 2019) and more recently to models optimized for embeddings like Sentence-BBin BERT (Reimers & Gurevych 2019) E5 (Wang et al.) 2022). and GTE
al. 2023). powering applications such as search, classification, and clustering.
To evaluate these models across its diverse use-cases, MTEB (Muennighoff et al. 2022 Enevoldsen ct al. 2025) has risen as the de facto benchmark framework for embeddings and consolidates evaluation across diverse tasks and datasets. Despite its breadth and commmunity-driver extensions
spanning multilingual, multimodal, and domain-specinii variants (Xiao et al.) 2025: Kasmaee et al.
2025| Tang & Yang 2025 Xiao et al. [2224] Ciancone al. 2024 Enevoldsen al. 2024 | Zinvandi
ct al. /025;/Wehrl et al. 2024 Poswiata et 2024; Snegirev et al. 2024). MTEB lacks human
performance baselines, making difficult to contextualize model achievements.
2

---

2.2 HUMAN EVALUATION IN NLP
Human evaluation is well established in NLP, especially for generative tasks like machine translation (Graham et al. 2013), summarization (Fabbri et al. (02I). and dialogue (Gupta al. 2019). In contrast, smbeddingbbasee tasks have relied almost exclusively on automated metrics, with little attention to human baselines.
In information retrieval, initiatives such TREC (Voorhees & Tice, 2000) collect human relevance judgments, but these serve as gold standards rather than benchmarks of human performance under model metrics (e.g., nDCG, MRR). Similarly, GLUE, SuperGLUE (Wang et al.) 2018 2011)) and MERA (Fenogenova et al. 2024) Russian GLUE-like benchmark report human baselines, but mainly for classification and reasoning tasks. For embeddings, works like STS (Cer et al. 2017) report inter-annotator agreement, yet these are not converted into model--coparatiee performance scores. This leaves gap: human performance embedding benchmarks such as MTEB remains largely unquantified.
3 METHODOLOGGY
3.1 FRAMEWORK DESIGN
Our framework builds on MTEB by establishing reproducible human evaluation protocols that align directly with model evaluation. It consists of task-specific annotation interfaces, principled dataset sampling, a standardized results format, and the use of aligned metrics. This design reveals where evaluation practices introduce ambiguity or inconsistency.
3.2 TASKS, DATASETS, AND METRICS
Our selection criteria ensures comprehensive coverage across multiple dimensions: (1) linguistic diversity: including both high-resource languages (English, Arabic, Russian) and lower-resource languages (Norwegian Bokmäl, Danish) to test cross-lingual generalization, (2) domain variety: spanning news, social media, encyclopedic content, scientific literature, and forum discussions to capture real-world application diversity, (3) construction methods: including both curated human annotations and synthetic dataset creation to understand how dataset origin affects human- model alignment, (4) task relevance: using tasks from established benchmarks widely adopted in the embedding evaluation community, and (5) task complexity variation: ranging from binary classification to fine- grained similarity judgments. This systematic selection ensures our findings generalize across the diverse landscape of embedding applications while maintaining direct relevance to existing evaluation frameworks.
Each task category uses primary evaluation metric enable consistent human moddl comparisons. We summarize the datasets, their domains, and the primary metrics applied in Appendix A Detailed task examples are provided in Appeendix C
Retrieval Proxy via Reranking We Reranking human-evaluable proxy for Information Retrieval. Direct human evaluation of large-scale retrieval is methodologically infeasible-requiring annotators evaluate thousands of candidate documents per query. Reranking preserves the core semantic challenge of discriminating query-candidate relevance while remaining tractable: humans evaluate only the top-k candidates, establishing baseline conceptually equivalent to embedding- space behavior and ensuring human model comparability.
3.3 INSTRUCTIONS
Human instructions are designed to match the task definitions exactly (e.g., identical label sets for classification, same 1-5 scale for STS) ensure valid comparisons. However, formal, detailed annotation protocols are not publicly available for many MTEB datasets, which limits our ability to verify alignment. To mitigate this, we designed instructions based on the original dataset papers'
With 0 being "The Leeit-Bchinds" and 5 heing "The Winners", cover eng: 5, ara: 5, rus: 4, dan: 3, nob: according the 0-5 scale by (Joshi al. 202T).
3

---

task descriptions, ensuring annotators understood the semantic distinctions required for each task. Instructions were piloted with a small subset before full annotation to identify and resolve ambiguities.
3.4 ANNOTATION PROCEDURE
Our annotations process follows a trend similar to recent embedding benchmarks Enevoldsen et al. (2025); Xiao et al. (2025) focusing on diverse set of tasks with fewer samples rather than large singular tasks. We choose this approach as it allow us to better cover the broad scopes of current benchmark. Annotators were recruited with a focus on cultural and language diversity.
All annotations are conducted in Argilla (Argilla Project Contributors 2025 using task-specific inter- faces: binary relevance for reranking, categorical labels for classification, free cluster ID assignment for clustering, and 0-5 similarity scores for STS. Sample sizes balance task complexity: reranking (20-49 queries), classification (40-48 examples), clustering (30 items), and STS (30-50 pairs).
All annotators were male, aged 20-35, from culturally diverse backgrounds, and experienced NLP practitioners with native or near-native proficiency in the evaluated languages. They followed structured guidelines and completed all annotations independently, without access to ground truth or model predictions. The downsampled task subsets used for comparisons are included in the MTEB package, with detailed task examples provided in Appendix C
English tasks were annotated by two annotators to enable agreement analysis. Multilingual tasks were annotated by a single annotator with corresponding language expertise. Inter-annotator agreement was assessed with aas-opppope metrics: Fleiss' kappa (Fleiss 1971) for classification, pairwise ARI (Strehl& & Ghosh| 2003) for clustering, pairwise Spearman correlation (Agirre et al. 2012a) for STS, and mean Spearman/Kendallss tau (Manning et al. 2008) for reranking. A detailed agreement analysis is provided in (npppendix F This controlled evaluation setup minimizes potential confounds from dataset variation and enables direct performance comparisons on identical evaluation instances.
3.5 MODEL SELECTION AND EVALUATION
We evaluate 13 embedding models chosen to cover multiple dimensions: (1) parameter scale (22M-7B), (2) architecture (encoder- and decoder-based), (3) instruction tuning (instruction-tunel and standard), and (4) multilingual capability (English and multilingual). This selection spans diverse computational budgets and training paradigms, capturing the current embedding landscape. All evaluated models provided in Appendixx H
All models are evaluated on the downsampled instances annotated by humans, using identical metrics, protocols, and computational settings. Human performance is computed using the metrics in Appendix A) mirroring MTEB protocols. For primary analyses, we report MAP for reranking, Accuracy for classification, V-Measure for clustering, and Spearman correlation for STS.
To account for sample size constraints, we determine statistical significance using 95% confidence intervals computed via Wilson Score Intervals (accuracy) and Fisher z-transformations (correlation), as detailed in Appendix E
3.6 LLM-AS-ANNOTATOR EVALUATION
To assess whether automated evaluation can proxy human judgment, we evaluate nine state-of-the-an Large Language Models (LLMs) as annotators on the exact same tasks. We employ GPT-5 (full and mini), G (full and mini), Gemini 2.5 Flash, Mistral 2maall244-Innttruct, and three Qwen3 variants (30B, 32B, Coder-30B), prompting them with identical instructions provided to human annotators (see Appendix C).
LLMs receive the same task instances, evaluation metrics, and scoring protocols human annotators, enabling direct performance comparisons. For classification and STS tasks, LLMs provide categorical labels or numerical similarity scores. For reranking, LLMs rank candidate documents by relevance to the query. Clustering tasks were excluded from LLM evaluation due to fundamental difficulties in eliciting consistent, structured cluster assignments from generative models.
This controlled setup determines whether LLMs can serve as scalable, low-cost proxies for human evaluation whether they exhibit systematic biases that limit their utility for benchmark development.
4

---

By evaluating LLMs on the same instances as humans and embedding models, we can directly compare their annotation quality and identify task-specific strengths and limitations.
4 RESULTS AND ANALYSIS
Figure provides an overview of human performance relative to 13 state-of-the-ar embedding models across 16 tasks. Human annotators rank 4th overall with a score of 0.776, trailing 3 large models but outperforming 10 others. However, and data reliability. As shown in Table humans neither represent a uniform performance ceiling nor lower bound, but rather occupy middle ground that varies significantly by task category, language, and dataset quality.
ranking obscures the nuance of task difficulty
We computed 95% confidence intervals for human performance using metric-approoriite methods (Wilson Score Intervals for classification accuracy, Fisher z-transformation for porrelation-based metrics, and empirical annotator ranges for clustering and reranking). Models perform outside human CIs in 14 of 26 tasks (p < 0.05), often on datasets with low inter-annotator agreement where "superhuman" performance may reflect artifact fitting rather than genuine capability (see (Appendix for methodology and complete results). Below we analyze performance patterns by task category and language, with full per-task results in Appendix B
Classification
Clustering eng
Reranking
STS
Overall
Model
ara
eng nob rus ara
rus dan eng nob ara
eng rus
Number of datasets
(1) (4) (1) (1)
(1) (4) (1) (1)
(4)
(1) (1) (4) (1)
(26)
M--L--2
$7.2 58.8 51.7 55.5 35.2
24.5
55.1
31.4
78.4
93.7
71.2
6.2
83,5
33.1
61.9
all-mppet-base-v2
53.5 62.0 47.0 60.5
21.4
22.9
59.7
36.9
79.0
93.3
80.5
13.2
83.0
42.2
63.4
c5-mistral-7b-instruct
74.5 70.0 70.5 70.0
68.5
76.0
82.7
77.7
90.6
96.4
86.1
16.0
85.9
63.0
78.2
emhxcdtinggemma-R0mm
71.0 58.6 54.0 73.5 19.2
43.7
65.1 36.9
74.3 86.9
71.4
36.7
69.9
66.2
64.2
75.2 76.5 70.8 74.5 73.7 67.1
75.9
72.2
84.3
95.3
87.8
28.8
84.0
54.2
77.5
63.5 87.1 70.5 79.8 64.3 54.7 83.2
43.7
90.1
95.8
90.0
40.9
88.1
69.5
80.1
multilingual-e5
----euallll--------- base
75.8 64.7 73.8 77.2 35.9 40.6
45.6
36.2
92.2
94.4 87.5
31.0
85.2
62.7
68.2
multilingual-e5--aa
77.0 64.9 75.0 80.0 34.6 31.0 52.5 46.9
95.0
95.3 92.2
33.8
86.3
68.8
70.4
multilingual-c5--e -maall
72.2 62.2 69.2 81.2 35.5 38.0
51.7 59.1
88.6
94.2 88.3
28.8
85.2 60.3
69.0
mxhai-embed-largeee
57.2 66.4 52.2 59.0 26.5 34.2 61.9 30.5
91.8
94.5
82.0
12.7 87.6 43.7
66.6
77.2 74.7 59.8 74.8 78.8 58.5 68.4
68.3 90.0
95.5
83.6
38.0
88.5
60.3
76.8
SFR EIRREEmedding Mistral
77.5 69,8 68.8 72.5 73.1 71.2
85.1
68.9 89.2
96.3
86.
1 15.3
86.4
64.0
78.3
@aell_ee__1.B____
65.8 84.0
67.0 79.2 36.8
42.6
78.6
46.7
91.7
96.0
88.6
37.2
86.7
62.1
76.9
Human
95.0 70.3 85.0 76.0 62.7 67.4 68.0 91.4 87.2 89.8 67.5 83.1 58.7
77.6
Table 1: Human performance compared to 13 embedding models across task categories and languages. Bold indicates highest performance (human or model), underline indicates best model performance. Humans achieve top performance in 5 of 14 aggregated task-language pairs, particularly excelling in non-English sentiment analysis and Arabic semantic similarity. Overall results are aggregated over the 26 lankulae pairs.
4.1 PERFORMANCE PATTERNS BY TASK CATEGORY
Figure shows human performance relative to model performance across all 26 task-language pairs. Each task shows human performance (point) positioned within the full spectrum from worst to best model performance (range bars). Humans consistently perform in the upper portion of model ranges, typically exceeding median model performance (61.5% of tasks) while rarely matching the best models (15.4% of tasks). Classification tasks show the strongest human performance, with humans outperforming all models in of tasks, while clustering and reranking reveal consistent gaps where humans fall short of poppeefforming models. Detailed gap analysis can be found in Ampppedix
Classification: Human performance averages 70.3, ranging from 45.8 on emotion classification (N = 0.39, fair agreement) to 95.0 on Arabic sentiment analysis. Models generally exceed human performance (best: 87.1), but humans outperform models nog-English sentiment analysis, particu- larly in Arabic (95.0 vs. 77.5) and Russian (92.5 vs. 81.2), likely benefiting from native cultural and linguistic understanding that current models fail to capture.
5

---

[EN
Nuttlingual Sentiment [M]]
Hutdingual Sentment: [EN]
Multlingual
Classification
Mu Bllnnuall Sendmeent: [U]
Cenversations [EN)
Tweet Sontientt
ArXiv [EN)
S18200 [AR]
SI8200 (DA)
Clustaring
S18200
SI8200 IRU)
MKKCI6ss|EN
SICKR
STS12 (EN)
51522 (EN)
[RU]
STSBcnchmari
Coro17 [EN]
Robusto4 [EN]
Human z Best
Reranking
WRippeta [DA]
Human fange
Mkkpesa
Humen Weras
Range (ow18))
Wik pedia IND]
100
Performance (%)
Figure 2: Comprehensive view of human performance relative all model performance ranges across 16 tasks by language.
Clustering: Humans average 67.4 V-measure with extreme variation. Near-perfect performance on WikiCities (97.6, ARI 0.91) contrasts sharply with poor performance on ArXiv papers (49.2, ARI -0.001). Models consistently outperform humans (best: 85.1%). The poor inter-annotator agreement ArXiv indicates fundamental task ambiguity rather than human limitation.
Reranking: Humans achieve strong performance (87.2 average MAP) with high inter-annotator agreement (p 0.64-0.85), demonstrating intuitive relevance understanding. Models exceed human performance (best: 96.4), but the high human agreement suggests these tasks align well with human judgment and provide reliable evaluation targets.
STS: Humans average 83.2 Spearman correlation, with notable variation: STS12 achieves 91.2 while STS22-Russiar drops 58.7. likely reflecting dataset quality issues discussed in $4.3 Models achieve comparable performance (best: 88.5), with moderate inter-annotator agreement (p- 0.58-0.77).
These results reveal critical insight that challenges conventional evaluation paradigms: human performance variation often reflects task quality rather than human limitations. Tasks with high human performance and agreement (reranking, toxieity classification) represent well-specified problems with clear ground truth, while tasks with low human agreement (emotion classification, academic clustering) may suffer from ambiguous annotation guidelines or inherently subjective judgments. Models achieve "superhuman" performance by reproducing consistent label patterns from training data, but this consistency may mask fundamental issues with task specification. Rather than treating low human performance ceiling to surpass, our findings suggest that it often signals the need for improved task design and clearer annotation frameworks.
4.2 CROSS-LINGUAL PERFORMANCE ANALYSIS
Human win rates vary substantially by language and comparison baseline (see Appendix for detailed breakdown). Against the best models, humans win only 15% of tasks overall, but this rises to 62% against median models. The advantage is strongest for non-English tasks: humans achieve a 29% win rate multilingual tasks versus 0% on English-only tasks against best models.
Arabic exhibits the strongest human advantage: a 67% win rate against the best models and 100% against the mean, with the largest gap in semantic similarity (67.5% 40.9%, 26.6-point margin).

---

Russian and Norwegian also show consistent human superiority in sentiment amalysis, where humans achieve 92.5% and 85.0%, respectively, substantially outperforming the best models. These advan- tages likely stem from cultural and contextual knowledge that models fail to capture, especially in lower-resource languages.
English tasks more balanced, with models generally matching or exceeding humans, reflecting the language's 's dominance in training data. Danish shows mixed outcomes, possibly due to stronger multilingual coverage for Germanic languages.
4.3 DATASET QUALITY AND EVALUATION CHALLENGES
Our analysis reveals systematic quality issues in several MTEB datasets that fundamentally com- promise their reliability as evaluation benchmarks. Human performance variation often correlates with underlying ambiguity rather than genuine human limitations, providing diagnostic tool for identifying problematic evaluation targets. See [Appendix Difor further qualitative analysis of these failure modes; below, we highlight two such examples.:
Emotion Classification Ambiguity: The emotion classification dataset exemplifies inherent labeling ambiguity, achieving only fair inter-annotator agreement (x 0.39) with 52.1% consensus. Real examples demonstrate the fundamental challenges: "I feel like such a noob when the customers make really dull and stupid jokes that im supposed to find funny" could reasonably be labeled as sadness (0), anger (3), or even surpris (5) depending on interpretation. Similarly, "I am feeling very indecisive and spontaneous" contains mixed emotional states that resist single-label categorization. Sarcastic expressions like "I got paid too much because get so many deliveries at work Im feeling bit shamed" present surface emotions that differ from intended meaning. When human experts fundamentally disagree correct answers for such inherently ambiguous cases, the apparent "superhuman" model performance (87.15 1% vs. 45.8% human) likely reflects consistent reproduction arbitrary majority label patterns rather than superior emotional understanding.
ArXiv Clustering Breakdown: Academic paper clustering shows complete breakdown of human agreement (ARI=0.001). indicating fundamental disagreement about how to categorize academic papers. Real examples illustrate the core ambiguity: papers like "Self- Solf--Suereeieed Audio Visual Representation Learning with Relaxed Cross- Modal Synchronicity" could legitimately cluster with computer vision, machine learning, or audio processing groups depending on the annotator's s perspec- tive on primary methodology versus application domain. "The architecture of innovation: Tracking face-to-face interactions with ubicomp technologies" spans social science, computer science, and architecture domains. Such interdisciplinary papers create fundamental disagreement about correct clustering approaches, with no objectively correct answer. The task uses derived labels from ArXiv categories, but the core issue that academic papers often span multiple domains, making any single clustering scheme inherently ambiguous. The high model performance (84.6% vs. 49.2% human) suggests that the models are reproducing consistent labeling patterns rather than solving the fundamental categorization challenge.
High-Qu Qualt-yultty Benchmark Identification: Conversely, tasks with high human agreement provide reliable evaluation targets. Reranking tasks achieve strong inter-annotator agreement (p 0.64-0.85) with clear performance targets, while toxicity classification shows moderate agreement (x = 0.55) with 77.8% annotator consensus. These represent genuine evaluation challenges where model improvements likely reflect meaningful progress rather than pattern matching to flawed labels.
These patterns suggest that apparent "suuerhumann" model performance often occurs precisely where human agreement is lowest, indicating that models excel not through superior understanding but through consistent reproduction of systematic labeling patterns. This raises concerns about the label quality in embeddings benchmarks, and we encourage future benchmark developers to critically examine the datasets before including them in benchmark, potentially using human annotations framework like HUME. Detailed analysis of specific quality issues is provided in Appendix D
4.4 CAN LLMS REPLACE HUMAN ANNOTATORS?
We benchmark nine LLMs as annotators to assess whether they can serve as reliable proxies for human judgment. As shown in Table the best-performing L.LM (GPT-4.1-mim) achieves 76. 1%
7

---

GPT-5
GPT-4.1
Gemini
Mistral Small-24B-
Qwen3
Best Emb.
Task
Human
Full
Mini
Full
Mini
2.5 Flash
30B
32B
Coder
Score
Classification
79.1
78.9
77.2
76.6
76.1
77.6
73.8
74.2
73.0
76.3
80.3 (jasper)
Reranking
88.3
75.1
75.5
75.7
77.2
76.2
78.0
75.6
74.8
73.8
94.8 (e5)
STS
76.5
73.0
69.0
74.9
74.9
69.3
75.0
67.1
68.6
71.3
77.1 (jasper)
Average 81.2
75.8
74.1
75.8
76.1
74.5
75.5
72.4
72.2
73.9
Table 2: LLM--as-annotator performance compared human annotators and best embedding models per task category. Human and LLM performance is computed over 19 task- languag pairs (clustering tasks excluded due difficulty eliciting cluster assignments). Best embedding model per task category shown with abbreviated name: jasper (jasper en_ vision_lant Laanalllanggee___ v1), SFR (SFR-E E-FR--mbedddng- Mistral), e5 Saalttlingualle---argeee Bold indicates best LLM performance (humans and embedding models consistently outperform LLMs and are not bolded).
average accuracy, falling short of human performance (81.2) (81.2%). We exclude clustering tasks in this comparison due to the difficulty of eliciting cluster assignments from generative models.
Task-specific patterns reveal important limitations. On classification, LLMs approach human perfor- mance (GPT-5: 78. 8.9% vs. Human: 79.1%). However, a substantial gap emerges in reranking, where humans achieve 88.3% compared to the best LLM at 78.0% (Mistral-Small)
10-point deficit on tasks where humans show high agreement (p = 0. 64-0.85)), For STS, humans (76.5%) outperform all LLMs (best: 75.0%, Mistral-Small). Embedding models achieve the highest scores across all categorics (Classification: 80.3%, Clustering: 79. 1 %. Reranking: 94.8%, STS: 77.1%). Detailed per-task LLM performance is provided in Appendixx I
To assess whether humans and LLMs face similar challenges, we computed Spearman rank correla- tions between human and LLM performance across the 19 task-language pairs. The moderate positive correlation (p= = 0.52, < 0.05, n = 19) indicates that tasks where humans perform well also tend to be tasks where LLMs perform well, suggesting partially shared difficulty patterns. However, the correlation is moderate rather than strong, indicating that humans and LLMs do not face identical challenges across all tasks (see [.1 for detailed correlation analysis).
DISCUSSION
5.1
TASK QUALITY AND EMBEDDING EVALUATION RELIABILITY
Our analysis reveals striking pattern: models achieve their highest relative performance precisely where human experts show the least agreement. This confirms that on low-quality datasets, current metrics do measure semantic understanding but rather the model's ability reproduce consistent annotation artifacts.
This finding reframes the role of human evaluation in benchmark design. Rather than serving merely as performance benchmark, human consensus establishes a validity threshold for evaluation tasks. When models significantly exceed this bound on low-agreement tasks, signals that the benchmark itself has lost its descriptive power: the task may be measuring annotation artifacts rather than the capability it claims assess. HUME provides the empirical mechanism to identify and deprecate these invalid evaluation targets, ensuring that leaderboards measure genuine capability rather than overfitting to noise.
Tasks with high human agreement, such as reranking and toxicity classification, provide reliable evaluation targets. Conversely, low-agreement tasks (e-g., ArXiv clustering, Emotion classification) suffer from ambiguous guidelines subjective judgments. Cultural factors add another dimension to evaluation reliability. Humans retain substantial advantages in Arabic semantic similarity and multilingual sentiment analysis, revealing genuine model limitations in cross-cultural understanding.
These findings suggest reliable evaluation depends as much on task quality as model capability. Rather than treating high model performance as automatic progress, recommend prioritizing high-agreement tasks for development, addressing cultural competence gaps, and critically examining whether apparent model superiority on ambiguous tasks reflects genuine capability evaluation
8

---

bias. Novel benchmarks should report human agreement alongside model scores: 85% accuracy emotion classification (185% of human performance, A = 0.39) represents a fundamentally different achievement than 85% on reranking (97% of human performance, p = 0.75).
5.2 LLM-BASED EVALUATION
Our LLM annotation experiments reveal important limitations for using LLMs as proxies for human judgment. While LLMs achieve competitive performance on classification tasks, substantial gap emerges reranking, where humans significantly outperform even the best LLMs. Notably, reranking tasks show strong inter-annotator agreement (p = 0.64-0.85). suggesting LLMs struggle with precisely the nuanced relevance judgments where human consensus is highest. This pattern contrasts with classification, where lower human agreement (x = 0.24-0.55) coincides with near- parity between humans and L.MMs.
This has important implications for benchmark development. The reranking gap suggests that LLMs may not reliably capture the semantic distinctions humans make well-defined tasks, even as they approach human performance on more ambiguous ones. Using LLMs for large-scale annotation may therefore introduce systematic biases, particularly for tasks requiring fine-graine ain-ggrined semantic judgments. The architectural mismatch between generative LLMs and discriminative evaluation tasks further limits their utility, as evidenced by our inability to evaluate clustering tasks.
While LLMs offer scalability advantages, these limitations suggest they should augment rather than replace human annotation, particularly for benchmark development where task quality directly impacts model development priorities. Future work should explore hybrid approaches that leverage LLM efficiency for initial annotation while reserving human judgment for high-agreement tasks and uncertain cases.
5.3 IMPLICATIONS FOR MODEL DEVELOPMENT AND EVALUATION PRACTICES
Our findings reveal concrete directions for both embedding model development and evaluation methodology that address the fundamental quality issues we've identified.
Development efforts should focus on tasks where humans demonstrate both high performance and agreement, as these provide the most reliable benchmarks for measuring genuine progress. Reranking tasks, with their clear performance targets and strong agreement (p = 0.64 0.85), offer dependable evaluation where the persistent model- human gap (96.4% 87.2%) represents meaningful challenges requiring better modeling of relevance relationships. Toxicity classification, despite moderate agreement (x = 0.55), provides another reliable target with 77.8% human consensus. In contrast, optimizing for tasks with poor human agreement (emotion classification x = 0.39, ArXiv clustering ART = -0.001) may lead models to excel at reproducing arbitrary labeling patterns rather than developing genuine semantic capabilities.
Prioritize High-A gitemmtt Tasks for Development:
Address Cultural and Linguistic Competence Gaps: The substantial human advantages in non- English tasks reveal critical model limitations that scaling training data alone cannot address. Arabic semantic similarity shows the largest human advantage (67.5% vs. 40.9% best model), while multilin- gual sentiment demonstrates consistent human superiority in non-English languages (95.0% Arabic, 92.5% Russian). These gaps suggest that current models lack the cultural and contextual knowledge necessary for cross- loos-linggall understanding. requiring architectural innovations or training approaches that go beyond simple data scaling to capture cultural nuances and contextual understanding.
Our analysis identifies specific datasets that compro- mise benchmark reliability and should be replaced in future MTEB iterations: emotion classification. ArXiv clustering, and STS22- Russian (systematic parsing artifacts). These tasks provide unreliable evaluation targets that may mislead model development efforts by rewarding pattern matching to flawed gold standards. Replacement datasets should demonstrate reasonable human agreement and clear task specifications, validated through human evaluation before inclusion in benchmark suites.
Replace Problematic Benchmark Datasets:
Report Dataset Quality Measures: Model performance should be interpreted in light of dataset quality indicators to provide proper context for evaluation results. We propose that benchmark leader- boards report human agreement metrics alongside model scores. A model achieving 85% accuracy

---

emotion classification (185% of human performance, 0.39) represents fundamentally different achievement than 85% on reranking (97% of human performance, = 0.75). High model performance low-agreement tasks should be viewed skeptically as potential artifacts of flawed evaluation targets rather than genuine capability improvements. For tasks where human agreement falls below established thresholds (N: 0.4 P 0.6), recommend either improving task specifications or removing the dataset from benchmark suites entirely. However, it is important to recognize that some degree of human disagreement reflects natural variability in judgment rather than dataset flaws. Future benchmarks could incorporate evaluation frameworks that preserve and leverage this variability rather than collapsing single gold labels (Plank 2022 Basile et al. (021).
5.4 LIMITATIONS
Our study has several limitations. First, our prioritization of breadth over depth- covering 16 diverse tasks tasks-resulted resulted in smaller sample sizes per task (20-50 instances). While we provide significance analyses to validate statistical conclusions, larger samples would better capture the full complexity of human performance variation and provide more robust estimates of human judgment distributions.
Second, multi-tast tast design constrained task-specific training. Annotators were average above- average raters without specialized training; experts would likely perform better, particularly on technical tasks. This was compounded by sparse annotation guidelines in original datasets, making alignment with original procedures difficult-thoug! this reflects realistic annotation scenarios where perfect replication is often infeasible.
Third, while three annotators participated overall, only two annotations were collected for most tasks, limiting our ability fully characterize agreement patterns. Additionally, while we ensured cultural and linguistic diversity among annotators, they were all male and aged 20-35, which does not fully represent human judgment distributions across broader demographic groups.
Fourth, While our study quantifies where models diverge from human performance, it does not fully explain why these gaps arise. Identifying the underlying factors such as gaps in training data coverage, domain cultural biases, and linguistic variability, particularly in low-resoure, settings remains a critical direction for future research. However, detailed information about model training corpora is often unavailable, limiting such analysis.
Finally, while our study evaluates human performance across diverse tasks, we did not systemat- ically investigate how task design features-suu as specification clarity versus meaningful chal- lenge-affeed affect human agreement and model discrimination. The field needs rigorous research on these design principles to avoid both ambiguity, which depresses human agreement, and oversinplification, which diminishes discriminative power.
CONCLUSION
We introduce HUME, comprehensive human evaluation framework for MTEB, addressing a critical gap in understanding empirical performance bounds for embedding models. By measuring human performance across 16 datasets spanning reranking, classification, clustering, and STS, we establish statistically robust baselines that reframe how model achievements should be interpreted.
Our findings show that human performance varies substantially by task categories. Tasks with high agreement provide reliable benchmarks, while low-agreement tasks often reveals design issues in the task formulation.
Finally, our benchmarking of nine LLM-as-annotator systems demonstrates that while they offer scalability, they cannot yet replace human judgment entirely. The best LLM (GPT-4.1-mini 76. 1%) falls short of human performance (81 1.2%% particularly on reranking. This suggests that future benchmarks should leverage LLMs to augment, but not replace, human evaluation.
6.1 ETHICAL CONSIDERATIONS
Annotators were co-authors who consented to the study. There were no external crowd workers involved in any part of the annotation process.
10

---

REFERENCES
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. arXiv preprint arXiv:2309.07445. 2023.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre, SemEval-2012 task 6: A pilot semantic textual similarity. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret (eds.). *SEM 2012: The First Joint Conference on Lexical and Computational Semantics Volume I: Proceedings of the main conference and the shared lask, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012). pp. 385-393, 393, Montréal, Canada, 7-8 June 2012a. Association for Computational Linguistics. URL https: ac wwaaaanntohoog.. /rg///// S 222 1051/
Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. Semeval-2012 task a pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics Volume Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Evaluation, SemEval '12, pp. 385-393, USA, 2012b. Association for Computational Linguistics.
Argilla Project Contributors. Argilla: A collaboration tool for building high-quality ai datasets. https://arqilaa.io) 2025. Version as of Sep. 2025; open-source data labeling / curation platform.
arXiv.org submitters. arxiv dataset, 2024. URL /Rt.ttpss///wwkeggglee...///////
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 258-266, Marseille, France, June 2022. European Language Resources Association. URL btttps://aclanthhe loyolog...... or9gy22.0000...... rrec-1
27
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. We need to consider disagreement in evaluation. In Kenneth Church, Mark Liberman, and Valia Kordoni (eds.), Proceedings of the /st Workshop on Benchmarking: Pust, Present and Future, pp. 15-21, Online, August 2021. Association for Computational Linguistics. URL http:s///acllantooooo..ooooe wrg/
doi: 10. 1/6633/w12021..ppptl
37
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Jopez-Gazpic and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens (eds.), Proceedings of the 1 1th International Workshop on Semantic Evaluation (SemEvul-2017), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 1/1552211117-2011. URLhttps: /ttpo:://accaaaao hat/a/aaannnttcccooo htt.:/ooooorrgo//11
S177 2001/
Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott Hale, David Jurgens, and Mattia Samory. SemEvval22022 task 8: Multilingual news article similarity. In Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, and Shyam Ratan (eds.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 1094-1 106, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18655/w1/20222.eeeee. https::/aclantho...
URL
ort. 2022. wtw.://....ee...aaaaae.......... semeval 155.
Mathieu Ciancone, Imene Kerboua, Marion Schaeffer, and Wissam Siblini. Mieb-french: Resources for french sentence embedding evaluation and analysis, 2024. URL https:t/axxii....ttt.m org/
abs 2tt//205..2000 20468
cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and nithum. Jigsaw unintended bias in toxicity classification, 2019. URL https::/kkaggee.cccoo com/

---

Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associa- tion for Computational Linguistics, 2019. URL httppp//aa...smmtttstoo....oooea
Ming-Wei
org/
CorpusTD: 52966339999
Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff, and Kristoffer Nielbo. The scan- dinavian embedding benchmarks: Comprehensive assessment of multilingual and monolin- gual text embedding. In Advances in Neural Information Processing Systems, 2024. URL htttps://nips.cc/// rtual /202al/ /ttpa/////////////////550
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiñíski, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ómer Çağatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafal Pošwiata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Bjöm Plüster, Jan Philipp Harries, Loic Magne, Isabelle Mohr, Mariya Hendriksen, Dawei Zhu, Hippolyte Gisserot-Boukblef, Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Suppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang, Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal Chitale, Simone Tedeschi, Nguyen Tai, Artem Snegirev, Michael Günther, Mengzhou Xia. Weijia Shi, Xing Han Lů, Jordan Clive, Gayatri Krishnakumar, Anna Maksimova, Silvan Wehrli, Maria Tikhonova, Henil Panchal, Aleksandr Abramov, Malte Ostendorff, Zheng Liu, Simon Clematide, Lester James Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Hongjín Su, Jimmy Lin, Howard Yen, Lasse Hansen, Sara Hooker, Chenghao Xiao, Vaibhav Adlakha, Orion Weller, Siva Reddy, and Niklas Muennighoff, MMTEB: Massive multilingual text embedding benchmark. arXiv preprint arXin:2502.133555 2025. doi: 00.48850/a2Xi..2222.3332
URL htthhttps:/ http///axvvv..o org/ abs/ wwwpp//wo/g//////2...aa.. 13595
Alexander R. Fabbri, Wojeiech Kryáciñíski, Bryan McCann, Caiming Xiong, Richard Socher, and
Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-400 2021. doi: 10.1162/tacl a 00373. URL
Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Denis Dimitrov, Alexander Panchenko, and Sergei Markov. Mera: A compre- hensive IIm evaluation russian, 2024. URL htt.iitttt:://arxtttat org abs 2243.
R. A. Fisher. Frequency distribution of the values of the correlation coeffients in samples from an indefinitely large population. Biometrika, 10(4):507-521, 05 1915. ISSN 0006-3444. doi: 10.1093/biomet/10.4.2
URI htt.ttttts://d..ooom www.:/i..orr///// 10. 1 1093/b0 comet/10. 4. 507
Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76 (5):378-388 1971. doi: 10.1037/h0031619.
Wikimedia Foundation. Wikimedia downloads. URL http:////amm.....iiiiiaaaaa.....
Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. Tweac: Transformer with extendable qa agent classifiers. arXiv preprint, abs/2104.07081, 2021. URL httt:://rxiiy
org/abs/ 2104.1110 07081
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Continuous measurement scales in human evaluation of machine translation. In LAW@ACI, 2013. URL https://apii
semanticscha
ozt//orpusID:112833
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P. Bigham. Investigating evaluation of open-domain dialogue systems with human generated multiple refer- ences, 2019. URL /RLnttps.//rriv.o//////
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 193-2199921. Dec 1985. doi: 10. (007/BF0190807)) URL. htt.::ttp://// tttttp:://doi.org/0.... 1007/ 908075
12

---

Kalervo Järvelin and Jaana Kekäläinen. Cumulated rain-based evaluation of ir techniques. ACM Trans. Inf. Syst. 20(4):422 446, October 2002. ISSN (046-8188. doi: 10.114555221555824118
582418
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world, 2021. URL https:
abs/ / 2004.2044.. 09095
Ali Shiraee Kasmaee, Mohammad Khodadad, Mohammad Arshi Saloot, Nicholas Sherck, Stephen Dokas, Hamidreza Mahyar, and Soheila Samiee. Chemteb: Chemical text embedding benchmark, an overview of embedding models performance & efficiency on specific domain, 2025. URL
https://arxiv: org abs/ 2412. 00532
Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. Open source strikes bread
new fluffy embeddings model, 2024. URL https:://wmw //wp:://wwwww. www.wixeedrrrdd.aa//..... mxbai Sesbai-embed-a wwwee-eeeeeeee largee--aaaeeeee v1
Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint vrXiv:2309.12877 2023.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL https:://aaaat
bxq/abs/ae 23088. 03281
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.
ArXiv, abs/1907.11692 2019. URL https:/ /aapi. semant icscholar. orggCorpusIDD 198953378
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) 14), pp. 216- 223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL /3tp1////////33333/ Paper_appr.._. pdf
Philip May. Machine translated multiiingu benchmark dataset. 2021. URL httts:///ttththt
com/ wtt.PPhiiiaaaatts-mmbmllll
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr- embediing-mistrallenhaare text retrieval with transfer learning. Salesforce Al Research Blog, 2024. URL https: http:::/wwwwwwwww/sssssse...e..
sa les wtt.c/cmcooooo.. httpeee/////oooooeeeeeeeeeeeeeee
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. In Proceedings of the 27th International
Conference on Neural Information Processing Systems - Volume 2, NIPS' 13, pp. 3111-3119,, Red Hook, NY, USA, 2013. Curran Associates Inc.
Sepideh Mollanorozy, Mare Tanti, and Malvina Nissim. Cross- Mooss-lingual transfer learning with (P) persian. In Lisa Beinborn, Koustava Goswami, Saliha Murado
uglu, Alexey Sorokin, Ritesh Kumar, Andreas Shcherbakov, Edoardo M. Ponti, Ryan Cot- terell, and Ekaterina Vylomova (eds.), Proceedings of the 5th Workshop on Research in Com- putational Linguistic Typology and Multilingual NLP, pp. 89-95, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 10018663/11/2022sssggyyp https://cclattholoogg www.:/rgggg//. . sigtyp-1.9
URL
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.073/6 2022.
Karl Pearson. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240-242. 2422, 1895. doi: 10. 1098/rspl.1895 1895.0041.
13

---

Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daclemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.31 10.3115//1
URL https: //acl/acccc antho www.anntwooooooooooooooo..e ogt.org//oo.111.////////
Barbara Plank. The "problem" of human label variation:
On ground truth in data, modeling and evaluation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10671-10682, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. 12658866/vll 2tt05////2222...ml-mmaaiai emm.pp ma in. 731/
URL https: MRRLattts///aclaatool..oo orgp://oooyoo/222222222//
Rafal Poáwiata, Stawomir Dadas, and Michal Perelkiewicz. Pl-mteb: Polish massive text embedding benchmark. arXiv preprint arXiv:2405.10133 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing, 2019. URL. https
7/api www.mmanniiccchoooooo
Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy- based external cluster evaluation measure. In Jason Eisner (ed.), Proceedings of the 2007 Joint Conference on Empir- ical Methods in Natural Language Processing and Compulational Natural Language Learning (EMNLP-CONLA) pp. 410-420, Prague, Czech Republic, June 2007. Association for Computa- tional Linguistics. URL https: www.aaonnannooooo..ooooooooo.ott wrg/Do7-10033
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujui (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3687-3697, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/w//D188 1404. URL httpt: httttp:://allantho..oo
org/ orq/D18-1404
Artem Snegirev, Maria Tikhonova, Anna Maksimova, Alena Fenogenova, and Alexander Abramov. The russian-focused embedders' exploration: rumteb benchmark and russian embedding model design, 2024. URLptttps://arx/t owtp:ss.rraabbb/b/eroo.oo... 2408. 2 12503
Marina Sokolova and Guy Lapalme. A systematic analysis of performance measures for classifi- cation tasks. Information Processing & Management, 45(4))427-437, 2009. ISSN 0306-4573, doi: httphh/s/oonnogggo...oo0o.....o0 .02. http::/www............ttee/////e
http::::://www.cceeeeeeeeeeeeee com/
C Spearrman. The proof and measurement of association between two things. International Journal of 110 10 2010. ISSN 0300-5771. doi: 10. 10.1093/jj/dyq191. URL
Alexander Strehl and Joydeep Ghosh. Cluster ensembles - a knowledge reuse framework for combining multiple partitions. J. Mach. Learn. Res., (tnull):583611,, March 2003. ISSN 1532-4435. doi: 10.16211524433331777775
URL /ttp:::///....or/10..11
153244303321888 21127889735
Yixuan Tang and Yi Yang. Finmteb: Finance massive text embedding benchmark, 2025, URL
https://arxiv: org/ abs / /at.s/2522..0 10990
Henrique Schechter Vera. Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam. Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sanseviero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, Weiyi Wang, Zhe Li, Gus Martins, Jinhyuk Lee, Mark Sherwood, Juyeong Ji, Renjie Wu, Jingxiao Zheng, Jyotinder Singh, Abheesht Sharma, Divya Sreepat, Aashi Jain, Adham Elarabawy. AJ Co, Andreas Doumanoglou, Babak Samari, Ben Hora, Brian Potetz, Dahun Kim, Enrique Alfonseca, Fedor Moiseev, Feng Han, Frank Palma Gomez, Gustavo Hernández Ábrego, Hesen Zhang, Hui Hui, Jay Han, Karan Gill, Ke Chen, Koert Chen, Madhuri Shanbhogue, Michael Boratko, Paul
14

---

Suganthan, Sai Meher Karthik Duddu, Sandeep Mariserla, Setarch Ariafar, Shanfeng Zhang. Shijie Zhang, Simon Baumgartner, Sonam Goenka, Steve Qiu, Tanmaya Dabral, Trevor Walker, Vikram Rao, Waleed Khawaja, Wenlei Zhou, Xiaoqi Ren, Ye Xia. Yichang Chen, Yi-Ting Chen, Zhe Dong, Zhongli Ding, Francesco Visin, Gaël Liu, Jiageng Zhang. Kathleen Kenealy, Michelle Casbon, Ravin Kumar, Thomas Mesnard, Zach Gleicher, Cormac Brick, Olivier Lacombe, Adam Roberts, Yunhsuan Sung, Raphael Hoffimann, Tris Warkentin, Armand Joulin, Tom Duerig, and Mojtaba Seyedhosseini. Embeddinggemma: Powerful and lightweight text representations, 2025, URL https: /t //w.ivv.aiivvv//aa//aaa org/ httg//bb////oo 2509. 20354
Nguyen Vinh, Julien Epps, and James Bailey. Information theoretic measures for clusterings Variants, properties, normalization and correction for chance. Journal of Machine Learning Research, 11:2837-2854, 10 2010.
comparison:
Ellen M. Voorhees and Dawn M. Tice. Building a question answering test collection. In Proceedings of the 23rd Anmual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '00, pp. 200-207, New York, NY, USA, 2000. Association for Computing Machinery. ISBN 5581132263. doi: 10.1 041114//345508.455577 URL https://doi
orq/10..1445345550 345577
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE; A multi-task benchmark and analysis platform for natural language understanding, In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP. pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.18653/w1W118-5
URL
org/ /w1 54467
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: a stickier benchmark for general-purpert language understanding systems. Curran Associates Inc., Red Hook, NY, USA, 2019.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-superriseed contrastive pre-training. ArXiv, abs/2212.03533, 2022. URLhhtps:://age www.nt
254366618
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.003: 00368, 2023.
Silvan Wehrli, Bert Amrich, and Christopher Imgang. German text embedding clustering benchmark. 2024. URL
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching information retrieval models to follow instructions, 2024. 2024.
Edwin B. Wilson. Probable Inference, the Law of Succession, and Statistical Inference. J. Am. Statist. Assoc., 22(158):209 212, 1927. doi: 10. 10.1000/01621455 191..499999...99299999
Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang. Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, and Niklas Muennighoff. Mieb: Massive image embedding benchmark, 2025. UtLtttts:://aatt
org abs /00./2504...0 10471
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian Yun Nie. C-pack: Packed resources for general chinese embeddings, 2024. URL https: 2309. 07597
Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models, 2025a. URL httttt::////ttttte
org/ abs /abs//112..e 2412219142
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embed- ding: Advancing text embedding and reranking through foundation models, 2025b. URL /tttps://arxile org
15

---

Erfan Zinvandi, Morteza Alikhani, Mehran Sarmadi, Zahra Pourbahman, Sepehr Arvin, Reza Kazemi, and Arash Amini. Famteb: Massive text embedding benchmark persian language. arXiv preprint arXiv:2502.11571, 2025.
A DATASET SPECIFICATIONS
Table 3 details the 16 datasets selected for this study, including their domains, descriptions, and the primary metries used for evaluation.
Datasets
Description
Metrics
Reranking
Core17, News21.
Information retrieval benchmarks (news, documents)
MAP, MRR @ 10. nDCG@ 10
Robust04 (Weller et al. 2024)
WikiMulti (Enevoldsen et al.
Wikipedia article reranking (eng. dan, nob)
2025
Classification
Emotion (Saaaiia et al. 2018
Emotion classification from social media text Sentiment analysis of tweets
Tweet Senti (Barbieri et al.
Accuracy, Fl, Weighted Fl
2022)
Toxicity (ejadams al. Multilingual Sentiment
2019
Toxic content detection
Sentiment classification (ara, eng, nob, rus)
Mollanorozy et al.
2023
Clustering
WikiCities (Foundation)
Entity clustering from Wikipedia Academic paper topic clustering (derived labels)
ArXiv (arXiv.org submitters
V-Measure, ARI, AMI
2024)
Reddit Geigle et al.
2021
Forum discussion topic clustering
S1B200 (Adelani
2023
Multilingual sentence clustering (ara, dan, eng, rus)
STSBenchmark MMay
2021
General semantic similarity benchmark
STS
SICK-R -R (Marelli al
2014
Semantic relatedness and entailment
Spearman, Pearson
STS12 Agire et al
20126
Shared task semantic similarity
STS22
hen et al.
2022
Multilingual semantic similarity (ara, eng, rus)
Table 3: Complete list of 16 datasets and evaluation metrics used for human annotation. For the
metrics we use MAP (Manning et al. 2008). MRR
(Manning et al. (008). nDCG (Jrvellin &
Kekäläinen 2002), Accuracy/Fi (Sokolova & Lapalme
2009). V-Measure (Rosenberg & Hirschberg 2007), ARI (Hubert & Arabie| 1985), AMI (Vinh et al. 2010 Spearman/Pearson (Spearman, 2010)
Pearson 1895), following MTEB implementations.
B DETAILED RESULTS BY TASK CATEGORY
This section provides comprehensive results for tasks, organized by category. Each table includes human performance alongside all 13 evaluated models, with inter-annotator agreement metrics where available.
Table 4 presents full results of the clustering tasks. Table presents full results of the classification tasks. Table 6 presents full results of the reranking tasks. Table 7 presents full results of the STS tasks.
C TASK EXAMPLES
This section provides screenshots of the actual Argilla annotation interfaces used in our study, illustrating the annotation challenges and interface design that human annotators encountered.
16

---

Fradent
Figure 3: Emotion Classification annotation interface showing the 6-category emotion labeling task. This task achieved fair inter-annotator agreement (x = 0.39) due to ambiguous emotional states and mixed emotions in social media text. Human performance: 45.8%, Best model: 87. %.
Figure 4: Tweet Sentiment Classification annotation interface demonstrating sentiment polarity annotation. This task achieved moderate inter-annotator agreement (ı: 0.48) with reasonable consensus on positive/negative sentiment. Human performance: 84.4%, Best model: 90.9%.
offectiy
acresting propiio dit teditage socks socore Emercen benhener 1e gomerye
Figure 5: ArXiv Clustering annotation interface showing academic papers that caused complete annotator disagreement (ARI -0.001) to interdisciplinary research overlap. Papers could be categorized by methodology, application domain, research community, leading to fundamental disagreement. Human performance: 49.2%, Best model: 84.6%.
Chake
thw inaale
Figure 6: Reddit Clustering annotation interface demonstrating thematic grouping of discussion topics. This task achieved fair agreement (ARI 0.34) due to overlapping themes across different discussion topics. Human performance: 68.8%, Best model: 100%.
stow rigents mentioe vonr arees inaale
Figure 7: SIB200 Clustering annotation interface showing multilingual sentence clustering task. This task achieved moderate inter-annotator agreement (ARI 0.42) with variation across languages depending cultural context and sentence complexity.
17

---

mork
Movt
Radi walkingg lak
dudud equadte dupans 2t'
de
Figure 8: Robust04 Reranking annotation interface showing document relevance assessment for information retrieval queries. This task achieved strong inter-annotator agreement (p 0.72). Human performance: 88.5%, Best model: 98.8%.
accroria to sobeer
Doar
Syrn Ange
Iedivvrr Encoue Sty delt Fagen miiing reth
Figure 9: Wikipedia Multilingual Reranking annotation interface demonstrating cross poos--lnggull rele- vance judgment. This task achieved moderate agreement (p = 0.64) due cross-lingual complexity.
Beitatty Suaert
Figure 10: STS12 annotation interface showing semantic similarity assessment using 0-5 scale. This well-curated dataset achieved strong inter-annotator agreement (p = 0.77). Human performance: 91.2%, Best model: 92.0%.
Figure 11: SICK-R annotation interface showing semantic relatedness and entailment task. This task achieved moderate agreement (p 0.68) due to task complexity. Human performance: 82.6%, Best model: 94.1 %.
18

---

Model
Arxiv
Reddit
SIB200
WikiCities
all-MiniLM-L6-v2
61.8
56.5
33.2
73.8
all-mpnet-base-v2
56.0
63.0
33.2
86.7
c5-mistral-7b-instruut
77.2
74.9
73.8
100.0
embeddinggemma-330m
73.0
72.8
36.0
70.3
gge-Qween-11.BB-insttttt
70.8
88.4
72.3
73.6
83.9
95.1
59.0
95.1
multilingual-c5-base
31.9
60.0
34.4
66.2
mulilinggua-c5-large
51.1
55.4
37.8
69.2
multiinggull-e5-small
30.5
76.9
38.7
72.6
mxbai-embeed-lrgee-l
48.3
72.1
33.0
90.8
Qwen3-Embedddig-00.6
69.1
76.3
65.5
74.8
SFR-EEmbeding-Mistra.
75.5
81.4
74.8
100.0
stella_enp _i__ee_____ en_1__B_v
84.6
100.0
44.2
86.4
Human
49.2
68.8
65.2
97.6
Table 4: Full clustering results.
Model
Emotion
MultilSenti
ToxicConvo
TweetSenti
all-MiniLM-L6-v2
42.1
57.9
66.4
59.6
all-mpnet-base-v2
44.0
61.5
60.4
58.4
e5-mistrra--7biinstuuu
47.3
75.9
68.0
75.8
embeldinggeemma-3000
40.4
64.6
66.4
67.8
ge--Qwen2-1.5B-insttrcc
56.2
78.1
78.7
79.3
jasper_en__ sin_vvsionn__o
laa__iiiion_annuaeeee. language_vi
75.4
77.3
86.7
90.9
multilingual-e5-base
36.2
78.6
66.0
69.1
multilinguaaleS-large
38.5
81.1
63.3
65.3
multilingual-e--smaall
33.8
75.8
65.1
69.6
mxbai-embed-largee-
42.1
64.4
68.7
65.8
Qwen3-Embeddingg-00
51.9
74.1
74.4
87.8
SFR-EEmbedding-Misttall
46.7
77.1
67.3
75.6
stella_ent
71.9
76.1
82.9
89.1
Human
45.8
87.5
73.3
84.4
Table 5: Full Classification results.
D TASK QUALITY ANALYSIS
D.1 DATASET QUALITY ISSUES
Our analysis revealed quality issues across multiple datasets that significantly impact human-model performance comparisons. These issues fall into several categories that help explain performance patterns observed in our study.
D.1.1
STS22- RUSSIAN
STS22-RUSSIAN
The Russian subset "STS22" dataset shows patterns that may help explain the comparatively low human agreement observed.
19

---

Model
Core17
News21
Robust04
Wikipedia
all-MiniLM-L6-v2
95.6
98.8
96.3
77.8
all-mpnet-base-v2
98.6
98.6
97.8
79.2
c5-mistral-7--instruut
98.8
99.5
98.8
88.4
embedddinggemma-300m
84.2
91.4
89.8
76.0
gte-Qwen2-1.BBiinstuut
97.5
99.2
98.5
86.1
casper_en_ie [esper_en__vsion___
wwwsiinnlagguaee.yy _l
98.2
100.0
98.7
88.8
multilinguullee--asse
96.2
98.6
96.9
88.5
multilingual-e5-large
95.7
97.8
97.2
92.6
multilingual-c--smalll
95.6
98.1
97.5
87.6
mxbai-embedd-arge--
97.2
98.0
98.6
85.6
Owen3-Emmeddding-0.66
97.0
100.0
98.5
86.8
Mirr
97.9
99.7
98.8
87.9
stella_en_ _i_____________
98.6
100.0
98.3
89.2
Human
85.2
92.7
88.5
87.9
Table 6: Full Reranking results.
Model
SICK-R
STS12
STS22
STSB
all-MiniLM--L6-v2
91.5
85.7
48.4
81.8
all-mpnet-base-v2
89.8
83.7
54.3
78.4
e5-mistral-77-instruut
93.2
89.1
58.5
85.9
embeddinggeemma-300m
72.6
77.9
57.9
58.3
gte-OWwen-11.B-innsttu
93.4
86.9
60.3
80.0
seaper_cen_viionnllaaeeee v1
93.8
92.0
67.2
88.7
multilingual-e5-basee
91.5
86.5
63.3
82.9
multilingual-eS-larg
89.4
89.9
65.9
83.6
multilingualee--small
88.6
87.6
63.3
81.9
mxbai-embed-large-vl
93.4
91.1
53.9
88.9
Qwen3-Embeddiing-0.6
93.3
91.8
63.6
90.9
SFR-Embediing-Miss
94.1
89.2
59.3
86.4
kella_en_1.5B__1
1atia_ee____5 1__B_v_3
92.3
89.1
64.3
87.1
Human
82.6
91.2
68.2
80.4
Table 7: Full STS results.
Context Expansion Issues:
Sentence pairs labeled as "4" (identical meaning) where sentence contains basic infor- mation and the paired sentence includes additional backstory or context
Translated example pattern: "Company reports earnings" vs. "Company reports earnings of SX million, exceeding expectations due to strong performance in sector Y*
Human annotators correctly identify these as semantically different (similarity 2-3), while gold labels mark them identical
20

---

This explains the low human performance (69.5%)
(58.5%) compared to models
Parsing and Processing Errors:
Incomplete sentence parsing affecting semantic interpretation
Parsing artifacts from web pages (e.g., page numbers, lists of automatically generated related news, ads)
D.1.2 MULTILINGUAL SENTIMENT CLASSIFICATION-RRSSIAA
The Russian subset of MMlltinngulSnniimenllssssccciii consists of news articles from different news sites. The task is to classify each text as "positive" or "negative". However, the dataset presents several challenges:
Neutral and Ambiguous Content:
Many samples are based on press releases from companies or government departments, which often neutral in tone and difficult to categorize as positive negative.
Translated example: "The total amount of pension savings accumulated in JSC "Unified Accumulative Pension Fund' (UAPF) as of September 1, 2016, amounted to about 6.41 trillion tenge, the press center of the pension fund said, KazTAG reports.
Such sentences more informative than sentiment-bearing.
Parsing and Processing Errors:
Similar to the issues described in $ D.I.I the dataset contains parsing artifacts from web pages.
D.1.3 EMOTION CLASSIFICATION
The emotion classification dataset suffers from inherent label ambiguity that explains the low human agreement (k = 0.39):
Mixed Emotional States:
Texts expressing multiple emotions simultaneously: "i am feeling very indecisive and spontaneous" (labeled as fear but could be surprise)
"i was feeling pretty anxious all day but my first day at work was very good day and that helped a lot" (contains both fear and joy)
"i am feeling crampy and cranky" (physical discomfort mixed with anger)
Sarcastic and Ironic Expressions:
"i got paid much because get many deliveries at work im feeling bit shamed so will curb the spending for bit" (sarcasm about being "overpaid")
"i feel like such a noob when the customers make really dull and stupid jokes that im supposed to find funny" (surface sadness but underlying anger/frustration)
"i feel very cheated since am supporting the family and doing all the other stuff while he spends hours day gaming" (labeled as joy but clearly expressing anger)
Contextual and Subjective Interpretation:
ii feel shame strange way" (ambiguous emotional context, labeled as surprise)
ii feel all funny sometimes" (vague emotional description that could be multiple categories)
"i feel underappreciated and under valued" (could be sadness, anger, or fear depending on interpretation)
21

---

D.1.4 ARXIV CLUSTERING CHALLENGES
Academic paper clustering presents fundamental categorization difficulties that explain the complete breakdown of human agreement (ARI = -0.001). This task uses derived labels from ArXiv paper categories:
Interdisciplinary Research Papers:
"Self-Superviset Audio- Visual Representation Learning with Relaxed Cross-Modal Syn- chronicity" (could cluster with computer vision, audio processing, or self-supervised learn-
ing)
"The architecture of innovation: Tracking face-to-face interactions with ubicomp technolo- gies" (spans social science, computer science, and architecture)
"PIINET: A 360-degree Panoramic Image Inpainting Network Using a Cube Map" (computer vision, graphics, or deep learning focus)
Methodological vs. Application Domain Confusion:
"Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approxima- tion" (reinforcement learning methodology vs. control theory application)
"Learning-Bane Basee Adaptive IRS Control with Limited Feedback Codebooks" (machine learning method vs. wireless communications application)
preeservne-reservingg numerical methods for stochastic Poisson systems" (numerical meth- ods vs. mathematical physics)
Emerging and Cross-Domain Research:
"The modularity of action and perception revisited using control theory and active inference" (cognitive science, control theory, or neuroscience)
"Food-chain competition influences gene's size" (evolutionary biology, computational biol- ogy, or mathematical modeling)
"Wavelet Analysis of Dengue Incidence and its Correlation with Weather and Vegetation Variables in Costa Rica" (epidemiology, signal processing, or environmental science)
D.2 IMPACT ON EVALUATION
These quality issues have several critical implications for embedding evaluation:
1. Artificial Model Advantages: Models may achieve "superhuman" performance by consis- tently reproducing systematic labeling patterns rather than demonstrating superior semantic understanding. This is particularly evident in tasks with low human agreement where models can exploit consistent but incorrect labeling patterns.
2. Misleading Benchmarks: Tasks with fundamental quality issues provide unreliable targets for model development. High model performance on such tasks may not indicate genuine capability improvements but rather successful pattern matching flawed gold standards.
3. Cultural and Linguistic Bias: Quality issues disproportionately affect non-English tasks, potentially masking genuine model limitations in cross-cultural understanding while artifi- cially inflating performance on problematic English datasets.
4. Evaluation Validity: The validity of using these datasets as benchmarks is questionable when human experts cannot agree on correct answers, suggesting fundamental issues with task specification rather than human limitations.
E STATISTICAL ROBUSTNESS ANALYSIS
E.1 CONFIDENCE INTERVAL METHODOLOGY
Given sample size constraints (N = 20 50), we computed 95% Confidence Intervals (CIs) for human performance using metric-specific analytical methods rather than generic approximations:
22

---

Classification (Aceuracy): Wilson Score Interval (Wilson 1927). which is robust for binomial proportions with small sample sizes and avoids the "zero-width" errors of normal approximations.
STS (Correlation): Fisher z-transformation Fiihee 1915) to compute CIs for Spearman correlations, ensuring valid bounds within [-1.1].
Clustering & Reranking: Empirical range between annotators conservative bound
given Namoottoors =
Statistical significance (*) determined by non-parametric overlap test: a model is considered significantly different if its score falls outside the human 95% CI (corresponding to < 0.05).
E.2 DETAILED RESULTS WITH CONFIDENCE INTERVALS
Table 8 presents human performance with 95% confidence intervals for all 26 lask-language pairs. Models perform outside the human confidence interval in 14 of 26 tasks, indicating statistically significant differences. Crucially, this separation often occurs tasks with low to moderate inter- annotator agreement. For example, in EmnotionClassiicttion. the best model (75.4) significantly exceeds human performance (45,8, CI:[32.6, 59.7J). but the low agreement (x 0.39) suggests this "superhuman" performance may reflect fitting annotation artifacts rather than genuine semantic superiority. Conversely, in MultilingualSentinen (Arabic) and STS22 (Arabic), humans significantly outperform models, highlighting genuine cultural gaps that current models fail bridge.
Task
Lang
N
K
Tot
Human (95% CT)
TAA
Best Model
Classification
EmotionClassificaton eng
45.8 [32.6, 59.7]
R.
75.4*
MultilnggullSentiment
ara
40
40
95.0 [83.5, 98.6]
N/A
77.5
MulilingualSentiment
eng
40
2
80
77.5 [62.5, 87.7]
K. = 0.24
95.5*
Multilingullenttmment
nob
40
40
85.0 [70.9, 92.91
N/A N/A
75.0
MultilingulSenniment
40
40
92.5 180.1, 97.41
81.3
Toverconvessations
eng
45
90
73.3 [59.0, 84.0]
K
86.7"
TweesSentimemEEExtaaiiooo
eng
45
2
90
84.4 [71.2, 92.3]
R 0.41
90.9
Clustering
ArxivClustering
eng
30
2
60
49.2 (35.3, 63.2]
ARI=-0.00
84.6*
ReddiiCllustering222P
eng
30
60
68.8 [63.2, 74.4]
ARI=0.42
100.0*
SIB:00CClsserrnngS25
30
30
76.0 [58.4, 87.8]
N/A
78.8
S1B2200ClusteringS22
dan
30
62.7 [44.9, 77.6]
N/A
76.0
SIBB20CCluteringSS2
eng
30
2
60
54.0 [41.8, 66.3]
ARI=0.15
83.3"
S1B200CCluseringgS25
rus
30
1
30
68.1 [50.2, 81.9]
N/A
77.7
WikiCitiesClustering
eng
30
60
97.6 [95.2, 100.0]
ARI=0.91
100.0
Reranking
Core IInstrunsto
eng
20
40
85.2 (83.6, 86.8]
P 0.80
98.8"
News2 Instruction
eng
31
92.7 (91.3, 94.1]
P =
100.0"
Robust04Innttuutionn
eng
49
2
98
88.5 [82.2, 94.8]
P = 0.75
98.8"
WikipediaMullilnggaal
dan
30
1
30
91.4 [76.2, 97.3]
N/A
95.0
WikipediaMuuttlinguall
eng
30
2
60
82.4 (75.6, 89.1
P 0.64 N/A
90.6*
WikipediaMuultilnguual
nob
30
89.8 174.1, 96.4]
92.3
STS
SICK-R
eng
82.7 [69.4, 90.5]
P = 0.63
94.1*
STS12
eng
50
2
100
91.2 [84.9, 94.9]
= 0.77
92.0
STS22
ara
30
1
30
67.6 [41.7, 83.3]
N/A
40.9"
STS22
eng
30
2
60
78.4 [59.0, 89.2]
P 0.75
82.9
STS22
rus
30
58.7 [28.8, 78.2]
N/
69.5
STSBenchmark
eng
100
80.4 [67.7, 88.4]
P = 0.58
90.9*
Table 8: Human performance with 95% confidence intervals. N number of samples; K number of annotators: Tot total annotations (NxK). CIs computed via Wilson Score Interval (Classification), Fisher's s z-transformation (STS), and Annotator Range (Cluserrig//Rrankinng)) indicates model score outside human 95% CI (p 0.05). IAA Inter-Annotator Agreement.
23

---

F INTER-ANNOTATOR AGREEMENT ANALYSIS
F.I AGREEMENT METRICS BY TASK CATEGORY
This section provides detailed inter-annotator agreement analysis using task-appropriate metrics. Agreement levels follow standard guidelines: к 0.8 (excellent), 0.6 < I 0.8 (substantial), 0.4 0.6 (moderate), 0.2 <, K S 0.4 (fair), S 0.2 (poor). For correlations: p 0.7 (strong), 0.4 < < 0.7 (moderate), - 0.4 (weak).
F.1.1 CLASSIFICATION TASKS
Emotion Classification: x = 0.39 (fair agreement)
annotators, items, 96 total annotations
Mean percentage agreement: 52.1%
Performance: Human 45.8%, Best model 87.1%
Toxicity Classification: y. 0.55 (moderate agreement)
2 annotators, 45 items, 90 total annotations
Mean percentage agreement: 77.8%
Performance: Human 73.3%, Best model 86.7%
Tweet Sentiment Classification: バ = 0.41 (moderate agreement)
annotators, items, 90 total annotations
Mean percentage agreement: 62.2%
Performance: Human 84.4%, Best model 90.9%
Multilingual Sentiment Classification: Agreement only for English
English: バ 0.24 (fair agreement), 2 annotators, 40 items, 62.5% agreement
Arabic, Norwegian, Russian: Single annotator (no agreement metrics)
Performance: Human advantages in non-English variants
F.1.2 CLUSTERING TASKS
ArXiv Clustering: ARI -0.001 (no agreement)
2 annotators, 30 items, 60 total annotations
Complete breakdown of consensus academic paper categories
Performance: Human 49.2%, Best model 84.6%
Reddit Clustering: ARI 0.42 (moderate agreement)
2 annotators, 30 items, 60 total annotations
Moderate consensus on discussion topic groupings
Performance: Human 68.8%, Best model 100%
WikiCities Clustering: ART 0.91 (excellent agreement)
2 annotators, items, 60 total annotations
High consensus geographical entity groupings
Performance: Human 97.6%, Best model 100%
STB200 Clustering: Agreement only for English
English: ARI = 0.15 (weak agreement), 2 annotators, 30 items
Arabic, Danish, Russian: Single annotator (no agreement metrics)
Performance varies significantly across languages
24

---

F.1.3 RERANKING TASKS
News21: p 0.85 (strong agreement)
annotators, items, 62 total annotations
Mean Kendall tau: 0.85, Binary kappa: 0.83
Performance: Human 92.7%, Best model 100% Core17: p 0.80 (strong agreement)
2 annotators, 20 items, 40 total annotations
Mean Kendall tau: 0.80, Binary kappa: 0.78
Performance: Human 85.2%, Best model 98.8%
Robust04: p 0.75 (strong agreement)
2 annotators, 49 items, 98 total annotations
Mean Kendall tau: 0.75, Binary kappa: 0.72
Performance: Human 88.5%, Best model 98.8%
Wikipedia Multilingual Reranking: Agreement only for English
English: P 0.64 (moderate agreement), annotators, 30 items
Mean Kendall tau: 0.64, Binary kappa: 0.60
Danish, Norwegian: Single annotator (no agreement metries)
Performance varies across languages
F.1.4
STS TASKS
STS12: p = 0.77 (strong agreement)
annotators, 50 items, 100 total annotations
Performance: Human 91.2%, Best model 92.0%
STSBenchmark: p == 0.58 (moderate agreement)
2 annotators, 50 items, 100 total annotations Performance: Human 80.4% Best model 90.9%
SICK-R: p = 0.63 (moderate agreement)
2 annotators, 40 items, 80 total annotations
Performance: Human 82.6%, Best model 94.1%
STS22: Agreement only for English
English: p = 0.75 (strong agreement), annotators, 30 items
Arabic, Russian: Single annotator (no agreement metrics)
Performance varies significantly by language
F.2
AGREEMENT PATTERNS AND TASK RELIABILITY
F.2.1
HIGII-AGREEMENT TASKS (RELIABLE BENCHMARKS)
Tasks with high human agreement (ı > 0.6 or 0.7) consistently demonstrate:
Clear, objective task specifications with minimal ambiguity
Adequate context for making informed judgments
Cultural and linguistic familiarity for annotators
Well-defined evaluation criteria with concrete examples
Minimal dataset quality issues or processing artifacts
Consistent performance patterns across annotators
Examples: WikiCities clustering, News22//Cone rerakknnnnntOrrrrkiinnn STS12, STSBenchmark
25

---

F.2.2 LOW-AGREEMENT TASKS (PROBLEMATIC BENCHMARKS)
Tasks with low agreement (k < 0.4 or < 0.6) often exhibit:
Ambiguous annotation guidelines subjective judgment requirements
Cross-cultural interpretation challenges
Insufficient context for accurate assessment
Systematic dataset quality issues or processing artifacts
Inherently subjective or multi-faceted concepts
Inconsistent contradictory gold standard labels
Examples: Emotion classification, ArXiv clustering, STS22-Russian
G ADDITIONAL HUMAN VS MODEL ANALYSIS
160
100
Best
Best
86%
71%
71%
67%
Win rate (%5
Win rate 1%5
43%
33%
33%
29%
20
20
Clessifiation
Guattring
Herenting
STS
English
Multfinuul
100
100
100%
100%
100%
100%
Best
Mein
Win rate (5)
62%
Win rate (56)
60
50%
33%
15%
Human win rates across task categories and languages. Top left: By task category shows humans perform moderately classification but struggle in clustering, reranking, and STS against best models. Top right: English-only vs multilingual tasks reveals humans perform better on multilingual tasks (29% vs 0% against best models). Bottom left: Performance varies dramatically by baseline comparison (15% best. 62% vs mean models). Bottom right: Language-speciff breakdown shows varying performance across different language codes.
Figure 12:
This section contains additional analysis on human vs model. Figure 14 shows the human performance
gaps versus mediann-performin models over all tasks by language. Figure 15 shows the task difficulty categorization based on human performance levels. Figure shows the model consistency analysis showing performance ranges across tasks. |Figure 2) shows comprehensive view human performance relative to all model performance ranges across 16 tasks by language.
26

---

(era)
+26.7
Classfication- Multilnnual Sentiment (aral
+17.5
Clasiiication Mulidingul Sentiment (rus]
Classifiationn Muttiingual Sentment (nob]
+10.0
Clustering- WidCities
2.4
Reranking- (nob]
Gustering SI2000 (ara)
2..8
Reranking- Idan)
STS STS-5 lengi
4.5
6.4
Beranking News21
7.3
Reranking lengi
8.2
Clustering SI8200 trus]
9.7
Robusin4
STS- 5T5BendFmark
-0066
STS
11.1
STS SIOKR
-1.55
SIB200
13.3
ClassFication Texicity
Reranking- Corel7
13.5
Classification Mindin Sentiment lengi
Clustering- SI8200
.29.3
Elassification Emotion
Clustering Reddit
-31.2
Clusterigg - ArXi
Gap vs Best model (percentage points)
Figure 13: Human performance gaps versus best cess-performin models across 26 task languaag pairs. Humans outperform the best models only 4 tasks (15.4%) with largest advantages in Arabic semantic similarity and sentiment analysis. The analysis reveals systematic model advantages in technical domains (clustering, reranking) versus human advantages in culturally-ifformed tasks.
H
MODELS EVALUATED
Table 9 shows information about each evaluated model.
Parameters (Millions)
Model
Li et al.
(2023)
1780
goggleembeddingeenn--0000
Vera et 2025
300
intfooat/e--mistall-bbiinss
Wang et al. (2023
2022
7111
inthooa//molddinuall---lgggg Wang et
2022
560
intfooa//mullillggualle---ssssss Wang et al.
2022)
278
introas/mmlldlingalle..smallllll Wang al.
(2022)
118
Lee et al.
2004
Li & Li
2023
335
NovaSenchhissperr.nnnviiiiooo http_/vviiioolaanaaeeee.o
ZZang el al. (2025a)
1999
Oveen(Queen--Embbeddiiggg-11
/Zaan al. (2025))
596
JalesforeeSSFRR-Embcdddngg (Meng et al. [0224)
7110
sentence-trance
Reimers & Gurevych (0019)
22.7
sentence-transfoon suneccctttanffommmerall-pppeeeee Reimers & Gurevych (2019)
109
Table List of all evaluated models. Model sizes in millions of parameters
27

---

Clsttring SI8200 (ara)
+40.1
+38.7
Clssdfication Multlingul Sentiment (ara)
+22.7
Clusteing WikiChies
+22.7
Clustering -IB2200 [rus]
+21.4
Clastering
+20.0
Classifiatiion Muklingull Sentiment (rus]
+18.0
Classifiationn Mutiingual Inob]
+16.2
Classficatioon Tweee Senti
+14.9
SI8200 leng!
+10.5
ClassPication Texicity
Reranking WikiMutii (nobl
+3.7
STS ST512
Classification Emotion
+1.9
Remanknng WklMulti [dan]
STS STS22 jeng)
+0.5
m75Benchmark
STS STS22 [rus]
-3.6
Reraniing
leng!
Clustering Readit
-6.1
Reranking
STS SICKR
-966
Classifcattinn Muldingual Sentiment lengl
-11.2
Beranking
Clustering - ArXi
19.8
-20
-10
10
20
40
Gap vs Median model (percentage points)
Figure 14: Human performance gaps versus median -ertiirn-proooinggggg models across 26 tasks by language. Humans achieve 61.5% win rate against median models, demonstrating competitive performance when compared to typical rather than best- models. This analysis reveals that human performance is much more competitive when compared against representative model performance rather than cherry-picked best results.
28

---

WikiGities
97.6
Multilingual Sentiment (ara)
95.0
News21
92.7
Multilingual Sentiment (rus)
WikiMutti (dan)
91.4
STS12
91.-2
89.8
RobustD4
88.5
Multilingual Sentiment (nob)
85.0
Tweet Senti
84.4
82.6
WikiMulti (eng)
82.4
ST5Benchmark
STS22 (eng)
78.4
Multilingual Sentiment (eng)
77.5
SIB200 (ara)
76.0
Toxidity
73.3
SIB200 (rus)
68.1
STS22 (ara)
SI8200 (clan)
62.7
STS22 (nus)
58.5
(eng)
54.0
ArXiv
49.2
Emotion
Human performance (%)
Figure 15: Task difficulty categorization based on human performance levels. The majority of tasks (69%) fall into the "easy" category (human performance 0.7), shown in green. Only two tasks fall below 0.5 (shown in red), both with notably low inter-annotator agreement, suggesting fundamental task ambiguity rather than limitations of human ability.
29

---

News21
7.3
Rebust04
STS22 (eng)
12.1
WikiMuti (eng)
12.5
14.1
Corel7
14.5
WikiMulti (dan)
WikiMuti (nob)
21.1
SICK-R
21.5
Multilingual (ara)
24.0
Multilingual Sentiment (rus)
25.7
Texicity
Multilingual Sentiment (nob)
28.0
Tweet Senti
32.4
ST5Benchmark
32.6
WikiCities
33.8
(ara)
Multilingual Sentiment (eng)
35.7
STS22 (rus)
Emotion
41.7
Reddit
44.6
(rus)
SI8200 (dan)
53.1
ArXiv
54.1
SIB200 (eng)
59.1
SIB200 (ara)
59.6
Model performance range (pp)
Figure 16: Model consistency analysis showing performance ranges across tasks. Higher value indicates greater variability across models (lower consistency). Tasks with small ranges (high con- sistency) often align with high human agreement, whereas tasks with large ranges (low consistency) typically correspond tasks where humans also struggle. This pattern suggests that both human and model performance reflect underlying task quality and clarity of task specification.
30

---

I DETAILED -----SS--NNTTTOR RESULTS
This section provides detailed LLMM-as-nnottato performance for each task-language pair. LLMs were not evaluated on clustering tasks due the difficulty of eliciting cluster assignments from generative models.
GPT-5
GPT-4.1
Gemini 2.5 Flash
Mistral Small -4B-1
Qwen3
Dataset
Human
Full
Mini
Full
Mini
30B
32B
Coder
Emotion [eng]
45,8
37.5
33.3
41.7
29.2
35.4
43.1
37.5
38.3
42.1
Multilingual Sentiment [ara]
95.0
92.5
95.0
90.0
90.0
92.5
90.5
92.8
85.5
91.2
Multilingual Sentiment [eng]
77.5
100.0
100.0
97.5
95.0
97.2
99.0
93.0
96.5
Multilingual Sentiment [nob]
85.0
92.5
85.0
87.5
82.5
85.0
86.8
86.8
77.2
88.5
Multilingual Sentiment [rus]
92.5
87.5
85.0
75.0
85.0
92.5
64.0
74.2
75.8
78.0
Toxic Conversations [engl
73.3
73.3
66.7
73.3
75.6
71.1
69.6
60.0
68.2
66.9
Tweet Sentiment [eng]
84.4
68.9
75.6
71.1
75.6
68.9
65.6
68.9
72.9
70.7
Average
79.1
78.9
77.2
76.6 76.1
77.6
73.8
74.2
73.0
76.3
Table 10: Detailed L.MM-as-amnotator results for classification tasks. Bold indicates best performance
per row.
GPT-5
GPT-4.1
Gemini
Mistral Small-24B-1
Qwen3
Dataset
Human
Full
Mini
Full
Mini
2.5 Flash
30B
32B
Coder
Core17 [engl
85.2
76.6
83.2
72.7
78.0
76.8
74.4
89.8
74.2
72.2
News21 [eng]
92.7
72.9
78.7
75.1
78.2
75.0
77.4
80.5
77.9
75.2
Robust04 [eng]
88.5
75.7
84.0
79.5
79.3
79.7
77.2
84.6
79.2
73.6
Wikipedia [dan]
91.4
84.7
78.3
85.8
83.6
82.8
84.0
76.6
79.8
79.9
Wikipedia [eng]
69.6
61.0
70.8
68.6
74.2
75.1
59.8
73.4
69.5
Wikipedia [nob]
89.8
71.5
68.1
70.6
75.8
68.6
79.6
62.6
64.2
72.3
Average
88.3
75.1
75.5
75.7
77.2
76.2
78.0
75.6
74.8
73.8
Table 11: Detailed
results for reranking tasks. Bold indicates best performance
per row.
GPT-5 Mini
GPT-4.1
Gemini 2.5 Flash
Mistral Small-24B-1
Qwen3
Dataset
Human
Full
Full
Mini
30B
32B Coder
SICK-R R [eng]
82.6
68.5
67.5
66.9
72.9
59.0
66.7
59.4
57.3
68.8
STS12 [engl
91.2
83.2
82.7
87.3
87.8
83.8
83.9
83.8
81.7
84.5
STS22 [ara]
67.5
57.1
45.5
56.9
55.8
43.1
56.9
35.3 50.4
47.9
STS22 [eng]
78.4
67.8
80.3
81.6
79.7
81.3
78.2
74.1
76.2
76.4
STS22 [rus]
58.7
71.9
51.3
67.2
66.6
64.9
78.6
65.9
62.4
67.0
STSBenchmark [eng]
80.4
89.3
86.7
89.7
86.3
83.8
85.4
84.3
84.0
82.9
Average
76.5
73.0
69.0
74.9
74.9
69.3
75.0
67.1 68.6 71.3
Table 12: Detailed LLM--s-annotator results for sts tasks. Bold indicates best performance per row.
1.1 HUMAN-LLM TASK DIFFICULTY CORRELATION
To assess whether humans LLMs face similar challenges, we computed Spearman rank correla- tions between human and LLM performance across the 19 lank-lan pairs (excluding clustering). positive correlation indicates that tasks where humans perform well also tend to be tasks where L.MMs perform well.
The overall correlation is moderate and statistically significant (p = 0.52, p = 0.023), suggesting partially shared difficulty patterns.
31

---

LLM Model
Spearman p p-value
GPT-5 Full
0.47
0.042
GPT-5 Mini
0.52
0.023
GPT-4.1 Full
0.43
0.066
GPT-4.1 Mini
0.57
0.012
Gemini 2.5 Flash
0.50
0.029
Mistral Small
0.30
0.215
Qwen3- 30B
0.50
0,031
Qwen3-32B
0.52
0,022
Qwen3-Coder
0.57
0,011
Average (all LLMs)
0.52
0.023
Table 13: Spearman rank correlation between human and LLM performance across task-language pairs (clustering excluded), A positive p indicates that tasks where humans score high also tend to be tasks where LLMs score high. Values with p < 0.05 indicate statistically significant correlations. The moderate positive correlations suggest partially shared task difficulty patterns between humans and L.L.s.
J LLM USAGE STATEMENT
Large language models were used to assist with formatting, citation integration, and writing polish during the preparation of this manuscript. Specifically, we used L.LMs for:
Formatting assistance for LaTeX tables and mathematical notation
Integration and standardization of citation formats
Minor writing improvements for clarity and flow
Code documentation and data processing script organization
All substantive content, including research design, data analysis, interpretation of results, and scientific conclusions, was developed entirely by the authors. The core contributions, methodology, and findings presented in this work are the original intellectual contribution of the research team. LLM assistance was limited to technical formatting and presentation improvements that did not influence the scientific content conclusions of the study.
32


Reviews:
[{"review_id": "qokMkHzIrI", "reviewer": "Reviewer_FCbW", "summary": "The paper introduces HUMA, a human performance evaluation on the MTEB benchmark. Results show that humans achieve an average score of 77.6%, compared to 80.1% for the best embedding model. Further analysis reveals that text embedding models underperform humans on low-resource languages, and that some datasets exhibit low quality or label ambiguity.", "questions": "(1) In L99-102, the paper states that \"Human evaluation is well established in NLP, especially for generative tasks like machine translation, summarization, and dialogue.  In contrast, embedding-based tasks have relied almost exclusively on automated metrics, with little attention to human baselines.\". This motivation is not convincing to me. For these generation tasks, automatic evaluation metrics are often unreliable, which requires human evaluation. However, for embedding-based tasks that have well-defined ground truth labels, human evaluation may not be that necessary.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The paper presents a comprehensive human performance evaluation of MTEB, covering reranking, classification, clustering, and STS datasets.\n\n(2) Based on the human performance results and inter-annotator agreement analysis, the authors identify several problematic datasets containing labeling ambiguities (e.g., the emotion classification dataset), which may lead to unrealistic model evaluations..", "weaknesses": "(1) The motivation of conducting a comprehensive human performance evaluation of MTEB is not very convincing to me. The two possible reasons I can get are (1) estimating an uppper bound of dataset performance, similar to how human performance on GLUE was previously used as a target to achieve. But this is not the case here since the paper shows that current models already surpass human performance; and (2) as the paper points out, identifying problematic datasets where humans perform poorly or where inter-annotator agreement is low. However, this second motivation seems somewhat weak to me.\n\n(2) The retrieval task, which is widely used in real-world applications, is not included in the human performance evaluation.\n\n(3) Most tasks includes only two annotations, and for each task there are only 20-50 instances annotated. This raise concerns on the reliablity of the evaluation.\n\n(4) Comparing the performance of native low-resource language speakers with text embedding models primarily trained on English data may not be a fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EOolwF2ByZ", "reviewer": "Reviewer_RywH", "summary": "This paper introduces HUME, a framework for measuring human-level performance on text embedding benchmarks. They compared human performance with 13 embedding models on 4 major categories of tasks (classification, clustering, semantic textual similarity, and reranking) in MTEB and analyzed the consistency of human annotators as the entry point, putting forward suggestions for the development of future evaluation benchmarks.\nThe main contributions include: (1) Proposing a framework for evaluating human embedding capabilities, which is currently relatively lacking; (2) The assumption that tasks with low human consistency may reflect the ambiguity of data or definitions rather than the true \"superhuman\" model capabilities offers a new guidance for the future development of benchmarks.", "questions": "How do you ensure that the low agreement among your annotators truly reflects data ambiguity, rather than annotation fatigue or lack of clear instructions? It is possible to supplement the analysis of model bad cases and whether the model as a whole leans towards certain labels.\n\nCould you share a concrete formula for your proposed “agreement-weighted evaluation”? It is suggested that several feasible agreement-based weighted evaluation methods be proposed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Supper cool idea!\n1. The paper is original in reframing benchmark evaluation: a human evaluation framework that quantifies human-level performance across text-embedding tasks and relates it to model results.\n2. The study is of good methodological quality, covering sixteen datasets and four task categories with transparent protocols and clear reporting.\n3. The paper’s significance lies in revealing that many “superhuman” model claims arise in tasks with low human agreement, highlighting the need for benchmark reform and more interpretable evaluation standards.", "weaknesses": "The small and homogeneous annotator pool (all male, limited in number, with single annotators for some languages) limits generalizability and the evaluation sets are small and lack confidence interval reporting, reducing statistical robustness.\n\nThere is a lack of a more direct and powerful analysis of the performance attribution of the superhuman model (directly attributing low human consistency to data/task design and quality issues is not rigorous enough). \n\nThe retrieval is the most widely used embeddinng application, which is not included in this work. I'm very curious and excited about how well this framework could guide the retriver evaluation.\n\nAdditionally, the proposed “agreement-weighted evaluation” idea is conceptually interesting but underdeveloped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VGmHYbDsgE", "reviewer": "Reviewer_zWFZ", "summary": "The paper introduces HUME, a framework for measuring human performance on text embedding tasks and comparing it directly to state-of-the-art embedding models. It evaluates human annotators across 16 MTEB datasets spanning classification, clustering, reranking, and semantic textual similarity in multiple languages. Results show that human performance is competitive but not dominant, often ranking around the upper-middle of model performance, with notable advantages in certain multilingual tasks. The authors analyze task difficulty, dataset quality, and human agreement, highlighting that low human performance often reflects dataset ambiguity rather than human limitations. The paper concludes with implications for benchmark design, cultural and linguistic gaps in current models, and recommendations for more reliable evaluation practices.", "questions": "1. (W1) Could the authors provide justification for using such a small number of annotators and annotated samples? Do they have any evidence or pilot studies suggesting that this limited pool and sample size are sufficient to produce stable and representative estimates of “human” performance?\n\n2. (W2) Can the authors provide a more detailed analysis of the underlying factors contributing to the observed human–model performance differences, such as training data coverage, domain bias, or cultural and linguistic variability, especially in multilingual tasks?\n\n3. (W3) Would it be feasible to evaluate LLMs by instructing them to follow the same annotation protocol as human annotators? If so, how might such an experiment serve as a complementary or cost-efficient reference point in future work?\n\n4. (W4) Why were retrieval tasks excluded from the study, and how might including them—given their centrality to text embedding applications—affect the overall findings and implications?\n\n5. (W5) How closely were the human labeling instructions matched to the original dataset annotation schemes, and could any inconsistencies between the two affect the validity of the human–model comparisons?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a timely and important problem by grounding text embedding evaluation in human performance. The motivation is well articulated, and the direction is practical and relevant for improving how benchmark results are interpreted.\n\n2. The experimental design is generally solid, with clear task selection and consistent evaluation protocols. While some aspects could be expanded (see weaknesses), it establishes a strong foundation for systematic human–model comparison across languages and task types.\n\n3. The discussion and implications are well developed, offering clear insights into dataset quality, evaluation reliability, and multilingual gaps. The proposed future suggestions are concrete and meaningful, adding to the significance and forward-looking value of the work.", "weaknesses": "1. The study’s reliance on only two annotators for most tasks (and, in some multilingual settings, just one) severely limits the robustness and representativeness of the claimed “human” performance. With such a small pool, the results risk reflecting individual annotator biases rather than general human ability, making the observed human–model gap less reliable as an empirical reference point. In addition, the number of annotated examples per dataset—ranging only from 30 to 50 items—is far too small relative to the scale of the original benchmarks. This raises concerns about statistical power and generalizability: subtle effects may be missed, while task-specific variability could be overstated or underrepresented in the final conclusions.\n\n2. Although the paper identifies several dataset quality issues, it does not sufficiently analyze their underlying causes from the perspective of model training. In particular, there is no supporting evidence or exploration of factors such as training data distributions, domain coverage, or cultural mismatches that might explain the observed human–model performance differences. This lack of analytical depth weakens the explanatory power of the findings, especially in multilingual settings where humans outperform models in some tasks but not others.\n\n3. The paper directly compares human annotators and embedding models but lacks a key experiment: whether large language models can follow the same annotation protocol. Evaluating LLMs under the same instructions would not only provide a useful reference baseline but also offer practical value by reducing the cost of future human evaluations. This experiment could significantly strengthen the paper’s claims regarding the human–model gap.\n\n4. While the paper covers classification, clustering, STS, and reranking tasks, it notably omits retrieval tasks, which are arguably the most central and widely used application of text embeddings in real-world systems. Excluding retrieval significantly limits the practical relevance and completeness of the study’s conclusions.\n\n5. The paper provides insufficient clarity on whether the human labeling instructions were fully aligned with the original dataset annotation schemes. Any divergence in labeling guidelines could introduce inconsistencies and confound the comparison between human annotators and model performance, raising questions about the validity of the measured human–model gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UpFkTUjF8x", "reviewer": "Reviewer_N1ED", "summary": "This paper introduces HUME, a framework for measuring human performance on text embedding tasks from MTEB. The authors evaluate human annotators across 16 datasets spanning four task categories: reranking, classification, clustering, and semantic textual similarity (STS).", "questions": "**Unclear Motivation for Human-Model Comparison**\n\nI'm struggling to understand the fundamental premise of this paper. Why should we compare embedding models to human performance at all?\n\nThis isn't like evaluating generative models, where we compare LLMs to humans because we want human-level intelligence and knowledge. Embedding models serve a completely different purpose. We never use humans to generate embeddings or perform large-scale semantic search in practice. The goal is optimizing downstream task performance, not replicating human judgment. So what does human performance actually tell us about embedding quality?\n\nThe paper repeatedly interprets superior model performance as suspicious, suggesting models are \"exploiting patterns rather than achieving true semantic understanding.\" But why can't models just be legitimately better at these tasks? Machine learning is designed to find patterns beyond human perception. When a model outperforms humans at emotion classification, maybe it's actually better at detecting emotional patterns consistently, rather than exploiting flawed data.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The selection of 16 diverse datasets across multiple languages, domains, and task types is well-motivated and thorough. The inclusion of both high-resource and low-resource languages adds an important cross-lingual perspective.", "weaknesses": "While the paper addresses an interesting question, I have concerns:\n1. **Necessity and Scope of the Study**\n\t- **Limited sample sizes**: With only 20-50 instances per task, can we really draw reliable conclusions about human performance on these benchmarks? \n\t- **Missing task categories**: The framework only covers 4 of MTEB's task types (reranking, classification, clustering, STS). What about retrieval, which is a very important task nowadays?\n2. **Methodological Concerns**\n\t- **Statistical significance**: The paper presents many performance comparisons but doesn't include significance testing. With small sample sizes, how confident can we be that observed differences are meaningful?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)