HUME: Measuring the Human-Model Performance Gap in Text Embedding Task
Authors: name: Adnan El Assadi; name: Isaac Chung; name: Roman Solomatin; name: Niklas Muennighoff; name: Kenneth Enevoldsen
Published: 2025-10-11 06:56:53

Abstract: Comparing human and model performance offers a valuable perspective for
understanding the strengths and limitations of embedding models, highlighting
where they succeed and where they fail to capture meaning and nuance. However,
such comparisons are rarely made, as human performance on embedding tasks is
difficult to measure. To fill this gap, we introduce HUME: Human Evaluation
Framework for Text Embeddings. While frameworks like MTEB provide broad model
evaluation, they lack reliable estimates of human performance, limiting the
interpretability of model scores. We measure human performance across 16 MTEB
datasets spanning reranking, classification, clustering, and semantic textual
similarity across linguistically diverse high- and low-resource languages.
Humans achieve an average performance of 77.6% compared to 80.1% for the best
embedding model, although variation is substantial: models reach near-ceiling
performance on some datasets while struggling on others, suggesting dataset
issues and revealing shortcomings in low-resource languages. We provide human
performance baselines, insight into task difficulty patterns, and an extensible
evaluation framework that enables a more meaningful interpretation of the model
and informs the development of both models and benchmarks. Our code, dataset,
and leaderboard are publicly available at
https://github.com/embeddings-benchmark/mteb.