BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive
  Retrieval
Authors: name: Hongjin Su; name: Howard Yen; name: Mengzhou Xia; name: Weijia Shi; name: Niklas Muennighoff; name: Han-yu Wang; name: Haisu Liu; name: Quan Shi; name: Zachary S. Siegel; name: Michael Tang; name: Ruoxi Sun; name: Jinsung Yoon; name: Sercan O. Arik; name: Danqi Chen; name: Tao Yu
Published: 2024-07-16 17:58:27

Abstract: Existing retrieval benchmarks primarily consist of information-seeking
queries (e.g., aggregated questions from search engines) where keyword or
semantic-based retrieval is usually sufficient. However, many complex
real-world queries require in-depth reasoning to identify relevant documents
that go beyond surface form matching. For example, finding documentation for a
coding question requires understanding the logic and syntax of the functions
involved. To better benchmark retrieval on such challenging queries, we
introduce BRIGHT, the first text retrieval benchmark that requires intensive
reasoning to retrieve relevant documents. BRIGHT is constructed from the 1,398
real-world queries collected from diverse domains (such as economics,
psychology, robotics, software engineering, earth sciences, etc.), sourced from
naturally occurring or carefully curated human data. Extensive evaluation
reveals that even state-of-the-art retrieval models perform poorly on BRIGHT.
The leading model on the MTEB leaderboard [38 ], which achieves a score of 59.0
nDCG@10,2 produces a score of nDCG@10 of 18.0 on BRIGHT. We further demonstrate
that augmenting queries with Chain-of-Thought reasoning generated by large
language models (LLMs) improves performance by up to 12.2 points. Moreover,
BRIGHT is robust against data leakage during pretraining of the benchmarked
models as we validate by showing similar performance even when documents from
the benchmark are included in the training data. We believe that BRIGHT paves
the way for future research on retrieval systems in more realistic and
challenging settings. Our code and data are available at
https://brightbenchmark.github.io.