Title: LitSearch: A Retrieval Benchmark for Scientific Literature Search
Benchmark: LitSearch
Summary: iterature search questions often require deep understanding and reasoning across research articles, posing challenges for modern search engines. LitSearch, a new benchmark with 597 literature search queries about recent ML and NLP papers, is introduced to address these challenges. The benchmark, constructed from GPT-4 generated and manually written questions, reveals a significant performance gap between traditional retrieval models like BM25 and state-of-the-art dense retrievers, with LLM-based reranking further improving retrieval performance, highlighting the limitations of commercial search engines on these complex queries.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Scientific literature search
Phenomenon defined by authors: Yes
Phenomenon definition: Finding literature via a specific search query—for
example, collecting related work, checking if a
method has been proposed before, or recalling a
previously seen paper
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Answer literature search questions related to a large corpus of scientific papers
Task item definition: question, answer, score
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 597
Dataset metadata available: Yes
Dataset metadata detail: in-line citation questions, hand written questions, broad, specific
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: in-line citation questions, hand written questions, broad, specific
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: General Science / 
New taxonomy: Specific Application / Physical Sciences