Title: CRAG - Comprehensive RAG Benchmark
Benchmark: CRAG
Summary: Retrieval-Augmented Generation (RAG) aims to improve Large Language Models (LLMs) by supplementing them with external knowledge, but existing datasets fail to capture the diverse and dynamic nature of real-world Question Answering (QA) tasks. The Comprehensive RAG Benchmark (CRAG) is introduced to address this gap, featuring 4,409 question-answer pairs and mock APIs for web and Knowledge Graph search, covering a wide range of domains and question types. Evaluation on CRAG shows that adding RAG improves LLM accuracy but still falls short of trustworthy QA, particularly with questions involving high dynamism, low popularity, or complexity, indicating key areas for future research.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: RAG performance in real-world QA settings
Phenomenon defined by authors: Yes
Phenomenon definition: RAG in the wild is more difficult than in contrived settings and models often fail
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Answer questions from an array of questions across domains and question categories
Task item definition: Question, retrieval contents, domain, category, answer, score
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 4409
Dataset size notes: train: 1760, val: 1320
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval