Title: RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
Benchmark: RAGTruth
Summary: This paper targets word-level hallucinations in various tasks and domains in the RAG setting. It presents approximately 18,000 responses generated using RAG from diverse LLMs which are annotated at the word level for hallucination intensity. Hallucination frequencies are benchmarked across various LLMs, and hallucination detection methods are assessed versus a small LLM fine-tuned using the proposed dataset, RAGTruth.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: hallucination detection, specifically for RAG applications
Phenomenon defined by authors: Yes
Phenomenon definition: "Hallucination in the context of LLMs usually refers to a situation where the
model generates content that is not based on factual or accurate information"
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: For a given reference-response pair, determine if it contains hallucinated content at the response level and span level.
Task item definition: A single item consists of source information (reference), an LLM-generated response (which may contain various degrees of hallucination), annotation of the location and type of hallucination (if any), and a brief annotated explanation of the hallucination observed.
Task definition detail: Additional meta-data regarding the model and inference hyperparameters used to generate each sample is provided, along with details regarding the source and task type for the reference texts.
Task source: Real task examples (e.g. GitHub issues), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 2700
Dataset size notes: 15090 (train)
Dataset metadata available: Yes
Dataset metadata detail: source information index, generating model, temperature, whether quality issues are present in the sample, task type of the data, source of the original content, prompt used to generate the response, base content for RAG
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: by task type (QA, summarization, data-to-text writing)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Benchmark statistics and quality checking are described. Hallucination density is assessed across models used to generate the data, in relation to context length, and versus position in the text.
Taxonomy: Retrieval / 
Alternate taxonomy: Factuality
New taxonomy: General Application / Document Retrieval