Title: WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia
Benchmark: WikiContradict
Summary: Retrieval-augmented generation (RAG) helps mitigate limitations in large language models (LLMs), but how LLMs handle knowledge conflicts from equally trustworthy sources remains unclear. The WikiContradict benchmark, consisting of 253 high-quality, human-annotated instances, evaluates LLM responses to contradictory passages from Wikipedia. Evaluations reveal that while LLMs struggle to generate answers reflecting the conflicting nature of contexts, especially with implicit conflicts, an automated model achieves an F-score of 0.8 in estimating LLM performance, highlighting areas for further improvement.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Real-world knowledge conflicts (intra-context)
Phenomenon defined by authors: Yes
Phenomenon definition: knowledge inconsistencies arise from the same or different retrieved
passages that originate from a single trusted source (Wikipedia) and are considered equally credible
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Answer questions about text passages under 5 different prompt templates
Task item definition: Question, context1, context2, answer, contradiction type, reference answer
Task source: Real task examples (e.g. GitHub issues)
Task source detail: Wikipedia articles
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 253
Dataset metadata available: Yes
Dataset metadata detail: question type, contradiction type
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: prompt template, question type, contradiction type
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Knowledge / Conflicts
New taxonomy: General Application / Knowledge Conflicts