Title: CONFLICTBANK: A Benchmark for Evaluating Knowledge Conflicts in Large Language Models
Benchmark: CONFLICTBANK
Summary: Large language models (LLMs) have made significant progress, but the issue of knowledge conflicts, which can lead to hallucinations, remains underexplored. To address this, CONFLICTBANK, a large benchmark of claim-evidence and QA pairs, is introduced to study conflicts arising from misinformation, temporal discrepancies, and semantic divergences. Through comprehensive experiments on various LLMs, the study provides insights into conflicts in retrieved and encoded knowledge, highlighting the importance of resolving these conflicts for developing trustworthy AI.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Knowledge conflicts in LLMs
Phenomenon defined by authors: Yes
Phenomenon definition: Retrieved conflicts arise during the inference stage when newly retrieved information contradicts the model’s parametric memory, while embedded conflicts occur during the training stage due to discrepancies within the training text itself
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Answer question-answer pairs
Task item definition: QA pair, conflict type, answer, memorization ratio
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 553,117
Dataset metadata available: Yes
Dataset metadata detail: conflict type
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Memorization score
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: conflict types
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Factuality / 
New taxonomy: General Capability / Factuality