ConflictBank: A Benchmark for Evaluating the Influence of Knowledge
  Conflicts in LLM
Authors: name: Zhaochen Su; name: Jun Zhang; name: Xiaoye Qu; name: Tong Zhu; name: Yanshu Li; name: Jiashuo Sun; name: Juntao Li; name: Min Zhang; name: Yu Cheng
Published: 2024-08-22 02:33:13

Abstract: Large language models (LLMs) have achieved impressive advancements across
numerous disciplines, yet the critical issue of knowledge conflicts, a major
source of hallucinations, has rarely been studied. Only a few research explored
the conflicts between the inherent knowledge of LLMs and the retrieved
contextual knowledge. However, a thorough assessment of knowledge conflict in
LLMs is still missing. Motivated by this research gap, we present ConflictBank,
the first comprehensive benchmark developed to systematically evaluate
knowledge conflicts from three aspects: (i) conflicts encountered in retrieved
knowledge, (ii) conflicts within the models' encoded knowledge, and (iii) the
interplay between these conflict forms. Our investigation delves into four
model families and twelve LLM instances, meticulously analyzing conflicts
stemming from misinformation, temporal discrepancies, and semantic divergences.
Based on our proposed novel construction framework, we create 7,453,853
claim-evidence pairs and 553,117 QA pairs. We present numerous findings on
model scale, conflict causes, and conflict types. We hope our ConflictBank
benchmark will help the community better understand model behavior in conflicts
and develop more reliable LLMs.