Benchmarking Chinese Knowledge Rectification in Large Language Models
Authors: name: Tianhe Lu; name: Jizhan Fang; name: Yunzhi Yao; name: Xin Xu; name: Ningyu Zhang; name: Huajun Chen
Published: 2024-09-09 17:11:51

Abstract: While Large Language Models (LLMs) exhibit remarkable generative
capabilities, they are not without flaws, particularly in the form of
hallucinations. This issue is even more pronounced when LLMs are applied to
specific languages and domains. For example, LLMs may generate nonsense
information when handling Chinese ancient poetry, proverbs, or idioms, owing to
the lack of specific knowledge. To this end, this paper introduces a benchmark
for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically,
we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of
knowledge from various sources, including classical texts, idioms, and content
from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony,
antithesis, and logical constructs inherent in the Chinese language. Through
the analysis of this dataset, we uncover the challenges faced by current LLMs
in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge
editing techniques on this dataset unveil the substantial scope for advancement
in the rectification of Chinese knowledge. Code and dataset are available at
https://github.com/zjunlp/EasyEdit.