KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Authors: name: Jifan Yu, name: Xiaozhi Wang, name: Shangqing Tu, name: Shulin Cao, name: Daniel Zhang-Li, name: Xin Lv, name: Hao Peng, name: Zijun Yao, name: Xiaohan Zhang, name: Hanming Li, name: Chunyang Li, name: Zheyuan Zhang, name: Yushi Bai, name: Yantao Liu, name: Amy Xin, name: Nianyi Lin, name: Kaifeng Yun, name: Linlu Gong, name: Jianhui Chen, name: Zhili Wu, name: Yunjia Qi, name: Weikai Li, name: Yong Guan, name: Kaisheng Zeng, name: Ji Qi, name: Hailong Jin, name: Jinxin Liu, name: Yu Gu, name: Yuan Yao, name: Ning Ding, name: Lei Hou, name: Zhiyuan Liu, name: Bin Xu, name: Jie Tang, name: Juanzi Li

The unprecedented performance of large language models (LLMs) necessitates
improvements in evaluations. Rather than merely exploring the breadth of LLM
abilities, we believe meticulous and thoughtful designs are essential to
thorough, unbiased, and applicable evaluations. Given the importance of world
knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark
(KoLA), in which we carefully design three crucial factors: (1) For ability
modeling, we mimic human cognition to form a four-level taxonomy of
knowledge-related abilities, covering 19 tasks. (2) For data, to ensure fair
comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs,
along with continuously collected emerging corpora, aiming to evaluate the
capacity to handle unseen data and evolving knowledge. (3) For evaluation
criteria, we adopt a contrastive system, including overall standard scores for
better numerical comparability across tasks and models and a unique
self-contrast metric for automatically evaluating knowledge hallucination. We
evaluate 21 open-source and commercial LLMs and obtain some intriguing
findings. The KoLA dataset and open-participation leaderboard are publicly
released at https://kola.xlore.cn and will be continuously updated to provide
references for developing LLMs and knowledge-related systems.