When Benchmarks Age: Temporal Misalignment through Large Language Model
  Factuality Evaluation
Authors: name: Xunyi Jiang, name: Dingyi Chang, name: Julian McAuley, name: Xin Xu
Organization: McAuley-Lab

The rapid evolution of large language models (LLMs) and the real world has
outpaced the static nature of widely used evaluation benchmarks, raising
concerns about their reliability for evaluating LLM factuality. While
substantial works continue to rely on the popular but old benchmarks, their
temporal misalignment with real-world facts and modern LLMs, and their effects
on LLM factuality evaluation remain underexplored. Therefore, in this work, we
present a systematic investigation of this issue by examining five popular
factuality benchmarks and eight LLMs released across different years. An
up-to-date fact retrieval pipeline and three metrics are tailored to quantify
benchmark aging and its impact on LLM factuality evaluation. Experimental
results and analysis illustrate that a considerable portion of samples in the
widely used factuality benchmarks are outdated, leading to unreliable
assessments of LLM factuality. We hope our work can provide a testbed to assess
the reliability of a benchmark for LLM factuality evaluation and inspire more
research on the benchmark aging issue. Codes are available in
https://github.com/JiangXunyi/BenchAge.