Evaluating LLMs at Temporal Generalization and Bias
Authors: ChenghaoZhu, Nuo Chen, Yufei Gao, Benyou Wang


Reviewer paper summaries: This paper addresses the issue that traditional benchmarks cannot effectively evaluate the temporal generalization of LLMs. They propose that LLMs should be evaluated on both temporal generalization and temporal bias dimensions. To this end, they introduce Freshbench, a benchmark designed to assess the temporal generalization of LLMs with real-time updates. The authors collected relevant and updatable data from sources such as Arxiv, BBC, Wikipedia, Github, Reddit, and Quora, and conducted detailed evaluations and analyses on 38 open-source LLMs.
Summary of strengths: - The motivation of this paper is straightforward and intuitive.
- The research problem addressed in this paper is significant, as it can drive the community towards more objective and accurate evaluation of LLMs' performance over time.
- The work involved is substantial, evaluating 38 open-source LLMs, and the data provided can serve as a valuable reference for other studies.
Summary of weaknesses: Question: Figure 2 shows two distinct patterns in the use of historical knowledge by Phi-1.5 and Zephyr-7b. However, when extended to a longer time dimension and more models are included, this difference seems to disappear (Figure 3 and 7). Based on your observations, what causes this discrepancy?
Comments, suggestions and typos: - The column headers in Table 2 need to have consistent capitalization.
Overall assessment scores: 3.5