Title: TemporalWiki: A Lifelong Benchmark for Training and Evaluating
Ever-Evolving Language Models
Benchmark: TEMPORALWIKI
Summary: Most LLM benchmarks are static yet real factutal knowledge changes, increases and depreciates. TemporalWiki addresses language models' temporal misalignment by providing a benchmark derived from consecutive Wikipedia snapshots to assess how well models adapt to evolving knowledge. The findings demonstrate that updating models using only the differences between snapshots achieves comparable or better perplexity than retraining on entire snapshots, while reducing computational costs by 12x.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Temporal Misalignment
Phenomenon defined by authors: Yes
Phenomenon definition: "temporal misalignment, which refers to neural networks showing poor performance due to misalignment in time between the train and test data"
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: TWIKI-PROBES:  "factual phrases synthetically generated from a naive concatenation of Subject, Relation, and Object" from English Wikipedia and Wikidata to evaluate temporal misalignment.
Task item definition: A naive concatenation of Subject, Relation from English Wikipedia and Wikidata  e.g. [Subject: Mario Chalmers] [Relation: member of sports team] where the model should generate [Object: Indios de Mayagüez] based on the following sentence in Wikipedia: "On September 27, 2021, Chalmers signed with Indios de Mayagüez of the Baloncesto Superior Nacional"
Task source: Real task examples (e.g. GitHub issues)
Task source detail: Tasks are sourced from English Wikipedia and English Wikidata.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Whilst the data to build the tasks is sourced from English Wikipedia and English Wikidata, the task itself is a naive concatenation of Subject, Relation from a real Wikipedia sentence where object is the model output that is evaluated.
Dataset size: It is an evolving dataset so there is no fixed size.
Dataset metadata available: Yes
Dataset metadata detail: Changed/Unchanged Facts
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: Simple average of perplexity for different snapshots of the wikipedia data.
Metric subscores: Yes
Metric subscores detail: For changed/unchanged facts and for different snapshots of the wikipedia data.
Metric access: Outputs alone
Task face validity: It is evaluating temporal misaglignment through the specific lens of factual information on Wikipedia.
Metric face validity: rima facie reason to believe that perplexity on factual completions is a valid metric for benchmarking a language model's ability to adapt to changing knowledge over time (the target phenomenon of temporal misalignment). But the task format is very synthetic.
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Author validity detail: Authors acknowledge that Wikipedia and Wikidata are not true reflections of real-world knowledge. They do not directly discuss the impact of their synthetic task format.
Taxonomy: Language Modelling / Updating
New taxonomy: General Application / Updating