Title: MULAN : A Study of Fact Mutability in Language Models
Benchmark: MULAN
Summary: The authors create MULAN , a benchmark for evaluating the ability of English language models to anticipate time-contingency, covering both 1:1 and 1:N relations. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: anticipating when facts are time-contingent
Phenomenon defined by authors: Yes
Phenomenon definition: time awareness in LLMs, specifically for encoding of fact mutability in their representations and for the comparative ease of editing of mutable facts versus immutable ones.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: Given a subject–relation query (input), the task is to predict the correct object(s) (output), where queries may involve either immutable or mutable facts. 
Task item definition: subject–relation query (input), correct object(s) (output). 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: Test: 35000 / 31410. 
Dataset size notes:  Train: 6230 / 6820; Validation: 5780 / 5910. 
Dataset metadata available: Yes
Dataset metadata detail: Immutable or not. 
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: Immutability-1 and Immutability-N. 
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Temporal
New taxonomy: General Capability / Temporal Reasoning