Title: Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization
Benchmark: TEMPOSUM
Summary: Propose a novel benchmark. Show that parametric knowledge stored in summarization models significantly affects the faithfulness of the generated summaries on future data. Discuss recommendations to the research community. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Text summarization
Phenomenon defined by authors: Yes
Phenomenon definition: "Abstractive summarization aims to generate a concise summary that contains the critical information of the source text while ensuring the generated text is fluent and faithful. This paper studies how PLMs that excel on summarizing data originating from the same temporal context as the pre-trained corpus generalize their summarization capabilities to OOD future data."-p16205
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: generalize to data of time that is in or out of distribution of model's training data
Task item definition: The model is given a news article then asked to summarize it.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 12734
Dataset metadata available: Yes
Dataset metadata detail: in/out of distribution, source
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric subscores: Yes
Metric subscores detail: in/out of distribution, source
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: the authors conduct human experiment
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization