Title: NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates
Benchmark: NewTerm
Summary: This paper introduces NewTerm, an adaptive benchmark designed for the real-time evaluation of new terms in large language models (LLMs) to address their struggle with real-time information due to knowledge cutoffs. The benchmark is constructed using a highly automated method allowing flexible and minimal human effort updates, revealing a performance reduction of over 20% on various LLMs with new terms and highlighting difficulties in generalizing to distant new terms. Annual updates to NewTerm, starting with 2022 and 2023, are planned to continuously assess and analyze the evolving challenge of new terms in LLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Updating of knowledge, real-time evaluation of new terms introduced after knowledge cutoff
Phenomenon defined by authors: Yes
Phenomenon definition: flexible updates for real-time information
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Answer questions about new terms from dictionary, introduced after knowledge cutoff
Task item definition: Question, multiple choice answers, response, correct
Task source: Real task examples (e.g. GitHub issues), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset metadata available: Yes
Dataset metadata detail: Domains: The Choice of Multiple Alter (COMA), The Choice of Similar Terms (COST), Common Sense Judgement (CSJ)
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: Domains: The Choice of Multiple Alter (COMA), The Choice of Similar Terms (COST), Common Sense Judgement (CSJ)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Updating
New taxonomy: General Application / Updating