Title: NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms
Benchmark: NEO-BENCH
Summary: The performance of Large Language Models (LLMs) declines due to the temporal drift between the training data and newer texts, notably impacted by the emergence of neologisms. A resource of recent English neologisms is created and analyzed, revealing that introducing new words significantly reduces model performance in tasks like machine translation. To address this, a benchmark is constructed to evaluate LLMs' ability to handle neologisms across various natural language understanding tasks, showing that models trained on more recent data perform better and highlighting the complexity neologisms pose for static LLMs
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LLM performance degradation due to temporal drift between data used for model training and newer text seen during inference
Phenomenon defined by authors: Yes
Phenomenon definition: language change causing data drift due to the emergence of neologisms – new word forms
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Answer multiple choice cloze questions based on example text with masked word, machine translation, definition generation, perplexity comparison of individual words
Task item definition: Text, answer, score
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 2162
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Updating
New taxonomy: General Application / Updating