Evaluating Robustness of Large Language Models with Neologisms (2024)

Abstract: The performance of Large Language Models (LLMs) degrades from the temporal drift between data used for model training and newer text seen during inference. One understudied avenue of language change causing data drift is the emergence of neologisms – new word forms – over time. We create a diverse resource of recent English neologisms by using several popular collection methods. We analyze temporal drift using neologisms by comparing sentences containing new words with near-identical sentences that replace neologisms with existing substitute words. Model performance is nearly halved in machine translation when a single neologism is introduced in a sentence. Motivated by these results, we construct a benchmark to evaluate LLMs’ ability to generalize to neologisms with various natural language understanding tasks and model perplexity. Models with later knowledge cutoff dates yield lower perplexities and perform better in downstream tasks. LLMs are also affected differently based on the linguistic origins of words, indicating that neologisms are complex for static LLMs to address. We will release our benchmark and code for reproducing our experiments.

heading: 1 Introduction
text: Neologisms – recent word forms representing a new meaning, sense, or connotation (Cartier, 2017) – consistently surface as language changes. Neologisms emerge to describe the ever-changing state of the world, such as new terms created during the COVID-19 pandemic. While humans easily adapt to language change, large language models (LLMs) struggle with the misalignment of training data and new test data distributions (Luu et al., 2022).
Prior work on temporal language change (Lazaridou et al., 2021; Onoe et al., 2022; Luu et al., 2022) observed model degradation when finetuning on older text and evaluating on newer data and named entities (Rijhwani and Preotiuc-Pietro, 2020; Agar-
wal and Nenkova, 2022; Liu and Ritter, 2023). However, as far as we are aware there has not been prior work that analyzes the robustness of LLMs on handling neologisms. We show that adding a neologism to text decreases machine translation quality by an average of 43% in a human evaluation (§2), even for popular words emerging before 2020.
In this paper, we present NEO-BENCH, a new benchmark designed to test the ability of LLMs to understand and process neologisms. We combine multiple methods and online text corpora to collect a diverse set of 2,505 neologisms based on the linguistic taxonomy devised by Pinter et al. (2020): (i) lexical neologisms – words representing new concepts, e.g., “long covid”; (ii) morphological neologisms – blends of existing subwords, e.g., “doomscrolling”; and (iii) semantic neologisms – existing words that convey a new meaning or sense, e.g., “ice” (a term that refers to petrolor diesel-powered cars taking electric car charging spots). We estimate word prevalence over time with Google Trends to obtain trending neologisms. We also create 4 benchmark tasks to evaluate the
1 13885
impact of neologisms on LLMs with Perplexity, Cloze Question Answering, Definition Generation, and Machine Translation.
We show that lower neologism perplexities correlate with higher downstream task performance. Older LLMs – BART, T5, GPT-J, and Flan-T5 – perform much worse with an average of 32.20% and 12.27% accuracy in question answering and definition generation, respectively. We also find that automatic metrics do not accurately measure the quality of translated sentences containing neologisms, evidenced by Spearman’s ρ rank correlation between COMETKiwi (a state-of-the-art metric) and human judgment, which is 0.491. This is lower than the average ρ of 0.629 for COMETKiwi across 5 language pairs reported in the WMT23 Quality Estimation task (Blain et al., 2023). LLM performance in NEO-BENCH also differs based on a word’s linguistic type, as lexical neologisms without derivations yield the highest perplexities and the most fragmented subword tokenization, while semantic neologisms that repurpose existing words result in literal definitions and translations.
NEO-BENCH evaluates a diverse set of LLM capabilities on handling neologisms in various tasks. Models must also understand compositionality for morphological neologisms, differentiate between word senses for semantic neologisms, and handle different contexts for lexical neologisms.

heading: 2 Motivation
text: We start by using machine translation as an example to illustrate the significant challenge neologisms pose on state-of-the-art NLP systems. We
manually collect 100 neologism words with sentential context from social media, news articles, and dictionaries. GPT-3.5, GPT-4 and commercial translation systems, e.g., Google Translate,1 Microsoft Bing,2 and DeepL Translator,3 only managed to correctly translate about 34-47% of these 100 sentences that contain neologisms based on our manual inspection (Figure 2; from English to Chinese). In stark contrast, when replacing the neologism with a common word in these sentences, the percentage of correct translations rises substantially to 67-73%. We observe similar trends in open-source translation models, such as ALMA (Xu et al., 2023) and M2M100 (Fan et al., 2020).
One thing to note is that these replacement words are not exact synonyms, but words that have been carefully chosen to create a near-identical, semantically plausible sentence; because new words emerge in areas not occupied by existing words (Ryskina et al., 2020), true synonyms would often be verbose and incompatible with the sentence context. Because the original sentences containing neologisms were collected in the wild, one might assume they would be even more natural in comparison to their modified counterparts, but yet, there is a large gap in translation quality between neologism and non-neologism words for all models.
A closer look reveals that six typical types of errors are made in mistranslated model outputs, which include (ordered by severity):
• Unnatural: Imperfect translation of the sen1https://translate.google.com/ 2https://www.bing.com/translator 3https://www.deepl.com/translator
2
tence due to grammatical errors; • Literal: Inaccurate output that literally trans-
lates the neologism or remaining sequence; • Partial: Part of the sentence is untranslated
and left out of the output; • Mistranslation: Incorrectly translated sen-
tence portion leads to a poor understanding of the overall sentence meaning;
• Copy: Part of the output is not translated and copied from the English input;
• Incomprehensible: Incoherent output that fails to capture any original sentence meaning;
Table 12 in the Appendix shows translations for each error type. The most common errors are mistranslations and literal translations with an average of 27.3% and 13.7% respectively. Model output for non-neologism sentences is more likely to have minor errors and be labeled unnatural by annotators.
Another interesting observation is that newer neologisms indeed show lower rates of good translations and often higher rates of mistranslations, as one may expect. Figure 3 shows the percentage of good translations and mistranslations over time for varied models. Compared to non-neologism sentences, models still yield lower rates of correct translations for neologisms that emerged before 2020. Many neologisms use existing words to convey meanings, such that the poor performance of models is not wholly explained by the absence of these word forms in training data. We propose a novel benchmark (§3) to systematically study the impact of neologisms on LLMs (§4).
3 NEO-BENCH: A Neologism Benchmark
We create NEO-BENCH, a benchmark that consists of 2,505 neologisms (both words and phrases) that newly emerged around 2020–2023 and 4 intrinsic/extrinsic tasks (Table 1) to evaluate LLMs’ abilities to generalize on neologisms. To facilitate continuous research on neologisms and language change, we intend to periodically update NEOBENCH with neologisms emerging after 2023.

heading: 3.1 Neologism Collection
text: A neologism is a term that represents a new meaning or sense (Cartier, 2017). Previous datasets (McCrae, 2019; Ryskina et al., 2020; Zhu and Jurgens, 2021) only collected specific word types, ignored neologisms conveying new meanings with existing words, and did not utilize word prevalence trends
20%
40%
60%
80% Google Translate Microsoft Bing
80% DeepL Translator
GPT-4
(more in Related Work §6). We design a more systematic collection process to quantify the effect of neologisms on a language’s data distribution. Filtering Reddit Data based on Google Trends (Method 1). New words commonly propagate in online communities (Zhu and Jurgens, 2021), thus, we count word frequencies in monthly Reddit data to find single-word neologism candidates. We set a frequency cutoff between 50 and 100 per month to obtain uncommon words and remove misspellings and named entities using SpaCy (Montani et al., 2023), resulting in 74,542 candidates. We further obtain word search frequencies from 2010 to 2023 on Google Trends4 and automatically filter out 87.13% of neologism candidates based on these trend lines (see Figure 4 for examples) by a combination of curve fitting, argmax detection, and integrals over time. Appendix §B.1 provides more details about trend filtering. From the set of 9,590 remaining candidates, we find that 10.48% are prevalent neologisms by manual inspection. In total, we collected 1,005 neologism words from Reddit (310 lexical, 588 morphological, and 107 semantic neologisms). Retrieving News Articles about Neologisms (Method 2). As Method 1 is only good at find-
4https://trends.google.com/trends/
3
ing single-word neologisms, we turn to news articles that explain the meanings of neologisms to collect multi-word expressions. We first manually get 100 neologisms from news articles, recording news headlines of neologisms. Then, based on the shared text patterns of headlines, we created 16 headline templates (e.g., “___: What is it?”) to retrieve Google News articles from 2019 to 2023. Using SpaCy, we identify 60,671 noun and verb phrases with a Part-of-Speech tagger and remove duplicates and named entities. We used the same aforementioned filtering method for these phrases using Google Trends. From the remaining 8,039 candidates, we manually extracted 1,100 neologisms (778 lexical, 222 morphological, and 100 semantic neologisms), of which 713 are multiwords.
Sampling Existing Neologism Datasets (Method 3). To supplement our dataset with additional neologisms, we also sample from two existing open-source resources that contain a lot of rare words, many of which have no Google Trends data available. The NYT First Said Twitter bot (@NYT_first_said) tweets out words when they are used for the first time in New York Times articles by using exclusion lists. We retrieve 1,100 of its tweets from 2020 to collect 200 derived neologisms (192 morphological and 8 semantic). We also sample 1,400 entries from another noisy, automatically constructed, dataset of 80,071 new slang dictionary entries (Zhu and Jurgens, 2021). We manually filter the sample and collect 200 derived neologisms (4 lexical, 194 morphological, and 2 semantic).
Overall. We collected 2,505 neologism words. While semantic neologisms are infrequent in all sources, Google Trends data enables the collection of them, as these words change in baseline prevalence when a new sense is being popularized. Only
5.04% of words from Reddit, 1.12% of phrases from news articles, and 3.09% of entries from previous datasets overlap with candidates from the other two sources — highlighting the importance of using multiple diverse data sources and methods for neologism collection. We also verified that 44.23% of these 2,505 words actually appear in the Urban Dictionary, a crowdsourced English-language online dictionary for slang words and phrases.

heading: 3.2 Benchmark Tasks
text: NEO-BENCH consists of 4 tasks – 3 downstream and 1 intrinsic metric – to evaluate models’ knowledge of neologisms: (i) Machine Translation with human and automatic evaluation; (ii) Cloze Question Answering to evaluate models in context; (iii) Definition Generation to evaluate models in a context-free setting; and (iv) perplexity to compare single-word neologisms to commonly used words. We describe the setup and result tables/figures in this section, then discuss the key findings based on these results more in-depth in §4. Machine Translation (Task 1). We sample from our collected neologisms (§3.1) and search for reference sentences containing these words on social media and Google. We construct 240 sentences, including the 100 used in §2. We work with in-house native speakers to create reference translations (English to Chinese) and evaluate system outputs in Table 5 with BLEU (Papineni et al., 2002), COMET (Rei et al., 2020), and MetricX-23 (XXL and XL) (Juraska et al., 2023). We also use the referencefree metrics COMETKiwi (Rei et al., 2022) and MetricX-23-QE (XXL and XL). Lower MetricX scores indicate higher performance. We report the correlations of metrics with human ratings for the MT models in §2 using Spearman’s ρ in Table 5. Cloze Question Answering (Task 2). We sample 750 neologisms to create text passages, where one sentence contains a neologism and the remaining passage serves as preceding or following context
4
a) Definition Generation Output Examples
Reference Definition: A stablecoin is a type of cryptocurrency where the value of the digital asset is supposed to be pegged to a reference asset, which is either fiat money, exchange-traded commodities, or another cryptocurrency.
(see example in Table 2). We mask out the neologism and provide four incorrect answers plus one distractor answer, which is a common word or phrase that is feasible in context. We evaluate BART-large (Lewis et al., 2019), T5-Large (Raffel et al., 2020), Flan-T5-Large (Chung et al., 2022), GPT-J 6B (Wang and Komatsuzaki, 2021), LLaMA-1 7B (Touvron et al., 2023a), Alpaca 7B (Taori et al., 2023), LLaMA-2, LLaMA-2-Chat (Touvron et al., 2023b), OLMo-7B (Groeneveld et al., 2024), OLMo-7B-Instruct, Mistral-7B (Jiang et al., 2023), Mistral-7B-Instruct, GPT 3.5 (Brown et al., 2020), and GPT-4 in multiple-choice Cloze Question Answering (QA). We experiment with 5-shot prompting and test three sizes of LLaMA-2 models. We show results in Figure 5 with the stratified and combined accuracies of selecting either the neologism or distractor answer.
Open-ended Definition Generation (Task 3). We evaluate the same models from Task 2 for their context-free knowledge of 750 neologisms with question prompts (i.e., “What is doomscrolling?”) to obtain neologism definitions. We construct human reference definitions and use GPT-4 to evaluate if model generations are semantically equivalent to the gold reference. We use 5-shot prompting and report results with accuracy in Figure 5. Table 3 shows example LLM-generated definitions.
Perplexity Rankings (Task 4). Using 422 Cloze passages that have both singular distractor and ne-
ologism answers, we use perplexity to evaluate GPT-J 6B, LLaMA-1 7B, Alpaca 7B, LLaMA-2 7B, LLaMA-2 Chat 7B, OLMo-7B, OLMo-7BInstruct, Mistral-7B, and Mistral-7B-Instruct. For each passage, we use rank classification (Brown et al., 2020), where we fill in the mask with the neologism and measure the perplexity of the passage. We replace the mask with the distractor answer and the top 5000 singular words from Reddit by frequency and measure perplexities of all 5002 sequences. The mask-filling words are sorted by the corresponding sequence perplexity, and the average rankings of neologisms and distractors are reported in Figure 7. Lower neologism rankings represent lower relative perplexities and show that models are likely to complete the passage with a neologism.
5

heading: 4 Key Findings
text: We utilize NEO-BENCH tasks to evaluate the ability of various LLMs to adapt to neologisms. The following are our key findings:
Current automatic metrics cannot accurately evaluate MT models that struggle with neologisms. In §2, MT models decrease in performance by 43% when translating neologisms with Bing being the best model based on human evaluation. However, COMET and COMETKiwi scores are notably high and MetricX-23 error scores are low for all models in Table 5. The best models are Google Translate for BLEU (0.507); DeepL for COMETKiwi (0.807) and MetricX-23-QE (1.260); and GPT-4 for COMET (0.854) and MetricX-23 (1.550), highlighting that automatic metrics show poor system-level correlations with human judgments. For sentence-wise correlation between
MT metrics and human evaluations, the average Spearman’s ρ of BLEU, COMET, COMETKiwi, MetricX-23 and MetricX-23-QE (XXL) is 0.244, 0.445, 0.491, 0.457, 0.451, respectively. In contrast, COMETKiwi, our highest correlating metric, has an average ρ of 0.629 for five language pairs on the WMT23 Quality Estimation task for direct assessment (Blain et al., 2023). From our reference sentences, translating neologisms often requires paraphrasing, resulting in low ρ for BLEU.

heading: GPT-4’s knowledge of neologisms is task specific.
text: Table 4 presents the human annotations of GPT-4 translations of neologisms, separated by the corresponding performance of neologisms in Cloze QA, Definition Generation, and Definition Prompting, where we ask GPT-4 if the provided human reference definitions of neologisms are correct. GPT-4 shows complete knowledge of a neologism if all tasks are correct; partial knowledge if one task is in-
6
correct; and unknown if multiple tasks are incorrect. There are higher rates of correct translations for neologisms GPT-4 understands, as good translations drop by 20.3% if GPT-4 does not fully know a neologism’s meaning. However, the rate of mistranslations are constant regardless of GPT-4 performance in other tasks, and GPT-4 only correctly translates 53.13% of neologisms it has complete knowledge of. GPT-4’s knowledge of neologisms is compartmentalized and does not result in similarly high performance in machine translation compared to other Neo-Bench tasks, emphasizing the difficulty of translating neologisms. Models perform worse on neologisms than preexisting words. For Cloze questions in Figure 5, neologism answers are designed to be more natural as the original passages contained these neologisms, yet all models select a large portion (27.99% on average) of distractor answers. Neologisms also have an average perplexity rank of 463 compared to distractor rankings of 45 in Figure 7, indicating much lower perplexity for pre-existing words. Older LLMs perform significantly worse. In Figure 5, the average performance of GPT-J, BART, T5, and Flan-T5 is 26.99% lower in Cloze QA and
5Mistral AI does not reveal training data details, so we provide our best estimation for the model’s knowledge cutoff.
47.78% lower in Definition Generation than other models. In Figure 7, GPT-J and LLaMA-1 models exhibit higher neologism rankings than newer open-source models, correlating with lower downstream performance. Newer models – GPT-3.5, GPT-4, LLaMA-2, OLMo, and Mistral – perform better as they are trained on data containing newer neologisms, generally have algorithmic improvements, and are trained with more resources than older models.
Perplexity rankings of older models increase drastically from 2019 until 2021. While neologisms are often gradually worked into a vocabulary (Zhu and Jurgens, 2021), we use trend lines to best estimate the date when a neologism becomes popular and report perplexity over time in Figure 6. Newer models dip in 2020 but increase afterward as 52% of neologisms from this period are now conventionalized terms related to COVID-19.
Larger models handle neologisms better. Increasing the sizes of LLaMa-2 and LLaMA-2 Chat leads to consistent improvements across both Cloze Question Answering and Definition Generation. On average, LaMA-2 70B and LLaMA-2 Chat 70B yield 10.13% higher accuracy in Cloze QA and 5.93% higher accuracy in Definition Generation than LLaMA-2 7B and LLaMA-2 Chat 7B.
Instruction-tuning results in high neologism perplexities. In Figure 7, LLaMA-1, LLaMA-2, OLMo, and Mistral models have, on average, 125 lower neologism rankings than their instruct-tuned counterparts. Instruct models are trained with dialogue (Wei et al., 2022), so uncommon generation is less desired.
7

heading: 5 Linguistic Taxonomy Analysis
text: We separate NEO-BENCH task results by neologism linguistic structure: lexical, morphological, and semantic. Figure 7 presents perplexity rankings, Figure 9 reports human evaluation for MT, and Figure 10 shows the results for the best models on Cloze QA and Definition Generation.
Lexical neologisms produce the highest perplexities, but yield the best downstream results. On average, lexical neologisms have 226 higher rankings than other words, indicating higher relative model perplexities. Figure 8 shows the distribution of characters per token of neologisms using the LLaMA tokenizer, and the average number of characters per token for lexical, morphological, and semantic neologisms is 2.36, 2.98, and 3.24 respectively. Lexical neologisms have more fragmented tokenizations, as these words have the highest proportion of 1-2 character tokens. Lexical neologisms are less likely to be separated into long, common word roots or segments representing subword information, instead producing uncommon token sequences that result in higher neologism rankings and perplexity. In downstream tasks, however, lexical neologisms yield 0.6% higher Cloze accuracy, 8.8% more correct definitions, and 21.5% more good translations than other neologisms.
Morphological neologisms produce low perplexities but yield poor downstream performance. Compared to lexical neologisms, morphological neologisms are, on average, segmented into longer tokens and constructed with common subwords, resulting in lower perplexity rankings. However, they yield 4.2% lower Cloze accuracy, 9.1% more incorrect definitions, and 26.1% less good translations than lexical neologisms. 76.8% of neologisms without trend lines are morphological. Compared to lexical and semantic neologisms that require preva-
to sentences without neologisms, replacing a word064 or phrase with a neologism decreases model per-065 formance by 44%. Motivated by this study, we066 create benchmark tasks in Neo-Bench to evalu-067 ate the impact of neologisms on LLMs through068 model perplexity and downstream tasks, includ-069 ing Cloze Question Answering, Definition Genera-070 tion, and Automatic Machine Translation. We find071 that model perplexities of neologisms are signifi-072 cantly higher than pre-existing words, and there is073 a positive correlation between later knowledge cut-074 off dates and downstream task performance, with075 an average of 54.7% improvement in accuracy be-076 tween the newest and oldest models.077 In summary, our key contributions include:078 • Motivating study of temporal drift from single in-079 stances of neologisms in machine translation (§2).080 • We introduce a new corpus of neologisms and081 benchmark tasks to evaluate LLMs (§3).082 • General results of LLMs on benchmark tasks (§4)083 and analysis of trends across tasks depending on084 the linguistic origin of neologisms (§5).085
2 Ne logism Benchmark Motivation086
There are no currently no benchmarks to observe087 the specific impact of neologisms on models. To088 determine if neologisms have an impact on LLMs,089 we evaluate machine translation (MT) models for090 English-to-Chinese translation. We collect 100091
popular neologisms from 2020-2023 and construct 092 input pairs for machine translation. Each pair con- 093 sists of a sentence containing a neologism and a 094 near-identical, logically feasible sentence that re- 095 places the neologism with a carefully constructed 096 substitute term. These replacement words are not 097 synonyms, as new words emerge in areas not occu- 098 pied by existing words (Ryskina et al., 2020), so 099 neologism synonyms are often verbose and incom- 100 patible with the sentence context. 101
We test commercial systems: Google Translate1, 102 Microsoft Bing2, and DeepL Translator3 and open 103 source models: ALMA 7B (Xu et al., 2023) and 104 M2M100 1.2B (Fan et al., 2020). MT output is 105 manually annotated with hand-crafted labels of in- 106 creasing error severity: 107
• Good: Correct translation that captures the 108 meaning of the neologism. 109 • Unnatural: Imperfect translation of the sen- 110 tence due to grammatical errors. 111 • Literal: Inaccurate output that literally trans- 112 lates the neologism or remaining sequence. 113 • Partial: Part of the sentence is untranslated 114 and left out of the output. 115 • Mistranslation: Incorrectly translated sen- 116 tence portion leads to a poor understanding of 117 the overall sentence meaning. 118
1https://translate.google.com/ 2https://www.bing.com/translator 3https://www.deepl.com/translator
2
Figur 9: Results of the human-an otated MT models for each linguistic type of neologism.
lenc to be differentiated from coh rent s ings,
morphological neologisms are created with polynomial combinations of common subwords. Many of these intelligible combinations are largely unused, resulting in lower downstream performance. Semantic neologisms produce the lo est perplexities and the worst performance in generation tasks. Since these neologisms use existing w rd forms, they have an average of 266 lower perplexity rankings than other neologism words. While se antic neologisms yield high Cloze QA accuracy, they also achieve the lowest percentages of correct definitions and translations. Models produce popular definitions and literal translations based on a word’s most common meaning, as the new sense of sema tic neologisms is often nuanced and difficult to capture.

heading: 6 Related Work
text: Temporal Drift in LLMs. Prior work has explored temporal data drift by creating temporal splits of training data data (Loureiro t al., 2022; Luu et al., 2022; Röttger and Pierrehumbert, 2021; Jin et al., 2022; Luu et al., 2022; Lazaridou et al., 2021). New factual updates of concepts are studied with temporal splits of text corpora and QA datasets (Jang et al., 2022; Margatina et al., 2023; Zhao et al., 2022; Vu et al., 2023). Other work has observed model degradation from new named entities (Onoe et al., 2022; Rijhwani and Preotiuc-Pietro, 2020; Chen et al., 2021). Temporal degradation occurs during short-term crisis events where information changes quickly (Pramanick et al., 2022). Studies have consistently found model degradation with perplexity and downstream tasks. There are no studies on model degradation from language change of neologisms, so we create a benchmark to evaluate models on neologisms with similar tasks.
8
Neologism Collection. Using reference texts as exclusion lists to filter common words from target corpora is the most documented method of neologism detection. Target texts and exclusion lists include news articles (Pinter et al., 2020; Falk et al., 2014), dictionaries (Kerremans et al., 2018; Langemets et al., 2020; Liu et al., 2013; Dhuliawala et al., 2016), social media (Pyo, 2023; Zalmout et al., 2019; Megerdoomian and Hadjarian, 2010) and other corpora (Cartier, 2017; Lejeune and Cartier, 2017). These texts are slow to curate, and semantic neologisms are filtered out. Moreover, no resource collects general semantic neologisms.
gSome resources measure word prevalence with time-series data of search queries to collect singleword neologism candidates (Broad et al., 2018) or cybersecurity neologisms (Li et al., 2021). They are limited in scope by collecting only one type of neologism based on rising popularity. Other methods collect neologisms with new slang dictionary entries (Dhuliawala et al., 2016; Zhu and Jurgens, 2021). Dictionaries are slow to update, so new entries are often conventionalized words. There is no resource that uses time-series data to collect a variety of neologisms rising in prevalence.
Previous work has utilized search templates of explanation patterns to collect automatically neologisms (Breen et al., 2018). A few efforts have used neural methods to automatically detect one specific type of neologism, such as adjective-noun neologism pairs (McCrae, 2019), blend words (Megerdoomian and Hadjarian, 2010), and grammatical neologisms (Janssen, 2012; Falk et al., 2014), which are existing words with new parts of speech. No automatic resource collects neologisms
from various topics and linguistic backgrounds. To address these limitations, NEO-BENCH uses multiple methods to semi-automatically collect a variety of neologisms, including multiword, semantic, and prevalent neologisms.
Unseen Words. Rare words or typos are typically unseen in data when training a model but may show up during inference. Prior work has measured model degradation from unseen words (Chirkova and Troshin, 2021; Nayak et al., 2020) and used contextual subword embeddings (Garneau et al., 2018; Chen et al., 2019; Hu et al., 2019; Wang et al., 2019; Araabi et al., 2022), similar surface forms of common words (Chen et al., 2022; Fukuda et al., 2020), and morphological structure (Lochter et al., 2020, 2022) to represent unseen words. Comparatively, the neologism lifecycle follows three stages: emergence, dissemination, and conventionalization (Cartier, 2017). New words often become prevalent and drastically shift a language’s distribution. Semantic neologisms also use existing word forms and are not classified as unseen words.

heading: 7 Conclusion
text: In this paper, we present NEO-BENCH, a new benchmark to test the ability of LLMs to generalize on neologisms. We use several methods to collect a variety of neologisms, including prevalent, multiword, and semantic neologisms. In our experiments, we find that models struggle with neologisms in both perplexity and downstream tasks. Machine Translation is especially difficult, as translating neologisms often requires paraphrasing the sentence. Current automatic metrics cannot measure translation quality, and human evaluation is still needed. Neologisms also affect models differently based on linguistic structure, indicating that this phenomenon is complex for LLMs to address.

heading: Limitations
text: Most of our neologisms largely originated in US and UK English, as we collect textual data from news articles from this region. We do not restrict our locations for Reddit data, but the majority of English-speaking Reddit users are also from the same regions. Given our limited expertise in other English dialects, especially regions whose English variations are largely influenced by other languages, we do not collect many neologisms from Englishspeaking regions outside these regions. However,
9
our computational framework for collecting neologisms can be applied to any language or local variation. For temporal drift of multilingual language modeling, we leave multilingual neologism collection and temporal drift analysis up for future work. Additionally, NEO-BENCH is static as we collect neologisms from mostly 2020-2023, which will become outdated over time as newer models will be exposed to new language in context. However, the semi-automatic collection methods require minimal human supervision and can be dynamically updated to continuously obtain neologisms. These methods require time-series data of words and online text corpora without needing human-curated information like updated dictionaries to filter words. The time-independent filters can collect recent neologisms without needing the time-consuming, manual curation of temporal splits of textual data. Provided that the Google and Reddit Terms of Service and API access enable NEO-BENCH collection, we intend on periodically updating the set of neologisms in our dataset.

heading: Ethical Considerations
text: We utilize Reddit monthly dumps to obtain uncommon words, which often include sensitive information such as account usernames. We take the appropriate measures to ensure that no personally identifiable information (PII) is included in our dataset. We use a named-entity recognition model via SpaCy to identify named entities that are potentially PII and largely remove this information automatically when filtering for neologism candidates. We also manually inspect all the candidates to ensure that no PII is included in our dataset. As we use natural references from Google to construct our model inputs, we also review our hand-crafted sentences to ensure that there is no PII contained in these sentences. Many neologism entries in our work emerge from slang, and some slang words have expletive or offensive meanings. The purpose of our dataset and benchmark is to obtain a representative sample of neologisms and comprehensively evaluate the impact of neologisms on Large Language Models. We present examples that do not contain such offensive information, but these offensive entries are nonetheless a consistent source of neologisms. For expletive neologisms, we strive to create input sentences that capture the meaning of the neologism while not perpetuating gender, racial, and other potential biases. We do
not collect any neologisms that directly reference stereotypes and demographic biases.

heading: Acknowledgments
text: We would like to thank Piranava Abeyakaran, Kaushik Sriram, Vishnesh Jayanthi Ramanathan, Yao Dou, Chao Jiang, Cai Yang, and Xiaofeng Wu for their help in constructing model inputs and translating reference sentences; Qihang Yao and Sam Stevens for their scripts for processing Google Trends and Urban Dictionary data, respectively; Graham Neubig and Paul Michel for valuable discussions. We would also like to thank Azure’s Accelerate Foundation Models Research Program for graciously providing access to API-based models, such as GPT-4. This research is supported in part by the NSF CAREER awards IIS-2144493 and IIS-2052498, ODNI and IARPA via the HIATUS program (contract 2022-22072200004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Related Work
text: Table 6 provides an overview and comparison of English neologism resources, including the types
of neologisms and collection methods used in each dataset. NEO-BENCH uses more collection methods and covers more types of neologisms, including multiword and semantic neologisms.

heading: B Data Collection
text: We base the categories of neologisms on the linguistic taxonomy used in previous literature. Based on our empirical studies, we observe that all neologisms fall under three broad categories: lexical, morphological, and semantic. We additionally label neologisms based on subcategories of these broad linguistic classes (e.g., word blends, derivations, acronyms, and novel phrases).
For Reddit neologism candidates, we collected 500 million utterances in December 2021 and 200 million utterances from January to May 2022. We tokenize the utterances with the NLTK package (Bird et al., 2009) to get individual word counts and update a generic word counter. Neologism candidates are selected by filtering out typos and extremely rare words with less than a frequency of 50. We further filter out named entities by utilizing a SpaCy named entity recognition (NER) model (Honnibal and Montani, 2017) (en_core_web_sm) to detect proper nouns and update a named entity counter. We compare the counts of words from the general counter and the named entity counter and filter out the word if the proportion that a general word is in a named entity is greater than 0.5. The remaining words with the lowest frequencies are the list of uncommon words that we treat as neologism candidates for a given month. In total, we collect 74,542 neologism candidates.
For news articles, we use a script to collect 11,412 headlines from Google News from 2019- 2023. In total, we get 60,671 noun and verb phrases with a Part-of-Speech Tagger via SpaCy (en_core_web_sm) that we treat as neologism candidates. We use an old dataset of 80,071 neologisms obtained from two slang dictionaries (Zhu and Jurgens, 2021) and sample 200 neologisms with interesting or no trend lines. Table 7 provides the breakdown of method overlap between each method pair in NEO-BENCH. Instead of the sample of 1,100 data points, we compare a total of all 6,908 words tweeted out by the NYT First Said bot from 2020 to 2023 with the other methods. Figure 11 provides the breakdown of NEO-BENCH by collection method and linguistic type.
14

heading: B.1 Google Trends Filtering
text: We collect Google Trends monthly data from January 2010 to July 2023. While Google Trends provides data from 2004, there are inconsistencies in word usage frequencies until 2010. To compare word prevalence between neologisms, we make a pairwise comparison of a neologism candidate with the misspelling ‘dangrous’, which provides a consistent baseline comparison for word usage data. We then use this normalized trend line for neologism candidate filtering.
In total, we create five differing methods that use a combination of filtering criteria, including curve-fitting, argmax detection, integral, line of best-fit, and maximum trend data values, to evaluate words as neologism candidates. We select the best combination based on which method yields both high precision and estimated recall in collecting neologisms. Using 20,000 words collected in February 2022, we filter this set through all five methods which filter out almost 90% of words. We sample each method for 100 candidates and manually annotate the samples for neologism classification, obtaining a sampled precision of each method. We combine all the neologisms from the manually-annotated samples to obtain a computationally derived neologism set. We evaluate each method for its estimated recall based on the proportion of words from the computational neologism
sample that is not filtered out. The sample precision is particularly low given the sparsity of neologisms that appear at a specific point in time, so we select the method with the highest precision of 0.2 and an estimated recall of 0.625 to reduce the amount of manual annotation required.
B.2 Dataset Analysis
With the best-performing filter method, we also estimate its recall with a set of 100 handpicked neologisms in our dataset. The estimated recall of our method is 0.55. This estimate is slightly lower than the estimated recall we used with the
15
neologism candidates we computationally gathered, but our collection methods remain consistent.
Analyzing the overlap of words in our dataset with Urban Dictionary by collection method, we find that 44.37% of Reddit neologisms, 38.4% of News neologisms, and 65.25% of neologisms obtained from dictionaries and exclusion lists overlap.

heading: B.3 Emergence Date Labeling for Perplexity-Based Rankings
text: Using Google Trends to record the month and year where word usage spikes, we estimate the date for when a neologism enters the dissemination stage of its lifecycle. We find that 68% of neologisms emerged during 2020-2023, and 17% of all words have no dissemination date or trend line. The remaining words were prevalent before 2020, but a new connotation or usage has recently emerged. These dates are potentially inaccurate as a trend line is collapsed into a single date. Compared to using the entire graph to evaluate long-term word trends when filtering for neologisms, a single date may not perfectly capture neologism growth for words that exhibit a steady rise in growth.
B.4 NEO-BENCH Tasks We sample 750 neologisms from our set of 2505 total neologisms. We stratify the sample based on the 5 collection methodologies used: handpicked, Reddit, News articles, NYT First Said Bot, and Slang dictionaries. We collect 500 neologisms from Reddit and News Articles, 100 from NYT First Said Bot, 100 from slang dictionaries, and 50 from our handpicked set.
We work with 3 in-house native English speakers to construct inputs for Cloze question answering, definition generation, and human-evaluated machine translation. Using a script, annotators are provided with Google Search Descriptions containing neologisms. With these reference sentences, annotators were instructed to create 750 multi-sentence Cloze passages and to create 4 multiple-choice options based on topical relevance to the neologism and 1 distractor answer that is a feasible substitute in the passage. Annotators also constructed 750 sentences containing neologisms and 750 questions asking for the definition of neologisms. From a subsample of 100 neologisms, annotators constructed minimal pair sentences using Google Search references for human-evaluated machine translation. Annotators were instructed to create any feasible substitute word, regardless of topic, to replace the
neologism in the minimal pair sentence. For perplexity rankings, we use 422 Cloze passages that have both a single-word neologism and distractor answer.
We additionally subsample 240 sentences containing neologisms for automatic machine translation. We work with 3 in-house native Chinese speakers to construct 240 reference translations for automatic MT evaluation. Annotators were provided the input sentence, the neologism, and the definition and were instructed to create fluent and accurate translations. If no associated term in Chinese exists, annotators were instructed to paraphrase sentences or use substitute terms with minimal information loss. Annotators were then instructed to retranslate the Chinese sentence back into English to enable further study on accurate translation techniques of neologisms, including paraphrasing the sentence. All 6 annotators we work with have a college-level education.

heading: C Experimental Details
text: All models are evaluated on two NVIDIA A40 GPUs for a single run since models are not finetuned for NEO-BENCH tasks.

heading: C.1 Machine Translation
text: For human-evaluated MT, our error categorization is partially adapted from the widely-used MQM framework (Freitag et al., 2021), which is adjusted based on the pilot studies we conducted on translating sentences containing neologism words. Translation errors are ordered by severity in affecting the understanding of the sentence, and annotators label sentences based on the most severe translation error. For instance, if there are grammatical mistakes and English words in the translation output, the output will be labeled “Copy” since that is the most severe translation error. Table 12 provides example outputs for each translation category.
We crowdsource annotations by screening for 5 fluent Mandarin speakers on Prolific6, a crowdsourcing website. All 5 annotators are fluent in English and Mandarin and reside in the United States and United Kingdom. All of the annotators have a college education and are informed about the nature of the study. Each annotator was given the same set of neologism sentences across all 5 models evaluated to ensure a standard comparison between models. Annotators were provided with
6https://www.prolific.com
16
neologism definitions and were instructed to select the label corresponding to the worst translation error in the sentence and highlight the corresponding spans in the input sentence and MT output. Annotators were also instructed to label the error severity and the confidence in their selection of translation categories on 3-point Likert scales. Finally, annotators marked if the translation error occurred from the neologism or from another portion of the sentence. The average time to annotate 20 minimal pairs was 80 minutes, and each annotator was paid $12.00 an hour, which is on the high end for standard pay on Prolific. We use the Thresh (Heineman et al., 2023) interface for annotating translation sentences, and Figure 13 provides a screenshot of the interface.
Based on our human reference translations of neologisms, we find that translation is difficult. For neologisms representing new concepts, there may be information loss if there is not an associated word in the target language (e.g. boyflux Ñ nonbinary). There are often no exact words associated with neologisms in the target language, so translation requires providing the neologism definition and rephrasing the entire sentence (e.g. fossilflation Ñ fossil fuel price increases). However, since neologisms are created with the same linguistic origins as common words, the novelty of neologisms results in lower MT performance.

heading: C.2 Rank Classification with Perplexity
text: We also tested T5 and Flan-T5 for perplexity ranking and find that Flan-T5 exhibits higher neologism rankings than T5. However, when sorting words by lowest perplexity and filling in the mask, we find that these models produced entirely incoherent sequences, so we do not report these models. Given the computational intensity of evaluating 5,002 sequence perplexities, we only evaluate the base size of models.

heading: C.3 Cloze Question Answering
text: Rank classification is used to select the lowest perplexity answer for BART, T5, and GPT-J. For other models, we shuffle the order of answers and conduct experiments with 5-shot prompting with the following format:
Fill in the blank with the options below: Question: [EXAMPLE CLOZE PASSAGE] a) [EXAMPLE INCORRECT ANSWER] b) [EXAMPLE DISTRACTOR ANSWER]
c) [EXAMPLE INCORRECT ANSWER]
d) [EXAMPLE INCORRECT ANSWER]
e) [EXAMPLE NEOLOGISM ANSWER]
f) [EXAMPLE INCORRECT ANSWER]
Answer: e) [EXAMPLE NEOLOGISM ANSWER]
...
Fill in the blank with the options below:
Question: [TEST CLOZE PASSAGE]
a) [TEST INCORRECT ANSWER]
b) [TEST INCORRECT ANSWER]
c) [TEST NEOLOGISM ANSWER]
d) [TEST DISTRACTOR ANSWER]
e) [EXAMPLE INCORRECT ANSWER]
f) [EXAMPLE INCORRECT ANSWER]
Answer:

heading: C.4 Definition Generation
text: We conduct experiments with 5-shot prompting with the following prompt:
Answer the question.
Question: [EXAMPLE DEFINITION QUESTION]
Answer: [EXAMPLE DEFINITION ANSWER]
...
Answer the question.
Question: [TEST CLOZE PASSAGE]
Question: [TEST DEFINITION QUESTION]
Answer:
One of the paper’s authors manually annotates 100 outputs. We measure the Cohen’s Kappa between automatic GPT-4 and human evaluation, and we obtain an average Cohen’s Kappa of 0.744, indicating high agreement between human judgment and GPT-4.
For automatic evaluation, we additionally use GPT-4 to determine if a correct model definition is better or worse than the reference definition provided by human input. For incorrect answers, we separate between incorrect and omitted generations, which are model outputs that are either left blank or, for GPT models, outputs where the model acknowledges that it does not recognize the neologism.
Table 13 provides the full results of the opendomain question-answering experiments, including the average length of definitions, manual evaluation and model-wise Cohen’s Kappa. While LLaMA-2 70B outperforms GPT-3.5 in Cloze QA, GPT 3.5 produces more correct definitions than LLaMA-270B. Instruction-tuned models produce a higher
17
proportion of correct answers that are deemed better than the human reference sentence. We report the average length of generations for each model and conclude that GPT-4 prefers instruction model outputs because the human reference sentences are on average 19.20 words long, which is more concise compared to the more elaborative responses of instruct-tuned models. We find that 81.07% of preferred answers across all models are longer than the alternative correct answer. Table 14 provides examples of instruct-tuned model responses that are evaluated as better than the human reference and examples of GPT-omitted responses. Even when prompted with shortened answers, instruction-tuned models produce longerform responses. We did not test for restraining the length of the model output as the complexity and reference definition length for each neologism varies extensively. Instruction-tuned models are less likely to produce omitted answers, and only GPT-3.5 and GPT-4 have generations that acknowledge that a term is unrecognized.

heading: D Linguistic Taxonomy
text: Table 8 provides the linguistic breakdown of neologism accuracies for each model in Cloze QA. Table 11 provides the stratified results of definition generation by linguistic type of neologism. Tables 9 and 10 provides the breakdown of automatic machine translation evaluation, and Figure 12 provides the
stratified results of manually labeling translations by linguistic type. We report each category as a proportion to the total amount of neologisms for a certain linguistic type. Model performance discrepancy between the open source models and commercial systems is highest for lexical neologisms. For automatic metrics, lexical neologisms do not yield higher BLEU scores but yield higher COMET and COMETKiwi scores. Morphological neologisms yield 5.3 lower BLEU scores, 3.4% lower COMET scores, and 4.4% lower COMETKiwi scores than lexical neologisms. BLEU scores for semantic neologisms are high as there is high token overlap between different senses of the same word form. However, semantic neologisms often yield similarly low COMET and COMETKiwi scores as Morphological neologisms.
18
19
20
Translation Output Examples
Fire Weather
Input: They will not issue official warnings until fire weather is forecast to occur. (Fire weather is the use of meteorological parameters such as relative humidity, wind speed, mixing heights, and soil moisture to determine whether conditions are favorable for fire growth and smoke dispersion.)
Model Output (Good): 在预测发生火灾天气之前，他们不会发布官方警告。 (Before predicting the occurrence of fire weather, they will not issue an official warning.)
Human Translation: 他们直到预测到火灾天气才会发布官方警告。 (They will only issue official warnings when a fire weather is forecasted.)
Dupe
Input: Discover new affordable dupes for luxury expensive makeup products. (A dupe is an abbreviation of the word "duplicate".)
Model Output (Unnatural): 为奢华昂贵的彩妆产品发现新的负担得起的复制品。 (Find new affordable replicas for luxurious and expensive makeup products.)
Human Translation: 探索奢侈昂贵化妆品的平价替代新品。 (Explore new affordable alternatives for luxurious and expensive makeup products.)
Input: When sleet started falling during Thanksgiving it was officially snowvember. (Snowvember refers to a particular November that experiences a lot of snowfall.)
21
22

heading: Future Work
text: For temporal drift of multilingual language modeling, we leave multilingual neologism collection and temporal drift analysis up for future work. Additionally, NEO-BENCH is static as we collect neologisms from mostly 2020-2023, which will become outdated over time as newer models will be exposed to new language in context. However, the semi-automatic collection methods require minimal human supervision and can be dynamically updated to continuously obtain neologisms. These methods require time-series data of words and online text corpora without needing human-curated information like updated dictionaries to filter words. The time-independent filters can collect recent neologisms without needing the time-consuming, manual curation of temporal splits of textual data. Provided that the Google and Reddit Terms of Service and API access enable NEO-BENCH collection, we intend on periodically updating the set of neologisms in our dataset