Venue: NAACL 2021

# GPT Perdetry Test: Generating new meanings for new words

 Nikolay Malkin\({}^{1}\)

Yale University

New Haven, CT &Sameera Lanka\({}^{2}\)

Microsoft Redmond, WA &Pranav Goel\({}^{3}\)

University of Maryland

College Park, MD

 Sudha Rao\({}^{2}\)

Microsoft Research

Redmond, WA &Nebojsa Jojic\({}^{2}\)

Microsoft Research

Redmond, WA

###### Abstract

Human innovation in language, such as inventing new words, is a challenge for pretrained language models. We assess the ability of one large model, GPT-3, to process new words and decide on their meaning. We create a set of nonce words and prompt GPT-3 to generate their dictionary definitions. We find GPT-3 produces plausible definitions that align with human judgments. Moreover, GPT-3's definitions are sometimes preferred to those invented by humans, signaling its intriguing ability not just to adapt, but to add to the evolving vocabulary of the English language.

## 1 Introduction

Humans are constantly expanding languages with new words. How are artificial language models, which are increasingly deployed 'in the wild', to handle the stream of neologisms that are appearing in slang or on social media Grieve et al. (2018)?

Today's most advanced language models, including GPT-3 Brown et al. (2020), use a subword tokenization of input text, rather than consuming it word by word. This allows them to process words never seen in their training data. For example, the word 'perdetry', which has never been used in English, is treated by GPT-3 as a sequence of two tokens (Fig. 1). The subword tokenization algorithm is designed for text compression and does not respect the natural morpheme boundaries.

We explore GPT-3's understanding of English at the subword level by prompting it to give definitions of nonce words1 (Fig. 1). We find in human studies that not only does GPT-3 generate realistic, original meanings for new words, but its definitions are sometimes preferred to those invented by humans. This finding sheds light on GPT-3's ability to adapt to and even extend a changing vocabulary.

Footnote 1: We use the term ‘nonce word’ for a new word not used in English. It becomes a ‘neologism’ once it acquires a meaning.

While we cannot ascertain GPT-3's exact reasons for assigning meanings to nonce words, our results prove that these reasons are not limited to morphology: many neologisms have no clear roots or derivational origin. The meanings of words may be imported by their phonological qualities - more precisely, their orthographic realizations - or by clues to their membership in certain lexical strata. Thus, at a high level, our findings suggest that GPT-3 has learned not only its world knowledge and capacity for long-range reasoning in text Brown et al. (2020), but also the nuances of etymology and the correspondences of sound and meaning that lie at the very base of language understanding.

Figure 1: **Above:** The neologism generation pipeline.



Below are some pairs of words together with their definitions. The goal is to guess, for each pair, which word goes with which definition. We will show you two options, and you will decide which of them is a better match. The words you'll get are rare, and we do not expect you to know many, or indeed any, of them. Make your best guess. For some pairs, there is no correct answer. We'll show you the expected answers at the end. Do not look up the words while doing the task: we are really interested in your gut feeling, right or wrong._

Footnote 10: [https://www.cds.org/en/](https://www.cds.org/en/)

## 2 Related work

The notion that some subword elements (phones-themes) carry meaning, but, unlike morphemes, do not play a part in word formation has caused controversy in linguistics for over a century [35, 47]. In his seminal work, de Sausure (1916) rejected this notion. Yet, later work identified a large set of English phonesthemes, such as the cluster /gl/ in 'glow', 'glutter', 'gloss', etc. meaning "light"; a notable list was compiled by Marchand (1959a,b). Recent studies found phonosemantic patterns that are common to many languages [8]. In practice, words are even engineered for subconscious reactions: certain sounds in brand names are correlated with associations such as size (of a gadget) or speed (of a courier) [13]. Our study suggests that GPT-3 may understand such patterns as well.

There is a body of work on joint modeling of (orthographic or phonological) word forms and grammatical classes such as noun gender and inflection pattern. In a recent study, Williams et al. (2020) used neural models to measure mutual information between _meanings_ and inflection classes of Czech and German nouns, which, for borrowed words, often depend on the language of origin. It is plausible that GPT-3 implicitly uses likely source languages of nonce words to generate meanings associated with some lexical strata, e.g., abstract nouns from Norman French, concrete nouns from the Germanic substrate, and artificially constructed terms with Greek or Latinate elements. (We direct the interested reader to the lexicon in Appendix C.)

Work on neologisms in NLP includes tracking their emergence and spread on the Internet [12, 24], mapping them into embedding spaces [10, 25, 26], and codifying and predicting stampologies [15, 26]. Others have studied definition generation [24] and the reverse task of mapping definitions to words [13], albeit with pretrained embeddings. Limited examples of a pretrained model's use of nonce words appear in Brown et al. (2020). In this work, we study GPT-3's ability to define words never seen in context.

## 3 Creation of new words and meanings

We trained a LSTM model [15] on a corpus of English words2 with a standard character-level objective, then sampled strings from the LSTM to create nonce words. The words were lemmatized and assigned parts of speech (POS): noun (n.), verb (v.), or adjective (adj.).

Footnote 2: github.com/dwyl/english-words

To produce definitions for these words, we generated text from GPT-3, primed with input in the format "**word (POS.)** -". Usually, GPT-3's outputs had the style of a dictionary definition (Fig. 1). The definitions were filtered by common-sense criteria and lightly edited for consistency, as explained in Appendix A. By this procedure, we obtained 146 word-definition pairs (67 n., 47 v., 32 adj.).

For comparison in our study, we also sampled a set of real but rare English words from a corpus.2 Definitions for these words were taken from a dictionary.3 This resulted in a combined set of 220 words (102 n., 70 v., 48 adj.), with a 2:1 ratio of fake to rare words in each POS. See Appen

Figure 2: The word-definition matching task instructions and a typical question. (GPT-3 happens to have generated the definitions in Option A. In the tests, the assignments of definitions to words in each pair were randomized.)



dices A and C for the full lexicon and generation details, including all points of human input.

## 4 Evaluation

### Machine-generated neologisms

We performed a study in which human subjects were presented with pairs of words of the same part of speech together with their definitions (generated by GPT-3, for fake words, or extracted from the dictionary, for rare words), but not told which definition matches with which word.4 Some questions contained two fake words, some two rare words, and some one fake and one rare word. Users were asked to decide which assignment of definitions to words is a better fit and to rate their confidence (Fig. 2); the choices were converted to a scale of 0 (confident in the incorrect match) to 5 (confident in the correct match). Each user received a random pairing of words, but saw each word exactly once. We collected 65 sets of annotations for each POS, for a total of \(65\cdot\frac{220}{2}=7150\) data points.

Footnote 4: Users were not told that some of the definitions were machine-generated. The full study details are in Appendix B.

Results.Humans prefer the pairing from our lexicon in 68% of cases. The scores by the POS and the kind of pair (fake-fake, fake-rare, or rare-rare) are shown in the top rows of Table 1. GPT-3's definitions align with human judgments far better than random choice (_p_-values below floating-point epsilon). Notably, humans' performance on pairs containing a fake and a rare word was about the same as on pairs of fake words.

Correlation in performance between different parts of the word-definition matching task is high. Considering only the fake-fake pairs, the score (number of correctly matched pairs) on the noun portion of the task is correlated with the score on verb and adjective pairs with Spearman \(\rho\approx 0.42\); a permutation test on rank correlation gives \(p\approx 0.01\). The verb and adjective portions are similarly predictive of the other two (\(p\approx 0.05\) for both). The correlation is even stronger (\(p<0.0001\) for nouns) when all pairs, not just fake-fake, are considered. This indicates that some users can be identified as 'better' at the task, perhaps due to their personal vocabulary, education, or effort. (For example, the average score on the fake-fake noun pairs is 70.7%. However, the average score on fake-fake noun pairs among users who scored above median on the fake-fake adjective pairs is 74.2%.) This is strong evidence that the values in Table 1 would be higher with a better selection of users.

There was significant agreement between annotators. In cases when the same pair of words was shown to two users, the mean difference between the two users' choices on the 0-5 scale was 1.5, and in 61% of cases the two users preferred the same assignment. Remarkably, the latter number is the same for rare-rare, rare-fake, and fake-fake pairs.

It is possible that the subjects knew some of the rare words - and the tables in Appendix C do suggest this. However, assuming that a subject will choose the correct match if they know the meaning of at least one word in a pair, and will do no worse than random guessing on pairs where they know neither word, the last 'human' row is consistent with less than a quarter of the rare words, on average, being known to the subjects.

Likelihood analysis.For each word \(w\) and definition \(d\) in the lexicon (where \(d\) may be the definition of a word different from \(w\)), we compute the likelihood under GPT-3 of the definition \(d\) to follow word \(w\), \(p(d|w)\). For each pair of words \((w_{1},w_{2})\) of the same POS, with definitions \((d_{1},d_{2})\), we compute the difference in log-likelihood between the proper match \((w_{1}-d_{1},w_{2}-d_{2})\) and the inverted assignment \((w_{1}-d_{2},w_{2}-d_{1})\):

\[\mathrm{LLD}(w_{1},w_{2})=\log\frac{p(d_{2}|w_{1})p(d_{1}|w_{2})}{p(d_{1}|w_{ 1})p(d_{2}|w_{2})}.\]

If GPT-3 were to perform the matching task done by our human subjects, it would choose the option with higher total likelihood. In other words, it would prefer the correct pairing if \(\mathrm{LLD}(w_{1},w_{2})\) is negative and the inverted pairing if it is positive.

Assuming that GPT-3 has seen the rare words in training, we expect it to score very well on rare-rare and rare-fake pairs. We also expect it to prefer

\begin{table}
\begin{tabular}{c l|l c c}  & & n. & v. & adj. \\ \hline  & fake-fake & 70.7\% & 59.8\% & 64.3\% \\ human & fake-rare & 72.6 & 60.9 & 65.3 \\  & rare-rare & 79.6 & 65.3 & 69.8 \\ \hline  & fake-fake & 92.4 & 83.2 & 87.3 \\ GPT-3 & fake-rare & 98.5 & 95.5 & 97.3 \\  & rare-rare & 99.4 & 98.4 & 100.0 \\ \end{tabular}
\end{table}
Table 1: Accuracies on the task of matching real and machine-generated words with definitions (Fig. 2), performed by study participants (‘human’) or the language model that created the fake definitions ( ‘GPT-3’).

 

the correct matches for fake-fake pairs, since the definitions of fake words were sampled from the same model of likelihood. Indeed, we see this in the bottom rows of Table 1. GPT-3's imperfect performance on fake-fake pairs is a byproduct of the sampling used in the generation and perhaps of the edits made in postprocessing. To maximize total likelihood of the lexicon, GPT-3 would prefer to enact some post-factum swaps of definitions.

LLD and human confidence.LLD is a good predictor of human judgments: confidence in the correct pairing for fake-fake pairs \((w_{1},w_{2})\) is strongly correlated with LLD\((w_{1},w_{2})\), a rank correlation test giving \(p<0.001\) for all POS.

One may object that this correlation - and indeed much of humans' performance - is due to the presence of simple disambiguating markers: for example, a word with suffix '-ist' is likely to denote a person, while an '-ism' is probably an abstract noun. However, examination of log-likelihood differences shows that this is not the case. We stratify the pairs of fake words by LLD and consider the distribution of humans' confidences for pairs with LLD falling in five ranges: \([-40,-30),[-30,-20),\ldots,[0,10)\). Confidence in the correct matching is inversely correlated with LLD, but humans tend to choose the correct assignment for pairs in all five strata (Table 2). For pairs with LLD in the ranges \([-10,0)\) and \([0,10)\), which form a majority, there tend to be no revealing morphological markers. (Table 2 shows pairs of words with LLD falling into these ranges; Table 7 in the appendix shows more examples.)

Conclusion.Finally, we observe that many of GPT-3's definitions are original: we are not aware of English words that describe the same concepts (see Table 2 and Appendix C). Some of the innovated meanings fill plausible lexical gaps ('drobbler'), while others require a degree of creativity ('subacitide'). This shows that GPT-3 is not simply aligning new words with existing words as in Zalmout et al. (2019), but inventing new meanings.

### Human-generated neologisms

We test GPT-3's ability to define new words on a set of human-proposed neologisms from the Dictionary of Obscure Sorrows.5 Many of these words were created out of real English morphemes. We sampled 20 words from this set, got GPT-3 defini

\begin{table}
\begin{tabular}{c|c l} \hline \multirow{2}{*}{\(-33.3\)} & **parademme :** & a person, party, or body that is not a participant in a dispute or controversy \\  & **calcanism :** & the study of extrusions of lava, volcanic rock, and ash \\ \hline \multirow{2}{*}{\(-23.2\)} & **carburst :** & a person with an abnormally large capacity for consuming substances \\  & **subacitide :** & the longest possible decimal number, continued indefinitely by periodic repetition of 0 \\ \hline \multirow{2}{*}{\(-13.2\)} & **stucenium :** & a little roof, the softit of a cornice, the median part of a pediment \\  & **helliact :** & a person with high level of knowledge and experience in a specific area \\ \hline \multirow{2}{*}{\(-9.4\)} & **prosemer :** & a vessel with rough edges or projecting parts \\  & **drobbler :** & a person who enjoys listening to music \\ \hline \multirow{2}{*}{\(+0.2\)} & **frequency :** & a person who has given up hope for fun \\  & **endosman :** & a performer in a minstrel duo \\ \hline \end{tabular}
\end{table}
Table 2: **Above: Histograms of confidence scores for fake-fake pairs with LLD in each of the five ranges defined in the text. Values of 3, 4, or 5 (orange) indicate preference for the correct matching. Below: Randomly sampled pairs of fake nouns with LLD falling in each of the five ranges. A negative number with large magnitude indicates that GPT-3 – and usually human subjects – strongly prefers this pairing to the reverse. A number close to 0 indices that the pairing and the reverse have similar likelihood.

 

tions for them using the same procedure as above, and also extracted human definitions (Appendix C).

We then ran a study with 25 users, in which each user was given words and both definitions (in random order, without being told how each definition was generated) and asked to pick the better match. The responses were converted to a scale of 0 (human-generated is much better) to 5 (GPT-3-generated is much better). Each user marked their definition preference for all 20 words (Fig. 3).

Results.Remarkably, users preferred GPT-3's definitions in 40% of cases, despite the fact that a human thought up each of these word-meaning pairs. This is not simply the result of random guessing by the workers: the result matrix (Fig. 4) shows a significant amount of structure. There are words on which most users agree that the better definition is the one generated either by the human inventor (top rows) or by GPT-3 (bottom rows).

Users most prefer GPT-3's definition for **backmasking**: _"the act of disguising messages within recordings via sound effects"_ to the human definition _"the instinctive tendency to see someone as you knew them in their youth"_, while the human definition of **lapyear**: _"the age at which you become older than your parents were when you were born"_ is preferred to GPT-3's _"a lazy person; someone of a low-energy lifestyle"_.

User clusters.These human-coined neologisms have a bias towards meanings with an existential slant, which results in additional structure in our results, reflecting the population structure of the subjects. Indeed, some workers prefer human-made definitions and others prefer GPT-3's definitions, which reflect a mixture of meanings seen in a crawl of the Internet.

To analyze the significance such preferences, we perform a randomization test. We define the polarization of a user as the absolute difference between the number of words for which they prefer the human-generated definition and the number for which they prefer GPT-3's definition. The average polarization over users is greater than that seen in 99% of random preference matrices, indicating that there may indeed be two types of users, with different preferences for the types of meanings they see in words.6

Footnote 6: A similar test could be performed taking the confidence into account. Here we define polarization as the absolute difference between a user’s mean confidence and 2.5. In each random sample, we flip a random subset of the entries in the confidence matrix to the opposite preference, while keeping the level of uncertainty the same: \(0\leftrightarrow 5\), \(1\leftrightarrow 4\), \(2\leftrightarrow 3\). This results in a \(p\)-value around 0.04.

## 5 Conclusion

A character-level model of English words composed with GPT-3 is a complete scheme for generating new words and innovative meanings. GPT-3 invents definitions for words it has not seen in training that are seen as reasonable by humans. These results have implications for language models' ability to adapt and even add to an evolving vocabulary. They can inspire future work on machine understanding of new slang, optimization of words and acronyms, creation of fictitious entries, and automatically generating word games.

Figure 4: Human subjects’ preference for GPT-3-generated definitions (bluer) or human-generated definitions (whiter). Each column represents a single user. The rows and columns have been sorted by their means. The full set of definitions can be found in Appendix C.

Figure 3: A typical question in the definition choice task. The instructions were similar to those in Fig. 2; the answer choices were identical. (In this case, Option A was generated by GPT-3, Option B by a human.)





## Ethics statement

The authors see no immediate negative societal consequences arising from this work.

As explained in Appendix B, we followed data privacy and anonymization procedures to the greatest extent possible and fairly compensated human subjects.

## References

* D. E. Blasi, S. Wichmann, H. Hammarstrom, P. F. Stadler, and M. H. Christiansen (2016)Sound-meaning association biases evidenced across thousands of languages. Proceedings of the National Academy of Sciences113 (39), pp. 10818-10823. Cited by: SS4.
* P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov (2017)Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics5, pp. 135-146. External Links: Document, ISSN 0021-9007 Cited by: SS1.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. arXiv:2005:14165. Cited by: SS1.
* Ferdinand de Saussure (1916)Cours de linguistique generale. Payot Paris. Cited by: SS1.
* Jim Feist (2013)"Sound symbolism" in English. Journal of Programatics45 (1), pp. 104-118. Cited by: SS1.
* J. Greive, A. Nini, and D. Guo (2018)Mapping lexical innovation on american social media. Journal of English Linguistics46 (4), pp. 293-319. Cited by: SS1.
* F. Hill, K. Cho, A. Korhonen, and Y. Bengio (2015)Learning to understand phrases by embedding the dictionary. Transactions of the Association for Computational Linguistics4, pp. . Cited by: SS1.
* S. Hochreiter and J. Schmidhuber (1997)Long short-term memory. Neural Computation9 (8), pp. 1735--1780. Cited by: SS1.
* R. Klink (2000)Creating brand names with meaning: The use of sound symbolism. Marketing Letters11, pp. 5-20. Cited by: SS1.
* H. Marchand (1959a)Phonetic symbolism in english word-formation. Indogermanische Forschungen64, pp. 146-168. Cited by: SS1.
* H. Marchand (1959b)Phonetic symbolism in english word-formation. Indogermanische Forschungen64, pp. 256-277. Cited by: SS1.
* G. De Melo (2014)Etymological wordnet: Tracing the history of words. In Proceedings of the 9th Language Resources and Evaluation Conference (LREC), Vol. , pp. 1148-1154. Cited by: SS1.
* T. Noraset, C. Liang, L. A. Birnbaum, and D. C. Downey (2017)Definition modeling: learning to define word embeddings in natural language. In 31st AAAI Conference on Artificial Intelligence (AAAI), pp. 3259-3266. Cited by: SS1.
* J. Nuckolls (1999)The case for sound symbolism. Annual Review of Anthropology28, pp. 225-252. Cited by: SS1.
* J. Pennington, R. Socher, and C. D. Manning (2014)Glove: global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543. Cited by: SS1.
* M. Ryskina, E. Rabinovich, T. Berg-Kirkpatrick, D. Mortensen, and Y. Tsvetkov (2020)Where new words are born: distributional semantic analysis of neologisms and their semantic neighborhoods. In Proceedings of the Society for Computation in Linguistics 2020, New York, New York, New York, New York, New York, New York, USA, pp. 367-376. External Links: ISBN 978-1-4503-319-1, Link, Document Cited by: SS1.
* A. Williams, T. Pimentel, H. Blix, A. D. McCarthy, E. Chodroff, and R. Cotterll (2020)Predicting declension class from form and meaning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 6682-6695. External Links: Link, Document Cited by: SS1.
* W. Wu and D. Yarowsky (2020)Computational etymology and word emergence. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, pp. 3252-3259. External Links: Link, Document Cited by: SS1.
* Q. Wurschinger, M. Fazleh Elahi, D. Zhekova, and H. Schmid (2016)Using the web and social media as corpora for monitoring the spread of neologisms. the case of 'rapefugee', 'rapeugeee', and 'rapugee'. In _Proceedings of the 10th Web as Corpus Workshop_, Berlin. Association for Computational Linguistics. External Links: ISBN 978-1-4503-319-1, Link, Document Cited by: SS1.



Lexicon creation details

The one-layer LSTM that generated nonce words had a 27-symbol alphabet ('a'-'z' and the start/stop token), embedding dimension 8, and hidden size 128. It was trained on ASCII lowercase words with a standard autoregressive modeling objective of maximizing next-token log-likelihood. Samples from this model were generated greedily with temperature 0.5. Samples that contained extremely rare (deemed unpronounceable) consonant clusters, those bearing a resemblance to obscene words, and those attested in English (as checked by a Web search) were removed. The part-of-speech assignments were chosen based on the word form by a fluent English speaker and cross-checked by three other fluent English speakers.

For all experiments with GPT-3 (generation and likelihood scoring), we used the davinci variant of the model, accessed through the free beta version of the API. In the generation of definitions from GPT-3, samples were taken with temperature 1 and truncated at the first line break or period. Because GPT-3 encountered dictionaries in its training data, acceptable samples were obtained for nearly all words. However, the following kinds of outputs were rejected:

* the dwarf sighed".
* one who frequays".
* Definitions containing sexual or violent content.

If three samples for a word failed the filters above, the word was discarded.

To clean the definitions, we removed technical (parenthesized) comments and alternative senses and made minor edits for consistent syntax and punctuation, such as making all definitions of verbs begin with 'to'. A similar processing was applied to the definitions from the Dictionary of Obscure Sorrows: the compendium contains extended explanations for each word, composed of the definition and examples of experiential manifestations of the given sorrow. We present only the definition to users.

For example, here are five samples for the fake word 'perbroil', which appears to trigger questionable content:

**perbroil (v.) -** Desperate to "lance all boils" on the ass of a superior _Rejected for obscene language and for not being a verb phrase. Otherwise, postprocessing would lowercase the 'D'.)_
**perbroil (v.) -** to have sex three times in one night _(Rejected for sexual content.)_
**perbroil (v.) -** To cook and then serve an already dead animal as though it were still alive. Results should include blood oozing from the creature's cherry buthtle. _(Rejected for violent and sexual content. Otherwise, postprocessing would lowercase the 'T'.)_
**perbroil (v.) -** broil on a plancha (type of griddle) _(Rejected for presence of 'broil'. Otherwise, postprocessing would insert 'to'.)_
**perbroil (v.) -** fry something gently on one side only [early 1990s.] _(This would be accepted, and postprocessing would insert 'to' and remove the bracketed comment.)_

This word would have been rejected, since the first three samples were unsuitable, but we would have found a suitable definition after five tries.

The rare words were randomly sampled and agreed upon as little-known by four fluent English speakers with postgraduate education. Words whose dictionary definitions did not meet the above criteria were rejected. Most of the words were known by none of them. 36 of the 74 words do not appear in the top 2 million words of the Common Crawl corpus, according to the GloVe embedding matrix (Pennington et al., 2014), and the median rank of the other words is 656565. The most common word is 'impala' (rank 89578).

## Appendix B Human study details

The studies with human subjects were performed on Amazon Mechanical Turk, with workers from the pool of native English speakers with at least 95% approval rating. Users were paid an average of US$0.07 per pair in the word-definition matching task and $0.10 per word in the definition choice task, equivalent to a wage of about $20/hour at 

the average speed of labeling. All data was collected anonymously and no information was retained other than the answers and time taken to complete the study.

## Appendix C Fake and rare word lexicon

The full set of 220 words used in our definition-matching experiments can be found in Tables 3, 4, and 5. For each word, we computed the average confidence in the correct matching for all pairs containing the word shown to humans in the study. The first column in each table shows the rank of this average confidence (a lower number indicating that the word's definition disambiguates it well).

The set of words from the Dictionary of Obscure Sorrows is shown in Tab. 6.

 



\begin{table}
\begin{tabular}{r|c c l}
85 & allotre & the outstanding opera singer \\
86 & ammors & the killing of a new theater \\
89 & ammors & bad smell, stunk, kenefe \\
69 & bathden & alone, director, or fachaudier imitation \\
91 & bathden & alip of tand that his up to form the surrounding land \\
19 & blemance & a systemeer capable of rebiter \\
76 & bathrenment & happiness or guide in having big plans and an presureureureure \\
25 & calcium & the duty of returns of law, wfecanic rock, and ash \\
40 & cathals & a reorder return \\
26 & carabely & weakness, reformeless, hotels, hotel \\
102 & earbar & a unitle which, e.u.a., jawack \\
46 & carabert & who cofinutes for curplus restaurants over other Polyvensia selling craft \\
41 & carabort & a person with abnormally large capacity for consuming substances \\
27 & careacion & a movement of the macks of the nose \\
39 & earuchock & an cable speed of scared common on the Atlantic coast of Europe \\
49 & document & a chemical tool could orderless afford \\
28 & demonactev & a case of pseudoreseness \\
74 & drawback & a lapse, face area that is blackboard \\
31 & default & nerve prices one enjoy listening to music \\
20 & endomans & a performer in a smartly day \\
60 & flashburn & a move-pipe \\
95 & finodite & a military solute \\
59 & finocone & a piece of cloth \\
35 & frequency & a person who has given up huge fee for \\
8 & garablin & the practice of eating animals \\
11 & horbened & abended herbber \\
73 & hehtilter & a person with high level of knowledge and experience in a specific area \\
98 & headbater & a person who sees no stand about relative, but how so considered a part of the family \\
37 & humidity & illlltemp, previsions \\
42 & harm & warm, gentle Mediterranean wind \\
72 & maldemetry & an altmas boi n Launch patch \\
77 & melanocytes & the study of geometric fingers, egres, for purpose of measuring or computing them \\
78 & manmoters & a physical domain of opposing in the same entity \\
14 & mandalidity & the extreme mental mobility command occurring prior to a homolacid or suicidal act \\
45 & markuski & a female friend: a female companion, an associate, a close friend \\
46 & mercatif & owe above vibration rights \\
101 & menory & owe shortly profits from another’s dishonovty \\
64 & actar & a woman that drifts in a bank attached to a rope, for assessment \\
34 & oppler & giant spider \\
80 & parafermme & a person, party, or body that is not a participant in a dispute or controversy \\
79 & parascalocalized a balance cost of air \\
47 & pathaway & an act of showing off \\
35 & perterey & an instance of inserting work, especially as a bbobby \\
75 & perspectice & a small dome-shaped structure resembling a shinkleon on the top of an ear of corn \\
48 & phillet & a motion entering under old career disposal \\
83 & pesticide & a person who is very special; only in novel situations \\
87 & premote & a person who is not wearing pasting during taking a bicycle \\
87 & premote & a vessel with global edges or projecting parts \\
96 & premoted & a sudden, freeward movement of a body part \\
90 & precommor & a plant dealer \\
97 & sacred & the shall of the back \\
39 & securism & resentence that includes outdoor activities that combat blight \\
2 & spudas & a severe year plan \\
88 & sextordan & a person who is sacrificed to the goods \\
66 & sherthth & a bedgeology of things or persons \\
79 & siteth & a short, vertical trade place above a letter or we used to indicate that it is a Hebrew abbreviation \\
84 & silchick & British theater spring for a comic actor \\
88 & sterlock & a stepping place: a place where there are open few enjoyment or rest \\
84 & settle & the place where they have also been slashed at once \\
85 & stepend & apendon, wardra, porencret, corducter \\
86 & sterecession & a little road, the widit of a cosmetic, the meathian part of a gelement \\
80 & subchaotic & the longest possible channel number, continued indirendingly by periodic repetition of 0 \\
87 & substrate & a base or exterior attribute \\
89 & taggulation & a technology be replaced by a link view or a way of thinking \\
81 & traapsality & excessive pressure for soil \\
86 & watergreen & the conventional perspective of participating in retrograde activities within a group \\
89 & waitfi & an Anglo-Sancecraft person which imites the stress patterns of an earlier poem \\
93 & assupplyk & a stem genome powered by rocket position for recognizing storm \\
100 & annoh & a horse-drawn carriage engine over each floor transportation \\
21 & hold & the rounded seatborne capsule of a cotton Act bar plate \\
92 & chott & a dry salt that stays in the summer between two centres some water in the winter \\
33 & crimployed & any banarer or similar environment \\
22 & codified & an addition argument that explains, modifies, or evokes a will \\
30 & conconcidry & the harmonious reinforcement or their advies parts of a wood of art \\
17 & errolig & a type of said, usually put in sandwicles \\
12 & dictrain & the condition can read out the whole consisting of raw waves \\
16 & strawcr & an allowance profound from an estate for a general support \\
6 & flagellum & a being, whighting membrane-reduced engender need for locomotion or feeding \\
77 & great & the element of poor strength should match metal mas into a mould in existing \\
32 & hoghboim & a source of thread, four or approaches \\
41 & impola & an atmosphere feed for its lighting ability \\
43 & labor & a coscing or unramed stimuli fall down the front of a tobacco, skirt, etc. \\
71 & length & a root, good glance, usually, an opaque level \\
23 & mobile & a non-headoid gel with a moderate left \\
68 & mountain & one of the separines between pure glass in compromise window \\
33 & enphalian & the genetic of balancing organically formed from the adiaxing of alcohol \\
24 & penemosan & a food made must meet such haue noted and heatous into a paste \\
55 & potetatic & a powder's hooked arm, a rather than a pipe \\
67 & poole & a gift, young women, especially seen as poisoniscons \\
15 & private & a swift, why reply, especially one that is missing \\
9 & rhombens & a dry drinking sound before drawing breathing \\
18 & roundle & normal, a step of a lake, a range \\
82 & scourable & an opaque shell of gaze, layer of paint \\
65 & selftorm & any of various vermiculients native to temperate zones \\
85 & steapble & the switch of the soft palate \\
11 & susumegator & a show-off \\
110 & sydarket & a peacehead to pleasure and luxury \\
38 & loctexta & a piece of mouse designed to emphasize the destersity of the performer \\
81 & takhar & a type of goal board for calling goat \\
62 & viiana & a mythological player whose character of \\
16 & wheery & a shift by word to originate behind warways \\
54 & voannow & a wooden statue used as a cell [SS50] \\ \end{tabular}
\end{table}
Table 3: The list of nouns and their definitions (fake words above the line, rare words below).





\begin{table}
\begin{tabular}{r l|l l}
68 & **accent** & to underestimate \\
31 & **batherine** & to talk up, based off, drag on \\
36 & **beckain** & to touch gently \\
41 & **bedeka** & to plant or sew seeks; to place in the ground \\
70 & **bedermae** & to augment a story or allegation with further details \\
25 & **beali** & to miss, be lacking in, to need \\
13 & **bethite** & to be endowed with, possess. \\
7 & **berstrow** & to observe, behold, soe, grape, store, look \\
15 & **cherestimate** & to go in quest of a wife \\
18 & **disaplane** & to become insubordinate or ebrellions \\
32 & **dischall** & to derive the truth of \\
17 & **domiore** & to renew, to recently the earth \\
52 & **drred** & to lie no two minds; to be undiscided \\
11 & **encrimm** & to be in love with ; desire intensely \\
63 & **encreen** & to draw attention to oneself with a display of bevery \\
48 & **enforce** & to begin, launch \\
14 & **excase** & to decrease, fall off, grow smaller, lessen, diminish \\
99 & **follate** & to award, grant \\
65 & **fortine** & to administer physical punishment for a trangression \\
40 & **dopaque** & to break at deform in any part of the body \\
30 & **bound** & to strike risk then one’s foot against the ground \\
53 & **infteen** & to dereven in blood \\
35 & **intensive** & to move or walk with one's eyes closed, guided only by the sense of touch \\
64 & **merzte** & to shoot fire an weapon \\
26 & **medister** & to pay special attention to someone for selfish reasons \\
8 & **malticat** & to be wrong, to be described \\
20 & **midown** & to disshape, squanged \\
67 & **mitreduce** & to hurry \\
9 & **pexcer** & to look on the bright side \\
61 & **phreem** & to be blind to \\
29 & **phrumb** & to mix up, muddle \\
50 & **purpoom** & to crush the game, be on point \\
21 & **regimple** & to work out the details of a complex matter \\
66 & **reprine** & to move about front enforcement, to fluster or to beside \\
57 & **respondor** & to dress in the latest fashion \\
42 & **supanglize** & to spin or what, it turn switity \\
22 & **sederze** & to turn to some \\
34 & **shoutee** & to laugh through half-open teeth \\
49 & **squires** & to be indecisive, to have second thoughts \\
16 & **subcoint** & to peel funds into a single account \\
37 & **subjecte** & to draw an inference; to drive meaning from context \\
44 & **superpedag** & to bring fuel, shed, subfigure \\
10 & **travent** & to cause to deviate at run aside from a come \\
51 & **trionine** & to speak with great clouquence or style \\
60 & **trovel** & to be in a state of confusion or disarray \\
19 & **unimple** & to break up, to dusaint \\
62 & **ward** & to comment or occree someone \\ \hline
47 & **apoptotic** & to remove the final sound or syllable \\
28 & **bediern** & to ramament something in shuoy, tasteless, or paddy finery \\
45 & **chime** & to crack, split, fissure, break \\
38 & **constate** & to relay information in a statement and say whether it is true or false \\
39 & **disimon** & to remove the outlines of; to efface \\
33 & **dicme** & to dress with others; metric, deck \\
2 & **diff** & to remove at rip hat, as in greeting \\
5 & **dense** & to strip the bluepler or skin from \\
55 & **intumeset** & to swell or enlarge \\
12 & **invicylic** & to convert, convince, or win over with flattery or wiles \\
3 & **javaniere** & to preserve wood from decay \\
69 & **luxer** & to beat again the wind \\
54 & **incobrate** & to work diligently by artificial light, to study at night \\
43 & **marage** & to make tough and malleable by means of heat treatment \\
56 & **madler** & to peplexe or bewilder \\
27 & **indifficate** & to make or build a net \\
24 & **oppliance** & to block, to stop up, to obstract \\
58 & **peculative** & to seal or misappappetic money that one has been trusted with \\
6 & **roudle** & to search for something from a frawer, closet, etc. \\
23 & **superpedag** & to do more than God commands \\
46 & **taile** & to maddle, confuse \\
1 & **traipse** & to walk in a messy or unattractively casual way \\
4 & **vacellate** & sway unstedally from one side to the other \\ \end{tabular}
\end{table}
Table 4: The list of verbs and their definitions (fake words above the line, rare words below).





\begin{table}
\begin{tabular}{r|l|p{142.3pt}|l} rank & word & GPT-3 definition & human definition \\ \hline
6 & **altschiner** & the pain of childbirth & weattiness with the same old issues that you’ve always had the active tendency to be someone as yet home in their youth the annotate metarography of being indores during a thoracstorm \\
8 & **dagermebine** & a clumsy feature filter & an imaginary interview with all and elaboration of yourself \\
18 & **ellipsium** & the quality of being only capable of feeling extreme empathy & sadness that you'll never be able to know how happy will turn out the tendency to give up trying to talk about an experience because people are unable to relate to it \\
11 & **eulamis** & a person piece of space rock & the feelings that everything original has already been done \\
15 & **hyshoprug** & someone who hunts into tears easily & the centre, before turn altogether of a place that’s usually blasting with people but is now abandoned and quiet \\
5 & **kenopsia** & the irrational fear of going without pants & the fear that you’ve lived an ordinary life \\
3 & **konkemian** & the organized resistance to the generally computer-based) changes that come with the digital world \\
10 & **hedesism** & excessive taste for symmetry & the desire to be struck by disaster \\
14 & **napyear** & a lazy person; someone of a low-energy lifestyle & the age at which you become older than your parents were when you were born \\
9 & **liberosis** & a slow lapse into partly and a subsequent execute for drinking all day & born \\
12 & **helatica** & trainer of yodelling and headingsings & the desire to care less about things \\
13 & **mimean** & the act of using to large aged when a small one will do & the frustration of knowing how exactly you fit into a stereotype \\
17 & **monchappisa** & the act of remembering a smell (especially an unpleasant odor) years later & the subtle bat persistent helping of being out of place \\
7 & **rochakimian** & a belief that personal poorer increases proportionally with our view height & the awareness of these aren't expecting \\
4 & **silience** & susceptibility to the illusion that notes we here are different from those heard by others & the unnoticed executive confidence that happens around us every day \\
13 & **velliher** & the chin tightness on the brink of darkness: twilight on the universe & the strange wistfulness of used bookstares \\
16 & **zenospace** & a classical name for iodide of potassium & the sense that time keeps going faster \\ \end{tabular}
\end{table}
Table 6: The list of words from the Dictionary of Obscure Sorrows. The first column is the rank of the frequency with which GPT-3’s definition was preferred.

\begin{table}
\begin{tabular}{r l|l|l}
22 & **antistar** & uncerically unpleasant in taste \\
34 & **bedatline** & friendly, genital \\
28 & **brawth** & rough and lacking in thirty \\
47 & **carbood** & keenly cert, attentive, punitaling \\
14 & **characterious** & observing or incoming dioxide delue \\
13 & **chartsii** & short and easy to understand \\
37 & **orientant** & likely to echo offie \\
7 & **climyan** & bluish, zare \\
27 & **connected** & decayed with much use or age: shudby \\
6 & **contetical** & commentre (e.g., tissues that support hearts or lungs) \\
32 & **deployion** & loudly stained or making \\
36 & **halitrotic** & having a tendency to destroy \\
16 & **forest** & arising from a mental vision, having visionary qualities \\
19 & **hardicine** & tending to build, taking immense \\
5 & **immaternal** & having a high risk of immediate death \\
18 & **imperfect** & critical, unfolding \\
48 & **imperfect** & blueswarm, menthistic \\
44 & **metaresoluse** & like to embark another suffering, hardened \\
41 & **ordisiulos** & having a feeling of fooling \\
33 & **params** & unwympathetically registered by other people’s problems \\
20 & **perpregnant** & mutually involved, or or allowing both parties. \\
11 & **perzheimer** & having cells that some can multiply into a variety of cell types \\
23 & **scripize** & charmingly dinchered \\
25 & **sprisitive** & driven by the need for independence \\
42 & **sterrobies** & bad-expected \\
30 & **sterolle** & afflicted with a disease of the bones \\
10 & **strotides** & unfounded by anxiety \\
17 & **supsupplementary** & never requiring reconsideration, settlement, or revision; irrevocable \\
40 & **quarant** & never satisfied \\
24 & **truspect** & sexually mature but not yet adult \\
45 & **tricky** & containing light, of nature of sight \\
4 & **trigropase** & alternating groups of three consecutive names in rhythm \\ \hline
21 & **accessent** & taming sour; really becoming true or acid \\
35 & **cante** & oblique, slanning \\
43 & **costive** & instelly, pneumissions \\
8 & **decotilute** & able to be bold down \\
26 & **fermial** & of or pertaining to the lap \\
9 & **gevm** & moves, stem, surty, sullen \\
33 & **known** & flexible and parsedful in movement \\
15 & **mutatine** & of or relating to early smoking \\
22 & **picxberry** & of returning to fishermen or fishing \\
29 & **rackelberry** & imminal: dissostoke \\
31 & **ready** & rusty and rancid \\
38 & **superpose** & causing or resulting from protection \\
12 & **serotential** & occuring in late summer) \\
1 & **sdology** & dial, defashioned \\
39 & **artidulant** & making a high-pitched chirping, granting, listening, or speaking sound \\
46 & **illicaceous** & of, pertaining to, or resembling the indien \\ \end{tabular}
\end{table}
Table 5: The list of adjectives and their definitions (fake words above the line, rare words below).





\begin{table}
\begin{tabular}{c|r l r l r l r l r l r l r l r l r l r l r l r l r l r l r l r l r l r l r l r l l r l l r l l r l l r l l r l l l r l l l r l l l r l l l l r l l l l r l l l l l r l l l l l l l r