SlowGuess/NAACL_2021_OCR huggingface dataset:
For all experiments with GPT-3 (generation and likelihood scoring), we used the davinci variant of the model, accessed through the free beta version of the API. In the generation of definitions from GPT-3, samples were taken with temperature 1 and truncated at the first line break or period. Because GPT-3 encountered dictionaries in its training data, acceptable samples were obtained for nearly all words. However, the following kinds of outputs were rejected:
- Outputs that were not in the form of dictionary definitions (were not lists of items that can be parsed as syntactic constituents of the appropriate type), e.g., "sacrel (n.) - the dwarf sighed".
- Definitions with words that differed from the word being defined by an affix, e.g., "frequayer (n.) - one who frequays".
- Definitions containing sexual or violent content.
If three samples for a word failed the filters above, the word was discarded.
To clean the definitions, we removed technical (parenthesized) comments and alternative senses and made minor edits for consistent syntax and punctuation, such as making all definitions of verbs begin with 'to'. A similar processing was applied to the definitions from the Dictionary of Obscure Sorrows: the compendium contains extended explanations for each word, composed of the definition and examples of experiential manifestations of the given sorrow. We present only the definition to users.
For example, here are five samples for the fake word 'perbroil', which appears to trigger questionable content:
(1) perbroil (v.) - Desperate to "lance all boils" on the ass of a superior (Rejected for obscene language and for not being a verb phrase. Otherwise, postprocessing would lowercase the 'D').
(2) perbroil (v.) - to have sex three times in one night (Rejected for sexual content.)
(3) perbroil (v.) - To cook and then serve an already dead animal as though it were still alive. Results should include blood oozing from the creature's cherry butthole. (Rejected for violent and sexual content. Otherwise, postprocessing would lowercase the 'T').
(4) perbroil (v.) - broil on a plancha (type of griddle) (Rejected for presence of 'broil'. Otherwise, postprocessing would insert 'to').
(5) perbroil (v.) - fry something gently on one side only [early 1990s.] (This would be accepted, and postprocessing would insert 'to' and remove the bracketed comment.)
This word would have been rejected, since the first three samples were unsuitable, but we would have found a suitable definition after five tries.
The rare words were randomly sampled and agreed upon as little-known by four fluent English speakers with postgraduate education. Words whose dictionary definitions did not meet the above criteria were rejected. Most of the words were known by none of them. 36 of the 74 words do not appear in the top 2 million words of the Common Crawl corpus, according to the GloVe embedding matrix (Pennington et al., 2014), and the median rank of the other words is 656565. The most common word is 'impala' (rank 89578).
B Human study details
The studies with human subjects were performed on Amazon Mechanical Turk, with workers from the pool of native English speakers with at least  $95\%$  approval rating. Users were paid an average of US\ $0.07 per pair in the word-definition matching task and \$ 0.10 per word in the definition choice task, equivalent to a wage of about \$20/hour at
the average speed of labeling. All data was collected anonymously and no information was retained other than the answers and time taken to complete the study.
C Fake and rare word lexicon
The full set of 220 words used in our definition-matching experiments can be found in Tables 3, 4, and 5. For each word, we computed the average confidence in the correct matching for all pairs containing the word shown to humans in the study. The first column in each table shows the rank of this average confidence (a lower number indicating that the word's definition disambiguates it well).
The set of words from the Dictionary of Obscure Sorrows is shown in Tab. 6.
Table 3: The list of nouns and their definitions (fake words above the line, rare words below).
Table 4: The list of verbs and their definitions (fake words above the line, rare words below).
22	antisard	unrelievedly unpleasant in taste
34	bedduline	friendly, genial
28	brawly	rough and lacking in clarity
47	carabodent	keenly careful, attentive, painstaking
14	chariocious	deserving or inciting ridicule
13	chartish	short and easy to understand
37	chorient	likely to catch on fire
7	climyrean	bluish, azure
27	concested	decayed with much use or age: shabby
6	ontical	connective (e.g., tissues that support hearts or lungs)
32	despious	loudly satirical or mocking
36	finitrotic	having a tendency to destroy
16	foreal	arising from a mental vision, having visionary qualities
19	harbolemic	tending to babble; talking nonsense
5	immortaunt	having a high risk of immediate death
18	imperful	critical, unyielding
48	importetical	lukewarm, unenthusiastic
44	memolessive	able to endure much suffering, hardened
41	oristitous	having a feeling of foreboding
33	paranory	unsynthetically aggrieved by other people's problems
20	perpagant	mutually involved; of or involving both parties.
11	reprimonic	having cells that some can multiply into a variety of cell types
23	scrappic	charmingly disheveled
25	spriative	driven by the need for independence
42	stereous	bad-tempered
30	storile	afflicted with a disease of the bones
10	strideless	untouched by anxiety
17	suspemptory	never requiring reconsideration, settlement, or revision; irrevocable
40	syncrant	never satisfied
24	trapecent	sexually mature but not yet adult
45	tricy	containing light, of the nature of light
4	trigopose	alternating groups of three consecutive notes in rhythm
21	aceous	turning sour; readily becoming tart or acid
35	cantic	oblique, slanting
43	costive	miserly, parsimonious
8	decoctible	able to be boiled down
26	gremial	of or pertaining to the lap
9	grum	morose, stern, surly, sullen
3	lissom	flexible and graceful in movement
15	matutine	of or relating to early morning
2	piscatory	of or pertaining to fishermen or fishing
29	rakehell	immoral; dissolute
31	reasy	rusty and rancid
38	saprogenic	causing or resulting from putrefaction
12	serotinal	occurring in late summer)
1	stodgy	dull, old-fashioned
39	stridulant	making a high-pitched chirping, grating, hissing, or squeaking sound
46	tiliaceous	of, pertaining to, or resembling the linden
Table 5: The list of adjectives and their definitions (fake words above the line, rare words below).
rank	word	GPT-3 definition	human definition
6	altschmerz	the pain of childbirth	weariness with the same old issues that you've always had
20	backmasking	the act of disguising messages within recordings via sound effects	the instinctive tendency to see someone as you knew them in their youth
8	chrysalism	a state of suspended development	the amniotic tranquility of being indoors during a thunderstorm
1	daguerreoluge	a clumsy fortune teller	an imaginary interview with an old photograph of yourself
18	ellipsis	the quality of being only capable of feeling extreme empathy	sadness that you'll never be able to know how history will turn out
11	exulansis	a pointy piece of space rock	the tendency to give up trying to talk about an experience because people are unable to relate to it
15	hiybbrpqag	someone who bursts into tears easily	the feeling that everything original has already been done
5	kenopsia	the creepy feeling that something is wrong, when everything is actually fine	the eerie, forlorn atmosphere of a place that's usually bustling with people but is now abandoned and quiet
3	koinophobia	the irrational fear of going without pants	the fear that you've lived an ordinary life
19	kudoclasm	the organized resistance to the (generally computer-based) changes that come with the digital world	the destruction of lifelong dreams
10	lachesism	excessive taste for symmetry	the desire to be struck by disaster
14	lapyear	a lazy person; someone of a low-energy lifestyle	the age at which you become older than your parents were when you were born
9	liberosis	a slow lapse into apathy and a subsequent excuse for drinking all day	the desire to care less about things
12	lutalica	mixture of yodelling and headbanging	the part of your identity that doesn't fit into categories
2	mimeia	the act of using too large a word when a small one will do	the frustration of knowing how easily you fit into a stereotype
17	monachopsis	the act of remembering a smell (especially an unpleasant one) years later	the subtle but persistent feeling of being out of place
7	occhiolism	a belief that personal power increases proportionally with one's height	the awareness of the smallness of one's perspective
4	silience	susceptibility to the illusion that noises we hear are different from those heard by others	the unnoticed creative excellence that happens around us every day
13	vellichor	the dim lightness on the brink of darkness; twilight on the universe	the strange wistfulness of used bookstores
16	zenosyne	a classical name for iodide of potassium	the sense that time keeps going faster
Table 6: The list of words from the Dictionary of Obscure Sorrows. The first column is the rank of the frequency with which GPT-3's definition was preferred.
LLD	word pair
-4.24	bellamen :	a strip of land that juts up from the surrounding land
blossard :	a garment made of cloth or leather
-6.7	stucenium :	a little roof, the soffit of a cornice, the median part of a pediment
persecole :	a small dome-shaped structure resembling a thimble on the top of an ear of corn
-8.0	flambuna :	a stove-pipe
carCNTion :	a movement of the muscles of the nose
+4.9	bellamen :	a strip of land that juts up from the surrounding land
silicily :	British theater jargon for a comic actor
+0.6	bellamen :	a strip of land that juts up from the surrounding land
parascound :	a shallow canoe or raft
+0.9	parascound :	a shallow canoe or raft
cantah :	a reindeer parka
-0.2	shoutze :	to laugh through half-open teeth
batherize :	to talk up, boast of, brag on
-5.7	encreen :	to draw attention to oneself with a display of bravery
bedrame :	to augment a story or allegation with further details
-4.3	batherize :	to talk up, boast of, brag on
bedeak :	to plant or sow seeds; to place in the ground
+1.0	influen :	to drench in blood
batherize :	to talk up, boast of, brag on
+2.0	disapplase :	to become insubordinate or rebellious
dreed :	to be in two minds; to be undecided
+0.2	beckain :	to touch gently
account :	to underestimate
-5.4	importical :	lukewarm, unenthusiastic
spriative :	driven by the need for independence
-4.7	despious :	loudly satirical or mocking
bedduline :	friendly, genial
-3.7	perpagant :	mutually involved; of or involving both parties.
carabodent :	keenly careful, attentive, painstaking
+1.7	tricy :	containing light, of the nature of light
despious :	loudly satirical or mocking
+4.5	foreal :	arising from a mental vision, having visionary qualities
paranory :	unsympathetically aggrieved by other people's problems
+6.4	sterebous :	bad-tempered
despious :	loudly satirical or mocking
Table 7: Additional random samples of word pairs with LLD between -10 and 10.