SlowGuess/NAACL_2021_OCR huggingface dataset:
To address this issue, we re-extracted full contexts from MASC. Given a sentence containing a target word in CoINCo, we located the sentence in MASC and used three non-overlapping adjacent MASC regions as our context. As a result, our context contains additional text that was erroneously omitted in CoINCo (including newlines), thereby reducing annotator confusion. The context of the above example in our benchmark is as follows:
"was kindly received," he said. "But an artist who would stay first among his fellows can tell when he begins to fail." "Oh?"
A.2 Retrieving substitutes from THESAURUS
We use thesaurus.com, which is based on Roget's Thesaurus (Kipfer, 2013), as a primary source of context-free substitutes for target words in SWORDS. This resource contains substitutes for 133K words, with an average of four senses per word (median of one—a fraction of words have dozens of senses) and 25 substitutes per senses.
To select substitutes for a particular target word, we gather all substitutes from all senses that have the same part of speech as the original target, in order to disentangle lexical substitution from the task of word sense disambiguation as well as to include
challenging distractors for evaluating models. $^9$  Because lemmas typically contain more substitutes than their associated word forms (e.g. "jump" has more substitutes than "jumping"), we lemmatize target words before querying the thesaurus.
B Crowdsourcing
B.1 Instructions and interface
Figures 2 and 3 show the instructions and interface we used for Amazon Mechanical Turk (AMT) to crowdsourced labels on substitutes. Following the practice of CoINCO, we showed a highlighted target word in the context, which consisted of three sentences to provide sufficient context. We instructed annotators to provide a negative label if the target word is a proper noun or part of a fixed expression or phrase.
Since our Human Intelligence Task (HIT) concerns acceptability judgement as opposed to substitute generation, we made the following modifications to the CoINCO's setup. First, we asked annotators whether they "would actually consider using this substitute" rather than whether the substitute "would not change the meaning" of the target word (Section 3.1). Second, we allowed annotators to abstain if they do not know the definition of the substitute, while asking them to return the HIT if they do not know the definition of the target word or more than three substitutes. Third, we asked annotators to accept a substitute which is "good but not quite grammatically correct." Lastly, we asked annotators to accept the substitute identical to the target word, in attempt to filtering out spammed HITs (Section B.3).
B.2 Setting on Amazon Mechanical Turk
Each HIT contained at most 10 candidate substitutes for a context-target word pair. When there were more than 10 candidate substitutes, we generated multiple HITs by partitioning the candidate substitutes into multiple subsets with potentially different length, using numpy.array_split. We randomized the ordering of substitutes so that each HIT is likely to contain substitutes from both CoINCO and the thesaurus. The following qualification conditions were used to allow experienced annotators to participate in our task:
$^{9}$ In COINCO, contexts, target words, and their part-of-speech tags all come from the Manually Annotated SubCorpus (Ide et al., 2008, 2010)
Note: Even if you have done this task before, please carefully read these updated instructions!
You will be shown a sentence with a highlighted word and a list of possible substitutes for the highlighted word. For each substitute, please select one of the following:
I would actually consider using this substitute as the author of the original sentence.
I would not consider using this substitute as the author of the original sentence.
I do not know the definition of the substitute.
Please use your best judgement when labeling each substitute. We are trying to capture what makes one word a good, natural substitute for another. Detailed criteria are listed below:
- Mark  $\checkmark$  if the substitute is good but not quite grammatically correct (see examples below).
- Ignoring grammar, please keep a high bar for marking a substitute as  $\checkmark$ .
- You may end up marking many substitutes as  $x$ . This is fine and expected.
- Mark  $\checkmark$  if the highlighted word itself appears as a substitute. Not doing so will make your HIT rejected.
- Mark  $x$  if the highlighted word is a proper noun or part of a fixed expression / phrase (see examples below).
- Return the HIT if you don't know the definition of the highlighted word or more than three substitutes.
Click to show examples
Sentence	Substitute	Label	Reason
The sky was gray.	overcast	✓	Good substitute
The cow jumped over the moon.	leap	✓	Ungrammatical ("leap" vs "lept") but good otherwise
They munched on the snacks.	ate	✓	Ungrammatical ("ate on") but good otherwise
She was fascinated by science.	sports	x	Different meaning
He said he was hungry.	pronounced	x	Not natural
We drank coffee.	liquid	x	Too general
He hit the sack.	punched	x	Highlighted word is part of an expression
She drank green tea.	emerald	x	Highlighted word is part of a phrase
The book was excellent.	gazint	?	Unknown definition
Figure 2: Our instructions ask annotators to accept a substitute if they would actually consider using the substitute as the author of the context, reject if not, or abstain if they do not know the definition of the substitutes. Examples are provided for reference, when annotators click the "click to show examples" buttons.
- HIT Approval Rate (\%) for all Requesters' HITs is greater than 98.
- Location is the United States.
- Number of HITs Approved is greater than 10,000.
Our target hourly wage for annotators was  $15. Based on our in-person pilot study with five native English speakers, we approximated the time per assignment (labeling at most twelve substitutes) to be 25 seconds. Then, we assumed that it may take 1-2x longer for crowd workers to complete the assignments and decided on the compensation of$ 0.10 to fall into the range of $7.25 (US federal minimum wage) and $15 per hour, which corre
sponds to 50 seconds and 24 seconds per assignment, respectively.
It may be surprising that our assignments only take 25 seconds on average, though there are several reasons why this is the case: (1) In general, making binary judgements about substitute words takes very little time for native speakers. (2) Annotators only have to read the target sentence once to provide judgements for all substitutes in an assignment. (3) Annotators usually do not need to read the two additional context sentences to make judgements. (4) Annotators can almost instantly judge two control substitutes (Section B.3), and are therefore only realistically evaluating at most ten candidates per assignment.
After this, we went up a flight of stairs and past a row of doors barred on the outside. "Excuse me," I said, ignoring Nepthys' warning look, "but may I ask what is kept in these rooms?" The woman laughed.
(original word: said)
voice	○✓	○×	○?
give voice	○✓	○×	○?
declare	○✓	○×	○?
speak	○✓	○×	○?
utter	○✓	○×	○?
communicate	○✓	○×	○?
state	○✓	○×	○?
unconveyed	○✓	○×	○?
pronounce	○✓	○×	○?
said	○✓	○×	○?
B.3 Filtering spam
In order to filter out work done by spammers, we included two additional control candidate substitutes in every HIT: the original target word and a randomly chosen dictionary word. Annotators were instructed to accept the substitute identical to the target word and were expected to either reject or abstain on the random word. We used these control substitutes to filter out spammed HITs. Concretely, we filtered out all the HITs with any wrong label assigned to the control substitutes as well as HITs completed by annotators whose overall accuracy on control substitutes across HITs was less than  $90\%$ . Then, we re-collected labels on these filtered HITs for Step 2 and Step 3.
C Data analysis
C.1 Annotator agreement
McCarthy and Navigli (2007) introduced two inter-annotator agreement measures, which assumes that a fixed number of annotators generate a set of substitutes for every context-target word pair. However, these measures are not designed for the case when there is only one collective set of substitutes for each context-target word pair, and every context-target word pair is labeled by various combinations
Figure 3: Our Human Intelligence Task (HIT) contains a highlighted target word in the context, which consisted of three sentences to provide sufficient context. For each HIT, we show at most 10 candidate substitutes for the target word and two control substitutes for filtering.
Figure 4: Annotator agreement between SWORDS and  $k$  additional annotators measured by rank-biased overlap (Webber et al., 2010). The standard deviations from 100 simulations are shown as error bars. We observe quite low RBO for  $k < 3$  and diminishing returns as  $k$  grows. This indicates that there is wide variation in opinions, and it is necessary to use sufficiently large  $k$  to capture the distribution.
of annotators.
Instead, we compute correlation between the two ranked lists using Rank-Biased Overlap (RBO) (Webber et al., 2010), which handles non-conjoint lists and weights high ranks more heavily than low unlike other common rank similarity measures such as Kendall's  $\tau$  and Spearman's  $\rho$ . With additionally collected 10 labels (Section 4.4), we computed RBO by comparing the ranked list of substitutes
10 McCarthy and Navigli (2007) had 5 annotators from the UK, where each of them annotated the entire dataset.
Model	GAP
HUMANS*	67.7
BERT	57.1
BERT-LS	53.8
BERT-LS w/o sp	53.0
BERT-LS w/o sv	51.9
GLOVE	49.8
RANDOM	32.6
Table 6: Evaluation of models on SWORDS in the ranking setting. Here, systems provide a score for every candidate in the benchmark. *Computed on a subset of the test data.
derived from the data to that of SWORDS and simulate the effect of having  $k$  annotators by sampling  $k$  labels per substitute without replacement a total of 100 times.
Figure 4 shows the correlation between SWORDS and  $k$  additional human annotators. We observe quite low RBO for  $k < 3$  and diminishing returns as  $k$  grows. Based on this observation, we argue that there is wide variation in opinions and it is necessary to use sufficiently large  $k$  to capture the distribution.
D Model evaluation
D.1 Ranking setting
As opposed to the generative setting where candidates must generate and rank substitutes, in the (easier) ranking setting, systems are given all candidate substitutes from the benchmark (including those marked as unacceptable) and tasked with ranking them by their appropriateness.
To evaluate ranking models, we adopt standard practice and report generalized average precision (GAP) (Kishida, 2005). GAP is similar to mean average precision, but assigns more credit for systems which produce substitutes that have higher scores in the reference list. Considering that our data collection procedure results in reference scores which correspond more to substitute appropriateness than ease-of-recollection, GAP is aligned with our high-level goals.
D.3 Baselines
We evaluate contextual embeddings from BERT and word embeddings from GLOVE (Pennington et al.,
2014), using cosine similarity of the target and substitute embeddings as the score. To compute the contextual embedding of a target or substitute with BERT, we mean pool contextual embeddings of its constituent word pieces. Because GLOVE discards contextual information, we expect it to perform worse than BERT, and is mainly used to assist interpretation of GAP scores. In the ranking setting, we are unable to evaluate GPT-3 and WORDTUNE, as we interface with these systems via an API which provides limited access to the underlying models. We report GAP scores in Table 6.
D.4 Results
We posit that contextual word embedding models should be invariant to contextual synonymy—they should embed acceptable substitutes nearby to one another. Hence, the SWORDS ranking setting may offer a useful perspective for evaluating this aspect of such models. In the ranking setting, our best contextual embedding model (BERT) achieves a GAP score of 57.1. While BERT outperforms a simple context-free baseline (GLOVE), it falls short of the 67.7 GAP score achieved by HUMANS. We interpret this as evidence that contextual embedding models have room to improve before attaining the aforementioned invariance.
D.5 Lexical substitution as natural language generation
GPT-3 is a language model which generates text in a left-to-right order, and is not designed specifically for the task of lexical substitution. To use GPT-3 to perform lexical substitution, we formulate the task in terms of natural language generation, and use in-context learning as described in (Brown et al., 2020). Specifically, we draw examples at random from the SWORDS development set to construct triplets of text consisting of (context with target word indicated using asterisks, natural language query, comma-separated list of all substitutes with score  $>0\%$  in descending score order) as follows:
Phone calls were monitored. An undercover force of Manhattan Project security agents \*\*infiltrated\*\* the base and bars in the little town of Wendover (population 103) to spy on airmen. Karnes knew the 509th was preparing for a special bombing mission, but he had no idea what kind of bombs were involved.
Q: What are appropriate substitutes for  $^{**}$ infiltrated\*\* in the above text?
A: penetrate, swarm, break into, infest, overtake, encompass, raid, breach
We construct as many of these priming triplets as as can fit in GPT-3's 2048-token context (roughly 12 examples on average), leaving enough room for a test example formatted the same way except without the list of answers. Then, we query the 175B-parameter davinci configuration of GPT-3 to generate a text continuation with up to 128 tokens. Finally, we parse the generated text from GPT-3, using its natural language ordering as the ordering for evaluation.
In an initial pilot study on a random split of our development set, we selected the sampling hyperparameters for GPT-3 as temperature 0, presence_penalty 0.5, and frequency_penalty 0, among possible candidates of  $\{0,1\}$  and  $\{0,0.5,1.0\}$ , and  $\{0,0.5,1.0\}$ , respectively. We used a grid search (18 runs) to select values based on highest  $F_{c}^{10}$ .
E Additional evaluation results
We include additional results from our evaluation. In Table 7, we break down  $F^{10}$  from Table 4 into  $P^{10}$  and  $R^{10}$ . In Table 8, we report performance of all generative baselines on traditional metrics for lexical substitution.
Model	Lenient	Strict
\(P_{c}^{10}\)	\(R_{c}^{10}\)	\(P^{10}\)	\(R^{10}\)	\(P_{c}^{10}\)	\(R_{c}^{10}\)	\(P^{10}\)	\(R^{10}\)
HUMANS*	75.3	77.5	44.9	60.6	-	-	-	-
COINCO	70.5	57.5	24.4	59.3	-	-	-	-
THESAURUS†	58.3	60.4	16.3	50.2	-	-	-	-
THESAURUS	49.4	51.1	11.6	35.9	-	-	-	-
WORDTUNE†	75.6	30.7	32.1	38.8	39.4	28.7	17.1	37.2
GPT-3	75.0	34.3	30.3	41.3	36.9	32.1	15.4	39.9
WORDTUNE	75.4	30.6	31.9	38.5	38.1	27.7	16.6	36.1
GPT-3†	75.2	34.3	30.1	41.0	35.3	30.7	15.1	39.1
BERT-K†	67.4	44.8	24.2	48.0	28.2	30.1	12.1	38.6
BERT-LS	67.2	44.2	23.9	46.8	25.3	27.0	11.0	35.0
BERT-K	66.6	44.3	23.7	46.9	23.0	24.5	10.0	31.8
BERT-M†	69.2	35.8	25.4	39.1	23.6	25.2	10.3	32.7
BERT-M	69.0	35.7	25.4	39.1	15.6	16.6	6.8	21.7
Table 7: Expansion on the results from Table 4, breaking down F-measures by precision and recall. *Computed on a subset of the test data. †Reranked by our best ranking model (BERT).
Model	BEST	BEST-M	OOT	OOT-M	P1
ORACLE	7.8	98.1	73.6	99.4	100.0
HUMANS*	3.9	29.8	56.1	93.0	91.0
COINCO	6.3	28.6	41.6	71.0	87.9
THESAURUS†	2.3	18.2	37.7	51.8	79.6
THESAURUS	2.3	7.3	30.3	37.9	59.1
WORDTUNE†	2.8	22.8	20.0	42.2	88.1
GPT-3	2.6	24.1	22.9	45.6	86.7
WORDTUNE	2.8	17.6	19.4	41.3	83.9
GPT-3†	2.6	19.8	22.1	44.3	86.2
BERT-K†	0.7	18.3	22.9	47.1	84.8
BERT-LS	0.7	17.9	20.6	41.7	81.2
BERT-K	0.7	14.9	18.8	40.4	77.7
BERT-M†	0.6	18.9	19.2	38.5	83.0
BERT-M	0.6	8.0	13.0	27.5	76.0
Table 8: Evaluation of models on SWORD in the generative setting using traditional evaluation metrics. We also include numbers for an ORACLE, as (unlike for  $F^{10}$  and GAP), the oracle does not achieve a score of 100. *Computed on a subset of the test data. †Reranked by our best ranking model (BERT).