SlowGuess/NAACL_2021_OCR huggingface dataset:
SWORDS  $\times$ : A Benchmark for Lexical Substitution with Improved Data Coverage and Quality
Mina Lee*
Chris Donahue*
Robin Jia
Alexander Iyabor
{minalee, cdonahue, robinjia, aiyabor, pliang}@cs.stanford.edu
We release a new benchmark for lexical substitution, the task of finding appropriate substitutes for a target word in a context. For writing, lexical substitution systems can assist humans by suggesting words that humans cannot easily think of. However, existing benchmarks depend on human recall as the only source of data, and therefore lack coverage of the substitutes that would be most helpful to humans. Furthermore, annotators often provide substitutes of low quality, which are not actually appropriate in the given context. We collect higher-coverage and higher-quality data by framing lexical substitution as a classification problem, guided by the intuition that it is easier for humans to judge the appropriateness of candidate substitutes than conjure them from memory. To this end, we use a context-free thesaurus to produce candidates and rely on human judgement to determine contextual appropriateness. Compared to the previous largest benchmark, our SWORDS benchmark has 3x as many substitutes per target word for the same level of quality, and its substitutes are 1.4x more appropriate (based on human judgement) for the same number of substitutes.
Imagine you are writing the message "I read an amazing paper today" to a colleague, but you want to choose a more descriptive adjective to replace "amazing." At first you might think of substitutes like "awesome" and "great," but feel that these are also unsatisfactory. You turn to a thesaurus for inspiration, but among reasonable alternatives like "incredible" and "fascinating" are words like "prodigious" which do not quite fit in your context. Ultimately, you choose to go with "fascinating," but reaching this decision required a non-trivial amount of time and effort.
Research on lexical substitution (McCarthy, 2002; McCarthy and Navigli, 2007; Erk and Padó, 2008; Szarvas et al., 2013; Kremer et al., 2014; Melamud et al., 2015; Hintz and Biemann, 2016; Zhou et al., 2019; Arefyev et al., 2020) considers the task of replacing a target word in context with appropriate substitutes. There are two widely-used English benchmarks for this task: SEMEVAL (McCarthy and Navigli, 2007) and COINCO (Kremer et al., 2014). For both benchmarks, data was collected by asking human annotators to think of substitutes from memory. Because lexical substitution was originally proposed as a means for evaluating word sense disambiguation systems (McCarthy, 2002), this data collection strategy was designed to avoid a bias towards any particular word sense inventory.
In this work, we consider a different use case for lexical substitution: writing assistance. For this use case, we are interested in evaluating a system's ability to produce appropriate substitutes that are likely to be difficult for humans to think of. We show that the data collection strategy used in past benchmarks yields low coverage of such uncommon substitutes—for our previous example, they might contain words like "awesome" and "great," but miss words like "incredible" and "fascinating." Furthermore, we observe that these benchmarks have low quality, containing words like "fun," which are easy to think of, but not quite appropriate in context.
We present SWORDS—the Stanford Word Substitution Benchmark—an English lexical substitution benchmark that raises the bar for both coverage and quality (Table 1). We collect SWORDS by asking human annotators to judge whether a given candidate word is an appropriate substitute for a target word in context, following the intuition that judging a given substitute is easier than producing that same substitute from memory. To bootstrap a set of candidates for humans to annotate, we
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4362-4379 June 6-11, 2021. ©2021 Association for Computational Linguistics
Context	My favorite thing about her is her straightforward honesty.
CoINCo	candid (3), artless (1), blunt (1), complete (1), direct (1), forthright (1), frank (1), outspoken (1), plainspon-ken (1), truthful (1), unreserved (1)
Thesaurus	honest, simple, easy, candid, forthright, genuine, sincere, truthful, unequivocal, unambiguous, clear-cut, clear, uncomplicated, aboveboard, barefaced, direct, frank, guileless, honorable, just, laid on the line, ... 33 more
SWORDS	sincere (80%), genuine (80%), frank (70%), candid (70%), direct (70%), forthright (70%), uncom-plicated (60%), up front (60%), clear-cut (60%), clear (60%), plainspon (60%), complete (50%), straight-arrow (50%), honest (50%), open (50%), blunt (50%), outspoken (50%), truthful (50%), plain-dealing (40%), undisguised (40%), unvarnished (40%), unreserved (40%), barefaced (40%), unequivocal (30%), upright (30%), simple (30%), veracious (30%), unconcealed (30%), like it is (30%), square-shooting (20%), upstanding (20%), undissembled (20%), manifest (20%), unambiguous (20%), pretense-less (20%), level (10%), laid on the line (10%), honorable (10%), guileless (10%), ... 20 more with 0%
Table 1: We consider lexical substitution, the task of finding appropriate substitutes for a target word in context. In CoINCo (the previous largest benchmark), humans are asked to think of substitutes from memory and result in low coverage (the number of annotators who produced each substitute is shown in parentheses; out of six annotators). On the other hand, looking up the target word in a thesaurus has higher coverage, but low quality, because it does not consider the context. In SWORDS, we combine the best of both worlds and provide a list of substitutes that has high coverage and high quality, along with fine-grained scores for each substitute (shown in parentheses). Substitutes with scores greater than  $50\%$  from SWORDS are bolded.
look up target words in an existing context-free thesaurus. $^{1}$  Because a thesaurus might miss substitutes that would not typically be synonymous with the target word outside of the provided context (e.g. "thought-provoking" for "amazing"), we also include human-proposed candidates from the previous COINCO benchmark.
Determining whether a substitute is appropriate is intrinsically subjective. To address this, we collect binary labels from up to ten annotators for each substitute, inducing a score for each substitute. In CoINCO, analogous scores are derived from the number of independent annotators who thought of a substitute—hence, as we will show in Section 4, these scores tend to correspond more to ease-of-recollection than appropriateness. In contrast, scores from SWORDS correspond to appropriateness, and also allow us to explicitly trade off coverage and quality, permitting more nuanced evaluation. Our analysis shows that compared to CoINCO, SWORDS has  $3\mathrm{x}$  more substitutes per target word for the same level of quality, and its substitutes are  $1.4\mathrm{x}$  more appropriate based on scores for the same number of substitutes.
We demonstrate that SWORDS is a challenging benchmark by evaluating state-of-the-art lexical substitution systems and large-scale, pre-trained language models including systems based on BERT (Devlin et al., 2019; Zhou et al., 2019) and GPT-3 (Brown et al., 2020). In our evaluation, we find
that humans substantially outperform all existing systems, suggesting that lexical substitution can be used as a downstream language understanding task for pre-trained models. We release SWORDS publicly as a benchmark for lexical substitution, coupled with a Python library that includes previous benchmarks in a common format, standardized evaluation scripts for prescribed metrics, and reproducible re-implementations of several baselines. $^2$
We describe lexical substitution and briefly introduce two widely-used benchmarks: SEMEVAL (McCarthy and Navigli, 2007), the first benchmark, and CoINCo (Kremer et al., 2014), the largest existing benchmark. For a survey of other benchmarks, we refer readers to Kremer et al. (2014), Hintz and Biemann (2016), and Miller (2016).
Lexical substitution. Lexical substitution is the task of generating a list of substitutes  $w'$  that can replace a given target word  $w$  in a given context  $c$  (McCarthy, 2002):
$$
\left(\text {c o n t e x t} c, \text {t a r g e t} w\right)\rightarrow \left[ \text {s u b s t i t u t e} w ^ {\prime} \right].
$$
The context  $c$  is one or more sentences where the target word  $w$  is situated. The target word  $w$  is one word in the context, which is either manually chosen by humans (McCarthy and Navigli, 2007) or
<sup>1</sup>Note that our use of a thesaurus makes SWORDS less appropriate for the original use case for lexical substitution: evaluating word sense disambiguation systems.
$^{2}$ SWORDS: github.com/p-lambda/swords  
All experiments reproducible on the CodaLab platform:  
worksheets.codalab.org/worksheets/0xc924392d555f4b4fbee47be92e3daa0b
automatically selected based on the part-of-speech of the target word (Kremer et al., 2014). The substitute  $w'$  can be a word or phrase. Note that the task of lexical substitution does not consider inflection and does not involve grammar correction; all benchmarks contain lemmas as substitutes (e.g. "run" instead of "ran").
SEMEVAL. The first lexical substitution benchmark, SEMEVAL-2007 Task 10 (McCarthy and Navigli, 2007), contains 201 manually chosen target words. For each target word, 10 sentences were chosen as contexts (mostly at random, but in part by hand) from the English Internet Corpus (Sharoff, 2006) and presented to five human annotators. The five annotators were instructed to produce up to three substitutes from memory as a replacement for the target word in context that "preserves the meaning of the original word." This resulted in 12,300 labels in total with four substitutes per target word on average.
CoINCO. The previous largest lexical substitution benchmark, CoINCO (Kremer et al., 2014), was constructed by first choosing 2474 contexts from the Manually Annotated Sub-Corpus (Ide et al., 2008, 2010). Then, all content words (nouns, verbs, adjective, and adverbs) in the sentences were selected to be target words in order to reflect a realistic frequency distribution of target words and their senses. Each target word was presented to six human annotators, who were asked to provide up to five substitutions or mark it as unsubstitutable. All the annotators were instructed to provide (preferably single-word) substitutes for the target that "would not change the meaning." This resulted in 167,446 labels in total and 7.2 substitutions per target word on average. For the rest of the paper, we focus on CoINCO (but not SEMEVAL) as our benchmark is built on CoINCO and it is the largest existing benchmark.
3 Our benchmark
SWORDS is composed of context, target word, and substitute triples  $(c, w, w')$ , each of which has a score that indicates the appropriateness of the substitute. We consider a substitute to be acceptable if its score is greater than  $50\%$  (e.g. bolded words in Table 1) and unacceptable if the score is less than
or equal to  $50\%$ . Similarly, a substitute with a score greater than  $0\%$  is considered conceivable, and otherwise inconceivable. Note that these terms are operational definitions for convenience, and different thresholds can be chosen for desired applications.
3.1 Addressing limitations of past work
Improving quality. In prior work, annotators were prompted to consider whether a substitute "preserves the meaning" (McCarthy and Navigli, 2007) or "would not change the meaning" (Kremer et al., 2014) of the target word. Instead, we ask annotators whether they "would actually consider using this substitute as the author of the original sentence." We believe this wording encourages a higher standard. In Section 4.1, we provide evidence that substitutes from SWORDS have higher quality than those from COINCO on average.
Improving coverage. For prior benchmarks, annotators were asked to generate a list of substitutes from memory. Psycholinguistic studies have shown that when humans are asked to predict the next word of a sentence, they deviate systematically from the true corpus probabilities (Smith and Levy, 2011; Eisape et al., 2020). Thus, we may reasonably expect that asking humans to generate substitutes would similarly lead to systematic omissions of some appropriate substitutes.
We observe that prior benchmarks exclude many appropriate substitutes that are difficult for humans to think of (Section 4.2). To address this limitation, we first obtain a set of candidate substitutes and then ask annotators to judge whether they would consider using a given candidate to replace the target word in the context. That is, given a context  $c$ , target word  $w$ , and candidate substitute  $w'$ , we ask humans to judge whether  $w'$  is a good replacement for the target word:
$$
\left(\text {c o n t e x t} c, \text {t a r g e t} w, \text {s u b s t i t u t e} w ^ {\prime}\right)\rightarrow \{0, 1 \},
$$
where a positive label 1 corresponds to "I would actually consider using this substitute as the author of the original sentence," and a negative label 0 as the opposite. As described in Section 3.2, we annotate a large pool of candidate substitutes to ensure high coverage of all possible substitutes. We confirm that this increases coverage compared to CoINCo in Section 4.2.
Redefining scores to reflect appropriateness. In past work, each substitute  $w'$  has an associated
3The reported number in Kremer et al. (2014) is 167,336 and 10.71, respectively. The latter differs as they counted the same substitute multiple times when suggested by multiple humans, whereas we report the number of unique substitutes.
Benchmark	Contexts	Targets	Substitutes	Labels	Substitutes per target
Inconceivable (score = 0%)	Conceivable (score > 0%)	Acceptable (score > 50%)
SEMEVAL	2010	201	8025	12,300	-	4.0	N/A
COINCO	2474	15,629	112,742	167,446	-	7.2	2.0*
SWORDS	1250	1250	71,813	395,175	37.4	20.1	3.9
COINCO (dev)	1577	10,179	67,814	98,950	-	6.7	2.0*
SWORDS (dev)	417	417	24,095	127,735	38.9	18.9	3.3
COINCO (test)	897	5450	44,928	68,496	-	8.2	2.0*
SWORDS (test)	833	833	47,718	267,440	36.6	20.7	3.3
Table 2: Benchmark statistics of SEMEVAL, COINCO, and SWORDS. Our benchmark contains 2x more acceptable substitutes (3.9) and 3x more conceivable substitutes (20.1) on average, compared to the previous largest benchmark COINCO. SWORDS has additional inconceivable substitutes which received a score of 0 from appropriateness judgement, which are useful for evaluation. Substitutes in the past benchmarks are listed as conceivable substitutes, and the number of acceptable substitutes for COINCO (numbers with *) is estimated based on the COINCO's substitutes scored under SWORDS.
score defined as the number of annotators who produced  $w'$  given the associated context  $c$  and target word  $w$ . Instead, we define the score as the fraction of annotators who judged the  $w'$  to be an appropriate replacement of  $w$ . We argue that the previous definition of score reflects ease-of-recollection, but not necessarily appropriateness. In Section 4.3, we show that our definition of score better represents the appropriateness of each substitute.
We collect substitutes and scores for a context and target word pair  $(c, w)$  via the following three steps.
Step 1: Select contexts, targets, and substitutes. We use the subset of contexts and target words from CoINCo. Concretely, we start with the  $(c, w)$  pairs in CoINCo and randomly select one  $w$  per  $c$  to annotate. Here, the context  $c$  consists of three sentences, where the middle sentence has the target word  $w$ . Next, we choose a set of candidate substitutes  $w'$  to annotate for each  $(c, w)$  pair, as framing annotation as binary classification requires determining the set of candidate substitutes a priori. We use human-generated substitutes from CoINCo, then add substitutes suggested by a thesaurus (see Appendix A.2 for details). In principle, candidate substitutes can be retrieved from any lexical resource or even sampled from a generative model, which we leave as future work. By combining candidates from CoINCo and the thesaurus, we increase the coverage of acceptable substitutes.
Step 2: Reduce the pool of substitutes. Given a list of candidate substitutes from the previous step, we collect three binary labels on each  $(c, w, w')$
triple (see Section 3.3 for details). Then, we pass any substitute with at least one positive label to Step 3 and further collect fine-grained scores. We show that the probability that an acceptable substitute gets incorrectly filtered out as an inconceivable substitute (three negative labels) is very low  $(0.8\%)$  in Section 4.4.
Step 3: Collect fine-grained scores. In the final step, we collect seven more binary labels on the substitutes which received at least one positive label from Step 2. This yields a total of 10 binary labels for the substitutes.
We used Amazon Mechanical Turk (AMT) to crowdsourced labels on substitutes. Each Human Intelligence Task (HIT) contained a target word highlighted in the context and at most 10 candidate substitutes for the target word. Each candidate substitute had three radio buttons for positive, negative, and abstain. Annotators were asked to choose positive if they would actually consider using the substitute to replace the target word as the author of the context, negative if they would not consider using the substitute, and abstain if they do not know the meaning of the substitute. We treated all abstain labels (1.24% of total labels) as negative labels, thereby making it binary. The benchmark includes abstain labels to maintain the option for them to be handled separately (e.g. excluded) in the future. The interface, instructions, qualification conditions, and filtering criteria used for crowdsourcing can be found in Appendix B.
Table 2 shows overall statistics of our benchmark. SWORDS comprises a total of 1250 context and target word pairs (494 nouns, 448 verbs, 189 adjectives, 119 adverbs) and 71,813 total substitutes that have been labeled (including both acceptable and unacceptable substitutes). For brevity, we defer an analysis of annotator agreement to Appendix C.1.
4.1 High quality
With our notion of acceptability, we first observe that  $75.4\%$  of the substitutes from  $\mathrm{COINCO}^4$  are considered unacceptable, and  $28.6\%$  of the substitutes are even inconceivable (receiving scores less than  $50\%$  and  $0\%$  from our human annotators). Table 3 shows examples of substitutes that received relatively high scores under  $\mathrm{COINCO}$ , yet were considered unacceptable under SWORDS. With the same size as  $\mathrm{COINCO}$  (by taking the subset of our benchmark with the highest scoring substitutes per target), the average score of the substitutes is 4.9 for SWORDS and 3.4 for  $\mathrm{COINCO}$ , resulting in  $1.4\mathrm{x}$  higher quality. Furthermore, SWORDS minimizes the potential noise by having fine-grained scores to account for appropriateness (Section 4.3) as well as explicit inconceivable substitutes, which is useful for evaluation (Section 5.2).
4.2 High coverage
We show that SWORDS achieves high coverage. Among the conceivable substitutes in SWORDS,  $14.4\%$  are only in CoINCo (CoINCo-only),  $14.6\%$  are common to both CoINCo and the thesaurus (CoINCo  $\cap$  Thesaurus), and  $71.1\%$  are only from thesaurus (Thesaurus-only). Among the acceptable substitutes,  $24\%$  are from CoINCo-only,  $37.1\%$  are from CoINCo  $\cap$  Thesaurus, and  $38.9\%$  are from Thesaurus-only. This suggests that a substantial number of substitutes are not present in CoINCo. Overall, SWORDS contains 3.9 acceptable and 20.1 conceivable substitutes per target word on average, increasing those numbers by nearly 2x and 3x over CoINCo, respectively.
In addition, we find that substitutes from CoINCo-only are more likely to be common words whereas substitutes from Thesaurus-only are more likely to be rare words. We compute the Zipf frequency (Speer et al., 2018) of each substitute based on the Google  $n$ -gram corpus (Brants and Franz,
2006) and threshold conceivable substitutes into three groups: uncommon  $(\leq 3.5)$ , neutral, common  $(>4.5)$ . We observe that substitutes from CoINCo-only are more likely to be common words  $(52.7\%)$  than those from Thesaurus-only  $(38\%)$ . On the other hand, the substitutes from Thesaurus-only tend to be more uncommon words  $(29\%)$  than those from CoINCo-only  $(17.5\%)$ .
4.3 Reflection of appropriateness in scores
We show that scores in SWORDS better reflect the appropriateness of each substitute compared to CoINCo both quantitatively and qualitatively. We find that if a substitute has a high score under CoINCo (score  $>1$ ), it is likely to be acceptable under SWORDS (score  $>50\%$ ) almost all the time  $(99.6\%)$ . However, the converse does not hold: the acceptable substitutes under SWORDS have low scores (score  $\leq 1$ ) under CoINCo half of the time  $(49.4\%)$ . Intuitively, this is because CoINCo's scores reflect the ease of producing the substitute from memory, whereas SWORDS's scores reflect the appropriateness of the substitute. Table 3 shows examples of context, target word, and substitute triples which received a low score from CoINCo but a high score from SWORDS.
4.4 Validation with additional data
We show that the probability of an acceptable substitute falsely filtered out in Step 2 is very low. To this end, we collected 10 additional labels on 100 context-target word pairs randomly selected from the test set, without reducing the pool of substitutes as in Step 2. By comparing the first three labels to the entire 10 labels, we find that  $35.5\%$  of substitutes without any positive labels in Step 2 could have received one or more positive labels if they were kept in Step 3. However, we find that  $99.2\%$  of these substitutes were eventually considered unacceptable (judged by 10 labels), indicating that the probability of an acceptable substitute incorrectly filtered out in Step 2 is very low  $(0.8\%)$ .
4.5 Score distribution
Figure 1 shows the score distribution of substitutes in SWORDS along with the source of substitutes: COINCO-only, COINCO  $\cap$  Thesaurus, or Thesaurus-only. Across scores, neither COINCO nor thesaurus completely dominates substitutes, and the overlap between COINCO and thesaurus is
5 1935 substitutes from CoINCo-only, 972 from both, and 43,806 from thesaurus-only received a score of  $0\%$
4For this analysis, we consider COINCO's substitutes that are used and labeled under SWORDS.
Context with target word	Substitute	COINCO's score (max: 6)	SWORDS's score (max: 100%)
I don't wish to be a spokesman for any campaign.	effort	3	0%
She was heading for a drink and slipped out of the crowd.	look	2	10%
"Name me," she said.	nickname	2	0%
The e-commerce free zone is situated in north Dubai.	district	1	90%
She will have reunions in the next few weeks.	forthcoming	1	60%
It's very reassuring that I'll not only be an outsider but a curiosity.	extraordinarily	0	70%
Table 3: Controversial examples of contexts, target words, and substitutes which have low scores under either CoINCo or SWORDS, but high scores under the other benchmark (score  $>1$  for CoINCo and score  $>50\%$  for SWORDS). The contexts are simplified for readability.
Figure 1: Score distribution of SWORDS's substitutes with the source of substitutes. We find that neither COINCO nor the thesaurus completely dominates substitutions across scores, indicating the necessity of both human-generated substitutes as well as substitutes from the thesaurus. Substitutes with score  $0\%$  are not shown to make the bars visually distinguishable.[5]