SlowGuess/NAACL_2021_OCR huggingface dataset:
quite small, thus indicating the necessity of both human-recalled substitutes as well as substitutes from a thesaurus. We also find that SWORDS adds more substitutes for all the scores, although substitutes from the thesaurus tend to have a lower range of scores compared to those from COINCO. Lastly, we observe that substitutes from COINCO roughly form a normal distribution, which suggests that even the substitutes provided by human annotators are incontrovertible, and that it is important to account for the intrinsically gradable nature of appropriateness with fine-grained scores.
5 Model evaluation
In this section, we evaluate several methods on SWORDS. The goals of this evaluation are threefold: (1) to prescribe our recommended evaluation practice for SWORDS, (2) to measure performance of existing large-scale pre-trained models and state-of-the-art lexical substitution systems, and (3) to measure human performance for the purpose of
comparing current and future systems.
5.1 Evaluation settings
There are two primary evaluation settings in lexical substitution research: the generative setting (McCarthy and Navigli, 2007) and the ranking setting (Thater et al., 2010). In the generative setting, systems output a ranked list of candidate substitutes. There are no restrictions on the number of candidates that a system may output. In the ranking setting, systems are given all candidate substitutes from the benchmark (including those marked as unacceptable) and tasked with ranking them by appropriateness. Here we primarily focus on the generative setting, as it is more relevant to writing assistance. We defer our experiments on the ranking setting to Appendix D.
In a writing assistance context, we envision that lexical substitution systems would be used to suggest a limited number of substitutes to users (e.g. 10 substitutes as opposed to 100). Hence, we consider evaluation metrics that examine the quality and coverage of the top-ranked substitutes from a system with respect to the substitutes that humans judged as acceptable (score  $>50\%$ ). Specifically, we compute precision  $(P^{k})$  and recall  $(R^{k})$  at  $k^{6}$ :
$$
P ^ {k} = \frac {\# \text {a c c e p t a b l e s u b s i t u t e s i n s y s t e m t o p - k}}{\# \text {s u b s i t u t e s i n s y s t e m t o p - k}}
$$
$$
R ^ {k} = \frac {\# \text {a c c e p t a b l e s u b s i t u t e s i n s y s t e m t o p - k}}{\min  (k , \# \text {a c c e p t a b l e s u b s i t u t e s})}
$$
Because we care about both quality (precision) and coverage (recall) when comparing systems, we report  $F^k$ , the harmonic mean of  $P^k$  and  $R^k$ . Likewise, we evaluate against the list of substitutes
Note that our definition of recall at  $k$  is non-standard; the min compensates for the fact that there are often fewer than  $k$  acceptable substitutes.
which humans judged as conceivable (score  $>0\%$ ).  $P_{c}^{k}$  and  $R_{c}^{k}$  constitute precision and recall of systems against this larger candidate list, and  $F_{c}^{k}$  their harmonic mean. Motivated by past work (McCarthy and Navigli, 2007), we primarily examine performance for  $k = 10$  and lemmatize system and reference substitutes during comparison.
We note that these metrics represent a departure from standard lexical substitution methodology, established by McCarthy and Navigli (2007). Like  $P^k$  and  $R^k$ , the previously-used BEST and OOT metrics are also measures of precision and recall, but do not take advantage of the negative labels from our binary data collection protocol as no such labels existed in the earlier benchmarks. Nevertheless, we report performance of all systems on these metrics in Appendix E as reference.
We evaluate both state-of-the-art lexical substitution systems and large-scale pre-trained models as baselines on SWORDS. We reimplement the BERT-based lexical substitution system (BERT-LS) from Zhou et al. (2019), which achieves state-ofthe-art results on past benchmarks. As another lexical substitution system, we examine WORD-TUNE (AI21, 2020), a commercial system which offers lexical substitution capabilities. $^7$
We also examine two large-scale pre-trained models adapted to the task of lexical substitution: BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020). To generate and rank candidates with BERT, we feed in the context with target word either masked (BERT-M) or kept intact (BERT-K), and output the top 50 most likely words according to the masked language modeling head. Because the target word is removed, BERT-M is expected to perform poorly—its main purpose is to assess the relative importance of the presence of the target word compared to the context. Note that both of these strategies for using BERT to generate candidates differ from that of BERT-LS, which applies dropout to the target word embedding to partially obscure it. To generate candidates with GPT-3, we formulate lexical substitution as natural language generation (see Appendix D.5 for details).
5.4 Human and oracle systems
Here we consider human and oracle "systems" to help contextualize the performance of automatic
lexical substitution systems evaluated on SWORDS. We evaluate the performance of HUMANS using labels from a separate pool of annotators as described in Section 4.4. Because this task is inherently subjective, this system represents the agreement of two independent sets of humans on this task, which should be thought of as the realistic upper bound for all metrics. We consider the substitutes that have score  $>0\%$  from the separate pool of annotators as HUMANS's substitutes in the generative setting.
We also consider both of the candidate sources, CoINCO and THESAURUS, as oracle systems. Each source contains a list of substitutes for every target word, and therefore can be viewed as a lexical substitution system and evaluated on SWORDS. CoINCO provides substitutes for a target word that were provided by (six) human annotators. This can be thought of as a proxy for how humans perform on lexical substitution when recalling words off the top of their heads (as opposed to making binary judgements as in HUMANS). THESAURUS provides context-free substitutes for a target word (regardless of their word senses) with the default ranking retrieved from the thesaurus. This represents the context-insensitive ordering that a user of the same thesaurus would encounter.
Because these oracle systems only produce candidates which are guaranteed to be in SWORDS, they have an inherent advantage on the evaluation metrics over other systems. Hence, to be more equitable to other systems, we additionally compute  $F^{10}$  and  $F_{c}^{10}$  in a "lenient" fashion—filtering out model generated substitutes which are not in SWORDS (we refer to the setup without filtering as "strict"). It is our intention that future systems should not use COINCO or THESAURUS in any way, as they leak information about the SWORDS benchmark.
5.5 Evaluation results
Table 4 shows that the performance of all methods falls short of that of humans on all metrics. We interpret this as evidence that SWORDS is a challenging benchmark, since strong (albeit unsupervised) baselines like BERT and GPT-3 do not reach parity with humans. We also observe that two models (WORDTUNE and GPT-3) achieve higher  $F^{10}$  than CoINCO. In other words, while all models perform worse than humans who are judging the appropriateness of substitutes (HUMANS), some models appear to slightly outperform humans who
WORDTUNE is not optimized for lexical substitution.
Model	Lenient	Strict
\( F^{10} \)	\( F_{c}^{10} \)	\( F^{10} \)	\( F_{c}^{10} \)
HUMANS*	51.6	76.4	-	-
COINCo	34.6	63.3	-	-
\( THESAURUS^† \)	24.6	59.3	-	-
THESAURUS	17.6	50.2	-	-
\( WORDTUNE^† \)	35.2	43.7	23.4	33.2
GPT-3	35.0	47.0	22.2	34.3
WORDTUNE	34.9	43.5	22.8	32.1
\( GPT-3^† \)	34.7	47.2	21.8	32.8
BERT-K†	32.2	53.8	18.5	29.1
BERT-LS	31.6	53.3	16.8	26.1
BERT-K	31.5	53.2	15.2	23.7
\( BERT-M^† \)	30.8	47.2	15.7	24.4
BERT-M	30.8	47.0	10.4	16.1
Table 4: Evaluation of models on SWORDS in the generative setting. Here, systems must both generate and rank a set of substitutes. We observe that the performance of all baselines on all metrics falls short of human performance. In the "lenient" columns, we filter out system candidates which do not appear in the reference list. *Computed on a subset of the test data.†Reranked by our best ranking model (BERT).
are thinking of substitutes off the top of their head (CoINCo). This implies that some lexical substitution models may already be helpful to humans for writing assistance, with room for improvement.
Overall, we find that there is no single system which emerges as the best on all metrics. We note that, despite BERT-LS representing the state-of-the-art for past lexical substitution benchmarks, its performance falls short of that of commercial systems like GPT-3 and WORDTUNE on most criteria. Also, the BERT-based methods output around 5x as many candidates as the other models on average, thus having an inherent advantage in recall with the lenient criteria (see Table 7 in Appendix E).
In Table 4, we additionally report the performance of generative models by re-ranking their lists of substitutes using the best ranker from our candidate ranking evaluation, BERT (see Appendix D for details). This procedure unilaterally improves performance for all systems on all metrics except for GPT-3. Hence, we speculate that improved performance on the ranking setting will be complementary to improved performance on the generative setting.
From a qualitative perspective, many of the systems we evaluate already produce helpful substi
tutes (Table 5). In examining errors, we find that BERT-based models and WORDTUNE tend to produce words that differ semantically from the target (e.g. "league" for zone). Substitutes generated by GPT-3 are often repetitive (e.g. for zone, GPT-3 produced 64 substitutes, out of which only 13 were unique)—we filter out duplicates before evaluating. Finally, we observe that some systems produce appropriate substitutes which are not present in SWORDS (e.g. GPT-3 produces "precinct" for zone), indicating that SWORDS still has gaps in coverage. However, the higher coverage and quality in SWORDS compared to past benchmarks still improves the reliability of our proposed evaluation.
As we already discussed previous lexical substitution benchmarks in Section 2 and models in Section 5, we use this section to draw connections to other related literature.
Word sense disambiguation. The task of word sense disambiguation consists of selecting the intended meaning (i.e. sense) from the pre-defined set of senses for that word in a sense inventory. The task of lexical substitution is closely related to word sense disambiguation, as many words are sense synonyms—some of their senses are synonymous, but others are not (Murphy, 2010). In fact, McCarthy (2002) proposed lexical substitution as an application-oriented word sense disambiguation task that avoids some of the drawbacks of standard word sense disambiguation, such as biases created by the choice of sense inventory (Kilgarriff, 1997).
Near-synonym lexical choice. Words are often near-synonyms—they can substitute for each other in some contexts, but not every context (DiMarco et al., 1993; Murphy, 2010). SWORDS can be viewed as a collection of human judgments on when certain near-synonyms are substitutable in a given context. The task of near-synonym lexical choice consists of selecting the original target word from a set of candidate words given a context where the target word is masked out (Edmonds and Hirst, 2002). The candidate words are composed of the target word and its near-synonyms which are often retrieved from a lexical resource such as Hayakawa (1994). In this task, systems are tested whether they can reason about near-synonyms and choose the best substitute that fits in the context, without knowing any direct semantic information about the
<sup>8</sup>We speculate that this is because GPT-3 produces many substitutes containing multiple word pieces, and mean-pooling several word pieces may result in lower-quality scores.
Context
Substitutes in SWORDS	The e-commerce free zone is situated in north Dubai, near the industrial free zone in Hebel Ali sector (90%), district (90%), area (90%), region (70%), section (70%), range (60%), strip (60%), ground (50%), segment (50%), territory (50%), sphere (40%), realm (40%), place (30%), tract (30%), city (30%), belt (20%), circuit (20%), band (0%)
Reference for Fk(7)
Reference for Fc(17)	sector, district, area, region, section, range, strip
sector, district, area, region, section, range, strip, ground, segment, territory
COINCO (9)
THESAURUS† (14)	area, region, district, section, city, place, range, strip, territory
district, area, belt, territory, region, realm, sector, section, circuit, segment
WORDTUNE† (11)
GPT-3† (13)
BERT-LS (50)
BERT-K† (50)
BERT-M† (50)	district, area, city, region, site, league, center, system, place, zona
district, area, territory, region, realm, sector, locality, section, quarter, precinct
belt, district, port, area, zones, city, park, center, strip, sector
zones, district, area, city, belt, region, park, ville, site, sector
zones, district, area, city, belt, territory, region, haven, park, site
Table 5: Qualitative comparison of top 10 candidates generated by best systems. From top to bottom, table sections show (1) a context, target word, substitutes, and scores from SWORDS (dev), (2) reference lists used to compute  $F^k$  and  $F_c^k$  (applying thresholds of  $>50\%$  and  $>0\%$  to scores), (3) candidates from oracle systems (data sources of SWORDS), and (4) candidates from best systems. For each system, we include the number of candidates produced by the system in parentheses (after removing duplicates), although we only consider the top 10 candidates for evaluation. Substitutes with scores greater than  $50\%$  from are bolded.
target word and without having to explicitly judge the appropriateness of other candidates.
Lexical and phrasal resources. Lexical resources such as thesauri are often used to identify possible word substitutes. WordNet (Fellbaum, 1998) is a widely used lexical resource for English that includes synonymy, antonymy, hypernymy, and other relations between words. PPDB (Pavlick et al., 2015) includes both word-level and phrase-level paraphrase rules ranked by paraphrase quality. These resources relate words and phrases in the absence of context, whereas lexical substitution requires suggesting appropriate words in context.
Paraphrase generation. Work on sentence-level paraphrase generation considers a wide range of meaning-preserving sentence transformations, including phrase-level substitutions and large syntactic changes (Madnani and Dorr, 2010; Wieting and Gimpel, 2018; Iyyer et al., 2018; Hu et al., 2019). Our work could be extended to phrases given appropriate methods for identifying target phrases and proposing candidate substitute phrases. One benefit of focusing on word substitutions is that we can cover a large fraction of all appropriate substitutes, and thus estimate recall of generative systems. Some word-level substitutions, such as function word variation and substitutions that rely on external knowledge, are also outside the scope of our work but occur in standard paraphrase datasets (Bhagat and Hovy, 2013).
Self-supervised pre-trained models. The task of suggesting words given surrounding context bears strong resemblance to masked language modeling, which is commonly used for pretraining (Devlin et al., 2019). However, for lexical substitution, appropriate substitutes must not only fit in context but also preserve the meaning of the target word; thus, additional work is required to make BERT perform lexical substitution (Zhou et al., 2019; Arefyev et al., 2020).
Modeling human disagreement. In SWORDS, we find considerable subjectivity between annotators on the appropriateness of substitutes. For the task of natural language inference, recent work argues that inherent disagreement between human annotators captures important uncertainty in human language processing that current NLP systems model poorly (Pavlick and Kwiatkowski, 2019; Nie et al., 2020). We hope that the fine-grained scores in SWORDS encourage the development of systems that more accurately capture the graded nature of lexical substitution.
We sincerely thank Frieda Rong, Nelson Liu, Stephen Mussmann, Kyle Mahowald, Daniel Jiang, and all reviewers for their help and feedback throughout this project. We also thank OpenAI and Wordtune for allowing us to evaluate their systems. This work was funded by DARPA CwC under ARO prime contract no. W911NF-15-1-0462.
AI21. 2020. Wordtune (accessed 2020 Oct 30). https://www(wordtune.com/.
Nikolay Arefyev, Boris Sheludko, Alexander Podolskiy, and Alexander Panchenko. 2020. A comparative study of lexical substitution approaches based on neural language models. arXiv.
Rahul Bhagat and Eduard Hovy. 2013. What is a paraphrase? Computational Linguistics, 39.
Thorsten Brants and Alex Franz. 2006. Web 1T 5-gram version 1.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171-4186.
Chrysanne DiMarco, Graeme Hirst, and Manfred Stede. 1993. The semantic and stylistic differentiation of synonyms and near-synonyms. In AAAI Spring Symposium on Building Lexicons for Machine Translation.
Philip Edmonds and Graeme Hirst. 2002. Near-synonymy and lexical choice. Computational Linguistics.
Tiwalayo Eisape, Noga Zaslavsky, and Roger Levy. 2020. Cloze distillation improves psychometric predictive power. In Computational Natural Language Learning (CoNLL).
Katrin Erk and Sebastian Padó. 2008. A structured vector space model for word meaning in context. In Empirical Methods in Natural Language Processing (EMNLP), pages 897-906.
Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press.
Samuel Ichiye Hayakawa. 1994. Choose the Right Word: A Contemporary Guide to Selecting the Precise Word for Every Situation. Collins Reference.
Gerold Hintz and Chris Biemann. 2016. Language transfer learning for supervised lexical substitution. In Association for Computational Linguistics (ACL).
J. Edward Hu, Rachel Rudinger, Matt Post, and Benjamin Van Durme. 2019. Parabank: Monolingual bitext generation and sentential paraphrasing via lexically-constrained neural machine translation. In Association for the Advancement of Artificial Intelligence (AAAI).
Nancy Ide, Collin Baker, Christiane Fellbaum, Charles Fillmore, and Rebecca Passonneau. 2008. MASC: the manually annotated sub-corpus of American English. In Language Resources and Evaluation (LREC).
Nancy Ide, Collin Baker, Christiane Fellbaum, and Rebecca Passonneau. 2010. The manually annotated sub-corpus: A community resource for and by the people. In Association for Computational Linguistics (ACL).
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In North American Association for Computational Linguistics (NAACL).
A. Kilgarriff. 1997. I don't believe in word senses. Computers and the Humanities.
Barbara Ann Kipfer. 2013. Roget's 21st Century Thesaurus, Third Edition. Random House Publishing Group.
Kazuaki Kishida. 2005. Property of average precision and its generalization: An examination of evaluation indicator for information retrieval experiments. Technical report, National Institute of Informatics.
Gerhard Kremer, Katrin Erk, Sebastian Pado, and Stefan Thater. 2014. What substitutes tell us - analysis of an "all-words lexical substitution corpus. In Association for Computational Linguistics (ACL).
Nitin Madnani and Bonnie J Dorr. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics, 36(3):341-387.
Diana McCarthy. 2002. Lexical substitution as a task for WSD evaluation. In SIGLEX/SENSEVAL Workshop on Word Sense Disambiguation: Recent Successes and Future Direction.
Diana McCarthy and Roberto Navigli. 2007. SemEval-2007 task 10: English lexical substitution task. In Workshop on Semantic Evaluations.
Oren Melamud, Ido Dagan, and Jacob Goldberger. 2015. Modeling word meaning in context with substitute vectors. In North American Association for Computational Linguistics (NAACL).
Tristan Miller. 2016. Adjusting Sense Representations for Word Sense Disambiguation and Automatic Pun Interpretation. Ph.D. thesis, Technische Universität Darmstadt.
M. Lynne Murphy. 2010. Lexical Meaning. Cambridge University Press.
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Empirical Methods in Natural Language Processing (EMNLP).
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics (TACL), 7.
Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Association for Computational Linguistics (ACL).
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
S. Sharoff. 2006. Open-source corpora: Using the net to fish for linguistic data. International Journal of Corpus Linguistics, 11:435-462.
Nathaniel Smith and Roger Levy. 2011. Cloze but no cigar: The complex relationship between cloze, corpus, and subjective probabilities in language processing. In Proceedings of the Annual Meeting of the Cognitive Science Society.
Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, and Lance Nathan. 2018. LuminosoInsight/wordfreq: v2.2. https://doi.org/10.5281/zenodo.1443582.
György Szarvas, R'obert Busa-Fekete, and Eyke Hüllermeier. 2013. Learning to rank lexical substitutions. In Empirical Methods in Natural Language Processing (EMNLP).
Stefan Thater, Hagen Fürstenau, and Manfred Pinkal. 2010. Contextualizing semantic representations using syntactically enriched vector models. In *Association for Computational Linguistics (ACL)*.
William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS).
John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Association for Computational Linguistics (ACL).
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2019. BERT-based lexical substitution. In Association for Computational Linguistics (ACL).
A Data collection
A.1 Deduplicating context
SWORDS uses the same contexts as CoINCo, but with slight modifications to avoid duplication issues and incomplete contexts found in CoINCo. CoINCo uses a subset of contexts from the Manually Annotated Sub-Corpus (MASC) (Ide et al., 2008, 2010), in which some sentences are erroneously repeated multiple times due to multiple IDs assigned to a single sentence. Consequently, CoINCo contains duplicate sentences in some contexts, as shown below:
"–was kindly received," "But an artist who would stay first among his fellows can tell when he begins to fail." "But an artist who would stay first among his fellows can tell when he begins to fail."
Furthermore, we found that some parts of the document context are missing in CoINCo because an ID was not assigned to the parts in MASC (e.g. "he said." is missing from the above passage after the word "received").