ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution (2024)

Abstract: Lexical Substitution discovers appropriate substitutes for a given target word in a context sentence. However, the task fails to consider substitutes that are of equal or higher proficiency than the target, an aspect that could be beneficial for language learners looking to improve their writing. To bridge this gap, we propose a new task — language proficiencyoriented lexical substitution. We also introduce ProLex, a novel benchmark designed to assess systems’ ability to generate not only appropriate substitutes but also substitutes that demonstrate better language proficiency. Besides the benchmark, we propose models that can automatically perform the new task. We show that our best model, a Llama2-13B model fine-tuned with task-specific synthetic data, outperforms ChatGPT by an average of 3.2% in F-score and achieves comparable results with GPT-4 on ProLex.1

heading: 1 Introduction
text: Nowadays, automatic English learning tools have become widespread across various educational settings. For instance, automatic grammar error correction systems (Omelianchuk et al., 2020; Yasunaga et al., 2021; Tarnavskyi et al., 2022; Cao et al., 2023), simplify grammar correction for learners and help enhance their writing skills. Besides grammar, enhancing vocabulary diversity is another crucial element in improving English writing (Smitherman and Villanueva, 2003; Johnson et al., 2016; González, 2017). Nevertheless, English second-language (L2) learners frequently struggle to use a diverse vocabulary in their writing (Gu and Johnson, 1996; Fan, 2020; Li et al., 2023; Sun et al., 2023). They tend to use the same set of words repeatedly, which may negatively impact
1Data and code available: https://github.com/ BillyZhang24kobe/LS_Proficiency
their performance on essay writing tests (Johnson et al., 2016).
The beginner L2 learners can leverage existing lexical substitution systems (Zhou et al., 2019; Lacerra et al., 2021; Yang et al., 2022; Wada et al., 2022; Qiang et al., 2023) to enhance their vocabulary breadth. These systems are designed to identify contextually suitable substitutes for a target word, thereby assisting learners in discovering appropriate substitutions. However, merely knowing these contextually appropriate substitutes is insufficient for L2 learners. Prior benchmarks (McCarthy and Navigli, 2007; Kremer et al., 2014; Horn et al., 2014; Lee et al., 2021) for lexical substitution focus solely on the contextual appropriateness of generated substitutes. To enhance the vocabulary diversity and writing proficiency of English learners, guided by the principles of the zone of proximal development (Vygotsky, 1978), we propose that the substitutions should also reflect an equal or higher level of language proficiency compared to the original target word.
In this work, we present ProLex, a novel lexical substitution benchmark that evaluates system performances on language proficiency-oriented lexical substitution, a new task that proposes substitutes that are not only contextually suitable but also demonstrate advanced-level proficiency. To construct ProLex, we begin by selecting target words and sentences according to their frequency in TOEFL-11 (Blanchard et al., 2013), a comprehensive corpus of essays written by non-native English learners. This approach ensures a data distribution that is more representative of L2 English learners. Subsequently, to effectively harness human expertise in annotating advanced proficiencyoriented lexical substitutions, we adopt the methodology recommended by Lee et al. (2021) — directly having humans judge the appropriateness of candidate substitutes instead of generating them ad hoc. To facilitate this, we employ GPT-4 to
8475
generate an initial pool of substitute candidates. Then, the human annotators assess these candidates based on their contextual appropriateness using our language-learning-oriented annotation scheme. The proposed scheme takes into account the preservation of semantic meaning, common English collocations, lexical diversity, and grammatical correctness. We specifically recruited two annotators who are pursuing their PhDs in Linguistics, specializing in English language teaching and education. Following human annotation, we use the Common European Framework of Reference (CEFR) (Council of Europe. Council for Cultural Co-operation. Education Committee. Modern Languages Division, 2001) to remove substitutions that have lower proficiency levels compared to the original target word.
In addition, we build models to perform language proficiency-oriented lexical substitution automatically on ProLex. We also evaluate the performance of existing state-of-the-art (SOTA) lexical substitution systems, as well as prompting LLMs in zero-shot and in-context learning settings. The results of our experiments on ProLex reveal that our top-performing model, Llama2-13B, which was instruction-tuned using a mix of synthetic and modified data, surpasses all existing SOTA lexical substitution systems by an average of 26.8% in F-score. It also outperforms ChatGPT by an average of 3.2% and attains results on par with GPT-4. We also demonstrate that, instruction tun-
ing using task-specific synthetic data yields better results compared to zero-shot LLMs on ProLex.
Overall, our contributions are threefold:
• We propose a new task, namely language proficiency-oriented lexical substitution, to help L2 English learners enhance their vocabulary diversity and writing proficiency.
• We present a novel benchmark ProLex to assess systems’ ability to perform the task by generating substitutes that are not only contextually appropriate but also reflect advanced language proficiency levels.
• We fine-tuned models with task-specific synthetic data and evaluated them with ProLex. The models attain results that are comparable to, or better than, the out-of-the-box LLMs, such as GPT-4 and ChatGPT.

heading: 2 Related Work
text: We first outline the task of lexical substitution, then introduce recent high-quality benchmark in this field, Swords, as detailed in (Lee et al., 2021). Next, we describe TOEFL-11 (Blanchard et al., 2013), a large-scale corpus of written essays composed by non-native English learners, which we adopt as the corpus for selecting context sentences and target words for ProLex.
Lexical Substitution Lexical substitution is a well-established task, which was originally defined in McCarthy (2002) - given a target word w in a given context c, one needs to generate a list of substitutes w′ that can be used to replace w in c. Specifically, the context c refers to one or more sentences encompassing the target word w. The target word itself, which may be selected manually by human annotators (McCarthy and Navigli, 2007), or automatically based on its part-of-speech as discussed in Kremer et al. (2014); Lee et al. (2021), is a single word within this context. The substitute w′ for the target word can be either a single word or a phrase. Although w′ can be ungrammatical in previous benchmarks, we want to incorporate only grammatically correct substitutes in ProLex. Moreover, the past benchmarks do not consider the language proficiency of the substitutes w′. In ProLex, our approach includes substitutes that are not only acceptable but also demonstrate equal or higher proficiency level compared to the target word.
Swords Different from other popular benchmarks (McCarthy and Navigli, 2007; Kremer et al., 2014) depending on recall from humans as the only source of data, Swords (Lee et al., 2021) enhanced the collection of data in lexical substitution by treating it as a classification problem. This method was guided by the intuition that it is easier for humans to judge the appropriateness of given substitutes than to create them spontaneously. This approach resulted in a higher-coverage and higherquality benchmark. In our human annotation process, we follow Swords’s established instructions, and propose a language-learning-oriented annotation scheme for the appropriateness judgement.
TOEFL-11 Essay Corpus The large-scale nonnative English writing corpus, TOEFL-11 (Blanchard et al., 2013), contains writing samples from non-native TOEFL test takers from 2006 to 2007. In particular, it contains 12, 100 essays, each of which is categorized into one of the three score levels: high, medium, and low. Focusing on enhancing the writing skills of English learning beginners, our methodology involves selecting context sentences and target words primarily from essays with medium and low scores. This approach is based on the assumption that beginner L2 English learners are likely to possess a more limited vocabulary range (Fan, 2000).

heading: 3 ProLex Benchmark
text: We propose ProLex, a benchmark for language proficiency-oriented lexical substitution. ProLex is composed of quadruplets, each containing a target word, a context sentence, a list of acceptable substitutes, and a list of proficiency-oriented substitutes (w, s, wa, wap). As indicated by Lee et al. (2021), previous benchmarks (McCarthy and Navigli, 2007; Kremer et al., 2014) gathered substitute words by prompting humans to generate them ad hoc. However, many viable substitutes, challenging for humans to devise, might ultimately be overlooked. Therefore, we follow Lee et al. (2021) by first obtaining a set of candidate substitutes and later asking annotators to judge their appropriateness. Formally, given a context s, target word w, and candidate substitute w′, the annotators are asked to judge whether w′ is an acceptable substitute for the target word: (w, s, w′) → {1, -1, 0}, with 1, -1 and 0 indicating "accept", "reject", and "uncertain" respectively. Figure 1 shows the complete benchmark creation process for ProLex. In the following sections, we will first illustrate the data creation process in Section 3.1. Then, in Section 3.2, we will describe our language-learningoriented annotation scheme. In Section 3.3, we show how we filter substitutes based on CEFR proficiency levels.

heading: 3.1 Data Creation
text: Since we focus on improving the writing proficiency of beginner L2 learners, we turn to TOEFL11 (Blanchard et al., 2013). We will select contexts, target words, correct basic grammar errors, and generate candidate substitutes through the following three steps.
Step 1: Select contexts, and target words. The TOEFL-11 dataset comprises 12, 100 essays, each scored as high, medium, or low. Specifically, there are 4, 202 high-scored essays, 6, 568 with medium scores, and 1, 330 categorized as low. Given the limited vocabulary range of low and mediumproficiency L2 English learners, we intend to select the words that are frequently used in their essays as the target words in our benchmark. For each essay of interest, we select mainly the content words (i.e. nouns, verbs, adjectives, and adverbs) that are used at least three times in the essay. Then, for each such target word, we randomly choose one sentence from all sentences containing that target word. The general statistics of all target words
and context sentences are presented in Appendix C Table 5. In sum, we selected 2, 531 unique wordcontext (w, s) pairs from low and medium-level essays from TOEFL-11 corpus.
Step 2: Correct basic grammar errors in selected context sentences. We observed that the chosen sentences frequently contain fundamental grammar mistakes, such as spelling errors, which can obscure the intended meaning of the context. Example grammar errors are demonstrated in appendix D. To correct these errors, we follow Rothe et al. (2021) and fine-tune a GPT-2 model (Radford et al., 2019) on CLang-8 dataset.2 The experiment details are also shown in appendix D. The resulting model is deployed to correct the basic grammar errors in all of our 2, 531 selected context sentences s. We then sample 1, 125 (w, s) pairs for human annotations.3
Step 3: Generate candidate substitutes. Powered by recent advances in LLMs (Brown et al., 2020; Wei et al., 2022), for each (w, s) we generate the candidates using GPT-4. Specifically, we prompt GPT-4 with five in-context examples to generate exactly five candidate substitutions for each of our 1, 000 (w, s) pairs — thus resulting in 5, 000 (w, s, w′) triples, with w′ representing each candidate substitute for each (w, s) pair. Figure 1 demonstrates an example of the generated candidates. The complete prompt for GPT-4 generation is shown in Appendix B.

heading: 3.2 ProLex Annotation Scheme
text: Different from previous schemes (McCarthy and Navigli, 2007; Kremer et al., 2014; Lee et al., 2021), ProLex deems a substitute valid if it maintains the original sentence’s meaning, aligns with common English usage, and is grammatically correct. Furthermore, ProLex encourages a diverse set of substitutes as it can expand the vocabulary of language learners. A detailed description of the annotation scheme is in Appendix A. Formally, as described in the previous section, for each (w, s, w′) pair, we ask the annotators to judge whether w′ is an acceptable substitute for w, by assigning one of the following classes to w′, 1 (acceptable), -1 (unacceptable) and 0 (uncertain). Subsequently, for each (w, s) pair, we will combine all w′ that
2https://github.com/ google-research-datasets/clang8 3We used 125 pairs for calculating inter-annotator agreement and 1, 000 for the annotation. We will include more pairs to expand the benchmark in future work.
are labeled as acceptable to form wa, leading to (w, s, wa) triples. In the following, we will discuss the features of the annotation scheme in detail.

heading: 3.2.1 Semantic Meaning Preservation
text: Given a triple (w, s, w′), namely a target word w, and the context sentence s where w is situated, the word substitute w′ should preserve the semantic meaning of the original context sentence s. Concretely, there are three cases of acceptable substitutes (i.e. labeled as "1") for meaning preservation in ProLex. Firstly, the annotators accept a substitute of w if they convey exactly the same meaning in s. Besides, a substitute is also acceptable if it is an entailment from the sense of the target word. For example, trust a person entails rely on a person in the following sentence: "I **trust** a person who has more knowledge than I do." Hence, rely on is a valid substitute to target word trust in the example. Lastly, a substitute can be used figuratively and convey a similar meaning as the target word w. In the following sentence: "Once the undergraduate studies are **pursued** by a student, the student is more aware of different subjects and the knowledge he has gained in the period of his studies.", embarked on is an acceptable substitute as it is a figurative use case, preserving the meaning of the target word pursued.

heading: 3.2.2 Common Collocations in English
text: Learners can improve their general levels of writing proficiency in the language through collocation knowledge (Howarth, 1998; Gitsaki, 1999; Boers and Webb, 2018). In ProLex, annotators accept substitutes only if they are common English collocations within the context sentence. For instance, consider the following example where the target word is defying: "Successful people are not only seeking new experiences, **defying** obstacles and hardship but also trying to be creative." Suppose we have a candidate substitute conquering, even though conquering obstacles conveys a similar meaning as defying obstacles, it will be rejected since it is not a common expression and collocation in English.
In addition, in cases where the annotators are unsure about the collocation of any expression, we encourage them to refer to an external collocation knowledge base, such as COCA, the Corpus of Contemporary American English (Davies, 2010), which contains 400 million words from various genres (e.g. spoken, fiction, newspapers and etc.)
distributed from 1990 to present time. In particular, we ask the annotator to accept certain expressions if the frequency4 of it queried from COCA is greater than 5. Take conquer obstacles from the previous example, since there is only one instance of conquer obstacles in COCA, it is rejected from the candidate list.

heading: 3.2.3 Lexical Diversity
text: A richer and more diverse vocabulary can improve both the quality and effectiveness of communication in English (Yu, 2010). To enhance lexical diversity in ProLex, annotators are asked to mark a substitute as acceptable if, in general, it matches at least one connotation of the target word in the context sentence. For instance, in the sentence "In fact they not only give the possibility to move in a short time but sometimes they are also **precious** goods.", precious conveys a connotation of high value in price or importance. Hence, the following candidates are all acceptable: valuable, prized, cherished, invaluable and treasured, as they express either "high value in price" or "importance", matching at least one connotation of the target word precious.

heading: 3.2.4 Grammar Correctness
text: Although previous benchmarks (McCarthy and Navigli, 2007; Kremer et al., 2014; Lee et al., 2021) do not involve grammar correction, ProLex considers the substitutes to be acceptable only if they are grammatically correct. For instance, in the sentence "Nevertheless, who are mostly responsible for these **research**?", all of the following candidates are rejected because none of them are in plural forms: study, investigation, analysis, examination and inquiry.

heading: 3.3 Substitutes Filtering based on CEFR Proficiency Levels
text: Taking the triples (w, s, wa) from human annotations in section 3.2, for each target word w in a context sentence s, and the list of acceptable substitutes wa, we perform filtering to select substitutes with equal or higher proficiency compared to the target word to form wap — resulting in the final quadruplets of ProLex (w, s, wa, wap). Concretely, we refer to the CEFR Checker (CathovenAI, 2023) to automatically determine the CEFR level of each target word w and its associated acceptable substitutes wa. An example filtering process is shown in
4https://www.english-corpora.org//coca/
Figure 1. Note that generally is removed from the set since its CEFR level (i.e. B1) is lower than that of overall (i.e. B2).

heading: 4 Dataset annotation process and statistics
text: Considering the necessity for substantial semantic understanding and knowledge of English collocations in the annotation process, we recruited two annotators currently pursuing PhDs in Linguistics, with specializations in English language teaching and education. On the 125 (w, s) pairs, namely 625 (w, s, w′) triples, the two annotators reached an inter-annotator agreement of κ = 0.60, indicating a near-substantial level of agreement.5 Then, they annotated the complete test set separately, with each annotating 2, 500 (w, s, w′) triples sampled from section 3.1. In total, among all 5, 000 candidate substitutes w′, 39% is labeled as 1 (accept), while 51% and 10% are annotated as -1 (reject) and 0 (uncertain), respectively. This entails that GPT-4 generated candidates may not always be valid in ProLex, indicating future research opportunities in using LLMs for generating semantically correct, collocationally appropriate English lexical substitutions.
5We perform Cohen’s Kappa (Cohen, 1960) to calculate the inter-annotator agreement. We exclude the labels with 0 to consider only the contexts that are conceivable to both annotators (i.e. labeled as 1 or -1).
We further process the annotations to compose the final quadruplets (w, s, wa, wap) in ProLex, where wa denotes a list of acceptable substitutes for w, and wap denotes the list of acceptable substitutes after filtering based on CEFR levels. We exclude the target words that have zero acceptable substitutes. Table 1 provides an overview of the statistics for the resulting data, grouped by low and mediumlevel proficiency. In total, there are 680 wordsentence (w, s) pairs in ProLex with at least one acceptable substitute.6 On average, for each (w, s) pair, there are 2.9 acceptable substitutes wa and 2.3 proficiency-oriented substitutes wap . Furthermore, we provide the distribution of CEFR levels of the target words in both low and medium sentences, and the result is shown in Figure 2. As evident, a significant proportion of the chosen target words are from A1 and A2 levels, comprising 75% in low-level sentences and 55% in medium-level sentences, respectively. Moreover, in Figure 3, we also present the CEFR level distributions for acceptable substitutes wa and proficiency-oriented substitutes wap . The distribution resonates closely with our research objective of ProLex, which aims to propose acceptable substitutions of high-proficiency level words for commonly used low-proficiency level words in L2 English sentences.

heading: 5 Language Proficiency-oriented Lexical Substitution
text: In this section, we outline recommended evaluation practices for ProLex, propose models that can automatically perform the task on ProLex, and assess
6The Part-Of-Speech distribution for the target word w in ProLex is shown in Table 6.
the performance of out-of-the-box LLMs and top lexical substitution systems.

heading: 5.1 Evaluation Settings
text: For the task of lexical substitution, there are two mainstream evaluation settings: generative setting (McCarthy and Navigli, 2007) and ranking setting (Thater et al., 2010). In the generative context, systems produce a sequence of potential substitutes, and there are no limits on the number of candidate substitutes they can generate. In the ranking scenario, systems receive the entire set of substitute options provided by the benchmark. The task then is to order these substitutes according to their appropriateness. In this work, we mainly focus on the generative setting and defer the ranking setting to future research.

heading: 5.2 Evaluation Metrics
text: We evaluate the system performance in terms of its substitute appropriateness and proficiency. For assessing appropriateness, we compare the system’s predictions with the acceptable substitutes, denoted as wa; similarly, for evaluating proficiency, we contrast the system’s predictions with proficiencyoriented substitutes, represented as wap . Inspired by Lee et al. (2021), we consider the evaluation metrics that measure the quality and coverage of the predicted substitutes from a system. Specifically, for appropriateness, we compute precision (P k), recall (Rk) and F-score (F k)7 at k:
7F k is calculated as the harmonic mean of P k and Rk.
P k = # acceptable subs wa in system top-k
# substitutes in system top-k
Rk = # acceptable subs wa in system top-k
min(k,# acceptable subs wa)
Similarly, we evaluate proficiency against the list of proficiency-oriented substitutes wap . P k p and R k p represent the precision and recall of system outputs against this smaller candidate list, with F kp as their harmonic mean. Also, we follow previous work (McCarthy and Navigli, 2007; Lee et al., 2021) to mainly examine performance for k = 10, and implement a soft evaluation setting, where all substitutes generated by a system are lemmatized before comparison. In addition, given that ProLex considers grammatically correct substitutes, we also apply a hard evaluation setting, where predictions should exactly match the reference without lemmatization.

heading: 5.3 Baselines
text: We evaluate systems in the following settings: 1) existing top lexical substitution systems, 2) instruction-tuning language models on taskspecific synthetic data, and 3) zero-shot prompting and in-context learning with LLMs.
Existing Top Lexical Substitution Systems Past lexical substitution systems were proposed and evaluated on three widely-known benchmarks: LS07 (McCarthy and Navigli, 2007), CoInCo (Kremer et al., 2014) and Swords (Lee et al., 2021). We evaluate the following two representative systems on ProLex:
• BERT-LS (Zhou et al., 2019): a BERT-based lexical substitution system. It once reached SOTA results on LS07 and CoInCo dataset.
• Para-LS (Qiang et al., 2023): a paraphraserbased system, achieving SOTA performance on all three benchmarks: LS07, CoInCo, and Swords.
Instruction Tuning with Task-specific Synthetic Data As instruction tuning has proliferated in building powerful instruction following models (Ouyang et al., 2022; Bai et al., 2022; OpenAI, 2023; Chiang et al., 2023), we experiment with two instruction-tuned large-scale language models, namely Vicuna8 (Chiang et al., 2023) and Llama-2 (Touvron et al., 2023). Note that we mainly focus on the smaller variants: Vicuna 7B/13B and Llama2 7B/13B. Since there is no existing training data for our language proficiency-oriented lexical substitution task, we adopt two approaches to synthesize the data: 1) directly generate data with GPT-4 through prompting, and 2) modify the existing lexical substitution benchmark, Swords. The prompts we employed to synthesize the data with GPT-4 are shown in section E.2.1. As for the second approach, we considered only the acceptable substitutes (i.e. score greater than 50%) from the Swords dataset and filtered the substitutes based on their CEFR levels, by only keeping the ones that have equal or higher proficiency levels compared to the target word. As a result, we denote the training data synthesized with GPT-4 as DL, the data modified from Swords as DS , and the aggregate of these two datasets as DLS .
Zero-shot and In-context Learning with LLMs Leveraging the potent zero-shot capabilities of Large Language Models (LLMs), we evaluate their performance on ProLex by prompting two widely recognized LLMs, GPT-4 and ChatGPT,9 to generate potential substitution candidates. Moreover, we also perform in-context learning with examples selected from DL.10 See appendix E.1 for more details.
8We used Vicuna v1.5 provided by (Zheng et al., 2023). 9Note that we used the version gpt-3.5-turbo-1106
10We randomly select 32 examples from DL for in-context learning with LLMs

heading: 5.4 Results and Analysis
text: Table 3 demonstrates the results of our experiments on ProLex. We take the top 10 (k = 10) predictions from each system to evaluate. In general, the appropriateness scores F 10 for both soft and hard settings are greater than the proficiency scores F 10p , indicating the challenges of generating proficiencyoriented substitutes. Additionally, all systems tend to achieve higher scores in soft setting, given that all predictions are lemmatized before comparison. This indicates that these systems might produce substitutes that are grammatically incorrect yet suitable and demonstrate advanced language proficiency.
Overall, in-context learning with GPT-4 achieved the best results across all systems in all evaluation settings. Note that the 32 in-context examples were randomly selected from our taskspecific synthetic dataset DL. This underscores the effectiveness of our data synthesis method, which can be applied in situations with limited data resources. Using DL, Vicuna-1.5-13B achieved the best performance among all fine-tuned model variants on single dataset and previous top-performing lexical substitution systems, even surpassing ChatGPT on both zero-shot and in-context learning scenarios, and GPT-4 on zero-shot setting.
In addition, in our instruction-tuning setup, training with dataset DS resulted in lower performance
compared to training with the synthetic dataset DL. This occurred because DS is derived from filtering Swords, which comprises context sentences not authored by L2 English learners, thus leading to a distribution shift. On the other hand, all context sentences in dataset DL originate from the TOEFL11 corpus, the same source used to develop ProLex. We also conducted fine-tuning experiments using the aggregate dataset DLS . Table 4 demonstrates that combining DS and DL enhanced the performance of all fine-tuned models. Notably, Llama-213B achieved scores nearly comparable to the best result of GPT-4.
As for previous lexical substitution systems, Para-LS consistently surpasses BERT-LS but lags far behind all other larger-scale systems. This highlights the limitations of these earlier methods and points towards a promising research direction in developing more efficient systems that can perform better in generating language proficiency-oriented substitutions.
For error analysis, Table 2 shows an example and the outputs generated by top systems. GPT-4 and ChatGPT produce words that have different semantics from the target (e.g. "hype" for **promotion**). Our fine-tuned models produce substitutes covering multiple connotations of the target (e.g. "advancement" and "marketing"), leading to lower precision. Para-LS generates words that have a more general meaning than the target (e.g. "event" and "show").

heading: 6 Conclusion and Future Work
text: We propose a new task language proficiencyoriented lexical substitution to improve the vocabulary diversity and writing proficiency of L2 English learners. We introduce ProLex, a novel benchmark designed to assess systems’ ability to generate appropriate and language proficiency-oriented substitutes for given target words in the context. Besides,
we also fine-tuned open-source language models on task-specific synthetic data, achieving results that are on par with, or better than GPT-4 and ChatGPT. In future, we will expand ProLex by leveraging larger L2 English writing corpora and incorporating more comprehensive sets of substitutes.

heading: 7 Limitations
text: Benchmark Size and Coverage Compared to previous Lexical Substitution benchmarks (McCarthy and Navigli, 2007; Kremer et al., 2014; Lee et al., 2021), the size of ProLex is relatively small. However, considering that ProLex draws its data from the TOEFL-11 corpus, it could be significantly extended by incorporating sentences from additional sources authored by L2 speakers.
In addition, the systems may produce valid substitutes that are not present in ProLex during evaluation, indicating that ProLex has limitations in coverage. By employing the annotation scheme described in Section 3.2, we can iteratively update and enhance ProLex with additional gold standard substitutes, thereby expanding its coverage in future versions.
Limits of CEFR Checker The CEFR Checker (CathovenAI, 2023) we used in this work is capable of assigning CEFR levels only at the word level. However, we noticed that in some cases, phrases could also serve as appropriate substitutes. Hence, extending the CEFR Checker’s functionality to include CEFR-level assignments at both the word and phrase levels would offer a more holistic evaluation in future versions of ProLex.
Phrases and Multi-word Substitutions In this work, we posit that the integration of multi-word expressions, such as idiomatic phrases, can enhance English language proficiency (Thyab, 2016; Yunus and Hmaidan, 2021; Al-Khawaldeh et al., 2016). Therefore, we treat all acceptable phrasal substitutes as valid substitutes for demonstrating better language proficiency compared to the single target word. In future research, we intend to investigate the methods for assessing proficiency levels in the use of phrases and multi-word expressions.

heading: 8 Ethics Statement
text: Reproducibility In this work, our data creation process utilized GPT-4. We also used ChatGPT for evaluation purposes. Although they are not open-sourced language models, to facilitate the
reproducibility of our results, we demonstrated all of prompts used in our paper. In addition, all the other models used in this research, are publicly available in peer-reviewed articles and referenced in this paper. All datasets, including our synthetic fine-tuning dataset and all annotated test data, will be released.
Biases Our models are built over Vicuna and Llama-2. In this work, we did not explicitly handle any bias that exists in the two pre-trained language models.
Human Annotators Both annotators were recruited from the doctoral programs in the linguistics department, and they specialize in English language teaching and education. They were paid at a rate of $13 per hour. To ensure the privacy and anonymity of all contributors, no personal or demographic information was collected.

heading: 9 Acknowledgement
text: We would like to thank Qingyang Wu, Kun Qian, Siyan Li, Matthew Toles, Yu Li, and Xiao Yu for their valuable discussions and suggestions on the paper. We also want to express our gratitude to Cathoven AI for providing free access to their CEFR checker APIs. In addition, we thank our expert annotators for their time and contributions to the completeness of the benchmark.

heading: A Detailed ProLex Annotation Scheme
text: In this section, we present the detailed annotation scheme for ProLex. In the annotation process, each annotator is presented with 2, 500 (w, s, w′) triplets. For each (w, s, w′), we ask the annotators to judge whether w′ is an acceptable substitute for w in s. In particular, they will choose among three labels: 1 (acceptable), −1 (unacceptable), and 0 (uncertain). The general guidelines are shown in Table 9. A candidate substitute w′ is labeled as 1 only if it satisfies all criteria in the table. It is labeled as −1 once it violates any one of them. We also provide label 0 to exclude the cases when either the target word w or context sentence s is extremely hard to understand as they are written by beginner L2 English learners. Besides the general guidelines, we also provide a detailed specification along with some examples for all three labels. See Table 10 for more illustration examples. The detailed specification is as follows:
• To determine if a substitute is a common collocation or expression in actual English use, annotators should refer to COCA to search for certain expressions:
– Mark 1 if the frequency of the expression is greater than 5 – Mark −1 otherwise
• Mark 1 if the substitute is an entailment from the sense of the target word.
• Mark 1 if the substitute can be used figuratively and also convey the same semantics as the target word does.
• Mark 0 if the substitute is a hyponym of the target word.
• A sentence can contain some grammar errors (e.g. spelling errors). If it is written in a way that it is hard for you to understand, mark it 0; otherwise, if you can understand, mark the substitute as 1 or −1.
• Mark 1 if the substitute is good even though with incorrect determiner like “a” vs “an”.
• Mark −1 if the substitute is good but not quite grammatically correct.
• Grammar of a substitute matters, please keep a high bar for making a substitute as 1.
• Mark 1 if the target word itself appears as a substitute.
• Mark −1 if the target word is a proper noun or part of a fixed expression/phrase.
• Mark 0 if you need more contexts of the original sentence to help you judge the acceptability.

heading: B Generate Candidate Substitutions from GPT-4
text: We generate the candidate substitutes for human annotation by prompting GPT-4. The prompt we used during generation is demonstrated in Table 11. In particular, we selected five instances from Swords (Lee et al., 2021) as the in-context examples to guide the generation process. In these examples, the target words consist exclusively of content words — nouns, verbs, adjectives, and adverbs — which represent the primary focus of ProLex.

heading: C Dataset Statistics
text: In this section, we illustrate the statistics of all target words and context sentences extracted from Section 3.1. The overall statistics are shown in Table 5 below. In general, there are 2, 531 unique (w, s) pairs combing sentences from low and medium-level essays. Note that on average, sentences from medium-level essays are longer than the ones from low-level essays. Furthermore, nouns comprise the majority of the target words, accounting for 57% of all selected words. In contrast, verbs, adjectives, and adverbs make up 23%, 16%, and 4% of the selection, respectively.
Note that we sample 1, 000 (w, s) pairs from the global dataset, generate candidate substitutes from GPT-4, and ask human annotators to judge the contextual appropriateness of these candidates. After filtering the ones labeled with 0, we end up with 680 (w, s) pairs in the final ProLex dataset. We also present the statistics of these (w, s) pairs in ProLex in Table 6. Among all 680 pairs, 52% of them are nouns, 25% are verbs, 18% are adjectives, and 5% are adverbs, closely mirroring the distribution found in the overall dataset described above.

heading: D Grammar Error Correction on TOEFL-11
text: Because the sentences in ProLex are extracted from low and medium-level essays from TOEFL-11 cor-
pus, these sentences frequently contain basic grammar errors that obscure the intended meaning of the context. To alleviate this problem, we first follow Rothe et al. (2021) and tested a T5 base model trained on cLang-8 on the CoNLL-14 test set. Finding its performance unsatisfactory, we further fine-tuned the model on ErAConD (Yuan et al., 2022) which slightly improved it. At last, we chose to fine-tune a GPT-2 model (Radford et al., 2019) on the proposed cLang-8 dataset, which provided a satisfactory F0.5-score as shown in table 7, allowing us to correct most grammatical errors in the TOEFL-11 essays.
In Table 8, we provide some examples to show the original sentences from TOEFL-11 and the corresponding corrected sentences output from our grammar model.

heading: E Details of Baseline Experiments
text: The details of the baseline experiments are presented in this section. We measure the performance of the following baseline systems: 1) zero-shot LLMs prompting, 2) instruction tuning with taskspecific synthesized data, and 3) current state-ofthe-art lexical substitution systems. In the following, the setups for each of these baselines are illustrated in great detail.
E.1 Zero-shot and In-context Learning with GPT-4 and ChatGPT
We evaluate the zero-shot performance of GPT-4 and ChatGPT on ProLex. The prompt we used in the experiment is shown in Table 12. Specifically, we used the model provided by OpenAI platform, namely gpt-4 and gpt-3.5-turbo-1106, to evaluate the performance of GPT-4 and ChatGPT, respectively. For each (w, s) pair, GPT-4 and Chat-
GPT outputs a list of substitutes that are separated by commas. In our evaluation, we take the top-k (k = 10) substitutes from the outputs and compare them with our labels in ProLex. Similarly, we also measure the performance of GPT-4 and ChatGPT under in-context learning settings. We randomly select 32 examples from the synthetic dataset DL and use these as in-context examples for both GPT-4 and ChatGPT. The prompt we used for this experiment is shown in Table 14.
E.2 Instruction tuning experiments
In our instruction tuning experiments, we conducted evaluations on the 7B and 13B variants of Vicuna and Llama-2. To synthesize task-specific training data, we applied two approaches: 1) generate data with GPT-4 and 2) modify the existing lexical substitution benchmark Swords. In the following, we will present detailed descriptions of the two approaches, along with the specific experimental settings used in the fine-tuning process.
E.2.1 Synthesizing task-specific training data Generate data from GPT-4 We start by selecting the context sentences from the TOEFL-11 dataset. Concretely, we randomly select 500 sentences from each proficiency level,11 rendering a total of 1, 500 sentences. We also make sure that the selected sentences are not in ProLex. Similar to section 3.1, we perform grammar error correction to the sentences extracted from low and mediumlevel essays, while sentences from high-level essays are retained in their original form. Subsequently, for each sentence, we prompt GPT-4 to select a target word based on a randomly pre-defined
11Note that there are three proficiency levels in the TOEFL11 dataset, namely high, medium and low.
part-of-speech tag and generate the proficiencyoriented substitutes. The complete prompt for this process is shown in Table 13. Specifically, we provide five in-context examples to guide the generation process. As a result, we take the final substitutes generated for each sentence as the proficiency-oriented substitutes. The resulting synthesized data is post-processed to fine-tune Vicuna 7B/13B and Llama-2 7B/13B models. In total, after post-processing,12 the synthesized dataset contains 1, 383 unique (w, s) pairs, with 6, 375 candidate substitutes and 4, 982 final substitutes. The partof-speech tags for all selected target words are uniformly distributed, with 361 nouns, 335 verbs, 357 adjectives, and 330 adverbs.
Modify existing benchmark Another way to synthesize training data is to modify the dataset provided by the Swords benchmark, filtering out lower proficiency level substitutes. Concretely, we only consider acceptable substitutes (i.e. score greater than 50%) in Swords, and perform filtering based on their CEFR levels.
The Swords dataset, with 1132 target word and context pairs, have an average of 60.7 substitutes per target, of which 4.1 are acceptable on average. After filtering out substitutes with a lower CEFR proficiency level than the target word, the dataset now contains an average of 3 substitutes per target. The average CEFR level of targets is A2, whereas the average level of substitutes after filtering is B1, indicating improvements on proficiency-level. The
12The post-processing involves dropping null values and duplicated rows.
CEFR level distributions of the target words and proficiency-oriented substitutes from the filtered dataset DS are shown in Figure 4.
E.2.2 Experiment setup We established our training pipeline based on the platform developed by (Zheng et al., 2023). Concretely, for the 7B models, we fine-tuned both Vicuna and Llama-2 for a maximum of 10 epochs each, using a single NVIDIA A100-80G GPU. For the 13B models, each variant was trained for up to 5 epochs on two NVIDIA A100-80G GPUs. In all cases, we configured the training batch size per device to 1 and established the initial learning rate at 1e-5, employing a linear learning rate scheduler. The best checkpoints were selected based on the performance on a separate development set of ProLex.

heading: Future Work
text: 2, we can iteratively update and enhance ProLex with additional gold standard substitutes, thereby expanding its coverage in future versions. Limits of CEFR Checker The CEFR Checker (CathovenAI, 2023) we used in this work is capable of assigning CEFR levels only at the word level. However, we noticed that in some cases, phrases could also serve as appropriate substitutes. Hence, extending the CEFR Checker’s functionality to include CEFR-level assignments at both the word and phrase levels would offer a more holistic evaluation in future versions of ProLex. Phrases and Multi-word Substitutions In this work, we posit that the integration of multi-word expressions, such as idiomatic phrases, can enhance English language proficiency (Thyab, 2016; Yunus and Hmaidan, 2021; Al-Khawaldeh et al. , 2016). Therefore, we treat all acceptable phrasal substitutes as valid substitutes for demonstrating better language proficiency compared to the single target word. In future research, we intend to investigate the methods for assessing proficiency levels in the use of phrases and multi-word expressions