Smart Word Suggestions for Writing Assistance
Year: 2023

Abstract:
Enhancing word usage is a desired feature for writing assistance. To further advance research in this area, this paper introduces "Smart Word Suggestions" (SWS) task and benchmark. Unlike other works, SWS emphasizes end-to-end evaluation and presents a more realistic writing assistance scenario. This task involves identifying words or phrases that require improvement and providing substitution suggestions. The benchmark includes human-labeled data for testing, a large distantly supervised dataset for training, and the framework for evaluation. The test data includes 1,000 sentences written by English learners, accompanied by over 16,000 substitution suggestions annotated by 10 native speakers. The training dataset comprises over 3.7 million sentences and 12.7 million suggestions generated through rules. Our experiments with seven baselines demonstrate that SWS is a challenging task. Based on experimental analysis, we suggest potential directions for future research on SWS. The dataset and related codes is available at https://github. com/microsoft/SmartWordSuggestions.

heading: 1 Introduction
text: Writing assistance is a widely used application of natural language processing (NLP) that helps millions of people. In addition to common features like grammatical error correction (Ng et al., 2014; Bryant et al., 2017), paraphrasing (Fader et al., 2013; Lin et al., 2014) and automatic essay scoring (Song et al., 2020), providing word suggestions is a desired feature to enhance the overall quality of the writing. As illustrated in figure 1, the word “intimate” in the first sentence should be replaced with “close”, as “intimate” is not suitable for describing relationships between colleagues.
In this paper, we introduce the task and benchmarks of Smart Word Suggestion (SWS). Figure
∗This work was performed during the first author’s internship at Microsoft Research Asia
†Corresponding Author
2 shows the definition of SWS. The goal of SWS is to identify potential improvable targets in the form of words or phrases within a given context, and provide substitution suggestions for every improvable target. These suggestions may include correcting improper word usage, ensuring that language usage conforms to standard written conventions, enhancing expression, and so on. Specifically, we categorize these suggestions into two types: refine-usage and diversify-expression.
Lexical Substitution (LS) (McCarthy and Navigli, 2007; Kremer et al., 2014; Lee et al., 2021) is the most relevant research benchmark in the field. LS systems aim to provide substitute words that
11212
maintain the original meaning of a given word within a sentence. However, in practical situations, it is important to recognize words that can be improved or replaced. Identifying these targets is crucial for practical use and a necessary step for making accurate substitution suggestions. In order to reproduce the real-world scenarios, we design SWS as an end-to-end process that takes a sentence as input and provides substitution suggestions for all improvable targets as output.
The SWS benchmark includes human-labeled data for testing, a large distantly supervised dataset for training, and a corresponding framework for evaluation. For testing, we collect 1,000 segments from English learners’ essays, and ask ten annotators to identify improvable targets and provide substitution suggestions. The high level of agreement among the annotators confirms the quality of the annotation. For weakly supervised training, we compile a large amount of distantly supervised data by using a synonym thesaurus to randomly substitute words in corpus. We also provide settings for both end-to-end evaluation and sub-task evaluation.
To investigate the challenges, we implemented seven baselines, including knowledge-driven methods, state-of-the-art lexical substitution methods, and end-to-end approaches for SWS. The experimental results show that the performance of the existing lexical substitution methods decreases significantly when applied to SWS. Additionally, the end-to-end methods we designed struggle to identify and improve targeted words or phrases. Detailed analysis and discussions on the results suggest several areas for further research.
To conclude, our contributions are as follows: • Introducing the SWS task for writing assis-
tance, and providing a benchmark with highquality human-labeled testing data and large
distantly supervised training data. • Developing the evaluation framework for
SWS, and conducting extensive evaluations on the provided baselines.
• Identifying several directions for further research on SWS through analysis.

heading: 2 Related Works
text: We begin by comparing SWS with three related tasks, highlighting the unique value of our work.

heading: 2.1 Lexical Substitution
text: Lexical substitution (LS) (McCarthy and Navigli, 2007; Kremer et al., 2014; Lee et al., 2021) is the task of providing substitute words for a specific word in a sentence. There are some major distinctions between the SWS and LS.
(1) In LS, the target word is already provided, while in SWS, the system needs to detect the improvable targets first.
(2) LS focuses on finding synonyms that maintain the meaning of both the word and the sentence. On the other hand, SWS is designed for writing assistance scenarios, so the substitutions aim to improve the writing of the sentences. LS focuses on word sense disambiguation in the context, which doesn’t require any "improvement". Here is an example in the LS07 dataset: This is clearly a terrible and shameful blot on UN peacekeeping. One of the substitutions is "terrible" → "very bad". This substitution doesn’t meet the SWS’s requirement as the use of "very bad" is less accurate, and the substitution worsens writing.
(3) LS uses lemmatized annotations for the target word and substitutions, while SWS extracts annotations directly from the sentence and requires that the substitutions fit grammatically within the
sentence to evaluate the model’s end-to-end performance.

heading: 2.2 Grammatical Error Correction
text: Grammatical error correction (GEC) (Ng et al., 2014; Bryant et al., 2017) also shares some similarities with SWS. Ng et al. (2014) pointed that more than 85% of the corrections in GEC are word-level and that these corrections improve users’ writing as well. However, the substitution suggestions provided by SWS do not include suggestions for correcting grammatical errors. Instead, SWS focuses on identifying and improving word or phrase usage. It is worth noting that the source sentences in the SWS test set are first processed by a GEC model (Ge et al., 2018) and then further checked by human annotators to ensure no grammatical errors in the inputs. In the writing assistant, SWS is the next step following GEC.

heading: 2.3 Paraphrase Generation
text: Paraphrase generation (PG) (Fader et al., 2013; Lin et al., 2014) aims to alter the form or structure of a given sentence while preserving its semantic meaning. PG has a variety of potential applications, such as data augmentation (Iyyer et al., 2018), query rewriting (Dong et al., 2017), and duplicate question detection (Shah et al., 2018). PG is different from SWS in two main ways: (1) SWS places a greater emphasis on improving writing by identifying and correcting inappropriate word usage or providing diverse expression options. (2) SWS focuses on substitution suggestions of words or phrases, and evaluations are based on word level. In contrast, PG directly measures performance at the sentence level.

heading: 3 Data Collection
text: This work is to construct a Smart Word Suggestion benchmark that accurately represents writing assistance scenarios. For evaluation, we collect sentences from English learners and use human annotations in accordance with McCarthy and Navigli (2007) and Kremer et al. (2014). For training, we compile a large-scale, distantly supervised dataset from Wikipedia (Erxleben et al., 2014; Vrandečić and Krötzsch, 2014).

heading: 3.1 Human-Annotated Data Collection
text: Human-annotated data is obtained through a threestage process: (1) cleaning corpus data from En-
glish learners’ essays, (2) labeling improvable targets and corresponding substitution suggestions, and (3) merging annotations and filtering out lowconfidence annotations.
Stage 1: Corpus Cleaning. We collect essays written by undergraduate English learners via an online writing assistance platform 1. We divide them into individual sentences. To avoid annotators making corrections beyond SWS, the sentences are refined with following actions: (1) removing sentences that have unclear meanings. (2) applying a correction model (Ge et al., 2018) to correct grammatical errors. (3) asking human reviewers to double-check for any remaining grammatical errors. Additionally, we filter out short sentences as they may not provide enough context or contain sufficient words to improve. We thoroughly reviewed all sentences to ensure that they do not contain any information that could identify individuals or any offensive content.
Stage 2: Human Annotation. Ten native English-speaking undergraduate students majoring in linguistics were recruited as annotators to independently annotate each sentence. To ensure annotation quality, all annotators were required to pass test tasks before participating in the annotation.
The annotators carried out the annotations in three steps: (1) identifying words or phrases in the sentence that could be improved, (2) offering one or more suggestions for each identified target, and (3) assigning a type of improvement after the substitution.
Specifically, we define the substitution suggestions as two types. (1) Refine-usage refers to instances where the use of a specific word or phrase is inappropriate in the current context, such as when it has a vague meaning, is a non-native expression, or is an incorrect usage of English. For instance, in the second sentence shown in figure 1, the word "possible" is intended to convey the meaning of "having the possibility", and is not appropriate in the context of the sentence. The annotators replaced "possible" with "likely." These suggestions are designed to help English learners understand the differences in word usage in specific contexts and to enable them to write in a way that is more consistent with native speakers. (2) Diversify-expression refers to instances where this word or phrase could be substituted with other words or phrases. These
1https://aimwriting.mtutor.engkoo.com/
suggestions aim to help users use a more diverse range of expressions. The last case in figure 1 is a corresponding example.
The annotators were required to provide at least three suggestions for each sentence. For the entire dataset of 1000 sentences, each annotator was required to provide at least 1500 refine-usage type suggestions. The detailed annotation instruction is in appendix A.
Stage 3: Merging and Filtering. Previous lexical substitution tasks (McCarthy and Navigli, 2007; Kremer et al., 2014) merged all the annotators’ results into a key-value dictionary, where the value indicates the number of annotators who provided this substitution suggestion. We merged the labeling results of 10 annotators in a similar way. Take the merging of two annotators’ annotations as an example. One is {happy: glad/merry, possible: likely}, and the other is {help: aid, possible: likely/probable}. The result after merging would be:
{happy: {glad: 1, merry: 1}, possible: {likely: 2, probable: 1}, help: {aid: 1}}
where happy, possible, help are improvable targets, and the sub-level dictionaries are the substitution suggestions after merging. We also collect the type of refine-usage or diversify-expression for each improvable target by taking the majority of the type labeling.
In order to reduce subjective bias among annotators, we discarded all improvable targets that were only annotated by one annotator. Finally, the dataset was split into a validation set of 200 sentences and a test set of 800 sentences.

heading: 3.2 Distantly Supervised Data Collection
text: We collect a large amount of distantly supervised data for weakly supervised training by using a synonym thesaurus to randomly substitute words in a corpus. The source corpus contains 3.7 million sentences from Wikipedia2 . The synonym thesaurus we use is the intersection of PPDB (Pavlick et al., 2015) and Merriam-Webster thesaurus3. The sentences are processed in 3 steps: (1) Selecting all the words or phrases in the synonym thesaurus, and treating them as improvable targets. (2) Using a tagger to find the part of speech of the improvable targets. (3) Randomly substituting the improv-
2https://dumps.wikimedia.org/enwiki/20220720/ 3https://www.merriam-webster.com/thesaurus
able targets with one synonyms of the same part of speech.
Note that the random substitution with the synonym dictionary may result in a more inappropriate word or phrase usage than the original text. Therefore, we treat the generated substitutions as the improvable targets, and the original targets as substitution suggestions.
In contrast to the human-annotated dataset, the distantly supervised dataset only includes one suggestion for each improvable target and does not have the annotation of suggestion type. The code for generating distantly supervised datasets will be released for further studies.

heading: 3.3 Data Statistics
text: Benchmark # Sentence # Target # Suggestion # Label
Table 1 shows the comparison between SWS and lexical substitution benchmarks. Our SWS dataset consists of 7027 instances of improvable targets and 16031 suggestions in 1000 sentences. The average length of the sentences in this dataset is 27.8 words. The improvable targets in this dataset includes 2601 nouns, 2186 verbs, 1263 adjectives, 367 adverbs, 267 phrases, and 343 other parts of speech. 3.8% of the targets and 3.3% of the suggestions are multi-word phrases. 63.0% of the targets are the type of refine-usage. Table 2 shows the proportion of refine-usage or diversify-expression targets with different part-of-speech.
The distantly supervised dataset SWSDS contains over 12.7 million suggestions in 3.7 million
sentences. 2.67% are multi-word phrases, and 0.3% of the suggestions are multi-word.

heading: 3.4 Inner Annotator Agreements
text: Previous studies on lexical substitution(McCarthy and Navigli, 2007; Kremer et al., 2014) evaluated the quality of the dataset with inter-annotator agreement (IAA). We adopt this approach and calculate pairwise inter-annotator agreement (PA) to assess the quality of the dataset. PAdet measures the consistency of identifying improvable targets:
PAdet = 1 |P | ∑
(i,j)∈P PAdetij
PAdetij = N∑
k=1
1
N
|sik ∩ s j k| |sik ∪ s j k|
where P is the set of annotator pairs. We have ten annotators, so |P | = C210 = 45. N is the number of all the sentences, and sik, s j k are the improvable target sets of sentence k identified by annotator i and j, respectively. PAsug measures the consistency of substitution suggestions of a same improvable target:
PAsug = 1 |P | ∑
(i,j)∈P PAsugij
PAsugij =
Mij∑
l=1
1
Mij
|til ∩ t j l | |til ∪ t j l |
where Mij is the size of the intersection of the improvable target sets identified by annotator i and j. til, t j l are the suggestions for target l given by annotator i and j, respectively. In the SWS benchmark, the PAdet and the PAsug are 23.2% and 35.4%, respectively. Our PAsug is significantly higher compared to previous LS datasets, 27.7% of SemEval (McCarthy and Navigli, 2007) and 19.3% of COINCO (Kremer et al., 2014), thereby confirming the annotation quality.

heading: 3.5 Data Quality of the Distantly Supervised Dataset
text: According to our statistics, 71.8% of the substitutions in the test set appear in the training set, and each substitution in the test set appears in the training set 10.4 times on average. Those data show the substitutions in the training set covers most of the substitutions in the test set, which verify the synthetic method is close to real-world scenarios.

heading: 4 Evaluation
text: In this section, we introduce the evaluation settings and metrics for SWS, including both the end-to-end evaluation and the sub-task evaluation.
For the end-to-end evaluation and the improvable target detection sub-task, we introduce precision, recall, and F0.5 as metrics. For the substitution suggestion sub-task, we utilize accuracy to evaluate the quality of the predicted substitutions. Examples of calculating the metrics can be found in appendix B.

heading: 4.1 End-to-end Evaluation
text: The end-to-end evaluation is computed based on each substitution suggestion. A true prediction is counted if and only if both the detected improvable target is in the annotated improvable target set and the suggested substitution is in the annotated substitutions of the target:
TPe2e = N∑
k=1
Mk∑
l=1
1 if skl ∈ Sk else 0
where N is the number of all the sentences, Mk is the number of targets in the sentence k, Sk is the set of annotated suggestions of sentence k, and skl is the l-th predicted suggestion of sentence k. The precision (Pe2e) and recall (Re2e) for end-to-end evaluation are calculated as follows:
Pe2e = TPe2e
NP , Re2e =
TPe2e
NG
where NP and NG are the number of predicted suggestions and annotated suggestions, respectively. In the writing assistance scenario, precision is more important than recall, so we calculate Fe2e0.5 as the overall metric.
Fe2e0.5 = 1.25 ·Pe2e ·Re2e 0.25 ·Pe2e +Re2e

heading: 4.2 Sub-Task Evaluation
text: Improvable Target Detection. In this task, model needs to find all the annotated improvable targets in the sentence. The precision (Pdet) and recall (Rdet) for detection are calculated as follows:
Pdet = ∑N k=1 |sk ∩ s′k|∑N
k=1 |s′k| , Rdet =
∑N k=1 |sk ∩ s′k|∑N
k=1 |sk|
where sk and s′k are the annotated improvable target set and predicted improvable target set for sentence k, respectively. Same with end-to-end evaluation, we compute Fdet0.5 to assess the performance for detection of improvable targets.
Fdet0.5 = 1.25 ·Pdet ·Rdet 0.25 ·Pdet +Rdet
Substitution Suggestion. In this task, model needs to give suggestions for each improvable target. We calculate accuracy of the suggestions on those correctly detected targets:
Accsug = 1
N
N∑
k=1
( 1
Mk
Mk∑
l=1
1 if t′l ∈ Tl else 0 )
where Tl is the annotated recommendation set of target l, t′l is the predicted recommendation for target l, and Mk is the total number of correctly detected targets in sentence k.

heading: 5 Experiments

heading: 5.1 Baselines
text: We test 7 methods on SWS. The methods could be divided into three groups: (1) Adopting external knowledge to give suggestions. (2) State-of-the-art lexical substitution methods. (3) End-to-end SWS baselines. We also list the human performance for reference.
External Knowledge Methods. Here are two methods that use external knowledge to give suggestions. (1) Rule-based synonyms replacement as how we construct the distantly supervised data. We adopt a greedy replacement strategy, where all entries are replaced. (2) ChatGPT4, a large language model trained on massive data and further fine-tuned with human feedback. We ask ChatGPT to directly generate the suggestions in every giving sentence. The prompt and details for utilizing ChatGPT can be found in appendix C.
Lexical Substitution Methods. Two state-of-theart lexical substitution methods are tested on SWS, i.e. BERTsp,sv (Zhou et al., 2019) and LexSubCon (Michalopoulos et al., 2022). We use the opensourced code of LexSubCon and re-implement BERTsp,sv . We let the model give a substitution for each word, and if the substitution is different with the original word, the word is regarded as a detected improvable target.
4https://openai.com/blog/chatgpt/
End-to-end Baselines. In the end-to-end framework, we treat SWS as three training paradigms, and provide one baseline for each. (1) Masked language modeling (MLM): We use BERT-baseuncased (Devlin et al., 2019) with an MLM head as the baseline. (2) Sequence-to-sequence generation: We use BART-base (Lewis et al., 2020) as the baseline. (3) Token-level rewriting: We use CMLM (Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer, 2019) as the baseline. The distantly supervised dataset is utilized to train the end-to-end baselines. For the improvable targets, the model is expected to learn the suggestions. Otherwise, the model is expected to keep the original words.

heading: 5.2 Main Results
text: Table 3 shows the experimental results of the baselines, from which we have the following observations:
(1) The rule-based approach is similar to the process of creating distantly supervised data. Both the rule-based method and end-to-end baselines, which are trained using distantly supervised data, have high Pdet and low Rdet values. This suggests that the synonym dictionary used in this work has high quality but low coverage.
(2) Compared with the rule-based method, the end-to-end models trained on distantly supervised dataset show a decline in performance for the improvable target detection, but an increase in performance for substitution suggestion. The improvable targets of the distantly supervised data do not accurately reflect the words or phrases that need improvement, resulting in difficulty in effectively training the models in detecting. However, the substitution suggestions in the distantly supervised data are derived from original words in Wikipedia, enabling the models to learn a relatively appropriate word usage in context.
(3) The results of the CMLM model show a decrease in performance compared to the pre-trained models, namely BERT and BART, particularly in terms of substitution suggestions. The pre-training of semantic knowledge may contribute to the superior performance of the pre-trained models for this task.
(4) There is a notable decrease in SWS for LS methods. Moreover, different LS methods have significant differences in detecting improvable targets. Only 2.1% of the words in the input sentence
are identified as improvable targets by BERTsp,sv , while LexSubCon detects 32.4%. The current LS methods are not compatible with the SWS task.
(5) The results from ChatGPT are comparable with the end-to-end baselines trained on 3.7 million sentences, but it is still has room for improvement.
(6) Human performance is significantly better than baselines. We believe there is a lot of room for the baselines to improve.

heading: 6 Analysis
text: We analyze the experimental results with two questions: (1) Does the model have the capability to accurately identify words that require improvement, or does it simply make random guesses? (2) Does the model have the ability to provide multiple useful suggestions for each target word?

heading: 6.1 Detection Analysis
text: Voting Index and Weighted Accuracy. After merging the annotations, we determine the voting index for each improvable target, i.e. the number of annotators who identified the word or phrase. The voting index reflects the necessary level of replacement for the word. Figure 3 shows Rdet for the improvable targets with different voting indexes. As depicted in Figure 3, improvable targets identified by a greater number of annotators are more easily detected by the models.
Then, we design weighted accuracy (WA) to evaluate the detection performance, using the voting index as weighting factors.
WAdet =
∑N k=1 ∑Mk l=1wkl if skl ∈ s′k else 0∑N
k=1 ∑Mk l=1wkl
where s′k is the predicted improvable target set of sentence k, skl is the l-th annotated target in sentence k, wkl is the voting index of skl, N is the number of total sentences, and Mk is the size of annotated improvable target set of sentence k.
Table 4 shows Rdet and WAdet of baseline methods. Consistent with the trend of Rdet for different voting indexes, the WAdet is relatively higher than Rdet. These results demonstrate that the baseline methods can detect the highconfidence improvable targets better.
Improvable Ratio. The improvable ratio (ImpR) is defined as the proportion of the number of detected improvable words to the total number of words in sentences. As shown in Table 4, Rdet, WAdet are positively correlated with ImpR.
To investigate how to control the model to achieve a desired ImpR, we build another distantly supervised dataset for training. Different from dataset construction described in section 3.2, we use the union of PPDB (Pavlick et al., 2015) and Merriam-Webster thesaurus as a large synonym thesaurus. As the thesaurus size increases, the artificial improvable targets in constructed data are increased to 25.4% from 13.2%.
The results of BERT trained on two datasets are presented in Table 5. Upon comparison of the two experiments, it is observed that the number of constructed improvable targets in the training set is nearly doubled, while the ImpR of the trained models only increases to 13.6% from 9.3%. It is challenging to control the ImpR. Thus, one direction under research is to control the model to attain a desired ImpR while maintaining a good performance.

heading: 6.2 Multiple Suggestions Analysis
text: It may be beneficial for users to have multiple suggestions for each improvable target. Therefore, we design a multiple-suggestion setting that allows the system to provide multiple substitution suggestions for each detected improvable target.
As the output suggestions are ranked in order, we propose using Normalized Discounted Cumulative Gain (NDCG), a metric commonly used in search engines, to measure the similarity between a ranked list and a list with weights.
NDCGm = 1
M
M∑
k=1
DCGm(T′k) DCGm(Tk)
DCGm(Tk) = m∑
i=1
∑ i′≤iwi
log(1 + i)
DCGm(T′k) = m∑
j=1
∑ j′≤j w ′ j
log(1 + j)
w′j = wi if t ′ kj ∈ Tk else 0
In this formula, M is the total number of true predicted improvable targets, and m is a parameter that specifies the number of suggestions for an improvable target. In the numerator, we accumulate the weights for the predicted suggestions from the first to the last. If recommendation i′ is not in human annotation, the weight is set to zero. Otherwise, the weight is set to its voting index. The denominator is a list sorted according to the voting index, which represents the optimal condition for giving m predictions. We provide an example of calculating NDCG in appendix D.
The average number of substitution suggestions for each improvable target in SWS benchmark is 3.3. When m exceeds the substitution number for a given target, DCGm(Tk) remains constant. Thus, NDCGm is only calculated for m = 1, 2, 3, 4. Figure 4 lists NDCGm for different baselines.
BERT may perform better than other methods, but as the number of suggestions m increases, the NDCGm of BERT drops significantly. This suggests that BERT struggles when providing multiple suggestions. This could be due to the lack of multiple substitution suggestions in the distantly supervised dataset. Future research could focus on improving the model’s ability to provide multiple substitution suggestions.

heading: 6.3 Case Study
text: Figure 5 gives two cases of BERT’s predictions. In the first case, BERT didn’t detect this improvable target. However, in our distantly-supervised training data, there are dozens of cases substituting "situations" to "circumstances". We think controlling the initiative of detecting is a direction worthy of research.
In the second case, BERT give the suggestion of "understanding", which is the closest word to "knowing" if ignores the context. However, it’s not the right meaning in the context of "knowing events". We think it’s hard to train a model aware of word usage in different contexts with the current distantly-supervised training data. Because we think the one-substitute-one data doesn’t provide enough information for model training on word usage. We regard this as a future research direction.

heading: 7 Conclusion
text: This paper introduces the first benchmark for Smart Word Suggestions (SWS), which involves detecting improvable targets in context and suggesting substitutions. Different from the previous benchmarks, SWS presents a more realistic representation of a writing assistance scenario. Our experiments and analysis highlight various challenges for future research and suggest opportunities for improvement in future work. We encourage further research on building more realistic training data, designing better data augmentation strategies, and developing
unsupervised or self-supervised methods for SWS.

heading: 8 Limitations
text: The SWS benchmark have two limitations: (1) The sentences in the SWS testing set come from students’ essays, which limits the system’s ability to test its performance in other specific domains such as laws or medicine. (2) the SWS corpus is at the sentence level, but some writing suggestions can only be made after reading the entire article, which are not included in our SWS dataset.

heading: A Annotation Instructions
text: We need to find at least 3 "words/phrases to change" in a sentence, and give "substitutes" for each. Every substitute should be classified as improve-usage or diversify-expression.
A.1 What is the word/phrase that needs to change?
Our aim is to find a word/phrase that needs to be better in writing scenarios. Suppose you are the teacher, and now you are helping the language learners to improve their English writing. We define a "word to change" as the substitution has influences as follows:
• To express the original semantic meaning more appropriately.
• To make the usage of the word much closer to the native speaker.
• To change spoken language into written language.
• To diversify the word usage for better expression.
The substitution should NOT cause the influence as follows:
• Rewrite the sentence, instead of words or phrases, into a better expression (e.g. "it is advisable" → "advisably,").
• Correct the mistakes in the sentence (e.g. "a lot" → "a lot of" in the sentence "There are a lot of valuable tips").
• Substitute the word with a synonym, but not help the English learners with better writing.
After the definition, we also give some rules that you could refer to:
• the word/phrase that needs to change is usually less than 3 words.
• the word/phrase that needs to change is usually an adj./adv./noun/verb.
• the word/phrase that needs to change is usually not a named entity.
A.2 How to give the substitutions?
The substitution should: • have the same semantic meaning as the "word
to change".
• keep the sentence’s meaning unchanged. Specifically, there are two scenarios for substitution: • If the word to change is general, and we can
clearly understand the sentence’s meaning. In this case, the substitution should be more precise. (e.g. "Schools in north-west China are our primary aiding individuals and we often start from our school when the summer vacation begins." "aiding"→"helping" is a good substitution)
• If the word to change is confusing, and we could only guess the sentence’s meaning. In this case, the substitution should be more general. (e.g. "Successful individuals are characterized by various merits including ..." "various"→"plentiful" is a bad substitution)
After the substitution, the sentence must be fluent as the original sentence. Errors in preposition collocations, tenses, and mythologies should be avoided. (e.g. "in a nutshell", "nutshell" → "essence" is not right, should be "in a nutshell" → "in essence")
A.3 Annotation Guidelines
• Substitutions in a grid should be connected with ";" (NOT ’,’ !).
• If the original sentence has grammar or typo problems, just discard the sentence.
• In the annotation table, the content in the column "word to change" should be EXACTLY THE SAME as the word/phrase in the original sentence, and there should not exist punctuation (except ";" to connect multiple substitutions)
• Substitute the smallest range of words, unless indivisible. (e.g. "I think you deserve it again" → "I think you deserve another chance" is a bad case, which should be "it again" → "another chance". "in a nutshell" → "in essence" is a good case, because "in a nutshell" is a phrase).
• We don’t need to paraphrase the sentence. • Please ensure that the "substitute" and "word
to change" have the same tense, plural forms, and part of speech.

heading: B Example of Evaluation Metrics
text: For example, given a sentence: “I am writing to answer the previous questions you asked.” The
annotation result of the sentence is as follows:
answer: {respond to: 3, reply to: 1}, writing: {connecting with: 3}, to answer: {in response to: 2}, questions: {queries: 2}
In improvable target detection, Sk is {answer, writing, to answer, questions}. If the prediction S′k is {answer, previous}, then Pdet = 1/2 and Rdet = 1/4.
In substitution suggestion metrics, take the true predicted target answer as an example. If the predicted suggestion is in {respond to, reply to, then Accsug = 1, otherwise Accsug = 0.
In end-to-end evaluation, if the predicted suggestions are {answer: respond, writing: connect with, asked: gave}, then Pe2e = 1/3 and Re2e = 1/4.

heading: C Prompt for ChatGPT
text: The prompt we use is as follows: In the following sentence, please give
some suggestions to improve word usage. Please give the results with the json format of “original word”: [“suggestion 1”, “suggestion 2”], and the “original word” should be directly extracted from the sentence. [s] where [s] is the sentence. Amazingly, ChatGPT can generate substitution suggestions with the keyvalue format. We use regular expression to extract the substitution suggestions. If the result is empty, we will re-generate until getting substitution suggestions.

heading: D Example of NDCG
text: Take an example of NDCG5: For a detected improvable target, if Tj with voting index is {respond to : 3, respond : 2, response : 1, reply to : 1} and T′j with order is {respond, respond to, tell, response, solution}, then DCG(T′j) and DCG(Tj) are calculated as follows, and NDCG5 = 4.4/5.1 = 86.3%.
Order Sub. Gain DCG5(T′k) 1 respond 2 2 = 2 × 1 2 respond to 3 3.9 = 2 + 3 × 0.63 3 tell 0 3.9 = 3.9 + 0 × 0.5 4 response 1 4.4 = 3.9 + 1 × 0.43 5 solution 0 4.4 = 4.4 + 0 × 0.39 Order Sub. Gain DCG5(Tk)
1 respond to 3 3 = 3 × 1 2 respond 3 4.2 = 3 + 2 × 0.63 3 response 1 4.7 = 4.2 + 1 × 0.5 4 reply to 1 5.1 = 4.7 + 1 × 0.43 5 NULL 0 5.1 = 5.1 + 0 × 0.39
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
section 8
3 A2. Did you discuss any potential risks of your work? section 3.1
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract & section 1
3 A4. Have you used AI writing assistants when working on this paper? We only use AI writing assistants to correct the grammar errors.
B 3 Did you use or create scientific artifacts? section 3, we use wiki data as the source corpus of training set
3 B1. Did you cite the creators of artifacts you used? section 3.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Wikipedia’s license is CC BY-SA, which is free to use / edit / share.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use wikipedia as the source corpus, which is common.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? section 3.1
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 3.3
C 3 Did you run computational experiments? section 5
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We use widely-known baselines.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
7 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? We use the default hyperparameters.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? section 3
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix A
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Due to insufficient space, it is not explained in the text. We found an outsourcing company SpeechOcean to provide labeling services for us, and the hourly salary is 30 dollars per working hour.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? section 3,4
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Our experiments and analysis highlight various challenges for future research and suggest opportunities for improvement in future work. We encourage further research on building more realistic training data, designing better data augmentation strategies, and developing
unsupervised or self-supervised methods for SWS