ParaLS: Lexical Substitution via Pretrained Paraphraser
Year: 2023

Abstract:
Lexical substitution (LS) aims at finding appropriate substitutes for a target word in a sentence. Recently, LS methods based on pretrained language models have made remarkable progress, generating potential substitutes for a target word through analysis of its contextual surroundings. However, these methods tend to overlook the preservation of the sentence’s meaning when generating the substitutes. This study explores how to generate the substitute candidates from a paraphraser, as the generated paraphrases from a paraphraser contain variations in word choice and preserve the sentence’s meaning. Since we cannot directly generate the substitutes via commonly used decoding strategies, we propose two simple decoding strategies that focus on the variations of the target word during decoding. Experimental results show that our methods outperform state-of-theart LS methods based on pre-trained language models on three benchmarks.

heading: 1 Introduction
text: Lexical substitution (LS) in context (Hintz and Biemann, 2016; Zhou et al., 2019; Arefyev et al., 2020) is an extremely powerful technology that can be used as a backbone of various NLP applications such as writing assistance (Lee et al., 2021), word sense disambiguation (McCarthy, 2002), and lexical simplification (Paetzold and Specia, 2016; Qiang et al., 2021a,b). Compared with traditional LS methods based on linguistic databases (e.g., WordNet) (Hassan et al., 2007; Yuret, 2007) or word embedding models (Melamud et al., 2015a,b), LS methods based on pretrained language models have made remarkable progress in generating substitutes by considering the context (Zhou et al., 2019; Qiang et al., 2021a; Michalopoulos et al., 2022; Seneviratne et al., 2022). These methods feed the sentence into BERT (Devlin et al., 2018)
∗Corresponding author.
or XLNet (Yang et al., 2019) to obtain the top probability words corresponding to the target word as the substitute candidates. However, they have the following two limitations.
(1) The predictability of words is greatly influenced by the surrounding context, with little regard for preserving the sentence’s meaning. As illustrated in Table 1, the utilization of pretrained models often leads to the generation of ill-suited words, such as "wet", "flat" and "cold", due to their contextual relevance and similarity to the target word.
(2) The utilization of subword techniques in pretrained models precludes the selection of multitoken words as substitutes, as they only generate the most probable single tokens. For instance, the words "desiccated" and "facilitated" would not be offered as a substitution for the target word "dry" as seen in Table 1.
To address the limitations mentioned above, we study how to generate substitutes via paraphrase modeling. Recent neural paraphrasers based on encoder-decoder framework (Wieting and Gimpel, 2017; Hu et al., 2019) produce fluent, meaningpreserving English paraphrases but contain variations in word choice. Therefore, our idea is whether we can decode the substitute candidates from the hidden representation of the target word. In this way, the substitutes are not only semantically consistent with the target word and fit in the context, but also can preserve the sentence’s meaning. The meaning-preserving properties of a paraphraser can aid in addressing the first limitation, while autoregressive paraphrasers can address the second limitation. To the best of our knowledge, paraphraser for LS task has not yet been explored, as current decoding methods focus on lexical variations within the entire sentence rather than the target word, resulting in a scarcity of appropriate substitutes for the target word.
To specifically focus on lexical variations of the
3731
target word during the decoding process, we propose two new decoding strategies. (1) Our first strategy, referred to as ParaLS, proposes fixing prefixes of the target word. This approach initiates the decoding process by mandating that the decoder begins with the target word’s prefixes in the sentence, to subsequently generate the probability distribution of the target word’s position. The words with the highest probabilities are then fixed and used when decoding the remaining words, with selected words of the target word’s position in the paraphrases being selected as substitute candidates. (2) The second strategy, referred to as ParaLS⋆, is proposed to address the oversight of suffixes in the first strategy. Inspired by NEUROLOGIC A⋆esque (Lu et al., 2022), which incorporates heuristic estimates of future cost, we adapt it to estimate of the words in suffixes.
To the best of our knowledge, ParaLS is the first LS method that can produce substitute candidates by considering preserving the sentence’s meaning. On three benchmarks, ParaLS and ParaLS⋆ achieve state-of-the-art performance across various evaluation metrics. Moreover, ParaLS⋆ without the step of substitute ranking outperforms all existing methods with the step of substitute ranking.
Additionally, we propose a novel strategy for the step of substitute ranking by text generation evaluation metrics BARTScore (Yuan et al., 2021)
and BLEURT (Sellam et al., 2020). Our method embeds each substitute into the original sentence to create an updated version. By using BARTScore and BLEURT to compute the relationship between the original and updated sentences, they can quantify the extent to which the meaning of the original sentence has been preserved by each substitute. Experimental results show that substitute ranking using only BARTScore outperforms the previous state-of-the-art ranking methods when the same substitution candidate lists are provided for two popular LS benchmarks. The code and the experimental results are source-opened in Github 1.

heading: 2 Related Work
text: Lexical Substitution. LS methods generally consist of two steps: substitute generation and substitute ranking. Previous LS methods utilize linguistic databases (e.g., WordNet) (Hassan et al., 2007; Yuret, 2007) or word embedding models (Melamud et al., 2015b,a; Qiang and Wu, 2021) to extract synonyms or highly similar words for a target word, and then sort them based on their appropriateness in context. These methods overlook the context of the target word while generating substitute candidates, thereby inevitably generating a plethora of irrelevant candidates that may impede the subsequent ranking phase.
Recent LS methods based on pretrained language models have attracted much attention (Zhou et al., 2019; Lacerra et al., 2021a; Michalopoulos et al., 2022), in which the pretrained BERT is the most widely used one. Zhou et al. (Zhou et al., 2019) apply dropout to the embeddings of target words, Michalopoulos et al. (Michalopoulos et al., 2022) propose a new mix-up embedding strategy by incorporating the knowledge of WordNet into the prediction process of BERT, and Lin et al. (Lin et al., 2022) proposed an auxiliary gloss regularizer module to BERT pre-training. Lacerra et al. (Lacerra et al., 2021b,a) tried to train pretrained language models by merging the development set of two LS datasets (CoInCo and TWSI). The current work (Arefyev et al., 2020; Seneviratne et al., 2022) sought to evaluate all existing pretrained language models, and found that combining the prediction of pretrained language models XLNet and Word2Vec achieved the best results.
Overall, pretrained language modeling-based LS methods consider contextual information of tar-
1https://github.com/qiang2100/ParaLS
get words when generating substitute candidates, but do not concern with the impact of applying substitutes on sentence meaning. In contrast to the aforementioned methods, we try to utilize the knowledge of a pretrained paraphraser to generate substitute candidates.
Lexical Substitution using Paraphrases. A few studies (Pavlick and Callison-Burch, 2016; Kriz et al., 2018) find substitute candidates for complex words from a large-scale paraphrase rule database, e.g., PPDB (Ganitkevitch et al., 2013) or its variations (Pavlick et al., 2015; Pavlick and Callison-Burch, 2016). A paraphrase rule database consists of large-scale lexical paraphrase rules (e.g., "berries→strawberries") that are extracted from large-scale paraphrase sentence pairs, such as ParaNMT (Wieting and Gimpel, 2017) or ParaBank (Hu et al., 2019). These methods do not take into account the context as linguistic resourcebased LS methods do. In this paper, we generate substitute candidates of target words using the pretrained paraphrase model instead of paraphrase rule databases or paraphrase databases.
Decoding Strategies. Paraphrase generation can be regarded as a monolingual machine translation task that transforms expressions of an input sentence while retaining its meaning (Wieting and Gimpel, 2017). Neural paraphrasers primarily rely on the encoder-decoder framework, achieving inspiring performance gains over the traditional approaches (Lu et al., 2022). Beam search decoding is the most common method for inference, which decodes the top-K sequences in a greedy left-toright fashion. When K is set to 1, beam search decoding is changed into greedy search decoding. In recent years, beam search decoding has had multiple variants to deal with various task-specific and diversity/fluency trade-off of outputs, such as noise beam decoding (Cho, 2016), iterative beam decoding (Kulikov et al., 2019), clustered beam decoding (Tam, 2020) and diverse beam decoding (Vijayakumar et al., 2018). To enable constrained generation, NEUROLOGIC A⋆esque (Lu et al., 2022) explicitly decodes future text to estimate the viability of different paths for satisfying constraints. In contrast to the above decoding strategies, our decoding strategies focus solely on enhancing the diversity of the target word’s variation.

heading: 3 Method
text: Given a given sentence x = {x1, x2, ..., xt, ..., xn} and the target word xt, we need a pretrained paraphraser based on an autoregressive model, instead of a pretrained language modeling like existing LS methods (Zhou et al., 2019; Michalopoulos et al., 2022; Seneviratne et al., 2022), e.g., BERT or XLNet. LS method consists of two steps: substitute generation and substitute ranking. After feeding sentence x into the paraphraser, we aim to extract substitute candidates for the target word xt by two novel decoding strategies (Section 2.2). Then, we rank the candidates to choose the most appropriate substitution without modifying the meaning of x (Section 2.3).

heading: 3.1 Motivation
text: Recent neural paraphrasers primarily rely on the encoder-decoder learning framework on a largescale paraphrase dataset, achieving inspiring performance gains over traditional methods (Meng et al., 2021; Kadotani et al., 2021). Many languages including English, French, German, Chinese, and Spanish own large-scale paraphrase datasets. For example, in English, ParaBank2 (Hu et al., 2019)
consists of 19,370,798 sentence pairs. Given an input sentence x and its corresponding paraphrase y, we consider standard left-to-right, autoregressive models, pθ(y|x) =∏|y|
t=1 pθ(yt|y<t, x), and omit x to reduce clutter. Decoding consists of solving,
y∗ = argmax Y⊆Y F (y) (1)
where Y is the set of all sequences, and the objective F (y) is log pθ(y).
If we want to generate multiple paraphrases of sentence x, the beam search decoding is widely used by the auto-regressive method, which maintains a beam of K possible generations, updating them incrementally by ranking their extensions via the model likelihood. Since beam search decoding aims to find the most-probable hypothesis for the whole sentence during decoding, it is difficult to extract multiple substitute candidates for the target word from the generated paraphrases, as shown in Figure 1(a).
Since beam decoding concerns lexical variations of the whole sentence instead of the target word, there are no sufficient appropriate substitutions that can be discovered for the target word if we directly extract the substitute candidates from the paraphrases using the beam decoding. We will propose two novel decoding strategies, ParaLS and ParaLS⋆, for the paraphraser that are specifically engineered to harness lexical variations of the target word.

heading: 3.2 Substitute Generation
text: Substitute generation aims to generate substitute candidates for the target word based on its context. We will generate the candidates during the process of decoding.
The process of decoding method can be treated as a discrete search, in which states are partial prefixes, y<t, actions are tokens in vocabulary V (i.e., yt ∈ V), and transitions add a token to prefixes, y<t ◦ yt. Each step of decoding consists of (1) expanding a set of candidate next-states, (2) scoring each candidate, and (3) selecting the best K candidates.
(1) ParaLS: Decoding by Fixing Prefixes. Given a sentence x and a target word xt, we force the decoder to begin with prefixes x<t of the target word, and decode succeeding token yt to estimate the probability distribution of the vocabulary
p(yt|x<t). We select the top K tokens Yt with the highest probability in the distribution as the results of decoding.
Y ′ t = {y<t ◦ yt|y<t = x<t, yt ∈ V}
Yt = arg topK (y<t◦yt)∈Y ′t
{f(y<t, yt)} (2)
where f(·) is scoring function that approximates the objective F .
The decoding phase by fixing prefixes x<t is crucial to generate substitute candidates since we forcibly generate K different tokens Yt with the highest probability. In this case, these generated tokens are not only semantically consistent with the target word and fit in the context, but also preserve the sentence’s meaning. Since one word may comprise two or more tokens, we adopt greedy search decoding to select the token that has the maximum probability for each preceding token, until reaching the end symbol "EOS" of one sentence. These K words are considered as substitute candidates, after eliminating the morphological derivations of the target word. As depicted in Figure 1(b), our decoding strategy concentrates on lexical variations of the target word.
(2) ParaLS⋆:Decoding with Lookahead Heuristics. ParaLS, by fixing prefixes, takes into account only prefixes x<t without accounting for suffixes x>t. In this manner, the top K tokens Yt by Equation (2) may be one word of suffixes. Drawing inspiration from the A∗ search algorithm (Hart et al., 1968) and NEUROLOGIC A⋆esque (Lu et al., 2022), ParaLS⋆ will incorporate an estimate of the words of suffixes into the prediction of p(yt|x<t), replacing Equation 2 with:
Yt = arg topK (y<t◦yt)∈Y ′t
{maxF (y<t, yt,y>t} (3)
where x>t represents suffixes, y<t equals x<t, and y>t equals x>t.
ParaLS⋆ enhances the ParaLS scoring function by incorporating an estimate of suffixes satisfaction. Our key addition is a lookahead heuristic that adjusts a candidate (y<t,yt)’s score proportional to the probability of satisfying additional suffixes constraints y>t. In reality, we need only estimate two or three words in suffixes without estimating suffixes.
Intuitively, our lookahead heuristic for decoding brings two benefits. (1) The yt can be a token that
would satisfy a multi-token constraint or a phrase as the lookahead computes the average score (yt,y>t). (2) When yt is one word in suffixes, the lookahead will help to decrease its score, thereby precluding it from being among the top K tokens.

heading: 3.3 Substitute Ranking
text: After obtaining substitute candidates, existing LS methods (Zhou et al., 2019; Lacerra et al., 2021b; Seneviratne et al., 2022) obtain a contextualized representation of each substitute by replacing the target word with the substitute, and rank the substitutes by computing the cosine similarity of the target word vector with respect to that of each substitute. The similarity between the target word and the substitute does not provide sufficient information about whether the substitute will modify the sentence’s meaning. After replacing the target word of the original sentence with the substitute to form the updated version, we attempt to evaluate the original sentence x and the updated sentence to rank the substitutes, as opposed to the target word and the substitute alone.
We formulate evaluating updated sentence as a text generation evaluation task. Assume that the updated sentence is denoted as x′ after replacing the target word xt in x into one substitute. To accurately calculate a similarity score between x and x′, we found BARTScore (Yuan et al., 2021) and BLEURT (Sellam et al., 2020) are specifically designed for text generation tasks, which aligns with the goal of lexical substitution. Therefore, they could be used to measure the quality of the substitutes.
BARTScore is a neural network-based evaluation metric that compares the likelihood of the original sentence and the updated sentence. It can assign higher scores to the sentences that are more likely to be original sentences. BLEURT is also a neural network-based evaluation metric, which is trained to predict how human-like a text is by comparing it with a large dataset of human-written texts. Those two metrics could assign a similarity or dissimilarity score, which allow the ranking of the substitutes based on how much similar to the original sentence they are, which might be better to rank the substitutes rather than other ranking methods (Zhou et al., 2019; Lacerra et al., 2021b; Seneviratne et al., 2022).
We have also incorporated the prediction scores of the substitute candidates generated by the para-
phraser. Ultimately, our method employs a linear combination of the aforementioned three features (Paraphraser, BARTScore, BLEURT) to compute the final score for each substitute candidate.

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: LS Benchmarks. Two widely used datasets, LS07 (McCarthy and Navigli, 2007) and CoInCo (Kremer et al., 2014), are chosen for the evaluation of LS methods. We also adopt the latest LS benchmark, Stanford Word Substitution Benchmark (SwordS) (Lee et al., 2021), which extends and improves CoInCo via crowdsourcing annotators in Amazon Mechanical Turk. Each instance in LS dataset is composed of a target word, its context, and corresponding substitutes. LS07 consists of 300 development examples and 1710 test instances for 201 polysemous words. CoInCo consists of 15K target instances with a given 35% development and 65% test. SwordS contains 762 test instances.
Metrics. For evaluating LS07 and CoInCo, we use the official metrics "best", "best-m", "oot", "ootm" in SemEval 2007 task as well as Precision@1 (P@1) as our evaluation metrics, following the previous LS methods (Zhou et al., 2019; Michalopoulos et al., 2022). Among them, "best", "best-m" and "P@1" evaluate the quality of the best predictions, while both "oot" (out-of-ten) and "oot-m" evaluate the coverage of the gold substitute candidate list by the top 10 predictions.
In SwordS, a word is regarded as acceptable if it is judged to be good by more than five out of ten annotators, and conceivable if selected by at least one annotator. For the evaluation metrics, the authors (Lee et al., 2021) use the harmonic mean of the precision and recall given the gold and top-10 system-generated substitutes. As gold substitutes, they use either the acceptable or conceivable words, and calculate the corresponding scores Fa and Fc, respectively.
Baselines. We compare our methods ParaLS and ParaLS⋆ with the following baselines, Word2Vec (Melamud et al., 2015b), BERT (Zhou et al., 2019), BERT+WordNet (Michalopoulos et al., 2022), GRRoBERT (Lin et al., 2022), and XLNet+Word2Vec (Arefyev et al., 2020; Seneviratne et al., 2022). Arefyev et al. (Arefyev et al., 2020) linearly combine the prediction of pretrained language models XLNet and Word2Vec. Afterward, Seneviratne et
al. (Seneviratne et al., 2022) adopt four features to rank the substitutes generated by XLNet and Word2Vec.
Implementation Details. To implement an English paraphraser, we fine-tune BART-base2 model in fairseq. The initial learning rate is set to lr = 3 × 10−5 and dropout is set to 0.1. We adopt the Adam optimizer with β1 = 0.9, β2 = 0.999, ϵ = 10−8. We choose an English paraphrase dataset ParaBank2 (Hu et al., 2019) to train the paraphraser. In our experiments, we duplicate all the samples by exchanging source sentence and target sentence. We use the BLEURT large model3 for the calculation of BLEURT score. BARTScore fine-tuned on ParaBank2 can be downloaded here4. We use the LS07 dev set for tuning the hyperparameters in our model. The weights for the prediction score (Paraphraser), BARTScore, and BLEURT for ParaLS and ParaLS⋆ are 0.02, 1, and 1, respectively. The number of outputted paraphrases K is set to 50. The lookahead length of ParaLS⋆ is 2. Following the existing work (Zhou et al., 2019; Michalopoulos et al., 2022; Seneviratne et al., 2022), only the top 10 substitutes are used for evaluation.

heading: 4.2 Experimental Results
text: Comparison of LS methods. The results of our methods as well as the state-of-the-art methods on
2https://dl.fbaipublicfiles.com/fairseq/models/bart.base.tar.gz 3https://huggingface.co/Elron/bleurt-large-512 4https://github.com/neulab/BARTScore
LS07 and CoInCo are displayed in Table 2. Typically, performance is evaluated by selecting the top substitutes after executing the substitute ranking step. To exclude the influence of substitute ranking, we also present the results without substitute ranking in parentheses.
As can be observed, our methods, ParaLS and ParaLS⋆, demonstrate superior performance over the latest LS methods (GR-RoBERT and XLNet+Word2Vec) across all metrics in the LS07 and CoInCo datasets. Notably, ParaLS⋆ without the step of substitute ranking outperforms all baselines, including the best baseline XLNet+Word2Vec, which utilizes four features for substitute ranking. ParaLS⋆ without substitute ranking significantly outperforms ParaLS without substitute ranking, which means that the decoding with lookahead heuristic in ParaLS⋆ is very useful.
The results on SwordS are presented in Table 3. Our method ParaLS and ParaLS⋆ achieve the highest Fa score and Fc score, largely outperforming the best baseline XLNet+Word2Vec as well as two commercial methods GPT-3 and WordTune. Unlike GPT-3, which is fine-tuned by a prompt-based learning framework from multiple samples of the development set in SwordS, ParaLS and ParaLS⋆ do not rely on any LS dataset.
In comparison to LS methods based on pretrained language models, our methods possess the following three advantages:
(1) The paraphraser has been specifically trained to learn lexical variations. This could give it an advantage over pre-trained language models, which are generally trained on a wide range of tasks and may not be as focused on lexical substitution.
(2) The paraphraser is better at preserving the original meaning and context of the text, as it has been specifically designed to rewrite text while maintaining its meaning. This could be particularly important for lexical substitution tasks, as the goal is often to find substitutions that are semantically similar to the target word.
(3) The paraphraser can generate more diverse or varied substitutions. Pre-trained language models, on the other hand, are more general-purpose and may not be as adept at generating diverse substitutions.
Comparison of Substitute Ranking. We also
evaluate our substitute ranking strategies on both the LS07 and CoInCo datasets. In this sub-task of LS task, assume that the substitute candidates are provided, each method aims to create the most appropriate ranking of the candidates. Following prior work (Zhou et al., 2019; Michalopoulos et al., 2022), we use GAP score 5 for evaluation in the subtask, which is a variant of MAP (Mean Average Precision). We also output the results of the proposed method ParaLS⋆ by removing one feature or two features.
The results are displayed in Table 4. XLNet+Word2Vec, BERT+WordNet, and CILEX3 utilize 2, 4, and 4 features respectively to rank the substitutes, which include Gloss-sentence similarity score, sentence similarity score, and WordNet similarity score, among others. Our results obtained solely by using the BARTScore or Paraphraser feature surpass those of the baselines, with BARTScore exhibiting particularly strong performance. BLEURT also demonstrates superior performance when compared to CILEX3 and BERT. These results confirm that text generation evaluation metrics (BARTScore or BLEURT) are better suited for substitute ranking than prior methods. The performance of ParaLS⋆ when one feature is removed demonstrates that all the features have a positive impact on the performance of ParaLS⋆.
The proposed strategy using BARTScore or BLEURT for ranking substitutes based on the change of the sentence’s meaning after embedding them into the original sentences is likely effective because it directly addresses the primary goal of lexical substitution, which is to preserve the meaning of the original sentence while replacing a word. By using text generation evaluation metrics such as BARTScore and BLEURT to compute the relationship between the original and updated sentences, the method can quantify the extent to which the meaning of the original sentence has been preserved by each substitute.
Ablation Study. To further evaluate the impact of each ranking feature on the performance of our method, we conducted an ablation study on ParaLS⋆. The results are presented in Table 5. Both BARTScore and BLEURT are observed to be beneficial in enhancing the performance of ParaLS⋆. The ablation study, by isolating and testing the performance of individual features, illustrates that the Paraphraser feature alone achieves the best perfor-
5https://tinyurl.com/gap-measure
mance, thereby highlighting the effectiveness of our decoding with lookahead heuristics in generating high-quality substitutes.
Case Study. To quantitatively evaluate the effectiveness of the substitutes generated by LS methods, we present five instances of CoInCo for analysis. Table 6 displays the top five generated substitutes. Upon examination, we find that many suitable substitutes, marked in blue, are not present in the Labels. As the labels are human-annotated, it is not possible to provide all suitable substitutes for each target word. We posit that the actual performance of ParaLS and ParaLS⋆ is superior to the results computed by the metrics.
Furthermore, we see that our methods generate more high-quality substitutes than the baselines. Even when the methods generate unsuitable substitutes, the changes to the semantic information of the sentence are minimal. In the future, our methods could be utilized to enhance the coverage of substitutes in existing LS datasets.

heading: 5 Conclusions
text: We introduce two novel paraphraser-based LS methods named ParaLS and ParaLS⋆, which generate substitute candidates by considering the context and preserving the sentence’s meaning. Specifically, we design two decoding strategies that center on lexical variations of the target word during decoding and propose a substitute candidate ranking strategy by utilizing the newest text generation evaluation metrics. Experimental results show that ParaLS and ParaLS⋆ significantly outperform the stateof-the-art LS methods. In the future, we will apply the methods to different languages, and verify our method on many downstream tasks to investigate
further the method’s general availability.
Limitations
Our method depends on a large-scale paraphrasing corpus. We only test our method on the English LS task. Excluding English, other languages have large-scale paraphrasing datasets available, e.g., French, German, Chinese, Spanish, etc. Our method can be easily extended to these languages. But, for some languages that cannot obtain enough paraphrasing datasets, our proposed method cannot be used. Another limitation is that our method may struggle to generate substitutions for rare or unusual words and phrases, as they may not have encountered sufficient examples of these words in the training paraphrase data.
Ethics Statement
One potential ethical consideration related to a LS method based on a paraphraser is the potential for biased or unfair language generation. If the training data used to develop the paraphraser is biased in some way (e.g., it disproportionately represents certain groups of people or uses certain words and phrases in a biased manner), this could lead to biased substitutions being generated by the model. It is important to ensure that the training data used to develop the model is diverse and free of bias in order to minimize the potential for unfair or biased language generation.
Another ethical consideration is the potential for the LS method to be used for malicious purposes, such as creating fake or misleading content. It is important to consider the potential consequences of the LS method’s outputs and to put safeguards in place to prevent the LS method from being used for nefarious purposes.

heading: Acknowledgement
text: This research is partially supported by the National Natural Science Foundation of China under grants 62076217 and 61906060, and the Blue Project of Yangzhou University.

heading: A Appendix A (More Experiments for
text: Ablation Study)
1. The baselines. We compare our methods ParaLS and ParaLS⋆ with the following baselines.
(1) Word2Vec: The words that have the highest similarities are selected as substitute candidates from the word embedding modeling whose vectors are closer in terms of cosine similarity with the target word (Melamud et al., 2015b).
(2) BERT: BERT proposed by (Zhou et al., 2019) applies dropout to the embedding of the target word for partially obscuring the target word.
(3) BERT+WordNet: Michalopulos et al. (Michalopoulos et al., 2022) integrated the knowledge from WordNet into the embedding of BERT.
(4) GR-RoBERT: Lin et al. (Lin et al., 2022) proposed an auxiliary gloss regularizer module to BERT pre-training, to enhance word semantic similarity.
(5) XLNet+Word2Vec(Arefyev et al., 2020; Seneviratne et al., 2022): (Arefyev et al., 2020) linearly combines the prediction of pretrained language models XLNet and Word2Vec. Afterward, Seneviratne et al. (Seneviratne et al., 2022) adopt four features to rank the substitutes generated by XLNet and Word2Vec.
2. Influence of different ranking features. In the paper, we give the results of CoInCo datasets. Here, we give the results of LS07 datasets. The results are shown in Table 7. The conclusions are consistent with the conclusions of CoInCo.
3. Influence of lookahead length. ParaLS⋆ has a parameter of lookahead length. In this experiment, we will analyze the influence of lookahead length on the performance of ParaLS⋆. We vary the length of the lookahead from 0 to 5. When lookahead length equals 0, ParaLS⋆ is transformed into ParaLS.
The results are displayed in Figure 2. We see that the performance ParaLS⋆ is robust when varying lookahead length.
4. The running time of LS method. We give the average running time of one instance in Table 8. We run 100 instances in CoInCo dataset, and compute the average time of one instance.
We see that ParaLS only need 1.05 second for one instance, close to BERT(Zhou et al., 2019). XLNet+Word2Vec (Seneviratne et al., 2022) is the slowest LS method.
5. Influence of different paraphrasers. We do these experiments to verify the influence of different paraphrasers on the performance of ParaLS. In our paper, we adopt pretrained modeling BART to fine-tune an English paraphraser. Here, we train a Transformer model in FairSeq with a 6-layer encoder and decoder, 512-dimensional embeddings,
8 encoder-decoder attention heads, and 0.1 dropout. The initial learning rate is set to lr = 3 × 10−4. We adopt the Adam optimizer with β1 = 0.9, β2 = 0.999, ϵ = 10−8.
The results are shown in Table 9. We see that the performance of our proposed ParaLS and ParaLS⋆ is not significantly affected by the specific paraphrase model that is used.

heading: B Appendix B (Case Study)
text: Here, we give the generated top 10 substitutes of 10 instances in CoInCo to analyze the generated substitutes by our method (ParaLS and ParaLS⋆) and the baselines (BERT (Zhou et al., 2019) and XLNet (Seneviratne et al., 2022).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will apply the methods to different languages, and verify our method on many downstream tasks to investigate
further the method’s general availability. Limitations
Our method depends on a large-scale paraphrasing corpus. We only test our method on the English LS task. Excluding English, other languages have large-scale paraphrasing datasets available, e. g. , French, German, Chinese, Spanish, etc. Our method can be easily extended to these languages. But, for some languages that cannot obtain enough paraphrasing datasets, our proposed method cannot be used. Another limitation is that our method may struggle to generate substitutions for rare or unusual words and phrases, as they may not have encountered sufficient examples of these words in the training paraphrase data