Venue: NAACL 2025

# Learning to Substitute Words with Model-based Score Ranking

 Hongye Liu

Duke University

hongye.liu@duke.edu &Ricardo Henao

Duke University

ricardo.henao@duke.edu

###### Abstract

Smart word substitution aims to enhance sentence quality by improving word choices; however current benchmarks rely on human-labeled data. Since word choices are inherently subjective, ground-truth word substitutions generated by a small group of annotators are often incomplete and likely not generalizable. To circumvent this issue, we instead employ a model-based score (BARTScore) to quantify sentence quality, thus forgoing the need for human annotations. Specifically, we use this score to define a distribution for each word substitution, allowing one to test whether a substitution is statistically superior relative to others. In addition, we propose a loss function that directly optimizes the alignment between model predictions and sentence scores, while also enhancing the overall quality score of a substitution. Crucially, model learning no longer requires human labels, thus avoiding the cost of annotation while maintaining the quality of the text modified with substitutions. Experimental results show that the proposed approach outperforms both masked language models (BERT, BART) and large language models (GPT-4, LLaMA). The source code is available at [https://github.com/Hyfred/Substitute-Words-with-Ranking](https://github.com/Hyfred/Substitute-Words-with-Ranking).

## 1 Introduction

In the current era of AI-driven technologies, the generation of natural language by machines has become a critical area of research within the field of natural language processing (NLP). The advent of advanced language models such as GPT Ouyang et al. (2022) and LLaMA Touvron et al. (2023, 2024) has brought about unprecedented improvements in generating coherent, contextually accurate text across a wide range of applications, including machine translation, summarization, and conversational agents. These models are revolutionizing industries by enabling machines to produce human-like text at scale, greatly expanding the possibilities for automation and interaction in both professional and creative domains.

As language generation capabilities improve, the need to refine and control the output of these models becomes more pressing. One key area in this direction is smart word suggestion (SWS) defined by Wang et al. (2023) as a task that focuses on enhancing writing by improving two aspects of NLP: identifying words (or tokens more generally) that if replaced can improve sentence quality, and suggesting alternatives for such words. Complementarily, Wang et al. (2023) introduced a dataset to benchmark models built for SWS. This task is essential in applications such as paraphrasing, machine translation, and writing assistance, where accurate word choice directly impacts the clarity and quality of the generated text. Despite the progress in full-text generation, word substitution presents unique challenges in maintaining grammatical correctness, contextual appropriateness, and semantic fidelity.

Previous SWS work can be divided into _task-specific_ models and _prompt-based_ methods. Task-specific models are trained directly to address the SWS task. One approach adapts models for lexical substitution (LS) McCarthy and Navigli (2007); Kremer et al. (2014), allowing them to generate substitutions for each word in a sentence. If the substitution differs from the original word, it is recognized and recorded as such. BERT-based Zhou et al. (2019) and the LexSubCon Michalopoulos et al. (2021) models were first explored in Wang et al. (2023). They also fine-tuned pretrained models like BERT Devlin (2018) and BART Lewis (2019), allowing them to handle SWS end-to-end. Importantly, these models were fine-tuned using supervised learning to provide suggestions for Wikipedia text using ground-truth substitutions obtained from a list of synonyms gathered from a thesaurus. Orthogonal to these machine learning techniques, rule-based methods that rely on paraphras

ing databases (_e.g._, PPDB Pavlick et al. (2015)) or a thesaurus (_e.g._, the Merriam-Webster thesaurus) have also been explored Wang et al. (2023).

Alternatively, prompt-based techniques leveraging large language models (LLMs), such as GPT Ouyang et al. (2022), have gained popularity in SWS. Provided that research in language generation has shifted from narrow tasks like substitution to generating coherent, contextually rich text, tasks such as SWS have become less of a priority. However, in general it is still key to enhance sentence quality, as precise word choices can significantly impact clarity, tone, and overall effectiveness in communication. Moreover, the success of these methods is highly dependent on prompt design.

From a different perspective, benchmarks for SWS predominantly rely on human-labeled data for both training and evaluating model performance. However, only a small group of annotators are recruited for data labeling, which can lead to a lack of diversity. For instance, in the SWS test dataset Wang et al. (2023), 12,883 suggestions for target words are available of which 7,827 (60.8%) were created by a single annotator, only 1632 (12.67%) of suggestions were created by at least three annotators, and merely 633 (4.91%) of suggestions were created by more than three annotators. While scaling up the number of annotators could potentially mitigate this issue, it significantly increases the costs of generating the annotations. In the context of substitution tasks, a large number of annotators is necessary to obtain a comprehensive representation of the substitution space, as evaluating them requires a diverse sample of high-quality alternatives. As a result, validation sets in such benchmarks may not fully capture model performance. Conversely, recent research has shown that model-based evaluation metrics align closely with human judgments Lu et al. (2022); Yuan et al. (2021); He et al. (2024), offering a viable alternative to alleviate the drawbacks of human evaluations.

To address these challenges, we propose an approach that eliminates the dependency on human-labeled data. Our contributions are listed below:

* We identified a gap in the evaluation methods for substitution models, which rely on human annotators. We propose a model-based evaluation strategy based on the BARTScore to assess SWS models without human annotations.
* We introduce an approach for generating and ranking token substitutions. Our method optimizes both the quality of the substitutions, via the model-based score, and the ranking of the substitutions relative to such a score. To the best of our knowledge, this is the first time SWS has been tackled by jointly optimizing the association between predictions and quality scores.
* We redefined the substitution task by forgoing the need for supervised learning relying on human annotations. Thus, our method eliminates the need for costly human-labeled data while maintaining high performance in SWS tasks.
* Our experimental results demonstrate that the proposed approach outperforms both masked language models (BERT, BART) and LLMs (GPT-4, LLaMA) in the SWS task.

## 2 Related Work

Enhancing word usage is a key feature of writing assistance. The SWS task in Wang et al. (2023) addresses it by emulating a real-world end-to-end writing scenario. Unlike traditional LS tasks McCarthy and Navigli (2007); Kremer et al. (2014), SWS requires systems to identify words that, if replaced, can improve sentence quality and suggest alternatives for such words. In the LS task, however, the word to be replaced is predefined and the model is only tasked with finding suitable substitutions without altering the overall meaning of the sentence. Further, in LS substitutions are typically lemmatized to match ground-truth labels, but in SWS they must be in the correct grammatical form.

Zhou et al. (2019) proposed a LS method using contextual word embeddings, modifying BERT with a dropout embedding policy to partially mask target words and introduced a validation score based on the top four layers of BERT for candidate evaluation. Michalopoulos et al. (2021) integrated external knowledge from WordNet into BERT for LS, combining scores from BERT, WordNet gloss similarity, and sentence embeddings, along with the validation score from Zhou et al. (2019), to generate substitutes. ParaLS Qiang et al. (2023) generated substitutes through a paraphrasing model, using a heuristics-based decoding strategy.

To adapt the substitution capabilities from LS to the SWS task, Wang et al. (2023) used an approach that allowed models to generate substitutions for each word. If the substitution differs from the original word, it is recognized as the word the model wants to improve and recorded as a suggestion. Further, they fine-tuned pretrained models like BERT Devlin (2018) and BART Lewis (2019) to address 

the task in an end-to-end manner, and as an alternative to machine learning, rule-based methods using dictionaries (theseurus) were also explored.

Recently, prompt-based methods have emerged as a powerful alternative for SWS Wang et al. (2023), by leveraging the capabilities of LLMs such as GPT Ouyang et al. (2022). These approaches shift the focus from traditional model fine-tuning to designing prompts that guide the model to produce desired outcomes. While this flexibility allows for easier adaptation to various tasks, the effectiveness of the method heavily relies on the prompt's structure. Despite the growing interest in generating fluent, contextually cohesive text with LLMs, prompt-based methods often overlook the importance of improving specific word choices. In contexts where sentence clarity and precision are key, refining word choices remains an important task that prompt-based approaches need to address.

Compared to traditional methods that rely heavily on human involvement, _e.g._, via manual annotations, in our work, we utilize model-based sentence scoring, specifically, BARTScore to quantify the quality of sentences, thereby eliminating the human-associated costs. We align the model's output with the scores through a ranking loss, ensuring that substitutions with higher scores are more likely to be generated by the model, resulting in higher-quality substitutions. This approach forgoes the need for human-dependent supervised training, thus without the need for human intervention.

## 3 Methodology

In SWS, achieving a representative sample of high-quality alternatives requires a broad pool of annotators, as diverse substitution options are essential for accurate evaluation. However, relying solely on human annotators is thus not only costly but also inefficient. Below we introduce a method to train and evaluate token substitution models that does not require human annotations, by instead solely relying on model-based scoring functions.

Problem DefinitionLet \(X\) denote a sentence composed of \(N\) tokens, _i.e._, \(X=(x_{1},\dots,x_{N})\). We aim to find a collection of \(K\) potential substitutes \(\{\tilde{w}_{k}\}_{k=1}^{K}\) for token \(x_{n}\), while estimating their likelihood given the sentence (_i.e._, its context) as \(p_{\theta}(x_{n}=\tilde{w}_{k}|X)\), where \(X\) is the original sentence and \(p_{\theta}(\cdot)\) is a model for the conditional likelihood for token \(x_{n}\) parameterized by \(\theta\). Note that technically, if \(p_{\theta}(\cdot)\) is a masked language model we should write \(p_{\theta}(x_{n}=\tilde{w}_{k}|X_{\setminus x_{n}})\), where \(X_{\setminus x_{n}}\) indicates that \(x_{n}\) has been masked out in \(X\), and alternatively, if we use an (autoregressive) language model we should write \(p_{\theta}(x_{n}=\tilde{w}_{k}|x_{<n},Z)\), where \(x_{<n}\) is the portion of the original sentence up to token \(n-1\) and \(Z\) is a prompt. However, we use \(p_{\theta}(x_{n}=\tilde{w}_{k}|X)\) in the following for notational simplicity. Based on these likelihood estimates, the token substitution rule is set as \(x_{n}\leftarrow\tilde{w}_{k}\) if \(p_{\theta}(x_{n}=\tilde{w}_{k}|X)>p_{\theta}(x_{n}=w_{n}|X)\), where \(w_{n}\) is the original value of \(x_{n}\). Moreover, if multiple replacement candidates satisfy the substitution rule, we select \(\tilde{w}_{k}\) according to \(\operatorname*{argmax}_{k}p_{\theta}(x_{n}=\tilde{w}_{k}|X)\), and alternatively, if none satisfy it, we leave the token unchanged, _i.e._, \(x_{n}\gets w_{n}\).

Importantly, \(p_{\theta}(x_{n}=\tilde{w}_{k}|X)\) reflects the likelihood of token \(\tilde{w}_{k}\) given its context \(X\), rather than the quality of the sentence. So motivated, we also seek to align such estimates with a score \(M(x_{n}=\tilde{w}_{k}|X)\) quantifying the quality of sentence \(X\) with \(x_{n}\) substituted with \(\tilde{w}_{k}\). For simplicity, in the following we denote the sentence \(X\) with \(x_{n}\) replaced with the \(k\)-th candidate, \(\tilde{w}_{k}\), as \(\tilde{X}_{k}=(\tilde{x}_{1},\dots,\tilde{x}_{N})\). Further, to make our objective scalable and general, the score \(M(x_{n}=\tilde{w}_{k}|X)\) ought not be obtained via human feedback, but via a _black-box model_ with which we can score sentences, but through which we cannot learn, _i.e._, gradients cannot be propagated through it to obtain learning signals.

Below we start by showing how we can use a model-based score to statistically characterize the suitability of a candidate as an alternative to the substitution rule introduced above, and then present an approach to estimate \(P(x_{n}=\tilde{w}_{k}|X)\) while accounting for scores \(M(x_{n}=\tilde{w}_{k}|X)\).

### Statistic for Model-based Scores

Model-based ScoreUsing automated scoring methods to quantify text generation quality without relying on human annotators can reduce costs associated with human labeling. Model-based scoring approaches such as BLEURT Sellam et al. (2020), BERTScore Zhang et al. (2019), GPTscore Fu et al. (2023), and BARTScore Yuan et al. (2021), have proven to be effective while closely aligning with human evaluation results. BARTScore is one of the most popular metrics for the evaluation of text generation. It has been shown that BARTScore outperforms other metrics such as BERTScore and BLEURT Yuan et al. (2021), while also being more efficient than GPTScore. Further, it has been used also as a ranking tool for substitution tasks, 

achieving good results (Qiang et al., 2023).

The BARTScore is a metric for universal natural language generation evaluation Yuan et al. (2021). It leverages the conditional likelihood from a pre-trained BART model Lewis (2019) to assess the quality of generated text. Since BART is an autoregressive model, the log-likelihood of tokens in a sentence is obtained one at a time conditioned on tokens before it. Specifically, we write

\[M(\tilde{X}_{k})=\sum^{N}_{n=1}\log p_{\hat{\theta}}\left(\tilde{x}_{n}\mid \tilde{x}_{<n},X\right)\,, \tag{1}\]

where \(\hat{\theta}\) represents the parameters of a pre-trained model (in this case, BART1), \(\tilde{x}_{n}\) is the \(n\)-th token of the modified sentence \(\tilde{X}_{k}\) which we wish to score, \(\tilde{x}_{<n}\) represents the first \(n-1\) tokens of \(\tilde{X}_{k}\), and \(X\) is the original (unmodified) sentence. In (1), the BARTScore \(M(x_{n}=\tilde{w}_{k}|X)\) has been simplified for notational convenience to \(M(\tilde{X}_{k})\), _i.e._, the score of modified sentence \(\tilde{X}_{k}\) in which a token has been replaced with the \(k\)-th candidate from potential substitutes \(\{\tilde{w}_{k}\}_{k=1}^{K}\). For now, we assume that such candidates are readily available, however, in the next section we will propose a learning strategy to train a model to generate substitutes given \(X\).

Footnote 1: We use BARTScore (ParaBank2 version) without incorporating any prompts.

Model-based Score StatisticOne potential issue with model-based scores including the BARTScore in (1) is that there is no clear standard to determine what score (or score range) qualifies as "good". It is clear that though scores can be used for pairwise or aggregate comparisons, the magnitude of the difference between two scores is not meaningful without a reference score distribution.

This underscores that numerical scores alone are insufficient to fully capture the significance of a score difference. Inspired by statistical hypothesis testing Moore (1999), we propose constructing a reference (null) distribution of \(M(\tilde{X}_{k})\) by sampling a collection of substitutes for a given token and then calculating their model-based score. By determining where \(M(\tilde{X}_{k})\) lies within this empirical distribution, we can more reliably assess the quality of a substitute. Since sampling candidates uniformly at random from a natural language vocabulary or from empirical token frequencies will be hugely inefficient, instead we use a model to generate substitute candidates. In this manner, we can evaluate the relative quality of the generated substitutes compared to other candidates from the model's own distribution. This constitutes a principled framework for assessing the quality of a model in generating token substitutes. Next, we introduce a hypothesis test to estimate a \(p\)-value for the significance of a candidate substitute relative to a reference (empirical) substitute distribution.

Given a model we wish to evaluate, let \(\{\tilde{w}_{k}\}_{k=1}^{K_{s}}\) be a set of \(K_{s}\) candidates for sentence \(X\) at a given position produced by such model, _e.g._, a masked language model like BERT Devlin (2018). Note that in general, \(K_{s}\) is a number much larger than the size of potential substitutes \(K\) described before. We can evaluate the quality of, for instance, the top candidate \(w_{1}\) (assuming they are ordered) as the proportion of times it produces a score that is larger than using all other candidates, \(\{\tilde{w}_{k}\}_{k=2}^{K_{s}}\). This frequency or \(p\)-value is formally written as

\[p_{\tilde{X}_{1}}=\frac{1}{K_{s}-1}{\sum^{K_{s}}_{k=2}I[M(\tilde{ X}_{k})>M(\tilde{X}_{1})]}\,, \tag{2}\]

where \(I[\cdot]\) is the indicator function and the \(\{w_{k}\}_{k=1}^{K_{s}}\) producing modified sentences \(\{\tilde{X}_{k}\}_{k=1}^{K_{s}}\) are obtained from a model \(p_{\theta}(x_{n}=\tilde{w}_{k}|X)\), which defines the reference distribution. In standard hypothesis testing fashion, we reject the hypothesis that candidate \(w_{1}\) has the same expected score as the reference distribution if \(p_{\tilde{X}_{1}}<\alpha\), for a significance level \(\alpha\), which we set to \(\alpha=0.01\) in our experiments. Since calculating model-based scores such as BARTScore is computationally expensive, we trade-off statistical accuracy with computational efficiency and let \(K_{s}=1000\) in the experiments.

### Preference-Aware Learning

Now that we have constructed a statistic based on model-based scores, in principle, we seek to train a model such that for each token \(x_{n}\), the resulting \(p_{\tilde{X}_{k}}\) from (2) for a given substitute candidate \(w_{k}\) producing \(\tilde{X}_{k}\) is as small as possible. According to (2), it is desirable for \(M(\tilde{X}_{1})\) to be larger than \(M(\tilde{X}_{k})\) for \(\{w_{k}\}_{k=1}^{K}\) candidates sampled from model \(p_{\theta}(x_{n}|X)\). This means that effectively we require the model being such that its outputs align, in likelihood, with the model-based score, BARTScore here. Consequently, it is desirable to learn parameters \(\theta\) so \(p_{\theta}(x_{n}=w_{k}|X)\) is ranked consistent to \(M(\tilde{X}_{k})\), which in turn will make \(p_{\tilde{X}_{k}}\to 0\) in (2) for \(k\to 1\). Below we consider two approaches to accomplish this: margin ranking Liu et al. (2023b); Chern et al. (2023); Liu et al. (2022) and direct preference optimization (DPO) loss Rafailov et al. (2024).

 

Ranking OptimizationRanking has been used as an optimization objective in many tasks including text summarization (Chern et al., 2023; Liu et al., 2022). Maximum likelihood estimation based on the standard cross-entropy loss can be effective at satisfying (2) by setting it as a classification problem by defining a vector of labels \(y\) where \(y_{k}=1\) if \(M(\tilde{X}_{k})\) is maximum among \(\{M(\tilde{X}_{k})\}^{K}_{k=1}\) or \(y_{k}=0\) otherwise. However, it does not take into account the ordering of the model-based scores or their magnitude differences. So motivated, we consider the following margin ranking (MR) loss (Liu et al., 2023b; Chern et al., 2023; Liu et al., 2022)

\[\mathcal{L}_{\mathrm{MR}}=\sum_{k}^{K}\!\sum_{j>k}^{K}~{}\max(0,s_{j}-s_{k}+ \lambda_{jk})\,, \tag{3}\]

where \(\{s_{k}\}^{K}_{k=1}\) are the logits from model \(p_{\theta}(x_{n}=w_{k}|X)\), _i.e._, before applying the softmax function, and we have sorted them such that \(s_{k}>s_{j}\) if \(M(\tilde{X}_{k})>M(\tilde{X}_{j})\) for \(i,j=1,\ldots,K\). Further, we set the _margin_\(\lambda_{jk}=\lambda\times(j-k)\) for some hyperparameter \(\lambda\), which in the experiments is set via cross-validation. Intuitively, (3) encourages the model to make predictions whose outputs \(\{p_{\theta}(x_{n}=w_{k}|X_{\setminus x_{n}})\}^{K}_{k=1}\) are consistent in order with \(\{M(\tilde{X}_{k})\}^{K}_{k=1}\), by penalizing pairwise order mismatches, while also enforcing predictions to be distanced by a fixed margin to improve robustness. The latter is justified by extensive results from the margin learning literature (Smola, 2000).

Improving Model-based ScoresOne unintended consequence of the loss function in (3) is that though it encourages model predictions to be aligned with model-based scores, it does so regardless of their values. This is so because only the order of \(\{M(\tilde{X}_{k})\}^{K}_{k=1}\) is considered in (3). In practice, we have observed that (3) effectively improves the ranking of predictions from the model, but it does so at the expense of producing predictions that have, on average, lower model-based scores relative to a reference model, _e.g._, the pre-trained model used as initialization for the refinement of \(\{p_{\theta}(x_{n}=w_{k}|X)\}^{K}_{k=1}\). To address this issue, we consider two approaches, one that seeks to improve the weighted average of model-based scores and another that maximizes the model-based score of the top prediction from the model relative to that of the reference model. Specifically, we write

\[\mathcal{L}_{\mathrm{AS}} =-\!\sum_{k}^{K}h(s_{k})M(\tilde{X}_{k})\,, \tag{4}\] \[\mathcal{L}_{\mathrm{BS}} =\max(0,(M(X)-M(\tilde{X_{1}}))f\left(s_{1}\right))\,, \tag{5}\]

where \(h(\cdot)\) is the softmax function, \(s_{k}\) is the logit corresponding to \(p_{\theta}(x_{n}=w_{k}|X)\), and \(M(X)\) and \(M(\tilde{X}_{k})\) are the model-based scores of the original and modified sentence, respectively. Recall that for \(M(\tilde{X_{1}})\), the token of interest \(x_{n}\) in \(X\) has been modified to the top prediction \(w_{1}\) from the model. Conceptually, \(\mathcal{L}_{\mathrm{AS}}\) in (4) seeks to maximize the (weighted) average of model-based scores from \(K\) predictions, while \(\mathcal{L}_{\mathrm{BS}}\) aims to improve the model-based score of the top prediction with respect to that of a reference model.

We then combine the margin ranking loss in (3) with the score-improving losses in (4) or (5) as

\[\mathcal{L}_{\mathrm{MR+AS}} =\mathcal{L}_{\mathrm{MR}}+\gamma\mathcal{L}_{\mathrm{AS}}\,, \tag{6}\] \[\mathcal{L}_{\mathrm{MR+BS}} =\mathcal{L}_{\mathrm{MR}}+\gamma\mathcal{L}_{\mathrm{BS}}\,, \tag{7}\]

where \(\gamma\) is a hyperparameter trading off ranking or model-based score improvement. In the experiments we will compare these two approaches to determine empirically whether is better to attempt to improve the scores of \(K\) predictions from the model as opposed to improving the score of one of the predictions from the model (the most likely) relative to a baseline prediction obtained from a reference model. Note that is tempting to use (5) with all predictions, not just the first one, however, we found empirically that it considerably increases the computational cost without significant performance gains. The cost overhead is caused mainly by the need to evaluate the model-based score \(K\) times rather than just \(2\) in (5).

Direct Preference OptimizationDirect Preference OptimizationDPO (Rafailov et al., 2024) is an efficient technique for aligning large language models with human feedback, which gained popularity due to its simplicity (Miao et al., 2024). For instance, it has demonstrated to be effective in chat benchmarks (Tunstall et al., 2023; Zheng et al., 2023). In our case, the model-based score serves as proxy for human feedback DPO (under the Plackett-Luce model) and is formally expressed as

\[\mathcal{L}_{\mathrm{DPO}}=-\mathbb{E}\left[\log\prod_{k=1}^{K}\frac{\exp( \delta r_{k})}{\sum_{j=k}^{K}\exp(\delta r_{j})}\right]\,, \tag{8}\]

where \(r_{k}=\log p_{\hat{\theta}}(\tilde{X}_{k})-\log p_{\hat{\theta}}(\tilde{X}_{ k})\), the expectation is over \(\{\tilde{X}_{1},\dots,\tilde{X}_{K},X\}\), and \(\theta\) and \(\hat{\theta}\) denote the parameters of the model being trained and that used for reference, respectively. Accordingly, only \(\theta\) are updated while learning while \(\hat{\theta}\) are kept fixed.

 

Since the magnitude of the logits drops significantly as \(k\to K\), we found that the sum in the denominator of (8) weakens the loss. Therefore, we removed the sum and instead compared the \(k\)-th and the \((k+1)\)-th substitution, rather than comparing it with all \(K\) values. Further, we approximate \(\log p_{\theta}(\tilde{X}_{k})\) with its logit \(s_{k}\), and let \(\delta=1\) for convenience. Then (8) simplifies (for a single token in sentence \(X\)) to

\[\mathcal{L}_{\text{DPO*}}=-\sum_{k=1}^{K-1}\left(s_{k}-\hat{s}_{k}-s_{k+1}+ \hat{s}_{k+1}\right)~{}, \tag{9}\]

where \(s_{k}\) and \(\hat{s}_{k}\) denote the logit of the \(k\)-th substitution from the model being trained and the reference model, respectively.

We also extend DPO (under the Bradley-Terry model) Rafailov et al. (2024) to multiple substitute candidates, where each candidate is compared with the next in the ordered list of substitute candidates. We write (for a single token in sentence \(X\))

\[\mathcal{L}_{\sigma\text{DPO*}}= \tag{10}\] \[\qquad-\sum_{k=1}^{K-1}\log\sigma\left(s_{k}-\hat{s}_{k}-s_{k+1}+ \hat{s}_{k+1}\right)~{},\]

from which we see that the only difference between (10) and (9) is that the comparison of logit values in the former is scaled with the log-logistic function. See Appendix A for a derivation of both losses.

## 4 Experiments

Below we illustrate the problem with evaluating with human annotations, then we present an ablation study comparing the optimization approaches in Section 3.2 and a benchmark comparing the proposed model to MLM and LLM approaches in terms of model-based score alignment, average score and the statistic in (2). Further, we explore the performance of top-2 predictions, _i.e._, when the top prediction is the original token, and results comparing LLMs with and without prompts encouraging token substitutes to be ranked by quality.

DatasetsWe consider four datasets. The SWS dataset by Wang et al. (2023) consists of three sets, validation and test labeled by human annotators, and an artificially generated training set. The training dataset was constructed from Wikipedia sentences, with labels obtained using a combination of PPDB Pavlick et al. (2015) and the Merriam-Webster theseaurus. We also consider two traditional lexical substitution datasets, LS07 McCarthy and Navigli (2007) and LS14 Kremer et al. (2014), which use lemmatized human annotations as ground truth substitutions. Further, we also consider a general dataset without ground-truth substitutions. Specifically, we use XSum Narayan et al. (2018), a summarization dataset consisting of BBC news articles and corresponding summaries. In the Appendix we show Table 6 summarizing these four datasets.

BaselinesWe compare our method with three classes of baseline models. _Masked language models (MLMs):_ BERT-base-uncased with original MLM head (BERT-naive) Devlin (2018). BERTSpsv Zhou et al. (2019), as a representative LS model. BERT-SWS and BART-SWS, both of which were fine-tuned on the SWS training dataset Wang et al. (2023). _Rule-based models:_ The rule-based approach introduced by Wang et al. (2023), which leverages a thesaurus. _Prompt-based large language models (LLMs):_ We leverage the power of pretrained LLMs via prompts to generate token substitutions. We consider two popular choices, GPT Ouyang et al. (2022) and LLaMA Touvron et al. (2023, 2023); Dubey et al. (2024), specifically, GPT-4o and LLaMA-3.1-8B-Instruct. Prompt-based techniques have gained popularity due to their flexibility and effectiveness in generating high-quality, contextually appropriate substitutions without requiring task-specific tuning Liu et al. (2023). We designed both ranking and non-ranking prompts, which we show in Appendix B.

Evaluation MetricsWe use the cosine similarity (CS) to measure the correlation between model predictions and model-based scores, _i.e._, BARTScores. Since BARTScore produces log-likelihoods, we log-transform model predictions accordingly. We also consider the quality of the substitutes \(\{\tilde{w}_{k}\}_{k=1}^{K}\). First we get the score ratio between the sentence with and without substitution and then calculate the average of BARTScore ratios as \(\text{ABR}=1/K\sum_{k=1}^{K}M(\tilde{X}_{k})/M(X)\).

ImplementationFor our model, we fine-tuned the BERT-base-uncased model Wang et al. (2023) on 100k randomly sampled sentences from the SWS training dataset using one Nvidia A100 GPU. For each sentence, we randomly selected 5 tokens, and for each token, we chose \(K=5\) candidates from the model's output to form a candidate pool and compute the training loss. The model was finetuned over 5 epochs. For DPO, we duplicated the same BERT model and weights at initialization, freezing one as the reference while only updating 

the weights of the other during training. During testing, we used the same token selection strategy, randomly selecting 5 tokens per sentence and choosing the top 5 candidates for each. Hyperparameter settings are described in Appendix C.

Illustrative ExampleWe stratify tokens in the SWS test set into five groups according to whether they were substituted by human annotators and/or the model. These groups are: change agreement (CA: both model and the human annotator agreed on the change), change disagreement (CD: both model and the annotator suggested a change, but the model substitution did not match that of the annotator), no change agreement (NCA: the model and the annotator kept the token unchanged), only model changed (OMC: the model suggested a change, but the annotator did not find it necessary), and only annotator changed (OAC: the annotator suggested a change, but the model did not).

Results in Table 1 show that most tokens remain unchanged (NCA: 94%) by both model (BERT-SWS) and annotator. The proportion of annotator changes for which the model agrees or disagrees is similar, CA: 11% CD: 15%, respectively. Interestingly, most annotator changes are not captured by the model (OAC: 74%), which is consistent with results in Wang et al. (2023) for a variety of models, and OMC: 6% tokens not changed by the annotator are changed by the model. These results underscore that \(i\)) a model is unlikely to match human annotations when these are largely subjective and incomplete; and \(ii\)) cases in which changes made by the model do not match the human annotator cannot be considered errors. These are consequences of having a few annotators each selecting which tokens need to be changed.

This is further illustrated below in Table 4 where we show that though tokens for which both model and annotator agree have better model-based statistics, there is also a non-insignificant portion for which model and annotator disagree, but the model-based statistic suggests that the model changes are of good quality. Some of these examples are shown in Table 7 of the Appendix.

Ablation StudySection 3.2 introduced two approaches to align the predictions of a trained model with model-based scores, namely MR in (3) and DPO in (8), (9) and (10). For the former, we consider two ways of improving the model-based scores, via a weighted average in (6) and improvement relative to a reference model in (7). Results in Table 2 show results comparing \(i\)) the alignment of model predictions with model-based scores using median CS, and \(ii\)) the average model-based scores using median ABR. Distributions of CS and ABR for all tokens are shown in the Appendix Figures 1 and 2. We see that \(i\)) the MR loss alone is not sufficient to deliver the best CS and produces the worst ABR; \(ii\)) DPO variants improve the ABR relative to MR, but not in terms of CS; \(c\)) MR+BS and MR+AS produce the best overall ABR and CS, respectively, however, MR+AS seems to provide the best trade-off between the two performance metrics, while outperforming both MR and DPO. Consequently, in the following experiments we will focus on MR+AS.

Model BenchmarkNext, we evaluate MLMs (BERT, BART) and LLMs (GPT-4o, LLaMA) on all datasets using median CS and ABR as metrics (distributions are shown in the Appendix Figures 1 and 2). Table 3 shows that MR+AS outperforms both MLMs (BERT, BART) and LLMs (GPT-4o, LLaMA), with top performances in terms of CS in all datasets and top ABR in two of the datasets (SWS and LS14). Also, when accounting for results variation across datasets, we see that MR+AS significantly outperforms the others in terms of CS,

\begin{table}
\begin{tabular}{r|l l r}  & Not change & Agreement & Disagreement \\ \hline Change & 4125 (0.74) & 631 (0.11) & 830 (0.15) \\ Not change & 14138 (0.94) & & 969 (0.06) \\ \end{tabular}
\end{table}
Table 1: Token changes in SWS test data. Rows are for annotator changes and columns indicate model changes and if they are in agreement with the annotator. In parenthesis are proportions relative to the row totals.

\begin{table}
\begin{tabular}{r c c|c|c|c|c|c|c}  & SWS & LSOT & LS14 & Xsbm & AVQ \\  & CAS & AR & CS & AS & AR & CS & ARR \\ \hline BERT-aware & 0.93 & 0.57 & 0.92 & 0.85 & 0.85 & 0.82 & 0.81 & 0.82 & 0.83 & 0.80 \\ BERT-aware & 0.92 & 0.93 & 0.85 & 0.85 & 0.82 & 0.81 & 0.81 & 0.82 & 0.83 & 0.80 \\ BERT-SWS & 0.92 & 0.93 & 0.91 & 0.92 & 0.92 & 0.93 & 0.94 & 0.91 & 0.91 & 0.70 & 0.70 \\ BERT-SWS & 0.93 & 0.94 & 0.92 & 0.81 & 0.93 & 0.76 & 0.83 & 0.89 & 0.80 & 0.81 & 0.81 \\ BM-SWS & 0.93 & 0.94 & 0.92 & 0.85 & 0.87 & 0.72 & 0.83 & 0.83 & 0.83 & 0.81 & 0.81 & 0.82 \\ BM-SWS & **0.93** & **0.98** & 0.92 & 0.93 & 0.92 & 0.93 & 0.83 & 0.83 & 0.83 & 0.81 & 0.82 \\ GPT-4o & 0.95 & 0.95 & 0.95 & 0.95 & 0.94 & 0.93 & 0.83 & 0.83 & 0.83 & 0.81 & 0.82 & 0.84 \\ LMLM & 0.95 & 0.95 & 0.95 & 0.94 & 0.93 & 0.94 & 0.86 & 0.87 & 0.81 & 0.80 & 0.84 \\ MH-AS & 0.98 & **0.99** & 0.95 & **0.99** & **0.93** & **0.99** & 0.84 & **0.99** & **0.81** & **0.80** & **0.86 & 0.81 \\ \end{tabular}
\end{table}
Table 2: Ablation CS and ABR metrics. Figures for each dataset are medians over all local detections. The last column shows averages over all datasets with standard deviations. The best results are highlighted in **bold**.

\begin{table}
\begin{tabular}{r|c c|c|c|c|c|c}  & SWS & LSOT & LS14 & Xsbm & AVQ \\  & CAS & AR & CS & AR & CS & AR \\ \hline BERT-aware & 0.93 & 0.57 & 0.92 & 0.85 & 0.82 & 0.83 & 0.81 & 0.82 & 0.82 & 0.83 & 0.80 \\ BERT-aware & 0.92 & 0.93 & 0.93 & 0.89 & 0.92 & 0.93 & 0.87 & 0.81 & 0.81 & 0.82 & 0.83 & 0.80 \\ BERT-SWS & 0.93 & 0.92 & 0.91 & 0.93 & 0.93 & 0.92 & 0.93 & 0.94 & 0.91 & 0.91 & 0.70 & 0.70 \\ BERT-SWS & 0.93 & 0.94 & 0.92 & 0.81 & 0.93 & 0.76 & 0.83 & 0.79 & 0.81 & 0.81 & 0.81 & 0.82 \\ BM-SWS & 0.93 & 0.94 & 0.92 & 0.85 & 0.97 & 0.72 & 0.83 & 0.83 & 0.83 & 0.81 & 0.81 & 0.82 \\ BM-SWS & **0.93** & **0.99** & 0.92 & 0.93 & 0.93 & 0.83 & 0.83 & 0.83 & 0.83 & 0.81 & 0.82 \\ GPT-4o & 0.95 & 0.95 & 0.95 & 0.94 & 0.93 & 0.86 & 0.86 & 0.87 & 0.81 & 0.88 & 0.84 \\ LMLM & 0.93 & 0.88 & 0.94 & 0.92 & 0.93 & 0.87 & 0.84 & 0.81 & 0.81 & 0.81 & 0.81 & 0.80 \\ Mn-AS & **0.98** & **0.99** & 0.95 & **0.99** & **0.83** & **0.99** & 0.84 & **0.89** & **0.81** & **0.86** & **0.81** \\ \end{tabular}
\end{table}
Table 3: Benchmark mean CS and ABR metrics. The last column shows averages over all datasets with standard deviations. The best results are highlighted in **bold**.

 

while being comparable with GPT-4o in terms of ABR. Importantly, for the latter, MR+AS and GPT-4o perform markedly better than other approaches.

Model-based Statistic BenchmarkAfter showing the effectiveness of our model in terms of CS and ABR, we now use the model-based statistic introduced in Section 3.1. Table 4 shows the proportion of model predictions that meet the significance threshold (\(\alpha=0.01\)) for three groups of tokens: CA, CD and OMC. We do not present results for the other two groups (NCA and OAC) because when using rule- and LLM-based models, we get no candidates for non-substitutions. Further, for these models we may not always get the same number of substitute candidates per token, thus we use their median (3 candidates) across all tokens in the SWS test set as the number of candidates for all other models when calculating the statistic, _i.e._, \(K_{s}=3\) in (2). In the Appendix Table 8 we show an extended table showing the NCA and OAC groups and \(K_{s}=1000\) for MLM-based models (ours included) and Figures 3-6 with \(p\)-value distributions for all models. To avoid problems associated with overfitting, thus making the evaluation more reliable, in Appendix Table 9 and Table 10 we extended the evaluation result by using the GPTScore2[7] with two backbone models: GPT2[5] and OPT[10]. The results show that our MR+AS still outperforms other baseline models.

Footnote 2: Similar to BARTScore, we adapted GPTScore for the paraphrasing task using the following prompt: “Rewrite the following text with the same semantics. {original sentence} In other words, {modified sentences}”. We also calculated the Spearman correlation between BARTSCore and GPTScore (GPT2-medium and OPT-350M), which yielded results of 0.929 and 0.921, respectively. These values indicate a strong correlation between BARTscore and GPTScore.

Ideally, we want proportions in Table 4 to be close to one, indicating that the top candidate is of better quality than the alternatives. Effectively, results demonstrates that our model (MR+AS) not only produces the largest proportions, but it does so irrespective of the group (CA, CD or OMC). This suggests that substitutions made by our model are of good quality (using BARTScore as proxy for quality) even if they do not agree with the human annotator. Moreover, in general, LLMs are better than the other MLM- and rule-based alternatives.

It is worth noting that LLM-based models make considerably more substitutions than our model. In the Appendix Table 11 we show that GPT-4o makes 24.4%, 24.8% and 11.5% changes in groups CA, CD and OMC, respectively, compared to 5.5%, 5.2% and 2.7% by our MR+AS. This suggests a trade-off between quality and quantity of substitutions between MR+AS and GPT-4o, respectively.

Top-2 Candidate PerformanceProvided that the substitution rates by our model are low relative to others as discussed above, we now examine the quality of the top-2 substitute. Note that in MLM-based models a change is produced only when the top substitute candidate is different than the original token and that for LLM-based models the top-2 candidate is only available is the model considers a substitution has to be made and more than one candidates are provided. To quantify the quality of the top-2 candidates, we randomly selected five tokens from each sentence where the top-2 candidate is available and calculate the BARTSCore ratio between the sentence modified with the top-candidate and that of the original (unchanged) sentence. We then averaged these ratios provide a summary of the quality of top-2 candidates. Results in Table 5 indicate that MR+AS achieved the best top-2 candidate quality across all four datasets followed (on average) by GPT-4o and the rule-based model.

Encouraging ranking in LLMsIt is well known that prompting LLMs to provide ordered responses is challenging [11]. We tried prompts with and without a specific request for ordered substitute candidates (see Appendix B for details). We found that both GPT and LLaMA produced slightly

\begin{table}
\begin{tabular}{l|c c c}  & CA & CD & OMC \\ \hline BERT-naive & 0.646 & 0.389 & 0.612 \\ BERT-spsv & 0.709 & 0.442 & 0.491 \\ BERT-SWS & 0.204 & 0.118 & 0.078 \\ BART-SWS & 0.225 & 0.139 & 0.121 \\ Rule Based & 0.602 & 0.241 & 0.344 \\ GPT-4o & 0.761 & 0.563 & 0.661 \\ LLAMA & 0.700 & 0.517 & 0.603 \\ MR+AS & **0.961** & **0.917** & **0.810** \\ \end{tabular}
\end{table}
Table 4: Proportion of \(p\)-values below the significance threshold (\(\alpha=0.01\)) on the SWS test data.

\begin{table}
\begin{tabular}{l|c c c c|c}  & SWS & LS07 & LS14 & XSum & AVG \\ \hline BERT-naive & 0.88 & 0.84 & 0.82 & 0.81 & 0.84\(\pm\) 0.025 \\ BERT-spsv & 0.82 & 0.80 & 0.73 & 0.73 & 0.77\(\pm\) 0.040 \\ BERT-SWS & 0.85 & 0.82 & 0.78 & 0.79 & 0.81\(\pm\) 0.028 \\ BART-SWS & 0.82 & 0.81 & 0.74 & 0.77 & 0.79\(\pm\) 0.035 \\ Rule Based & 0.89 & 0.84 & 0.82 & 0.85 & 0.85\(\pm\) 0.025 \\ GPT-4o & 0.88 & 0.86 & 0.80 & 0.85 & 0.85\(\pm\) 0.028 \\ LLaMA & 0.86 & 0.83 & 0.77 & 0.79 & 0.81\(\pm\) 0.035 \\ MR+AS & **0.91** & **0.87** & **0.85** & **0.85** & **0.87 \(\pm\) 0.024** \\ \end{tabular}
\end{table}
Table 5: Median BARTScore ratios for top-2 candidates. The last column shows averages over all datasets with SDs. The best results are highlighted in **bold**.

 

better median CS values without the prompt specifying ranked candidates (0.954 and 0.943, respectively) relative to the alternative (0.953 and 0.940, respectively). Importantly, these are significantly lower than the median CS value for MR+AS (0.983). CS distributions for all models can be found in the Appendix Figure 7. We note that it may be possible to optimize the prompt for better ranked results, however, this is beyond the scope of our work.

Human StudyWe recruited five annotators to carry out the study. First, we randomly sampled 25 cases from the SWS test data and provided the top two candidate replacements proposed by our model. We randomly flipped the order of these two candidates before presenting them to the annotators. Annotators were asked to indicate their preference for the order and whether they preferred to replace the target word with either of the two candidates3.

Footnote 3: Example of questionnaire 1 can be accessed at [https://tinyurl.com/SWSQuestionnaire1](https://tinyurl.com/SWSQuestionnaire1).

To assess inter-annotator agreement, we calculated the (unweighted) Cohen's kappa coefficient for both order agreement and replacement agreement. The average kappa values for the order agreement and the replacement agreement were 0.40\(\pm\)0.21 (standard deviation) and 0.13\(\pm\)0.21, respectively. These results align with our assumption that asking annotators to suggest replacements for a specific word in a given sentence is a subjective task. In Appendix Table 12, we present the complete results. Furthermore, we observed that in 83% of the cases, the annotators did not choose to replace the target word.

To mitigate the potential bias caused by providing the target word in advance, we designed a second questionnaire consisting of 50 cases. Twenty-five cases were sampled where our model and BARTScore agreed, and twenty-five cases where they disagreed. In this setting, the target word was hidden and only the top two candidate replacements (with randomly flipped order) were provided. Annotators were then asked to choose their preference from three options4: (0) the top-1 and top-2 candidates are equally good. (1) the top-1 candidate is better. (2) the top-2 candidate is better.

Footnote 4: Example of questionnaire 2 can be accessed at [https://tinyurl.com/SWSQuestionnaire2](https://tinyurl.com/SWSQuestionnaire2).

Subsequently, we made a comparison between the model predictions and the responses of the annotators. The results showed that in 38.8% of the the instances, annotators considered both candidates equally good. In the remaining cases, when the model and BartScore agreed, 79% of annotators favored the same choice. In cases where the model and BartScore disagreed, annotators sided with BartScore 67% of the time and with the model 33% of the time. In the SWS test data, the model and BartScore agreed on the ranking of the top two candidates in 67% of instances. In Appendix Table 13, we present the complete results.

## 5 Discussion

This paper addressed the problem of training and evaluating SWS models without requiring human annotations. Specifically, we leveraged a model-based score (BARTScore) to define evaluation metrics and to serve as a label-proxy for our loss function. Extensive experimental results demonstrate the effectiveness of our approach relative to both MLM- and LLM-based models.

### Limitations

The proposed method leverages BARTScore as a sentence scoring mechanism, essentially acting as a proxy for human annotators in guiding model optimization during preference-aware learning. However, we cannot eliminate the possibility that BARTScore may not always accurately quantify the quality of token substitutions. While BARTScore has shown effectiveness in our experiments, it could be readily replaced by alternative scoring metrics deemed more reliable or better suited to other word substitution contexts. We leave this exploration as interesting future work.

### Ethical Considerations

In the process of developing and deploying smart word suggestion systems it is crucial to consider the broader ethical implications of relying on automated tools for language optimization. These systems may inadvertently reinforce biases present in training data, leading to unintended or inappropriate word suggestions. Moreover, the replacement of human judgment with automated scoring models like BARTScore can risk overlooking or misrepresenting nuances in language, particularly in sensitive contexts. Therefore, we recommend ongoing human oversight and continuous evaluation of the model's output to ensure fairness, inclusivity, and alignment with ethical standards.

 



## References

* Chern et al. (2023) I-Chun Chern, Zhiruo Wang, Sanjan Das, Bhavuk Sharma, Pengfei Liu, Graham Neubig, et al. 2023. Improving factuality of abstractive summarization via contrastive reward learning. _arXiv preprint arXiv:2307.04507_. Devlin (2018) Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_. Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. _arXiv preprint arXiv:2302.04166_. He et al. (2024) Zhiwei He, Xing Wang, Wenxiang Jiao, Zhuosheng Zhang, Rui Wang, Shuming Shi, and Zhaopeng Tu. 2024. Improving machine translation with human feedback: An exploration of quality estimation as a reward model. _arXiv preprint arXiv:2401.12873_. Kremer et al. (2014) Gerhard Kremer, Katrin Erk, Sebastian Pado, and Stefan Thater. 2014. What substitutes tell us-analysis of an "all-words" lexical substitution corpus. In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_, pages 540-549. Lewis (2019) M Lewis. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. _arXiv preprint arXiv:1910.13461_. Liu et al. (2023a) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023a. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. _ACM Computing Surveys_, 55(9):1-35. Liu et al. (2022) Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. Brio: Bringing order to abstractive summarization. _arXiv preprint arXiv:2203.16804_. Liu et al. (2023b) Yixin Liu, Kejian Shi, Katherine S He, Longtian Ye, Alexander R Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. 2023b. On learning to summarize with large language models as references. _arXiv preprint arXiv:2305.14239_. Lu et al. (2022) Qingyu Lu, Liang Ding, Liping Xie, Kanjian Zhang, Derek F Wong, and Dacheng Tao. 2022. Toward human-like evaluation for natural language generation with error analysis. _arXiv preprint arXiv:2212.10179_. Lu et al. (2023) Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2023. Error analysis prompting enables human-like translation evaluation in large language models. _arXiv preprint arXiv:2303.13809_. McCarthy and Navigli (2007) Diana McCarthy and Roberto Navigli. 2007. Semeval-2007 task 10: English lexical substitution task. In _Proceedings of the fourth international workshop on semantic evaluations (SemEval-2007)_, pages 48-53. Miao et al. (2024) Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. 2024. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. In _The Thirty-eighth Annual Conference on Neural Information Processing Systems_. Michalopoulos et al. (2021) George Michalopoulos, Ian McKillop, Alexander Wong, and Helen Chen. 2021. Lexsubcon: Integrating knowledge from lexical resources into contextual embeddings for lexical substitution. _arXiv preprint arXiv:2107.05132_. Moore (1999) Jason H Moore. 1999. Bootstrapping, permutation testing and the method of surrogate data. _Physics in Medicine & Biology_, 44(6):L11. Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. _arXiv preprint arXiv:1808.08745_. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744. Pavlick et al. (2015) Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 425-430. Qiang et al. (2023) Jipeng Qiang, Kang Liu, Yun Li, Yunhao Yuan, and Yi Zhu. 2023. Parals: lexical substitution via pretrained paraphraser. _arXiv preprint arXiv:2305.08146_. Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9. Rafael Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. _Advances in Neural Information Processing Systems_, 36. Tellham et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. _arXiv preprint arXiv:2004.04696_. Smola (2000) Alexander J Smola. 2000. _Advances in large margin classifiers_. MIT press

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Bapistse Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clementine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. _arXiv preprint arXiv:2310.16944_.
* Wang et al. (2023) Chenshuo Wang, Shaoguang Mao, Tao Ge, Wenshan Wu, Xun Wang, Yan Xia, Jonathan Tien, and Dongyan Zhao. 2023. Smart word suggestions for writing assistance. _arXiv preprint arXiv:2305.09975_.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. _Advances in Neural Information Processing Systems_, 34:27263-27277.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _preprint arXiv:2205.01068_.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.
* Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohuan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. _Advances in Neural Information Processing Systems_, 36:46595-46623.
* Zhou et al. (2019) Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2019. Bert-based lexical substitution. In _Proceedings of the 57th annual meeting of the association for computational linguistics_, pages 3368-3373.

 

DPO Derivation Details

Dpo*Direct Preference Optimization (DPO) Rafailov et al. (2024) is an efficient technique for aligning large language models (LLMs) with human feedback, which gained popularity due to its simplicity Miao et al. (2024). For instance, it has demonstrated to be effective in chat benchmarks Tunstall et al. (2023); Zheng et al. (2023). In our case, the model-based score serving as proxy for human feedback DPO (under the Plackett-Luce Model) is written as

\[{\mathcal{L}_{\mathrm{DPO}}=-\mathbb{E}\left[\log\prod_{k=1}^{K}\frac{\exp( \delta r_{k})}{\sum_{j=k}^{K}\exp(\tilde{r}_{j})}\right]~{},} \tag{11}\]

where \(r_{k}=\log p_{\theta}(\tilde{X}_{k})-\log p_{\hat{\theta}}(\tilde{X}_{k})\), the expectation is over \(\{\tilde{X}_{1},\dots,\tilde{X}_{K},X\}\), and \(\theta\) and \(\hat{\theta}\) denote the parameters of the model being trained and that used for reference, respectively. Accordingly, only \(\theta\) are updated while learning while \(\hat{\theta}\) are kept fixed.

Since the magnitude of the logits drops significantly as \(k\to K\), we found that the sum in the denominator of (11) weakens the loss. Therefore, we removed the sum and instead compared the \(k\)-th and the \((k+1)\)-th substitution, rather than comparing it with all \(K\) values. The Derivation of DPO* is shown below

\[{\mathcal{L}_{\text{DPO}^{*}}=-\log\prod_{k=1}^{K-1}\frac{\exp( \delta r_{k})}{\exp(\sigma r_{k+1})}}\] (12) \[\qquad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \ 

While being adapted only by first writing, the data gradually became aware of the critical role of law in our daily life.

Annotation: Human-("inspired," important," [...]," Medical ("cratest").

The residential theory argues that there is no Ethics choice strictness; and the success is big enough for bankless of people.

Annotation: Humans-("duscuscated," [...]," [.

 



\begin{table}
\begin{tabular}{c|c|c|c|c|c|c}  & OAC & CA & CD & OMC & NCA \\ \hline BERT-naive & **1** & 0.893 & 0.439 & 0.351 & **1** \\ BERT-spsv & **1** & 0.927 & 0.506 & 0.498 & **1** \\ BERT-SWS & **1** & 0.975 & 0.749 & 0.655 & **1** \\ BART-SWS & **1** & 0.897 & 0.618 & 0.625 & **1** \\ Rule Based & - & 0.531 & 0.247 & 0.332 & - \\ GPT-4o & - & 0.745 & 0.554 & 0.673 & - \\ LLaMA & - & 0.694 & 0.531 & 0.625 & - \\ MR+AS & **1** & **0.981** & **0.759** & **0.657** & **1** \\ \end{tabular}
\end{table}
Table 10: Proportion of \(p\)-values below the significance threshold (\(\alpha=0.01\)) for \(K_{s}=1000\) using GPTscore with OPT-350M (350M).

Figure 4: \(p\)-value distributions.

Figure 5: \(p\)-value distributions.

\begin{table}
\begin{tabular}{c|c|c|c|c|c}  & OAC & CA & CD & OMC & NCA \\ \hline BERT-naïve & **1** & 0.888 & 0.425 & 0.47 & **1** \\ BERT-spsv & **1** & 0.891 & 0.474 & 0.438 & **1** \\ BERT-SWS & **1** & 0.948 & 0.717 & 0.621 & **1** \\ BART-SWS & **1** & 0.869 & 0.6 & 0.595 & **1** \\ Rule Based & - & 0.552 & 0.248 & 0.309 & - \\ GPT-4o & - & 0.734 & 0.557 & 0.655 & - \\ LLaMA & - & 0.654 & 0.518 & 0.624 & - \\ MR+AS & **1** & **0.963** & **0.809** & **0.719** & **1** \\ \end{tabular}
\end{table}
Table 9: Proportion of \(p\)-values below the significance threshold (\(\alpha=0.01\)) for \(K_{s}=1000\) using GPTScore with GPT2-medium (355M).

\begin{table}
\begin{tabular}{l|c|c|c|c|c}  & OAC & CA & CD & OMC & NCA \\ \hline BERT-naive & **1** & 0.888 & 0.425 & 0.47 & **1** \\ BERT-spsv & **1** & 0.891 & 0.474 & 0.438 & **1** \\ BERT-SWS & **1** & 0.948 & 0.717 & 0.621 & **1** \\ BART-SWS & **1** & 0.869 & 0.6 & 0.595 & **1** \\ Rule Based & - & 0.552 & 0.248 & 0.309 & - \\ GPT-4o & - & 0.734 & 0.557 & 0.655 & - \\ LLaMA & - & 0.654 & 0.518 & 0.624 & - \\ MR+AS & **1** & **0.963** & **0.809** & **0.719** & **1** \\ \end{tabular}
\end{table}
Table 8: Proportion of \(p\)-values below the significance threshold (\(\alpha=0.01\)) for \(K_{s}=1000\) using BARTScore.





\begin{table}
\begin{tabular}{l|r r r r}  & \multicolumn{3}{c}{Order Agreement} \\  & Annotator1 & Annotator2 & Annotator3 & Annotation4 & Annotator5 \\ \hline Annotator1 & - & 0.28 & 0.51 & 0.75 & 0.44 \\ Annotator2 & - & - & 0.12 & 0.19 & 0.2 \\ Annotator3 & - & - & 0.41 & 0.44 \\ Annotator4 & - & - & - & 0.68 \\ Annotator5 & - & - & - & - \\ Average+SD & & & & 0.40\(\pm\)0.21 \\  & & & & \\  & & & & \\  & & & & \\  & & & & \\  & & & & \\  & & & & \\  & & & & \\ \end{tabular}
\end{table}
Table 12: Inter-annotator agreement using (unweighted) Cohen’s kappa coefficient. SD is the standard deviation.

Figure 6: \(p\)-value distributions.

\begin{table}
\begin{tabular}{l|r r r r}  & \multicolumn{3}{c}{Select same} & Select different & Select 0 \\ \hline Annotator1 & 7 & 1 & 17 \\ Annotator2 & 12 & 5 & 8 \\ Annotators3 & 14 & 9 & 2 \\ Annotator4 & 18 & 5 & 2 \\ Annotator5 & 8 & 0 & 17 \\ Average+SD & 11.8\(\pm\)4.5 & 4\(\pm\)3.6 & 9.2\(\pm\)7.5 \\  & Model and BARTScore Disagreement (25 cases) & & Age with BadTScore & Select 0 \\ \hline Annotator1 & 6 & 3 & 16 \\ Annotator2 & 9 & 7 & 9 \\ Annotator3 & 14 & 7 & 3 \\ Annotator4 & 14 & 8 & 3 \\ Annotator5 & 4 & 1 & 20 \\ Average+SD & 9.4\(\pm\)4.6 & 5.2\(\pm\)3.0 & 10.2\(\pm\)7.7 \\ \end{tabular}
\end{table}
Table 13: Human study results. The Model is our (MR+AS) model. SD is the standard deviation. Based on the last column, we calculated the proportion of cases in which humans selected 0 by dividing the last column’s value by 25 and computing the average, resulting in 38.8%. For the remaining cases (where the selection is either 1 or 2), we analyzed agreement between the annotator, the model, and BARTScore. In the top panel, the proportion of cases where the annotator agree with model and the BARTScore is calculated as the value in the first column divided by the sum of the first and second columns, resulting in an average of 79%. In the bottom panel, we computed two proportions: the agreement between the annotator and BARTScore, calculated as the value in the first column divided by the sum of the first and second columns, and the agreement between the annotator and the model, calculated as the value in the second column divided by the sum of the first and second columns. The resulting agreement proportions are 67% for annotator-BARTScore agreement and 33% for annotator-model agreement.