Identifying Reliable Evaluation Metrics for Scientific Text Revision
Authors: Leane Jourdan, Florian Boudin, Richard Dufour, Nicolas Hernandez
Accepted at: acl25

Identifying Reliable Evaluation Metrics for Scientific Text Revision

Léane Jourdan1, Florian Boudin1,2, Nicolas Hernandez1, Richard Dufour1,
1Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France
2JFLI, CNRS, Nantes University, France
Correspondence: leane.jourdan@univ-nantes.fr

Abstract

Evaluating text revision in scientific writing remains a challenge, as traditional metrics such as ROUGE and BERTScore primarily focus on similarity rather than capturing meaningful improvements. In this work, we analyse and identify the limitations of these metrics and explore alternative evaluation methods that better align with human judgments. We first conduct a manual annotation study to assess the quality of different revisions. Then, we investigate reference-free evaluation metrics from related NLP domains. Additionally, we examine LLM-as-a-judge approaches, analysing their ability to assess revisions with and without a gold reference. Our results show that LLMs effectively assess instruction-following but struggle with correctness, while domain-specific metrics provide complementary insights. We find that a hybrid approach combining LLM-as-a-judge evaluation and task-specific metrics offers the most reliable assessment of revision quality.

1 Introduction

Effective revision is a critical step in scientific writing, ensuring clarity, coherence, and adherence to academic standards. The writing process typically consists of four stages: 1) Prewriting, 2) Drafting, 3) Revising, and 4) Editing (Jourdan et al., 2023). The revision stage involves substantial modifications to improve readability, style, and formality (Du et al., 2022; Li et al., 2022). This step is particularly critical, as poor writing quality can obscure research findings and often contributes to paper rejection (Amano et al., 2023). As illustrated in Figure 1, the revision task takes an original paragraph and an instruction specifying the required modification as input. The expected output is a revised paragraph that aligns with the given instruction.

Given the importance of this task, reliable evaluation is crucial. Like other text generation tasks, text revision is assessed using well-established metrics such as ROUGE (Lin, 2004) or BERTScore (Zhang et al., 2020). While embedding-based metrics (e.g., BERTScore) capture some semantic similarity, they still primarily focus on surface-level features and lexical overlap, rather than capturing deeper aspects of text quality.

In text revision, similarity-based metrics alone fail to fully capture revision quality. Beyond surface similarity to a reference text, revision assessment requires considering improvements over the original version, meaning preservation, and adherence to the instruction. Several studies have relied on human evaluation to assess text revision systems (Du et al., 2022; Raheja et al., 2023, 2024; Ito et al., 2020; Schick et al., 2023). However, human evaluation is costly and time-consuming, making it impractical for large-scale or iterative assessments during system development. To address this limitation, we explore alternative automatic evaluation approaches that provide a more reliable and scalable assessment of revision quality.

Since text revision encompasses various sub-tasks (e.g., paraphrasing, summarization, text simplification, style transfer, grammar error correction (GEC)) (Li et al., 2022; Raheja et al., 2024; Ito et al., 2019; Kim et al., 2022), we first explore reference-free evaluation metrics commonly used to assess these tasks. These metrics compare the original and revised texts directly, rather than relying on a gold reference. Additionally, we ex-

![Overview of the text revision task](page_784_355_377_181.png)

Figure 1: Overview of the text revision task
plore different LLM-as-a-judge approaches, which can incorporate the revision instruction and potentially approximate human reasoning. With the rapid growth of LLMs, these approaches are increasingly used for evaluating diverse tasks (Gu et al., 2024). However, prior studies have shown that LLMs experience performance drops when no gold reference is provided, sometimes being outperformed by simpler methods, making them less appealing (Doostmohammadi et al., 2024; Mita et al., 2024). In this work, we aim to evaluate whether these results generalise to the text revision task and investigate the impact of providing a gold reference. Our contributions are as follows:

• We release ParaReval, a dataset of human pairwise evaluations of generated revisions.1

• We show that traditional similarity metrics fail to accurately evaluate text revision.

• We demonstrate that LLM-as-a-judge can effectively assess instruction following without the need for a gold reference.

• We find that similarity metrics complement LLM-as-a-judge in addressing challenging cases.

• While LLM-as-a-judge performs best, we show that the ParaPLUIE metric (Lemesle et al., 2025) can serve as a cost-effective alternative for measuring meaning preservation.

2 Related Work

In this section, we categorise evaluation approaches into three types: n-gram similarity metrics, embedding-based similarity metrics, and LLM-as-a-judge methods.

2.1 N-grams Similarity Metrics

N-gram-based similarity metrics have been the standard for evaluating text generation tasks. These metrics primarily measure lexical overlap between the generated text and the reference. However, they cannot capture semantic equivalence or improvements made over the original text. The most commonly used n-gram-based metrics are:

• BLEU (Papineni et al., 2002): Initially developed for machine translation evaluation, BLEU has been widely used in text revision tasks (Du et al., 2022; Raheja et al., 2024; Jourdan et al., 2024; Dwivedi-Yu et al., 2024; Mücke et al., 2023).

• ROUGE (Lin, 2004): Designed for summarization evaluation, ROUGE includes several variants, with ROUGE-L being the most commonly used in text revision (Du et al., 2022; Jourdan et al., 2024, 2025; Dwivedi-Yu et al., 2024).

• METEOR (Banerjee and Lavie, 2005): A unigram matching metric for machine translation, less sensitive to paraphrasing than BLEU. Used in text revision by Mücke et al. (2023).

• GLEU (Napoles et al., 2015): A variant of BLEU tailored for GEC, and used for text revision in (Dwivedi-Yu et al., 2024; Raheja et al., 2023). It takes the source text into account, rewarding corrections and crediting unchanged parts, while also penalizing ungrammatical edits.

• SARI (Xu et al., 2016): Designed for automatic text simplification, it is the most commonly used metric for evaluating text revision (Du et al., 2022; Raheja et al., 2023, 2024; Jourdan et al., 2024, 2025; Dwivedi-Yu et al., 2024). It compares the system’s output to both the reference and the source texts, rewarding correct additions, deletions, and retention of words.

BLEU, ROUGE and METEOR metrics score the similarity between the generated output and a reference text. SARI and GLEU are the only metrics that consider the source text, which is essential for assessing improvements in text revision. However, while they are interpretable, they struggle with tasks requiring deeper semantic understanding, such as evaluating instruction following or measuring improvements over the original text.

2.2 Embedding Similarity Metrics

Embedding-based metrics like BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019) or SemDist (Kim et al., 2021) are designed to capture semantic similarity beyond surface-level lexical overlap. These methods compare the embeddings of generated and reference texts to assess their alignment. In text revision, only BERTScore has been used (Mücke et al., 2023; Jourdan et al., 2024). It computes cosine similarity between contextualised embeddings from BERT for corresponding words in reference and generated sentences.

1https://github.com/JourdanL/parareval
2.3 LLM-as-a-Judge Approaches
Recent works have explored the use of LLM-as-a-judge for evaluating generation tasks to go beyond surface similarity. These approaches treat evaluation as a judgment task, where an LLM assesses generated text based on multiple criteria. Several classification schemes have been proposed: Gu et al. (2024) propose to categorise them into Scores, Yes or No, Pair and Multiple choice. Zheng et al. (2023) propose three different variations: pairwise comparison, single-answer grading(score) and reference-guided grading.

Notably, Doostmohammadi et al. (2024) proposed evaluating generated text on three dimensions: naturalness (does the generation sound natural and fluent?), relatedness (is the generation related to the prompt and follow the required format?), and correctness (is the generation correct?, with meaning varying by task). For text revision evaluation, Mita et al. (2024) designed a one-question pairwise comparison prompt and tested it in a zero and few-shot settings. However, their results showed that this approach underperformed compared to a fine-tuned BERT classifier.

For our LLM-as-a-judge approaches, we build on these works and we propose a combination of the three variations from (Zheng et al., 2023).

3 Experimental Setup
To examine the limitations of traditional similarity metrics, we first generate multiple revised outputs using various LLMs and manually evaluate them.

3.1 Dataset
We use the evaluation split of the ParaRev dataset (Jourdan et al., 2025), which contains 258 pairs of revised paragraphs extracted from scientific articles revised by the authors themselves. Each paragraph is annotated with two different revision instructions, resulting in a total of 516 data points. Additionally, each paragraph is labelled with its revision intention type, which will be used in our analysis, the taxonomy is provided in Table 1.

3.2 Revision Models
To ensure a diverse set of revision outputs in terms of quality, we generate revised paragraphs for each original paragraph + instruction pair using 6 different models. The models used are the following:

• CoEdIT-XL, a T5-based model fine-tuned for sentence revision (Raheja et al., 2023)

<table>
  <tr>
    <th>Type</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Light</td>
    <td>Minor changes in word choice or phrasing.</td>
  </tr>
  <tr>
    <td>Rewriting Medium</td>
    <td>Complete rephrasing of sentences within the paragraph.</td>
  </tr>
  <tr>
    <td>Heavy</td>
    <td>Significant rephrasing, affecting at least half of the paragraph.</td>
  </tr>
  <tr>
    <td>Concision</td>
    <td>Same idea, stated more briefly by removing unnecessary details.</td>
  </tr>
  <tr>
    <td>Content Deletion</td>
    <td>Modification of content through the deletion of an idea.</td>
  </tr>
</table>

Table 1: Taxonomy of revisions at paragraph level

• Open LLMs: Llama 3 8B Instruct, Llama 3 70B Instruct, Mistral 7B Instruct v0.2
• Closed-source LLMs: GPT 4o mini, GPT 4o

The prompts are provided in Appendix A.

3.3 Annotation Task
To identify which metrics best reflect the true quality of revisions, we conducted a manual evaluation comparing human judgments with automatic metric scores. For this, we designed an annotation task where human annotators compared pairs of revision candidates and selected their preferred version.

We carried out the annotation with the help of 10 annotators: 3 professors and 7 PhD students, all non-native English speakers but experienced in both reading and writing scientific papers in NLP. Each annotation instance consisted of two revision suggestions for a given paragraph, produced in Section 3.1, along with the corresponding revision instruction. Annotators answered a series of questions to assess the quality of the revisions:

• Q1A and Q1B Relatedness x2: Did model A/B address the instruction? {Yes strictly, Yes with additional modifications, No}
– If it was your article and your instruction:
  – Q2 Correctness: Which revisions would you consider acceptable? {Both, A only, B only, None}
  – Q3 Preference: Which revision would you prefer to include in your paper? {Both, A, B, None}

• Category-Specific Evaluation, {Both, A, B, None} are possible answers for each question:
  – Rewriting light: Which model improves the academic style and English the most?
  – Rewriting medium: Which model improves the readability and structure the most?
– Rewriting heavy: Which model improves the readability and clarity the most?
– Concision: Which model manages the most to give a shorter version while keeping all the important ideas?

A screenshot of the annotation environment is available in Appendix B. To ensure fair evaluation, we balanced the pairwise comparisons across models, ensuring that each model was compared to the others a similar number of times.

3.4 Annotation Phase Results

From the evaluation subset of the dataset, we generated 1,548 pairs of revised paragraphs for annotation. Among these, 129 pairs (8.33%) received double annotations to measure inter-annotator agreement. The agreement scores (Cohen’s Kappa \( \kappa \)) for each question are reported in Table 2.

For our analysis, as we are studying the metrics’ capacity to identify the best revision among two propositions, we introduce the notion of Extended Preference. Even if annotators select None for the Preference question, a model is still considered preferable if it is the only one Correct or Related to the instruction. We then consider the leftovers Both and None as Ties.

<table>
  <tr>
    <th>Question</th>
    <th>\( \kappa \)</th>
    <th>Agreement</th>
  </tr>
  <tr>
    <td>Relatedness</td>
    <td>0.54</td>
    <td>Moderate</td>
  </tr>
  <tr>
    <td>Correctness</td>
    <td>0.55</td>
    <td>Moderate</td>
  </tr>
  <tr>
    <td>Preference</td>
    <td>0.33</td>
    <td>Fair</td>
  </tr>
  <tr>
    <td>Concision</td>
    <td>0.22</td>
    <td>Fair</td>
  </tr>
  <tr>
    <td>Rewriting light</td>
    <td>0.41</td>
    <td>Moderate</td>
  </tr>
  <tr>
    <td>Rewriting medium</td>
    <td>0.48</td>
    <td>Moderate</td>
  </tr>
</table>

Table 2: Cohen’s Kappa (\( \kappa \)) for each question.

Figure 2 presents the distribution of human preferences across models. Based on human annotations, GPT-4o emerges as the best-performing revision model, being strictly favoured in 58.33% of comparisons. Llama 3 70B follows, with a preference rate of 53.68%. When doing pairwise comparison of revision models, we observe that Llama 3 70B is in a tie with GPT-4o. For more details, we also report the pairwise preferences on revision models in Appendix C.

![Bar chart showing the distribution of human extended preference for each revision model.](page_1012_120_388_246.png)

Figure 2: Distribution of human extended preference for each revision model. The green area indicates cases where the model is preferred.

4.1 Performance of Revision Models with Similarity-based Metrics

To determine the best revision model, we evaluate each generated revision using traditional similarity-based metrics by comparing them to a reference. Additionally, we compare the scores of these models with a no edits baseline that simply recopies the input as output. Results are presented in Table 3. We observe that all metrics, except GLEU, consider the no edits baseline to be the best-performing approach, with CoEdIT-XL also being a strong contender. However, upon manual inspection, we find that CoEdIT-XL tends to perform minimal revisions, such as correcting grammar and typos or, in some cases, excessively deleting parts of the paragraph. This suggests that these metrics favour not making any changes rather than rewarding meaningful, in-depth revisions.

4.2 Redundancy and Correlation Among Metrics

To further investigate this issue, we analyse the correlation between different metrics and their relationship with edit distance (Levenshtein distance). We compute the Pearson correlation coefficient between all metrics (see Figure 3). We observe that most metrics are highly correlated, suggesting they provide redundant information. The only exception is SARI, which differs from most other metrics because it considers the original text, the generated revision, and the reference. These results suggest that, although we aim to use different metrics to study the revisions from various angles, most of them ultimately convey the same information.
<table>
  <tr>
    <th>Rev. Model</th>
    <th>BLEU</th>
    <th>ROUGE-L</th>
    <th>METEOR</th>
    <th>GLEU</th>
    <th>SARI</th>
    <th>Bertscore</th>
    <th>BLANC</th>
    <th>BETS</th>
    <th>ParaPLUIE</th>
  </tr>
  <tr>
    <td>no edits | gold</td>
    <td><b>66.00</b></td>
    <td><b>78.30</b></td>
    <td><b>83.80</b></td>
    <td>25.78</td>
    <td><b>60.63</b></td>
    <td><b>95.95</b></td>
    <td><b>55.21</b></td>
    <td><b>2.461</b></td>
    <td><b>20.93</b></td>
  </tr>
  <tr>
    <td>CoEdit-XL</td>
    <td>50.24</td>
    <td>67.46</td>
    <td>66.66</td>
    <td>23.84</td>
    <td>39.60</td>
    <td>93.90</td>
    <td><b>58.96</b></td>
    <td>1.554</td>
    <td>19.35</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>27.77</td>
    <td>50.79</td>
    <td>54.02</td>
    <td>15.38</td>
    <td>31.63</td>
    <td>92.14</td>
    <td>41.59</td>
    <td><b>2.491</b></td>
    <td><b>23.02</b></td>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>41.66</td>
    <td>62.07</td>
    <td>62.00</td>
    <td>25.78</td>
    <td>39.33</td>
    <td>93.53</td>
    <td>49.09</td>
    <td>2.364</td>
    <td>22.67</td>
  </tr>
  <tr>
    <td>Llama-3-70B</td>
    <td>46.78</td>
    <td>65.61</td>
    <td>67.20</td>
    <td>30.31</td>
    <td>42.74</td>
    <td>93.90</td>
    <td>52.27</td>
    <td>2.386</td>
    <td>22.58</td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>51.68</td>
    <td>69.54</td>
    <td>72.70</td>
    <td><b>32.67</b></td>
    <td>45.06</td>
    <td>94.80</td>
    <td><b>54.89</b></td>
    <td><b>2.497</b></td>
    <td>22.74</td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td>49.34</td>
    <td>68.20</td>
    <td>69.88</td>
    <td>31.35</td>
    <td>43.54</td>
    <td>94.45</td>
    <td>53.62</td>
    <td>2.454</td>
    <td><b>22.86</b></td>
  </tr>
</table>

Table 3: Results on the paragraph revision task with traditional and alternative metrics.

![Correlation matrix showing similarity metrics against each other and edit distance](page_1012_670_377_312.png)

Figure 3: Correlation of the similarity metrics against each other and to the negative edit distance.

4.3 Sensitivity of Metrics to Edit Distance

The first two columns of Figure 3 show a strong correlation between similarity metrics and negative edit distance, both in relation to the original and the reference paragraph. This relation is further illustrated in Appendix D. Two key observations emerge:

• First, metrics only capture surface similarity. The high correlation with the distance between the reference and generated revision suggests that traditional metrics mostly reflect how closely a model replicates the reference revision, rather than evaluating the quality of the revision itself. Even BERTScore, despite being based on embeddings and computationally more expensive, ultimately provides similar information to simpler distance-based metrics.

• Second, substantial revisions are penalised. The strong correlation between metric scores and the distance between the original and generated text indicates that the more a revision deviates from the original paragraph, the lower its score. This suggests that traditional metrics do not reward substantial, qualitative improvements, such as restructuring sentences or enhancing clarity. Instead, they encourage conservative edits that closely match the reference.

This phenomenon creates a major evaluation bias: Models that produce minimal edits receive higher scores and valid, but different, improvements are penalised. In many cases, making no revision at all results in a higher score than making meaningful changes, as exemplified in Figure 4.

Among all metrics, SARI and GLEU stand out by having a lower correlation to edit distance (\( \leq 0.52 \)), as they explicitly penalise unchanged text edits, thereby encouraging revision.

5 Exploring Alternative Evaluation Approaches

The goal of this section is to identify evaluation metrics that better correlate with human assessments of text revision quality.

5.1 Metrics from Related NLP Domains

We hypothesise that in text revision, an essential factor is the comparison to the original text, as the metrics computing the similarity to a reference revision tend to overlook whether the modification effectively improves the original text. SARI and GLEU are widely used in text revision because they consider both the original and reference texts.

Additionally, text revision encompasses various subtasks depending on the type of modification conducted. Raheja et al. (2024) classified revisions into three main categories: GEC, Simplification, and Paraphrasing. They evaluated each with a distinct set of metrics. This suggests that a single metric may not be sufficient for text revision, as we are not trying to capture the same phenomenon depending on the type of revision.

Inspired by these ideas, we explore metrics from related NLP domains, selecting those that consider the original text and align with specific types of
Figure 4: Overview of the evaluation in a case where automatic evaluation (here SARI) and human judgment don’t align.

revision (e.g., text summarization metrics for concision tasks or paraphrase evaluation metrics for rewriting tasks). We identify three candidate metrics, taking the original and generated revised paragraph as input:

• BETS (Zhao et al., 2023): Designed for text simplification to assess meaning preservation and comparative simplicity at the level of modified word pairs, using BERT embeddings.

• BLANC (Vasilyev et al., 2020) (BLANC-help variant): Designed for document summarization as a replacement for ROUGE. It measures how helpful a summary is to understand a text, using a BERT-based model.

• ParaPLUIE (Lemesle et al., 2025): A metric for paraphrase detection that prompts Mistral 7B and uses perplexity scores when suggesting a Yes or No answer- instead of the generated text.

Table 3 presents the evaluation results using these three candidate metrics, we also add the results comparing the original paragraph to the gold revision for comparison. BETS and ParaPLUIE present a similar ranking of their preferred models and rank CoEdIT-XL last like human annotation. Conversely, BLANC follows the one of similarity-based metrics.

Figure 5 reports the correlations between these new metrics and negative edit distance. Except for BLANC, these metrics showcase a low correlation to the edit distance. Additionally, in Appendix D, we visualise these correlation relations.

BETS and ParaPLUIE emerge as promising candidates for evaluating text revision, while BLANC appears less suitable. We further investigate their alignment with human annotations to confirm their effectiveness in Sections 5.3 and 6.

![Correlation matrix showing the relationships between out-of-domain metrics and edit distance](page_1012_1092_388_246.png)

Figure 5: Correlation of the out-of-domain metrics against each other and to the negative edit distance.

5.2 LLM-as-a-Judge

An additional hypothesis is that text revision should not only account for the original text but also assess the model’s ability to follow instructions effectively. We explore LLM-as-a-Judge approaches to evaluate both these aspects.

We experiment with different approaches for LLM-as-a-judge, based on the work of Doostmohammadi et al. (2024), who employed GPT-4 as a judge to evaluate three key criteria in generated text: 1) Naturalness, 2) Relatedness, and 3) Correctness. Since our task involves modifying existing text rather than generating it from scratch, Naturalness is not relevant to our evaluation. However, Relatedness (whether the revision follows the instruction), aligns with Q1A and Q1B from our
human annotation and Correctness (whether the revision is acceptable) aligns directly with Q2. We structure our prompt similarly to the human annotation task to evaluate these two aspects.

We explore two approaches from Gu et al. (2024) for using LLMs as judges: Generating scores (*LLM-Likert*) where the model is presented individual revisions to grade them, and Yes or No questions + Pairwise comparisons (*LLM-Choice*) where the model is presented with pairs of revisions to select its preferred one or declare a tie. As Doostmohammadi et al. (2024) pointed out a drop in performance when GPT-4o was not provided a gold reference, we experimented with these two approaches with and without a gold reference. The prompts are provided in Appendix E.

Since our revision candidates were generated by multiple LLMs, we ensure a fair evaluation by also using multiple LLMs as judges. This helps to reduce potential bias, where a model might favour its own outputs. In Appendix F, we analyse the preferences of each LLM judge and discuss this potential bias.

We averaged agreements over three runs for all models except GPT-4o due to cost constraints. However, in the main body of the paper, we further average the results of all models per approach to present them more concisely.

The results with LLM-as-a-judge approaches are reported in Table 4. All approaches agree with human annotation on CoEdIT-XL being the less performing revision model and on LLama70B being a strong option for the task. However, LLM-likert scores are really close.

<table>
  <tr>
    <th rowspan="2">Rev. Model</th>
    <th colspan="2">LLM-choice</th>
    <th colspan="2">LLM-likert</th>
  </tr>
  <tr>
    <th>base</th>
    <th>w. gold</th>
    <th>base</th>
    <th>w. gold</th>
  </tr>
  <tr>
    <td>CoEdIT-XL</td>
    <td>6.15</td>
    <td>8.82</td>
    <td>3.385</td>
    <td>3.487</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td><b>64.95</b></td>
    <td>56.73</td>
    <td><b>4.816</b></td>
    <td><b>4.497</b></td>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>58.88</td>
    <td>57.08</td>
    <td>4.789</td>
    <td>4.472</td>
  </tr>
  <tr>
    <td>Llama-3-70B</td>
    <td>59.34</td>
    <td><b>60.20</b></td>
    <td>4.784</td>
    <td>4.479</td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>50.52</td>
    <td>52.33</td>
    <td>4.750</td>
    <td>4.447</td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td>50.85</td>
    <td>51.41</td>
    <td>4.743</td>
    <td>4.443</td>
  </tr>
</table>

Table 4: Results on the paragraph revision task with LLM-as-a-judge evaluation. For LLM-choice, we report the extended preference and for LLM-likert, we report the average of all criteria.

5.3 Results

After identifying all the candidate metrics, we assess their alignment with human judgement using three distinct measures to convey this agreement: Cohen’s Kappa (\( \kappa \)), Cramér’s V (\( V \)), and Pairwise Accuracy to account for ties (Deutsch et al., 2023). Table 5 reports the alignment of automatic evaluation methods with human judgments, we add a random baseline for comparison. As BLEU, ROUGE-L and METEOR are highly correlated, we report only ROUGE-L results in the main body of this paper.

LLM-Choice emerges as the most reliable evaluation option, followed by ParaPLUIE. LLM-Likert and GLEU also exhibit strong alignment. However, while LLM-Likert achieves higher accuracy when making a decision, it tends to overclassify cases as ties (See Appendix G).

<table>
  <tr>
    <th>Judge</th>
    <th>Pair acc.</th>
    <th>V</th>
    <th>\( \kappa \)</th>
  </tr>
  <tr>
    <td>Avg. LLM choice</td>
    <td><b>0.496</b></td>
    <td>0.239</td>
    <td><b>0.247</b></td>
  </tr>
  <tr>
    <td>Avg. LLM likert</td>
    <td>0.338</td>
    <td><b>0.240</b></td>
    <td>0.181</td>
  </tr>
  <tr>
    <td>ParaPLUIE</td>
    <td><b>0.477</b></td>
    <td>0.225</td>
    <td><b>0.197</b></td>
  </tr>
  <tr>
    <td>BETS</td>
    <td>0.437</td>
    <td>0.152</td>
    <td>0.127</td>
  </tr>
  <tr>
    <td>BLANC</td>
    <td>0.312</td>
    <td>0.117</td>
    <td>-0.080</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>0.395</td>
    <td>0.161</td>
    <td>0.034</td>
  </tr>
  <tr>
    <td>SARI</td>
    <td>0.416</td>
    <td>0.184</td>
    <td>0.071</td>
  </tr>
  <tr>
    <td>GLEU</td>
    <td>0.448</td>
    <td>0.193</td>
    <td>0.138</td>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>0.373</td>
    <td>0.179</td>
    <td>-0.013</td>
  </tr>
  <tr>
    <td><i>Random</i></td>
    <td><b>0.264</b></td>
    <td><b>0.027</b></td>
    <td>-0.006</td>
  </tr>
</table>

Table 5: Alignment of automatic metrics with human judgements across all data. Pairwise accuracy and Cramér’s V are defined on [0:1] and Cohen’s Kappa on [-1:1].

6 Performance by Aspects

In this section, we further analyse performance at a finer granularity on two aspects to see if the performances vary with the type of revision or the difficulty to discriminate the pair of propositions.

6.1 Performance by Revision Category

To test our hypothesis from Section 5.1, we analyse the alignment of human and automatic judgments across different revision types using ParaRev label annotations (see Figure 6). For most categories, LLM-Choice is the most reliable evaluation approach. However, for cases where paragraph content is minimally altered (Rewriting Light, Rewriting Medium and Concision), ParaPLUIE appears as a good alternative to capture meaning-preservation, as it is less costly than LLM-as-a-judge approaches. For instance, it processed our
Figure 6: Alignment of automatic metrics with human annotations by revision category

dataset in just 11 minutes, compared to 1 hour and 22 minutes required for Mistral 7B-Choice on a V100 GPU.

For Content Deletion, n-gram similarity-based metrics such as GLEU and SARI offer cost-efficient alternatives, aligning as well as or better than LLM-Choice with human preferences by leveraging reference-based deletion information.

Finally, for Rewriting Heavy, BETS outperforms other metrics and aligns better to human annotation than in other categories of revisions. In this category, the meaning of the paragraph must remain the same, while undergoing in-depth restructuring and rephrasing of most of the content. In the dataset, many of the instructions associated with these paragraphs focus on making them clearer, more readable, or easier to understand, which can be linked to the task of text simplification. BETS is a balanced score between meaning preservation and text simplification, which likely explains its strong performance in this category.

6.2 Performance by Difficulty
To further assess metric effectiveness, we analyse performance across varying difficulty levels. We categorise revision pairs based on human annotation difficulty levels:

• Easy Cases (530 pairs): defined by Q1A and Q1B, one model followed the instruction while the other did not.

• Medium Cases (214 pairs): defined by Q2, both models followed the instruction, but only one produced an acceptable revision.

• Hard Cases (575 pairs): defined by Q3: Both revisions were acceptable, but one was preferred.

We report the alignment by difficulty in Figure 7. We observe that \( LLM\text{-}Choice \) performs best in easy cases, achieving 0.821 accuracy. This suggests that LLMs are particularly effective at recognising whether a revision follows the given instruction, an ability that none of the other metrics possess. However, for medium cases, where both revisions comply with the instruction, similarity-based metrics outperform LLMs in identifying the best option. We posit that these metrics can leverage information from the gold reference to assess the expected revision more effectively.

For hard cases, none of the metrics perform well, with all methods showing low alignment with human judgments, as the task becomes even more subjective. In such situations, the preference for one revision over another may depend largely on the original author’s writing style and intent, making automatic evaluation difficult. However, Para-PLUIE seems to be the best option for evaluating these hard cases, ensuring that the original meaning of the paragraph is preserved during the revision process and preventing the revision model from hallucinating.

Since LLMs struggle with correctness assessment, we further analyse this aspect in Appendix H, correlating results with preliminary human annotation questions. A full alignment breakdown including all metrics is available in Appendix I.

6.3 Impact of Providing Gold Reference for LLM-as-a-Judge Approaches
We examine whether providing the gold reference influences the performance of LLM-as-a-judge methods and find minimal impact. \( LLM\text{-}Choice \) accuracy vary slightly from 0.496 to 0.494 when provided with the gold reference and \( LLM\text{-}Likert \) from 0.338 to 0.363. Our findings contrast with Doostmohammadi et al. (2024), who reported that in the absence of a reference, GPT-4o exhibited weaker alignment with human judgments. This suggests that LLMs rely heavily on their own internal reasoning rather than direct comparisons to a gold-
Figure 7: Alignment of automatic metrics with human annotations, by difficulty. The triangles in the first column represent the agreement on the total dataset from Table 5

standard revision. More details in Appendix J.

7 Discussion and Conclusion

In this article, we identified the most reliable metrics to evaluate scientific text revision. The identification of those metrics will allow a better analyse of existing text revisions strategies and designing new models for this task.

Our results suggest that LLMs-as-a-judge methods effectively assess whether a revision follows instructions but struggle to distinguish between two strong candidates and need to be completed by other metrics. Traditional similarity metrics, while not designed to assess instruction-following, prove valuable in differentiating between valid revisions. Their ability to compare revisions against a reference provides a tie-breaking mechanism when LLMs fail to make a clear distinction.

However, LLM-as-a-judge methods remain computationally expensive. To mitigate this, we recommend using a smaller, complementary set of metrics that strikes a balance between cost, interpretability, and alignment with human judgments. This subset could include a small LLM to evaluate instruction-following, ParaPLUIE for meaning preservation, and similarity-based metrics like SARI and GLEU, which leverage information from the gold reference to help differentiate between more challenging cases.

8 Limitations

The primary limitation of this work is the size of the dataset, as we were limited by the size of the evaluation split of ParaRev dataset and only had a limited number of volunteer researchers for manual annotation. As it need to be conducted with qualified annotators (researchers) our human resources are very limited, leading to a smaller double-annotated subset. A larger amount of annotated data would enhance the reliability of our analysis, strengthening the claims we made in this paper.

Additionally, preference-based annotation is inherently subjective, as reflected in the Cohen’s Kappa scores in Table 2. For the ParaReval dataset, we first annotated a double-annotated subset and retained the annotations from researchers with the highest agreement. Those with the highest agreement scores continued the annotation process to enhance reliability. The choice of annotators (similar background) may also introduce a bias.

We were also limited by computational resources, as LLMs use a lot of energy. Our priority was to ensure that our results were model independent, so we conducted the same experience with different models but using only one prompt. However, it would have been preferable to also check the prompt-independent aspect (Mizrahi et al., 2024).

Finally, numerous methods and metrics have been proposed to evaluate tasks in text generation throughout the years. To keep our analysis clear, we considered a limited number of them. For the LLM-based approaches, a larger number of runs would have been preferable for GPT-4o but due to cost issues we had to limit it to one run per approach.

9 Ethical Considerations

Data availability All the data are from the ParaRev corpus, the paragraphs are extracted from scientific articles collected on OpenReview where they fall under different "non-exclusive, perpetual, and royalty-free license" 2.

Computational resources

2https://openreview.net/legal/terms
• To generate revisions with Co-edit and experiment with BERT-based metrics, we used a local GeForce RTX 2080 11Go GPU for approximately 12 hours.

• To use ParaPLUIE and the different open LLMs to generate revisions and evaluation, we used V100 and A100 GPUs for a total of 249 hours on a supercomputer, equivalent to 0.009 tons of \( CO_2 \).

• To use GPT-4o mini and GPT-4o we spent 29.01$ spent on GPT API credits.

Acknowledgments

We thank Anas Belfathi, Maël Houbre, Trung Hieu Ngo, Xavier Pillet, Mohamed Reda Marzouk and Thomas Sebbag for their participation in human evaluation.

We thank Thomas Sebbag for proof-reading this paper.

This work was granted access to the HPC resources of IDRIS under the allocations 2023-AD011013901R1, 2024-AD011013901R2 and 2024-AD011014882R1 made by GENCI.

References

Tatsuya Amano, Valeria Ramírez-Castañeda, Violeta Berdejo-Espinola, Israel Borokini, Shawan Chowdhury, Marina Golivets, Juan David González-Trujillo, Flavia Montaño-Centellas, Kumar Paudel, Rachel Louise White, et al. 2023. The manifold costs of being a non-native english speaker in science. PLoS Biology, 21(7):e3002184.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.

Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914–12929.

Ehsan Doostmohammadi, Oskar Holmström, and Marco Kuhlmann. 2024. How reliable are automatic evaluation methods for instruction-tuned LLMs? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6321–6336, Miami, Florida, USA. Association for Computational Linguistics.

Wanyu Du, Zae Myung Kim, Vipul Runderstandahjea, Dhruv Kumar, and Dongyeop Kang. 2022. Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 96–108, Dublin, Ireland. Association for Computational Linguistics.

Jane Dwivedi-Yu, Timo Schick, Zhengbao Jiang, Maria Lomeli, Patrick Lewis, Gautier Izacard, Edouard Grave, Sebastian Riedel, and Fabio Petroni. 2024. EditEval: An instruction-based benchmark for text improvements. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 69–83, Miami, FL, USA. Association for Computational Linguistics.

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594.

Takumi Ito, Tatsuki Kuribayashi, Masatoshi Hidaka, Jun Suzuki, and Kentaro Inui. 2020. Langsmith: An interactive academic text revision system. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 216–226, Online. Association for Computational Linguistics.

Takumi Ito, Tatsuki Kuribayashi, Hayato Kobayashi, Ana Brassard, Masato Hagiwara, Jun Suzuki, and Kentaro Inui. 2019. Diamonds in the rough: Generating fluent sentences from early-stage drafts for academic writing assistance. In Proceedings of the 12th International Conference on Natural Language Generation, pages 40–53, Tokyo, Japan. Association for Computational Linguistics.

Léane Jourdan, Florian Boudin, Richard Dufour, Nicolas Hernandez, and Akiko Aizawa. 2025. ParaRev : Building a dataset for scientific paragraph revision annotated with revision instruction. In Proceedings of the First Workshop on Writing Aids at the Crossroads of AI, Cognitive Science and NLP (WRAICOGS 2025), pages 35–44, Abu Dhabi, UAE. International Committee on Computational Linguistics.

Léane Jourdan, Florian Boudin, Nicolas Hernandez, and Richard Dufour. 2024. CASIMIR: A corpus of scientific articles enhanced with multiple author-integrated revisions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2883–2892, Torino, Italia. ELRA and ICCL.

Léane Jourdan, Florian Boudin, Richard Dufour, and Nicolas Hernandez. 2023. Text revision in scientific writing assistance: An overview. In 13th International Workshop on Bibliometric-enhanced Information Retrieval (BIR), number 3617 in CEUR Workshop Proceedings, pages 22–36, Aachen.

Suyoun Kim, Abhinav Arora, Duc Le, Ching-Feng Yeh, Christian Fuegen, Ozlem Kalinli, and Michael L.
Seltzer. 2021. Semantic distance: A new metric for asr performance analysis towards spoken language understanding. In Interspeech 2021, pages 1977–1981.

Zae Myung Kim, Wanyu Du, Vipul Raheja, Dhruv Kumar, and Dongyeop Kang. 2022. Improving iterative text revision by learning where to edit from other revision tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9986–9999, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Quentin Lemesle, Jonathan Chevelu, Philippe Martin, Damien Lolive, Arnaud Delhay, and Nelly Barbot. 2025. Paraphrase generation evaluation powered by an LLM: A semantic metric, not a lexical one. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8057–8087, Abu Dhabi, UAE. Association for Computational Linguistics.

Jingjing Li, Zichao Li, Tao Ge, Irwin King, and Michael Lyu. 2022. Text revision by on-the-fly representation optimization. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 58–59, Dublin, Ireland. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Masato Mita, Keisuke Sakaguchi, Masato Hagiwara, Tomoya Mizumoto, Jun Suzuki, and Kentaro Inui. 2024. Towards automated document revision: Grammatical error correction, fluency edits, and beyond. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 251–265, Mexico City, Mexico. Association for Computational Linguistics.

Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949.

Justin Mücke, Daria Waldow, Luise Metzger, Philipp Schauz, Marcel Hoffman, Nicolas Lell, and Ansgar Scherp. 2023. Fine-tuning language models for scientific writing support. In Machine Learning and Knowledge Extraction, pages 301–318, Cham. Springer Nature Switzerland.

Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 588–593, Beijing, China. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Vipul Raheja, Dimitris Alikaniotis, Vivek Kulkarni, Bashar Alhafni, and Dhruv Kumar. 2024. mEditT: Multilingual text editing via instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 979–1001, Mexico City, Mexico. Association for Computational Linguistics.

Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. 2023. CoEditT: Text editing by task-specific instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5274–5291, Singapore. Association for Computational Linguistics.

Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2023. PEER: A collaborative language model. In The Eleventh International Conference on Learning Representations.

Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020. Fill in the BLANC: Human-free quality estimation of document summaries. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11–20, Online. Association for Computational Linguistics.

Wei Xu, Courtney Napoles, Ellie Pavlick, Quanzhe Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578, Hong Kong, China. Association for Computational Linguistics.

Xinran Zhao, Esin Durmus, and Dit-Yan Yeung. 2023. Towards reference-free text simplification evaluation with a BERT Siamese network architecture. In Findings of the Association for Computational Linguistics:
ACL 2023, pages 13250–13264, Toronto, Canada. Association for Computational Linguistics.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
A  Text Revision Prompt

Prompt segment 1: Text revision prompt messages

system_message= """You are a writing assistant specialised in academic writing.
Your task is to revise the original paragraph from a research paper draft that will be given according to the author's instruction. The input will follow the pattern ' <author_instruction> : "<Original_paragraph>" '. Please answer only by "Revised paragraph: <revised_version_of_the_paragraph>". Please limit your modifications only to what is requested in the author's instruction. Do not make any other modifications to the rest of the paragraph ."""

user_message= """{instruct} : "\{parag\}" """

B  Annotation Environment

See Figure 8

<table>
  <tr>
    <th>sidKm05v3wS98erM_03</th>
    <th>5220</th>
    <th>Instruction</th>
    <th>Model A</th>
    <th>Model B</th>
  </tr>
  <tr>
    <td>Original paragraph</td>
    <td></td>
    <td>Hyperparameter tuning: For our PHN method we select hyperparameters based on the HV computed on a validation set. Selecting hyperparameters for the baselines is non-trivial as there is no clear criteria that is reasonable in terms of runtime. In order to select hyperparameters based on HV, each approach needs to be trained multiple times on all rays. We therefore select hyperparameters based on a single ray, and apply those for all rays. Our selection criterion is as follows: we collect all models trained using all hyperparameter configurations, and filter out the dominated solutions. Finally, we select the combination of hyperparameters with the highest uniformly.</td>
    <td>Hyperparameter tuning: For our PHN method, we select hyperparameters based on the HV computed on a validation set. Selecting hyperparameters for the baselines is non-trivial as there is no clear criteria that is reasonable in terms of runtime. To select hyperparameters based on HV, each approach would need to be trained multiple times on all rays. Therefore, we select hyperparameters based on a single ray, and apply those for all rays. Our selection criterion is as follows: we collect all models trained using all hyperparameter configurations and filter out the dominated solutions. Finally, we select the combination of hyperparameters with the highest uniformly.</td>
    <td>Hyperparameter tuning: For our PHN method, we determine hyperparameters based on the Hypervolume (HV) computed on a validation set. Selecting hyperparameters for the baselines is non-trivial as there is no clear criteria that is reasonable in terms of runtime. Since each approach requires multiple training iterations to select hyperparameters based on HV, we instead choose hyperparameters based on a single ray. Our hyperparameter selection criteria involves collecting all models produced using all configurations and filtering out the dominated solutions. The combination of hyperparameters achieving the highest uniformly is subsequently selected from the remaining solutions.</td>
  </tr>
  <tr>
    <td>Did model A address the instruction?</td>
    <td>Yes strictly</td>
    <td></td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>If it was your article and your instructions:</td>
    <td>Yes with additional modifications</td>
    <td></td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Would you consider this revision acceptable?</td>
    <td>Both</td>
    <td></td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Which revision would you prefer to include in your final version?</td>
    <td>B</td>
    <td></td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Which model improves the readability and structure the most?</td>
    <td>B</td>
    <td></td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Figure 8: Screenshot of the annotation environment
C  Human Pairwise Revision Model Preferences

See Figure 9

![Pairwise comparison of human preferences on revision models.](page_324_186_1047_453.png)

Figure 9: Pairwise comparison of human preferences on revision models.
D  Relation between Metrics and Levenshtein Distance

See Figures 10, 11, 12 and 13.

![Scatter plots showing the distribution of similarity metrics scores based on Levenshtein distance between the reference and generated paragraph.](page_186_184_1077_1012.png)

Figure 10: Distribution of similarity metrics scores based on Levenshtein distance between the reference and generated paragraph.
Figure 11: Distribution of similarity metrics scores based on Levenshtein distance between the original and generated paragraph.
Figure 12: Distribution of alternative metrics scores based on Levenshtein distance between the reference and generated paragraph.

Figure 13: Distribution of alternative metrics scores based on Levenshtein distance between the original and generated paragraph.
E  LLM-as-a-Judge Prompts

Our prompts are inspired by the ones used in Doostmohammadi et al. (2024). See Figures 14, 15 and 16 for longer prompt sections.

Prompt segment 2: LLM Choice System message without Gold

You are an evaluator of academic writing on the task of text revision.
In this task, two revision models have been provided with the original paragraph written for a scientific article and a revision instruction on how to revise the paragraph.
You will be given the proposition from the two different models and several questions to determine the quality of those propositions and identify the best one. In your answer please only provide the answers to the questions.

Prompt segment 3: LLM Choice System message with Gold

You are an evaluator of academic writing on the task of text revision.
In this task, two revisions models have been provided with the original paragraph written for a scientific article and a revision instruction on how to revise the paragraph.
You will be given the proposition from the two different models and several questions to determine the quality of those proposition and identify the best one. To help you in this task you will also be given the gold paragraph which is the version revised by the author themselves.
In your answer please only provide the answers to the questions.

Prompt segment 4: Category Questions for LLM Choice with and without gold

Rewriting_light:"""Which model improves the academic style and English the most?"""

Rewriting_medium:"""Which model improves the readability and structure the most?"""

Rewriting_heavy:"""Which model improves the readability and clarity the most ?"""

Concision:"""Which model manages the most to give a shorter version while keeping all the important ideas?"""
Prompt segment 5: LLM Choice User message with and without Gold

[BEGIN DATA]
***
[Original paragraph]: "{original_paragraph}"
***
[Revision instruction]: "{instruction}"
***
[Model A]: "{modelA_generated_revised_paragraph}"
***
[Model B]: "{modelB_generated_revised_paragraph}"
***
[END DATA]

1. Did model A address the instruction? Answer "Yes strictly", "Yes with additional modifications" or "No":
   - Yes strictly : The model proposition matches what is required in the instruction. Here, the quality of the revision does not matter.
   - Yes with additional modifications : The model proposed additional modifications to the one required in the instruction. But some of the modification address the needs stated in the instruction.
   - No : The model proposition does not match the instruction.

2. Did model B address the instruction? (Answer "Yes strictly", "Yes with additional modifications" or "No")

3. Is model A revision acceptable? Answer "Yes" or "No". Answer "Yes" if the model made a good quality revision proposition that should replace the original paragraph in the scientific article.

4. Is model B revision acceptable? (Answer "Yes" or "No")

5. Which model proposed the best revision? (Answer preferably "A" or "B", you can answer "Both" if it is really a tie. Answer "None" if you answered "No" to question 3 and 4.)""
<Additional category questions depending on the revision intention labels of the instance>

""For all questions, you do not need to explain the reason.

Your response must be RFC8259 compliant JSON following this schema:
{"1": str, "2": str , "3": str , "4": str , "5": str """
< """"6": str """" and """, "7": str """" can be added depending on the number of labels of the instance.>
"""}}

Figure 14: User message for prompting LLM Choice with and without Gold

Prompt segment 6: Category Questions for LLM Likert with and without gold

Rewriting_light:"""The academic style and english has been improved.""""

Rewriting_medium:"""The readability and structure has been improved.""""

Rewriting_heavy:"""The paragraph has been rewritten in a more well organized and clear version, fitting the academic style.""""

Concision:"""The generated revision is a shorter version that kept all the important ideas.""""
Prompt segment 7: LLM Likert System message without Gold

You are an evaluator of academic writing on the task of text revision.
In this task, a revision model have been provided with the original paragraph written for a scientific article and a revision instruction on how to revise the paragraph.
You will be given the proposition from the revision model and several affirmations to determine the quality of this proposition.
You will answer each affirmation with a grade (int) from 1 to 5 as following: 1 = Strongly disagree , 2 = Disagree , 3 = Neutral ,4 = Agree , 5 = Strongly agree
In your answer please only provide the answers to the affirmations.

Prompt segment 8: LLM Likert User message without Gold

[BEGIN DATA]
***
[Original paragraph]: "{\"original_paragraph}"
***
[Revision instruction]: "{\"instruction}"
***
[Model proposed revision]: "{\"model_generated_revised_paragraph}"
***
[END DATA]

1. Relatedness: The generated revision correctly addressed the instruction.

2. Correctness: The generated revision is better than original version in my opinion."
<Additional category questions depending on the revision intention labels of the instance>

""For all questions, you do not need to explain the reason.

Your response must be RFC8259 compliant JSON following this schema:
{"1": str, "2": str , "3": str , "4": str , "5": str "}
< "" "6": str "" and """, "7": str "" can be added depending on the number of labels of the instance.>
"""}}

Figure 15: User message for prompting LLM Likert without Gold

Prompt segment 9: LLM Likert System message with Gold

You are an evaluator of academic writing on the task of text revision.
In this task, a revision model have been provided with the original paragraph written for a scientific article and a revision instruction on how to revise the paragraph.
You will be given the proposition from the revision model and several affirmations to determine the quality of this proposition.
You will answer each affirmation with a grade (int) from 1 to 5 as following: 1 = Strongly disagree , 2 = Disagree , 3 = Neutral ,4 = Agree , 5 = Strongly agree
To help you in this task you will also be given the gold paragraph which is the version revised by the author themselves.
In your answer please only provide the answers to the affirmations.
Prompt segment 10: LLM Likert User message with Gold

[BEGIN DATA]
***
[Original paragraph]: "{original_paragraph}"
***
[Revision instruction]: "{instruction}"
***
[Model proposed revision]: "{model_generated_revised_paragraph}"
***
[Gold revised paragraph]: "{gold}"
***
[END DATA]

1. Gold similarity: The generated revision is similar to gold revision.

2. Relatedness: The generated revision correctly addressed the instruction.

3. Correctness: The generated revision is better than original version in my opinion."
<Additional category questions depending on the revision intention labels of the instance>

"""For all questions, you do not need to explain the reason.

Your response must be RFC8259 compliant JSON following this schema:
({"1": str, "2": str , "3": str , "4": str , "5": str "}
< """" ""6": str """" and """, "7": str """" can be added depending on the number of labels of the instance.>
"""}}

Figure 16: User message for prompting LLM Likert with Gold
F  Bias of LLM Models on their own Revisions

As we used several LLM for revision that we reused as judges, we check in Table 6 if they are biased towards their own proposition. We don’t observe such bias, and even notice that results tend to be consistent across judge models. However, as they all tend to favour Mistral 7B we also computed the average edit distance between the original and generated revised texts for all revision models. As Mistral has the highest average, this could indicate an opposite bias as the one conveyed in similarity metrics: LLM-as-a-judge approaches tend to favour propositions with more important revisions.

<table>
  <tr>
    <th>Judge./Revision model→</th>
    <th>CoEdIT</th>
    <th>Mistral 7B</th>
    <th>Llama 3 8B</th>
    <th>Llama 3 70B</th>
    <th>GPT-4o mini</th>
    <th>GPT 4o</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>3.29</td>
    <td>42.83</td>
    <td>46.12</td>
    <td>53.68</td>
    <td>52.13</td>
    <td>58.33</td>
  </tr>
  <tr>
    <td>Mistral 7B Choice</td>
    <td>21.58</td>
    <td><b>60.92</b></td>
    <td><b>56.20</b></td>
    <td>55.17</td>
    <td>50.65</td>
    <td>52.65</td>
  </tr>
  <tr>
    <td>Llama 3 8B Choice</td>
    <td>3.94</td>
    <td><b>64.66</b></td>
    <td>58.98</td>
    <td><b>62.02</b></td>
    <td>58.08</td>
    <td>52.07</td>
  </tr>
  <tr>
    <td>Llama 3 70B Choice</td>
    <td>1.49</td>
    <td><b>73.00</b></td>
    <td><b>61.76</b></td>
    <td>59.23</td>
    <td>46.06</td>
    <td>48.13</td>
  </tr>
  <tr>
    <td>GPT-4o mini Choice</td>
    <td>1.81</td>
    <td><b>66.86</b></td>
    <td>58.53</td>
    <td>57.30</td>
    <td>47.61</td>
    <td>48.90</td>
  </tr>
  <tr>
    <td>GPT-4o Choice</td>
    <td>1.94</td>
    <td>59.30</td>
    <td>58.91</td>
    <td><b>62.98</b></td>
    <td>50.19</td>
    <td>52.52</td>
  </tr>
  <tr>
    <td>Mistral 7B Gold Choice</td>
    <td>23.64</td>
    <td>54.65</td>
    <td><b>56.33</b></td>
    <td><b>56.52</b></td>
    <td>51.16</td>
    <td>54.07</td>
  </tr>
  <tr>
    <td>Llama 3 8B Gold Choice</td>
    <td>14.67</td>
    <td>51.62</td>
    <td>59.24</td>
    <td><b>60.06</b></td>
    <td>53.68</td>
    <td>50.74</td>
  </tr>
  <tr>
    <td>Llama 3 70B Gold Choice</td>
    <td>1.49</td>
    <td><b>64.47</b></td>
    <td>58.85</td>
    <td>61.95</td>
    <td>51.10</td>
    <td>48.19</td>
  </tr>
  <tr>
    <td>GPT-4o mini Gold Choice</td>
    <td>2.00</td>
    <td><b>62.73</b></td>
    <td>58.08</td>
    <td><b>59.50</b></td>
    <td>51.61</td>
    <td>51.16</td>
  </tr>
  <tr>
    <td>GPT-4o Gold Choice</td>
    <td>2.33</td>
    <td>50.19</td>
    <td>52.91</td>
    <td><b>62.98</b></td>
    <td>54.07</td>
    <td>52.91</td>
  </tr>
  <tr>
    <td>Mistral 7B Likert</td>
    <td>5.04</td>
    <td><b>25.65</b></td>
    <td>24.55</td>
    <td>23.84</td>
    <td><b>25.97</b></td>
    <td>23.25</td>
  </tr>
  <tr>
    <td>Llama 3 8B Likert</td>
    <td>6.98</td>
    <td>38.11</td>
    <td><b>39.99</b></td>
    <td>36.24</td>
    <td><b>38.56</b></td>
    <td>36.05</td>
  </tr>
  <tr>
    <td>Llama 3 70B Likert</td>
    <td>2.26</td>
    <td><b>40.38</b></td>
    <td>37.40</td>
    <td>32.69</td>
    <td>29.20</td>
    <td>30.10</td>
  </tr>
  <tr>
    <td>GPT-4o mini Likert</td>
    <td>1.87</td>
    <td><b>36.95</b></td>
    <td>33.85</td>
    <td>31.98</td>
    <td>31.52</td>
    <td>29.39</td>
  </tr>
  <tr>
    <td>GPT-4o Likert</td>
    <td>1.36</td>
    <td>41.28</td>
    <td><b>47.87</b></td>
    <td><b>49.61</b></td>
    <td>47.67</td>
    <td>46.90</td>
  </tr>
  <tr>
    <td>Mistral 7B Likert Gold</td>
    <td>11.24</td>
    <td>18.41</td>
    <td>17.89</td>
    <td>19.64</td>
    <td>19.05</td>
    <td><b>19.77</b></td>
  </tr>
  <tr>
    <td>Llama 3 8B Likert Gold</td>
    <td>10.92</td>
    <td><b>46.25</b></td>
    <td>41.15</td>
    <td><b>42.96</b></td>
    <td>42.18</td>
    <td>39.60</td>
  </tr>
  <tr>
    <td>Llama 3 70B Likert Gold</td>
    <td>2.45</td>
    <td><b>46.19</b></td>
    <td><b>47.22</b></td>
    <td>44.51</td>
    <td>39.28</td>
    <td>38.76</td>
  </tr>
  <tr>
    <td>GPT-4o mini Likert Gold</td>
    <td>2.58</td>
    <td>44.06</td>
    <td>43.99</td>
    <td>45.41</td>
    <td><b>46.51</b></td>
    <td>44.19</td>
  </tr>
  <tr>
    <td>GPT-4o Likert Gold</td>
    <td>3.88</td>
    <td>42.25</td>
    <td><b>49.22</b></td>
    <td><b>49.22</b></td>
    <td>48.45</td>
    <td><b>51.16</b></td>
  </tr>
  <tr>
    <td>ParaPLUIE Mistral</td>
    <td>17.83</td>
    <td><b>72.48</b></td>
    <td><b>56.20</b></td>
    <td>52.91</td>
    <td>47.67</td>
    <td>50.39</td>
  </tr>
  <tr>
    <td>ParaPLUIE Llama3 8B</td>
    <td>20.35</td>
    <td><b>70.74</b></td>
    <td>52.33</td>
    <td>54.84</td>
    <td>52.33</td>
    <td>46.90</td>
  </tr>
  <tr>
    <td>Edit distance (Original-Generated)</td>
    <td>190.82</td>
    <td><b>342.69</b></td>
    <td>270.47</td>
    <td>234.95</td>
    <td>175.02</td>
    <td>197.36</td>
  </tr>
</table>

Table 6: Distribution of extended strict preference of each LLM judge by revision model
G  Distribution of Extended Preference of each LLM Judge

See Table 7.

<table>
  <tr>
    <th>Judge./Choice→</th>
    <th>Tie</th>
    <th>A</th>
    <th>B</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>14.53</td>
    <td>44.25</td>
    <td>41.21</td>
  </tr>
  <tr>
    <td>Mistral 7B Choice</td>
    <td>0.95</td>
    <td>82.43</td>
    <td>16.63</td>
  </tr>
  <tr>
    <td>Llama 3 8B Choice</td>
    <td>0.08</td>
    <td>53.92</td>
    <td>46.00</td>
  </tr>
  <tr>
    <td>Llama 3 70B Choice</td>
    <td>03.45</td>
    <td>52.97</td>
    <td>43.58</td>
  </tr>
  <tr>
    <td>GPT-4o mini Choice</td>
    <td>06.33</td>
    <td>39.28</td>
    <td>54.39</td>
  </tr>
  <tr>
    <td>GPT-4o Choice</td>
    <td>04.72</td>
    <td>53.62</td>
    <td>41.67</td>
  </tr>
  <tr>
    <td>Mistral 7B Gold Choice</td>
    <td>01.21</td>
    <td>77.76</td>
    <td>21.04</td>
  </tr>
  <tr>
    <td>Llama 3 8B Gold Choice</td>
    <td>03.33</td>
    <td>23.79</td>
    <td>72.85</td>
  </tr>
  <tr>
    <td>Llama 3 70B Gold Choice</td>
    <td>04.65</td>
    <td>51.14</td>
    <td>44.21</td>
  </tr>
  <tr>
    <td>GPT-4o mini Gold Choice</td>
    <td>4.98</td>
    <td>39.27</td>
    <td>55.75</td>
  </tr>
  <tr>
    <td>GPT-4o Gold Choice</td>
    <td>08.21</td>
    <td>49.94</td>
    <td>41.86</td>
  </tr>
  <tr>
    <td>Mistral 7B Likert</td>
    <td>57.24</td>
    <td>22.37</td>
    <td>20.39</td>
  </tr>
  <tr>
    <td>Llama 3 8B Likert</td>
    <td>34.69</td>
    <td>34.28</td>
    <td>31.03</td>
  </tr>
  <tr>
    <td>Llama 3 70B Likert</td>
    <td>42.66</td>
    <td>28.71</td>
    <td>28.64</td>
  </tr>
  <tr>
    <td>GPT-4o mini Likert</td>
    <td>44.81</td>
    <td>27.99</td>
    <td>27.2</td>
  </tr>
  <tr>
    <td>GPT-4o Likert</td>
    <td>21.77</td>
    <td>39.02</td>
    <td>39.21</td>
  </tr>
  <tr>
    <td>Mistral 7B Likert Gold</td>
    <td>64.66</td>
    <td>17.29</td>
    <td>18.05</td>
  </tr>
  <tr>
    <td>Llama 3 8B Likert Gold</td>
    <td>25.65</td>
    <td>38.20</td>
    <td>36.16</td>
  </tr>
  <tr>
    <td>Llama 3 70B Likert Gold</td>
    <td>27.20</td>
    <td>35.44</td>
    <td>37.36</td>
  </tr>
  <tr>
    <td>GPT-4o mini Likert Gold</td>
    <td>24.42</td>
    <td>37.96</td>
    <td>37.62</td>
  </tr>
  <tr>
    <td>GPT-4o Likert Gold</td>
    <td>18.60</td>
    <td>41.54</td>
    <td>39.86</td>
  </tr>
</table>

Table 7: Distribution of extended preference of each LLM judge

H  Distribution on Relatedness and Correctness for LLM-as-a-Judge Approaches

See Table 8

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Relatedness</th>
    <th rowspan="2">Correctness Acc.</th>
  </tr>
  <tr>
    <th>Strict Acc.</th>
    <th>Soft Acc.</th>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td><b>67.03</b></td>
    <td>84.69</td>
    <td>76.78</td>
  </tr>
  <tr>
    <td>gpt-4o-gold</td>
    <td>62.99</td>
    <td>84.66</td>
    <td><b>76.94</b></td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>62.26</td>
    <td><b>85.30</b></td>
    <td>76.04</td>
  </tr>
  <tr>
    <td>gpt-4o-mini-gold</td>
    <td>58.95</td>
    <td><b>85.14</b></td>
    <td>75.91</td>
  </tr>
  <tr>
    <td>llama3-70b</td>
    <td><b>66.66</b></td>
    <td>82.11</td>
    <td>75.96</td>
  </tr>
  <tr>
    <td>llama3-70b-gold</td>
    <td>66.48</td>
    <td>82.37</td>
    <td>76.10</td>
  </tr>
  <tr>
    <td>llama3-8b</td>
    <td>45.41</td>
    <td>80.16</td>
    <td>72.30</td>
  </tr>
  <tr>
    <td>llama3-8b-gold</td>
    <td>40.55</td>
    <td>77.70</td>
    <td>61.93</td>
  </tr>
  <tr>
    <td>mistral-gold</td>
    <td>58.78</td>
    <td>75.19</td>
    <td>62.37</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>58.98</td>
    <td>75.27</td>
    <td>62.58</td>
  </tr>
</table>

Table 8: Accuracy of LLM-Choice on the preliminary Relatedness and Correctness questions. For relatedness, in soft accuracy, we merge "Yes stricly" and "Yes with additional modifications" categories.

I  Alignment of all Metrics

See Figures 17 and 18
Figure 17: Alignment of metrics to human annotations by metric by type of metric and type of revision.
Figure 18: Alignment of metrics to human annotations by metric by types of difficulty
J   Impact of using Gold References on the Alignment of LLM-as-a-Judge Approaches

See Figures 19 and 20.

![Alignment of LLM-as-a-judge approaches with human annotations by revision category](page_184_153_1080_273.png)

Figure 19: Alignment of LLM-as-a-judge approaches with human annotations by revision category

![Alignment of LLM-as-a-judge approaches with human annotations, by difficulty. The triangles in the first column represent the agreement on the total dataset.](page_184_464_1080_273.png)

Figure 20: Alignment of LLM-as-a-judge approaches with human annotations, by difficulty. The triangles in the first column represent the agreement on the total dataset.

Reviewer paper summaries: This paper conducts a comparative evaluation and analysis of metrics for scientific text revision. The investigated metrics include both reference-based traditional metrics, metrics imported to the task from other domains, and LLM-as-a-judge approaches.
For their evaluation, they create a new meta-evaluation dataset based on the outputs of six revision systems on the pararev dataset. Revisions are evaluated for correctness, instruction-adherence and for personal preference of the annotators.
Summary of strengths: 1. The paper provides a human annotated dataset of scientific article simplification for multiple simplification models, which will be very useful to future work on developing metrics for this task.

2. The set of investigated metrics is very extensive, including a large set of variations on the LLM as a judge paradigm. This will support future researchers in selecting appropriate evaluation metrics.

3. The analysis regarding the preference of similarity-based metrics for high-overlap instances is interesting and provides important guidance for which metric to use.

4. The authors highlight that different metrics work well for different quality aspects, particularly showing that LLM-as-a-judge models work well for instruction-following, but are outperformed by some competitors on correctness, showing the need for including multiple metrics in evaluation.
Summary of weaknesses: 1. There are only very few instances with repeat annotations in the dataset and the limited amount of repeat annotations have only two repeat annotations. While this is likely fine for relatedness and correctness, where agreement is acceptable, it is less ideal for preference, which, as the authors correctly note, is a highly subjective task. Without such repeat annotations, it will remain difficult to conduct any meaningful meta-evaluation for this category.

2. ~The dataset subset with repeat annotations that is used to measure agreement is distributed differently than the overall dataset. Specifically, Llama-70b is overrepresented, with the comparison between Llama-70b and coedit-xl being especially much more frequent (~~18% of pairs) than expected. Since there is a large performance gap between Llama-70b and coedit-xl, this mismatch carries a risk of overestimating agreement.~
Update after discussion: This turned out to be a misinterpretation of the dataset. The duplicate annotations are unfortunately not part of the submitted dataset. Instead, there are duplicate instances for some paragraph/model pairs.

3. While the variation in LLM-selection is extensive, there is no variation in LLM-prompts, which is not ideal for comparing LLM performance (see, e.g. https://aclanthology.org/2024.tacl-1.52.pdf).

4. The annotation and analysis mixes objective properties that should be independent of the comparison to another model, specifically correctness and instruction following, with a pairwise evaluation. This creates a risk of interaction effects (e.g. a strong model_A leading to a more strict assessment of model_B or vice-versa), which might confound results on this dataset.

5. While the individual sections of the paper are well written, I found the overarching argumentation unnecessarily difficult to follow.
This is in part because related results are highly segmented between sections.
For example, both Table 3 and Table 4 report average scores for simplification models according to different automatic metrics and both Figure 3 and 4 show intra-metric correlation. Neither of the two analyses includes the LLM judges, even though this would be straightforward for the Likert-setting and possible with some minor modifications (e.g. using ranking agreement between annotated pairs for the Figures). While some of the information can be recovered from the appendix, this is not ideal for following along with the argumentation of the paper and readability would profit from another editing pass.
Comments, suggestions and typos: 1. BLEU and METEOR are included in the initial inter-metric correlation analysis, but not in the human/metric alignment analysis. While this is understandable given the high correlation, it would be nice to either include them for completeness or make it obvious in the text why they were dropped.

2. Figure 3, 4 would be visually much easier to parse if you used the negative Levenshtein distance instead, so colors indicate the magnitude of each correlation.

3. Instead of easy/medium/hard as a categorization in 6.2 I feel it would be much clearer to instead clearly label that you evaluate the sub-aspects in 

4. It would be useful to include annotator IDs in the annotations. This might support future work, particularly with regard to working with the subjective quality aspects.

Line 204 Rewritting –> Rewriting
Overall assessment scores: 4
Metareview score: 4
Metareview: This paper investigates the limitations of similarity-based metrics (e.g., ROUGE, BERTScore) for scientific text revision and compares them with other reference-free and LLM-based approaches. The authors provide a new human-annotated dataset of revision outputs and demonstrate that different metrics excel in different dimensions (e.g., correctness vs. instruction-following). They conclude that combining a domain-specific metric with an LLM-based judge yields the most reliable assessment.
Summary of reasons to publish: 1. The paper contributes an annotated dataset focused on scientific text revision, enabling robust comparisons of evaluation metrics and facilitating further work in this emerging area.

2. The authors evaluate both traditional (e.g., n-gram) and modern (e.g., LLM-as-a-judge) approaches, highlighting each one’s strengths and weaknesses. This breadth of coverage helps clarify which metrics are best suited for specific revision aspects.