From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications (2024)

Abstract: Evaluating large language models (LLMs) is fundamental, particularly in the context of practical applications. Conventional evaluation methods, typically designed primarily for LLM development, yield numerical scores that ignore the user experience. Therefore, our study shifts the focus from model-centered to human-centered evaluation in the context of AIpowered writing assistance applications. Our proposed metric, termed “Revision Distance,” utilizes LLMs to suggest revision edits that mimic the human writing process. It is determined by counting the revision edits generated by LLMs. Benefiting from the generated revision edit details, our metric can provide a self-explained text evaluation result in a human-understandable manner beyond the context-independent score. Our results show that for the easy-writing task, “Revision Distance” is consistent with established metrics (ROUGE, Bert-score, and GPT-score), but offers more insightful, detailed feedback and better distinguishes between texts. Moreover, in the context of challenging academic writing tasks, our metric still delivers reliable evaluations where other metrics tend to struggle. Furthermore, our metric also holds significant potential for scenarios lacking reference texts.

heading: 1 Introduction
text: You can’t manage what you can’t measure well.—Cruz-Cázares et al. 2013
With the continuous development of large language models (LLMs) such as ChatGPT1, GPT4(OpenAI), and Llama(Touvron et al., 2023), a plethora of application research and development work based on LLMs has emerged.
*This work was done when Yongqiang Ma interned at Alibaba.
†Corresponding Authors. 1https://openai.com/blog/chatgpt
During the model training phase, the main focus is optimizing the model’s loss in an isolated environment. However, LLM-based applications should be human-centered, prioritizing user experience and utility. This raises a key question: How do we evaluate LLM-based applications from a human-centric perspective?
Imagining the scenario where developers employ automatic evaluation metrics (Lin, 2004; Papineni et al., 2002; Zhang et al., 2020; Zhao et al., 2019) like Rouge(Lin, 2004) to evaluate LLM-generated text for writing assistance debugging. Rouge only provides a high-level evaluation score to measure textual surface similarity. Since these metrics disregard end-users, the evaluation result is inadequate and misaligns with user needs and preferences. To address this gap, we explore alternative humancentered evaluation metrics, putting the user at the forefront of our evaluation.
2127
This paper focuses on the prevalent application scenario for LLMs, specifically, the LLM-powered writing assistant in scenarios from email and letter writing to academic writing.2 During the AI-human collaborative writing process, AI-generated text often requires extended revisions. Additionally, recent studies suggest that LLMs can produce humanlike behavior, such as providing human preferences feedback (Bai et al., 2022; Lee et al., 2023), conducting text quality evaluation (Chiang and Lee, 2023; Fu et al., 2023). Therefore, we assume that the LLM can be a proxy user for generating revision edits that align with actual human editing behaviors.
Drawing from these insights, our proposed metric, RevisionDistance, incorporates the iterative process of user-driven text revision. It quantifies the number of edits a user must take to an LLM-generated text to achieve a predefined quality threshold. Compared to typical metrics where higher scores often indicate better quality, our proposed metric operates on the principle that a smaller distance signifies superior text quality. This design is premised on the notion that high-quality texts require fewer revisions to align with a predefined standard of excellence.
In the reference-based evaluation setting, we compared our metric with Rouge, BERT-Score, and GPT-Score across two writing tasks: the easywriting task and the challenge-writing task. For each task, we sample texts from two models to form a comparison group. Then, we apply text evaluation metrics to assess the text quality. (1) For the easy-writing task, our metric consistently aligns with baseline metrics, supporting the intuition that a stronger model should produce texts with superior evaluation scores. (2) For more challenging tasks, our metrics can still provide stable and reliable evaluation results even if most of the baseline indicators encounter different issues.
In reference-free scenarios, the “Revision Distance” metric aligns closely with human judgment in approximately 76% of cases in the dataset from the ultrafeedback dataset (Bartolome et al., 2023). Furthermore, by categorizing the types of edits made, our metric provides a more fine-grained analysis than those metrics that only yield scores. Additionally, there is no significant difference in format
2We use the “Related Work” section Generation (RWG) (Liu et al., 2023; Chen et al., 2021) as the testbed for academic writing, which requires heavy knowledge reasoning work and complex concept understanding ability.
between texts written by humans and AI-generated texts. Our metric is naturally extendable to evaluating human-authored texts in scenarios involving human writing. For example, our metric can be used in educational settings.
The contributions are listed as follows: 1) We highlight the significance of the end-user’s perspective in the text evaluation in the context of LLM-power writing assistant. 2) Aligning with real-world human editing behaviors, we propose a human-centered text evaluation metric, which provides a self-explain and fine-grained insight for developers and end-users. 3) Based on broad and various test tasks, we conduct an experiment to demonstrate the utility of our proposed humancentered metrics.

heading: 2 Related Work
text: The text evaluation methods can be categorized into human evaluation and automated approaches. Human evaluation is widely recognized as the most natural way to evaluate the quality of a given text, which often involves human annotators and qualitative analyses (Clark et al., 2021; Belz et al., 2023). This method is often expensive and timeconsuming work and requires extensive domain expertise for domain-specific scenarios. On the other hand, current automated evaluation methods tend to generate a comprehensive score that is facilitated in comparing new models with established stateof-the-art approaches. These include metrics such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019), BARTScore (Yuan et al., 2021), and DiscoScore (Zhao et al., 2023a) typically compute a similarity (or dissimilarity) score between a model-generated text and a reference text.
Large language models have been adeptly utilized for roles such as aiding in data annotation (Li et al., 2023) and delivering feedback that mirrors human preferences (Bai et al., 2022; Lee et al., 2023; Pang et al., 2023). For the evaluation stage, Chiang and Lee (2023) found that the LLM evaluation is consistent with the human evaluation results. The GPTScore (Fu et al., 2023) has been proposed to score the model-generated text. Similarly, Jain et al. (2023) also studied the efficacy of LLMs as multi-dimensional evaluators.
In conclusion, current metrics tend to yield a comprehensive score that detaches the task context
for model development and optimization. However, the ultimate application of LLMs is humancentered, prioritizing the user experience and utility. Consequently, a context-independent numerical score is insufficient in LLM application scenarios. Our proposed metric shifts the text evaluation to a human-centered perspective, which incorporates the iterative process of user-driven text revision.

heading: 3 Revision Distance
text: As depicted in Figure 2, we frame the context of AI-powered writing assistance, with LLMs serving dual functions: as the proxy of the user (LLMUser) and as the generator (LLMgen). The LLMgen is the pivotal component of the writing assistant application. Given the user input content, the LLMgen generates a Ydraft, such as emails, letters, articles, and “Related Work” sections. The DRevision quantifies the number of edits from LLMUser(Ydraft, Y ).
To avoid the generated revision focusing excessively on surface-level textual features, we have explicitly required LLMUser to focus on coherence and logical flow, as shown in Appendix D. Additionally, we have calibrated the revision generation process by setting the temperature parameter to 0, aiming to reduce the likelihood of generating unstable outputs. We also separated complex revisions that contained multiple revisions, such as “simplify and reorganize,” into individual actions to ensure consistency in the granularity of revisions. By setting a consistent scope and clear objectives for revisions, our metric promotes a uniform revision process that is less prone to variance, thus enhancing reproducibility and reliability.
For the reference-based evaluation setting, we utilize the human-written text or ChatGPT output
as the ground truth. The LLMUser is designed to produce structured revision edits, improving the consistency of the YDraft to the ground-truth text Y . In scenarios where no ground truth text is available, we require the LLMUser to refine the given text towards an ideal form, as envisioned by the LLMUser itself.3 These revision edits are produced to improve Ydraft to closer align with the ideal version, which mimics the revision process of human writers.

heading: 4 Results and Discussion

heading: 4.1 Evaluation for Reference-based Setting

heading: 4.1.1 Task and Dataset
text: To validate the utility of our proposed metric, we have constructed two distinct datasets to address both the easy-writing task and the challengewriting task. The challenge-writing task refers to the scenario that requires heavy knowledge reasoning and complex concept understanding. For the easy-writing task, we use emails, letters, and article generation as a testbed. For the challenge-writing task, we employ academic writing as the testbed. The test dataset details in this evaluation setting are described in Appendix A.
3This ideal version is not explicitly generated but rather serves as an implicit standard within the revision edits generation prompt.

heading: 4.1.2 Text Generation Models
text: To assess the discriminative capacity of our revision distance metric, we designed strong and weak writing applications. The terms “strong” and “weak” refer to the generation ability of utilized LLM, as detailed in Table 1. (1) For the easy-writing task, we employ two Mistral-series models (Jiang et al., 2023); (2) For the challenge-writing task, we employ GPT-4 and its variant.4
We have validated the text generated by both strong and weak models. For most of the text pairs, the strong model consistently produced higherquality text. Aligning with human evaluations, a good metric should yield higher evaluation scores for texts generated by stronger models compared to those from weaker models. Here, we analyze the evaluation capability of different metrics by assessing texts generated by strong and weak models.

heading: 4.1.3 Result Analysis
text: As shown in Table 2, our metric shows utility for easy and challenging writing tasks. Different from other metrics, smaller DRevision indicate better text quality. To assess the metric’s ability to differentiate between models, we calculate the relative change rate from the “Weak” model to the “Strong” model. Notably, existing metrics have reached saturation for the easy-writing tasks, exhibiting a limited relative change rate regarding the performance of distinct models. For example, the GPT score tends to assign higher scores to texts, primarily clustering between 85 to 95, resulting in smaller relative variations.
4The models employed in both tasks are detailed in Appendix B and Appendix C, respectively.
Conversely, our metric demonstrates better efficacy in discerning the nuanced capabilities of diverse models. Our metric operates within a range of 0 to n, which, by its nature, may exhibit large relative changes for small numeric variations. It’s observed that DRevision yields a larger change rate, highlighting the enhanced discriminative capacity of our metric.
Additionally, for the complex academic writing task, we conducted a human evaluation, as shown in Appendix G. Based on the evaluation results, we categorized texts as “Chosen” or “Rejected.” As shown in Table 3, DRevision metric aligns with human preferences, indicating superior text quality with fewer revisions for “Chosen Texts.” In contrast, the Rouge metric often misaligns with human judgments, erroneously assigning higher scores to “Rejected Texts.”
Our metric shows a negative Spearman correlation (R = -0.38) with human evaluation scores, unlike the near-zero or marginally positive coefficients of Rouge, BERTScore, and GPTScore. This indicates our metric’s unique approach, where
lower scores mean better text quality.

heading: 4.2 Evaluation for Reference-free Setting
text: To demonstrate the performance of our evaluation method in scenarios, where ground truth is unavailable, we extracted 41 cases related to writing tasks from the UltraFeedback dataset(Bartolome et al., 2023). Each case contains a chosen response and a rejected response.
When applied to the selected cases, our “Revision Distance” metric aligns with human judgments in 76% of instances, indicating that chosen responses typically necessitated fewer revisions.

heading: 4.3 Qualitative Analysis
text: Based on the analysis of revision edit details from the Challenge-writing Task (writing the “Related Work” section), we classify the revision actions into three categories: (1) Reference Order Revision, (2) Reference Comparison Revision, and (3) Reference Description Revision. The description of three categories is shown in Appendix F.
For complex writing tasks, the challenge often lies in knowledge reasoning of concepts. CoT prompting can dramatically improve the multi-step reasoning abilities of LLMs (Wang et al., 2023). As shown in Table 4, for texts generated by Vanilla GPT-4, the average revisions required are 0.80 for Order, 0.84 for Comparison, and 2.29 for Description. For the texts generated by CoT GPT-4, the average revisions required are 0.67 for Order, 0.71 for Comparison, and 2.36 for Description.
Based on the fine-grained analysis of revision edits, we find that CoT-based GPT-4 can provide text with fewer revisions related to Order and Comparison issues in “Related work” writing tasks. A slight decline in the reference description dimension exists compared with Vanilla GPT-4. In conclusion, the fine-grained analysis revision edits can provide insightful feedback for future model improvement.

heading: 5 Conclusion
text: With the rapid advancement of LLM-based applications, the pivotal question arises: “how can we evaluate LLM-based applications from a humancentered perspective?” Existing evaluation metrics, typically used for model development, merely yield a context-independent numerical score, lacking user relevance. Our research shifts text evaluation from a predominantly model-centered perspective to a human-centered one.
Using the LLM-powered writing assistant as a test scenario, we take a comprehensive experiment on diverse writing tasks to validate the effectiveness and reliability of our “Revision Distance” metric. This metric converts text evaluation into contextualized text revisions, highlighting textual discrepancies and offering users a detailed, transparent rationale for the scores. Our findings demonstrate the metric’s applicability and dependability in both reference-based and reference-free contexts.
Limitations
This paper introduces a metric that leverages GPT4, specifically applied to evaluating LLM-powered writing assistants. In our initial attempts, we utilized the GPT-3.5 API to generate revisions. However, GPT-3.5 struggled to follow the instructions for generating revisions that improve logic flow and coherence. However, the computational and financial costs of using GPT-4 are considerable. Exploring the use of a smaller, specialized model to generate initial edits could reduce costs and improve efficiency.
LLMs have a wide array of applications, and for this study, we have chosen the “Related Work” section generation task as a testbed for challenging writing scenarios. As a knowledge-intensive cognitive task, writing the “Related Work” section requires writers to integrate multi-source knowledge into the manuscript. Therefore, writing a comprehensive “Related Work” section is a laborintensive and time-consuming endeavor. Future studies could explore the application of our metric in longer text generation tasks, such as code generation and scientific reports, to validate its effectiveness and applicability across different domains.
In this study, each generated revision item is assigned equal weight. Future research should focus on developing a dynamic revision edit weighting method to evaluate textual differences more finely.
Ethics Statement
In this paper, we propose a new automatic evaluation metric, RevisionDistance, to evaluate the LLM-generated text in an AI-power writing assistant setting. The positive impact of Revision Distance is that it can provide a more nuanced and self-explain representation of the quality of LLM-generated text. Notably, our metric is humancentered and transparent, which can help demystify the evaluation process for LLM-generated text, making it more accessible to a wider user, including those who are not experts in AI. The negative impact is that over-reliance on RevisionDistance might lead to the overlooking of qualitative aspects of text generation that are harder to quantify, such as creativity. Additionally, if the reference texts within RevisionDistance are biased or of low quality, this could amplify the biases in the LLMsgenerated text.

heading: Acknowledgements
text: This work was supported by the National Science and Technology Major Project (2022ZD0116204), the National Natural Science Foundation of China (72174157), and Alibaba Group through Alibaba Innovation Research Program.

heading: A Dataset for Reference-based Setting
text: • For the easy-writing task, we use the task of emails, letters, and articles generation as testbeds. Specifically, we extracted 147 relevant instances of emails, letters, and articles written from Wildchat (Zhao et al., 2023b), a real-world user-ChatGPT interactions corpus, as the easy-writing dataset.
• For the challenge-writing task, we employ the “Related Work” generation task (Liu et al., 2023; Chen et al., 2021) as the testbed. The “Related Work” sections in academic manuscripts position the authors’ contributions within the existing academic context. They highlight the novelty and advantages of the presented research compared to existing works. Specifically, we randomly selected 90
“Related Work" paragraphs from scholarly articles within the ACL dataset (Rohatgi et al., 2023) as the challenge-writing dataset.

heading: B Text Generation Models for Easy Writing Task
text: We use the APIs of Mistral-7B5 and Mistral-8x7B6, hosted on Huggingface, to generate responses for the prompts within our constructed easy-writing task dataset. The parameters for the inference process are shown in Table 5.

heading: C Text Generation Models for Challenge Writing Task
text: In academic manuscripts, the “Related Work” sections position the authors’ contributions within the existing academic context. They highlight the novelty and advantages of the presented research in comparison to existing works.
In the “Related Work” generation task, the model utilizes a set of reference papers, denoted as Ref , along with a description of the user’s current research denoted as D, to generate a “Related Work” draft, denoted as YDraft, for the user. In this work, the input data is sourced from the ACL papers (Rohatgi et al., 2023). We select the related work section paragraph as the test data based on the section title. Notably, the abstracts of both the reference papers and the user’s target paper are utilized to encapsulate the core content of the respective research, thereby assisting in the generation process. For both Vanilla GPT-4 and CoT-based GPT-4, the temperature is set as 1.0 in the generation process.
YDraft = LLMgen(Ref,D) (1)
5https://huggingface.co/mistralai/Mixtral-8x7B-Instructv0.1
6https://huggingface.co/mistralai/Mixtral-8x7B-Instructv0.1
Vanilla GPT-4 We concatenate the task instruction and metadata of reference and source papers to directly prompt the LLM to get the final “Related Work”.
CoT-based GPT-4 We initially prompt the LLM to generate learned relevant knowledge in the training stage(Sun et al., 2023) and then create three segments for different perspectives. Finally, these segments, along with the recalled knowledge and the metadata of the input papers, are integrated to generate the comprehensive “Related Work” paragraph. Based on the intermediate step, the LLM can better capture interrelationships among scientific publications and concepts.

heading: D Prompt for Revision Generation
text: In the prompt, we require the GPT-4 to output in JSON. Because the output format setting is the same across multiple prompts, it is explained separately here, as shown in Figure 6.

heading: E Example for Revision Action Item
text: E.1 Revision Regarding the Text Content
###Human-written Text: To deal with the STS task, previous studies have resorted to various features (e.g. word overlap, synonym/antonym), linguistic resources (e.g. WordNet and pre-trained word embeddings) and a wide assortment of learning algorithms (e.g. Support Vector Regression (SVR), regression functions and NNs). Among these works, several techniques extract multiple features of sentences and apply regression functions to estimate these similarity scores (Lai & Hockenmaier, 2014; Zhao et al., 2014; Bjerva et al., 2014; Severyn et al., 2013). Lai & Hockenmaier (2014) analyzed distinctive word relations (e.g. synonyms, antonyms, and hyperonyms) with features based on counts of co-occurences with other words and similarities between captions of images. Zhao et al. (2014) predicted the sentence similarity from syntactic relationship, distinctive content similitudes, length and string features.
Bjerva et al. (2014) also utilized a regression algorithm to foresee the STS from different features (WordNet, word overlap, and so forth). Finally, Severyn et al. (2013) combined relational syntactic structures with SVR.
###AI-generated Text: Semantic Textual Similarity (STS), the task of measuring the degree of semantic equivalence between two pieces of text, has been extensively explored in the literature. The development of compositional distributional semantic models (CDSMs) forms an integral part of STS investigations, which employ meaning-rich computational systems to better understand and quantify semantic relatedness. The work done by Marelli et al. went a step further, focusing
{ "action_name": "simplify", "revision_description": "Simplified the text by removing details regarding
on producing a large English benchmark, SICK, for the deep evaluation of CDSMs, significantly contributing to the body of tools available for STS analysis. . . ..
E.2 Revision Regarding the Text Structure
In this case study, we compare a human-written text and an AI-generated text that superficially appears very similar but exhibits subtle differences in the structure and order of information. The humanwritten text follows a logical sequence, while the AI-generated text swaps the order of key details.
###Human-written Text: First, Authors A proposed a BERT-based method. Second, Authors B proposed a GPT-based method.
###AI-generated Text: First, Authors B proposed a GPT-based method. Second, Authors A proposed a BERT-based method.
This case is used to demonstrate the limitations of current automated metrics in capturing such structural differences and demonstrates the superiority of our novel metric in evaluating text quality.
As shown in Table 6, current commonly used metrics tend to assign near-perfect scores to AIgenerated texts, implying a high degree of equivalence with their human-written counterparts. However, this fails to capture the underlying structural differences between the texts.
As depicted in Figure 8, our metric can better capture the text’s structural differences. Notably, our metric can offer users a coherent and transparent explanation of the scores assigned, benefiting
from the detail of revision actions.
{ "action_name": "Reorder", "revision_description": "Reordered the
sequence of references to match the human -written text",
... }

heading: F Revision Categories for the LLM-Generated “Related Work”
text: to modifying the description of a particular reference paper, either by elaborating it or by making it more concise.

heading: G Human Evaluation for Metrics in Challenge-writing Task
text: We selected 20 paragraphs from both methods for expert analysis. Five AI field specialists assessed the LLM-generated content, focusing on content quality, structural coherence, and argumentative strength. Evaluators assign scores across three dimensions on a scale from 1 to 5. The sum of the scores from the three dimensions is taken as the final score for the text.
The rate of alignment with humans is 50% for our metric. The human alignment rate for Rouge 1, Rouge-2, Rouge-L, and BertScore is lower than 50%, which is 40%. The GPT-Score’s human alignment rate is 60%. However, the GPT-Score’s evaluation scores for human-chosen and rejected samples are 87.4 and 87.7, respectively. This demonstrates that the GPT-Score overall lacks discriminative power. Additionally, compared to our metric, the GPT-Score also falls short in terms of interpretability.
While their potential optimization revision edits might differ for two texts, they can have the same number of optimization steps, leading to instances where their Revision Distance is equal (accounting for 20% of cases). Therefore, our metric achieved only a 50% human alignment rate. We will refine our metric by implementing a weighting method to address this issue.

heading: H Stability Analysis
text: The stability of evaluation results when employing LLMs as proxies of evaluators is a concern echoed by other metrics such as the GPT-Score. Based on the “TextAttack” framework (Morris et al., 2020), we utilized word embeddings to make slight alterations to the words in the text (originally with a Revision Distance of 4). This yielded four perturbed samples, with resultant revision edits numbering 4, 4, 5, and 4, illustrating our metric’s stability.

heading: Future Work
text: Future studies could explore the application of our metric in longer text generation tasks, such as code generation and scientific reports, to validate its effectiveness and applicability across different domains. In this study, each generated revision item is assigned equal weight. Future research should focus on developing a dynamic revision edit weighting method to evaluate textual differences more finely