RevisEval: Improving LLM-as-a-Judge via Response-Adapted References

## Abstract

Abstract With significant efforts in recent studies, LLM-as-a-Judge has become a cost-effective alternative to human evaluation for assessing the text generation quality in a wide range of tasks.
However, there still remains a reliability gap between LLM-as-a-Judge and human evaluation. One important reason is the lack of guided oracles in the evaluation process.
Motivated by the role of reference pervasively used in classic text evaluation, we introduce RevisEval , a novel text generation evaluation paradigm via the response-adapted references. RevisEval is driven by the key observation that an ideal reference should maintain the necessary relevance to the response to be evaluated.
Specifically, RevisEval leverages the text revision capabilities of large language models (LLMs) to adaptively revise the response, then treat the revised text as the reference ( response-adapted reference ) for the subsequent evaluation.
Extensive experiments demonstrate that RevisEval outperforms traditional reference-free and reference-based evaluation paradigms that use LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks.
More importantly, our response-adapted references can further boost the classical text metrics, e . g ., BLEU and BERTScore, compared to traditional references and even rival the LLM-as-a-Judge.
A detailed analysis is also conducted to confirm RevisEval ’s effectiveness in bias reduction, the impact of inference cost, and reference relevance.

## 1 Introduction

As the large language model (LLM) already exhibits strong alignment with humans (Gilardi et al., ; OpenAI et al., ), LLM-as-a-Judge (Chang et al., ; Li et al., ; Gao et al., ), aka. LLM-evaluator, has emerged as a viable alternative to human evaluation in assessing text generation quality.
Given the task instruction and the corresponding model-generated responses, LLMs are prompted to predict preferences or scores for these responses.
Despite considerable efforts have been made, such as chain-of-thought (Zheng et al., ), specialized rubrics (Liu et al., ), and extensive evaluation-specific training datasets (Li et al., ; Wang et al., ; ),
human evaluation remains the gold standard in text quality assessment (Zeng et al., ) and LLM-as-a-Judge struggles with particular biases (Huang et al., ) and being vulnerable to the misleading context (Dubois et al., ; Chen et al., ).
One important reason is the lack of an oracle to direct the evaluation process.
Fortunately, classical text evaluation metrics, like BLEU (Papineni et al., ) and ROUGE (Lin, ), offer valuable prompts in mitigating such a gap: given an appropriate reference, i.e., the ground-truth answer to the task, calculating the similarity between the references and the model-generated responses can achieve a satisfactory correlation with human evaluations.
Furthermore, several studies highlight the reference could prevent being overly sensitive to semantic deficiency  (Sheng et al., ) and overcoming bias (Deutsch et al., ) in certain cases.

Figure: Figure 1: Performance comparison of reference-free and reference-based evaluation paradigms across different similarity groups in MT-Bench, using GPT-4-as-a-Judge. In the reference-based evaluation, the GPT-4 direct response is used as the reference, and the evaluated response with a higher BERTScore with the reference is regarded as the preferred one. As the similarity between the reference and the response increases, the human agreement accuracy of the reference-based evaluation significantly improves, while the reference-free evaluation maintains relatively consistent performance across all similarity levels.
Refer to caption: x1.png

However, straightforward leveraging references in LLM-as-a-Judge can also be challenging. In addition to the availability of high-quality references (Rei et al., ), previous works (Mehri & Eskenazi, ; Gómez-Rodríguez & Williams, ; Guan & Huang, ) find that pre-existing references introduce noise across various text evaluation tasks due to the one-to-many problem, where for a given task input, there exist many diverse yet valid responses.
In this case, particular pre-existing references could negatively penalize many appropriate but dissimilar responses in the evaluation process (Ji et al., ). Thus, we hypothesize that an effective reference should be closely relevant to the response to be evaluated.
We further verify this on MT-Bench (Zheng et al., ), an open-ended instruction-following dataset. As shown in Figure 1, we use GPT-4 direct responses to the instructions as the references and quantify relevance by the similarity between the references and the responses using BERTScore (Zhang et al., ). We find that higher relevance simulates greater utility from the reference, resulting in more effective evaluations than a reference-free evaluator.

Motivated by the above findings, we deem that an effective reference should maintain high quality while ensuring relevance to the response, which led us to consider that revising the response adaptively could be a good candidate for this reference (Guo et al., ).
Therefore, we propose a novel evaluation paradigm Revise-and-Evaluation, abbreviated as RevisEval.
Specifically, given the (instruction, response) pair, RevisEval first revise the response using the instruction and evaluation rubric, resulting in the response-adapted reference. RevisEval further leverages this generated response-adapted reference to guide final evaluation (e.g., scoring or pairwise comparison).
By revising the original response, we can ensure that the generated reference is both high-quality and closely relevant to the original content. The comparison between the original and revised responses offers valuable insights for evaluation. Orthogonal to previous work that only focuses on the discrimination abilities of LLMs, RevisEval stands out by fully utilizing the generative potential by revision.

We conduct comprehensive experiments to validate the effectiveness of our proposed RevisEval. Using both proprietary and open-source LLMs, RevisEval consistently achieves better evaluation performance compared to reference-free and reference-based evaluation paradigms in both NLG tasks and open-ended instruction-following tasks.
Moreover, we seek to verify the effectiveness of the response-adapted references in the classic metrics, e.g., BERT and BERTScore, showing that each metric exceeds itself by up to 3%-10% accuracy compared to using direct response as references.
We then combine LLM-as-a-reviser with multiple classic metrics and find that it outperforms LLM-as-a-Judge (reference-free setting) by over 1.5% on average when using weak LLMs and is comparable when using GPT-4.
Finally, we analyze how our paradigm achieves overall superiority. In reducing verbosity and positional bias, our approach offers clear advantages in adversarially designed LLMBar and swapping position testing. Merely increasing inference cost of reference-free evaluation still lags behind RevisEval, demonstrating our method’s efficiency does not rely on naively accumulating cost. Meanwhile, we validate the relationship between reference relevance and efficiency using response-adapted references.

## 2 Related Work

### 2.1 Evaluation of Large Language Models

Instruction-tuned LLMs (Ouyang et al., ; Dubey et al., ; Team et al., ) have revolutionized the field of natural language processing (NLP) due to their ability to handle a wide range of language-related tasks. Unlike traditional NLP tasks, such as Machine Translation and Summarization, which can be evaluated using N-gram-based metrics like BLEU (Papineni et al., ), ROUGE (Lin, ), and METEOR (Banerjee & Lavie, ) by comparing responses with reference texts, LLMs excel at open-ended language generation tasks (e.g., story generation and open-ended instruction-following generation), where no single reference response exists. Consequently, several studies embrace the potential of LLM-as-a-Judge and shift toward reference-free metrics, advocating the abandonment of conventional reference-based evaluation methods (Sheng et al., ; Chen et al., ). In this paper, we re-visit the significance of reference in LLM evaluation. Furthermore, to address the challenge of the absence of a single standard answer in certain evaluation tasks, we propose leveraging LLMs to generate response-adapted intermediate references, thereby improving the evaluation performance of both traditional metrics and LLM-as-a-Judge.

### 2.2 LLM-as-a-Judge

Recent progress in NLP has introduced model-based evaluation metrics like BERTScore (Zhang et al., ) and BARTScore (Yuan et al., ). However, these methods also depend on the availability of human-annotated references, which can be expensive, time-consuming, and labor-intensive (Zheng et al., ). With the emergence of large language models (LLMs), several studies (Zheng et al., ; Dubois et al., ) have harnessed their robust evaluation capabilities for assessing natural language generation (NLG), particularly by employing proprietary models like GPT-4 (OpenAI et al., ). To avoid information leakage caused by external API calls, some efforts advocate finetuning LLMs with evaluation data to obtain evaluator models (Vu et al., ; Li et al., ; Wang et al., ; Kim et al., ). A wide variety of techniques are used to enhance the performance of LLM-as-a-Judge, such as Chain-of-Thoughts (CoT; Wei et al. ()) to first generate concise reasoning and then the final decision, adding pre-defined rules (Zeng et al., ) in prompts to list some general rules for LLM-as-a-Judge to follow explicitly, and swapping the two responses to avoid positional bias (Wang et al., ). Zheng et al. () found that, even with the use of a CoT prompt, LLM-as-a-Judge can still be misled by the surrounding context, particularly by erroneous response text. Therefore, they propose a reference-guided method where the LLM-as-a-Judge’s response is first generated independently based on the given instruction and then presented as a reference answer within the evaluation prompt. To the best of our knowledge, we are the first to generate response-adapted references from both the instruction and the response to be evaluated.

## 3 Methodology

In this section, we introduce our novel evaluation paradigm, RevisEval, which enhances the evaluation by generating response-adapted references. Illustrated in Figure 2, RevisEval consists of two components, response-adapted reference generation and reference-based evaluation, which we will discuss in Sec. 3.1 and 3.2, respectively.

Supposing $y$ is the response generated by a model for a given task instruction $x$, RevisEval assesses the quality of $y$ on a specific rubric $a$. Firstly, in the generation phase, conditioned on ($x,a$), RevisEval deploys a LLM reviser $\mathcal{R}$ to revise $y$ to generate a response-adapted reference $r^{\star}$.
Secondly, in the evaluation phase, taking the ($x,a,y$) and generated $r^{\star}$ as input, RevisEval adopts LLM-as-a-Judge $\mathcal{F}_{E}$ to assess $y$ using $r^{\star}$ as the reference. Besides, we further expand RevisEval to support traditional reference-based metrics $\mathcal{F}_{M}$. The evaluation objective is to ensure that the automated evaluations align closely with human evaluations, which is introduced in Appendix G.

Figure: Figure 2: Illustration of our proposed RevisEval. Given an instance ($x,y,a$), we use RevisEval to assess $y$ in rubric $a$. In RevisEval, (i) reviser generates a response-adapted reference $r^{\star}$ by revising the $y$ to enhance the (ii) following LLM-as-a-Judge, even classic text metrics.
Refer to caption: x2.png

### 3.1 Response-adapted Reference Generation

LLMs have already demonstrated their surprising revision capabilities in various tasks, including improving specific attribution (e.g., writing style and grammar) of passages (Gao et al., ), correcting hallucination (Akyurek et al., ), post-editing the generated story (Yang et al., ) and generating higher-quality revised responses to complement preference pairs for DPO (Guo et al., ; Yoon et al., ; Jiang et al., ; Xu et al., ). Thus, unlike previous works treating LLMs as discriminators (Hu et al., ), we leverage the revision capabilities of LLMs to unlock the generative potential to offer richer and more valuable insights for evaluation.
RevisEval deploys LLMs to revise the response $y$ from the instruction $x$ on a evaluation rubric $a$,

$$ $r^{\star}=\mathcal{R}(y|x,a),$ (1) $$

where $r^{\star}$ is the generated response-adapted reference for the subsequent evaluation. Notably, when RevisEval must give the preference on two responses, $y_{1}$ and $y_{2}$, in pairwise comparison, we randomly select one for primary text to be revised while using the other as revision guidance. We remind reviser that this revision guidance may not be perfect and should be used with caution in the revision prompt,

$$ $r^{\star}=\begin{cases}\mathcal{R}(y_{1}|y_{2},x,a)&\text{if }\mathcal{C}=1\\ \mathcal{R}(y_{2}|y_{1},x,a)&\text{if }\mathcal{C}=2\\ \end{cases}$ (2) $$

where $\mathcal{C}$ is a random variable that decides which response is chosen for revision. $C$ can be either 1 or 2, with each having an equal chance of occurring. Notably, we discuss other possible revision strategies in Appendix I.2, which are less effective comparably. Introducing a qualified reference can reliably guide the evaluation process, for instance, acting as an “anchor” to reduce biases. Our strategy, which incorporates revision guidance and randomly sampling one as the primary text for revising, further reinforces fairness. This will be validated in Sec. 4.2 and 4.5.

### 3.2 Reference-based Evaluation

In the evaluation phase, RevisEval supports LLM-as-a-Judge $\mathcal{F}_{E}$ in a reference-based setting and remains compatible with previous metrics $\mathcal{F}_{M}$.

#### LLM-as-a-Judge.

With powerful generalization capabilities, LLMs can serve as discriminators for evaluation, referred to as LLM-as-a-Judge. In this case, the evaluation can be operated using $\mathcal{F}_{E}$,

$$ $s=\mathcal{F}_{E}(y|x,a,r^{\star}).$ (3) $$

This can be easily accomplished by simply using a general prompt for inference, where we clarify the instruction, response, response-adapted reference, and evaluation rubric in the prompt, as shown in Appendix A. Moreover, we implement this for open-source LLMs through finetuning evaluation data in this task format, with the detailed process provided in Appendix C.

#### Metrics.

Once we get the reference $r^{\star}$, we can also implement classic metrics, regardless of statistical n-gram or model-based metrics. The evaluation score $s$ is:

$$ $s=\mathcal{F}_{M}(y,r^{\star}|[x,a]).$ (4) $$

More specifically, when $\mathcal{F}_{M}$ are n-gram metrics, e.g., BLEU, ROUGE, and METEOR, we can directly compute the similarity between $y$ and $r^{\star}$; when $\mathcal{F}_{M}$ are model-based metrics, e.g., BERTScore and BARTScore, we can optionally input $x$ and $a$ to the metrics. The effectiveness of metrics heavily relies on the reference. When the response-adapted reference is appropriate, even a simple metric can revive its evaluation functionality in open-ended tasks. We validate this point in Sec. 4.3 and 4.4.

## 4 Experiments

In this section, we first present the comprehensive experimental settings in Sec. 4.1 and evaluate the RevisEval in LLM-as-a-Judge across various tasks in Sec. 4.2; we then verify the effectiveness of classic text evaluation metrics when using response-adapted references in Sec. 4.3; building on above findings, we compare two evaluation paradigms, combining LLM-as-a-reviser with multiple metrics and LLM-as-a-Judge, when using weak LLM in Sec. 4.4; finally, we conduct detailed comparative analysis of RevisEval in Sec. 4.5, such as bias reduction, inference cost impact and reference relevance.

### 4.1 Evaluation Settings

#### Evaluation benchmarks.

We evaluate our approach on multiple classic NLG benchmarks by measuring the correlation between the evaluators/metrics and human annotations in a scoring rating task. We follow the experimental setting of Jiang et al. () and select four representative NLG tasks: Data-to-Text, Machine Translation, Text Summarization, and Story Generation. Table 9 shows the details of these benchmarks. Additionally, we test our approach on the more challenging open-ended instruction-following benchmarks (MT-Bench, Alpacafarm, and LLMBar), which primarily rely on pairwise comparison task. Unlike the NLG benchmarks, these preference datasets contain more general instructions covering a broader range of tasks with more diverse responses.

**Table 1: Kendall ($\tau$) and Spearman ($\rho$) correlation results comparing reference-free, reference-based, and RevisEval methods across natural language generation tasks. This table demonstrates that, without human-annotated references, our proposed RevisEval significantly outperforms reference-free and reference-based methods involving both open-source and proprietary LLM-as-a-Judge.**
| Methods | Summarization | Translation | Data2Text | Story Generation | Avg. |
| --- | --- | --- | --- | --- | --- |
| $\tau$/$\rho$ | $\tau$/$\rho$ | $\tau$/$\rho$ | $\tau$/$\rho$ | $\tau$/$\rho$ |  |
| N-gram Metrics |  |  |  |  |  |
| BLEU | 10.66/14.42 | 14.50/19.73 | 23.13/33.29 | -1.93/-2.70 | 11.59/16.19 |
| ROUGE | 10.81/14.85 | 13.19/17.83 | 24.74/35.49 | -1.53/2.34 | 11.80/17.63 |
| METEOR | 12.37/16.72 | 16.52/18.80 | 25.58/36.27 | -1.87/-2.65 | 13.15/17.29 |
| Model-based Metrics |  |  |  |  |  |
| BERTScore | 17.50/23.83 | 31.57/42.41 | 30.74/43.75 | 16.00/23.79 | 23.95/33.45 |
| BARTScore | 29.12/35.50 | 7.01/12.83 | 22.32/34.33 | 14.15/33.48 | 18.15/29.04 |
| UniEval | 35.89/47.52 | 16.08/21.90 | 28.56/38.38 | 31.22/44.46 | 27.94/38.07 |
| GPTScore | 28.20/37.41 | 6.50/8.90 | 19.81/28.82 | 16.36/23.91 | 17.72/24.76 |
| InstructScore-7B | 20.86/38.68 | 40.44/50.43 | 30.21/38.54 | 13.50/16.13 | 26.25/35.94 |
| TIGERScore-7B | 28.79/35.11 | 33.65/41.50 | 32.44/42.39 | 29.72/39.26 | 31.15/39.56 |
| Llama-3.1 8B-Inst | 27.49/31.02 | 19.59/23.54 | 28.46/36.24 | 26.13/29.97 | 25.42/30.19 |
| Open-Source LLM-as-a-Judge |  |  |  |  |  |
| Ref-Free | 27.83/31.89 | 30.84/38.66 | 38.75/49.32 | 25.74/31.72 | 30.79/37.90 |
| Ref-Based | 34.09/39.53 | 35.76/41.12 | 39.24/50.87 | 8.79/10.44 | 29.47/35.49 |
| RevisEval (Ours) | 32.41/37.73 | 33.14/39.66 | 39.02/49.92 | 25.95/32.11 | 32.63/39.86 |
| Proprietary LLM-as-a-Judge |  |  |  |  |  |
| Ref-Free | 31.82/38.98 | 34.62/43.38 | 37.99/49.50 | 23.81/33.29 | 32.06/41.29 |
| Ref-Based | 32.56/40.01 | 41.47/45.29 | 37.35/49.02 | 17.58/24.86 | 32.24/39.80 |
| RevisEval (Ours) | 33.63/41.15 | 40.72/45.32 | 37.90/50.93 | 25.11/35.26 | 34.34/43.17 |

#### Base LLMs and metrics.

Our proposed RevisEval aims to improve evaluation performance across both LLM-as-a-Judge and classic metrics, offering enhanced results. For proprietary LLMs, we adopt GPT-4 as the base model and focus on implementing our paradigm during the inference stage. For open-source LLMs, we implement our method via finetuning the Llama 3.1-8B model. Following the Jiang et al. ()’s setting on open-source models, we distill the evaluation outputs generated by GPT-4 when inputting task instructions and corresponding evaluated responses, and tune them in our models. Notably, our training data has no overlap with evaluation benchmarks.
For classic metrics, we cover various metrics, like n-gram based BLEU, ROUGE, METEOR and model based BERTScore, MOVERScore (Zhao et al., ), BARTScore, which rely heavily on references. We validate our method by assessing the utility of reference texts.

We ensure that our approach maintains versatility and fairness across models, with further details on prompts, inference, finetuning, and baselines in the Appendix A, B, C, D, E and F.

### 4.2 Enhancing LLM-as-a-Judge Performance

We present the main results of our proposed RevisEval across NLG evaluation tasks and instruction-following preference benchmarks in Table 1 and 2. We summarize the conclusions below.

#### RevisEval achieves stronger performance across various NLG tasks.

For the powerful proprietary LLMs, RevisEval outperforms reference-free and human-annotated reference-based evaluation across tasks with approximately 0.02 in Kendall correlation on average, as demonstrated in Table 1. Notably, in story generation, high-quality human references hinder evaluation for LLM-as-a-Judge, which decreases by 0.06 compared to the reference-free method in the Kendall correlation. In contrast, RevisEval shows that generated response-adapted reference can still significantly enhance evaluation by about 0.08 in Kendall than human reference-based evaluation. An exception is machine translation, where RevisEval aligns closely with reference-based methods, and we analyze this result exists a consistent rationale about reference relevance in Sec 4.5.

For the open-source LLMs, RevisEval not only outperforms the reference-free method but also beats the reference-based methods by over 0.03 on average in Kendall correlation. Especially in story generation, the reference-based approach is consistent with the above conclusion, trailing by approximately 0.17 in Kendall compared to our RevisEval. Furthermore, the LLM with specialized finetuning also performs better than the LLM with general instruction finetuning (i.e., Llama 3.1-8B Inst) on NLG evaluation tasks, leading by about 0.02 in Kendall.

**Table 2: Results of LLM-as-a-Judge on instruction-following preference tasks. Our proposed RevisEval significantly enhances the performance of both open-source and proprietary LLM-as-a-Judge across various general evaluation tasks. Here, D.R. denotes Direct Response to instruction.**
| Methods | # of Training Samples | MT-Bench | AlpacaFarm | LLMBar | Avg. |
| --- | --- | --- | --- | --- | --- |
| Open-Source LLM-as-a-Judge |  |  |  |  |  |
| JudgeLM-7B (Zhu et al., ) | 100,000 | 64.1 | 53.9 | 36.3 | 51.4 |
| PandaLM-7B (Wang et al., ) | 300,000 | 75.0 | 54.9 | 31.7 | 53.9 |
| Auto-J-13B (Li et al., ) | 4,396 | 75.2 | 64.6 | 36.0 | 58.6 |
| Prometheus-7B (Kim et al., ) | 100,000 | 52.8 | 33.5 | 30.1 | 38.8 |
| Prometheus-2-7B (Kim et al., ) | 300,000 | 55.0 | 37.3 | 26.3 | 39.5 |
| Llama 3.1-8B-Tuned |  |  |  |  |  |
| –Ref-Free | 9,800 | 67.4 | 61.1 | 51.1 | 59.9 |
| –Ref-Based (Llama-D.R.) | 9,800 | 74.9 | 61.5 | 58.9 | 65.1 |
| –Ref-Based (GPT-4-D.R.) | 9,800 | 78.0 | 65.5 | 63.0 | 68.8 |
| –RevisEval (Llama-as-a-Reviser) | 9,800 | 75.2 | 64.7 | 57.8 | 65.9 |
| –RevisEval (GPT-4-as-a-Reviser) | 9,800 | 79.3 | 67.1 | 64.9 | 70.4 |
| Proprietary LLM-as-a-Judge (GPT-4) |  |  |  |  |  |
| Ref-Free | - | 81.2 | 70.9 | 72.6 | 74.9 |
| Ref-Based (GPT-4-D.R.) | - | 81.5 | 67.7 | 79.9 | 76.4 |
| RevisEval (GPT-4-as-a-Reviser) | - | 83.0 | 72.9 | 79.0 | 78.1 |

#### RevisEval excels on open-ended instruction-following preference benchmarks.

As shown in Table 2,
whether implemented on open-source or proprietary models, RevisEval consistently surpasses all baselines by at least 6.3% on average. The details of these baselines are listed in Appendix F, and they are tuned with tens of thousands of data for the LLM-as-a-Judge. On the same base LLM, RevisEval exceeds reference-free evaluation by 3%-6%. We compare RevisEval to reference-based evaluations followed by Zheng et al. ()’s setup, whose reference is the LLM’s direct response to the instruction. Our approach performs better than reference-based evaluation on average when both references are generated by the same base LLMs. Furthermore, using GPT-4 as the reviser boosts Llama 3.1-8B-as-a-Judge by over 4.5% compared to Llama-as-a-Reviser, highlighting the importance of reference quality.

LLM-as-a-Judge can be biased toward longer, verbose answers (Saito et al., ; Dubois et al., ) or answers that match a similar format (Huang et al., ). LLMBar (Zeng et al., ) is a challenging benchmark to meta-evaluate such superficial quality biases.
As demonstrated in Table 2, these baselines, even after tuning $>100K$ samples, struggle to exceed 50% accuracy, exposing bias challenge significantly.
By using response-adapted references from RevisEval, the weak open-source LLM-as-a-Judge’s performance improves substantially by about 6%, showing RevisEval can address superficial quality bias.
On proprietary LLM, RevisEval achieves a 3.2% improvement in accuracy compared to reference-free evaluation. Our proposed response-adapted references perform slightly worse than direct responses from the same proprietary model. This is likely because LLMBar emphasizes “instruction-following precision,” where a single ground-truth response exists for the instruction.

In summary, RevisEval consistently outperforms traditional reference-free and reference-based methods, and our revision significantly provides guidance for LLM evaluation by significantly utilizing the generative advantages of LLM.

### 4.3 Activating classic metrics Performance

Figure: Figure 3: Comparative analysis of reference-based metrics performance using references generated by Human/GPT-4 and RevisEval on NLG and instruction following benchmarks. RevisEval can greatly enhance traditional reference-based metrics, even n-gram metrics, reaching performance levels comparable to GPT-4-as-a-Judge in reference-free evaluations.
Refer to caption: x3.png

Since response-adapted references have enhanced LLM-as-a-Judge in Sec. 4.2, this motivates us to ask could these references directly improve the performance of classic metrics, even n-gram metrics?

As the classic text metrics are not compatible with the pairwise comparison, we extend them as follows:

$s=\mathbb{I}(\mathcal{F}_{M}(y_{1},r)>\mathcal{F}_{M}(y_{2},r)),$

where $\mathbb{I}(\cdot)$ is the indicator function, determining which $y_{1}$ or $y_{2}$ is preferred based on the metric scores.
We use human references in NLG tasks and GPT-4 direct responses in instruction-following benchmarks as the baseline references $\hat{r}$, comparing them with references $r^{\star}$ generated by RevisEval. As shown in Figure 3, RevisEval enables each classic metric to significantly surpass its performance based on baseline references across all tasks. This increase is especially evident in more complex open-ended tasks like story generation and AlpacaFarm. Additionally, we find that using $r^{\star}$ generated by GPT-4 as-a-Reviser combined with classic metrics can yield comparable evaluation performance to GPT-4’s reference-free evaluation. Notably, in AlpacaFarm, when using the response-adapted references, MOVERScore achieves 1.2% improvement compared to the reference-free GPT-4-as-a-Judge.

### 4.4 Potential Evaluation Paradigm for Weak Large Language Models

**Table 3: Comparative analysis of weak LLM-as-a-Judge and weak LLM-as-a-Reviser+classic metrics on instruction-following tasks. Under the same finetuning training resources, a weak LLM-as-a-Reviser combined with classic metrics can produce better results.**
| Metrics | MT-Bench | AlpacaFarm | LLMBar | Avg. |
| --- | --- | --- | --- | --- |
| LLM-as-a-Reviser |  |  |  |  |
| BLEU | 64.5 | 63.9 | 51.6 | 60.0 |
| ROUGE | 62.0 | 63.5 | 51.6 | 59.0 |
| METEOR | 66.4 | 67.7 | 46.3 | 60.1 |
| BERTScore | 62.3 | 62.3 | 54.4 | 59.7 |
| MOVERScore | 61.5 | 68.3 | 51.6 | 60.5 |
| BARTScore | 66.9 | 61.9 | 51.3 | 60.0 |
| Majority Voting | 63.4 | 68.5 | 52.5 | 61.4 |
| LLM-as-a-Judge | 67.4 | 61.1 | 51.1 | 59.9 |

We observe that 1) in the Sec. 4.2, all weak LLMs still exhibit a significant gap compared to GPT-4-as-a-Judge, even after extensive training with high-quality, evaluation-specific data, and 2) in the Sec. 4.3, classic metrics combined with response-adapted references generated by GPT-4 can achieve performance close to the reference-free GPT-4-as-a-Judge. Thus, should we consider a potential evaluation paradigm of “weak LLM-as-a-Reviser + metrics” instead of “weak LLMs-as-a-Judge”?

To explore this, we compare “Llama-as-a-Judge” with “Llama-as-a-Reviser + metrics,” as shown in Table 3. When using references generated by “Llama-as-a-Reviser”, we find that BLEU, METEOR, MOVERScore, and BARTScore can surpass “Llama-as-a-Judge” on average across 3 tasks. Furthermore, we apply a majority voting across multiple metrics, outperforming “Llama-as-a-Judge” over 1.5% on average. This suggests that instead of continuously training weak LLMs to improve their evaluative discrimination capabilities, leveraging their generation abilities for revision may be more effective. Without extra inference costs, this approach can lead to better evaluation outcomes.

**Table 4: Positional bias analysis in pair comparison evaluations when applying different evaluation paradigms. This table presents the ratio of changed evaluation results after swapping the response position. A lower proportion indicates less positional bias.RevisEval stands out as the best, exhibiting the lowest bias among all paradigms.**
| Paradigms | MT-Bench | Alpacafarm | LLMBar |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Llama 3.1-8B | GPT-4 | Llama 3.1-8B | GPT-4 | Llama 3.1-8B | GPT-4 |  |
| Ref-Free | 49.1 | 10.3 | 61.1 | 20.0 | 44.6 | 17.9 |
| Ref-Based | 22.8 | 6.5 | 34.1 | 22.2 | 32.5 | 11.2 |
| RevisEval | 20.5 | 5.9 | 30.1 | 19.9 | 30.3 | 7.9 |

### 4.5 Comparative Analysis to Other Evaluation Paradigms

#### Positional bias analysis.

Positional bias (Wang et al., ; Zheng et al., ; Wu & Aji, ; Chen et al., ) occurs when human or LLM evaluators tend to favor one side in a pairwise comparison, regardless of answer quality. We investigate this bias by swapping answer positions and taking the LLM to re-evaluate, as shown in Table 4. The results indicate that reference-based evaluation decisions have less variation than reference-free ones. RevisEval is generally 2%-4% lower on the reference-based evaluation, showing better consistency. This result is probably because RevisEval provides references more closely aligned with the answers, further minimizing bias.

**Table 5: Ablation study on the impact of inference cost. Increasing evaluation cycles to match or exceed RevisEval’s inference cost in reference-free did not improve accuracy. This shows that RevisEval’s superior performance is not from twice inference.**
| Inference Cost | MT-Bench | AlpacaFarm | LLMBar |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Llama 3.1-8B | GPT-4 | Llama 3.1-8B | GPT-4 | Llama 3.1-8B | GPT-4 |  |
| 1-Cycle Ref-Free | 67.4 | 81.2 | 61.1 | 70.9 | 51.1 | 72.6 |
| 3-Cycle Ref-Free | 64.1 | 81.2 | 54.9 | 71.9 | 53.2 | 74.9 |
| RevisEval (2-Cycle) | 71.3 | 83.0 | 64.7 | 72.9 | 54.9 | 79.0 |

#### The impact analysis of inference cost.

Compared to the reference-free approach, RevisEval requires two cycles of inference (i.e., revision and evaluation). To show the impact of extra inference cost, we further
conduct three cycles of reference-free evaluations using extra different temperatures ($0.3$ and $0.7$), followed by majority voting (see Table 5). For GPT-4, additional evaluation cycles slightly improve accuracy but still lag behind ours by 1%-4%. For weak LLMs, more cycles led to worse performance. The above results indicate that our approach provides more valuable guidance.

**Table 6: Comparative analysis of how reference-based evaluation effectiveness varies with changes in the similarity between the response and reference texts across different constructing reference strategies. Here, Effectiveness, $\mathcal{P}_{\text{ref}}/\mathcal{P}_{\text{free}}$, refers to the performance ratio between reference-based and reference-free evaluation, where $\mathcal{P}$ denotes the evaluation performance, e.g., Acc and Corr; similarity is still measured by BERTScore.**
|  | Reference Source | WMT-22(en-zh) | WebNLG | MT-Bench | Summeval | AlpacaFarm | ROC |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Similarity | Human/GPT-4 | 65.12 | 59.09 | 25.00 | 23.51 | 13.04 | 12.86 |
| RevisEval | 63.03 (-3.3%) | 76.41 (+29.3%) | 30.29 (+21.2%) | 35.63 (+51.6%) | 35.72 (+173.9%) | 27.57 (+114.4%) |  |
| Effectiveness | Human/GPT-4 | 1.20 | 0.98 | 1.00 | 1.02 | 0.95 | 0.74 |
| RevisEval | 1.18 (-0.02) | 1.00 (+0.02) | 1.02 (+0.02) | 1.06 (+0.04) | 1.03 (+0.08) | 1.05 (+0.31) |  |

#### Evaluation performance improves with increased relevance between reference and responses.

It’s been observed that the less relevant a reference is to the response, the less effective it is for evaluation in previous work and Figure 1. We further verify whether this trend holds true with our method.
We define effectiveness to describe whether reference-based evaluation is more effective than reference-free evaluation.
As shown in Table 6, the increasing similarity between reference and evaluated responses generally leads to better evaluation effectiveness. This explains why our method doesn’t perform as well in translation, where human references are already highly similar to the response. For other tasks, human or GPT-4 direct-response references have lower similarity than references of RevisEval, leading to a lower effectiveness. Additionally, for different tasks, the similarity between the human/GPT-4 reference and the evaluated response varies, reflecting the open-ended generative degree of this task. A lower similarity indicates a greater diversity of potential valid responses. In this context, as the task becomes more open-ended, the effectiveness of RevisEval shows a greater improvement than the human/GPT-4 reference.

## 5 Case Study: How does it work?

**Table 7: An instruction-following open-ended case of pairwise comparison and a story-generation case of score rating evaluated by different paradigms. Score/Preference refers to accurate evaluation result and Score/Preference refers to wrong result. We emphasize the noisy evaluation elements in red, while highlighting the useful elements of the evaluation in green. Here, the base LLM is GPT-4.**
| Instruction: Classify the following to the correct word group. \n\n Happy, Joyful |  |
| --- | --- |
| Response 1 | Response 2 |
| Happy -> Emotions | Emotions |
| Reference-free Evaluation:<br>Both Assistant A and Assistant B . . . . . . Both responses are correct,<br>but one might argue that including the words with the categorization, as Assistant A does, provides a clearer answer and specifically indicates that both words have been classified. . . . . . .<br>- Final Verdict: [[A]] |  |
| Referfence-based Evaluation:<br>Direct Response-as-Reference: Happy and Joyful belong to the word group of synonyms, as they both express similar feelings of pleasure and contentment.<br>. . . . . . Assistant A,<br>while correct in stating that "Happy" belongs to "Emotions," did not mention "Joyful" or classify them together as synonyms. Rather, it only assigned a category to one word.<br>Assistant B . . . . . .<br>but fails to classify both words as instructed, missing the relationship between "Happy" and "Joyful."<br>Neither assistant directly addressed the classification as "synonyms," which was central to the user’s instruction.<br>- Final Verdict: [[A]] |  |
| RevisEval Evaluation:<br>Response-adapted Reference: Emotions<br>. . . . . . Assistant A . . . . . .<br>However, it is clear from the classification provided that Assistant A understands the context and correctly categorizes "Happy" but overlooks "Joyful".<br>Assistant B simply responds with "Emotions", effectively covering both "Happy" and "Joyful". This response uses a more concise and generalized approach, which aligns perfectly with the reference answer.<br>Based on the reference answer and the clarity and completeness in addressing both terms, Assistant B’s response is superior. . . . . . .<br>- Final Verdict: [[B]] |  |
| Instruction: Generate a reasonable ending for the following story. \n Source Input: my 2 year old cousin was eating an apple . |  |
| Response:<br>he brought apples from his class. he dropped the apples. he began to throw them back up. he had a hard time. |  |
| Gold Score:<br>3.8 |  |
| Reference-free Evaluation:<br>. . . . . .<br>but it lacks coherence and logical flow. The sentences seem disconnected, and the events described do not build upon each other in a meaningful way. The output does not demonstrate a clear understanding of the story’s context or the character’s actions.<br>- Rating: [[1]] |  |
| Referfence-based Evaluation:<br>Human Reference: he walked around between everyone at the party . he started coughing and choking . everyone panicked and started hitting him on the back . he eventually spit out the apple piece .<br>The model-generated output is not fluent and lacks coherence in comparison to the reference output.<br>. . . . . . . . . . . .<br>Overall, the model-generated output fails to meet the fluency criteria due to its lack of coherence and grammatical errors.<br>- Rating: [[1]] |  |
| RevisEval Evaluation:<br>Response-adapted Reference: He had brought apples from his class, but accidentally dropped them. Struggling, he attempted to pick them up and in the process, began tossing them around. He was having a hard time managing the apples.<br>The model-generated output is somewhat fluent but lacks coherence and completeness in comparison to the reference output. . . . . . The reference output provides a more detailed and logical sequence of events, enhancing the narrative’s coherence and clarity.<br>. . . . . .<br>Overall, the model-generated output is fluent but could benefit from improvements in coherence and detail to match the quality of the reference output.<br>- Rating: [[3]] |  |

We show two representative examples in Table 7, one from the instruction-following task (AlpacaFarm) and the other from the NLG evaluation task (story generation).
The reference-free evaluation tends to be influenced by verbose text, often leading to incorrect judgments, e.g.,
“provides a clearer answer…”. While the LLM’s direct response or human reference might not be erroneous and can fulfill the instructions well, for instance, “Happy and Joyful belong to the word group of synonyms, …,” in the first case, it has no relevance to responses to be evaluated and resulting
“neither assistant directly addressed the classification,” judgment. In contrast, the reference generated by RevisEval is a more adaptive guidance, “Emotions”, accurately helping the LLM to align the crucial elements,
“the clarity and completeness”. This is also evident in the second case, where differences between the response and the revised text, such as
“had brought,”
“accidentally dropped,” and others, directly highlight fluency issues with the response.
In other words, this demonstrates a transparent potential.
Furthermore, we provide evaluation discrepancy statistics between RevisEval and other evaluations to observe how different evaluation methods, as demonstrated in Appendix I.3.

## 6 Conclusion

In this work, we introduce a novel yet general evaluation paradigm that leverages the revision capabilities of LLMs to revise evaluated responses to be response-adapted references for evaluation. This approach significantly enhances the reliability of the versatile LLM-as-a-Judge, particularly in effectively reducing bias.
The references generated through our RevisEval greatly improve even the simplest n-gram metrics, achieving performance comparable to LLM-as-a-Judge. This proves especially advantageous for weaker LLMs, which often struggle to improve despite extensive training, providing an efficient method to enhance their evaluation capabilities. Our findings demonstrate that (1) the importance of references has been underestimated, and (2) harnessing the generative strengths of LLMs can substantially support evaluation tasks by increasing reference relevance.

## Appendix A Prompt Template

We provide the prompt templates used for evaluation and revision. These prompts are either taken directly from MT-Bench or adapted from it, ensuring the universality of our proposed paradigm.

Figure: Figure 4: The prompt of reference-free pairwise comparison evaluation.
Refer to caption: x4.png

Figure: Figure 5: The prompt of reference-based pairwise comparison evaluation.
Refer to caption: x5.png

Figure: Figure 6: The prompt of reference-free score rating evaluation.
Refer to caption: x6.png

Figure: Figure 7: The prompt of reference-based score rating evaluation.
Refer to caption: x7.png

Figure: Figure 8: The prompt of LLM-as-a-reviser for pairwise comparison.
Refer to caption: x8.png

Figure: Figure 9: The prompt of LLM-as-a-reviser for score rating.
Refer to caption: x9.png

Figure: Figure 10: The prompt of LLM direct response to instruction.
Refer to caption: x10.png

## Appendix B Inference Setting for Proprietary Model

#### Base Model.

We choose GPT-4 as the base model for our evaluation and revision. For reproducibility, we used the GPT-4 version GPT-4-Turbo-2024-04-09, with a temperature setting of $0.0$.

## Appendix C Finetuning Setting for Open-sourced Model

#### Base Model.

We choose Llama 3.1-8B-Inst (^1^11[https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)) as the base model for our evaluation and revision. Here, we want to clarify that we choose the Instruct model as the base model because finetuning on this model yields better evaluation and revision results than the pretrained model.

#### Training Setting.

We followed the common setup for supervised instruction finetuning, with a $\textit{context length}=2048$, $\textit{epochs}=3$, $\textit{batch size}=128$, and $\textit{learning rate}=2e-5$.

#### Distilling Setting.

Whether finetuning open-source models for evaluation or revision capabilities, the training data comes from the generation of a powerful model prompted by the same data source. The model we choose to distill is still GPT-4, with the same version and inference settings as mentioned above.

**Table 8: The Statistics of NLG Evaluation Training Data.**
| Task | Aspects | Samples Items | Evaluation Items |
| --- | --- | --- | --- |
| Summarization | fluency,consistency,coherence,relevance | 2886 | 11544 |
| Translation | accuracy | 6000 | 6000 |
| Data2Text | accuracy,fluency | 3098 | 6196 |
| Story Generation | fluency,consistency,style matching | 1052 | 3156 |

Figure: Figure 11: In our data distillation process for open-source LLMs, we utilize HH-rlhf and MetricInstruct as the primary data sources. We then employ a proprietary LLM to perform RevisEval, generating both the revisions and corresponding evaluation outputs. Finally, we fine-tune the open-source LLM using this enriched dataset.
Refer to caption: x11.png

#### Distilling Data Source.

We depict the distilling data flow in Figure 11. For NLG evaluation, ideally, we would have a variety of erroneous samples along with human evaluation scores for them. However, such data typically exists only in test sets, making it unavailable for training and often limited in quantity. Therefore, we choose MetricInstruct (^2^22[https://huggingface.co/datasets/TIGER-Lab/MetricInstruct](https://huggingface.co/datasets/TIGER-Lab/MetricInstruct)), proposed by Jiang et al. (), as our training data source. This dataset provides a large volume of diverse erroneous texts, which serve as the basis for evaluation. From the $40K+$ data points, we filter out other NLG tasks and apply our previously mentioned prompts with corresponding aspects, maintaining the same inference settings to generate evaluation scores and reasoning for these error samples. Detailed statistics are presented in the Table 8. Although the overall dataset size is relatively small compared to other works specifically designed to train evaluators, the NLG evaluation data we assess remains held-out from these training samples.

Unlike NLG evaluation data that lacks human-labeled evaluation, preference data typically contains substantial human-labeled preference annotations without evaluation. We choose the most commonly used hh-rlhf (^3^33[https://huggingface.co/datasets/Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf)) (Bai et al., ) dataset, applying the aforementioned prompts and inference settings to conduct evaluations on this data to get the evaluation. We select the preference correctness intersection of reference-free evaluation, reference-based evaluation, and gold preference annotations, ensuring both accuracy and fairness when comparing the performance of LLMs under different evaluation methods post-training. In the end, we selected $10,000$ samples, each containing corresponding revisions, reference-free evaluation, and reference-based evaluation. These three sets were then used to train the same model separately, ensuring that no new information is introduced to alter the distribution.

#### Finetuning LLM-as-a-Judge adaptive to NLG.

Each instance in filtered MetricInstruct is a tuple (response, input context, task instruction, aspect), then use GPT-4 to get the corresponding (reference-free evaluation, revision text, RevEvo evaluation) to each instance. Subsequently, we separately use these data to finetune LLM to get the reference-free evaluator, LLM-as-a-reviser, and reference-based evaluator.

Each instance in filtered hh-rlhf is a tuple (instruction, response a, response b), then use GPT-4 to get the corresponding (reference-free evaluation, revision text, RevEvo evaluation) to each instance. Subsequently, we separately use these data to finetune LLM to get the reference-free evaluator, LLM-as-a-reviser, and reference-based evaluator.

#### Decoding Setting.

When the finetuned model executes the evaluation and revision tasks, the decoding setting uses a greedy decoding strategy with a $\textit{max output length}=1024$ and $\textit{temperature}=0.01$.

## Appendix D Benchmarks

### D.1 NLG Evaluation Benchmarks

**Table 9: The Statistics of NLG Evaluation Benchmarks.**
| Task | Benchmark | Response Source | Inputs Items | Samples Items |
| --- | --- | --- | --- | --- |
| Summarization | SummEval (Fabbri et al., ) | 16 Models | 100 | 1600 |
| Translation | WMT-22 (zh-en) (Kocmi et al., ) | 18 Models | 1875 | 33750 |
| Data2Text | WebNLG-2020 (Zhou & Lampouras, ) | 18 Models | 179 | 2848 |
| Story Generation | OpenMEVA (ROC) (Guan et al., ) | 5 Models | 200 | 1000 |

Traditional text generation tasks and their corresponding evaluation benchmarks are highly diverse. Based on the varying degrees of freeform in text generation tasks, we select four representative Machine Translation, Data-to-Text, Summarization, and Story Generation. We follow the experiment setting of Tigerscore and choose specific benchmarks for each task, and their statistics are shown in the Table 9.

### D.2 Instruction Following Preference Benchmarks

With their powerful generalization capabilities, LLMs have become the focus of research on NLP generation abilities. Evaluating LLMs requires more challenging tasks that can assess their generalization capabilities. Here, we selected three representative benchmarks: MT-Bench (abbr., MT-Bench_Human-Judgement), AlpacaFarm, and LLMBar.

#### MT-Bench.

This dataset comprises $3.3K$ expert-level pairwise human evaluations of model responses, generated by six models across 80 MT-Bench questions. The six models include GPT-4, GPT-3.5, Claude-v1, Vicuna-13B, Alpaca-13B, and LLaMA-13B, offering a diverse representation of powerful language models. The topic of subtasks is consisted of Writing, Roleplay, Reasoning Math, Coding, Extraction, STEM and Humanities. MT-Bench is the most common benchmark for evaluating LLM-as-a-Judge, and our results validate the feasibility of our experiments. We select the first round of dialogues from this dataset as the evaluation data, containing $1284$ cases.

#### AlpacaFarm.

We utilize Human-Crossannotation (^4^44[https://huggingface.co/datasets/tatsu-lab/alpaca_eval/blob/main/alpaca_farm_human_crossannotations.json](https://huggingface.co/datasets/tatsu-lab/alpaca_eval/blob/main/alpaca_farm_human_crossannotations.json)) set specifically designed for evaluating the reliability of evaluators, following the AlpacaFarm process. Each instance in this dataset contains cross-annotations from 4 human experts. Additionally, the tasks in this dataset are more diverse, open-ended, and challenging, making the preference annotations more reliable. Notably, since four experts conduct the preference annotations, some instances resulted in ties, where two annotators favored the first option and the other two favored the second. We filtered out these tied cases, leaving a final evaluation dataset of $501$ instances.

#### LLMBar.

LLMBar is a meta-evaluation benchmark designed to test how well LLM evaluators can identify instruction-following outputs. It consists of two parts: (1) The Natural set, which gathers instances from existing human-preference datasets, filtered and adjusted to ensure a clear preference for each instance. (2) The Adversarial set, where the authors intentionally create misleading outputs that superficially seem good but deviate from the instructions, to challenge the evaluators. The Natural set measures performance in real-world conditions, while the Adversarial set tests evaluators’ ability to detect true instruction-following. The overall size is $419$.

## Appendix E Baselines in NLG Evaluation Tasks

#### N-gram Metrics.

N-gram text generation metrics are commonly used to evaluate the text quality generated by models, especially in tasks like machine translation and summarization. While these metrics are simple and efficient, they come with notable limitations: they are highly sensitive to surface-level differences, such as word order or vocabulary choice, which may fail to capture the true meaning or fluency of the generated text. In our evaluation, we use the most widely adopted metrics: BLEU, ROUGE-L, and METEOR.

#### Model-based Metrics.

To capture the semantic-level meaning of generated text, researchers have started using models like BERT and BART as the foundation for text evaluation. Typical examples include BERTScore, BARTScore, and Moverscore. BERTScore computes the similarity between two text sequences based on the contextual embeddings from BERT, while Moverscore enhances this by adding many-to-one alignment. BARTScore, on the other hand, uses BART to calculate the probability of converting the response to the reference text as a score. All of these are reference-based model metrics.

Additionally, there are reference-free model-based metrics. These metrics are trained on specific task datasets, allowing the model to internalize relevant information. As a result, the model can generate evaluations without needing reference texts and transform them into scores. For instance, UniEval uses data augmentation to expand a task-specific dataset to 30K examples and fine-tunes on T5, which is why it performs exceptionally well in summarization tasks.

#### LLM-as-a-Judge.

Following the exciting advancements in large language models (LLMs), the most straightforward approach has been to replace the models in previous model-based metrics, such as BART, with larger models. GPTScore follows this concept, and despite its simplicity, it delivers notable results. Moreover, leveraging the vast internal knowledge of open-source LLMs, more powerful and interpretable evaluators can be developed, such as INSTRUCTScore and TIGERScore.

Xu et al. () argue that performing error analysis on given reference texts enhances evaluation explainability and reliability. They used NLG evaluation data and employed GPT-4 to perform error-based assessments. These outputs were then paired with the response to train an Llama model, resulting in a training dataset of 10K examples. TIGERScore goes a step further by proposing a reference-free approach. Using a similar strategy, they collected 40K data points for training, masking the reference text during the process.

## Appendix F Baselines in Instruction Following Preference Benchmarks

**Table 10: The Lists of weak LLM-as-a-Judge.**
| Model | Base Model | Instruction | Response Annotation | Evaluation Scheme | Training Samples |
| --- | --- | --- | --- | --- | --- |
| JudgeLM | Vicuna-7B | Alpaca-GPT4,<br>Dolly-15K… | 11 models<br>(Alpaca, Vicuna…) | GPT-4 Pairwise Grading | $100K$ |
| PandaLM | LLaMA-7B | Alpaca 52K | 5 models<br>(LLaMA, Bloom…) | GPT3.5 Pairwise Selection | $300K$ |
| Auto-J | LLaMA2-13B-chat | Chatbot Arena,<br>OpenAI WebGPT… | Preference Datasets | Human Pairwise Selection,<br>Pointwise Grading | 4396 |
| Prometheus | LLaMA2-7B-chat | GPT-4 Generated | GPT-4 Generated | GPT-4 Pointwise Grading | $100K$ |
| Prometheus-2 | Mistral-7B-v2.0 | GPT-4 Generated | GPT-4 Generated | GPT-4 Pointwise Grading<br>Preference | $300K$ |

#### LLM-as-a-Judges.

The baseline models are listed in the Table 10. Considering scalability and cost, researchers have long sought to achieve evaluation performance on weaker LLMs that is comparable to that of stronger LLMs. The most straightforward approach to this challenge has been to automatically generate more preference-related data, and many of these efforts have followed this strategy.

#### N-gram Metrics & Model-based Metrics.

Unlike the NLG-Evaluation benchmark, the LLM-as-Judge benchmark has largely moved away from using n-gram metrics and model-based metrics. This shift is due to several characteristics of the test samples in such benchmarks: (1) The response space is extremely large and unconstrained, making reference annotations both unhelpful and prohibitively expensive; (2) These metrics do not perform well for tasks such as coding or math, where even models like BERT struggle to capture semantic-level meaning. In this study, we apply these metrics to demonstrate that, when reference texts are highly relevant, these otherwise “inapplicable” metrics can be reactivated and produce meaningful results.

## Appendix G Meta Evaluation

Meta-evaluation aims to assess the performance of automated metrics by measuring how well the automated evaluations $y_{\text{auto}}$ align with human evaluations $y_{\text{human}}$. For score ratings, we calculate the correlation values across all $N$ samples, represented as:

$Corr=g\left([y_{\text{auto}}^{1},\dots,y_{\text{auto}}^{n}],[y_{\text{human}}^
{1},\dots,y_{\text{human}}^{n}]\right),$

where $g$ can adopt various correlation functions, e.g., Spearman. For pair-wise comparison evaluations, accuracy is typically used as the evaluation metric,

$Acc=\frac{1}{|N|}\sum_{(i,o_{1}^{i},o_{2}^{i})\in N}\mathbb{I}[y_{\text{auto}}
^{i}=p^{i}]$

#### NLG Tasks.

In the NLG evaluation task, it’s crucial to assess various rubrics of the text during the evaluation process. For the four selected tasks, we’ve outlined the specific rubrics to be evaluated in the accompanying Table 8. In both the SummEval and Story-Generation tasks, we evaluate multiple rubrics independently, calculating the correlation coefficient for each one. Subsequently, we compute the average correlation coefficient across all rubrics to obtain an overall assessment for each task. This comprehensive approach ensures a more nuanced and accurate evaluation of the model’s performance across different dimensions.

## Appendix H Pearson Correlation in NLG Evaluation Tasks

We also supplement the Pearson Correlation results in the NLG Evaluation Tasks.

**Table 11: Pearson correlation coefficients comparing non-reference, static-reference, and dynamic-reference methods across various text generation tasks and Instruction-Following Benchmarks. This table summarizes the performance of these methods in generating summarization, translation, data-to-text, and story-generation tasks.**
| Methods | Summarization | Translation | Data2Text | Story Generation | Avg. |
| --- | --- | --- | --- | --- | --- |
| n-gram Metrics |  |  |  |  |  |
| BLEU | 14.13 | 17.47 | 34.29 | -3.89 | 15.50 |
| ROUGE | 15.36 | 16.26 | 35.85 | -0.22 | 16.81 |
| METEOR | 18.69 | 18.80 | 36.30 | -1.02 | 18.19 |
| Reference-free Metrics |  |  |  |  |  |
| BERTScore | 26.26 | 37.65 | 48.22 | 26.58 | 34.68 |
| BARTScore | 19.73 | 29.04 | 47.89 | 17.76 | 28.61 |
| UniEval | 53.22 | 23.11 | 51.14 | 44.88 | 43.09 |
| GPTScore | 13.47 | 21.05 | 48.70 | 18.94 | 25.54 |
| InstructScore-7B | 27.40 | 51.55 | 47.28 | 12.81 | 34.76 |
| TIGERScore-7B | 43.95 | 37.70 | 49.13 | 39.90 | 42.67 |
| Llama-3.1 8B-Instruct | 25.89 | 27.84 | 31.15 | 31.04 | 28.98 |
| Open-Sourced LLM-as-a-Judge |  |  |  |  |  |
| Ref-Free | 33.61 | 25.14 | 53.36 | 35.02 | 36.78 |
| Ref-Based | 42.32 | 29.99 | 48.52 | 11.92 | 33.19 |
| RevEvo(Ours) | 39.69 | 29.58 | 53.87 | 29.04 | 38.05 |
| Proprietary LLM-as-a-Judge |  |  |  |  |  |
| Ref-Free | 42.12 | 41.35 | 54.26 | 33.50 | 43.56 |
| Ref-Based | 43.31 | 44.11 | 53.98 | 24.63 | 41.51 |
| RevEvo(Ours) | 43.81 | 43.92 | 54.25 | 35.07 | 44.26 |

## Appendix I Other Analysis

### I.1 The benefits of training resource scale for LLM-as-a-Judge are questionable.

As shown in Table 2 and 10, Despite being trained with extensive evaluation-specific resources, these LLM-as-a-Judge baselines fail to achieve evaluation performance comparable to GPT-4, particularly on the adversarially designed LLMBar, where they perform worse than random selection. While substantial effort is put into designing and generating a large amount of training data for these LLMs, the results are even less effective than our evaluator trained on just 10,000 samples from the hh-rlhf dataset. The possible reasons for this could be: 1. The inherent capabilities of the base model play a more crucial role; 2. Simply increasing the volume of training data does not yield significant benefits; 3. Efforts should be focused on other potentials to enhance the evaluator, such as the method we propose.

### I.2 Other Revision Strategies for Pairwise Comparison

When revising two given responses to generate a reference, we experiment with two different revision strategies. In our work, we adopt a strategy where one text is randomly selected as the primary text to be revised, while the other serves as the revision guidance. In addition to this, we try the following two prompt strategies:
a) Revising a single text based on both responses.
b) Revising each response separately.
However, the outcomes of these two strategies are unsatisfactory. Strategy (a) exhibit a tendency to forcibly merge the two responses during the revision process, resulting in a generated text that lacked logical consistency. Strategy (b), on the other hand, lead to a revised text with very low similarity to the other response, inevitably favoring the one chosen for revision. Neither method is capable of producing a satisfactory reference.
We anticipate that future research might provide more revision strategies or approaches to more effectively combine multiple responses and generate a high-quality reference. We look forward to seeing further developments in this area.

### I.3 Evaluation Differential of Different Methods

We analyse the differential between RevisEval and two other evaluation methods, showcasing the proportion of samples where the evaluation decisions differed. A higher proportion indicates a greater difference in the evaluation mechanisms of the two methods. The Table 12 present that RevisEval and the other two evaluation methods make different decisions on 22% of the samples in AlpacaFarm and 16% of the samples in LLMbar. This indicates that our evaluation method has a significant difference in mechanism-level, compared to the other two methods.
Furthermore, it suggests aggregating these differing decisions could potentially lead to a more reliable final evaluation.

**Table 12: Statistics on the differential proportion between RevisEval and each of the other two evaluation methods. A higher ratio indicates a greater evaluation difference between the two mechanisms.**
| Comparative Evaluation | MT-Bench | AlpacaFarm | LLMBar |
| --- | --- | --- | --- |
| Ref-Free | 8.1 | 22.8 | 16.9 |
| Ref-Based (GPT-4 Direct Response) | 9.7 | 21.6 | 16.2 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. While the overall paper is well-written, mentioning what the numbers mean in each table and how they have been calculated in the captions or in the text may improve the readability of the paper to a general user. For eg:  mentioning that the values in Table 2 is the accuracy against human preferences... 
2. As mentioned in the weaknesses, please provide details of any experiments that were conducted to study the soundess of this approach for factual responses (where the generated response contains errors which get normalised in the adapted reference).
soundness: 1
strengths: 1. The paper motivates the problem very well by identifying the issues with current reference-based evaluation paradigms. The idea of dynamically generating contextually relevant references is creative and interesting. It aims to address very important and quite relevant aspects of using LLM as evaluators. 
2. Extensive experiments have been conducted across different tasks as well as various metrics have been evaluated. The authors also show the generalizability of their approach to different metrics.
3. The paper also considers and accounts for the various biases present in LLM Evaluators and also considers the cost of conducting evaluations (which is often ignored in a lot of works)
4. Many interesting insights have been reported by the authors, including using these contextually relevant references to improve the standard n-gram and model-based metrics.
summary: The paper proposes an interesting method “RevisEval” which explores a new approach to performing reference-based evaluation by modifying references based on the responses to be evaluated. The authors show that this improves the reliability of LLM-based evaluators as compared to using static references, by hypothesizing that an effective reference must be closely relevant to the response to be evaluated. Authors show many interesting observations and analysis across various standard NLG tasks as well as open-ended generative tasks and also evaluate various metrics (both standard and LLM-based). Authors also show that these adapted references can even boost the efficacy of standard metrics.
weaknesses: While I agree with the motivation behind the paper, I am not sure about the soundness of the  methodology followed to generate the reference answers:
1. Using the response itself to generate an "adapted reference", the evaluation might indirectly validate the response’s content and structure. This may lead to artificially inflated evaluations, as the evaluator is essentially comparing the response against a modified version of itself, which serves as the reference.
2. If the response contains subtle errors, the adapted reference “might” effectively validate or normalize these errors. These is no study around whether the reviser indeed accounts for or corrects for these errors.
3. While this approach may work well for evaluations of standard NLG tasks as well as some open-ended tasks that care about the language generation capabilities, but for evaluations that care about the factual accuracy of the responses (something where LLMs are overall known to hallucinate), this simple revision may not be robust.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: * How would the proposed method work for multiple references? For open-ended text generation tasks, including MT, multiple references are often used.
* What's the metric used in Table 2? Accuracy?
* Figure 3 — it looks like which metrics are most effective (and closest to GPT-4 performance) vary based on the specific metrics used. Would you provide some general guidelines which metric(s) are most effective in general when combined with RevisEval? Or simply doing majority voting is a good strategy?
* In the last paragraph of Section 4.2 — "significantly" is used two times. Are they used in a statistical sense? If not, they are simply very subjective adverb and I would advise against using it in this context
* Same for the section title of 4.3 — what do you exactly mean by "activating?" I would rephrase with something simpler, e.g, "Improving"
* What are some examples of future work of RevisEval? The conclusion section only provides the summary of the findings.
soundness: 4
strengths: * Simple yet effective method — the core idea of the proposed method, RevisEval, is very simple—simply "rewrite" the response based on the human-written reference (and the rubric) and use it as a new reference. The method is also effective for many settings including LLM-as-a-Judge and traditional reference-based metrics. It is easy to imagine that the proposed method is used in evaluation of many NLG tasks going forward.
* Good ablation studies — the paper provides a wide set of ablation studies to show the proposed method's effectiveness. It shows evaluation results on scoring tasks as well as pairwise preference benchmarks. I also liked the bias analysis (Section 4.5) as well as the detailed analysis of concrete examples (Section 5).

Overall, the paper is overall well written and provides enough evidence that the proposed method is simple, effective, and widely applicable.
summary: Recently, LLM-as-a-Judge has been gaining popularity for evaluating language generation tasks, but still has a few reliability challenges compared to human evaluation. This paper proposes RevisEval, a text evaluation method that can be used for LLM-as-a-Judge methods as well as more traditional reference-based evaluation metrics, such as BLEU and BERTScore. The core of the method is to use LLMs to revise the response (system output) based on the human reference, called reponse-adapted references, which is then used as a new reference in the downstream evaluation, be it LLM-as-a-Judge or traditional evaluation metrics. Through experiments, the authors showed that 1) the proposed method RevisEval showed improved correlation with gold standard compared to reference-free and baseline reference-based evaluation methods, and 2) on preference tasks, RevisEval outperforms baselines including fine-tuned LLM-as-a-Judge models, 3) the proposed method reduces the positional bias compared to reference-free methods as well as conventional reference-based method.
weaknesses: No major weakness as far as I see. Here are some minor weakness points:

* Unclear names—personally I find "response-adapted references" very confusing. It sounds like the method adapt references based on response, but actually it's the other way around. It is actually reference-adapted responses, but I'm not sure if this is a better way of describing it (I don't have any better ideas).

* Unclear description of the experiment settings—the main body of paper benefits a bit of description about the benchmarks. It is based on Tigerscore, but the paper provides very little information re: the specific datasets used and their sizes. Importantly, I think the variety and the quality distribution of responses matter a lot for the evaluation of evaluation methods, and a few sentences about the quantity and the quality of the benchmark datasets would be very helpful.

* Future prospect—this is a bit hypothetical, but the very reason why RevisEval works at all is that current LLMs are in general better at generation rather than discrimination, as the authors state in Section 4.4. Does this mean that in the future, if we have more powerful LLMs at discrimination, would the proposed method still be useful, since future LLMs can simply "guess better" using the reference and the response? 

* Applicability — the paper already shows experimental results on a wide range of tasks and benchmarks, but I'm suspecting they are all English tasks (only exception is the source sentences of MT, which are in Chinese). It doesn't have to be done in this paper, but it would be valuable to test the effectiveness of RevisEval in a wider range of tasks (e.g., image captioning) and languages.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to Weaknesses.
soundness: 4
strengths: The proposed method is intuitive and reasonable, with a straightforward implementation that advances previous work using LLMs to generate references for evaluation. They also consider a comprehensive range of experimental setups, baseline methods, and evaluation benchmarks to verify the effectiveness of their method, resulting in solid experimental analyses.
summary: The work proposes a simple and straightforward evaluation method that involves modifying and enhancing the output text to be evaluated, using it as the reference for further evaluation, motivated by the potentially unsatisfactory quality of traditional references. They experiment with various setups, including using strong and weak LLMs as revisors and employing both traditional evaluation metrics and LLM-based evaluators.
weaknesses: Given that previous studies have already utilized LLMs to generate higher-quality references as replacements for traditional references (Tang et al., 2024), the innovation and contribution of this method are somewhat diminished. I believe they could further enhance the analysis by more comprehensively comparing these two approaches for generating references (generation as reference vs. revision as reference). Additionally, I suggest exploring the use of more refined response-adapted references, such as having the revisor focus on specific dimensions during evaluation, to allow for a richer and more diverse discussion.

The experiments in this work are thorough, but they may be somewhat distracting. First, the main experimental results presented in Tables 1 and 2 involve some inconsistent demonstrations; for example, both tables include the "Open-Source LLM-as-a-Judge" part, but the types of methods involved seem different. In Table 1, it’s unclear whether "Ref-Based" refers to references generated by the corresponding LLMs or the original references, which is important. And Sections 4.3 and 4.4 may not be as critical and could be moved to the appendix, given the availability of stronger evaluation methods; this would allow space for more in-depth experiments and analysis.

**Reference**

Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References (Tang et al., NAACL 2024)

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. While the improvements look promising, there are some questions about the effectiveness of the proposed solution. Recent meta-evaluation works like Reward Bench [1] show that reward models are much more powerful than LLM-as-Judges in proxying human responses. Does the proposed methodology have a benefit against RMs? 

2. Automated evaluators are also widely used as a proxy for human preference in RLHF. An additional step to generate revisions makes the whole process slower and expensive. Hence, while the performance may be promising, it seems like it limits the usage of automated evaluators. Where do you expect this methodology to be used? 

3. Mandating a revision step before evaluation assumes that the revising model can refine the answer better. What if the question-response pair is too difficult for the model to revise? Will the methodology still be effective?


[1] https://arxiv.org/abs/2403.13787
soundness: 2
strengths: 1. The paper proposes a simple and straightforward solution to improve reference-based evaluation. The methodology is easy to implement and shows promising improvement in quality on different benchmarks. 

2. The methodology shows strong robustness, naturally controlling for style, and shows nice performance on adversarial benchmarks like LLM Bar, despite relatively small training.
summary: The paper proposes a simple solution to enhance reference-based evaluation of LLM-as-a-Judges. Instead of using pre-made references, the work introduces a novel evaluation paradigm, "Revise-and-Evaluation," where an LLM revises the provided input to generate a reference answer. The authors note that this method is effective in creating a reference similar to the response in terms of style and different artifacts, effectively accounting for the quality of the answer only. The methodology can be expanded to classical evaluation methodologies like BLEU, ROUGE, and more. The methodology is tested on diverse benchmarks.
weaknesses: Please see the questions section.

Meta-review: The paper proposes RevisEval which improves on the standard LLM-as-a-judge pipeline by modifying the reference response to align better with the generated output. Reviewers generally believe the idea has promise and it improves over the baseline LLM-as-a-judge methods. They do raise some issues: (1) revising the LLM output to generate the reference might introduce unintended biases, (2) the improvement over the baseline simply uses a strong LLM to generate a output-agnostic reference is quite minimal. Adding more analysis to study point 1 would strengthen the paper.

Decision: Accept (Poster)