Title: CriticEval: Evaluating Large-scale Language Model as Critic
Venue: NeurIPS 2024
Accepted: yes
Keywords: Critique ability

LLM

evaluation
Abstract: Critique ability, i.e., the capability of Large Language Models (LLMs) to identify and rectify flaws in responses, is crucial for their applications in self-improvement and scalable oversight. While numerous studies have been proposed to evaluate critique ability of LLMs, their comprehensiveness and reliability are still limited. To overcome this problem, we introduce CriticEval, a novel benchmark designed to comprehensively and reliably evaluate critique ability of LLMs. Specifically, to ensure the comprehensiveness, CriticEval evaluates critique ability from four dimensions across nine diverse task scenarios. It evaluates both scalar-valued and textual critiques, targeting responses of varying quality. To ensure the reliability, a large number of critiques are annotated to serve as references, enabling GPT-4 to evaluate textual critiques reliably. Extensive evaluations of open-source and closed-source LLMs first validate the reliability of evaluation in CriticEval. Then, experimental results demonstrate the promising potential of open-source LLMs, the effectiveness of critique datasets and several intriguing relationships between the critique ability and some critical factors, including task types, response qualities and critique dimensions.

CRITICEVAL: Evaluate Large Language Model as Critic

Abstract

Critique ability, i.e., the capability of Large Language Models (LLMs) to identify and rectify flaws in responses, is crucial for their applications in self-improvement and scalable oversight. While numerous studies have been proposed to evaluate critique ability of LLMs, their comprehensiveness and reliability are still limited. To overcome this problem, we introduce CRITICEVAL, a novel benchmark designed to comprehensively and reliably evaluate critique ability of LLMs. Specifically, to ensure the comprehensiveness, CRITICEVAL evaluates critique ability from four dimensions across nine diverse task scenarios. It evaluates both scalar-valued and textual critiques, targeting responses of varying quality. To ensure the reliability, a large number of critiques are annotated to serve as references, enabling GPT-4 to evaluate textual critiques reliably. Extensive evaluations of open-source and closed-source LLMs first validate the reliability of evaluation in CRITICEVAL. Then, experimental results demonstrate the promising potential of open-source LLMs, the effectiveness of critique datasets and several intriguing relationships between the critique ability and some critical factors, including task types, response qualities and critique dimensions.

1 Introduction

Critique ability is crucial for the self-improvement [1, 2] of LLMs, as it enables the effective analysis and correction of flaws in responses [3, 4]. This capability also facilitates a more robust framework, i.e. , scalable oversight [5, 6], for ensuring the AI systems growing in scale and capability remain aligned with human-desired outcomes and ethical standards.

So far, while numerous works have been proposed to evaluate critique ability of LLMs in downstream tasks, like common NLP tasks [7, 8] and reasoning tasks [4, 9], their comprehensiveness and reliability are limited. Specifically, existing works [10, 11, 12] typically evaluate only specific aspects of critique ability, resulting in limited evaluated critique dimensions [13, 11, 12, 10, 14], insufficient analysis of response qualities and task types [4, 9]. Besides, while GPT-4 is frequently used to evaluate textual or natural language critiques [3, 10, 13], its reliability across all critique dimensions and tasks remains unverified [10, 15]. In summary, a comprehensive and reliable benchmark for assessing critique capability of LLMs is still under-explored, significantly impeding the in-depth analysis.

To fill this gap, we propose a novel benchmark, CRITICEVAL, designed to comprehensively and reliably measure critique capability of LLMs. Specifically, to ensure comprehensiveness, CRITICEVAL evaluates critique ability of LLMs from following dimensions: evaluating a single response (feedback), comparing pairs of responses (comparison), correcting the response based on feedback (correction) and evaluating one feedback of LLM (meta-feedback). These critique dimensions cover all categories of critiques in previous works [11, 13, 4] and the necessary capabilities for self-improvement of LLM [1] and scalable oversight [6]. These critique dimensions are measured under nine diverse task scenarios, including three common NLP tasks, two alignment tasks, and four math reasoning and coding tasks. Moreover, evaluated responses in each critique dimension and each task are collected using various open-source and closed-source LLMs with different capabilities, with human annotation ensuring varied quality levels. Furthermore, since both the scalar-valued and textual formats of critique are commonly used in these scenarios [3], CRITICEVAL evaluate the critiques in both formats, equipped with objective [18, 19] and subjective [10, 11] evaluations, respectively. Note that scalar-valued critiques typically refer to Likert scores and preference labels, while textual critiques refer to more fine-grained textual analysis about response quality [11, 3]. Overall, as shown in Table 1, CRITICEVAL exhibits significant advantages in comprehensiveness compared to previous benchmarks. It demonstrates great diversity in critique formats, critique dimensions, response qualities and the data size of textual critique.

To ensure the reliability of evaluating textual critiques in CRITICEVAL, a large number of high-quality critiques are annotated, serving as references for GPT-4 to evaluate textual critiques automatically. To annotate these textual critiques efficiently, we employ a human-in-the-loop pipeline [20], first generated by GPT-4 and then rigorously reviewed and refined by human experts.

Extensive evaluations of 35 widely used open-source and closed-source LLMs prove the reliability of CRITICEVAL. Specifically, GPT-4 with human-annotated reference critiques achieves close correlations with human judgments, while removing them results in significant performance loss. Additionally, critiques with higher scores consistently lead to superior improvements, illustrating a clear correlation between the real critique ability of LLMs and their evaluation scores within CRITICEVAL. Then, extensive evaluations results also demonstrate that some open-source LLMs, such as Qwen [21] and InternLM2 [22], are approaching state-of-the-art closed-source LLMs in critique capabilities, and their critique ability could be further improved through scaling strategy. Besides, the effectiveness of critique datasets is also validated. Finally, these evaluation results also reveal several intriguing phenomena:

Critique difficulty varies by task type. For instance, math reasoning and coding tasks are more challenging for feedback, comparison, while they are easier for meta-feedback.

There is an inverse relationship between the quality of critiques and responses. For example, high-quality responses pose a greater challenge to critique effectively.

Critique difficulty correlates with the critique dimensions; notably, comparison and metafeedback dimensions present greater challenges than feedback dimension.

These observations and phenomena promote an in-depth understanding of critique ability of LLMs. We hope the discoveries could spur future research in this field.

2 Related Work

2.1 Application of Critique Ability

Automatic Evaluation Automatic evaluation, also known as critique ability in recent works [3, 10], has been well studied in the past few years [23, 18]. It aims to accurately judge the evaluated responses in numerous NLP tasks and reduce the high cost of human annotations [18, 17, 24, 25, 26]. Recently, advanced LLMs, like GPT-4, have exhibited very close correlation with human judgments [18, 27, 13], assign textual critiques with corresponding quality scores, i.e., the scalar-valued score, in a chain-of-thought inference manner [28]. To further mitigate the high inference cost of closed-source LLMs, numerous works propose to improve critique ability of open-source LLMs by fine-tuning them on critique datasets generated by GPT-4 [14, 29, 30], like Auto-J [11] and UltraFeedback [12].

LLM Self-improvement So far, critique ability has been widely used for self-improvement of LLMs in two stages: (1) Inference stage : Given textual critiques that analyze the flaws in the response and provide suggestions, LLMs can iteratively improve the response quality [6, 15, 31, 32, 33].; (2) Training stage : Scalar-valued critiques are frequently used to compile responses with a clear performance gap for rejective fine-tuning (RFT) or preference learning (RLHF [34]), which further enhances LLM capabilities [1, 2, 5, 35, 36]. For instance, Self-rewarding [1] improves Llama-2-70B by fine-tuning it on samples selected based on its rewards. Similarly, ChatGLM-Math [36] fine-tunes a math-critique model for scoring generated answers, which are used for rejective fine-tuning [37] and direct preference optimization [38].

2.2 Benchmarking Critique Ability

So far, numerous meta-evaluation benchmarks have been proposed to evaluate the critique ability of models [18]. Early benchmarks mainly focus on evaluating scalar-valued critiques [30, 24] on common NLP tasks [18, 23], like translation [7] and summary [8] by computing the correlations between model and human judgments. Recent works also assess scalar-valued critiques of LLMs on reasoning and coding tasks [24, 9]. For example, CRITICBENCH [9] built from 3 reasoning tasks, analyzes important properties of critique ability of LLMs. Our concurrent work, CRITICBENCH [4] analyzes several intriguing findings among generation, critique and correction capability on responses collect from five reasoning tasks. 3 Compared with these existing works, our proposed CRITICEVAL exhibits advantages on several crucial factors, like critique dimensions, response qualities and diverse task types, leading to more comprehensive evaluations for critique ability. Although CRITICBENCH [9] collect high-quality responses, their quality levels are still unclassified.

Beyond scalar-valued critiques, evaluating textual critiques is more challenging [10, 15]. Most existing works coarsely evaluated textual critiques using GPT-4 [11, 12], proven unreliable [10, 13]. Unlike them, our extensive results prove that GPT-4 with human-annotated critiques is reliable for evaluating textual critiques. Although our concurrent work, MetaCritique [13], demonstrates the reliability of evaluating textual critiques by verifying their Atomic Information Units, it is unclear whether their conclusions could be extended to more critique dimensions and tasks.

3 Preliminaries

We first formally define the key concepts and their corresponding notions in CRITICEVAL. Figure 1 shows a specific case to understand these concepts.

Task Input ( I ) and Response ( R ) represent the queries and generations of LLMs, respectively.

Critique aims to analyze and refine the generated responses. Formally, this paper studies the critique capabilities in four dimensions: (1) feedback F s involves textual analysis and a quality score. Good feedback should not only find flaws in responses but also provide helpful suggestions for correction [6]; (2) correction or refinement CR aims to revise responses with or without feedback. Previous evaluations [11, 12, 9] overlook this dimension, although it is an inevitable step when letting the model improve itself [35]; (3) comparison F c contains a textual critique and a preference label for a pair of responses ( R a , R b ) ; (4) Meta-feedback F s ( F s ) , i.e. , the feedback of feedback itself [6], involves a rating score reflecting the quality of F s and corresponding textual analysis, which is a high-level critique dimension. Such an ability is necessary to improve critique ability of LLMs [1, 13]. Due to the complexity of the meta-feedback dimension, textual critiques are not collected in this paper, and we leave it for future research.

Compar-

ision

Response B is more helpful than A ... On the contrary

,response A only claims it cannot solve this input .

B

/AB

Feeback

ion

Generated response fails to fulfill user

'

s query

... Following suggestions could be considered ...

Here is the revision for response A: Of course! To

make a pizza,the following ingredients are needed: ...

Please give me a

recipe for pizza

Sure,here is a

recipe for pizza: ...

I do not know

any recipe ...

Response A

Response B

Meta

Feeback

8

/10

The feedback point out some errors ...

It also provides suggestions ...

1

/10

To the best of our knowledge, these four critiques cover all categories of critiques examined in prior research [11, 4, 13]. Although the feedback of correction and comparison dimensions are also important, they are not essential for the self-improvement of LLMs. Thus, this study mainly focuses on studying the feedback of feedback F s ( F s ) and leaving the rest of them for our future work.

4 CRITICEVAL Construction

Given the challenge of crafting scalar-valued and textual critiques from scratch, we construct CRITICEVAL using a human-in-the-loop data construction pipeline as shown in Figure 2. 4

4.1 Task Input Collection

Task inputs for 9 distinct tasks are collected to evaluate critique capabilities comprehensively (Step 1 in Figure 2). Specifically, CRITICEVAL includes three widely used tasks for evaluating critique ability: (1) representative classical language tasks: summary [39], translation [40], and question-answering [41]; (2) LLM alignment: general chat scenarios [19] and harmlessness cases [35]; (3) reasoning and code capabilities: math reasoning with chain-of-thought (CoT) and program-of-thought (PoT), and coding with and without execution results. We hereinafter refer to 'code w/ execution' as 'CodeExec' and 'code w/o execution' as 'CodeNE'. For each task, we collect around 100 task inputs from the test sets of some widely used benchmark datasets to ensure the task input quality and avoid data contamination. Please refer to Appendix D for more details about the data source.

4.2 Response and Critique to be Evaluated

For each collected I in each task, LLMs of different scales and capabilities are first employed to generate responses with diverse flaws (Step 2 (a) in Figure 2). The complete list of LLMs is in Appendix E. Then, low-, medium-, and high-quality responses with diverse quality differences are collected according to the quality score annotated by the human raters with GPT-4-turbo as the assistant (Step 2 (b)). Moreover, we also collect golden or correct responses, which have been proven challenging for critiques [15]. More details about how to select low-, medium, high-quality and correct responses can be found in Appendix F.

After collecting responses, we further collect critiques to be evaluated for the meta-feedback dimension by utilizing four LLMs that are known powerful for critiques (Step 3 (d) in Figure 2): (1) GPT-4; (2) GPT-3.5-turbo; (3) Auto-J-13B [11]; (4) UltraCM-13B [12].

4.3 Reference Critique Generation and Annotation

After collecting task inputs and responses, four kinds of reference critiques are collected on these responses to make the objective and subjective evaluation in our proposed CRITICEVAL more reliable.

Feedback and Correction GPT-4-turbo is utilized to generate feedback and corrections sequentially (Step 3 (c) and (e) in Figure 2). The scalar-valued and textual critiques for feedback dimension

revise

中

A

Harm-

lessness

General

Chat

2+1=?

Math

PoT

(x)

Code

$_

$_

Pool of Large-scale

Language Models

Translation

Generation Tasks

Response Qualities

Critique Dimensions

CRITICEval

LLMs and Diverse

Task Input Preparation

Responses Generation

and Human Re-check

Critiques Generation

and Human Annotation

(a)

LLM

Response

Uneven-quality

Responses

Human Experts

Check the

Generated Response Ratings

(c)

Feeback

(b)

LLM

Rating

1 by 1

score

(f)

Compar

-

ision

(d)

Meta

Feedback

Human Experts

All Above Generated Scalar (    )

or Textual (       ) Critiques

text

input

input

preferred

label

text

inputs

pairs

inputs

task

inputs

Question

Answering

Medium

-quality

Response

Correct

Response

Low-quality

Response

Summary

Math

CoT

Code with

Execution

score

(e)

ion

High-quality

Response

cross

our

bench

bench

our

Step

Step

Step

XN

Q

xN

L

SN

are collected, denotes as 'score' and 'text' in Figure 2. Since responses in math reasoning and coding tasks pose significant challenges for critiques during our annotation, ground-truth answers are provided for GPT-4 as references to generate high-quality feedback and corrections. Then, they are carefully reviewed and revised by human annotators.

Comparison Our empirical finding suggests that pairs of responses pose greater challenges for comparison if they perform similarly. Therefore, two kinds of pairs are first created: ( R low , R high ) and ( R med , R high ) , designated as the easy and hard samples, respectively. Then, GPT-4-turbo is prompted to provide scalar-valued and text critiques on these pairs (Step 3 (f) in Figure 2). These outputs, labeled as 'preferred label' and 'text' in Figure 2, are then refined by human annotators.

Meta-Feedback Since GPT-4 has been proven unreliable to evaluate critiques [10, 15], three human experts are asked to provide their quality scores for generated critiques (Step 3 (d) in Figure 2).

During human annotation, multiple human experts are asked to follow a rigorous annotation protocol, detailed in Appendix H.1, and the statistics of human annotation for reference critiques are described in Appendix H.6. Besides, several case studies are shown in Appendix I to facilitate a clear understanding of our proposed CRITICEVAL.

5 Evaluation Metrics

5.1 Objective Evaluation

Feedback and meta-feedback evaluation aim to evaluate the consistency between generated scores and human judgments. This setup facilitates the generation in a chain-of-thought manner, followed by the quality score of the evaluated critiques. For the meta-feedback dimension, LLMs are prompted with annotated reference critiques. The widely-used Spearman correlations [42] are computed [23, 24], which ranges from -1 to 1 (normalize to ( -100 , 100) ). Higher scores indicate a higher consistency with human judgments. The p -value of spearman correlation are recorded, and p < 0 . 05 is typically considered to be statistically significant [23, 43].

Comparison evaluation assesses the accuracy of LLM in deciding preferences between two responses. It is well known that current LLMs exhibit significant positional bias [17, 44, 45], i.e., LLMs tend to prefer responses based on their specific position in the prompt. We implement a rigorous verification process to mitigate the effects of positional bias. Specifically, given responses R a and R b to be compared, we obtain the comparison based on two orders, noted as F a c = F c ( R a , R b )

̸

and F b c = F c ( R b , R a ) . The objective scores are computed by: s = 1 N ∑ N i =1 1 ( L ( F a c , F b c )) , where 1 → { 0 , 1 } is the indicator function. L ( F a c , F b c ) is true if and only if F a c = F b c and F a c , F b c align with ground-truth preference label. N is the number of test samples.

Correction evaluation is only conducted on math reasoning and coding tasks since the revision could be easily verified with the ground-truth answers and the test cases [4, 9]. Thus, the objective evaluation metric is implemented as the pass rate: N Pass N , where N and N Pass are the number of the total samples and passed samples, respectively.

5.2 Subjective Evaluation

The subjective evaluation aims to evaluate the quality of the generated textual critiques. Since responses in math reasoning and coding tasks can be verified, we only conduct subjective evaluations on other 5 tasks for the correction dimension. In our work, GPT-4 evaluates the generated critiques by generating the chain of thought followed by the score, with our human-annotated critiques as references. It is well-known that LLMs prefer longer generations during their automatic evaluation [46, 45]. However, Figure 7 in Appendix J proves that there is no clue that GPT-4, with our concise and precise reference critiques as input, would give higher scores to longer critiques. The subjective scores range from 1 to 10 . Following previous work [20], the human-annotated reference critiques are anchored to 8 , serving as a relative scoring pivot.

5.3 Overall Score

The overall score of subjective and objective evaluation is computed as averaging on all the critique dimensions, and more details about computing the overall score can be found in Appendix K.

6 Evaluation and Analysis

The critique abilities of representative LLMs are analyzed in this section, and the overview results are shown in Table 2. Firstly, the reliability of evaluation in CRITICEVAL are proven in Section 6.2

and Section 6.3. Then, overall analysis is described in Section 6.4. Furthermore, several intriguing phenomena about some critical factors are described, including task types (Section 6.5), the response quality (Section 6.6) and the critique dimensions (Section 6.7). Finally, we elaborate and analyze the fine-grained error patterns of model-generated critiques in Section 6.8. The complete experimental results of all evaluated LLMs on the test/dev set for each task and each critique dimension are placed in Appendix L.

6.1 LLMs to be Evaluated

35 widely used open-source and closed-source LLMs of different sizes are evaluated on CRITICEVAL, including the instruction-tuned LLMs [21, 50, 22, 51], critique-tuned LLMs that fine-tuned on critique datasets generated by GPT-4 [11, 12, 30], and reward models [12, 52, 53]. Please refer to Appendix N for all evaluated LLMs and the inference details. The prompt templates for LLMs on critique dimensions are shown in Appendix I with score rubrics listed in Figure 18 in Appendix H.3.

6.2 Reliability of Subjective Evaluation in CRITICEVAL

As LLMs are prompted with human-annotated critiques, their performance in meta-feedback could reveal their reliability for evaluating generated textual critiques. As shown in Table 2 ( F s ( F s ) column) and Table 3, GPT-4-turbo achieves very high correlations ( 62 . 90 , and 66 . 18 ) with human judgment. Although there is still a gap compared to the average human level ( 66 . 18 < 79 . 03 ), the strong correlations ensure the reliable evaluation for tex-

Table 3: Results of meta-feedback dimension in CRITICEVAL dev set. p -value < 0 . 05 .

tual critique ability [23]. Furthermore, we also conduct the ablation study to prove the contribution of our human-annotated reference critiques. As shown in Table 3, it can be found that all LLMs perform worse when the reference critiques ( ref. ) are removed (average 13 . 36 performance decrease), proving their significant contribution for reliable subjective evaluation in our proposed CRITICEVAL.

Moreover, except for the feedback critique dimension, we also test the reliability of subjective evaluation on the correction and comparison critique dimensions. Specifically, we ask three human annotators to annotate the quality score of 450 critiques generated by five representative LLMs (GPT-3.5-turbo, Qwen72B-Chat, InternLM2-20B-Chat, Mistral-7B and ChatGLM36B) from 9 tasks in CRITICEVAL, and all the human annotators

Table 4: Correlations in CR and F c dimensions. p -value < 0 . 05 .

are guided by the same evaluation protocol in our subjective evaluation. The results are shown in Table 4. It can be found that GPT-4-turbo, with our human-annotated critiques as references, could achieve a very strong correlation with human judgments, close to the average human level. This observation proves the robust and reliable subjective evaluation of the textual critiques in the correction and comparison dimensions. Besides, the correlation scores on the correction and comparison critique dimension are higher than the feedback dimension. This phenomenon suggests that the feedback of the feedback is more challenging than the feedback of correction and comparison.

6.3 More Effective Critiques Consistently Lead to Superior Corrections

Although the reliability of subjective evaluation has been proven in Section 6.2, it is still unknown whether real feedback critique ability of LLMs is consistent with the evaluation results in CRITICEVAL . To explore this, we prompt the InternLM2-20B-Chat and Llama2-70B-Chat models to revise responses from CRITICEVAL using three sources of feedback with varying quality levels. As

illustrated in Table 5, a clear and consistent trend emerges: as the quality of the feedback increases, both the objective and subjective revision performance improves. This finding underscores that real critique ability of LLMs aligns closely with the evaluation results in our proposed CRITICEVAL, i.e., critiques of LLMs with higher scores are more accurate and effective for corrections.

In summary, the reliability of evaluation in CRITICEVAL has been well proven. Following sections will describe the overall analysis and relationships between critique ability and several crucial factors.

6.4 Overall Analysis of LLMs

As shown in Table 2, GPT-4 significantly outperforms other LLMs on most critique dimensions, while slightly underperforms our human-annotated critiques ( 7 . 81 < 8 ). Surprisingly, open-source LLMs are approaching state-of-the-art closed-source LLMs. For example, InternLM2-20B-Chat surpasses GPT-3.5-turbo on overall subjective scores ( 6 . 20 > 5 . 89 ). Furthermore, there is a clear relationship that the critique ability of LLMs improves steadily as the number of parameters increases (Table 2), suggesting that the critique ability of LLMs highly correlates with their capability.

Table 6: Critique-tuned LLMs results in feedback dimension.

We also provide a clear diagram to show this relationship in Figure 14 in Appendix R. Beyond the average scores, we also categorize the textual critiques of LLMs into multiple quality intervals for more interpretable analysis, which are described in Appendix O.

The results of critique-tuned LLMs in the feedback dimension on the test set are shown in Table 6. It can be found that critiquetuned LLMs fine-tuned from Llama-2-13B significantly outperform even the Llama-2-70B-Chat model, proving the effectiveness of critiques datasets [11, 12]. The results of representative reward models are shown in Table 7. From these results, it can be found that reward models like UltraRM-13B achieve impressive performance in scoring the quality of responses, significantly

Table 7: Reward model objective results in F s and F c dimensions.

outperforming GPT-3.5-turbo. This observation aligns with findings in recent works [14].

6.5 Relationship with Task Type

Effective critiques usually require domain knowledge and understanding of given tasks. We analyze the relationship between critique ability and task type in Table 8, which shows the average performance of all evaluated LLMs.

Feedback, Comparison LLMs achieve much higher scores in the first five tasks than on math reasoning and coding tasks, indicating math reasoning and code tasks are more challenging.

Meta-Feedback LLMs achieve much higher consistency with human judgments on code and math reasoning tasks, indicat-

Table 8: Two Avg. rows represent the average scores of all LLMs on the first 5 tasks and the last 4 tasks.

ing that evaluating textual critiques in math reasoning and code tasks is more reliable.

Correction Math reasoning tasks are more challenging than coding tasks, and CodeExec is easier to revise than CodeNE due to the richer information in execution results. Except for math reasoning and coding tasks, the translation is the most challenging task because professional domain knowledge is required, while harmlessness is the easiest to refine since most LLMs have been trained to avoid harmful generations [35]. Furthermore, we explore the variance in correction quality on reasoning and coding tasks ( Obj. in Table 9)

Table 9: Average performance of evaluated LLMs on test set.

and other subjective tasks ( Sub. in Table 9). 5 Specifically, three kinds of feedback are used for correction: (1) H uman-annotated F eedback ( HF ); (2) E mpty F eedback ( EF ), where LLMs are prompted to

improve responses without any feedback; and (3) LLMs S elf-generated F eedback ( SF ). As shown in Table 9, it can be found that self-generated feedback is beneficial to corrections on subjective evaluation (HF (7 . 12) > SF (5 . 48) > EF (5 . 16) ), while it might negatively affect corrections on objective evaluation of math reasoning and coding tasks (SF (13 . 01) < EF (14 . 44) < HF (43 . 66) ). This observation proves that LLMs struggle in self-improvement on challenging reasoning tasks, aligning with recent findings [54, 4].

6.6 Relationship with Response Quality

Before analyzing the relationship between response qualities and critique ability, it is essential to categorize the error patterns in responses. We highlight that the error patterns are related to the task type, complicating the classification of errors. To conduct a representative analysis of errors in all tasks, human annotators are asked to categorize errors into three patterns, which collectively encompass nearly

all the cases: (1) Obvious error is easy to critique and correct, like apparent misuses of words in translation task; (2) Complex error is challenging to correct, regardless of whether critiques are easy to critique, like logical reasoning error in reasoning tasks; (3) Subtle error is hard to critique, while it is usually easier to revise than complex error, like slight misunderstandings of context in general chat. The distribution presented in Table 10 reveals distinct primary errors across different response qualities. More details about these error patterns in each task are described in Appendix P.

Given the distribution of error patterns, we analyze critique ability of LLMs on responses with varying qualities. As shown in Table 11, highquality responses are the hardest for feedback since they contain lots of subtle errors (Table 10). Note that the medium-quality responses have higher objective feedback scores than lowquality ones, which is inconsistent with our expectations. This phenomenon is because low-

quality responses often receive very low human-annotated quality scores (near 1), while the scoring of LLMs tends to be higher, leading to a discrepancy. For the correction dimension, low-, and high-quality responses are easier to correct than medium-quality due to the most obvious and subtle errors. There are two kinds of qualities for comparison dimension: easy and hard. Most LLMs perform better on easy samples than on hard samples. Specifically, the subjective and objective scores of easy samples are 4.78 and 39.73, respectively, higher than those of hard samples (4.55 and 29.80). For the meta-feedback dimension, LLMs achieve the highest consistency with human judgments on high-quality responses while performing worst on medium-quality responses.

6.7 Relationship with Critique Dimensions

The average scores of all evaluated LLMs on different critique dimensions are shown in Table 12. Objective scores of comparison and correction are not recorded because they are not correlations. Several conclusions can be made: (1) correction is the easiest critique dimension, followed by feedback, and then comparison. This observation demonstrates that comparison requires accurate analysis of both responses, which is more complex than the feedback dimension; (2) As a high-level critique dimension, meta-feedback is more challenging than the feedback.

6.8 Fine-grained Failure Modes in Model-Generated Critiques

This section analyzes the fine-grained failure modes in model-generated critiques across feedback, comparison and correction dimensions. As illustrated in Table 13, human annotators summarize the 12 main failure modes in model-generated critiques. Then, we compute the distribution of these failure modes of all evaluated LLMs. Figure 3 demonstrate that the most frequent failure modes are missing errors (E1, E2), lacing effective comparison analysis (E7) and worse revision than references (E10) for feedback, comparison and correction dimensions, respectively. Furthermore, as shown in Figure 4, it can be observed that missing errors/suggestions (E1, E2) and inaccurate critiques (E3, E4, E8) usually lead to lower subjective scores.

Table 12: Average performance on test set.

















0

5

10

15

20























0

5

10

15

20











0

5

10

15

20

25

30

35





















3.0

3.5

4.0

4.5

5.0























3.00

3.25

3.50

3.75

4.00

4.25

4.50

4.75

5.00











3.0

3.5

4.0

4.5

5.0

5.5

6.0

6.5

7 Conclusion and Future Work

In this paper, we introduce a comprehensive and reliable benchmark for evaluating the critique abilities of LLMs, named CRITICEVAL. Extensive experimental results first prove the reliability of CRITICEVAL, and reveal the promising potential of open-source LLMs, the effectiveness of critique datasets and intriguing relationships between critique capabilities and some factors: task types, response qualities and critique dimensions. These observations significantly promote an in-depth understanding of the critical ability of LLMs and LLM's self-improvement. In the future, we plan to enhance our benchmark in several key areas: (1) Broadening the scope to include more tasks, such as tool-using; (2) Extending the benchmark to encompass other languages, like Chinese; (3) Improving the subjective evaluation protocol to allow for more fine-grained analysis; (2) Continue to evaluate LLMs and track their critique ability, like Llama-3 models; (5) Improving the quality of reference critiques by incorporating additional high-quality critiques from advanced LLMs if and only if their quality surpasses the existing reference critiques.

Acknowledgments and Disclosure of Funding

First, the authors would like to express their sincere gratitude to all the anonymous reviewers and meta-reviewers for their insightful comments and constructive feedback. Besides, we would also like to thank many senior researchers for their valuable comments before our submission, which greatly improved the quality of our paper: Leyang Cui, Yan Wang, Yong Hu, Rongcheng Tu, Hongli Mao, Fanshu Sun and Chen Xu. The names are listed in no particular order. Furthermore, this project was funded in part by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is a PI of CPII under the InnoHK. This work was also supported in part by the National Natural Science Foundation of China (No. 62172039, U21B2009, 62276110, and 62450100), the MIIT Program (CEIEC-2022ZM02-0247), the Postdoctoral Fellowship Program of CPSF (No. GZC20233403), and the China Postdoctoral Science Foundation (No.2024M764142).

References

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.

Weiwen Xu, Deng Cai, Zhisong Zhang, Wai Lam, and Shuming Shi. Reasons to reject? aligning language models with judgments, 2023.

Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies, 2023.

Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations , 2024.

Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil˙ e Lukoši¯ ut˙ e, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, and Jared Kaplan. Measuring progress on scalable oversight for large language models, 2022.

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022.

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André FT Martins. Results of wmt22 metrics shared task: Stop using bleu-neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 46-68, 2022.

Alexander R Fabbri, Wojciech Kry´ sci´ nski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics , 9:391-409, 2021.

Liangchen Luo, Zi Lin, Yinxiao Liu, Lei Shu, Yun Zhu, Jingbo Shang, and Lei Meng. Critique ability of large language models, 2023.

Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane DwivediYu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Shepherd: A critic for language model generation, 2023.

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Representations , 2024.

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.

Shichao Sun, Junlong Li, Weizhe Yuan, Ruifeng Yuan, Wenjie Li, and Pengfei Liu. The critique of critique, 2024.

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535 , 2024.

Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. Self-contrast: Better reflection through inconsistent solving perspectives, 2024.

Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. Criticbench: Benchmarking llms for critique-correct reasoning, 2024.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, 2023.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval , 2023.

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. Alignbench: Benchmarking chinese alignment of large language models, 2023.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.

Tian Lan, Xian-Ling Mao, Wei Wei, Xiaoyan Gao, and Heyan Huang. Pone: A novel automatic evaluation metric for open-domain generative dialogue systems. ACM Trans. Inf. Syst. , 39(1), nov 2020.

Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. INSTRUCTSCORE: Towards explainable text generation evaluation with automatic feedback. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 5967-5994, Singapore, December 2023. Association for Computational Linguistics.

Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. Dissecting human and llm preferences, 2024.

Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization, 2023.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.

Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. Critiquellm: Scaling llm-as-critic for effective and explainable evaluation of large language model generation, 2023.

Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore: Towards building explainable metric for all text generation tasks. arXiv preprint arXiv:2310.00752 , 2023.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR) , 2023.

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation , pages 1066-1083, Singapore, December 2023. Association for Computational Linguistics.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.

Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, Jie Tang, and Yuxiao Dong. Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline, 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS , 2020.

Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André F. T. Martins. Findings of the WMT 2020 shared task on quality estimation. In Loïc Barrault, Ondˇ rej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri, editors, Proceedings of the Fifth Conference on Machine Translation , pages 743-764, Online, November 2020. Association for Computational Linguistics.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP , 2018.

Jerrold Zar. Spearman Rank Correlation , volume 5. 07 2005.

Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In AAAI Conference on Artificial Intelligence , 2017.

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023.

Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations , 2024.

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. Benchmarking foundation models with language-model-as-an-examiner. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM , 2023.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier,

Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.

DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 , 2024.

OpenAI. Gpt-4 technical report, 2023.

Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhongshen Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing Zhang. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. CoRR , abs/2209.02970, 2022.

Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V -usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 59886008. PMLR, 17-23 Jul 2022.

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations , 2024.

Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models, 2023.

Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. Kilt: a benchmark for knowledge intensive language tasks, 2021.

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's song in the ai ocean: A survey on hallucination in large language models, 2023.

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787 , 2024.

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.

Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. A data-driven analysis of workers' earnings on amazon mechanical turk. In Proceedings of the 2018 CHI conference on human factors in computing systems , pages 1-14, 2018.

Giulio Zhou and Gerasimos Lampouras. Webnlg challenge 2020: Language agnostic delexicalisation for multilingual rdf-to-text generation. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+) , pages 186-191, 2020.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv , abs/2110.14168, 2021.

Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning, 2023.

Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. Openmeva: A benchmark for evaluating open-ended story generation metrics. arXiv preprint arXiv:2105.08920 , 2021.

François Mairesse, Milica Gasic, Filip Jurcicek, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , pages 1552-1561, 2010.

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. arXiv preprint arXiv:1911.03705 , 2019.

Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. FLASK: Fine-grained language model evaluation based on alignment skill sets. In The Twelfth International Conference on Learning Representations , 2024.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence , 2020.

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. ACL , 2017.

Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2357-2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. ACL , 2022.

Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A theorem-driven question answering dataset. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7889-7901, Singapore, December 2023. Association for Computational Linguistics.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents, 2023.

Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs, 2024.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys , 55(12):1-38, March 2023.

Rajani Nazneen, Lambert Nathan, Sheon Han, Wang Jean, Nitski Osvald, Beeching Edward, and Tunstall Lewis. Can foundation models label data like humans? Hugging Face Blog , 2023. https://huggingface.co/blog/llm-v-human-data.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023.

LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy , 2023.

Baichuan. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305 , 2023.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations , 2024.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2024.

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations , 2023.

A Limitations

A.1 Sub-optimal Reference Critiques

Following previous work [20], our work construct CRITICEVAL with the human-in-the-loop annotation pipeline, i.e., multiple human annotators are asked to review and revise the critiques generated by GPT-4 model. Even though we have established a rigorous annotation process to ensure the high quality of annotated critiques, human annotators are inevitably influenced by GPT-4's initial generated critiques in some more open-ended tasks, like general chat and QA tasks. This problem may result in the quality of revised critiques by human annotation still being sub-optimal. To address this problem, we plan to enhance CRITICEVAL in our next version. Specifically, we will replace existing reference critiques with potential better critiques generated by advanced evaluated LLMs, like Claude-3 6 , if and only if their qualities surpasses reference critiques.

A.2 GPT-4 Model for Subjective Evaluation

CRITICEVAL mainly utilizes the advanced GPT-4-turbo model for subjective evaluation. Despite integrating high-quality annotated reference critiques to guide GPT-4 toward more accurate assessments, it's essential to acknowledge that the model's evaluations may not always align perfectly with human judgment. While GPT-4 have yet to reach the level of precision of human annotation - they currently represent the most effective approach for balancing the trade-offs between evaluation cost and quality. It is still a significant challenge to accurately and automatically evaluate critiques across all scenarios. Recognizing this, we aim to address these issues in our future work by progressively refining our benchmark and evaluation protocols.

A.3 More Tasks for Critique

Compared with existing benchmarks [12, 10, 14, 13], our proposed CRITICEVAL exhibits significant advantages in the diversity of evaluated task scenarios. Although we strive to cover a wide range of diverse generation tasks, there are still some tasks have yet to be considered, such as tool learning [55], knowledge-intensive tasks [56] and hallucination [57]. In our future work, we will continue to include more tasks in the next version of CRITICEVAL.

A.4 More LLMs to be Evaluated

Some newly proposed LLMs and reward models [58] have not been added yet, like Llama-3 7 and Claude-3 series models. Since our conclusions are summarized from evaluation results of over 35 open-source and closed-source LLMs, lacking these LLMs does not affect conclusions. We will continue to evaluate their critique ability in our future work.

A.5 Limited Inference Strategy

In this paper, all the evaluated models generate the critiques by using the greedy-search decoding method. There exist some inference strategies to potentially improve the model's performance, like structured generation [59]. The primary goal of our proposed CRITICEVAL in the current stage is to construct a comprehensive and reliable benchmark for evaluating the critique ability of LLMs, and we will explore these inference strategies to improve the critique ability in our future work.

B Ethical Considerations

Most task inputs in CRITICEVAL are collected from publicly available datasets, free from any possible harm toward individuals or groups. Moreover, humans carefully select and process the responses and critiques generated by LLMs to secure privacy and confidentiality. No personal identification information is involved. However, it should be noted that the task input, responses, and critiques in the Anthropic-HHH dataset [60] of the harmlessness task contain harmful materials and hate

speech. Despite the risks involved, it is essential to disclose this research fully, and materials in the Anthropic-HHH dataset have been widely used for safety research in the LLM community. All raters have been paid adequate wages. The hourly wage of our human annotators is about 5.69 USD, which is much higher than average hourly wage 3.13 USD on Amazon Mechanical Turk [61].

C Comparison and Statistics of CRITICEVAL

The comparison between CRITICEVAL and existing benchmarks can be found in Table 14, which proves the advantages of our proposed CRITICEVAL for critique evaluation. Compared with existing benchmarks for critique evaluation, our proposed CRITICEVAL contains 3 , 608 textual natural language critique samples (textual critiques) that are well annotated by multiple human annotators, leading to a more stable and reliable assessment in our subjective evaluation. The scale of objective evaluation data (scalar data) in our dataset is second only to Chat Arena 8 and SummEval [8]. However, compared to these two datasets, our dataset contains a more diverse range of critique dimensions and tasks (nine diverse tasks). Moreover, the statistics of CRITICEVAL in the test and dev set are shown in Table 15.

D Source Data for Different Tasks

The benchmark includes three representative classical language tasks: summary [39], translation [40], and question-answering [41]. Since a popular application of LLMs is to serve as a chatbot, where alignment is important to ensure the safe application of LLMs, we collect instructions from general chat scenarios [19] and harmlessness cases [35] to evaluate the LLMs' critique ability for alignment. Furthermore, the reasoning and code capabilities are also fundamental for augmenting LLMs as agents [78], another important and promising application of LLMs. Thus, we also collect

instructions for math reasoning with chain-of-thought and program-of-thought, and coding with and without execution results.

To ensure the difficulty of CRITICEVAL, we only collect coding and math reasoning questions that some 70B LLMs cannot correctly answer, which is proven effective in previous works [9]. Our motivation is to collect questions that could easily raise responses with diverse flaws. Simple questions pose challenges for us in achieving this goal since most LLMs can easily solve them. Collecting these questions that 70B LLMs cannot answer correctly makes the difficulty of questions become moderate or complex, which aligns with our motivation.

The details of selected datasets for 9 tasks are listed in Table 16, covering

Table 16: Source of 9 tasks in CRITICEVAL. Most tasks contain diverse samples from multiple test sets.

the well-known NLP tasks (translation, summary, and question answering), reasoning tasks (mathematics and coding), and alignment (general chat and harmlessness). These datasets' test sets are used for CRITICEVAL construction, avoiding data contamination. For each task, we collect around 100 instructions from the test sets of some widely-used benchmark datasets to ensure the instruction quality and avoid data contamination.

Our dataset is under Apache 2.0 License.

E List of Used LLMs for Response and Critique Generation

Our study uses several LLMs with different capabilities to generate diverse feedback, listed in Table 17. Besides, we also use some critique-tuned LLMs to generate textual feedback, like Auto-J-13B and UltraCM-13B models.

F Details of Responses Generation

F.1 How to Collect Low-, Medium-, High-quality Responses

To identify the quality of these responses efficiently, GPT-4 is utilized to initially assign quality ratings ranging from 1 to 7 (Step 2 (b) in Figure 2) then let human annotators meticulously review and adjust these scores, which are used in the objective evaluation in the feedback dimension (Section 5.1). then, three responses with distinct quality differences for each I are sampled based on their human-verified quality scores, including low-, medium-, and high-quality responses (noted as R low , R med , R high , respectively). Due to partial simple or hard queries, there might be queries where the scores of

three responses are close. However, there is a distinct quality difference between low, medium, and high-quality responses overall.

The statistical of responses' quality scores on 9 tasks can be found in the Figure 5. Figure 5 demonstrates the discernible performance disparities in responses for each task. Since automatic execution leaks quality information, we do not collect the correct responses for the CodeExec task. Such variation is instrumental in analyzing the impact of response quality on the feedback.

F.2 Correct or Golden Response Generation

Golden or correct responses are collected for each task input I , which are proven challenging for critiques [10, 15]. We use GPT-4 to generate correct responses using ground-truth rationales or codes as hints for coding and mathematical tasks. Since executions leak information about













































response quality, correct responses are not collected for the CodeExec task. In tasks beyond coding and mathematics, GPT-4 is prompted to refine its past generations, given its feedback during multiple turns, and the last revision is collected as golden response.

G Reasons of Utilizing Human-in-the-Loop Data Generation Pipeline

We construct the high-quality reference critiques by using human-in-the-loop data generation pipeline, which is motivated by two essential considerations (effectiveness and efficiency).

Human-written Critiques are Usually Insufficient - Sub-optimal (Effectiveness) Our trial human annotation reveals that human annotators might neglect some apparent or severe issues when writing critiques from scratch, consistent with findings in recent studies from OpenAI [79, 6]. Our experimental result in Appendix M demonstrate that neglecting issues usually leads to low-quality critiques. In contrast, despite the possibility of generating wrong critiques, LLMs like GPT-4 offer more comprehensive and detailed critiques [79]. By revising LLM's errors by human experts, the final critiques could be more comprehensive and accurate, leveraging the strengths of both human annotators and LLMs.

Annotating Challenging Critique Task from Scratch Cost A Lot (Efficiency) Writting critiques from scratch is a significant challenge [20, 10]. For instance, Shepherd [52] incurred an annotation

Domain

cost of 8$ per sample, leading to over 28,864$ and 1,350 work hours to annotate the entire CRITICEVAL, which is unbearable for our project. Thus, we have to employ advanced LLMs to generate draft critiques, followed by human annotation. GPT-4 was chosen because our preliminary studies indicate that it is the most reliable LLM for producing draft critiques, while other LLMs are much worse. Consequently, a diverse set of LLMs introduces more noise in generated critiques, bringing more difficulties to human annotators.

In conclusion, the human-in-the-loop pipeline achieves the trade-off between annotation cost and quality. We promise to add these details to the revised paper to emphasize the motivation of using the human-in-the-loop pipeline.

H Human Annotation Details

H.1 Evaluation Protocol

Three to five human annotators annotate each scalar-valued critique in each task. Biases among human annotators may arise from factors such as the annotator's gender and professional background. To minimize the impact of annotators' biases on the quality of our human annotations, we first selected a diverse group of annotators from a crowd-sourcing platform to annotate each critique sample collaboratively. All human annotators have been paid adequate wages. The hourly wage of our human annotators is about 5.69 USD, which is much higher than average hourly wage 3.13 USD on Amazon Mechanical Turk [61].

Before annotation, we designed a rigorous data annotation verification process to iteratively train these annotators, thereby ensuring the stability and reliability of the annotation quality. During annotation, these human annotators first annotate each textual critique, which are summarized by another supervisor annotator. After annotation, the supervisors (authors in our paper) conducted a 5% sample inspection. If the error rate exceeds the threshold, annotators are asked to revise their work until the error rate is lower than the threshold.

H.2 Inspect Errors in Datasets

It should be noted that some underlying datasets contain inaccuracies that may lead to compounding effects during evaluation. For example, we have noticed some incorrect solutions and rationales for mathematics and coding questions during our human evaluation process. In our work, to mitigate the effects of such errors, human annotators are asked to meticulously examine each question, the provided golden answers (only for mathematics and coding tasks) and the evaluated responses and critiques. They are asked to exclude instances where the golden answers or questions are flawed or incorrect, like wrong solutions to mathematics and coding questions.

H.3 Score Rubrics for Different Tasks

The annotators are entrusted with the detailed score rubrics to evaluate the different dimensions [14]. Table 18 lists the score rubrics designed for different tasks. Note that math and code tasks only need to check the correctness.

H.4 Internet Search

Task inputs in the QA and chat tasks often require specific factual knowledge for responses. However, GPT-4 sometimes produces spurious knowledge or fails to effectively identify factual inaccuracies within these responses, a common issue known as hallucination [80, 57]. Consequently, we strongly urge human annotators involved in the feedback, comparison and correction annotations in QA and general chat tasks to verify factual content through internet searches. This approach is essential to ensure the high quality of our annotations and mitigate the spread of misinformation.

H.5 Exclude Mention of Ground-Truth

To generate correct responses R corr . and critiques for challenging tasks, like mathematics and coding, we provide the ground-truth rationales as reference for GPT-4. Then, these generated responses and

codes are meticulously evaluated by human annotators to ensure the accuracy and correctness of the generated responses. Human annotators are asked to exclude any direct references of ground-truth rationals in these generated R corr . and critiques.

H.6 Statistics of Human Annotation on Reference Critiques

The average revision rates of textual critiques for the feedback, comparison, and correction critique dimensions are 25 . 22% , 34 . 83% , and 48 . 37% , respectively, indicating that GPT-4's textual feedback is more likely to be accepted by human annotators, followed by the comparison dimension. In contrast, GPT-4-turbo's correction might contain more flaws that need to be corrected. Moreover, the average inner correlation score among annotators is 0 . 79 , indicating solid inner agreement.

I Case Study

This section shows some cases in CRITICEVAL, subjective evaluation of feedback, comparison and correction critique dimensions.

I.1 Examples in CRITICEVAL

In this subsection, we have provided some cases about four critique dimensions: (1) feedback; (2) comparison; (3) correction; and (4) meta-feedback. From these cases, it can be found that samples in our proposed CRITICEVAL contains rich metadata. For example, the data source, quality, and used LLMs are listed.

I.2 Case Study Proves Reference Critique

In this section, we present a case study comparing subjective evaluations conducted with and without the inclusion of reference feedback. Figure 6 illustrates that, without reference feedback, GPT-4 erroneously attributed very high scores to its generated feedback. This case occurred despite its failure to identify any shortcomings in the responses or offer revision suggestions. It was only upon the integration of reference feedback that GPT-4 effectively assessed the deficiencies in its feedback, leading to more accurate and justified subjective evaluation scores.

I.3 Subjective Evaluation Cases

The case for feedback subjective evaluation is shown in Figure 8. The generated response contains factual errors: 'cool music from 1920 are King of the Jazz Band and In the Mood.'. High-quality reference feedback points out these errors, while Qwen-72B-Chat's feedback fails to find them, leading to a low-quality score 2 . Figure 9 shows the case for comparison subjective evaluation. The case for correction subjective evaluation is shown in Figure 10.

I.4 Objective Evaluation Cases

The case for the feedback dimension is shown in Figure 11, and the case for the comparison dimension is shown in Figure 12.

# Question:

I like to host guests at my home from time to time, and I am gathering recipes of different dishes and

drinks to keep things interesting. I am interested in trying some Danish dishes. Can you give me a

recipe for Koldskål?

Here is a recipe for Koldskål: Ingredients: 4 Quart Jar, 3/4 cup chopped onion 1/2 cup chopped carrot

1/2 cup chopped celery 1 1/2 cups chopped cabbage 1/2 cup chopped mushrooms 1/2 cup chopped

tomatoes 1/2 cup chopped zucchini 1/2 cup chopped ... (multiple dull repetitions)

# Reference Feedback:

The response provided is not satisfactory and contains several issues.

1. Factual Error

: The recipe provided is not for Koldskål, a traditional Danish dish. Koldskål is a

sweet dish typically made with buttermilk, eggs, sugar, vanilla, and lemon, often served with small

biscuits called kammerjunkere. The ingredients listed in the response do not reflect this.

2.

Relevance

:

The response does not answer the user's request for a Koldskål recipe.

Instead, it

provides a list of ingredients that appear to be for a different type of dish entirely.

3.

Intelligibility

:

The response is difficult to read and understand due to the repetitive listing of

ingredients. It's also missing instructions on how to prepare the dish.

4. Logical Error

: The response contains repetition of ingredients, specifically dried apricots and dried

figs, which are listed multiple times. This is not logical in a recipe context.

To improve the response, the model should provide an accurate recipe for Koldskål, including the

correct ingredients and preparation instructions. The model should also avoid repetition of ingredients

and ensure the response is clear and easy to understand.

# Generated Feedback

The recipe for Koldskål is not provided.

-----------------------------------------

# Subjective Evaluation w/. Reference Feedback

The generated feedback is extremely brief and fails to address the multiple issues present in the response.

It only states that the recipe for Koldskål is not provided, which is accurate but not comprehensive.

The reference feedback, on the other hand, provides a detailed analysis of the problems, including

factual errors, relevance, intelligibility, and logical errors. It also offers constructive suggestions for

improvement.

The generated feedback does not mention the factual inaccuracy of the ingredients listed, the irrelevance

of the response to the user's request, the repetitive listing of ingredients, or the lack of preparation

instructions. It also does not provide any guidance on how the response could be improved. Therefore,

the generated feedback is of much lower quality than the reference feedback.

Score: 2

-----------------------------------------

# Subjective Evaluation w/o. Reference Feedback

The generated feedback correctly identifies that the response does not provide a recipe for Koldskål.

The response lists ingredients that are not related to the traditional Danish dish Koldskål, which is a

sweet cold soup typically made with buttermilk, eggs, sugar, vanilla, and lemon. Instead, the response

includes a nonsensical list of ingredients that are repeated multiple times and are more in line with

a vegetable or fruit salad rather than a Danish dessert soup.

The feedback is succinct and directly

addresses the main issue with the response, which is the absence of an actual Koldskål recipe. However,

the feedback could have been more detailed by explaining what Koldskål is and why the ingredients

listed do not correspond to the dish in question.

Score: 8

Figure 6: One case in subjective evaluation of QA task. GPT-4 provides a more reasonable rationale and score given the high-quality feedback as a reference. Generated feedback is brief, cannot point out flaws, and provides rich information for revision, indicating its low quality. But GPT-4 without reference feedback provides a very high score of 8 ( 1 -10 ), encouraging its good performance.

J Analysis about Length Bias in Subjective Evaluation

Following previous works [46, 81], the Likert scores and the average number of unique tokens in generated critiques are shown in Figure 7. It can be found that there is almost no relationship between the number of tokens and the Likert scores generated by GPT-4 in the subjective evaluation of three critique dimensions, owing to the concise and precise reference critiques in the prompt.

K Overall Score Computation

The overall scores of LLMs on these four critique dimensions are calculated to reflect their overall capability. For objective evaluation, the overall scores are the average scores over four critique dimensions, ranging from 0 to 100 . Since Spearman scores range from -1 to 1 , they are first normalized into (0 , 100) before averaging. For subjective evaluation, since meta-









































feedback does not require textual critiques, the overall scores are set as the average of feedback, comparison and correction scores, ranging from 1 to 10 .

# Evaluation Task

You are given:

⋆

a given user query or instruction

⋆

a generated response by a large-scale language model

⋆

a textual feedback analyzes the response

⋆

a reference high-quality feedback as reference

# Given Data

-

###

Question

: What is some cool music from the 1920s?

### Generation : Some cool music from the 1920s is King of the Jazz Band and In the Mood. ### Reference Feedback (Score of 8) : The response contains a factual error. 'In the Mood' is not from the 1920s, but was recorded in 1939 by Glenn Miller. Also, 'King of the Jazz Band' is not a recognizable song title from any era. This could lead to confusion for the user. The model should ensure it provides accurate and recognizable song titles from the 1920s.

### Generated Feedback : The response provides two examples of cool music from the 1920s, which are 'King of the Jazz Band' and 'In the Mood.' However, it would be more helpful if the response could provide additional context or information about these songs or the genre of music they belong to. For example, mentioning that 'King of the Jazz Band' was performed by Paul Whiteman's orchestra, which was one of the most popular jazz bands of the time, or that 'In the Mood' was a hit song by Glenn Miller's orchestra and became an iconic piece of swing music, would make the response more informative and engaging. Additionally, providing a few more examples of cool music from the 1920s would give the reader a broader understanding of the musical landscape of that era. Overall, while the response is relevant and understandable, it can be improved by adding more depth and context.

-

# Very Important Detail!!!

1. If you think the generated feedback is better then reference feedback, you could provide score higher

than 8.

2. please first generate your feedback and explanation for the generated feedback by comparing it with

reference feedback, and then you must generate the score following the format 'Score: x', where x is

the score range from 1 to 10.

----------------------------------------

# Chain-of-thought evaluation by GPT-4: The generated feedback fails to identify the factual inaccuracies present. The reference feedback correctly points out that 'In the Mood' is not from the 1920s but from 1939, and it also notes that 'King of the Jazz Band' is not a recognizable song title. The generated feedback, on the other hand, does not address these issues at all. Instead, it suggests adding more context and examples to make the response more informative, which, while potentially useful, does not correct the fundamental errors. The generated feedback's suggestion to provide additional context about the songs and the era is a good one, but it is based on the assumption that the provided song titles are correct and relevant, which they are not. Therefore, the generated feedback is not helpful in improving the translation's accuracy or fluency and does not align with the high-quality reference feedback.

#

Score: 2

# Evaluation Task

You are given:

⋆

a source sentence in English

⋆

a generated response A by a large-scale language model

⋆

a generated response B by another large-scale language model

⋆

a textual comparison analyzes the response

⋆

a reference high-quality comparison-based feedback as reference

# Given Data

-

###

Translation

: Milhaud also used jazz idioms, as in his Suite for clarinet, violin and piano.

###

Translation A

:

米

哈

伊

尔

德

也

曾

使

用

爵

士

乐

的

音

调

，

如

他

的

长

笛

、

小

提

琴

和

钢

琴

的

组

曲

。

###

Translation B

:

米

尔

豪

德

也

使

用

爵

士

乐

的

习

语

，

如

他

在

长

号

、

小

提

琴

和

钢

琴

的

四

重

奏

中

所

使

用

的

。

###

Reference Feedback (Score of 8)

:

Both translations have inaccuracies. In Translation A, 'clarinet'

is incorrectly translated as '

长

笛

' (flute). In Translation B, 'clarinet' is incorrectly translated as '

长

号

' (trombone). Moreover, in Translation B, 'Suite' is incorrectly translated as '

四

重

奏

' (quartet).

However, Translation A correctly translates 'Suite' as '

组

曲

'.

Considering these, Translation A is

slightly better due to its correct translation of 'Suite'.

###

Generated Feedback

:

Translation A is more accurate than Translation B. The original sentence

mentions 'jazz idioms' which is translated correctly as '

爵

士

乐

的

音

调

' in Translation A. However,

Translation B uses '

爵

士

乐

的

习

语

' which does not convey the same meaning as 'idioms.' Additionally,

the original sentence mentions 'Suite for clarinet, violin and piano' which is translated correctly as '

长

笛

、

小

提

琴

和

钢

琴

的

组

曲

' in Translation A, but incorrectly as '

长

号

、

小

提

琴

和

钢

琴

的

四

重

奏

' in

Translation B. Therefore, my preference would be for Translation A as it is more accurate and faithful

to the original sentence.

-

# Very Important Detail!!!

1. If you think the generated feedback is better then reference feedback, you could provide score higher

than 8.

2. Please first generate your feedback and explanation for the generated feedback by comparing it with

reference feedback, and then you must generate the score following the format 'Score: x', where x is

the score range from 1 to 10.

-----------------------------------------

# Chain-of-thought evaluation by GPT-4:

The generated feedback correctly identifies that Translation A is more accurate than Translation B.

However, it makes a significant factual error by stating that '

长

笛

' is the correct translation for 'clarinet,'

which is incorrect. The correct translation for 'clarinet' should be '

单

簧

管

' or '

黑

管

.' This error is

critical as it misleads the evaluation of the accuracy of the translations. The generated feedback also

correctly points out the mistranslation of 'jazz idioms' in Translation B. However, it fails to mention

that '

爵

士

乐

的

音

调

' in Translation A is also not an entirely accurate translation of 'jazz idioms,' as

'

音

调

' refers more to musical tones rather than idioms or phrases. Additionally, the generated feedback

does not address the error in Translation B where 'Suite' is translated as '

四

重

奏

' (quartet), which

is a significant mistake as it changes the type of musical composition being discussed. The reference

feedback is more accurate as it correctly identifies the mistranslation of 'clarinet' in both responses and

the mistranslation of 'Suite' in Translation B. It also provides a clear comparison that acknowledges

the correct translation of 'Suite' in Translation A, which the generated feedback fails to do. In terms

of intelligibility, both feedbacks are clear and easy to understand. They both maintain relevance by

addressing the accuracy of the translations in relation to the original sentence. Considering these points,

the generated feedback has a factual error and misses a critical translation error, making it less effective

than the reference feedback.

#

Score: 4

# Evaluation Task

You are given:

⋆

a source sentence in English

⋆

a generated translation

⋆

a high-quality feedback for generated translation

⋆

a textual comparison analyzes the response

⋆

a reference high-quality comparison as reference

Please carefully analyze the source, translation, and feedback for translation, please generate your

correction/revision.

# Given Data

-

###

Source

: Yet, the first rush of the rebels carried the entrenchments of Speen Hill.

###

Translation

:

然

而

，

第

一

批

起

义

者

攻

占

了

Speen Hill

的

防

御

工

事

。

###

Reference Correction (Score of 8)

:

然

而

，

起

义

者

的

第

一

波

破

了

斯

平

山

的

壕

沟

。

###

Correction

:

然

而

，

第

一

批

起

义

者

攻

占

了

Speen Hill

的

防

御

工

事

。

-

# Very Important Detail!!!

1. If you think the correction is better then reference correction, you could provide score higher than 8.

2. Please first generate your explanation for the generated correction by comparing it with reference

correction, and then you must generate the score following the format 'Score: x', where x is the score

range from 1 to 10.

-----------------------------------------

# Chain-of-thought evaluation by GPT-4:

The generated correction '

然

而

，

第

一

批

起

义

者

攻

占

了

Speen Hill

的

防

御

工

事

。

' addresses the

feedback points effectively: 1) Fluency: The generated correction replaces '

冲

了

过

去

' with '

攻

占

了

,' which is more formal and appropriate for the historical context of a military action. This improves

the fluency of the sentence. 2) Readability: By using '

攻

占

了

' (captured), the generated correction

clarifies the action taken by the rebels. It removes the confusion caused by '

带

着

Speen Hill

的

工

事

冲

了

过

去

' in the original translation, making it clear that the rebels overran the entrenchments rather than

physically carrying them. 3) Translationese: The term '

带

着

' is removed in the generated correction,

thus eliminating the direct translation issue and making the sentence sound more natural in Chinese.

Comparing the generated correction to the reference correction '

然

而

，

起

义

者

的

第

一

波

破

了

斯

平

山

的

壕

沟

。

', both sentences convey the same meaning and are free from translationese. The

reference correction uses '

破

了

' (assault broke through), which is also a clear and accurate

description of the rebels' action. The reference correction includes the term '

壕

沟

' (trenches), which is a

specific type of entrenchment, while the generated correction uses '

防

御

工

事

' (defensive fortifications),

a more general term. Both terms are acceptable, but the reference correction may provide slightly more

detail about the nature of the entrenchments. Overall, the generated correction is faithful to the feedback

and is an improvement over the original translation. It is on par with the reference correction in terms

of clarity and appropriateness for the context.

#

Score: 8

L Complete Results

L.1 Complete Results for Different Qualities

Table 19 and Table 20 show the overall evaluation results for different qualities granularities on the subjective evaluation of the test and dev set the feedback, comparison, and correction critique dimensions. Table 21 and Table 22 show the overall evaluation results for different qualities granularities on the objective evaluation of the test and dev set the feedback, comparison, and correction critique dimensions.

L.2 Complete Results for Different Tasks

Table 23 and Table 24 show evaluation results on each tasks (test set) for the feedback and comparisonbased feedback dimensions. Table 25 and Table 26 show complete evaluation results on each tasks (dev set) for the feedback and comparison dimensions. Table 27 and Table 28 show complete evaluation results on each task (test and dev set) for the correction dimension.

M Human Performance in CRITICEVAL

In this section, we provide more details and comparison between LLMs and human performance. Specifically, we conduct the human annotation of the subjective tasks on the CRITICEVAL test set, and the overall human-level performance is shown in Table 29. Note that the cohort and corresponding set of human critiques does not represent the best possible human performance; instead, they represent the capability of annotators selected for this human performance annotation of the CRITICEVAL test set.

It can be found that the human-level significantly outperforms GPT-4 on the objective task, while it is inferior to GPT-4 on the subjective evaluation. Therefore, we conduct the Quantitative and Qualitative Analysis to understand the performance gap between humans and GPT-4 in subjective evaluation.

Quantitative Analysis We conduct the fine-grained failure modes analysis, and the distribution of each failure mode for feedback, comparison and correction dimensions are shown in Table 30, Table 31 and Table 32. The numbers in following tables indicate the frequencies of error types in GPT-4 and human-written critiques. The detailed description of each failure mode can be found in Section 6.8. As for the feedback and comparison dimensions, the distribution of E1 (missing issues), E2 (missing suggestions or low-quality suggestions), and E7 (insufficient analysis) in humanwritten critiques is significantly higher than that of GPT-4. In contrast, the distribution of other

error types is significantly lower. As for the correction dimension, human annotators usually do not follow suggestions in the provided feedback (E9) and generate additional errors (E11). Through communicating with the annotators, we notice that the primary cause of this issue is that some tasks require domain-specific knowledge, and the lack of this knowledge among human annotators results in lower-quality corrections. In summary, the human-written critiques are often less comprehensive than GPT-4, significantly reducing the quality. In contrast, the mistakes in human-written critiques are significantly less than that of GPT-4. This phenomenon is consistent with our preliminary study and recent findings [79, 6], further proving the effectiveness and reasonableness of leveraging the human-in-the-loop pipeline to construct comprehensive and accurate reference critiques.

Qualitative Analysis We inspect the human-written critiques in the subjective evaluation tasks to understand the source of the performance difference. In general, human annotators write fewer comments than LLMs, and the comments are usually general and brief. Besides, many tasks involve domain-specific knowledge that humans may lack, but GPT-4 excels in (albeit with potential hallucinations).

N Evaluated LLMs

We extensively evaluate widely used open-source and closed-source LLMs of different sizes on CRITICEVAL to understand the current progress in this field, including (1) instruction-tuned LLMs; (2) critique-tuned LLMs; and (3) reward models. To reproduce evaluation results, the greedy search is employed for open-source LLMs, and the temperature factor is set as 0 for closed-source LLMs, i.e., decoding randomness is minimum.

The inference procedures of all these evaluated LLMs in this paper are conducted in an A800 server with 8 GPU cards, each with 80G CUDA memory. The vLLM [82] and LMDeploy [83] packages are used to speed up the inference, and the average inference time cost for each LLM is 1.25 hours.

N.1 Instruction-tuned LLMs

For closed-source LLMs, we test GPT-4, Claude, Gemini-Pro, PaLM, GPT-3.5-turbo, etc. For open-source LLMs, we test numerous LLM series including Mistral [49], LLaMA2 [37], Baichuan2 [84], Qwen 9 [21], InternLM2 [48], WizardLM [85], Vicuna [86], Yi, and DeepSeek [50], etc.

N.2 Critique-tuned LLMs

Recent works have proven that fine-tuning LLMs on critiques generated by GPT-4 significantly improves LLM's critique ability [11, 30, 24, 12]. Llama-2-13B fine-tuned on GPT-4's critique could









0.0

0.2

0.4

0.6

0.8





















0.0

0.2

0.4

0.6

0.8



















0.0

0.2

0.4

0.6

0.8











achieve close critique ability to GPT-4, and even outperform much larger model, like Llama-2-70B. In this paper, we name the LLMs fine-tuned on critiques dataset Critique-tuned LLMs, and we comprehensively evaluate critique-tuned LLMs on our proposed CRITICEVAL: (1) TigerScore [30]; (2) Auto-J-13B [11]; and (3) UltraCM [12].

However, there are two popular critique-tuned LLMs are not evaluated in CRITICEVAL. The reasons are listed as follows: (1) InstructScore [24] can only be used to evaluate limited tasks, like data2text and commonsense, thus we donot test InstructScore in our work; (2) Prometheus [14] are not evaluated because of its high dependence on the criteria question, score rubrics and reference answers, which is not fully covered in our benchmark.

N.3 Reward Models

Moreover, we show CRITICEVAL can also be used to evaluate reward models [87]. There are lots of reward models that can be publicly accseed. We only evaluate three representative reward models, and leave the evaluation on other reward models in our future work [58]: (1) UltraRM-13B [12]; (2) Ziya-7B [52]; (3) SteamSHP [53].

O Interpretable Analysis of the Quality of Textual Critiques

Beyond the simplified average scores from 1 to 10 in the subjective evaluation of CRITICEVAL, we also categorize the textual critiques of each LLM into three quality intervals for more interpretable analysis: (1) Low-quality critiques (1-3); (2) Medium-quality critiques (4-6); (3) High-quality critiques (7-10). The results of five representative LLMs on feedback, correction, and comparionbased feedback dimensions are shown in Figure 13. It can be found that GPT-4-turbo exhibits strong critique ability and barely generates low-quality critiques. In contrast, the critiques generated by the Llama-2-70B-Chat are usually low-quality. Besides, the ratio of low-quality critiques generated by some LLMs, like Llama-2-70B-Chat and Qwen-72B-Chat, are very high, indicating that they have a lot of room for improvement.

P Error Patterns in Responses

In this section, we analyze the details and cases about three kinds of error patterns in responses: (1) obvious error; (2) complex error; (3) subtle error. Specifically, we ask human annotators to summarize the common error cases after they annotate all the textual critiques in CRITICEVAL, and categorize them into obvious error, complex error, and subtle errors in Table 33. It should be noted that these three error patterns may have other specific error cases in nine domains. However, it is difficult to exhaust all the error case. Thus, our motivation is to list, annotate, and analyze as many as possible to ensure the accuracy of our experimental results.

Q Likert Score for Responses

Figure 5 demonstrates the discernible performance disparities in responses for each task. Since automatic execution leaks quality information, we do not collect the correct responses for the





































































CodeExec task. Such variation is instrumental in analyzing the impact of response quality on the feedback.

R Visualization of Relationship between Model Scales and Critique Ability

Following previous works [9], we provide the diagrams to demonstrate the relationship between LLM's critique ability and model scales, which are shown in Figure 14. It can be easily found that the critique ability of all LLM series steadily increase with the number of the parameters (scale) increasing.

S How Few-shot Examples Affect Performance

In this paper, we have studied the few-shot prompting strategy. However, our results demonstrate that few-shot prompting reduces performance across various LLMs. As illustrated in Table 34, when using 1-5 examples in objective feedback evaluations, we observed a significant decline in LLMs' performance as the number of examples increased. This intriguing phenomenon may be due to the complexity of the critique task, where few-shot examples might impede the LLM's understanding of the evaluated responses. Consequently, CRITICEVAL currently does not utilize few-shot prompting by default. Given the emerging interest in critique ability research, we look forward to future works investigating advanced inference strategies to improve critique ability of LLMs.

T Scalability and Cost about CRITICEVAL

In this section, we provide the cost of constructing one task and inferencing one LLM in CRITICEVAL.

T.1 Construction Cost

Collect Evaluated Responses for All Tasks

Open-source LLMs: a GPU server with 8 A100 (80G) cards is used to generate evaluated responses, and the total GPU hours are 4.26 hours, approximately 82.88$ (refer to the price of Alibaba Cloud).

Closed-source LLMs: the average cost for each LLM is 0.89$.

Generate and Revise GPT-4 Critiques The cost of the human annotation is computed under these settings: (1) Four human annotators (3 annotators and one supervisor); (2) 5.69$ hourly wage for each annotator; (3) Average 400 samples in one task. The overall construction cost are shown in Table 35, which is affordable [88].

T.2 Average Computation Cost for One LLM

As shown in Table 36The overall cost of the test and dev set is 13.19+9.94=23.13$, comparable to the evaluation cost on the AlpacaEval benchmark (5-15$) [19]. These costs are essential for CRITICEVAL, as they guarantee the reliability of critique evaluation. We promise to add these details to the Appendix of our revised submission.

U Multilingual Support

The primary goal of CRITICEVAL in the current stage is to construct a reliable and comprehensive evaluation for critique ability. We agree that it is essential to study multilingual critiques and intend to broaden CRITICEVAL to include other languages in future work. The following content briefly introduces our preliminary solution on how to achieve this goal.

Construct Multilingual CRITICEVAL Following the previous work [89], CRITICEVAL could be translated to various languages, especially low-resource languages, with human annotation for revising translation inaccuracies. the most direct way is to translate CRITICEVAL into various languages, with human annotation for revising translation inaccuracies.

Evaluate Multilingual CRITICEVAL While the reliability of objective evaluation could be ensured, the reliability of subjective evaluation is limited by the multilingual capability of the judge model (GPT-4). We recommend back-translating multilingual critiques into English and evaluating them within English CRITICEVAL.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have well discussed the main claims in Section 4 and Section 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are well described and discussed in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our experimental results could be easily reproduced because we have already discussed the necessary details in Section 6.1, Appendix I and Appendix L.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We upload dataset and evaluation toolkit of our proposed CRITICEVAL with this submission to reproduce our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have well described all the details about inference procedure in Section 6, Appendix H, Appendix F, Appendix E, Appendix D, Appendix N and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the statistical significance of our objective evaluation in CRITICEVAL in Section 5 and Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The time cost, compute worker, CUDA memory for LLM inference in our proposed CRITICEVAL dataset are well described in Appendix N.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We promise our research conforms with the NeurIPS Code of Ethics, which is listed in Section B, Section D.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We have described the ethical consideration in Appendix B, and our paper doesn't raise the potential positive or negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We have discussed this part of content in Appendix B.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have well discussed the Licenses of existing assets in Appendix D, while we do not discuss more details for the used LLMs in this paper due to the vast number of LLMs.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We upload the documentation and some important information about our new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We have discussed this part of the content carefully in Appendix H.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: We have submitted an application to the Institutional Review Board (IRB). After evaluation, the committee determined that the human annotation for critique task posed no ethical concerns, and accordingly, we conduct the human annotation. Besides, we have adequately addressed minimal ethical concerns by following efforts: ensuring the fairness of the data collection by selecting a diverse group of annotators, providing clear guidelines, and compensating the annotators well, as described in Appendix B and Appendix H. Therefore, these efforts make sure our work does not violate the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This work introduces a benchmark for using LLMs are critics. 

The benchmark covers four settings:
1. Providing feedback
2. Correction of a response with/without feedback
3. Comparison of two responses for a given query
4. Providing meta-feedback (feedback on feedback)

STRENGTHS:
Extensive experiments across models and setups.

WEAKNESSES:
For weakness I would repeat the limitations I list below.

QUESTIONS:
Did you consider structured generation? 

Besides positional bias, did you explore any other biases such as bias to length, style or certain words?

Did you consider few-shot performance?

LIMITATIONS:
Two minor limitations:
- As far as I can tell structured generation was not used which has been shown to improve performance
- As far as I can tell few-shot performance was not considered.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CriticEval, a benchmark designed to comprehensively and reliably evaluate the critique ability of large language models (LLMs). It assesses critique capabilities across four dimensions (feedback, comparison, correction, and meta-feedback) and nine diverse task scenarios, using both scalar-valued and textual critiques for responses of varying quality. The benchmark was constructed using a human-in-the-loop pipeline, with initial critiques generated by GPT-4 and refined by human experts. CriticEval employs both objective metrics and subjective evaluation by GPT-4 with human-annotated reference critiques. Key findings from evaluating 35 LLMs include: GPT-4's high correlation with human judgments when using reference critiques, some open-source LLMs approaching closed-source models in performance, and insights into how critique difficulty varies by task type, response quality, and critique dimension.  CriticEval could be used as a comprehensive and reliable tool for assessing LLM critique capabilities.\n\nSTRENGTHS:\n- CRITICEVAL evaluates critique ability across multiple dimensions (feedback, comparison, correction, meta-feedback) and diverse task scenarios, providing a more holistic assessment than existing benchmarks.\n- The benchmark combines GPT-4 generated critiques with human expert refinement and employs both objective metrics and subjective evaluation with human-annotated reference critiques. This approach ensures a more reliable and accurate evaluation of LLM critique abilities.\n- The paper presents results from evaluating 35 open-source and closed-source LLMs, offering valuable insights into the current state of LLM critique capabilities. It reveals interesting relationships between critique difficulty and factors like task type, response quality, and critique dimension.\n\nWEAKNESSES:\n- The biggest concern of the dataset is that it relies heavily on GPT-4 for initial critique generation and evaluation. This could introduce a bias favoring models such as GPT-4 and models trained on GPT-4 distilled data. The human-in-the-loop process might not fully mitigate this bias, especially if annotators are influenced by GPT-4's initial outputs. A more diverse set of models or purely human-generated critiques for the benchmark could have provided a more neutral evaluation framework. \n- The paper doesn't adequately address the scalability of CRITICEVAL for evaluating future language models. Additionally, the reliance on GPT-4 and human experts for evaluation might make it challenging for other researchers to fully reproduce or extend the benchmark. \n- Insufficient analysis of failure modes: While the paper presents overall performance metrics, it doesn't delve deeply into the specific ways in which models fail at critique tasks. A more detailed error analysis could provide valuable insights into the limitations of current LLMs and guide future research more effectively.\n- Lack of comparison to human performance: The paper doesn't provide a clear comparison between LLM performance and human performance on these critique tasks.\n\nQUESTIONS:\nHow did you evaluate the human's performance during the benchmark construction phase?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the need for a comprehensive evaluation of the critique ability of large language models (LLMs) for self-improvement and alignment with human outcomes. Current evaluation methods are critiqued for their limited scope and reliability. The authors propose CRITICEVAL, a benchmark designed to evaluate LLMs across four critique dimensions: feedback, comparison, correction, and meta-feedback, covering nine diverse task scenarios including NLP, alignment, and reasoning tasks. While CRITICEVAL incorporates human-annotated references to enhance reliability, it heavily relies on GPT-4 for evaluations, raising concerns about generalisability across other LLMs. Evaluations of 35 LLMs demonstrate CRITICEVAL's effectiveness but also highlight the inherent difficulties in critiquing complex tasks and the inverse relationship between critique and response quality. Although promising, the reliance on human annotation and the potential biases introduced by using a single LLM for baseline evaluations could limit its broader applicability. The release of datasets and evaluation tools is a positive step towards fostering further research.\n\nSTRENGTHS:\nThe work is a very comprehensive study on the critical evaluation problem and provides insight for those who would be in situations where they would want to build better LLM pipelines for use-cases where accuracy is important for fit for use. \n\nThe exploration of 35 different LLM providers more diversity and incorporating human feedback for quality ranking, it allows for wider evaluation than prior benchmarks. \n\nI believe the work is well presented, written and argued. It is a very daunting project to get all the pieces together. It is harder, ironically, to evaluate it just as a paper because of all the moving parts, but the authors evaluations across the LLMs are appreciated.\n\nWEAKNESSES:\nEven with the mention of incorporating Chinese, I believe there should be a serious discussion on how such evaluation pipelines and datasets would function for low-resource languages (Chinese is not one). Many of the large LLMs have multilingual capabilities and their generation of responses are more likely to have higher error rates and as such being able to do critical evaluation in such use-cases is important. \n\nThe computational cost and real cost of setting up the CRITICEVAL pipelines should be spelled out.\n\nIt is not clear if IRB/Ethical clearance was obtained as the checklist response states that IRB *would* be easy to obtain. This is a concern as such I will be referring this for ethics review. It is appreciated that information was provided in Appendix B and G, but whether IRB was obtained or not should have a clear YES or NO statement.\n\nQUESTIONS:\n1. How do you protect against errors in the underlying tasks datasets. The explosion of many task evaluation sets, even with the best intentions of researchers means that we have compounding effects of source errors (e.g. in Translation), how would this affect your work and how would you mitigate against it?\n2. Ultimately how many people are involved and computational cost of such an exercise as this affects replication ability of this study to cover some of the limitations you hilighted?\n\nLIMITATIONS:\nThe limitations are written in Appendix A. They are clear.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study constructs a comprehensive framework for LLM-based evaluation, encompassing data construction, human/machine annotation, and result analysis. It defines a single evaluation framework that covers various tasks and response types. Although previous studies have evaluated different tasks and response types, they often lacked comprehensive analysis, making it difficult to understand the relationships between various factors during evaluation. This research aims to address this gap. Additionally, it examines the reliability of evaluations based on the types of judge models and target models, as well as the importance of human annotation data.\nThrough extensive experiments and comprehensive analysis, this study demonstrates consistent trends across various tasks and highlights the importance of utilizing human-annotated data in evaluations.\n\nSTRENGTHS:\n1. The proposed framework allows for the evaluation of various tasks and response types.\n2. Through various experiments, the high evaluation capability of closed-source LLMs, regardless of response type and task, is confirmed.\n3. The importance of human-annotated data is evident, showing consistency regardless of the evaluation model.\n4. Additionally, extensive experiments analyze various aspects such as response quality, difficulty, and the capability of reward models as judge models.\n\nWEAKNESSES:\n1. One important metric for evaluation models, the ability to revise its generation with critique, is excluded. As mentioned in the paper, there is active research on improving generation performance through LLM’s self-feedback during the inference stage. It is necessary to evaluate each model’s ability to revise existing outputs when provided with critique.\n2. Although this study addresses the translation task, it excludes the critique ability in other multilingual contexts. The critique ability of LLMs in various languages other than English is crucial for ensuring diversity, and additional data collection on this aspect seems necessary.\n\nQUESTIONS:\n1. What is the pattern of score changes when using feedback from other LLMs in Table 5, based on the quality of the feedback from these other LLMs?\n\nLIMITATIONS:\nYes, this study clearly mentions its limitations in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}