Training Language Models to Critique with Multi-Agent Feedback

## Abstract

Abstract Critique ability, a meta-cognitive capability of humans, presents significant challenges for LLMs to improve.
Recent works primarily rely on supervised fine-tuning (SFT) using critiques generated by a single LLM like GPT-4.
However, these model-generated critiques often exhibit flaws due to the inherent complexity of the critique.
Consequently, fine-tuning LLMs on such flawed critiques typically limits the model’s performance and propagates these flaws into the learned model.
To overcome these challenges, this paper proposes a novel data generation pipeline, named MultiCritique, that improves the critique ability of LLMs by utilizing multi-agent feedback in both the SFT and reinforcement learning (RL) stages.
First, our data generation pipeline aggregates high-quality critiques from multiple agents instead of a single model, with crucial information as input for simplifying the critique.
Furthermore, our pipeline improves the preference accuracy of critique quality through multi-agent feedback, facilitating the effectiveness of RL in improving the critique ability of LLMs.
Based on our proposed MultiCritique data generation pipeline, we construct the MultiCritiqueDataset for the SFT and RL fine-tuning stages. Extensive experimental results on two benchmarks demonstrate:
1) the superior quality of our constructed SFT dataset compared to existing critique datasets;
2) additional improvements to the critique ability of LLMs brought by the RL stage.
Notably, our fine-tuned 7B model significantly surpasses other advanced 7B-13B open-source models, approaching the performance of advanced 70B LLMs and GPT-4.
Codes, datasets and model weights will be publicly available.

## 1 Introduction

The critique ability, i.e., the capability to identify and refine flaws in responses, has been widely used to facilitate reliable automatic evaluation and self-improvement of LLMs (Lan et al., ; Wu et al., ).
As a meta-cognitive capability (Toy et al., ; Wang & Zhao, ), critique ability requires LLMs to possess a deep understanding of user queries and evaluated responses beyond mere criticism (Kim et al., ; Zheng et al., ).
Therefore, it is challenging to improve the critique ability of LLMs (Lan et al., ; Lin et al., ).

Recent researches indicate that open-source LLMs usually demonstrate weaker critique abilities compared to advanced closed-source models (Lan et al., ; Lin et al., ).
Therefore, numerous works have been proposed to improve the critique ability of LLMs, primarily through Supervised Fine-Tuning (SFT) using critiques generated by one single LLM, typically GPT-4 (Li et al., ; Cui et al., ).
However, these model-generated critiques often exhibit flaws like inaccurate descriptions or suggestions about flaws in responses due to the complexity of the critique (Verga et al., ; Lan et al., ; Liu et al., ).
Consequently, the fine-tuned LLMs on these datasets are constrained by these flawed critiques, and these flaws in critiques are propagated into the learned models during the SFT stage, leading to the potential amplification of these issues.

To address these challenges, in this paper, we develop a novel data generation pipeline named MultiCritique. This approach utilizes multi-agent feedback to construct the high-quality dataset for the SFT and RL fine-tuning stages.
First of all, to mitigate the effects of flawed critiques generated by one single LLM, we develop the MultiCritique-SFT pipeline, as shown in Figure 1 (Step 2).
Specifically, it first collects the multi-agent analytical critiques from four advanced LLMs rather than one single LLM, with three crucial information as input for simplifying the critique: (1) A detailed task description of the user query; (2) A customized two-tier structured evaluation criteria; and (3) A reference response tailored to satisfy these criteria.
Each model-generated analytical critique consists of a list of Analytical Critique Units (ACUs), with each ACU identifying one specific flaw in the evaluated responses.
Then, GPT-4 conducts the meta-critique classification process to classify these ACUs into seven quality categories automatically (Lan et al., ), given all multi-agent analytical critiques as context.
The process concludes by summarizing a final analytical critique that aggregates high-quality ACUs while discarding flawed ones, effectively combining the advantages of multiple models.

Figure: Figure 1: The overview of our proposed MultiCritique data generation pipeline. First, we prepare queries and evaluate responses and crucial information (Step 1). Then, we conduct the MultiCritique-SFT pipeline to construct the high-quality SFT critique dataset (Step 2). Finally, we conduct the MultiCritique-RL pipeline to construct the preference critique dataset for the RL stage (Step 3). An ACU is a structured unit for identifying one specific flaw in the evaluated response. A list of model-generated ACUs denotes the analytical critique.
Refer to caption: x1.png

Second, to move beyond the mere behavior cloning on model-generated critiques in the SFT dataset,
we develop the MultiCritique-RL data generation pipeline to construct the high-quality preference critique dataset via multi-agent feedback, facilitating the effectiveness of RL in improving the critique ability of LLMs.
Specifically, as shown in Figure 1 (Step 3), chosen (+) and rejected (-) analytical critiques are naturally paired via meta-critique classification in the MultiCritique-SFT pipeline, where chosen analytical critiques exhibit fewer and minor flawed ACUs than rejected ones.
However, the preference between these chosen and rejected analytical critiques might be inaccurate due to the complex meta-critique analysis and limited model capability (Lan et al., ; Sun et al., ).
To address this issue, we then propose the Multi-Agent-Revision-Scoring (MARS) filtering to refine the preference critique dataset, ensuring the chosen analytical critiques lead to superior revisions compared to rejected ones across multiple models.
These preference pairs are used to fine-tune a reward model, guiding the RL process to improve critique ability further.
Unlike recent works that utilize preference learning to improve the critique ability of LLMs, such as Themis (Hu et al., ) and CriticGPT (McAleese et al., ), our proposed MultiCritique-RL pipeline does not require any human annotations, thereby demonstrating better scalability.

Based on our proposed MultiCritique data generation pipeline, we construct the MultiCritiqueDataset for improving critique ability, consisting of two splits for the SFT and RL fine-tuning stages (right part of Figure 1).
Extensive experiments on two benchmarks demonstrate the superior quality of our proposed MultiCritiqueDataset compared to other critique datasets.
Specifically, even if the scales of existing datasets are over three and eight times larger than our proposed dataset, the model fine-tuned during the SFT stage on our proposed MultiCritiqueDataset still significantly outperforms those fine-tuned on other datasets, with 21.48% and 22.50% average performance gain on CriticEval (Lan et al., ) and CriticBench (Lin et al., ) benchmarks, respectively.
Furthermore, the RL fine-tuning stage on MultiCritiqueDataset further boosts the critique ability of the SFT model, resulting in 6.3% and 0.51% absolute improvements on CriticEval and CriticBench.
Besides, ablation studies confirm the positive contributions of crucial information in simplifying the critique and the roles of our proposed MultiCritique data generation pipeline in enhancing dataset quality.
In summary, the InternLM2-7B-Chat-SFT model that fine-tuned by SFT and RL stages on our proposed MultiCritiqueDataset, significantly outperforms advanced 7B-13B instruction-tuned and critique-tuned LLMs as well as the GPT-3.5-Turbo model, approaching advanced 70B LLMs and GPT-4. For example, our model approaches GPT-4 on CriticBench, with an F1 score of 75.66% vs. 78.75%, significantly outperforming average performance of 7B-13B instruction-tuned and critique-tuned LLMs (55.58%).

## 2 Related Work

Critique Ability of LLMs
The critique ability of LLMs has been applied in three key areas:
(1) Reliable Automatic Evaluation: LLMs can achieve high correlation with human annotators in response evaluation (Liu et al., ; Zheng et al., );
(2) Self-Improvement of LLMs: This ability facilitates the self-improvement of LLMs during inference and training stages (Yuan et al., ; Wu et al., );
(3) Robust Reward Modeling: Textual critiques contribute to robust reward modeling by providing detailed feedback (Ye et al., ; Zhang et al., ).
Recently, to bridge the gap between the weak open-source LLMs and advanced closed-source models,
various datasets have been developed to enhance the critique capability of LLMs by mimicking GPT-4 behavior (Lan et al., ; Lin et al., ).
For example, datasets such as UltraFeedback (Cui et al., ), Auto-J (Li et al., ) and Prometheus (Kim et al., ), collect critiques from GPT-4 to fine-tune open-source LLMs like Llama2-13B.
Despite their potential, these datasets often suffer from quality issues due to the inherent complexity of critiques (Verga et al., ).
Compared with these existing works, we first propose a novel data generation pipeline, MultiCritique, to collect high-quality critiques from multi-agents.
Then, reinforcement learning is used to improve the critique ability of LLMs, moving beyond the mere behavior cloning on the supervised critique dataset.

Preference-based Reinforcement Learning (RL)
Reinforcement learning (RL) algorithms, like PPO (Schulman et al., ), are widely utilized to guide LLMs to generate responses that are more preferred by humans (Yang et al., ; Xu et al., ; Yuan et al., ).
RL algorithms typically employ a reward model as a proxy for human judgment, learning through human-annotated pairwise comparison of responses. This is often called Reinforcement Learning from Human Feedback (RLHF) (Stiennon et al., ).
Recently, some works have been proposed to improve the critique ability through reinforcement learning. For example, CriticGPT (McAleese et al., ) improves the critique ability of ChatGPT through RLHF.
Similarly, Themis (Hu et al., ), Self-Taught Evaluator (Wang et al., ), SFR (Wang et al., ) collect the preference critique pairs by examining consistency between the LLM’s judgment scores and the human-annotated judgment scores.
However, their solution is limited due to the high cost of human annotations. In contrast, our proposed MultiCritique pipeline automatically collects preference critique datasets via a multi-agent framework without human supervision.
Our concurrent work, Meta-Rewarding (Wu et al., ), collects preference critique pairs using optimized LLM as the meta-judge model.
Compared with Meta-Rewarding, we propose the Multi-Agent-Revision-Scoring (MARS) to filter high-quality critique pairs by using the quality of their revisions as an indicator of critique quality, thereby facilitating a more robust RL.

Multi-agent Framework
The multi-agent framework has been shown to foster more divergent thinking and aggregate the diverse opinions of multiple LLMs (Liang et al., ; Zhang et al., ), effectively improve the alignment of LLMs. For example, Arena learning (Luo et al., ) and Stable Alignment (Liu et al., ) improve the alignment of LLMs via multi-agent battle results or social simulation.
Recently, the multi-agent framework has also been used for improving critique ability. For exmaple, PoLL (Verga et al., ) addresses the model-specific biases by pooling judgment scores of a panel of small models.
ChatEval (Chan et al., ) enhances the reliability of LLM-based automatic evaluations via multi-agent debate.
However, the quality of their multi-agent critiques are not assessed, which may lead to incorrect critiques influencing other LLMs.
Compared to their work, we explicitly evaluate the quality of critiques through (1) GPT-4-based meta-critique; and (2) evaluating revision quality as an indicator of critique quality.

## 3 Method

### 3.1 Data Preparation

As shown in Figure 1 (Step 1), we collect diverse queries and evaluated responses before running our proposed MultiCritique data generation pipeline. Besides, we also elicit crucial information from the queries to simplify the critique. Details are elaborated as follows.

Diverse Queries Collection We first compile queries from several well-established datasets. These include alignment datasets such as OpenHermes-2.5, DEITA (Liu et al., ) and OpenAssistant (Köpf et al., );
mathematical and coding reasoning datasets like MetaMathQA (Yu et al., ) and CodeFeedback (Zheng et al., );
as well as the critique dataset Auto-J (Li et al., ).
In total, we sample 10.7K queries covering 123 diverse task scenarios.

Diverse Evaluated Responses Collection
Once the queries are collected, we use eleven LLMs with different capabilities to generate responses with varying response qualities. These responses are then evaluated using the robust reward model—InternLM2-20B-reward (Cai et al., ), which performs well in RewardBench (Lambert et al., ).
We select three low-, medium- and high-quality responses for each query, ensuring a significant performance gap.
In total, 10.7K$\times$3=32.1K query-response pairs are collected.

Crucial Information Collection
Once the query-response pairs are collected, we elicit three crucial information sequentially using GPT-4 to simplify the critique:
(1) Task Description: Our preliminary findings indicate that LLMs may misunderstand the objectives of queries. By prompting GPT-4 to describe the task, we can mitigate this issue to some extent;
(2) Customized Evaluation Criteria:
Once the task description is obtained, we propose generating customized two-tier structure evaluation criteria tailored to each query to guide effective critiques.
The first tier outlines the fundamental evaluation dimensions for the task, while the second tier offers more customized criteria. Each criterion is structured with a name, description and level of importance (Li et al., );
(3) Reference Response: Finally, we generate reference responses that satisfy all customized evaluation criteria. Since reference responses tend to produce critiques that lack diversity for mathematical and coding questions, we set reference answers as empty for these two tasks.

Following these three steps, we collect $\mathbb{N}$=32.1K samples $\{(q_{i},r_{i},\mathcal{CI}_{i})\}_{i=1}^{\mathbb{N}}$, where $q_{i},r_{i},\mathcal{CI}_{i}$ represents the $i$-th query, evaluated response and corresponding crucial information in the dataset. Please refer to Appendix C for more details and statistical information about the collection of queries and evaluated responses.

### 3.2 MultiCritique-SFT Data Generation Pipeline

After collecting query-response and crucial information, we introduce our proposed MultiCritique-SFT data generation pipeline to construct the high-quality SFT dataset.
The overview of MultiCritique-SFT pipeline is shown in Figure 1 (Step 2).
This pipeline gathers multi-agent critiques to mitigate flawed critiques generated by one single LLM.

To achieve this goal, we aim to aggregate the high-quality content in multi-agent critiques into a comprehensive and accurate critique.
Therefore, we first collect detailed analytical critiques simultaneously from multiple LLMs (Multi-Agent Analytical Critique). Each analytical critiques consist of a list of Analytical Critique Units (ACUs). Then, the GPT-4 classifies each ACU into one of seven quality categories (Meta-Critique Classification).
Finally, a final critique is aggregated by summarizing the accurate ACU while discarding the flawed ones (Critique Summarization).
Unlike previous works allowing multi-agent discussions or debates (Chan et al., ), our preliminary study observes that LLM’s critiques could be easily influenced by other critiques, potentially reducing the diversity and comprehensiveness of critiques.
Thus, we do not allow the inner-model discussions. Instead, we utilize GPT-4 as a summarizer to evaluate each critique content given all LLMs critiques as context, since previous works (Lan et al., ; Lin et al., ) have proven that only GPT-4 could effectively conduct the meta-critique.

Multi-agent Analytical Critiques
We employ four LLMs to simultaneously critique the evaluated responses: GPT-4, Claude-1-instant, Qwen-1.5-72B-Chat and InternLM2-20B-Chat, which exhibits strong performance on CriticEval benchmark (Lan et al., ).
Each LLM performs sentence-by-sentence and cross-sentence critique and generates a structured analytical critique, which is a list of Analytical Critique Units (ACUs).
An ACU is a structured unit for identifying and addressing a specific flaw in the evaluated response, consisting of five values about the flaw:
(1) the location(^1^11We introduce a pre-processing step to label sentences in evaluated responses, as detailed in Appendix C.1.);
(2) the description;
(3) suggestions for revision;
(4) the criteria type;
and (5) the severity.
The structured ACUs not only benefits the robust subsequent meta-critique process (Sun et al., ) but also exhibit great interprebability.

Meta-Critique Classification
Since LLMs often produce flawed critiques due to the limited model capability (Lan et al., ; Wang et al., ), it is crucial to identify these flaws before aggregating them into a final comprehensive and accurate analytical critique.
To achieve this goal, we employ GPT-4 to judge the quality of these model-generated analytical critiques, a concept known as meta-critique (Sun et al., ; Lan et al., ).
Specifically, GPT-4 classifies each ACU into one of seven quality categories by considering all the multi-agent analytical critiques in the context.
These quality categories are determined by human annotators and are associated with human-annotated severity scores ranging from 1 to 5.
Each quality category reflects one specific flaw in an ACU.
Please refer to Appendix H.5 for more details.
Importantly, for one model-generated analytical critique, the accumulated severity scores of its ACUs could indicate its quality, whereas a higher accumulated severity score indicates lower analytical critique quality.

Critique Summarization
After the previous steps, GPT-4 summarizes these ACUs into a comprehensive analytical critique.
This process involves retaining and merging accurate ACUs generated by multiple LLMs while modifying or excluding those identified as flawed.
Except for the final summarized analytical critique, we also prompt GPT-4 to generate an overall description and judgment score for the evaluated response.
Finally, the final analytical critique, description and judgment score are concatenated as the final critique for the evaluated response, denoted as $\mathcal{C}$.

By following previous steps of MultiCritique-SFT, we construct a supervised fine-tuning dataset MultiCritiqueDataset-SFT, consisting of $\mathbb{N}$=32.1K samples: $\{(q_{i},r_{i},\mathcal{CI}_{i},\mathcal{C}_{i})\}_{i=1}^{\mathbb{N}}$.

### 3.3 MultiCritique-RL Data Generation Pipeline

Beyond the behavior cloning on the supervised dataset, we also conduct the MultiCritique-RL data generation pipeline to construct the preference critique dataset for training reward models.
As shown in Figure 1 (Step 3), the MultiCritique-RL pipeline involves two steps.

Preference Pairs Collection
Four model-generated analytical critiques could be naturally collected from our proposed MultiCritique-SFT data generation pipeline.
As described above, the quality of ACUs in each analytical critique is classified by meta-critique, and each model-generated analytical critique is associated with an accumulated severity score.
Except for four model-generated analytical critiques, we also supply the summarized final analytical critique with its accumulated severity score of 0.
For $i$-th sample in the SFT dataset, the chosen and rejected critiques, denoted as $C_{i,j_{+}},C_{i,j_{-}}$, are paired when there exists a significant performance gap between them, i.e., the difference in their accumulated severity scores is greater than the threshold, which we determined to be 5.
It should be noted that the descriptions and final judgment score of response quality are primarily based on the final summarized analytical critique. Since they are easy instruction-following tasks, we do not collect their preference pairs for the RL fine-tuning stage.

Multi-Agent-Revision-Scoring (MARS) Filtering
The meta-critique analyses might be inaccurate due to the complex meta-critiques and limited GPT-4 capability (Lan et al., ), leading to the noise in the preference dataset.
To address this issue, we leverage the quality of revision as an indicator of critique quality, and propose the Multi-Agent-Revision-Scoring (MARS) filtering to refine the preference dataset.
Specifically, four 7B LLMs first revise the evaluated response based on each critique, each performing eight revisions, resulting in a total of 4$\times$8=32 revisions.
The reason for using multiple LLMs rather than one single model for revisions is to ensure the reliability and robustness of the evaluation.
These revisions are then evaluated using the InternLM2-20B-reward (Cai et al., ).
Finally, critique pairs in the preference dataset are reserved if the chosen critique’s average reward score is higher than the rejected critique’s score.
For mathematical problems, we compute the exact answer matching rather than reward model scores.

In summary, the preference dataset MultiCritiqueDataset-RL for the RL fine-tuning stage is constructed, consisting of $\mathbb{M}$=19.7K samples: $\{(q_{i},r_{i},\mathcal{CI}_{i},C_{i,j_{+}},C_{i,j_{-}})\}_{i=1}^{\mathbb{M}}$.

## 4 Experimental Setup

### 4.1 Implementation Details

Fine-tuning on MultiCritiqueDataset consists of two sequential stages:

SFT Stage
To ensure a deep understanding of the critiques, we optimize the LLMs using the concatenation of the crucial information $\mathcal{CI}_{i}$ and final critiques $\mathcal{C}_{i}$ by minimizing Maximum Likelihood Estimation (MLE):

$$ $L_{\text{MLE}}=-\frac{1}{\mathbb{N}}\sum_{i=1}^{\mathbb{N}}\log p_{\theta}( \mathcal{CI}_{i},\mathcal{C}_{i}|q_{i},r_{i})$ (1) $$

RL Stage
After the SFT fine-tuning stage, a reward model is first trained to classify chosen and rejected analytical critiques $C_{i,j_{+}},C_{i,j_{-}}$ by optimizing the focal ranking loss, following previous works (Cai et al., ).
Then, the SFT model is optimized by PPO (Schulman et al., ) algorithm to generate the analytical critiques with fewer flaws, guided by this reward model.

Please refer to Appendix C for implementation details about fine-tuning stages.

### 4.2 Benchmarks and Evaluation Metrics

We utilize CriticEval (Lan et al., ) and CriticBench (Lin et al., ) benchmarks to evaluate the critique ability of LLMs.

#### CriticEval

evaluates critique ability across nine tasks, covering alignment, common NLP and reasoning capabilities.
We first evaluate the critique quality: (1) The objective feedback evaluation ($\bm{F}_{\text{obj.}}$) calculates the Spearman correlation between LLM and human judgments on response quality; (2) The subjective feedback evaluation ($\bm{F}_{\text{sub.}}$) involves GPT-4 assessing the textual critiques quality. These scores range from 1 to 10.

Furthermore, we evaluate the quality of revisions generated by critiques as the indicator of critique quality: (1) The objective revision evaluation ($\bm{R}_{\text{obj.}}$) measures the average Pass Rate of five LLMs’ revisions for mathematical and coding questions. CriticEval evaluates the chain-of-thought (CoT) and program-of-thought (PoT) approaches for mathematics. For coding tasks, it compares two settings: with execution (CodeExec) and without execution
results (CodeNE);
(2) The subjective revision evaluation ($\bm{R}_{\text{sub.}}$) is assessed by GPT-4, with scores ranging from 1 to 10.

Importantly, CriticEval has proven a strong correlation between GPT-4 and humans in subjective evaluation, given the human-annotated critiques and revisions as references.

#### CriticBench

consists of 3,825 queries and evaluated responses for five challenging reasoning tasks: (1) mathematical reasoning; (2) commonsense reasoning; (3) symbolic reasoning; (4) algorithm reasoning; and (5) code generation.
The correctness of the evaluated responses is annotated based on the ground-truth responses. The F1 score is used to evaluate whether LLMs can accurately identify the correctness of evaluated responses.
Since critique-tuned LLMs cannot utilize few-shot samples, all models are tested under the zero-shot setting to ensure a fair comparison.

### 4.3 Baseline Datasets and Models

#### Baseline Datasets

Three critique datasets constructed by GPT-4 are compared : (1) Auto-J (Li et al., ); (2) UltraFeedback (Cui et al., ) and (3) Feedback-Collection (Kim et al., ).

#### Baseline Models

We evaluate both advanced closed-source and open-source instruction-tuned LLMs, including GPT-3.5-Turbo and GPT-4 (GPT-4-1106-preview), Llama3 and the Qwen2 (Yang et al., ) series.
Additionally, we assess several critique-tuned LLMs, such as Themis (Hu et al., ), TIGERScore (Jiang et al., ), Auto-J (Li et al., ), UltraCM (Cui et al., ).
Prometheus requires criteria and reference responses as inputs, which are unavailable in the two benchmarks.
To address this, we fine-tune the InternLM2-7B-Chat-SFT model using our processed dataset, moving the evaluation criteria and reference responses into the output.
Some critique-tuned LLMs, such as JudgeLM (Zhu et al., ), are excluded, and reasons are detailed in Appendix E.1.

More details about our evaluation setup can be found in Appendix C.

**Table 1: Overall experimental results. Results for GPT-3.5-Turbo and GPT-4-Turbo are from original paper (Lin et al., ). For 7B-13B models, the best performance is highlighted in bold. Results comparable to the best performance (no more than 0.2% performance gap) are also highlighted.**
| Models | CriticEval | CriticBench |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\bm{F}_{\text{obj.}}$ | $\bm{F}_{\text{sub.}}$ | $\bm{R}_{\text{obj.}}$ | $\bm{R}_{\text{sub.}}$ | Math | Comm. | Symb. | Algo. | Code | Overall |  |
| Closed-source LLM |  |  |  |  |  |  |  |  |  |  |
| GPT-3.5-Turbo | 61.47 | 5.06 | 15.54 | 6.20 | - | - | - | - | - | 51.44 |
| GPT-4-Turbo | 76.09 | 7.90 | 26.88 | 7.71 | - | - | - | - | - | 78.75 |
| 70B instruction-tuned LLMs |  |  |  |  |  |  |  |  |  |  |
| Qwen2-72B-Instruct | 75.44 | 7.83 | 23.89 | 7.21 | 82.15 | 59.64 | 78.22 | 51.35 | 85.81 | 75.86 |
| Llama3-70B-Instruct | 73.28 | 7.05 | 21.97 | 6.90 | 82.35 | 60.22 | 86.31 | 54.90 | 86.16 | 76.80 |
| 7B-13B instruction-tuned and critique-tuned LLMs |  |  |  |  |  |  |  |  |  |  |
| Qwen2-7B-Instruct | 50.49 | 5.47 | 16.21 | 5.42 | 52.25 | 26.20 | 29.55 | 9.35 | 70.23 | 45.66 |
| Llama3-8B-Instruct | 37.20 | 5.04 | 17.69 | 5.98 | 78.33 | 62.64 | 62.05 | 62.19 | 76.41 | 70.71 |
| Themis-8B | 38.07 | 4.07 | 14.43 | 2.63 | 53.34 | 27.35 | 33.16 | 35.64 | 44.33 | 42.57 |
| Prometheus-7B (Ours) | 38.06 | 2.54 | 18.78 | 4.57 | 59.43 | 54.28 | 31.98 | 22.82 | 67.07 | 54.25 |
| TIGERScore-7B | 0.64 | 3.24 | 12.89 | 4.36 | 66.62 | 38.21 | 44.52 | 27.34 | 52.49 | 52.83 |
| TIGERScore-13B | -2.31 | 3.39 | 15.45 | 4.54 | 68.91 | 45.47 | 53.04 | 42.86 | 44.13 | 56.28 |
| UltraCM-13B | 21.51 | 4.12 | 16.19 | 4.85 | 76.54 | 35.59 | 50.51 | 25.17 | 54.73 | 59.39 |
| Auto-J-13B | 36.05 | 4.21 | 17.69 | 5.62 | 80.02 | 50.64 | 53.06 | 52.06 | 75.61 | 67.41 |
| InternLM2-7B-Chat-SFT | 38.78 | 3.73 | 14.48 | 3.32 | 27.08 | 17.48 | 18.82 | 14.29 | 36.13 | 24.71 |
| + MultiCritiqueDataset-SFT | 58.15 | 5.71 | 19.33 | 5.78 | 89.49 | 62.60 | 57.04 | 51.85 | 79.51 | 75.15 |
| + MultiCritiqueDataset-RL | 63.28 | 6.07 | 19.26 | 6.33 | 89.36 | 60.56 | 61.51 | 57.76 | 79.32 | 75.66 |

## 5 Experimental Results

The overall experimental results are presented in Section 5.1.
We then demonstrate the superiority of our constructed MultiCritiqueDataset by comparing it with other datasets in Section 5.2.
Finally, we analyze the scaling phenomenon on these datasets in Section 5.3.

### 5.1 Overall Experimental Results

Table 1 demonstrates that both SFT and RL fine-tuning stages on our proposed MultiCritiqueDataset significantly improves the critique ability of the InternLM2-7B-Chat-SFT model.
Specifically, in the CriticEval subjective feedback evaluation ($\bm{F}_{\text{sub.}}$), the SFT stage alone yields an absolute improvement of 19.8%, with RL stage adding an additional 6.3% absolute improvement.
Consequently, our fine-tuned model not only significantly outperforms other 7B-13B instruction-tuned and critique-tuned LLMs but also uniquely surpasses GPT-3.5-Turbo model across all evaluation metrics.
Furthermore, in the CriticBench benchmark, our model’s critique ability even approaches that of advanced 70B instruction-tuned LLMs and GPT-4, highlighting its competitive performance.

**Table 2: Detailed results for mathematical and coding tasks in CriticEval.**
| Models | CriticEval ($\bm{F}_{\text{sub.}}$) |  |  |  |
| --- | --- | --- | --- | --- |
| Math<br>CoT | Math<br>PoT | Code<br>Exec | Code<br>NE |  |
| SFT | 4.64 | 5.21 | 4.72 | 5.56 |
| RL | 5.70 | 6.21 | 4.87 | 5.33 |

It can be observed that our model fine-tuned by RL stage has slightly lower objective revision evaluation score in CriticEval for mathematical and coding tasks than the SFT model (19.26 $<$ 19.33).
However, when we evaluate the textual critique quality ($\bm{F}_{\text{sub.}}$) for these tasks in CriticEval (Table 2), RL significantly improves the critique quality across most mathematical and coding tasks.
This observation suggests the inherent instability in evaluating revisions within mathematical and coding tasks.

Figure: Figure 2: The correlation between the number of training samples in the SFT dataset (from 1K to 256K) and critique ability. low, medium, high and full represent the models that are trained on critiques in MultiCritiqueDataset-SFT for low-, medium-, high-quality, and all three response qualities (full), respectively.
Refer to caption: x2.png

### 5.2 Comparison with Existing Critique Datasets

**Table 3: Comparison between our constructed MultiCri tiqueDataset-SFT and existing critique datasets. C-Bench is the abbreviation of CriticBench.**
| SFT Models | CriticEval | C-Bench |  |  |  |
| --- | --- | --- | --- | --- | --- |
| $\bm{F}_{\text{obj.}}$ | $\bm{F}_{\text{sub.}}$ | $\bm{R}_{\text{obj.}}$ | $\bm{R}_{\text{sub.}}$ | Overall |  |
| InternLM2-7B-Chat | 38.78 | 3.73 | 14.48 | 3.32 | 24.71 |
| + Auto-J | 45.44 | 3.56 | 14.63 | 3.47 | 67.76 |
| + UltraFeedback | 52.95 | 4.42 | 15.81 | 3.54 | 58.67 |
| + Feedback-Collection | 33.00 | 2.54 | 18.78 | 4.57 | 49.76 |
| + Ours | 58.15 | 5.71 | 19.33 | 5.78 | 75.15 |

We fine-tuned the InternLM2-7B-Chat-SFT model on each dataset individually.
Table 3 shows that the model fine-tuned on MultiCritique Dataset-SFT significantly outperforms those fine-tuned on other datasets, with 21.48% and 22.50% average performance gain on CriticEval (subjective feedback evaluation) and CriticBench, respectively.
Despite the fact that the data scales of Feedback-Collection and UltraFeedback datasets are over three and eight times larger than MultiCritiqueDataset-SFT, the superior results of ours indicate its better quality.

### 5.3 Scaling Phenomenon on Critique-tuned Datasets

Figure 2 illustrates two findings:
(1) The model trained on MultiCritiqueDataset-SFT consistently outperforms those trained on other datasets across most data scales, regardless of the response quality used;
(2) As the number of training samples increases, the critique ability of models fine-tuned with MultiCritiqueDataset-SFT improves steadily, leveling off beyond 12K samples.
Besides, models utilizing critiques encompassing all response quality types (full) outperform those trained on critiques of individual quality types, indicating that diverse response qualities benefit the generalization of critique ability.

## 6 Analyze

In this section, we perform several ablation studies to evaluate the contributions of:
(1) MultiCritique-SFT in collecting high-quality analytical critiques;
(2) Crucial Information in simplifying of critiques;
and (3) Multi-Agent-Revision-Scoring (MARS) filtering in improving the quality of preference critique pairs for the RL stage.

#### Ablation Study on MultiCritique-SFT

**Table 4: Ablation study on MultiCritique-SFT.**
| SFT Models | CriticEval |  |  |  |
| --- | --- | --- | --- | --- |
| $\bm{F}_{\text{obj.}}$ | $\bm{F}_{\text{sub.}}$ | $\bm{R}_{\text{obj.}}$ | $\bm{R}_{\text{sub.}}$ |  |
| MultiCritique-SFT | 59.74 | 5.17 | 20.92 | 6.05 |
| GPT-4-Turbo | 58.53 | 5.07 | 18.39 | 5.87 |
| Claude-1-instant | 56.77 | 5.01 | 19.00 | 5.79 |
| Qwen-1.5-72B | 57.30 | 4.89 | 17.74 | 5.81 |
| InternLM2-20B | 54.73 | 4.84 | 17.52 | 5.82 |

To examine the effectiveness of MultiCritique-SFT pipeline, we also fine-tune LLMs with analytical critiques generated by each model involved in MultiCritique-SFT data generation pipeline.
Table 4 demonstrates that models fine-tuned with critiques generated by MultiCritique-SFT outperforms those optimized with critiques from individual models.
Furthermore, there is a notable performance gap in the critiques generated by different models. For example, GPT-4 surpasses Qwen-1.5-72B-Instruct and InternLM2-20B-Chat, and all three outperforms Claude-1-instant, aligning with findings from CriticEval (Lan et al., ).

#### Ablation Study on Crucial Information

**Table 5: Ablation study on crucial information.**
| SFT Models | CriticEval |  |  |  |
| --- | --- | --- | --- | --- |
| $\bm{F}_{\text{obj.}}$ | $\bm{F}_{\text{sub.}}$ | $\bm{R}_{\text{obj.}}$ | $\bm{R}_{\text{sub.}}$ |  |
| Full | 58.15 | 5.71 | 19.33 | 5.78 |
| - w/o Task | 55.01 | 5.12 | 18.72 | 5.73 |
| - w/o Criteria | 57.28 | 5.46 | 19.12 | 6.17 |
| - w/o Ref. | 57.72 | 5.21 | 16.42 | 5.74 |
| - w/o All | 57.11 | 5.12 | 13.86 | 5.73 |

In this section, we evaluate the contributions of three crucial information in simplifying the critiques. Specifically, we remove each crucial information during training and evaluate them in the same setting.
Table 5 shows that removing each crucial information leads to a significant performance drop on most metrics in CriticEval.
This observation suggests that crucial information play a vital role in simplifying the critiques.
Interestingly, training without evaluation criteria (w/o Criteria) leads to the best performance on the subjective revision evaluation in CriticEval ($\bm{R}_{\text{sub.}}$). This observation suggests that while criteria benefit critiques, they might have side effects for revisions. We will explore the reasons behind this intriguing phenomenon in our future work.

#### Ablation Study on MARS Filtering

**Table 6: Ablation study on MARS filtering.**
| Models | CriticEval |  |  |  |
| --- | --- | --- | --- | --- |
| $\bm{F}_{\text{obj.}}$ | $\bm{F}_{\text{sub.}}$ | $\bm{R}_{\text{obj.}}$ | $\bm{R}_{\text{sub.}}$ |  |
| SFT Stage | 58.15 | 5.71 | 19.33 | 5.78 |
| RL Stage | 63.28 | 6.07 | 19.26 | 6.33 |
| - w/o MARS | 63.05 | 4.84 | 18.79 | 5.99 |

To examine the contribution of our proposed MARS pipeline, the SFT model is also fine-tuned by RL, guided by a reward model trained on the preference dataset without MARS filtering, denoted as w/o MARS.
Table 6 illustrates that exclusion of MARS pipeline results in a notable decline in performance.
For example, the model (w/o MARS) falls short of the SFT baseline on the subjective feedback evaluation (4.84 $<$ 5.71). This suggests that RL fine-tuning becomes instable without the MARS filtering, highlighting its contributions.

## 7 Conclusions and Future Works

In this paper, we propose a novel data generation pipeline, MultiCritique, to automatically construct the dataset to improve the critique ability of LLMs through SFT and RL fine-tuning stages.
Extensive experiments demonstrate that MultiCritique significantly surpasses existing datasets. Additionally, the RL fine-tuning stage on MultiCritique further improves the critique abilities of LLMs.
In the future, we plan to expand MultiCritique to the pairwise response comparison (Lan et al., ), enhancing further LLMs’ ability to evaluate paired responses.

## 8 Ethics Statement

Our paper proposes a novel automatic data generation pipeline associated with a critique dataset for the SFT and RL fine-tuning stages, named MultiCritiqueDataset.
As described in Appendix C, all the leveraged datasets in our work are from the publicly available datasets, which have been well processed to protect user privacy.
To ensure the diversity of our datasets, we source over 32.1K user queries from 123 diverse task scenarios, and each user query consists of three evaluated responses with varying qualities, like low-, medium, and high-quality.
We have proposed numerous components to ensure the quality of collected critiques, such as multi-agent revision scoring (MARS) filtering.
In the future, we will continue to improve the quality and enlarge the scale of our constructed MultiCritiqueDataset. For example, we plan to extend our MultiCritique data generation pipeline to the pairwise response comparison evaluation protocol. Besides, we also plan to collect more diverse mathematical and coding queries to enhance the critique ability of fine-tuned models on these challenging reasoning tasks.

## Appendix A Limitations

### A.1 Limitations in Critique Protocol

So far, there are two kinds of protocols for critiques (Li et al., ; Lan et al., ): (1) single-response evaluation; (2) pairwise response comparison. Our proposed MultiCritiqueDataset only critiques the quality of one response given the conversation history or user query, while the pairwise response comparison is not considered.
Our proposed data generation pipeline could be easily modified to collect high-quality datasets for pairwise response comparison.
In the future, we will extend our proposed MultiCritiqueDataset to pairwise response comparison protocols and evaluate the models under some preference benchmarks, like RewardBench (Lambert et al., ), PandaLM (Wang et al., ) and Preference Collection (Kim et al., ).

### A.2 Limitations in MultiCritique-SFT Pipeline

This work uses four LLMs to critique the evaluated responses in our proposed MultiCritique-SFT pipeline. The capability of these four models may be sub-optimal at present. Our work began in April 2024, and these four models exhibited strong critique capabilities at that time (Lan et al., ). In the future, we will continue to supplement more powerful models in the MultiCritique-SFT pipeline to improve the quality of MultiCritiqueDataset, such as Llama-3.1 and OpenAI o1 series.
Besides, our work only leverages four models, and more models in the MultiCritique-SFT pipeline would introduce more perspectives for critiques. Since the cost of critique and meta-critique continue to increase given more models, we only selected four models due to the limited budget. In the future, we will explore how the number of used models affects the quality of critiques in the MultiCritique-SFT pipeline.

Unlike the existing multi-agent framework Chan et al. (); Zhang et al. (), our proposed MultiCritique-SFT does not allow the inner-model debate and discussions. The reasons are as follows:
(1) The inference efficiency of inner-model debate is much lower. It not only costs a huge amount of API quota for evaluating each response but also slows down the data generation speed;
(2) Our preliminary study observes that other models easily influence LLM opinions during critique generation. This may cause the multiple LLM critiques to degrade into the opinions of one or two LLMs, and the inaccurate opinions could influence other LLMs, thereby reducing the comprehensiveness and diversity of the critiques.

In contrast, our preliminary study observes that GPT-4-based meta-critique could effectively calibrate the flawed critiques with correct critiques of other LLMs.
In future work, we will further address this issue to unlock the potential of multi-agent frameworks in the critique task.

### A.3 Limitations in MultiCritique-RL Pipeline

In MultiCritique-RL data generation pipeline, we introduce the Multi-Agent-Revision-Scoring (MARS) filtering to refine preference critique pairs obtained from the MultiCritique-SFT pipeline, using revision quality as an indicator of critique quality. Currently, revision quality is assessed by the InternLM2-20B-reward model (Cai et al., ), which, despite its strong performance on the RewardBench (Lambert et al., ) benchmarks, may not consistently reflect the true quality of responses across all task scenarios. The most reliable method would be human annotation, but it is costly and not scalable. Therefore, we adopt this trade-off approach to balance effectiveness and efficiency in measuring revision quality. In future work, we aim to enhance the accuracy of reward modeling, facilitating better MARS filtering.

### A.4 Limitations in Evaluation Efficiency

Compared to existing critique-tuned LLMs, like Auto-J and UltraFeedback, the models fine-tuned on our proposed MultiCritiqueDataset will generate longer sequences, consisting of the task description, two-tier criteria, reference response and critiques. Therefore, the inference efficiency of our fine-tuned models may be worse than that of existing works.
However, our ablation study in Section 6 reveals that crucial information is essential for more robust and accurate critiques. In future work, we will explore how to compress the crucial information and improve the inference efficiency.

## Appendix B Differences between Recent Works

This section will discuss the primary differences between our designed data generation pipeline and existing works, like Prometheus (Kim et al., ).

#### Difference in Data Preparation

Although Prometheus collects five responses with quality scores ranging from 1 to 5 (Kim et al., ), these responses are synthesized using a GPT-4 reference response, leading to responses that are very similar to the reference, which is a significant deviation from real-world scenarios.

#### Difference in Crucial Information

Although Prometheus also employs customized criteria for better critiques (Kim et al., ), our work differs significantly.
Our evaluation criteria are organized into a hierarchical two-tier structure, providing clear definitions for diverse evaluation dimensions—a method proven effective in automatic evaluation (Lee et al., ; Liu et al., ).
In contrast, Prometheus synthesizes one criterion using GPT-4, lacking sufficient guidelines for high-quality reference response and critique generation.

## Appendix C Implementation Details

### C.1 MultiCritiqueDataset Construction

#### Query Preparation

All the queries in Auto-J (Li et al., ) and DEITA (Liu et al., ) are collected.
For OpenHermes-2.5(^2^22[https://huggingface.co/datasets/teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5)), we sample 1K queries for its 28 categories, leading to 28K queries.
Following previous work (Yuan et al., ), we use 3.2K examples from the OpenAssistant dataset by sampling only the first conversation turns in the English language that achieves the highest human-annotated scores.
Besides, we also sample 2K mathematical and coding questions from MetaMathQA (Yu et al., ) and CodeFeedback (Zheng et al., ) datasets to collect critiques for reasoning tasks. Only the first conversation utterance (the coding question) in CodeFeedback is used.
None of the training samples are from the test set in CriticEval and CriticBench benchmarks.

#### Collect Evaluated Responses

**Table 7: The average reward model scores for each response quality.**
| Resp. Quality | Avg. |
| --- | --- |
| Low | -1.41 |
| Medium | 0.70 |
| High | 1.69 |

To collect diverse evaluated responses for queries, we use eleven widely-used LLMs with varying scales and capabilities in this work: (1) Qwen-1.5-72B-Chat; (2) Qwen-1.5-7B-Chat; (3) InternLM2-20B-Chat; (4) Yi-34B; (5) Mixtral-8x7B-Instruct; (6) Llama2-13B-Chat; (7) Llama2-7B-Chat; (8) Gemma-2B; (9) Baichuan2-13B-Chat; (10) Vicuna; (11) WizardLM-7B-v0.1.
The LMDeploy tookit (Contributors, ) is used to inference these LLMs by random sampling decoding method, and the hyper-parameters are 0.95 top-p and 0.8 temperature.
Besides, the InternLM2-20B-reward (Cai et al., ) model is used to score the quality of responses, and the reward scores are used to classify responses into three quality levels.
The average reward scores for each response quality are shown in Table 7. It can be observed that there exists a significant performance gap among these response qualities.

Given that reward models fail to accurately evaluate the quality of responses in mathematical and coding questions, we only collect two kinds of response qualities: (1) high-quality responses generated by GPT-4o and (2) low-quality responses generated by eight 7B-20B open-source LLMs.

#### Collect Crucial Information

The prompt for LLMs to generate task description, two-tier structured criteria and reference response are described in Appendix H.
Our preliminary study reveals that reference responses tend to produce critiques that lack diversity for mathematical and coding questions. As a result, we set reference responses as empty for these two tasks.

Most previous works rely on human-annotated criteria for each task (Hu et al., ; Li et al., ), which do not scale well.
We propose generating a customized two-tier structure evaluation criteria tailored to each query using GPT-4.
Besides, the user pre-defined criteria are provided as input optionally for better flexibility.

#### Pre-process Evaluated Responses

Our proposed ACUs contain the location of flaws in the evaluated response for better interpretability.
To achieve this goal, we pre-process the evaluated responses by appending labels for sentences in evaluated responses.
For most tasks, punctuation marks such as periods, exclamation marks, and semicolons are used to divide sentences.
For code-related task scenarios, the sentence is divided by the line breaks to represent lines of the evaluated code.

#### Collect Preference Dataset

The threshold of differences in accumulated severity scores is set as 5 in this paper.
Besides, we leverage four additional 7B LLMs to revise the evaluated response eight times, given the model-generated critiques: (1) InternLM2.5-7B-Chat; (2) Llama-3.1-8B-Instruct; (3) Qwen2-7B-Chat; (4) Mistral-7B-Instruct.
The random sampling decoding method is used to generate diverse revisions, and the hyper-parameters are (1) 0.95 top-p, (2) 50 top-k, and (3) 1.0 temperature.
The vLLM toolkit (Kwon et al., ) is used to speed up the inference.

### C.2 Experimental Details

#### SFT

During the SFT training stage, the InternLM2-7B-Chat model is fine-tuned by optimizing the Maximum Likelihood Estimation (MLE) loss:

$$ $L_{\text{MLE}}=-\frac{1}{\mathbb{N}}\sum_{i=1}^{\mathbb{N}}\log p_{\theta}( \mathcal{CI}_{i},\mathcal{C}_{i}|q_{i},r_{i})$ (2) $$

The training process is running on 2 A800 GPU Serves (16 GPUs) by using DeepSpeed(^3^33[https://github.com/microsoft/DeepSpeed](https://github.com/microsoft/DeepSpeed)). To achieve a fair comparison, we set the training hyper-parameters as follows: (1) 4e-5 learning rate; (2) 6e-6 minimum learning rate; (3) 32,768 maximum sequence length; (4) 2 epoch; (5) 1 batch size; (6) AdamW optimizer.

We explore the effect of instruction format and data recipe of crucial information during the SFT stage in Appendix G. We fix the following experimental setup for supervised fine-tuning: (1) the proportion of the single-turn template is 5% and left 95% training samples for SFT are multi-turn conversations, consisting task description, two-tier structured evaluation criteria, reference responses, critiques consisting of a list of ACUs generated by MultiCritique-SFT pipeline and summarization of the final judgment for the evaluated response; (2) the crucial information for each query is only optimized once in 2 epochs.

In Section 6, we analyze the contributions of our proposed MultiCritique-SFT pipeline. We only collect the summarization of final judgments for the critiques generated by MultiCritique-SFT , and the critiques generated by each LLM do not have the corresponding summarizations. Thus, in this experiment, we do not fine-tune the model to predict the summarization of final judgments in 95% multi-turn training samples.

#### Reinforcement Learning

During the reinforcement learning stage, we first train the InternLM2-7B-Chat as the reward model on MultiCritiqueDataset-RL by using xtuner toolkit (Contributors, ), and the hyper-parameters are as follow: (1) 32,768 maximum sequence length; (2) 1 epoch; (3) 1 batch-size; (4) AdamW optimizer; (5) 2e-5 learning rate; (6) focal loss (Lin et al., ).
For the $i$-th sample, the focal ranking loss is computed to optimize the reward model:

$$ $L_{\text{ranking }}=-(1-2\times\max(0,P_{j_{+},j_{-}}^{i}-\frac{1}{2}))^{2} \log(P_{j_{+},j_{-}}^{i})\text{,}$ (3) $$

where $P_{j_{+},j_{-}}^{i}=\sigma(r_{j_{+}}^{i}-r_{j_{-}}^{i})$ represents the probability that the reward score of $C_{i,j_{+}}$ is greater than that of $C_{i,j_{-}}$.
The difficulty decay coefficient only takes effect when the model correctly predicts the preference
of $i$-th training sample, i.e., $P_{j_{+},j_{-}}^{i}>0.5$, otherwise it equals to 1.

Subsequently, we conduct the PPO algorithm to optimize the SFT model on six nodes of A800 GPU servers (48 GPU cards) with the ray toolkit.(^4^44[https://github.com/ray-project/ray](https://github.com/ray-project/ray)) The hyper-parameters during reinforcement learning are listed as below: (1) 30,000 maximum sequence length; (2) 64 batch-size; (3) deepspeed zero-2; (4) 0.9 top-p and 1.0 temperature sampling parameters for policy model.

#### Evaluation

We leverage the publicly available codebase of CriticEval and CriticBench for evaluation.
To ensure the robust objective evaluation of the revision critique dimension, we leverage five LLMs with varying capabilities to revise the responses given feedback generated by each baseline: InternLM2-7B-Chat, InternLM2.5-7B-Chat, InternLM2-20B-Chat (Cai et al., ), Mixtral-7x8B-Instruct (Jiang et al., ) and Llama-3.1-70B-Instruct.
Due to the limited OpenAI API budget, we only conduct the subjective evaluation on the revision dimension to evaluate the quality of revisions generated by the Llama-3.1-70B-Instruct model.

In CriticBench benchmark, the responses with $\geq 7$ Likert Scores generated by our fine-tuned models are treated as the positive samples since responses with $\geq 7$ are comparable or better than the reference answers in our defined score rubrics, which is described in Appendix H.6.
The responses with $>2$ quality scores are treated as positive samples for the Prometheus model since the overall score range is 1 to 5.

To ensure reproducibility, the greedy search decoding strategy is used for inference.
As for the models we fine-tuned on our proposed MultiCritiqueDataset, the optional user pre-defined criteria are empty during inference.

## Appendix D Statistics of MultiCritiqueDataset

The statistical information of our proposed MultiCritiqueDataset is shown in Table 8. Our proposed MultiCritiqueDataset significantly outperforms existing critique datasets from multiple dimensions, like response quality and the number of tasks.
Although the size of UltraFeedback and Feedback-Collection are greater than our proposed MultiCritiqueDataset, the models fine-tuned on them are much worse than that fine-tuned on MultiCritiqueDataset, demonstrating the better quality of our proposed dataset.
Although Feedback-Collection and Preference-Collection consist of 5 response qualities, they are synthesized by GPT-4, resulting in very similar content with reference responses.

**Table 8: The comparison between our proposed MultiCritiqueDataset and existing critique datasets. Avg. Turn represents the average number of utterances in the multi-turn conversation history, and the user query is the last utterance in it. The number of tokens is counted based on the InternLM2-7B-Chat tokenizer.**
| Dataset | Type | Task<br>Desc. | Criteria | Ref. | Tokens | Resp.<br>Quality | Num.<br>Task | Num.<br>Query | Num.<br>Resp. | Avg.<br>Turn | Public |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Auto-J | SFT | ✗ | ✗ | ✗ | 3.8M | - | 58 | 4.4K | 4.4K | 1 | ✓ |
| UltraFeedback | SFT | ✗ | ✗ | ✗ | 227M | - | 9 | 257K | 257K | 1 | ✓ |
| TIGERScore | SFT | ✗ | ✗ | ✗ | 23.7M | - | - | 42.5K | 42.5K | 1 | ✓ |
| Feedback-Collection | SFT | ✗ | ✓ | ✓ | 191.5M | 5 | - | 20K | 100K | 1 | ✓ |
| Preference-Collection | SFT | ✗ | ✓ | ✓ | 382.9M | 5 | - | 40K | 200K | 1 | ✓ |
| Themis | SFT,RL | ✗ | ✓ | ✗ | - | - | 9 | 67K | 67K | 1 | ✗ |
| JudgeLM | SFT | ✗ | ✗ | ✓ | - | - | - | 100K | 200K | 1 | ✓ |
| MultiCritiqueDataset-SFT | SFT | ✓ | ✓ | ✓ | 531.1M | 3 | 123 | 10.7K | 32.1K | 2.40 | ✓ |
| MultiCritiqueDataset-RL | RL | ✓ | ✓ | ✓ | 352.9M | 3 | 123 | 19.7K | 39.4K | 2.35 | ✓ |

The complete list of the task scenarios in our proposed MultiCritiqueDataset is shown in Table 9, consisting of 123 tasks.
Except for 58 fine-grained tasks defined in Auto-J (Li et al., ), our proposed dataset includes 65 categories defined in the OpenHermes-2.5 dataset.

**Table 9: The complete list of task scenarios in our proposed MultiCritiqueDataset-SFT. The number of samples is also listed.**
| Task | Num. | Task | Num. | Task | Num. |
| --- | --- | --- | --- | --- | --- |
| default | 9362 | math reasoning | 6228 | code generation | 5280 |
| explaining general | 3452 | open question | 3048 | seeking advice | 2674 |
| value judgement | 2586 | roleplay | 1210 | functional writing | 958 |
| verifying fact | 838 | brainstorming | 828 | analyzing general | 780 |
| code correction<br>rewriting | 720 | chemistry | 718 | physical | 710 |
| chitchat | 702 | bio | 702 | asking how to question | 690 |
| creative writing | 632 | rejecting | 540 | planning | 538 |
| counterfactual | 528 | awareness | 480 | editor | 480 |
| misconception | 480 | general | 480 | cot | 480 |
| experience | 480 | song | 480 | plan | 480 |
| joke | 480 | rp | 480 | multiple choice | 480 |
| trivia | 480 | counterfactual<br>contextual | 478 | stylized response | 478 |
| theory of mind | 478 | writing | 478 | greeting | 478 |
| orca | 478 | riddle | 478 | wordgame | 478 |
| gtkm | 468 | recommendation | 462 | solving exam question<br>without math | 456 |
| coding | 452 | writing personal essay | 432 | text summarization | 430 |
| summarization | 424 | explaining code | 408 | agent | 406 |
| text to text translation | 396 | writing email | 372 | question generation | 372 |
| card | 372 | instructional rewriting | 360 | ranking | 358 |
| writing song lyrics | 318 | writing cooking recipe | 314 | information extraction | 300 |
| post summarization | 300 | data analysis | 294 | writing job application | 294 |
| writing presentation script | 292 | classification<br>identification | 276 | solving exam question<br>with math | 276 |
| paraphrasing | 240 | detailed writing | 222 | writing advertisement | 142 |
| writing social media post | 138 | title generation | 132 | text correction | 120 |
| language polishing | 114 | writing product description | 108 | writing blog post | 96 |
| code to code translation | 92 | writing legal document | 90 | writing technical document | 74 |
| reading comprehension | 66 | text simplification | 60 | writing scientific paper | 48 |
| keywords extraction | 40 | writing marketing materials | 36 | topic modeling | 18 |
| writing news article | 18 | quiz | 18 | writing chapter | 16 |
| code simplification | 12 | note summarization | 12 | writing letter | 12 |
| writing history essay | 6 | predicting general | 6 | writing feature story | 6 |
| criticism | 6 | challenges | 6 | writing social<br>responsibility report | 6 |
| impact | 6 | impact analysis | 6 | changing mindset | 6 |
| overview | 6 | writing consumer complaint | 6 | writing dialogue | 6 |
| writing sequel | 6 | writing historical document | 6 | exit planning | 6 |
| writing screenplay | 6 | writing deployment script | 6 | data conversion | 6 |
| time zone conversion | 6 | language history | 6 | writing press release | 6 |
| writing survival manual | 6 | writing movie review | 6 | writing biography | 6 |
| reward | 6 | writing comedy skit | 6 | writing note | 6 |
| writing love note | 6 | writing love letter | 6 | writing config file | 6 |
| writing script | 6 | writing kubernetes<br>deployment file | 6 | writing code | 2 |

The overall quota for using OpenAI and Claude API to construct our proposed MultiCritiqueDataset are 9,180$ and 125.6$, respectively.
The average API cost for each sample is 0.29$. Given that the average price of one human-annotated critique is 8$ (Wang et al., ), our data generation pipeline is much cheaper and easier to scale to more diverse task scenarios.

## Appendix E Excluded Baselines and Benchmarks

### E.1 Excluded Baselines

Some baselines are excluded during our evaluation, and the reasons are described as follows:
(1) InstructScore (Xu et al., ) is trained on samples with limited tasks, failing to extend to other diverse tasks, like mathematics reasoning and code generations;
(2) JudgeLM (Zhu et al., ) is mainly trained to compare two responses with critiques. Although it can be used to score the single responses, the reference responses should be supplied(^5^55[https://github.com/baaivision/JudgeLM/tree/main/judgelm/llm_judge](https://github.com/baaivision/JudgeLM/tree/main/judgelm/llm_judge)), which are unavailable in CriticEval and CriticBench;
(3) Reward models (Lambert et al., ) are also widely used to evaluate the quality of responses. However, their scores can only reflect the relative differences in response quality, so reward models cannot be assessed in CriticBench. Additionally, due to the lack of textual critiques, reward models are unsuitable for evaluation under CriticEval.

### E.2 Excluded Benchmarks

Existing benchmarks for evaluating the critique ability of LLMs could be classified into two categories: (1) single-response evaluation; (2) pairwise response comparison (Li et al., ; Kim et al., ).
Single-response evaluation aims to evaluate the quality of a single response given the context of the conversation or user query. For example, CriticEval and CriticBench evaluate whether LLMs could accurately score the quality of responses.
Pairwise response comparison selects the better response from a pair of responses. For example, RewardBench (Lambert et al., ), Feedback-Bench (Kim et al., ) and PandaLM test set (Wang et al., ) consist of numerous pairs of responses with clear performance gap.
Since pairwise response comparison is much simpler than comparing the scores corresponding to the two responses, it is unfair for our models under the pairwise response comparison benchmarks.
Therefore, these benchmarks are not used in our paper.
We will extend our proposed MultiCritiqueDataset from single-response evaluation to the pairwise response comparison (Li et al., ).

## Appendix F Preliminary Study on Crucial Information

During designing our data generation pipeline, we conducted a preliminary study to verify whether crucial information helps reduce the complexity of critique tasks and improve the quality of collected critiques.
Specifically, we conducted the self-critique prompting (Pan et al., ) to collect critiques and corresponding revisions and evaluated the quality of the critiques by measuring the quality of their corresponding revisions.

### F.1 Experimental Setup

We first random sample 1,280 queries and evaluated responses from MultiCritiqueDataset.
Then, four LLMs are prompted to generate critiques and subsequent revisions with or without each crucial information: (1) InternLM2.5-7B-Chat; (2) Qwen2-7B-Chat; (3) Llama-3.1-8B-Instruct; and (4) Mixtral-7B-Instruct.
Each model generates critiques and revisions eight times, leading to overall $4\times 8=32$ revisions, and the advanced InternLM2-20B-reward model judges the quality of these revisions.

### F.2 Experimental Results

**Table 10: Avg. rewards.**
|  | Reward |
| --- | --- |
| Ref. w/ Criteria | 0.54 |
| Ref. w/o Criteria | 0.45 |

First of all, as shown in Table 10, it can be found that the
quality of reference responses generated given the customized evaluation criteria is much better, indicating the effectiveness of our proposed two-tier structure evaluation criteria.

**Table 11: Avg. reward scores.**
|  | Reward |
| --- | --- |
| Origin Response | -0.11 |
| w/ All | 0.085 |
| w/o Task | 0.076 |
| w/o Ref. | 0.029 |
| w/o All | -0.005 |

Besides, as shown in Table 11, it can be found that the quality of revisions becomes worse when task descriptions and reference answers are removed. Besides, removing all the crucial information leads to the worst performance (-0.005 $<$ 0.085). Note that we do not evaluate the contributions of customized evaluation criteria since its contribution is proven in Table 10, i.e., improves the quality of reference responses.

## Appendix G How Factors Affect Performance During SFT Stage

In this section, we analyze two factors that influence the performance of models fine-tuned on our proposed MultiCritiqueDataset-SFT: (1) instruction format; and (2) the data recipe of crucial information.

### G.1 Instruction Format

**Table 12: The propor tion of single turn prompts.**
| Rate | $\bm{F}_{\text{obj.}}$ |
| --- | --- |
| 1.0% | 61.14 |
| 2.5% | 57.32 |
| 5.0% | 63.85 |
| 10.0% | 60.21 |

Our proposed MultiCritiqueDataset-SFT consists of mult-turn conversations for generating critiques.
To ensure the generalization of fine-tuned models, in this paper, we construct the single-turn and multi-turn prompt templates in the instruction dataset for the supervised fine-tuning (SFT) stage. Our experimental results reveal that the proportion of single-turn and multi-turn templates in the training data significantly affects the model’s performance. As shown in Table 12, it can be observed that when the proportion of single-turn templates is 5% of the total data size, the fine-tuned models could achieve optimal performance on the feedback objective evaluation of CriticEval during the SFT stage.
Therefore, this setting is used in all the experiments in our paper.

### G.2 Data Recipe of Crucial Information During SFT stage

**Table 13: The proportion of training volume on crucial information.**
| Rate | $\bm{F}_{\text{obj.}}$ |
| --- | --- |
| 16.67% | 63.85 |
| 33.33% | 60.19 |
| 66.67% | 56.99 |
| 100.0% | 57.47 |

As described in Appendix D, our proposed MultiCritiqueDataset-SFT consists of 32.1K evaluated responses and 10.7K queries, and the same query has the same crucial information: task description, criteria, and reference response. Therefore, the training volume on crucial information will be three times larger than that of critiques. This might lead to overfitting crucial information, influencing the optimization of critiques.
To address this problem, we mask the loss of crucial information at varying rates.
As shown in Table 13, it can be found that the performance of fine-tuned model on CriticEval benchmark decreases when the proportion of training volume on crucial information increases, and the best proportion of training volume is 16.67%, i.e., the crucial information for each query is only optimized once in 2 epochs.
We leverage this experimental setting in all our experiments.

## Appendix H Designed Prompts in MultiCritique

In this section, we provide the detailed prompts that used in our proposed MultiCritiqueDataset data generation pipeline.

### H.1 Task Description

The prompt for GPT-4 model to generate the task description is shown in Figure 3, while the multi-turn conversations are not provided.

Figure: Figure 3: The prompt for generating task description about the last user query in conversation.

### H.2 Criteria Generation

The prompt for GPT-4 to generate the two-tier structured criteria is shown in Figure 7.
Note that the user could provide their pre-defined criteria.
If the criteria provided by users are not empty, GPT-4 is asked to generate the two-tier structured criteria from scratch; otherwise, GPT-4 is asked to expand on the criteria provided by the user and must not generate content that conflicts with the user’s provided criteria.
Besides, it can be found that each item of criteria consists of 3 fundamental values: (1) criteria name, (2) criteria fine-grained description, and (3) importance degree of the criteria (normal, medium, important).

### H.3 Reference Generation

Given the two-tier structured criteria, GPT-4 is asked to generate high-quality reference responses that satisfy all the evaluation criteria, as shown in Figure 4.

Figure: Figure 4: The prompt for generating reference response given the criteria.

### H.4 Multi-Agent Analytical Critique

After generating the three crucial information, multiple LLMs are asked to follow the instructions in Figure 8 to critique the evaluated responses.
It can be found that LLMs are asked to critique the evaluated responses sentence by sentence and generate a list of Analytical Critique Units (ACUs) consisting of 5 key values: (1) citation symbol of the sentence in evaluated response; (2) description of this flaw; (3) which criteria this flaw belongs to; (4) severity of this flaw; (5) revision suggestions.

### H.5 Meta-Critique Classification

As shown in Figure 5, after collecting multiple critiques generated by LLMs, the GPT-4 model is asked to conduct the meta-critique to analyze the quality of each ACU. Each ACU is classified into seven categories.

Figure: Figure 5: The prompt for generating meta-critiques for all the critiques generated by multiple LLMs.

The detailed descriptions of each meta-critique and corresponding severity score are shown in Table 14.

**Table 14: Our human-annotated quality categories of ACUs. A higher severity score indicates the worse performance of corresponding ACUs.**
| Label | Meaning | Detailed Description of Quality Category | Severity<br>(1-5) |
| --- | --- | --- | --- |
| L0 | Correct ACU | This feedback is accurate and provide helpful suggestions. | 0 |
| L1 | False Negative ACU | The content is free from any flaws and should not be critiqued. | 5 |
| L2 | Wrong Criteria | The type of criteria of feedback is misclassified or does not exist. | 2 |
| L3 | Wrong Severity | The severity of this flaw is misclassified. | 1 |
| L4 | Wrong Description | The descriptions of flaws are unreasonable or inaccurate. | 4 |
| L5 | Wrong Suggestion | The suggestions for revisions are unreasonable or introduce errors. | 4 |
| L6 | Unhelpful Suggestion | Revision suggestions are reasonable but not helpful. | 1 |

### H.6 Critique Summarization

Finally, the GPT-4 model is asked to summarize the critiques from multiple LLMs and generate the final critiques and summarization for the evaluated responses. As shown in Figure 6, it can be found that the judgment scores for evaluated responses are the floating numbers ranging from 1 to 10, and the $\geq 7$ scores indicate the comparable and even better qualities of responses than reference responses.

Figure: Figure 6: The prompt for generating final critiques and summarization for the evaluated responses, which is used for the supervised fine-tuning stage.

Figure: Figure 7: The prompt for generating two-tier structured criteria. We also allow user to input their specific evaluation criteria.

Figure: Figure 8: The prompt for generating analytical critiques for evaluated responses.

## Appendix I Case Study in MultiCritiqueDataset

### I.1 Case Study of Customized Evaluation Criteria

We provide one case of two-tier structured evaluation criteria for one query in MultiCritiqueDataset in Figure 9. Compared with existing works, like Themis and Auto-J, our evaluation criteria contain a more diverse and customized evaluation dimension for the user query, which is beneficial for robust and accurate evaluation.

### I.2 Case Study of Critiques

We provide one case of analytical critique units (ACU), summarization, and judgment of critiques in Figure 10. Each ACU points out one flaw in a located sentence in the evaluated responses.

Figure: Figure 9: One case of customized evaluation criteria for one query. ## represents the first tier criteria and ### represents the second tier criteria, which is a more fine-grained and specific evaluation criteria for the first tier criteria.

Figure: Figure 10: One case of critiques in MultiCritiqueDataset-SFT. Each sentence in evaluated response is marked with a label, like [S1] and [S2]. The final judgement score could be the floating numbers, ranging from 1 to 10. Except for flaws in each sentence, GPT-4 also locate flaws across sentences, like the “Moderate —Style” flaw across sentence 2 and sentence 3 “[S2] - [S3]”.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 8
presentation: 3
questions: - Table 4 is somewhat unclear to me. Could you elaborate on what was specifically done in this ablation study? Are you fine-tuning from the same baseline models and then using different datasets, such as MultiCritique-SFT versus critiques generated solely by GPT-4-turbo and other individual models? Additionally, are these individual critiques extracted from the MultiCritique dataset, or are they generated separately?
soundness: 3
strengths: - The paper is well-written and effectively presented, with technical details that are clearly explained and easy to follow. Figure 1 is particularly effective in illustrating the entire MultiCritique pipeline.
- The evaluations and ablation studies are comprehensive, providing a thorough analysis of the model's performance and the impact of various components.
- The MultiCritique SFT dataset, if made open-source, could be a good contribution to the field due to its careful construction aimed at reducing variances and biases.
- The limitations section is well-prepared, addressing most concerns that might arise from the study.
summary: This paper proposes a critique data generation pipeline called MultiCritique to enhance the critique capabilities of language models. The process is divided into three steps. In Step 1, the authors selected queries from six datasets and used 11 language models to generate responses. These responses were rated by InternLM2-20B-reward into three quality levels, and GPT-4 was used to extract and summarize crucial information (CI) for each query-response pair. In Step 2, the query-response-CI tuples were input into multiple language models to generate aspect-specific critiques, referred to as Analytical Critique Units (ACUs). GPT-4 then classified these ACUs into seven quality categories and aggregated them into a final critique summary, which could be used for SFT of language models. In Step 3, the ACUs generated in Step 2 were used to construct preference pairs, which were then employed for reinforcement learning fine-tuning using PPO.
The evaluation of this pipeline was conducted on two benchmarks: CriticEval (unpublished) and CriticBench (ACL Findings '24).
weaknesses: - Dependence on GPT-4: The project heavily relies on GPT-4 for classifying and summarizing critiques, which may introduce biases into the final critique summaries. It is not hard to say that the entire project is feasible because of the existence of GPT-4. This reliance potentially undermines the benefits of using multiple agents in earlier steps. I know it might be a bit too much to ask, but it would be interesting to explore the impact of using a different model, such as Claude 3.5 Sonnet, to see how the results might differ.
- Specificity Towards Critique Ability: The model's focus on improving critique ability raises concerns about overfitting to these specific tasks. It would be beneficial to assess whether this specialization affects other capabilities, such as chat performance. Evaluating the model on benchmarks like AlpacaFarm Arena-Hard could provide insights into any potential degradation in general capabilities.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - What is the generalizabilty of this kind of fine-tuning of the critiquing ability. Can the authors show whether finetuning on critiquing dataset generated from general alignment datasets (like OpenHermes, OpenAssistant) alone can still help in generating critiques in unseen domains like math and code.
- The performance improvement of the MultiCritiqueDataset-RL over MultiCritiqueDataset-SFT is not very clear - for some datasets there seems to be improvement while for others it is hurting or improvement is negligible. Can the authors comment on why this happening? Also what about including both MultiCritiqueDataset -SFT and -RL?
- Minor concern about terminology - I am not quite sure why the authors stress on “agent” feedback — this work on critique generation seems like yet another instruction tuning task, this in general can be helpful in agentic tasks but I don’t see why this is termed “multi-agent”
soundness: 3
strengths: - It is a Practically useful work
- Good improvement in results in CriticEval and CriticBench benchmarks
summary: This paper proposes a data generation pipeline utilising some of the SoTA LLMs (OpenAI, Claude, Qwen etc) both for SFT and RL tuning in order to improve the critiquing ability of LLMs. The authors experiment on CriticEval and CriticBench, where their SFT and RL datasets provides a good boost in performance.
weaknesses: - Mostly this paper feels like just a bit of a formalisation for distilling “critiquing” ability from GPT-4 by generating critiquing data both for SFT and RL tuning. The data generation process is also quite heavily dependent on GPT-4 abilities.
- Novelty of this work is not very clear to me

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - No major questions, I think the paper ideas a straightforward and easy to follow, the presentation is good, and the ideas are well executed
- From a research perspective, I think novelty is lacking a bit, neverthless it is an important contribution
soundness: 3
strengths: - overall addressing an important problem of developing small high qulaity critique model
- new multi-agent feedbakc approach to curate dataset
- creation of a new high qulaity multi-critique dataset, incorporating diverse queries,and crucial information collection
- naturla integration and use of RL
- Strong experimental valiadtion, showing 7b FT models perform at par or better with closed source and 70b model on criticeval and critic bench
summary: The paper, tackles the inherent difficulty in enhancing critique capabilities within large language models (LLMs). Critique ability, a key aspect of human meta-cognition, is essential for LLMs as it allows them to identify and address flaws in responses. This skill is pivotal for reliable automatic evaluation and model self-improvement. However, training models to critique effectively poses challenges. Most existing methods rely on supervised fine-tuning (SFT) using critiques generated by a single LLM, typically a model like GPT-4. Although effective to a degree, these single-model critiques can propagate the original model’s biases and limitations, leading to inaccuracies and diminished critique quality.

To address these issues, the authors introduce a novel data generation pipeline, MultiCritique, that uses multi-agent feedback to enhance the quality and diversity of critiques. Rather than depending on a single model’s perspective, MultiCritique aggregates feedback from multiple advanced LLMs, allowing for a richer, more balanced dataset. This multi-agent approach is implemented in both the SFT and reinforcement learning (RL) stages, producing a dataset called MultiCritiqueDataset. With this dataset, the model undergoes fine-tuning and then reinforcement learning, where preference scoring further refines the critique quality based on feedback from several agents.

The study’s experiments show that a 7B model fine-tuned with the MultiCritiqueDataset significantly outperforms other models in critique tasks, even rivaling the performance of much larger, closed-source models like GPT-4. The authors argue that the multi-agent approach not only yields a higher-quality critique dataset but also preserves feedback diversity, making it a robust solution for training LLMs in critique ability. This advancement offers an effective pathway toward more self-improving, reliable, and critique-capable language models
weaknesses: - not very technically novel, looks like a fairly stratightforward principle of ensembling for dataset collection
- Performance is still limited by an upperbound of the the base LLMs, and it is not clear if we could sclae to 70b, would the performance be better

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. While the pipeline effectively aggregates critiques, the reliance on multiple LLMs (including GPT-4) for meta-critique classification may introduce considerable computational and financial overhead. Could you provide quantitative result and  a discussion on efficiency, potential trade-offs, or future work focused on optimizing this aspect?

2. Although the multi-agent framework is well-justified, could you provide a more explicit differentiation from other multi-agent critique frameworks (e.g., Arena learning and Stable Alignment). Clearer explanations on how MultiCritique uniquely contributes to critique generation and feedback quality.
soundness: 3
strengths: 1. The paper introduces a multi-agent framework that significantly improves critique dataset quality by aggregating and refining feedback from diverse LLMs. The structured use of Analytical Critique Units (ACUs) and meta-critique classification addresses the limitations of single-agent feedback and provides a robust critique foundation. The structured use of Analytical Critique Units (ACUs) and meta-critique classification addresses the limitations of single-agent feedback and provides a robust critique foundation.

2. The MultiCritiqueDataset improves LLM critique ability considerably over traditional supervised datasets, evidenced by impressive gains on CRITICEVAL and CRITICBENCH

3. The paper has quite extensive evaluation and analysis that support the method.
summary: The paper proposes MultiCritique, a novel data generation pipeline aimed at enhancing language models' (LLMs) critique abilities by leveraging multi-agent feedback during supervised fine-tuning (SFT) and reinforcement learning (RL) stages. The proposed MultiCritique pipeline aggregates critiques from multiple advanced LLMs, utilizing key meta-critique processes to classify and consolidate high-quality feedback. This results in the construction of a new MultiCritiqueDataset, which shows empirical improvements on CRITICEVAL and CRITICBENCH benchmarks, demonstrating superior performance in critiquing abilities. Notably, the approach effectively competes with models up to ten times its size, approaching the critique capabilities of models like GPT-4.
weaknesses: See question.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See above.
soundness: 2
strengths: The authors leverage multi-agent feedback to train models capable of critique, claiming that multi-agent feedback provides more accurate and critical information, which could address the inherent complexity of critique. However, the reviewer has expressed doubts about this claim. While the experimental setup is fairly robust, the descriptions of the experimental settings lack detail in most parts.
summary: Overall, this paper is poorly written, describing certain motivations and methods that are difficult to understand.

> For example, in the abstract, the authors emphasize that the inherent complexity of model-generated critiques limits model performance, which they cite as a primary motivation for this work. However, MultiCritique originates from Multi-agent Feedback, which clearly adds to this complexity by involving multiple models.
weaknesses: 1. In terms of research on critique, other researchers have already explored similar ideas with LLMs. For instance, Jan Leike at OpenAI proposed "Self-critiquing models for assisting human evaluators," which this paper does not discuss. Additionally, while the authors discuss RLHF, they surprisingly omit foundational literature on RLHF, such as *Training language models to follow instructions with human feedback*. It appears that the authors did not thoroughly investigate relevant literature, which results in some contributions being overstated.
   
2. The terms “meta-cognitive” and “critique as a greater challenge than criticism” are ambiguous. Could the authors clarify these?

3. In Line 173, the authors state, "In total, we sample 10.7K queries covering 123 diverse task scenarios." However, the reviewer does not know what these 123 tasks entail or where to find this information in the appendix in this paragraph.

4. The authors use InternLM2-20B for scoring. How accurate is this reward model in the authors’ study, and how does it align with human evaluations at the three labeled response levels? Furthermore, Section 3.1 lacks a clear definition of these three levels.

5. In Section 4.1, the authors present Equation 1, where they condition \( Q_i \) and \( R_i \) to generate \( C_I \) and \( C \). How does this differ from the fine-tuning approach in the Aligner [3] paper, or are the two approaches identical?

6. In the experiments, I noticed that the authors applied SFT on InternLM2-7B-Chat-SFT, essentially layering SFT on a model that was already SFT-trained. How did they address potential issues of hallucination in this setup? Moreover, why did they not consider using Llama3-8B-Instruct as a base model?

7. This paper is based on multi-agent feedback, but it is unclear how the multi-agent approach is implemented. My understanding is that the feedback primarily comes from multiple language models. How does this approach differ from the multi-agent debate setups described in [4][5]? If the multi-agent approach in this paper merely relies on feedback from multiple LLMs without any designed communication between the agents, it appears to lack sufficient innovation compared to existing works on critique and debate. I recommend that the authors clarify this aspect.

---

### References:
[1] Self-critiquing models for assisting human evaluators
[2] Training language models to follow instructions with human feedback
[3] Aligner: Efficient Alignment by Learning to Correct  
[4] Debating with More Persuasive LLMs Leads to More Truthful Answers  
[5] Improving Factuality and Reasoning in Language Models through Multiagent Debate

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The multi-agent critique and meta-critique process likely incurs substantial computational expenses, especially given the high reliance on GPT-4 for critique refinement. A practical analysis of the computational cost relative to the quality gains would provide more context on the feasibility of scaling this approach for widespread or real-time applications.
soundness: 2
strengths: The paper presents using multi-agent feedback through the MultiCritique pipeline, a deviation from previous single-model critique training methods. This generates critique data by aggregating feedback from multiple advanced models, thus raising the quality and robustness of the generated critiques. 

This paper validates the model improvements on benchmarks CRITICEVAL and CRITICBENCH, with positive results. The inclusion of ablation studies demonsrates the contributions of each aspect of the proposed pipeline.

The authors provide a clear breakdown of the MultiCritique pipeline, detailing each stage from data preparation to the meta-critique and RL stages. The structured format of the ACUs is explained clearly, allowing readers to follow the critique generation and refinement processes.
summary: The paper introduces MultiCritique, a data generation pipeline designed to enhance the critique ability of LLMs by using multi-agent feedback during both SFT and RL. MultiCritique aggregates feedback from multiple models, filtering out errors to produce a critique dataset. This method uses a structured approach where each model identifies specific flaws, followed by a meta-critique stage that refines the critiques for accuracy. The reinforcement learning stage further boosts critique quality using a preference-based dataset refined through multi-agent scoring of revision effectiveness. Experiments demonstrate that MultiCritique-trained models outperform existing models on benchmarks such as CRITICEVAL and CRITICBENCH.
weaknesses: The improvements demonstrated by the MultiCritique pipeline, while statistically significant, are relatively modest in scale, particularly in relation to the substantial computational complexity introduced by multi-agent / GPT4 feedback. 

Additionally, the paper would benefit from exploring where the MultiCritique pipeline's improvements become saturated. By examining performance gains across a broader range of model sizes or critique tasks, the authors could better identify the conditions under which MultiCritique delivers more substantial gains.

Meta-review: After carefully considering the six expert reviews and the subsequent author-reviewer discussions, I recommend rejecting this submission. While the paper presents an interesting approach to improving language models' critique abilities through multi-agent feedback, several concerns prevent it from meeting ICLR's acceptance standards.

The paper's primary technical contribution centers on MultiCritique, a data generation pipeline that leverages multiple language models to generate and refine critiques. While this approach demonstrates some improvements over existing methods, the reviewers raised significant concerns about both the novelty of the technical contribution and the cost-effectiveness of the proposed solution.

A major concern, highlighted particularly by reviewers fcKr and CtUU, is the substantial computational overhead introduced by the multi-agent feedback process and its heavy reliance on GPT-4. The authors' response does not fully address these efficiency concerns.

Several reviewers also questioned the methodological novelty of the work. As reviewer CtUU noted, the paper appears to build upon existing multi-agent debate and critique frameworks without introducing substantially new technical innovations. While the authors mentioned that their unified framework differs from previous work by addressing both SFT and RL phases, the core mechanisms still rely heavily on established techniques.

The discussion period revealed additional concerns about the method's scalability and generalizability. While the authors provided new experiments showing results with different base models and on unseen domains, questions remain about the approach's effectiveness with larger models and more diverse critique tasks. The authors' acknowledgment that improvements on 70B models were limited further suggests constraints on the method's broader applicability.

In weighing these factors, the combination of substantial computational overhead, limited technical novelty, and questions about broader applicability make this submission not ready for publication at ICLR 2025.

Decision: Reject