Training Language Models to Critique with Multi-Agent Feedback

Authors: Tian Lan, Wenwei Zhang, Chengqi Lyu, Shuaibin Li, Chen Xu, Heyan Huang, Dahua Lin, Xian-Ling Mao, Kai Chen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Critique, LLM, preference-based RL

Abstract:
Critique ability, a meta-cognitive capability of humans, presents significant challenges for LLMs to improve.
Recent works primarily rely on supervised fine-tuning (SFT) using critiques generated by a single LLM like GPT-4.
However, these model-generated critiques often exhibit flaws due to the inherent complexity of the critique. 
Consequently, fine-tuning LLMs on such flawed critiques typically limits the model's performance and propagates these flaws into the learned model.
To overcome these challenges, this paper proposes a novel data generation pipeline, named MultiCritique, that improves the critique ability of LLMs by utilizing multi-agent feedback in both the SFT and reinforcement learning (RL) stages.
First, our data generation pipeline aggregates high-quality critiques from multiple agents instead of a single model, with crucial information as input for simplifying the critique.
Furthermore, our pipeline improves the preference accuracy of critique quality through multi-agent feedback, facilitating the effectiveness of RL in improving the critique ability of LLMs.
Based on our proposed MultiCritique data generation pipeline, we construct the MultiCritiqueDataset for the SFT and RL fine-tuning stages. Extensive experimental results on two benchmarks demonstrate:
1) the superior quality of our constructed SFT dataset compared to existing critique datasets;
2) additional improvements to the critique ability of LLMs brought by the RL stage.
Notably, our fine-tuned 7B model significantly surpasses other advanced 7B-13B open-source models, approaching the performance of advanced 70B LLMs and GPT-4.
Codes, datasets and model weights will be publicly available.

Decision: reject
Reviewer scores: 8, 6, 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730519747801

thread_type:
official_review

<thread_object>
creator: reviewer_fckr

<content>
summary:
The paper introduces MultiCritique, a data generation pipeline designed to enhance the critique ability of LLMs by using multi-agent feedback during both SFT and RL. MultiCritique aggregates feedback from multiple models, filtering out errors to produce a critique dataset. This method uses a structured approach where each model identifies specific flaws, followed by a meta-critique stage that refines the critiques for accuracy. The reinforcement learning stage further boosts critique quality using a preference-based dataset refined through multi-agent scoring of revision effectiveness. Experiments demonstrate that MultiCritique-trained models outperform existing models on benchmarks such as CRITICEVAL and CRITICBENCH.

strengths:
The paper presents using multi-agent feedback through the MultiCritique pipeline, a deviation from previous single-model critique training methods. This generates critique data by aggregating feedback from multiple advanced models, thus raising the quality and robustness of the generated critiques. 

This paper validates the model improvements on benchmarks CRITICEVAL and CRITICBENCH, with positive results. The inclusion of ablation studies demonsrates the contributions of each aspect of the proposed pipeline.

The authors provide a clear breakdown of the MultiCritique pipeline, detailing each stage from data preparation to the meta-critique and RL stages. The structured format of the ACUs is explained clearly, allowing readers to follow the critique generation and refinement processes.

weaknesses:
The improvements demonstrated by the MultiCritique pipeline, while statistically significant, are relatively modest in scale, particularly in relation to the substantial computational complexity introduced by multi-agent / GPT4 feedback. 

Additionally, the paper would benefit from exploring where the MultiCritique pipeline's improvements become saturated. By examining performance gains across a broader range of model sizes or critique tasks, the authors could better identify the conditions under which MultiCritique delivers more substantial gains.

questions:
The multi-agent critique and meta-critique process likely incurs substantial computational expenses, especially given the high reliance on GPT-4 for critique refinement. A practical analysis of the computational cost relative to the quality gains would provide more context on the feasibility of scaling this approach for widespread or real-time applications.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730525420294

thread_type:
official_review

<thread_object>
creator: reviewer_ctuu

<content>
summary:
Overall, this paper is poorly written, describing certain motivations and methods that are difficult to understand.

> For example, in the abstract, the authors emphasize that the inherent complexity of model-generated critiques limits model performance, which they cite as a primary motivation for this work. However, MultiCritique originates from Multi-agent Feedback, which clearly adds to this complexity by involving multiple models.

strengths:
The authors leverage multi-agent feedback to train models capable of critique, claiming that multi-agent feedback provides more accurate and critical information, which could address the inherent complexity of critique. However, the reviewer has expressed doubts about this claim. While the experimental setup is fairly robust, the descriptions of the experimental settings lack detail in most parts.

weaknesses:
1. In terms of research on critique, other researchers have already explored similar ideas with LLMs. For instance, Jan Leike at OpenAI proposed "Self-critiquing models for assisting human evaluators," which this paper does not discuss. Additionally, while the authors discuss RLHF, they surprisingly omit foundational literature on RLHF, such as *Training language models to follow instructions with human feedback*. It appears that the authors did not thoroughly investigate relevant literature, which results in some contributions being overstated.
   
2. The terms “meta-cognitive” and “critique as a greater challenge than criticism” are ambiguous. Could the authors clarify these?

3. In Line 173, the authors state, "In total, we sample 10.7K queries covering 123 diverse task scenarios." However, the reviewer does not know what these 123 tasks entail or where to find this information in the appendix in this paragraph.

4. The authors use InternLM2-20B for scoring. How accurate is this reward model in the authors’ study, and how does it align with human evaluations at the three labeled response levels? Furthermore, Section 3.1 lacks a clear definition of these three levels.

5. In Section 4.1, the authors present Equation 1, where they condition \( Q_i \) and \( R_i \) to generate \( C_I \) and \( C \). How does this differ from the fine-tuning approach in the Aligner [3] paper, or are the two approaches identical?

6. In the experiments, I noticed that the authors applied SFT on InternLM2-7B-Chat-SFT, essentially layering SFT on a model that was already SFT-trained. How did they address potential issues of hallucination in this setup? Moreover, why did they not consider using Llama3-8B-Instruct as a base model?

7. This paper is based on multi-agent feedback, but it is unclear how the multi-agent approach is implemented. My understanding is that the feedback primarily comes from multiple language models. How does this approach differ from the multi-agent debate setups described in [4][5]? If the multi-agent approach in this paper merely relies on feedback from multiple LLMs without any designed communication between the agents, it appears to lack sufficient innovation compared to existing works on critique and debate. I recommend that the authors clarify this aspect.

---

### References:
[1] Self-critiquing models for assisting human evaluators
[2] Training language models to follow instructions with human feedback
[3] Aligner: Efficient Alignment by Learning to Correct  
[4] Debating with More Persuasive LLMs Leads to More Truthful Answers  
[5] Improving Factuality and Reasoning in Language Models through Multiagent Debate

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730560645161

thread_type:
official_review

<thread_object>
creator: reviewer_x97p

<content>
summary:
The paper proposes MultiCritique, a novel data generation pipeline aimed at enhancing language models' (LLMs) critique abilities by leveraging multi-agent feedback during supervised fine-tuning (SFT) and reinforcement learning (RL) stages. The proposed MultiCritique pipeline aggregates critiques from multiple advanced LLMs, utilizing key meta-critique processes to classify and consolidate high-quality feedback. This results in the construction of a new MultiCritiqueDataset, which shows empirical improvements on CRITICEVAL and CRITICBENCH benchmarks, demonstrating superior performance in critiquing abilities. Notably, the approach effectively competes with models up to ten times its size, approaching the critique capabilities of models like GPT-4.

strengths:
1. The paper introduces a multi-agent framework that significantly improves critique dataset quality by aggregating and refining feedback from diverse LLMs. The structured use of Analytical Critique Units (ACUs) and meta-critique classification addresses the limitations of single-agent feedback and provides a robust critique foundation. The structured use of Analytical Critique Units (ACUs) and meta-critique classification addresses the limitations of single-agent feedback and provides a robust critique foundation.

2. The MultiCritiqueDataset improves LLM critique ability considerably over traditional supervised datasets, evidenced by impressive gains on CRITICEVAL and CRITICBENCH

3. The paper has quite extensive evaluation and analysis that support the method.

weaknesses:
See question.

questions:
1. While the pipeline effectively aggregates critiques, the reliance on multiple LLMs (including GPT-4) for meta-critique classification may introduce considerable computational and financial overhead. Could you provide quantitative result and  a discussion on efficiency, potential trade-offs, or future work focused on optimizing this aspect?

2. Although the multi-agent framework is well-justified, could you provide a more explicit differentiation from other multi-agent critique frameworks (e.g., Arena learning and Stable Alignment). Clearer explanations on how MultiCritique uniquely contributes to critique generation and feedback quality.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730617431402

thread_type:
official_review

<thread_object>
creator: reviewer_3orz

<content>
summary:
The paper, tackles the inherent difficulty in enhancing critique capabilities within large language models (LLMs). Critique ability, a key aspect of human meta-cognition, is essential for LLMs as it allows them to identify and address flaws in responses. This skill is pivotal for reliable automatic evaluation and model self-improvement. However, training models to critique effectively poses challenges. Most existing methods rely on supervised fine-tuning (SFT) using critiques generated by a single LLM, typically a model like GPT-4. Although effective to a degree, these single-model critiques can propagate the original model’s biases and limitations, leading to inaccuracies and diminished critique quality.

To address these issues, the authors introduce a novel data generation pipeline, MultiCritique, that uses multi-agent feedback to enhance the quality and diversity of critiques. Rather than depending on a single model’s perspective, MultiCritique aggregates feedback from multiple advanced LLMs, allowing for a richer, more balanced dataset. This multi-agent approach is implemented in both the SFT and reinforcement learning (RL) stages, producing a dataset called MultiCritiqueDataset. With this dataset, the model undergoes fine-tuning and then reinforcement learning, where preference scoring further refines the critique quality based on feedback from several agents.

The study’s experiments show that a 7B model fine-tuned with the MultiCritiqueDataset significantly outperforms other models in critique tasks, even rivaling the performance of much larger, closed-source models like GPT-4. The authors argue that the multi-agent approach not only yields a higher-quality critique dataset but also preserves feedback diversity, making it a robust solution for training LLMs in critique ability. This advancement offers an effective pathway toward more self-improving, reliable, and critique-capable language models

strengths:
- overall addressing an important problem of developing small high qulaity critique model
- new multi-agent feedbakc approach to curate dataset
- creation of a new high qulaity multi-critique dataset, incorporating diverse queries,and crucial information collection
- naturla integration and use of RL
- Strong experimental valiadtion, showing 7b FT models perform at par or better with closed source and 70b model on criticeval and critic bench

weaknesses:
- not very technically novel, looks like a fairly stratightforward principle of ensembling for dataset collection
- Performance is still limited by an upperbound of the the base LLMs, and it is not clear if we could sclae to 70b, would the performance be better

questions:
- No major questions, I think the paper ideas a straightforward and easy to follow, the presentation is good, and the ideas are well executed
- From a research perspective, I think novelty is lacking a bit, neverthless it is an important contribution
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684858664

thread_type:
official_review

<thread_object>
creator: reviewer_zqhf

<content>
summary:
This paper proposes a data generation pipeline utilising some of the SoTA LLMs (OpenAI, Claude, Qwen etc) both for SFT and RL tuning in order to improve the critiquing ability of LLMs. The authors experiment on CriticEval and CriticBench, where their SFT and RL datasets provides a good boost in performance.

strengths:
- It is a Practically useful work
- Good improvement in results in CriticEval and CriticBench benchmarks

weaknesses:
- Mostly this paper feels like just a bit of a formalisation for distilling “critiquing” ability from GPT-4 by generating critiquing data both for SFT and RL tuning. The data generation process is also quite heavily dependent on GPT-4 abilities.
- Novelty of this work is not very clear to me

questions:
- What is the generalizabilty of this kind of fine-tuning of the critiquing ability. Can the authors show whether finetuning on critiquing dataset generated from general alignment datasets (like OpenHermes, OpenAssistant) alone can still help in generating critiques in unseen domains like math and code.
- The performance improvement of the MultiCritiqueDataset-RL over MultiCritiqueDataset-SFT is not very clear - for some datasets there seems to be improvement while for others it is hurting or improvement is negligible. Can the authors comment on why this happening? Also what about including both MultiCritiqueDataset -SFT and -RL?
- Minor concern about terminology - I am not quite sure why the authors stress on “agent” feedback — this work on critique generation seems like yet another instruction tuning task, this in general can be helpful in agentic tasks but I don’t see why this is termed “multi-agent”
</content>
</thread_object>

</thread>
<thread>
creation date:
1730785318597

thread_type:
official_review

<thread_object>
creator: reviewer_rdx7

<content>
summary:
This paper proposes a critique data generation pipeline called MultiCritique to enhance the critique capabilities of language models. The process is divided into three steps. In Step 1, the authors selected queries from six datasets and used 11 language models to generate responses. These responses were rated by InternLM2-20B-reward into three quality levels, and GPT-4 was used to extract and summarize crucial information (CI) for each query-response pair. In Step 2, the query-response-CI tuples were input into multiple language models to generate aspect-specific critiques, referred to as Analytical Critique Units (ACUs). GPT-4 then classified these ACUs into seven quality categories and aggregated them into a final critique summary, which could be used for SFT of language models. In Step 3, the ACUs generated in Step 2 were used to construct preference pairs, which were then employed for reinforcement learning fine-tuning using PPO.
The evaluation of this pipeline was conducted on two benchmarks: CriticEval (unpublished) and CriticBench (ACL Findings '24).

strengths:
- The paper is well-written and effectively presented, with technical details that are clearly explained and easy to follow. Figure 1 is particularly effective in illustrating the entire MultiCritique pipeline.
- The evaluations and ablation studies are comprehensive, providing a thorough analysis of the model's performance and the impact of various components.
- The MultiCritique SFT dataset, if made open-source, could be a good contribution to the field due to its careful construction aimed at reducing variances and biases.
- The limitations section is well-prepared, addressing most concerns that might arise from the study.

weaknesses:
- Dependence on GPT-4: The project heavily relies on GPT-4 for classifying and summarizing critiques, which may introduce biases into the final critique summaries. It is not hard to say that the entire project is feasible because of the existence of GPT-4. This reliance potentially undermines the benefits of using multiple agents in earlier steps. I know it might be a bit too much to ask, but it would be interesting to explore the impact of using a different model, such as Claude 3.5 Sonnet, to see how the results might differ.
- Specificity Towards Critique Ability: The model's focus on improving critique ability raises concerns about overfitting to these specific tasks. It would be beneficial to assess whether this specialization affects other capabilities, such as chat performance. Evaluating the model on benchmarks like AlpacaFarm Arena-Hard could provide insights into any potential degradation in general capabilities.

questions:
- Table 4 is somewhat unclear to me. Could you elaborate on what was specifically done in this ablation study? Are you fine-tuning from the same baseline models and then using different datasets, such as MultiCritique-SFT versus critiques generated solely by GPT-4-turbo and other individual models? Additionally, are these individual critiques extracted from the MultiCritique dataset, or are they generated separately?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732113600402

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank all the reviewers for their helpful comments and suggestions. Overall, we are delighted that reviewers found our work is "clear and well-written" (Reviewer rDX7, fcKr, 3oRZ), and our contributions and improvements are "significant" (Reviewer rDX7, zQhf, 3oRZ, X97P, fcKR). Besides, we are also delighted that all reviewers thought our experiments are "comphrensive and robust".

Following suggestions of reviewers, we have updated our submission with additional experimental results showing the generalization to general capabilities (Section 6 "MultiCritique Improves the General Capability of LLMs"), generalization to unseen tasks (Section 6 "Generalization to Unseen Tasks"), and results on more base LLMs (Appendix H). We also re-written some paragraphs in our main text and Appendix for answering some reviewer questions. **All the revisions in PDF are highlighted in blue. In the following content, we will answer some common questions of reviewers.**

---

## **Novelty**

We appreciate the comments of Reviewer zQhf and 3oRZ regarding the novelty of our work. We will clarify as following aspects:

### **Fundamental Challenges and Limitations in Prior Work**

We want to highlight the challenges in previous works that we are trying to solve.

- **Inherent Complexity:** Critique requires LLMs to understand the tasks, and then provide constructive feedback on response, making it challenging even for advanced models like GPT-4.
- **Bias of One Single Model:** Previous critique-tuning approaches rely on single-model (GPT-4) critiques for SFT. When the teacher model (GPT-4) makes errors, these flaws are propagated to the student model during fine-tuning.
- **Upper bound Limitation of SFT:** The reliance on SFT creates an inherent upper bound on critique performance.

### **Distinct Targets and Task Formulation**

As described in related work, most previous multi-agent works primarily focus on improving LLM's alignment and response quality. While there are some multi-agent works that improve the LLM-based evaluation during the inference stage, our motivation is to design a unified multi-agent feedback framework to improve LLM's critique ability through both the SFT and RL fine-tuning stages.

### **Technical Contributions**

To address these challenges, our work introduces three solutions:

- **Introducing Crucial Information:** Our framework introduces three crucial information to **reduce the complexity of the critique task.**
- **MultiCritique-SFT Framework:** To address the bias of one single model, we propose to aggregate diverse and accurate critiques from multi-agents and mitigate the flawed critiques. **Experiments in Section 6 (Table 4) demonstrate that our critiques is better than each model's critique.**
- **MultiCritique-RL framework:** Beyond the behavior cloning of SFT, we propose to calibrate the preference critiques by utilizing the multi-agent feedback from downstream revision task, facilitating the robust RL for improving LLM's critique ability.

**Note that experiments in Section 6 verify the effectiveness of three solutions.** In conclusion, although our work builds upon existing critique and multi-agent research, it introduces novel technical solutions that significantly differ from previous works.

---

## **GPT-4 Dependency**

We appreciate the reviewer rDX7 and zQhf's concerns about GPT-4 dependency and novelty. We would like to clarify these points:

### **Flexible Framework**

**Our MultiCritique framework is a general and model-agnostic framework.** GPT-4 serves as one possible meta-critique judge model. While we chose GPT-4 due to its advanced meta-critique capabilities [1], any sufficiently advanced LLM can fulfill this role.  To demonstrate this flexibility, we conducted experiments with Claude-3.5-Sonnet and Qwen2.5-72B-Instruct. Specifically, we randomly sample 200 samples from MultiCritiqueDataset-SFT, and prompt Qwen2.5-72B-Instruct and Claude-3.5-Sonnet to re-run the Meta-Critique Classification and Critique Summarization processes. Subsequently, we assess two metrics to reflect the correlations between GPT-4 and these models: (1) Spearman correlation scores between GPT-4 and these models on assessing the response quality; (2) Agreement on judging the correctness of each ACU in Meta-Critique classification. 

|  |Correlation| Agreement|
| - | - | - |
| Claude-3.5-Sonnet|0.58| 0.71|
| Qwen2.5-72B-Instruct |0.60|0.72|

These models achieve not only high meta-critique agreement (> 70%, the random baseline is 50%) with GPT-4 but also strong correlation 
 (≈ 0.6) in final judgment scores. Therefore, other advanced models could also be used in our proposed MutiCritique.

> [1] The Critique of Critique

### **GPT-4-Free MultiCritique-RL**
More importantly, our MultiCritique-RL phase operates entirely without GPT-4, instead utilizing multiple 7B LLMs.

**In conclusion, while we leverage GPT-4's capabilities for practical implementation, any advanced models can fulfill this role.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
After carefully considering the six expert reviews and the subsequent author-reviewer discussions, I recommend rejecting this submission. While the paper presents an interesting approach to improving language models' critique abilities through multi-agent feedback, several concerns prevent it from meeting ICLR's acceptance standards.

The paper's primary technical contribution centers on MultiCritique, a data generation pipeline that leverages multiple language models to generate and refine critiques. While this approach demonstrates some improvements over existing methods, the reviewers raised significant concerns about both the novelty of the technical contribution and the cost-effectiveness of the proposed solution.

A major concern, highlighted particularly by reviewers fcKr and CtUU, is the substantial computational overhead introduced by the multi-agent feedback process and its heavy reliance on GPT-4. The authors' response does not fully address these efficiency concerns.

Several reviewers also questioned the methodological novelty of the work. As reviewer CtUU noted, the paper appears to build upon existing multi-agent debate and critique frameworks without introducing substantially new technical innovations. While the authors mentioned that their unified framework differs from previous work by addressing both SFT and RL phases, the core mechanisms still rely heavily on established techniques.

The discussion period revealed additional concerns about the method's scalability and generalizability. While the authors provided new experiments showing results with different base models and on unseen domains, questions remain about the approach's effectiveness with larger models and more diverse critique tasks. The authors' acknowledgment that improvements on 70B models were limited further suggests constraints on the method's broader applicability.

In weighing these factors, the combination of substantial computational overhead, limited technical novelty, and questions about broader applicability make this submission not ready for publication at ICLR 2025.