Understanding Prejudice and Fidelity of Diverge-to-Converge Multi-Agent Systems

Authors: Zhen Tan, Song Wang, Shyam Marjit, Zihan Chen, Yinhan He, Xinyu Zhao, Pingzhi Li, Jundong Li, huan liu, Tianlong Chen
Primary area: foundation or frontier models, including LLMs
Keywords: Large language model agents, Multi-Agent System, Benchmark

Abstract:
Large language model (LLM) agents have demonstrated substantial potential across various tasks, particularly in multi-agent systems. Among these, \textit{Diverge-to-Converge} (D2C) frameworks stand out for their ability to iteratively diversify and converge intermediate thoughts to improve problem-solving. In this paper, we conduct a comprehensive study on the \textit{\textbf{prejudice}} and \textit{\textbf{fidelity}} of typical D2C frameworks, including both model-level and society-level frameworks. 
\ding{182} In the \textit{prejudice} section, we uncover an inherent \textit{confirmation bias} in D2C systems, which not only leads to suboptimal performance, but also amplifies social biases, such as gender discrimination and political partisanship. Surprisingly, we find that by reframing open-ended problems into controlled initialized problems, this bias can be leveraged to foster more equitable and effective agent interactions, ultimately improving performance.
\ding{183} In the \textit{fidelity} section, we explore the scaling laws of D2C frameworks at different granularities, revealing that increasing the number of agents enhances performance only when the system is not yet saturated---such as in complex tasks or with weaker agents. In saturated scenarios, however, adding more agents can degrade performance. 
To facilitate further study, we develop \texttt{APF-Bench}, a benchmark specifically designed to evaluate such inherent weaknesses of D2C frameworks. 
We hope our findings offer instructional insights into the strengths and limitations of D2C multi-agent systems, offering guidance for developing more robust and effective collaborative AI systems.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730026131386

thread_type:
official_review

<thread_object>
creator: reviewer_xvhp

<content>
summary:
This paper focuses on the Diverge-to-Converge (D2C) frameworks and highlights the challenges of prejudice and fidelity in D2C frameworks. The authors define prejudice and fidelity as the performance variation under changed conditions and scaling laws, respectively. To evaluate prejudice and fidelity, this paper introduces APF-Bench using the proposed Dataset Refinement. The results confirm the findings.

strengths:
1. The paper is well-structured and easy to follow. The inclusion of informative figures and tables enhances clarity.
2. This paper reveals two key challenges: the impact of initial conditions and the number of agents on the final performance.
3. The experiments span many task-domains and multiple models.

weaknesses:
1. This paper mainly proposes a benchmark to test and validate these challenges rather than further addressing them.
2. In more complex scenarios, problem reframing is difficult.

questions:
Minor comments:
1. In Figure 1, should the question be "A ship travels 80 miles east/west and 150 miles north. How far is the ship from its starting point?".
2. D2C instead of C2D, e.g. Section 5 Debatepedia, Dataset Problem Reframing, etc.
3. Inconsistent symbol representation. In Section 3, C stands for the total number of calls, whereas in Section 4, C stands for Agent Count.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730761325444

thread_type:
official_review

<thread_object>
creator: reviewer_qy19

<content>
summary:
The paper conducts a study on confirmation bias from initial responses in different multi-agent LLM setups and comes up with a technique to prevent this bias (and thus improve benchmark performance) by changing the framing of questions. It then presents very initial work on how multi-agent system performance scales with the number of agents and tokens.

strengths:
- The paper presents a very comprehensive review of existing multi-agent LLM research and fits in prejudice and fidelity quite well in these settings. This provides useful context to understand the paper’s key contributions.

- The problem reframing method is reasonably novel and the experimental evaluations are comprehensive enough to demonstrate improvements with this method.

- It presents initial interesting results around differences in scaling the number of agents/LLM calls versus the number of tokens per generation. This could allow for a lot more future work in multi-agent LLM research.

- APF-Bench encompasses other benchmarks and can act as a useful starting point for similar research directions.

weaknesses:
- The paper explores only problem reframing as a bias mitigation strategy. However, not every problem can be converted into a binary problem, and other strategies are not explored at all.

- The paper does not perform evaluations on any open source models.

- The refinement strategy for datasets could introduce selection bias and skew results. I would be interested in seeing results across a random subset of the test set on the benchmarks used.

- The paper spends its first 5.5 pages providing a background on the problem and multi-agent LLM settings. This takes away from its key contributions, which are limited to the problem reframing strategy and very introductory work on scaling laws around fidelity. Section 6.2 is extremely limited and does not back up its claims with linked experiments.

- The appendix presents examples of model outputs, however it does not provide examples of inputs to the models (especially in the problem reframing setting). I’ve posed questions around these examples in Questions section of my review.

questions:
- Page 18, Case 2, GSM8k: Could the authors provide complete inputs to the models and their outputs for each iteration?

- Is there a hypothesis around why the results hold and such biases occur in language models? Are there reasonable tests that can be conducted around this?

- Could there exist better reframing techniques? Why was the binary reframing technique selected? Will it work for all tasks?

Update - these questions have been answered by the authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731117818813

thread_type:
official_review

<thread_object>
creator: reviewer_eh5m

<content>
summary:
The authors investigate reasoning pathologies of a certain class of multi-agent LLM systems, Diverge-to-Converge (D2C) frameworks, both at the model- and society-level. The authors identify an inherent confirmation bias in D2C systems but results in social biases and task underperformance which can be alleviated if open-ended questions are re-phrased as binary. The authors then study the scaling laws of D2C frameworks, finding that more agents does only result in performance improvements if the system is not yet saturated but can otherwise even degrade performance. The authors suggest remedies for both these pathologies and release APF-Bench to specifically evaluate these weaknesses.

strengths:
* Very timely and can provoke thought - e.g. trade-off bias/compute
* Robust evaluation across multiple datasets
* the idea to use reframing to tackle biases seems novel

weaknesses:
* Could have discussed a greater variety of biases other than confirmation bias
* It isn't clear how questions of real-world importance that are open-ended can always be brought into binary form.
* conceptual advances are limited - scaling laws / reframing techniques themselves feel rather incremental

line 216 "menifest"

questions:
* How do you prevent bias in the debate judgements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731124171723

thread_type:
official_review

<thread_object>
creator: reviewer_z9fh

<content>
summary:
This paper examines the limitations of Diverge-to-Converge (D2C) frameworks in large language model (LLM) agents, focusing on prejudice and fidelity. It reveals a confirmation bias in D2C systems that hampers performance and amplifies social biases, but reframing open-ended problems as binary questions mitigates these effects. The study also shows that increasing the number of agents only improves performance under unsaturated conditions. Additionally, the authors introduce APF-Bench, a benchmark to evaluate these weaknesses, providing insights for building better collaborative AI systems.

strengths:
1. It uncovers and addresses *confirmation bias* in D2C frameworks, providing practical solutions to mitigate performance issues and social biases.
2. The study examines both*prejudice* and *fidelity*, offering a detailed understanding of D2C frameworks across multiple levels.
3. By demonstrating how reframing problems into binary questions improves fairness and effectiveness, the research has real-world applicability.
4. The development of *APF-Bench* as a dedicated tool for evaluating D2C systems is a valuable resource for future research.
5. The analysis of scaling laws provides essential guidelines for optimizing agent collaboration in different task scenarios.

weaknesses:
1. Limited Real-World Testing: The findings might lack generalizability if not tested in diverse, real-world multi-agent scenarios.
2. Potential Oversimplification: Reframing problems as binary questions may oversimplify complex tasks, possibly limiting the depth of solutions.
3. Scalability Constraints: The performance degradation observed in saturated systems indicates a limitation in scaling D2C frameworks effectively.
4. Bias Mitigation Trade-offs: While the approach reduces biases, it may inadvertently introduce new limitations or biases in certain contexts.

questions:
* Why do you study D2C frameworks rather than other MAS frameworks? Is D2C a typical and widely adopted MAS framework? What are the incentives behind this choice?
* What is the main *contribution in scientificity* that the paper claims? This paper does a lot of evaluation and analysis on different LLMs, but they are the existing ones. Could you provide insights into designing LLMs that can inherently avoid or mitigate confirmation bias? Or, can you give a discussion on the underlying causes of such bias, which could possibly arise at the data level or the pre-training/fine-tuning level instead of solely empirical discovery? 
* See also Weaknesses for other questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732409260517

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
### Common Questions

- **`W1 (z9FH):`** Limited Real-World Testing: The findings might lack generalizability if not tested in diverse, real-world multi-agent scenarios.  
- **`W2 (z9FH):`** Potential Oversimplification: Reframing problems as binary questions may oversimplify complex tasks, possibly limiting the depth of solutions.  
- **`W2 (EH5M):`** It isn't clear how questions of real-world importance that are open-ended can always be brought into binary form.  
- **`W1 (QY19):`** The paper explores only problem reframing as a bias mitigation strategy. However, not every problem can be converted into a binary problem, and other strategies are not explored at all.  
- **`W2 (xVhp):`** In more complex scenarios, problem reframing is difficult.  

---

**Comment**:  

Apologies for any confusion regarding the oversimplification of reframing problems as binary questions. We acknowledge that not all tasks can be reduced to binary questions such as “yes/no” or “true/false.” Instead, we present **controlled initialization** through problem reframing for multi-agent systems by leveraging the **potential solution space** of a given problem.  

### Dataset Examples  
In this study, we consider four diverse datasets (for confirmation bias) that address unique, real-world task types of importance [line 266]:  

1. **PIQA** `[Lines: 270–273]`:  
   Contains questions with a **binary solution space**, where either "statement-1" or "statement-2" is correct.  

2. **StrategyQA** `[Lines: 274–277]`:  
   Also contains questions with a **binary solution space**, where the answer is "Yes" or "No."  

3. **GSM8K** `[Lines: 278–281]`:  
   Includes questions with a **non-binary solution space**; the solution lies in $ \mathbb{R} $ (real numbers).  

4. **Chess Move Validity** `[Lines: 282–285]`:  
   Similar to GSM8K, this dataset has a **non-binary solution space**. While it is commonly stated that there are 64 possible answers formatted as $[a-h][1-8]$, representing potential chess moves, the actual number of valid moves may vary depending on the specific state of the chessboard (e.g., piece positions, legal moves). Therefore, the solution space dynamically adjusts to the context of the game. Each generated answer was deemed correct as long as it was one of the valid answers in the sequence.


### Controlled Initialization Framework  
Let $Q$ denote a question, $C_A$ the correct answer space, and $C_W$ the wrong answer space. Controlled initialization is structured as follows:  
- For **controlled wrong initialization**, the prompt is:  
  _“Is $C_W$ the correct answer to the question?”_  
- For **controlled right initialization**, the prompt is:  
  _“Is $C_A$ the correct answer to the question?”_  

For binary tasks like **PIQA** and **StrategyQA**, where $C_W = \sim C_A$, the solution space is straightforward. However, for non-binary tasks:  
- **GSM8K**: $C_A \subset \mathbb{R}$ (correct numerical solution), and $C_W = \mathbb{R} \setminus C_A$.  
- **Chess Move Validity**: $C_A$ is the set of valid answers out of 64 possible moves, and $C_W$ encompasses the complement of valid moves. 

### Broader Applicability and Frameworks  
These controlled initializations are initiated on the **affirmative side** during the starting round for both:  
1. **Multi-label frameworks** (e.g., Self-consistency, Debate, Consultancy).  
2. **Society-label frameworks** (e.g., CAMEL, LangChain, AutoGen).  

By explicitly tailoring the initialization to the solution space, we maintain flexibility to address task-specific complexity. In the revised manuscript, we have incorporated these clarifications and further detailed the methodologies discussed above to emphasize the adaptability of our approach.

---

### Manuscript Updates
We have marked out the updated contents in blue in the pdf. Specifically, the updates include:

1. More detailed dataset description on the searching space of solutions, in Section 5.

2. Detailed explanation on the controlled initialization, in Appendix A.

3. Further discussion on contribution, with the comparison with other LLM Debias Directions, in Appendix E.

4. Numerical results for figures, in Appendix F.

5. More detailed revisions including typos, layout adjustment, revised figure captions, etc.
</content>
</thread_object>

</thread>


Metareview:
metareview:
While this work tackles an important and interesting topic in so-called multi-agent LLM systems, I believe the current work is not ready to be published. The current work somewhat overstates its contributions by framing the paper as a broad study of prejudice and bias in multi-agent systems (as evident in the title), but only looks at confirmation bias (as pointed out by Reviewer EH5M). The experiments in the paper primarily make speculative claims about the underlying reason for the observed bias, based on the final task performance, while foregoing the opportunity to look deeper into the inference chains leading to these results. A quantitative analysis of the intermediate outputs of these multi-agent systems would provide a richer understanding of the bias studied in this work.