Title: Can Dependencies Induced by LLM-Agent Workflows Be Trusted?
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-LLM-Agent System
Abstract: LLM-agent systems often decompose high-level objectives into subtask dependency graphs, assuming that each subtask’s output is reliable and conditionally independent of others given its parent responses. 
However, this assumption frequently breaks during execution, as ground-truth responses are inaccessible, leading to inter-agent misalignment—failures caused by inconsistencies and coordination breakdowns among agents. 
To address this, we propose SeqCV, a dynamic framework for reliable execution under violated conditional independence. 
SeqCV executes subtasks sequentially, each conditioned on all prior verified responses, and performs consistency checks immediately after agents generate short token sequences. 
At each checkpoint, a token sequence is accepted only if it represents shared knowledge consistently supported across diverse LLM models; otherwise, it is discarded, triggering recursive subtask decomposition for finer-grained reasoning. 
Despite its sequential nature, SeqCV avoids repeated corrections on the same misalignment and achieves higher effective throughput than parallel pipelines. 
Across multiple reasoning and coordination tasks, SeqCV improves accuracy by up to 30\% over existing LLM-agent systems.
Code is available at https://github.com/tmllab/2025_NeurIPS_SeqCV.


Peer reviews:
comments: SUMMARY:
The authors delve into the root cause of inter-agent misalignment and argue that it stems from the failure of the “conditional independence” assumption in existing workflow designs. Building on this observation, they introduce SEQCV, a dynamic framework that tackles the problem through sequential execution, cross-verification, and recursive decomposition.

STRENGTHS AND WEAKNESSES:
## Strength

1. This paper is generally in a good structure and easy to follow.
2. This paper demonstrated performance gains on several benchmarks, suggesting its effectiveness.

## Weakness

- Token cost would be high. Every trajectory and answer is verified, which appears to incur excessive token costs and risks exceeding the model’s context window. Even for easy query such as 1+1, the workflow method still needs to decompose and verify, which introduces unnecessary cost.

- The novelty of the proposed method is unclear. Currently, it seems like another one manually-designed workflow.

- The paper does not explain how tasks are decomposed.

- No ablation study: subsequent experiments do not directly validate whether inter-agent misalignment has been solved, making the link between the stated problem and the experiments unclear.

- The framework’s reliance on strictly sequential execution would make the workflow slow during inference.

QUESTIONS:
na

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SEQCV, a dynamic framework designed to address the issue of inter-agent misalignment in LLM-agent systems. The authors identify the violation of conditional independence assumptions—where subtask responses are assumed to be reliable and dependent only on parent responses—as the root cause of these misalignments. SEQCV tackles this by executing subtasks sequentially, conditioning each on all prior responses, and immediately verifying them through consistency checks across diverse LLM models. If a response is deemed unreliable, a recursive splitting mechanism breaks down the subtask into smaller components. Experiments demonstrate that SEQCV improves accuracy by up to 17.3% and reduces execution time by more than half on complex tasks compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper pinpoints the violation of conditional independence assumptions in LLM-agent workflows as a core issue leading to inter-agent misalignment, which significantly impacts quality and runtime efficiency. This highlights a fundamental challenge in current LLM-agent systems.\n\n- SEQCV introduces a dynamic framework that ensures reliable execution despite violated conditional independence assumptions. It achieves this through sequential subtask execution, conditioning on prior responses, consistency checks, and a recursive splitting mechanism for unreliable outputs.\n\n- SEQCV shows substantial gains in both accuracy and efficiency across various tasks and datasets, including mathematical reasoning, knowledge-intensive reasoning, logical reasoning, and multi-hop reasoning. It improves accuracy by up to 17.3% and reduces execution time by more than half on complex tasks.\n\n## Weaknesses\n\n- The paper does not provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments. This hinders reproducibility for other researchers.\n\n- The evaluation relies on an LLM (gpt-4.1-nano) to determine whether predictions match ground truths and calculate accuracy. While this can be efficient, it introduces a potential bias or limitation if the evaluating LLM itself has errors or biases, or if its \"ground truth\" assessment isn't truly objective.\n\n- SEQCV's verification step relies on \"semantically consistent across diverse LLM models\" and \"peer-reviewed by agents of other LLM models\". While the paper mentions using \"gpt-4o-mini, gpt-4.1-mini, and gpt-4.1-nano as the core models\" for evaluation and \"three gpt-4o-mini models\" or \"gpt-4o-mini, o4-mini and o3-mini\" as backbone models, it doesn't explicitly define what constitutes \"diverse\" enough models to ensure robust cross-model validation. The effectiveness of this mechanism heavily depends on the actual diversity and complementary strengths of the chosen LLMs, which isn't thoroughly explored.\n\n- While the paper claims SEQCV \"avoids costly misalignment corrections and delivers higher effective throughput than parallel pipelines\" , the fundamental sequential nature of subtask execution and the consistency checks at each token sequence checkpoint  inherently introduce overhead. Although it aims to avoid costly misalignment corrections, the frequent verification and potential recursive splitting could still lead to higher overall latency for simple tasks where misalignment is less likely. The efficiency gains might primarily be realized on complex tasks prone to significant misalignment errors in parallel setups.\n\nQUESTIONS:\n- How is \"semantic consistency\" precisely defined and measured during the cross-model validation process, especially given that LLMs may generate varied but equally valid responses for certain subtasks?\n\n- What is the practical impact of the \"maximum recursion depth\" in the recursive splitting mechanism? Does reaching this limit imply a failure to solve the subtask, and if so, how does SEQCV handle such scenarios and their impact on the overall task objective?\n\n- Could the paper provide a more detailed breakdown of the latency implications of SEQCV's sequential execution and frequent verification steps across different task complexities, perhaps with a comparison to the latency of re-execution after misalignment in parallel approaches?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper shows that graph‑based multi‑LLM workflows break down because subtasks’ outputs are not conditionally independent; this causes style, reasoning, or objective misalignment among agents. The authors propose SEQCV, which executes subtasks sequentially, conditioning each on all verified history. In add ition, it performs segment‑level cross‑model voting to accept or discard partial outputs, and recursively splits a failing subtask into simpler ones. Across six benchmarks, SEQCV improves accuracy by 3‑4pp and cuts runtime on complex tasks despite the sequential flow.

STRENGTHS AND WEAKNESSES:
strengths:
1. Solid empirical gains; diverse tasks; transparent methodology and clear figures, pseudocode, and motivation.
2. Addresses a real reliability gap in agent systems; technique is broadly applicable.
3. Fine‑grained cross‑model verification 、 dynamic splitting is novel.


Weaknesses:

1. Accuracy gains are moderate; sequential flow may bottleneck very wide graphs.
2. Relies on established ideas (voting, recursion) stitched together.

QUESTIONS:
1. How much do sequential execution, cross‑validation, and recursive splitting each contribute?
2. Can independent branches of a DAG run in parallel without re‑introducing misalignment?
3. What exact criterion defines “agreement,” and how do you guard against unanimous but wrong answers?

LIMITATIONS:
SEQCV requires an acyclic task graph and may struggle with feedback loops .
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers failures in multi-agent systems caused by inter-agent misalignments. When complex tasks are decomposed into a graph of subtasks, the system often assumes conditional independence between them. However, this assumption is frequently violated. Since LLMs produce approximate outputs without access to ground truth, hidden dependencies can emerge, leading to cascading errors. To address this, the authors propose a framework called SEQCV. It executes subtasks sequentially, conditioning each new step on the full history of previously verified outputs. SEQCV also introduces a sequential generation and cross-validation mechanism. This allows the system to selectively split complex subtasks into smaller ones. The authors demonstrate that SEQCV improves accuracy across several benchmarks. It also reduces overall execution time by catching errors early, thus avoiding expensive full-context corrections later.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The efficiency of multi-agent LLM systems is an important and understudied research area. This paper provides an insightful study on this topic.\n2) The paper proposes a new framework, SEQCV, to improve the reliability and efficiency of multi-agent LLM systems.\n3) The paper validates SEQCV on six standard reasoning benchmarks (including MATH and HotpotQA) and show the proposed solution performs well. In addition, the paper includes a challenging, domain-specific (game development) task. Finally, I found the results provided in the Appendix being insightful as well.\n\nWeaknesses:\n1) Questionable claims and assumptions:\n- The claim that existing literature assumes subtasks in a DAG decomposition are conditionally independent is quite strong. You seem to draw inspiration from reinforcement learning and optimal control theory by suggesting that LLM-based workflows resemble partially observable Markov decision processes (POMDPs) rather than Markov decision processes (MDPs). However, this is not a novel insight. It is already widely accepted that LLM (or any neural model) outputs are approximations, not ground truths. Therefore, ML-based workflows are inherently non-Markovian.\n- You should clearly justify the differences in core assumptions. In particular, many prior works limit inter-agent input sharing not due to a belief in conditional independence, but for context management reasons. Conditioning on full dialogue history can indeed resolve non-Markovian issues, but it is memory-intensive.\n\n2) Generality: The main contribution of this paper is a practical method for managing context. By conditioning on previously verified outputs, the method helps mitigate non-Markovian effects in LLM workflows. This is a useful idea. However, it is unclear how the approach would generalize to more autonomous agentic systems. In such settings, agents choose their communication partners, and the interaction graph may not be a DAG, e.g., include cycles to implement things like self-refinement.\n\n3) Lack of ablation studies: SEQCV introduces several interesting components: Global-Context Construction, Subtask-Context Generation, Cross-Model Validation, and Dynamic Recursive Splitting. However, the paper lacks ablation studies. Without them, it is hard to assess the individual contribution of each module. This limits the overall evaluation and interpretability of the system.\n\n4) Clarity: You should introduce and define all symbols before using them in Section 2. Otherwise, the formalism is difficult to follow.\n\nQUESTIONS:\n1) What are the latency measurements for the accuracies reported in Table 1? It would be useful to provide an accuracy-latency trade-off curve. Such a curve could illustrate performance across the different system configurations you tested.\n\n2) How was cross-validation performed? Specifically, how does this methodology apply to a system where agents have distinct and interdependent roles, such as the CEO, Software Engineer, and Code Reviewer in ChatDev?\n\n3) The paper uses a \"consensus of LLMs\" as a proxy for ground truth. What is the justification for this assumption? In particular, \n- How did you account for shared systemic biases given the LLMs were likely trained on similar data? \n- There is previous work showing that majority voting does not always scale [3]. In this light, can you better justify this method?\n- Were the LLM consensus results ever grounded? For example, were they validated against a benchmark created by human experts? Please specify.\n\n4) The fact that LLM outputs are approximations and that errors can cascade in a sequential process is a well-understood challenge in ML-based decision processes [1,2]. Could you clarify wether and how your paper provides a new perspective when it comes to framing the problem beyond previously identified challenges, e.g., (a) LLMs are unreliable and (b) workflows need robust error handling?\n\n[1] Sun, Chuanneng, Songjun Huang, and Dario Pompili. \"LLM-Based Multi-Agent Decision-Making: Challenges and Future Directions.\" IEEE Robotics and Automation Letters (2025).\n[2] Dai, Peng, and Daniel Weld. \"Artificial intelligence for artificial artificial intelligence.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 25. No. 1. 2011. \n[3] Chen, Lingjiao, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. \"Are more llm calls all you need? towards scaling laws of compound inference systems.\" arXiv preprint arXiv:2403.02419 (2024).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}