DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems
Authors: Ming Ma
Jue Zhang
Fangkai Yang
Yu Kang
Qingwei Lin
Saravan Rajmohan
Dongmei Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM-based agent systems
failure analysis
intervention
Primary area: foundation or frontier models, including LLMs
TL;DR: IIntervention-driven debugging advances beyond log-based attribution by validating and repairing failures in LLM-based multi-agent systems.

Abstract: Large language model (LLM)–based multi-agent systems are challenging to debug because failures often arise from long, branching interaction traces. The prevailing practice is to leverage LLMs for log-based failure localization, attributing errors to a specific agent and step. However, this paradigm has two key limitations: (i) log-only debugging lacks validation, producing untested hypotheses, and (ii) single-step or single-agent attribution is often ill-posed, as we find that multiple distinct interventions can independently repair the failed task. To address the first limitation, we introduce DoVer, an intervention-driven debugging framework, which augments hypothesis generation with active verification through targeted interventions (e.g., editing messages, altering plans). For the second limitation, rather than evaluating on attribution accuracy, we focus on measuring whether the system resolves the failure or makes quantifiable progress toward task success, reflecting a more outcome-oriented view of debugging. On the datasets derived from GAIA and AssistantBench, DoVer flips 18–28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses. Our findings highlight intervention as a practical mechanism for improving reliability in agentic systems and open opportunities for more robust, scalable debugging methods for LLM-based multi-agent systems.

Published as a conference paper at ICLR 2026
DOVER: INTERVENTION-DRIVER. AUTO DEBUGGING FOR LLM MULTI-AGENT S SYSTEMS
Ming Ma Jue Zhang Fangkai Yang
Yu Kang', Qingwei Lin", Saravan Rajmohan', Dongmei Zhang?
Institute of Neuroscience,
State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Center for Excellence in Brain Science and Intelligence Technology,
Chinese Academy of Sciences, Shanghai, 200031, China
2 School of Future Technology. University of Chinese Academy of Sciences, Beijing. 100049, China 3 Microsoft
mam2 wwwp2222iioncco wwt.einnn.c.... en,
( (juezhang, fangkeilyanggymlcccoof..cc
ABSTRACT
Large language model (LLM)-based multi-agent systems are challenging to de- bug because failures often arise from long, branching interaction traces. The pre- vailing practice is to leverage LLMs for log-based failure localization, attributing errors to specific agent and step. However, this paradigm has two key limitations: (i) log-only debugging lacks validation, producing untested hypotheses, and (ii) single-step single-agent attribution is often ill-posed, as find that multiple distinct interventions can independently repair the failed task. To address the first limitation, introduce DoVer, intervention-driven debugging framework, which augments hypothesis generation with active verification through targeted interventions (e.g., editing messages, altering plans). For the second limitation, rather than evaluating on attribution accuracy, we focus on measuring whether the system resolves the failure or makes quantifiable progress toward task success, reflecting a more outcome-oriente, view of debugging. Within the Magnetic- One agent framework, on the datasets derived from GAIA and AssistantBench. DoVer flips 18-28% % of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses. DoVer also per- forms effectively a different dataset (GSMPlus) and agent framework (AG2), where it recovers 49% of failed trials. These results highlight intervention as a practical mechanism for improving reliability in agentic systems and open oppor- tunities for more robust, scalable debugging methods for LLM--ased multi-agent systems. Project website and code will be available at https::/akaa.m/Doo///
INTRODUCTION
The advancement of Large Language Models (LLMs) has led to the rapid rise of LLM-based agent systems, particularly multi-agent architectures where agents different roles work collaboratively solve complex tasks Li al. (2023); Wu al. (2023a); Hong al. (022)). As these systems are increasingly developed and deployed in production, the need to debug their failures becomes inevitable during their lifecycle. Importantly, by "failures" we do not refer to conventional software errors (e.g., exceptions or crashes), but rather to the errors where the system executes without in- terruption yet produces incorrect or unsatisfactory results Mialon et al. (2024); Yoran et al. (2024). Such failures frequently arise in scenarios where one diagnoses why agent system underperforms on benchmark tasks during the development phase, or when one addresses user-reported dissatisfac- tion (e.g.. 'thumbs-down' signal with textual feedback) from an online deployed system.
Work is done during an internship at Microsoft. Corresponding authors.

---

Published as a conference paper at ICLR 2026
Debugging failures in LLM- agent systems presents unique challenges. These systems typi- cally involve multiple rounds of LLM calls, each with extensive textual context, making manual log inspection labor-intensive. Furthermore, in multi-agent tasks, tracking inter-agent information flow is crucial, as failures often stem from Inter-Agent Misalignment Cemri et al. (2025). Recent efforts address these issues by using LLMSs to analyze system failures ZZhuge al. (2024); Zhang et al. 2202cc a) often via single-agent, single-step failure attribution. In this method, an execution log is input to an LLM tasked with identifying the agent and step responsible for the failure Zhang et al. 2025c). However, as shown in our reproduction study (Section 3), log-based failure attribution is fundamentally limited by the uncertainty of ground truth annotations. This uncertainty arises for several reasons: agent systems often employ multiple strategies (e.g.. ReAct Yao et al. (2023) with distinct failure points in single session, and inter-agent misalignment can render the assignment of responsibility to a single agent or step ambiguous.
To circumvent the limitations of uncertain ground-truth attribution, we propose explicit validation via intervention, introducing Do (Do-then- (Do-then-Verify), an intervention-driven framework for auto- mated debugging. Do explicitly validates failure attribution hypotheses derived from session logs by intervening at suspected failure points, modifying agent instructions task plans, while preserving prior context. The system is then re-executed from the intervention point onward. If the failure resolves, the hypothesis is supported; if it persists despite faithful intervention, the hypothesis is refuted. This process enables an iterative cycle of hypothesis generation and validation.
Do also supports interventions across multiple steps rather than restricting to single-point edits. By decomposing the failure trace into separate trials, we intervene at each and assess the impact. Within the Magnetic-One (M1) Fourney et al. (2024) agent framework, our experiments show that Do recovers 18% and 28% of failures on datasets from AssistantBBecc Yoran et al.) (2024) and GAIA Mialon et al. (2024), respectively. It further enables the validation or refutation of 30-60% of failure hypotheses, depending on task complexity. To demonstrate the generality of Do we apply to another agent framework, AG2 Autooien2 (2025), using a different dataset, GSMPlus Li et al. (2224). Ver again performs effectively, achieving a 49% flip rate on failure cases.
To summarize, our main contributions are: (i) We propose Der,r, an intervention framework for automatically debugging failures in LLM-based multi-agent systems; (ii) We identify and ana- lyze the challenges posed by uncertain ground-truth annotations in log-based failure attribution; (iii) We demonstrate experimentally that Do not only recovers significant portion of failure cases but also enables explicit validation and refutation of failure attribution hypotheses.
2 RELATED WORK
2.1 FAILURE ANALYSIS AND ATTRIBUTION FOR LLM-BASED AGENT SYSTEMS.
L.LM-based agent systems exhibit diverse and frequent failure patterns that accumulate along long execution logs. To characterize why and where errors arise, MAST Cemri et al. (2025) catalogs failures across task interpretation, planning, tool/environment interaction, and verification, while evaluation frameworks Zhuge et al. Arabzadeh al. (2024) argue that end-to-end pass/fail is too coarse and introduce requirement graphs or task-utility criteria that reveal where progress stalls. Extending this perspective to execution logs, TRAIL Deshpande et al. (2025) creates turn- level traces and a fine- -raiine taxonomy (reasoning, planning, execution), empirically showing that even strong long-context models struggle trace debugging.
A parallel line of work seeks failure attribution: identifying the earliest decisive step or agent that is responsible for the earliest sufficient cause of failure (Zhang et al.] (2025c a). However, these attributions are inferred from logs and remain an untested hypothesis unless validated by execution. In Ver, we treat attribution as a hypothesis to be tested. We apply a targeted edit at the implicated location (message, plan, tool call), and rerun the system, judging success by milestone/utility gains. This places emphasis on verified repair and is consistent with trajectory- aware evaluation advocated in Arabzadeh et al. (2024); Deshpande et al. (2025).
Recent contemporancous s further advance failure attribution by introducing reasoning- driven judges Zhu et al. (2025a). abduct-ace act-pre counterfactual scaffolding West et al. (2055)
'hhse works appeared during the final stage of this work preparation the subsequent peer-review period.
2

---

Published as a conference paper at ICLR 2026
causal-inferene formulations Ma et al. 22025) spectrum- failure attribution approach Ge al. (2025), hierarchical error- attributionn method Banerjee et al. (2025). and graph- failure tracing approach Zhang et al. (2025b). Notably, the observation in West et al. (2025) that adding explicit step indices improve attribution aligns with our own prompt-refinemet study in Section
Ge
Emerging works have also explored specialized failure-tracer models trained curated suc- cess/failure trajectories Zhang et al. (2025a); Kong et al.) (2025). These approaches are orthogonal Ver and could be incorporated into the initial failure-attribution stage of Do Dover to strengthen its diagnostic signal.
2.2 DEBUGGING APPROACIES FOR LLM-BASED AGENT SYSTEMS
Beyond failure analysis and attribution, several systems explore how to debug trajectories, i.e., in- tervention and replay.
tools such as AGDebugger Epperson et al. (2025) en- able rewind/edit/re- execute with trace visualization, and graph runtimes like LangGraph LangChain (2025) provide checkpoints, interrupts, and "time-travel" branching. These demonstrate that small, targeted interventions often work, but they are manual and hard to scale. The recent AgentDe- bug Zhu et al. al.. (2025b) work employs an intervention-driven methodology similar to Do although it does not place particular emphasis on multi-agent settings.
From the software-repair perspective, Rahardja et al. (2025) package real agent-system issues into exccutable environments with failing tests (AgentIssue -sune-Bnch) and find low resolution rates for current software engineering agents, highlighting the difficulty of maintaining agent software. In- dustrial experience at Google similarly evaluates agent- based program repair production bugs, showing promise but also current limits Rondon et al. (2025). This motivates auto debugging that verifies edits within the original run and measures intermediate progress. DoVer does by choos- ing minimal intervention, re-running the trajectory, and scoring milestone/utility gains, in line with Zhuge et al. (2024); Deshpande al. (2025).
3 FROM LOG- BOG-BASED ATTRIBUTION TO INTERVENTION-BASIEE DEBUGGING
In this section, we surface several subtleties in existing bag-based failure analysis that motivate our intervention-basee based auto-debugging system. We begin by recapping the prevailing task formulation for log-based single-agent/stpp failure attribution, and then revisit evaluation on an existing bench- mark to highlight sources uncertainty in ground-truth annotation that shape our design.
Log-based Saigll--geen/SSngl-SSttee Failure Attribution. A common setup for log-based fail- ure attribution proceeds as follows. The input is session log, typically from LLM-driven multi- agent system, covering the full trajectory from an initial user query to either a final answer or ter- mination caused by system- -efined stopping rules (e.g., maximum number of replanning rounds). The log is sequence of agent messages organized by turns The task is to identify the earliest single agent and single step responsible for the failure. failure step defined as decisive er- ror: if one were to replace the agent's incorrect action at that step with correct one, the trajectory would subsequently succeed. To evaluate this setup, the Who& When (WW) dataset was introduced in Zhang et al. (202cc). In WW, multiple annotators independently reviewed session logs and then reconciled discrepancies through discussion. Reported results show that state-of tf-th--rr LLMSs at the time achieved below 10% accuracy failure step attribution.
Reproduction and Prompt Refinements. We reproduced the WW step/agent attribution protocol to understand why step-level accuracy is low. Through failure case analysis, we identified two minimal, non-invasive prompt refinements that consistently improve accuracy: (i) adding explicit step indices to the failure log, and (ii) embedding a concise reminder of the annotators' guidance as instructions. With these refinements, attribution accuracy rises noticeably. For example, on the Hand Crafted category of WW (WW-HC) with the setting of including ground-truth answers and adopting
We do consider logs from asynchronous agent executions this work.
3

---

Published as a conference paper at ICLR 2026
Trial 1: Step (1-38) Baseline Plan: APOO City Landmark Firm
User Query
EiccutionArrriiee Navigo Scrolling Info Outhoring
Step 53 (Ambiiguuu Fulture Alt uutibu
36
Please click 2019 APOO
Initial plan
Enier Wanfuutter did diseclly sp of to the bottor Westt uttee.
sheetry age showing the up acdyy
Curing first wesk
Auguot 2015, orearthe
NAdA Astronomy
Trial 2: Step (39-45) Plan Pivot APOD Calendar for Dirnct Dates
Pictures en Day Day
ellicked control..
Excoution: Culendar Foctura Eiporatto!
Unclickablei
Thana strtee abyalis haw andmart Ievisiegin
3D
40
54
65
Plan Updata
Eir Orc" estrutic crror instruction; WesS.ifer inco
named atter Whet lethe of aechiress.m mitfi that building?
Trial 3: Step (8S-57) Reline and Leverage APOD URL Patterns
Exseution: i: Repoatedia patt lik Trsat Proposed Ufi patternneer the end
66
B7
77
87
liput Node
Plan Updatt
Eron Orail estrour stulled
Excoution Node
Stop (88-92) Apply URL-Patterned and Verify
Fror Kode
Error Node WWW
Exccution: Explored the pottr pontessoooonnn stronca)
Dechestriior
Apmove of 410i
WatSurfer
Update
LLH emor
Figure 1: Failure trace of Case in WW-HC, illustrating ambiguity in failure attribution. The session consists of four distinct trials, each initiated by a plan update and executed via ReAct-style Yao jet al. (2023) loop. Different strategies (e.g., direct scrolling in Trial vs. calendar navigation in Trial 2) yield separate error points, making single-step attribution across the session inherently ambiguous. Trial 2 (Steps 53-55) further shows inter-agent misalignment: the Orchestrator issued an invalid instruction, while the WebSurfer compounded the error by executing unrelated action.
the All-at-Once prompt (a single LLM call over the full session log), step attribution accuracy for GPT-4o increases from 6% to 24% Reproduction details are provided in Appendix A
Impact of Uncertain Ground-Truth Labels on Attribution. Despite these gains, absolute step- level accuracy remains low for practical deployment. By comparing model outputs against the ground-truth (GT) labels, we found that uncertainty in the GT annotation is major contributing factor. Specifically, for the 29 GAIA Mialon et al.] (2024) cases WW, our independent review suggests that 14 of 29 cases exhibit GT uncertainty. This is consistent with WW, which reports annotator uncertainty of 15-30% and an average initial disagreement of 20% Zhang et al. (025cc).
To assess the impact of GT uncertainty model performance, we stratified evaluation into two subsets. For the 14 uncertain cases, average step attribution accuracy is 24% for GPT-4o and 7% for GPT-5. In contrast, for the remaining certain 15 cases, average accuracy increases to 44% for GPT-46 4o and 53% for G These results indicate that GT uncertainty substantially affects model performance. Per-case annotation notes and results are provided in Appendix A
Sources of Uncertainty in Failure Agent/Step Annotations. Our re- re-annotation surfaced three sources of GT uncertainty among the 14 uncertain GAIA cases WW:
(1) Multiple trials within single session. Modern agentic systems frequently employ ReAct- style Yao et al. (2023) loops with repeated planning-executtee cycles, producing multiple trials of task solving. We define trial as the contiguous span starting from planning step and continuing through the execution steps for that plan. Each trial may contain its own decisive error step(s), espe- cially when exploring new strategies or branches. For instance, Figure illustrates the failure trace of Case 3 in WW-HC, which comprises four distinct trials within a single session. Different strate- gies were explored (e.g., Trial attempted to locate the target webpage via direct scrolling, whereas Trial used the calendar feature), each introducing distinct potential error points. Consequently. enforcing single-step attribution across the entire session is intrinsically ambiguous. In our review, 9 of the 14 uncertain cases exhibit this pattern.
Unless otherwise specified, GPT-40 and GPT-5 refer to the versions of 201--44-20241222 and "GPT- 5-chat-20250807", respectively. All T.I.M API calls made through Azure OpenAl using default parameters.
4

---

Published as a conference paper at ICLR 2026
Hypothesis Ag
Triad
Hypothesis Ag
osevutin
fullure
Hypothesis kg
Trat
Trial 5 intervention
lask succoeed
NEW CEMEOsAttio
Trial Segmentation
Failure Attribution
Intervention Generation
Intervention Execution
Figure 2: Dover (Do-then-Verifyy Debugging Pipeline. (1) Trial segmentation: split the failed session log into trials using re-plan steps as cut points. (2) Failure attribution: for each trial, propose
a hypothesis hy that marks a faulty step or agent. (3) Intervention generation: turn h into an actionable intervention that edits either the plan or the attributed message or step in the original log- (4) Intervention execution: replay the trajectory in place, i.e., preserve all steps before the intervened step, then execute the intervention and measure progress of the new log. Colors indicate plan/re-plan (blue), execution (green), attributed failure (red), terminal failure (dark red), terminal success (dark green), intervention (yellow), new plan/re-plan (blue hatch), and new execution (green hatch).
(2) Ambiguity from multi-agent coordination. Attribution to specific agent/step can unclear when the underlying issue stems from inter-agent coordination. For instance, when a sub-agent fails to carry out an instruction, responsibility may lie with the sub-agent's capabilities or with ambiguous/misaligned guidance from a supervising agent. In Trial 2 of Figure D for example, the Orchestrator agent instructed the WebSurfer agent to click on a non-existent control. Rather than reporting the issue, the WebSurfer instead clicked the target year "2015". In this case, both agents exhibited failures, making attribution to either one alone inappropriate. Such ambiguous attribution aligns with the Inter-Agent Misaligument category identified in Cemri et al. (2025). We observed this phenomenon in 5 of the 14 uncertain cases.
unrelated control that had connection to
(3) Cross-annotator alignment challenges. Even with adjudication, achieving fully aligned labels can be difficult. For 7 of the uncertain cases in WW, we found no clear eπror at the GT-designated step, suggesting that differing interpretations can persist despite careful protocol design.
Implications for Intervention-Baae Based Debugging. The above analysis suggests that log-only fail- ure attribution can fundamentally suffer from the issue of uncertain ground-truth labels. Conse- quently, we propose explicit validation via intervention: hypothesize the failure step, intervene on (e.g.. replace the action or instruction), and verify whether the trajectory subsequently succeeds. This protocol operationalizes the "decisive error" definition while simultaneously eliminating de- pendence on noisy human labels and thus reducing annotation burden. Moreover, two design take- aways follow from uncertainty study. (i) Trial awareness. Because logs often contain multiple planning-exeee execution trials, interventions must be applied at the trial level, motivating our rial-based intervention framework. (ii) Role-specific interventions. Given ambiguity in attributing responsi- bility between an orchestrator and its sub-agents, we adopt a clear taxonomy interventions: (a) interventions the orchestrator's 's plan and its instructions sub-agents, and (b) direct capability improvements for sub-agents (e.g., skills). These implications motivate our intervention-base auto debugging system; in the next section, detail the framework and its concrete instantiations.
4 METHODOLOGY
Following the implications for intervention-based debugging, we present DoVer, a do-then-verify debugging pipeline that turns failure-attribution hypotheses into controlled edits and checks whether those edits change outcomes. As shown in Figure 2 Do Ver consists of four stages: (1) trial segmen- tation to break an execution log into trials, (2) hypothesis generation to hypothesize a failure step or agent, (3) intervention generation to synthesize a testable change, and (4) intervention execution with differential evaluation against the original using task success and a progress score.
5

---

Published as a conference paper at ICLR 2026
4.1 DOVER PIPELINE
Trial Segmentation. After a task is executed by an agentic system, we have obtained a long ex- ecution session log t {(ae, m,) H"= where or represents the active agent that produces the message m, at step 1, and de keeps stateful information that is necessary for state restore and replay. For example, this includes historical context (e.g.. prior messages sent LLMs) as well as browsing history for agents acting in web-browsing rolc. Since modern LLM-based agent systems Fourney al. (2024): SSmolAgents 2025): AutoGen2 (025) have the self-reflection capabilities Yao et al. 2023 Shinn et al. (2023) and re-plan after reflection, we then segment the trace T into trials T² using re-plan steps as segmentation point, as illustrated in Figure This trial segmentation shortens context so LLMs can reason about a single causal chain, and enables independent and parallel inter- ventions for efficiency. Thus, multiple hypotheses can be proposed for single session trace, which essential many failures admit more than one viable repair.
While one could implement trial segmentation by leveraging system-speciif message patterns (c.g., in MI, plan re-plan steps often begin with "We are working to address the following user quest"), ) we instead adopt a prompt-based approach. Specifically, we employ LLMs to reason over the full session log and identify planning-related steps. This method generalizes more effectively to other agent systems whose log patterns for planning steps are not known a priori. The prompt used for trial segmentation is provided in Figure 5 in Appendix B
Failure Attribution. For each trial 7", we generate candidate failure attribution hypothesis
h4
(a) rt),
where 7 the step index of attributed failure step, i is the trial index, ä' the suspected agent, and r natural-language rationale. We build on existing log-based attribution methods (e.g.. our improved All-at-Once prompt from Section 3 but crucially do not require perfect precision, since correctness will later be tested via explicit intervention. Here we adapt the All-at-Once prompt (see Figure 6 in the Appendix B) so that can be applied session logs segmented into trials by the preceding step.
Intervention Generation. In this stage, failure attribution hypotheses are transformed into concrete interventions. Each intervention I represents a targeted edit to the failing context, informed both by the specific failure hypothesis and by the broader context of the intervention step. As noted in Section 3 ambiguity may arise in attributing responsibility between the orchestrator and its sub- agents. We thus distinguish interventions directed at orchestrator from those directed at sub-agents.
To keep our method more agnostic to the underlying agent architecture, we focus primarily on interventions the orchestrator level, i.e., interventions applied at the message-pas passing layer via direct edits to agent messages. This choice improves generality and simplicity but introduces a trade- off: we cannot directly intervene sub-agen sub-agents to enhance their capabilities, which would typically require substantial system-level modifications (e.g., extending the WebSurfer agent to support in- page search). In summary, the interventions considered in this work fall into two categories:
Modified Instructions to Sub-Agents: Adjusting the orchestrator's messages to sub-agents to clar- ify intent, correct arguments, or supply missing context. This approach indirectly influences sub- agent behavior.
Plan Updates: Revising the orchestrator's high-level plan, e.g., reordering, decomposing, placing steps, to route around the identified failure.
The prompt used for intervention generation is provided in Figure g in Appendix B
Intervention Execution. The agentic system replays each trial with interventions applied in-situ at the suspected failure step. All steps are preserved before the intervened step and execute the inter- vention 4, yielding a counterfactual trace ř = {F1, T2, ... TT-1, Note that one intervention creates new trace, verifying the failure attribution of each trial.
4.2 EVALUATION METRICS
We consider two sets of evaluation metrics to address our research questions: (1) whether failed cases be turned into successful ones through intervention and (2) how effective our debugging system is at validating or refuting the initial failure hypothesis. To reduce the impact of execution randomness (e.g., LLM stochasticity), we perform three independent runs for each intervention.

---

Published as a conference paper at ICLR 2026
Metrics for Turning Failures into Successes. To answer the first question, introduce two mallure-flipping related metrics: Trial Success Rate and Progress Made. The Trial Success Rate is defined as the ratio of trial runs that successfully complete the task after intervention. The Progress Made metric captures whether intervention brings a failed trial closer to success, even does not ultimately succeed. Thus, it provides more fine-grained measure of improvement when the intervention makes the failed case "more correct".
Specifically, when an intervention does not yield full success and human-annotated solution steps are available (e.g.. in the GAIA benchmark Mialon et al. (2024), we could evaluate the degree of progress by comparing the new execution trace against auman-annotater steps. For each log we extract up to K S 5 milestones {m") new trace accomplishes. Both the extraction of milestones and their evaluation are performed using LLMSs with carefully designed instructions. The actual prompts are provided in Figures and T0 in Appendix B We then define the milestone achievement count for a trace y as:
{m²} k==1 and measure how many of these milestones the
A(7)
Σ Il milestone m² is achieved in 7|
We measure Progress Made as the ratio of additional milestones achieved:
Prog(T -i) A(tr) -A(T) E [--1,1]
where řy is the new execution trace after intervention, K is the number of extracted milestones from human-annotated trace.
In settings where human-annotated intermediate steps unavailable, the progress metric above cannot be directly applied. In such cases, one may instead directly compare execution traces before and after intervention with LLM-as-a-judge, We leave this alternative evaluation for future work.
Metrics for Validating Failure Hypotheses. To evaluate the effectiveness our debugging sys- tem in validating or refuting the initial failure hypothesis, we classify each trial after intervention into four categories: Validated, Partially Validated, Refuted and Inconclusive. The Inconclusive cat- egory necessary because we frequently observe that agents fail follow the intervened instruction, resulting in unsuccessful trials. In such cases, it is unclear whether the outcome stems from an in- correct failure hypothesis or from other limitations of the system that prevent the intervention from being carried out.
To handle this ambiguity, we conduct a comparative analysis of traces before and after intervention. We leverage LLMS to determine whether the intervention was faithfully executed by the agent, resulting in boolean metric "is intervention ffulfileedd for each trial run. The prompt employed in this work can be found in Figure in the Appendix B Together with the overall Trial Success Rate and Progress Made metrics, we define four outcomes as follows:
Validated: At least 2 of repeated rums succeed.
Partially Validated: Fewer than 2 of 3 runs succeed, and at least 2 of 3 runs both fulfill the intervention and show additional 20% progress (i.e., they advance by one key milestone).
Refuted: Same as "Partially Validated," except that progress does not exceed 20%.
Inconclusive: All other cases.
In summary, these metrics allow to quantify both the extent which interventions make failed cases more correct and how effectively the system validates or refutes hypotheses. We now apply them in the subsection analyze experimental results.
EXPERIMENT RESULTS
5.1 EXPERIMENT SETUP
Agent System. We consider two distinct agent frameworks in this work. Following Zhang et al. 2025c), begin by conducting experiments using Magentic-One (M1) Foorney et al. (202)),
7

---

Published as a conference paper at ICLR 2026
popular LLM-based multi-agent framework that was also used for collecting failure traces in WW.
We enable Ver on MI by adapting the manual debugging tool AGDebugger Epperson so that Do Ver can (without human intervention): (i) save the state at each step as a checkpoint: (i)
Epperursonnppnnnnnnnnoooo et al. (2025)
load checkpoints from prior failed run; (iii) intervene by modifying an agent message at specified step; and (iv) resume execution from the intervened step.
To evaluate Vor'r' generality, we further construct a MathChat multi-agent system using sec- ond framework, AutoGen2 (AG2) AutooGen2 (2025); Wu et al. (2023b). This MathChat system is instantiated using the prompts provided in MAST Cemri et al. (2025). and we extend AG2 with checkpointing and re-execution capabilities analogous to those in MI. Implementation details and practical lessons for reducing the integration burden on new frameworks are provided in Appendix(C
Datasets. We first follow Zhang et al. (2025c) and include all cases in the Hand Crafted category of WW: 28 cases from AssistantBench Yoran al. (2024) and 29 cases spanning all three GAIA
levels Miilon et al. 2024). To increase data volume, we additionally include all 53 Level-1 cases from GATA's validation set; after excluding those already present in WW, this yields 45 extra cases. We refer to these three sets as WW-AB, WW-GAIA, and GAIA-Level-I, respectively.
For the MathChat system, following MAST Cemri al. (025). we additionally use the GSMPlus dataset Li et al. 2024). Concretely, we adopt the 2,400 examples in the "testmini" split and re- collect execution traces with checkpoints for all problems, forming the GSMPlus setting used in our AG2-based experiments.
Failure Trace Collection. We begin with an initial run over all cases and evaluate outcomes to identify failure traces. The failure logs published with WW and MAST are not directly usable for our purposes: logs of agent messages alone are insufficient to support replay and targeted intervention. The number of failed cases per dataset is given in Table I ("Failed Cases" column). The overall success rate on the full GAIA Level-1 set matches the value reported for Magentic-One Fourney al.] (2024), suggesting that our collected failure traces faithfully reflect MI's capabilities. As in WW and MAST, we use GPT-4c both to generate traces and to power the intervened session runs.
Total
Failed Cases
Intervened Cases
Intervened Trials
Trials per Case
WW-AB
23
72
3.1
WW-GAIA
29
26
25
99
4.0
GAIA-Level-1
45
26
25
63
2.5
GSMPlus
2400
214
141
198
1.4
Table 1: Summary of failed and intervened cases across datasets, showing the total number of cases, failed cases, intervened cases, total intervened trials, and the average number of trials per case.
Auto Debugging with Dor.r. We apply DoVer to each failed trace using GPT-4o while obtain- ing the progress made metric and failure hypothesis validation results with GPT-5. As described in Section 4 Do first segments the full trace into distinct trials, then performs failure attribution and intervention generation for each trial. Finally, it collects the intervened session traces and conducts comparative analysis. Table reports the number of cases for which an intervention was success- fully generated (L.LMs may occasionally conclude, incorrectly, that no mistake occurred), the total number of intervened trials, and the average number of intervened trials per case. On average, we perform about 3 (1.5) intervened trials per case in the AB and GAIA (GSMPlus) datasets, indicating that most cases contain multiple trials and multiple potential failure points, making them worthwhile targets for debugging.
5.2 QUANTITATIVE EVALUATION RESULTS
Table 2 presents the experimental results for the flipping metrics. For making failure cases
more correct, the Trial Success Rate across all intervened trials is 17.6% for cases in WW, compared to a higher success rate of 27.5% for GAIA-Level-1 cases. This difference can be explained by
the fact that WW contains more challenging cases (e.g., Level-2/3 GAIA tasks). A similar pattern is observed for the Progress Made metric: interventions yield a 15.7% improvement (i.e., nearly one key milestone) in GAIA-Levvl-1, but considerably less progress in WW-AB and WW-GAIA.
8

---

Published as a conference paper at ICLR 2026
Intervened Trials
Trial Success Rate
Progress Made
WW-AB
72
17.6%
+0%
WW-GATA
99
17.6%
+8.8%
GAle-Level-1
27.5%
+15.7%
GSMPlus
198
49.0%
Table Experimental results on failure-fliping metries across settings. The table reports the num- ber of Intervened Trials, the Trial Success Rate, and the average Progress Made.
Notably, for WW-AB, almost no progress is achieved after intervention, suggesting that in some situations interventions may even hinder progress toward success. Finally, in the GSMPlus setting, Do achieves nearly a 50% trial success rate, underscoring that its effectiveness generalizes well across datasets and agent frameworks. Note that the Progress Made metric cannot be computed for GSMPlus due to the absence of human-annotated solution steps in the dataset.
Intervened Trials
Partially Validated
Validated
Inconclusive
Refuted
WW-AB
72 11 (15.3%)
48 (66.7%)
3 (4.2%)
10 (13.9%)
WW-GAIA
99 16 (16.2%)
57 (57.6%)
5 (5.1%)
21 (21.2%)
GAIA-Levela
22 (34.9%)
18 (28.6%)
(12.7%)
(23.8%)
Validation outcomes of failure hypotheses across datasets. The table reports the number and percentage of trials classified as Validated, Inconclusive, Partially Validated, or Refuted.
Table 3:
Turning to the validation of failure hypotheses, Table 3 shows that for both WW-AB and WW- GAIA, the proportions of Validated and Refuted hypotheses are similar, each around 15%, while the majority (about 60%) fall into the Inconclusive category. contrast, GAIA-Level-1 exhibits higher rates of both validated and refuted hypotheses, with inconclusive cases reduced to about 30%. This pattern suggests that the more difficult cases in WW make harder for the agent system reliably carry out the intended interventions.
5.3 ABLATION STUDY
Impact of Different DoVer Underlying Models. To test whether DoVer depends proprictary frontier model, we vary its debugging model while keeping failure traces, the agent system, and all prompts fixed. In the WW-GAIA setting, replace GPT-4o with two locally hosted open-source models, 3wen3--8 8B and Qwen3-32B in thinking mode. As shown Table[4 Qwen3-88 recovers 11.3% of 77 trials and Qwen3-32B recovers 16.9% of 87 trials, compared 17.6% for GPT-4o 4o over 99 trials. These results show that (i) DoVer does not rely on a single proprietary backend and works with substantially narrow the gap to GPT-4o, underscoring DoVer's generality and practicality for open- source deployments.
local models, and (ii) larger open-source models (e.g., Qwen3-32B)
Do Model
Intervened Trials
Trial Success Rate
Qwen 88 (0-shot)
77
11.3%
Qwen3-8B (3-shot)
77
14.3%
Owen (0-shot)
87
16,9%
GPT-4c (0-shott
99
17.6%
Table 4: Ablation of DoVer models and few-shot prompting in the WW-GAIA setting.
Effect of Few-Shot Prompting for Smaller DoVer Models. To assess whether prompt-based guidance can mitigate the limitations smaller models, we compare Do performance with and without few-shot examples. In the WW-GAIA setting described above, enhance the intervention- generation prompt for the Qwen3-82 model by adding three manually curated few-shot examples, each demonstrating how the orchestrator refines initially suboptimal instruction into a clear and effective one, along with brief context before and after the intervention. The resulting trial success rate is reported "Owen3-88 (3-shot)" in Table 4 The results show that Qwen 8B with this

---

Published as a conference paper at ICLR 2026
3-shot prompt achieves 14.3% trial success rate, compared to 11.3% in the zero-shot setting. This improvement indicates that even small models can benefit considerably from lightweight in-context supervision, suggesting that richer prompt design future supervised or reinforcement learning on intervention data may further narrow the gap to larger models.
Compare with other Self-Improvement Methods. To contextualize DoVer's gains, we compare against two self- imfrooeereeettt style methods adapted from Self-Refine MMdaan et al. (2023) and CRITIC Gou al. (2023) in the WW-GAIA setting. In the Self-Refine-style baseline, the under- lying LLM first critiques the final answer given the full session log and then generates revised answer in a second pass. In the CRITIC-Style baseline, we similarly elicit feedback from the final answer and log, but inject this feedback as additional agent message and allow the agent system to run for one extra round so that all tools and sub-agents can react to it. Across all 26 failed WW- GAIA cases, neither baseline is able to flip any failure into success (0% recovery), whereas Do Ver recovers 17.6% of trials (Table 2 Further examination reveals why these self seff-improvement meth- ods fall short: trial trajectories between the initial failure point and the final answer often long, noisy, and highly divergent, making end-of-trace refinement insufficient for reliably redirecting the system. In contrast, Do performs in-situ interventions at potential failure points, enabling timely and targeted corrections that are essential in multi-agent settings.
Qualitative Analysis. Detailed case vignettes and tool enhancement analysis are provided in Ap- pendix D and E.
6 CONCLUSION
We introduced Do an intervention-driver framework that reframes debugging in LLM-based multi-agent systems as a do-then-verify process. Rather than relying solely on log-based attribution, DoVer operationalizes debugging by formulating and directly testing failure attribution hypotheses via targeted interventions. Across datasets from AssistantBench and GAIA under M1, DoVer re- covers 18-28% of previously failed trials, and it recovers 49% of failures in GSMPlus when applied within a different agent framework (AG2). Beyond these recoveries, DoVer consistently produces measurable improvements and verifies or falsifies most hypotheses. This outcome-oriented perspec- tive highlights intervention as a practical and scalable tool for improving reliability, while reducing dependence on ambiguous human annotations. By bridging failure analysis with practical repair, Vever takes a step toward more robust, verifiable, and celf-improving multi-agent systems.
7 LIMITATIONS AND GENERALIZABILITY
Our empirical evaluation provides evidence of feasibility across two frameworks, though the ered tasks and model families represent only a subset deployed agentic systems. Consequently, do not evaluate long-running production workloads or settings with striet safety-critical quirements.
Methodologically, the two components of Do trial segmentation and intervention- based val- idation, are conceptually general, but our current instantiation imposes several preconditions. First, the agent framework must expose sufficiently rich interaction logs and a checkpoint/replay interface so that we can reconstruct contiguous planning-execution trials and splice in edited messages. In- tegrating Ver into systems without such facilities (c.g.. asynchronous or black-box orchestrators) would require additional instrumentation or higher-level abstractions over the interaction history, and our experience in MI and AutoGen2 shows that adding checkpointing still requires non-trivial engineering effort. Second, in this work we restrict interventions to the orchestrator's 's textual mes- sages; we neither modify sub-agent code nor synthesize new tools, which limits our ability to repair failures rooted in missing capabilities rather than mis-specified coordination. Finally, part of our analysis (milestone progress and "intervention fulfilled" labels) relies on LLM-as-a-judde assess- ments, which may introduce biases despite careful prompt design. Exploring broader domains and architectures, richer intervention spaces (c.g., tool augmentation or code changes), and more human- grounded evaluation protocols are potential directions for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their constructive comments and suggestions, which helped improve the clarity and quality of this work. We also thank Tianming Yang. Yanjie Gao and Anyan Chen for helpful discussions. We further thank the creators and maintainers of GAIA, Assis- tantBench, GSMPlus, Magentic-One AutoGen2, AGDebugger and related open-source tools and benchmarks for making their datasets and systems publicly available, which enabled the experiments in this paper.
REFERENCES
Negar Arabzadeh, Siqing Huo, Nikhil Mehta, Qingyun Wu, Chi Wang. Ahmed Hassan Awadal- lah, Charles L. A. Clarke, and Julia Kiseleva. Assessing and verifying task utility in LLM- powered applications. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings the 2024 Conference on Empirical Methods Natural Language Processing, pp- 21868-21888, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.11665/w1/2024ecmmllppoo mainopmma.m..................... 1211. 1219.
URL https://aclanthol..o org/
1219/
AutoGen2. Autogen 0.2: An open-source programming framework for agentic ai. https:// microsoft. 1tw.:/tuib..//aattoee////...... 2025, Accessed: 2025-09-01.
Adi Banerjee, Anirudh Nair, and Tarik Borogovac. Where did it all go wrong? a hierarchical look into multi-agent error attribution, 2025. URL https::/ org/ wowg/abs... abs 22sgg/bb//251... 04886
Mert Cemri, Melissa Z. Pan, Shuyi Yang. Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent Ilm systems fail?, 2025. URL http::ttps:atttt org/abs/ 2503.... 13657
Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. TRAIL: trace reasoning and agentic issue localization. CoRR, abs/2505.086388 2025. doi: 10448500/ARXIV.2555..668 08638
URL httth:ttt///dtt w0t.//...0//.0../........ 2505.
Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang (Eric) Zhu, and Saleema Amershi. Interactive debugging and steering of multi-agent ai systems. In Proceed- ings of the 2025 CHI Conference on Human Factors Computing Systems, CHI '25, pp. 1-15. oow.oooo/....100000000000.0011.0
ACM, April 2025. doi: 000114553706599837733888 URL UtLhttp::// 3706598. 3713581
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. Magentic-one: A generalist multi-agent system for solving complex tasks. CoRR, abs/2411 1.0468, 2024. doi: 40.48550/ARXIV2211 .04468. URL 710.8555/arXiiv. 2411 04468
Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang. Who is introducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis, 2025.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint vrXiv:2305.11732 2023.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative framework. In The Twelfth International Conference Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024. URL https: //openreview. wtt.rrmmmmm.....mAAAA.....

---

Published as a conference paper at ICLR 2026
Fanqi Kong, Ruijie Zhang, Huaxiao Yin, Guibin Zhang, Xiaofei Zhang, Ziang Chen, Zhaowei
Zhang, Xiaoyuan Zhang, Song-Chun Zhu, and Xue Feng. Aegis: Automated emror generation and attribution for multi-agent systems, 2025. URLhttts://aaxxv..a http///xxivvvo/oo wwtpp.wwww.w.////.s.s.....
LangChain. Langgraph: Time travel (checkpoints & forking) human-in-the-loop ww.pio//gggaaa https::/langchainn aai-aai.. httpa//oooooaaggggaaaaooooo----o homaaaaaa///ma.n.ooa thetthe_________________________
coop/timettavee 2025, Accessed: 2025-09-01.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicative agents for "mind" exploration of large language model society. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA. USA, December 10 16, 2023, 2023. URL httptttt:////peeeeee nips. wttp::/...ps.....ppppppp.a cc/papez filees/ppperr//222/aaa a3621 753000/0000777711100302222222243
html
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of lims as mathematical problem solvers. arXiv preprint arXiv:2402. 19255, 2024.
Guoqing Ma, Jia Zhu, Hanghui Guo, Weijie Shi, Jiawei Shen, Jingjiang Liu, and Yidan Liang- Automatic failure attribution and critical step prediction method for multi-agent systems based on causal inference, 2025.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534-46594, 2023.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GATA: benchmark for general Al assistants. In The Twelfth International Conference Learning Representaatins, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL hhttps://cpenrew//
n www.:eeet//ooeeeeeeee ForumReact bbxaai
Alfin Wijaya Rahardja, Junwei Liu, Weitong Chen, Zhenpeng Chen. and Yiling Lou. Can agents fix agent issues? CoRR, abs/2505.20749, 2025. doi: 1044550/ARXIV22555227749 URL https:
7/doi. aoooioorg/10.. w...448855aiii.. 2505. 20749
Pat Rondon, Renyao Wei, José Cambronero, Jürgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano, and Satish Chandra. Evaluating agent-based program repair at google. In 47th IEFF/ACM International Conference on Software Engineering: Software Engineering in Prac- tice, SEIP ICSE 2025, Ottawa, ON, Canada, April 27 MMay 3, 2025, pp. 365-376. IEEE, URL
2025. doi: 10.1 SEWVICSSEEEEIP6355.02222 ICSE SEIP66354 22IIP66354.025.... 00038.
www.eeii..ooog/ooo 1109/ 1109/
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz. Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL (tt.///8886822222666188888888888 0e90 htt.://2essacaoeo-oooaaaaat-t--- Con.:racctttootteeeeeeeee.ttt html html
nips wttp:esssspps....ppppeee..e www.ss/papeee///////e hhahh/
SmolAgents. smolagents: barebones library for agents that think in code. https: /ttt:s:///ithbb.
com/
2025. Accessed: 2025-09-01.
Alva West, Yixuan Weng, Minjun Zhu, Zhen Lin, Zhiyuan Ning, and Yue Zhang. Abduct, act, predict: Scaffolding causal inference for automated failure attribution in multi-agent systems, 2025.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, hls/2308.08155, 2023a. doi: 10.48550/ARXIV. 2308.08155. ht..sttss::/ tonttss://dof http://ooo.org/././/// 0tt1111..885500 48550/ 2308 081 08155.
12

---

Published as a conference paper at ICLR 2026
Yiran Wu, Feiran Jia, Shaokun Zhang. Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu. and Chi Wang. Mathchat: Converse to tackle challenging math problems with Ilm agents. arXiv preprint arXiv:2306.000 00330000033,,.. 2023b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023. net/
URL wttppttss.//ppeeeeiiiiicec hwtp:/eeeeffooooeeo forum? id=WE_ SIINEE_llYULL--
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024. pp. 8938 8968. Association for Computational Linguistics, 2024. doi: URL
emn p-ma 505
Guibin Zhang, Junhao Wang. Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. Agentracer: Who is inducing failure in the Ilm agentic systems? arXiv preprint arXiv:2509.03312, 2025a.
Heng Zhang, Yuling Shi, Xiaodong Gu, Haochen You, Zijian Zhang, Lubin Gan, Yilei Yuan, and Jin Huang. Graphtracer: Graph-guided failure tracing in Ilm agents for robust multi-turn deep search, 2025b. URL
org. abs/ 2500 10581
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which agent causes task failures and when? on automated failure attribution of LLM multi-agent systems. In Forty-second In- ternational Conference on Machine Learning. 2025c. URL /ettps::/cpenreviers /eett/
forum? f d=Gaz
TYx
Chenyang Zhu, Spencer Hong, Jingyu Wu, Kushal Chawla, Charlotte Tang, Youbing Yin, Nathan Wolfe, Erin Babinsky, and Daben Liu. Raffles: Reasoning-bas attribution of faults for Ilm systems, 2025a.
Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang. Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, Xiaoteng Ma, Xiaodong Yu, Gowtham Ramesh, Jialian Wu, Zicheng Liu, Pan Lu, James Zou, and Jiaxuan You. Where Ilm agents fail and how they can learn from failures, 2025b.
Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang. Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-judee Evaluate agents with agents. CORR, abs/2410. 2024. doi: 10445500ARXXI.241000000 48550/arXiv. 2410. 10934
URL
org/10.
A REVISIT LOG-BASED FAILURE ATTRIBUTION ON THE WHO& WHEN DATASET
In this section, we revisit the log-based failure attribution task on the Who& When (WW) dataset. We present the details of our reproduced evaluation on WW as well as two minimal, non-invasive prompt refinements. We further examine the ground-truth labels in WW. providing detailed annotation notes for each examined case and discussing sources of uncertainty during ground-trus -tuth annotation.
Our reproduced evaluation on WW focuses on the All-at-Once (AAO) method, in which a single LLM call made over the full input log. While we also reproduced results for the other two methods, Step-by-Steep and "Binary Search" introduced in WW, focus AAO because it requires only one LLM call and achieves performance comparable to the other methods. our reproduction, experiments with both GPT-4o and GPT-5, repeating each run three times
13

---

Published as a conference paper at ICLR 2026
Hand Crafted
Algorithm Generated
Method
Agent-Level Ace.
Step-Level Aec.
Agent-Level Ace.
Step-Level Acc.
Random
12.00
4.16
29.10
19.06
Baseline (WW)
55.17
5.26
54.33
12.50
Baseline (GPT-4o)
55.17=8.009
6.04+22.23
55.16+2.771
15.28+2.70
Step Index (GPT-4c)
52.30=:1.00
20.69+2.98
58.73+3.46
40,47:+4.00
+ Guidance (GPT-4o)
59.19=1.99
23.56:14,98
1..41=1883
35,45
+ Guidance (GPT-5)
19.19
2..56++1.00
62,43-+11.6
45.77++1.65
Table 5: Reproduced evaluation results on the WW dataset using the All-at-Once method with the ground-truth annotation. Results are reported for both Hand-Crafted and Algorithm-Generated scenarios at agent-level and step-level accuracy. Rows show baseline results from WW, our repro- duction with GPT-4o, and refinements with explicit step indices and guidance reminders, well as the latest GPT-5 model.
reduce randomness. Detailed results are shown in Table Following WW, we include the ground- truth annotation in the AAO prompt (the "With Ground-Truth setting); results without ground-truth are similar and omitted for brevity.
Table first lists the "Random" setting and the reported results in WW (denoted as "Baseline"). Examining the baseline failure cases yields two observations. First, for some cases the predicted failure step index is hallucinated, i.e the index exceeds the total number of steps in the log. Specif- ically, we find that the percentages of cases exhibiting this issue are 13.8% and 20.6% for the "Hand Crafted" and "Algorithm Generated" scenarios, respectively. To mitigate this, add explicit step indices to the failure log (see Figure B). The corresponding results appear in the "+Step Index" row of T This simple change substantially improves step-level attribution accuracy. We addition- ally verity that no outputs have out-of-range indices after this modification. Second, the baseline AAO prompt does not explicitly restate the guidance used during ground-truth annotation, which may cause LLMs and human annotators to apply different criteria for the failure agent/step. To align the criteria, we embed concise reminder of the annotators' guidance into the baseline prompt (see Figure 3). With this addition, attribution performance improves further; see the "+Guidance" row in Table
Despite these prompt refinements, step-level attribution accuracy remains only around 20%. even with the latest GPT-5 model. To better understand this gap, we compare model outputs with ground- truth labels and find that major contributing factor is uncertainty in the ground-truth annotation. As discussed in Section 3 we identify three sources of annotation uncertainty. Table 6 provides detailed annotation notes for each GAIA case presented in WW. For example, for Case 3, we list the step ranges for all 4 trial presented in the log. In each trial, we also provide the annotation notes for key steps (c.g., for the current ground-truth annotation Step 32, we agree that it can be potential error step). Brief trial summary for later Trials 2/3/4 are also given in which we explicitly point out whether new strategies of solving task are explored. Note that exclude Case 25 due ambiguity in its question (i.e., two different ways of interpreting the year of 2019).
In our independent annotation, we also tag each case along the following dimensions: (i) whether the ground-truth failure step appears correct; (ii) whether alternative failure steps can be justified when the log contains multiple trials exploring different strategies; (iii) whether agent/step attribution is ambiguous; and (iv) whether API errors or flaky behavior are present. We then mark a case as uncertain any of the first three tags positive. The full tagging results are summarized in Table which also includes auxiliary information such as the number of trials, the GT failure step from WW, and the outputs from each model run. Outputs matching the GT failure step are highlighted in green. As shown, agreement between model outputs and GT labels is substantially higher for cases with uncertain GT annotation (i.e., where the "Is Current GT Uncertain?" column equals 1) than for cases with more certain GT annotation. This finding indicates that current log-based failure attribution can be confounded by uncertainty in ground-truth truth labels, motivating the intervention- based debo-debggg approach presented in the main text.
14

---

Published as a conference paper at ICLR 2026
Optimal Prompt Template
assistant tasked with analyzing multi-agen conversation when solving real world probler.
Innotation Guideline.
Responsible Agent:
s) Select the single agent that should be directly responsible for this foilure in your wind. Allow for sone subjectivity, but be prepared to give your reasons.
b) Don't be too strict. If there exist agents that do redundant steps and agents that nake mistakes, choose the agent who nakes nistakes.
If there are agents that make dbvinus eistaces, decide one single in mind. multiple
nake mistakes, choose that rade the nost sericus mistake.
step:
a)
single sistake agent, decide single step. Mistake step sust by agent.
b) If the nistake agent nakes nistakes in miltiple steps, choose the first step.
<) Incex fron B.
Failure Reasons:
5) First, use natural language to describe the reason. E.g.>
"The agent wrete the wrong code, "
b) Make the reader could understand the annotations.
Others:
a) Accurately record
labeling. b) Mark annotation you have any uncertain,
vatee discuss later.
Mille follouing the Amotation Guideline above, you must still strictly produce the final answer in the exact output Format specified below.
The problem is: (prablan)
The Answer for the problen (grourd_trvffe truth)
Identify which
explain the
conversation:
(step_idx(_)) (agort_name_t): (content_1)..
Based conversatioo,, please prediet folloxing:
1. The name the agent who made a nistake that should be directly responsible for the wrong solution to the real world probler.
If there are agents that nake abvious mistakes, decide one single agent in your wind. Directly output the nane of the Expert. 2. In which step the sistake agent first made sistake. for exarple, in conversation structured as follows:
[Stop
"xx""
"xxx""
[Step "agent
[Stop 'agent
}.
each entry represents "step' where agent provides input. The 'x' symbolizes the speech of each agent, If the mistake is in agent ets speech, the stop number is the second speech by 'agant at contains the nistake, the step nueber is 3, and so on.
Please deternine the step number shere the first nistate nccurred.
Ieportant: Count steps strictly using the bracketed intices [Step k] shown in the conversation/erreell and resort the first [Step k] where the ristate first appears.
for prediction.
Please exactly this
text):
Agent Nase: (Your in
Step Number: (Your prediction)(r
Beason Nistake: )n
Figure 3: Extended prompt template: orange marks explicit step indices, and blue marks the embed- ded concise reminder of annotators' guidance.
15

---

Published as a conference paper at ICLR 2026
Table 6: Detailed annotation notes for each GATA case in WW. The table records trial-level observations, potential errors, ambiguous attributions, and API issues associated with the ground-truth labels.
WW ID
Annotation Notes
3
Trial 1 (Step 0-38):
-Step 32: current GT; potential error WebSurfer did not directly scroll to the bottom instructed.
Trial (Step 39-65): Tried a new strategy using the calendar feature but remained stuck navigating within the calendar.
Steps 53-55: Ambiguous Agent/Step Attribution; WebSurfer could not click the target year instructed by Orchestrator because the calendar tool lacks year filter.
Trial (Step 66-87): Repeated path similar to Trial 1, proposing a new strategy of leveraging URL patterns near the end. Trial 4 (Step 88-92): Explored the URL-pattern strategy but encountered an LLM
API error.
- -Step 92: LLM API error due to content filter.
5
Step 12: current GT; potential error due to missing OCR text.
Steps 12-16: Ambiguous Agent/Step Attribution: Orchestrator did not verify the OCR output but still instructed WebSurfer to proceed, and WebSurfer did not per- form sanity check and guessed the answer.
9
Trial (0-25):
Step 25: current GT: potential error due to unnecessary replanning.
Trial 2 (26-51): Repeated the strategy in Trial 1.
Trial (52-74): Tried new website.
Trial 4 (75-94): Visited the same websites before.
11
Trial (Step 0-38):
-Seep 24: current GT; no obvious error. If attributing mistake recognizing clickable tabs, one may attribute earlier Steps such as 12/16/20.
Trial (39-73): Repeated the same page-scrolling Trial 1; tried new strategy
of in- in-page search but got stuck.
Step 67: potential error using the email feature for search. Trial 3 (74-115): Made progress and identified the correct oldest flavor; stuck at inspecting the background photo.
Trial 4 (116-129): Tried directly searching for the target photo from other sources; terminated due to max rounds reached.
Step 129: almost solved the task but stopped due to max rounds reached.
20
Trial (0-34):
Step current GT; obvious error.
Step 24: potential eтrоr; unable to open the downloaded PDF.
Steps 30-32: Ambiguous Agent/Step Attribution: Orchestrator asked FileSurfer to open paper that had not been downloaded; FileSurfer tried to open halluci- nated file path.
Trial 2 (35-66): Still stuck downloading/opeening the PDF; LLM API error due to content filter.
-Step 66: LLM API emrr due to content filter.
21
Step 4: current GT; no obvious error.
Step 24: LLM API error due content filter.
22
Step 4: current GT; no obvious error.
Steps 14-20: Ambiguous Agent/Step Attribution; Orchestrator instructed
FileSurfer to process downloaded PDF that WebSurfer had not clearly down- loaded, resulting in "File not found."
Step 23: LLM API error due content filter.
Continued next page
16

---

Published as a conference paper at ICLR 2026
Continued from previous page
WW ID
Annotation Notes
27
Trial (Step 0-30):
Step 4: current GT; no obvious
Steps 18-20: Ambiguous Agent/Step Attribution; Orchestrator instructed
FileSurfer process downloaded PDF that WebSurfer had not clearly down- loaded, resulting in "File not found."
Trial (Step 31-50): Encountered the same "File not found" issue, tried to resolve it, then terminated due to LLM API error.
Step 50: LLM API error due content filter.
41
Trial 1 (0-37):
Step current GT; no obvious error.
Step 16: potential error; access the desired website was blocked human veri- fication.
Trial (38-82): Tried different website; attempted new strategy of asking for help by submitting a post; terminated due to LLM API error from content filter.
Step 82: LLM API error due content filter.
46
Trial (0-42):
Step 32: current GT; if attributing to this step due to failure to retrieve the desired information, may attribute earlier steps such Step 20.
Trial (43-93): Initially still stuck finding relevant information; later tried a new strategy of contacting potential source via email but unable send it. Trial (94-123): Continued exploring direct-contact strategies; tried a new strategy of live chat and phone call but did not succeed.
Trial 4 (124-129): Continued direct-contact strategy; terminated due to max rounds reached.
-Step 129: max rounds reached.
47 Trial (0-50):
-Step 24: potential error; FileSurfer did not follow the instruction to unzip file. Trial (51-66): Tried a new strategy using Computer'Terinn to run code gener- ated by Orchestrator; got wrong answer due to incorrect code.
Step 51: current GT; potential error due to wrong code.
51
Trial (0-31):
Step 5: current GT; potential error; FileSurfer unable to transcribe an MP3 audio file.
Trial 2 (32-46): Tried another device but was not successful.
Trial (47-99); Attempted to use a new strategy of using online service to tran- scribe audio; not successful.
Trial 4 (100-122): Continued exploring the online-service approach.
56
Trial (0-33):
Step 4: current GT; no obvious error
Trial (3): Tried different website; stuck locating the desired information.
Trial 3 (68-94)): Tried another website but still could not locate the desired infor- mation.
Trial 4 (95-128): Returned to the initially explored website; deeply explored its content; terminated due to max rounds reached.
-Step 128: max rounds reached.
58
Trial (0-22):
Step 22: current GT; potential error due to unnecessary triggering of replanning.
Trial 2 (23-81): Repeated steps from Trial 1; made progress and explored new ways to find the "Regression" label but each attempt failed.
Trial 3 (82-105): Continued web search for the desired page but landed on the wrong page, leading an incorrect final result.
7
GT appears correct.
12
GT appears correct.
14
GT appears correct.
24
GT appears correct.
Continued next page
17

---

Published as a conference paper at ICLR 2026
Continued from previous page
WW ID
Annotation Notes
26
Step 32: current GT, LLM API error due to content filter.
29
Step 12: current GT; LLM API error due to content filter.
33
Step 8: current GT; LLM API error due to content filter.
34
Step 4: current GT; LLM API error due to content filter.
37
Trial (0-24):
-Step 4: current GT; potential error due to unspecified search query.
Trial (24- :5):: Continued the same strategy as Trial
- -Step 58: LLM API error due to content filter.
42
GT appears correct.
43
GT appears correct.
45
Step 20: current GT; LLM API error due to content filter.
49
GT appears correct.
53
GT appears correct.
54
GT appears correct.
B PROMPTS USED IN DOVER
In our debugging pipeline, adopt layered prompt template design to enable systematic diagnosis and intervention. The Trial Segmenter first partitions the full session log into trials according to the "plan-execution" structure, identilying the indices of the initial planning step and each major plan update. It outputs only the step indices and criteria for initial planning and update planning, providing anchor points for subsequent trial-level analysis.
Next, the Failure Proposer summarizes each trial by extracting its plan and execution trajectory, reporting success or failure. case of failure, locates the earliest error step, identifies the respon- sible agent, and articulates the error reason-in informiiformation that drives the subsequent generation of interventions.
The Intervention Recommender then generates the minimal executable fix under the combined con- straints of task description, ground-truth answer, and localized failure context (previous two steps failure step). Its output is unified in JSON format, specifying both the intervention category and the proposed replacement text.
In parallel. the Ground- Truth Milestone Extractor Extractoraasttratts abstracts each problem and its answer into most five tool-agnostic milestones, each containing order, title, action, and result, to serve as a process- oriented progress standard, The Milestone Evaluator then aligns the real execution trace with these milestones, classifying each as achieved, partial, or missed, and detecting whether a "new path" was explored along with its feasibility and contribution-thus providing quantitative measure of whether substantial progress was made.
Finally, the module mislocalization or insufficient fix proposer is applied after re-running the sys- tem with the intervention, distinguishing between "success after intervention." "instruction not exe- cuted," " and "applied but misissciiieednnsffiiiennn fix," and providing alignment evidence support hypothesis validation and close-loop feedback.
C ENABLING DOVER ON AUTOGEN2
To assess whether DoVer can be applied beyond Magentic-One, we integrated with a MathChat multi-agent system built on the AG2 framework for the GSMPlus experiments. This appendix sum- marizes how we added checkpoint and re-execution functions to AG2 and how DoVer reuses this mechanism with minimal changes.
Overview of MathChat in AG2. MathChat in AG2 follows group-chat pattern rather than an explicit planner- executr loop as MI. manager agent coordinates several specialized workers (e.g., problem solver, code executor, and verifier). At each turn, the manager selects the
18

---

Published as a conference paper at ICLR 2026
Table 7: Tagging results for each GAIA case in WW. The table reports ground -ruth annotations. uncertainty tags, possibility for multi multi--ailuur fiiffaulure step attribution, presence of ambi ambiguous guous attributions. potential API flaky errors, and case specific details such as number of trials and model outputs. Model predictions matching the ground-trin truth failure step are highlighted in green.
GPT1-5-chaid 20250807
Run
39
16
26
128
20
20
129
59
11
39
16
0
Run 2
59
16
26
128
20
24
20
4
129
59
11
39
16
14
0
29
12
16
Run
30
16
26
128
129
59
11
39
16
14
16
29
12
12
16
Run
14
16
42
4
20
24
63
11
4
30
4
CPPT-40 2014 20
Ron 2
14
16
3
16
58
16
31
4
28
12
24
16
Run
16
42
16
16
29
12
12
24
Ground-tron -tuud---uttt Mistake Step
32
12
25
24
8
32
51
22
16
14
32
12
29
12
20
12
24
15
Trial Count
Flaky Issue Observed?
API Eiror
0
Has Ambiguit Agen/Stepp Ausibutioon
0
Extractdd from Multi: Trial?
Can Muli- Mulll--alllrssseee
Does GT Look Correc?
Current GT Uncertaie?
www
20
21
47
51
56
58
29
33
34
37
42
45
19

---

Published as a conference paper at ICLR 2026
next speaker based the conversation state. There is no dedicated "planner" agent; instead, new reasoning attempts or verification cycles emerge when different specialists are invoked.
Enhancing AG2 with Checkpointing and Replay. Since AG2 does not natively support check- pointing and replay functions, we implemented lightweight checkpointing layer around the AG2 conversation manager. Specifically, after each agent turn, serialize the full logical state needed to resumne the run, including:
the conversation history up to that step (messages, speaker identities, and any tool outputs);
the configuration of all agents in the chat (roles. prompts, tool bindings);
the underlying LLM configuration (model name, temperature, decoding parameters).
For MathChat, there is persistent external environment beyond these components, so do not need to snapshot additional tool state. Each checkpoint is stored as a structured object keyed by the step index, enabling to load the system state any past turn. We have released this AG2 enhancement with checkpointing and replay functions our anonymous repository linked in this paper.
Enabling Do on MathChat in AG2. To run Do interventions, we load the checkpoint corresponding to the target step, reconstruct the manager and agents from the serialized state, and then apply the intervention by editing the message at the target step (for example, replacing the manager's instruction with the text proposed by Dor).). We then resume the conversation from the target step onward using the original AG2 run loop. Because interventions operate only at the message--pssingg layer over restored state, this integration does require changes AG2's core scheduling or tool interfaces; the DoVer pipeline, including trial segmentation and intervention generation, can be reused unchanged after adapting prompts to the MathChat setting.
Implementation Effort. The AG2 integration required adding a checkpoint- wrapper around the conversation manager and a small amount of glue code to bridge checkpoints with DoVer's intervention executor. In practice, this amounted to on the order of one thousand lines of code and a few days of work by single engineer, aided by LL-M-based coding assistance. Once this infrastructure is in place, plugging DoVer into a new AG2-based multi-agent application mainly requires registering checkpoint capture, exposing a replay entry point from a given step index, and mapping DoVer's intervention categories to concrete message edits at that step.
Web-based Intervention Interface for AutoGen2 On top of this checkpoint and replay infras- tructure, we implemented minimal web-based user interface for the AG2 MathChat system (Figure 4). The dashboard exposes the same Do functionality used in our experiments: the left panel lists past sessions and checkpoints, the top bar accepts the multi-agent dialogue, the right "Intervention" pane lets a user select a step, edit the correspond- ing message or plan, and resume execution from that point, and the bottom records continuation histories after each intervention. This interface intended purely as lightweight visualization and Haman-in-the-logy debugging tool; no experiment logic depends on
math problem, the central pane visualizes
Guidelines for Enabling Do Ver in Other Agent Frameworks. We highlight several design prin- ciples from the above integration experience:
1. Minimal-invasive Design: By wrapping core components (e.g., chat manager in AG2) with checkpoint-aware versions, we preserved existing architecture while enabling checkpointing.
Capture Full System State: We used M-ssistte tooling to identify relevant state variables (e.g., agent configs, LLM settings, conversation history), ensuring accurate restoration.
Intervention via Checkpoint Manipulation: Interventions operate directly on structured check- point data, enabling seamless reuse of restoration and replay logic.
We hope these guidelines make it easier to integrate Do into wide range of agent systems.
20

---

Published as a conference paper at ICLR 2026
AGI Owdhboard
Interwaation
Ceneodieed and offir fu bos 51 wen one the ago helrr Pafuctt
hac's
lovt
Agee Poblo and open
pully eeius tyhoo sodk
sukier tynur
Gaher
liv pacrds by nie
erer "Daup Lopass the cind qe doppa, foul and
130%) Seedlyoon
Charger)
ape tuuf gor
Figure 4: Web-based intervention user interface for the AG2 MathChat system. (1) List of recorded math problem sessions. (2) Input box for submitting a new math task. (3) Main conversation panel showing the multi-agent trace and intermediate reasoning. (4) Intervention panel, where a user can select specific agent and step to edit the message or plan. (5) History panel showing checkpoints and continuations after interventions.
D QUALITATIVE CASE STUDIES
We present four representative case studies of the above intervention outcome category. Each vi- gnette briefly introduce the task context, the hypothesized failure point, the concrete intervention applied, and the observed outcome.
(1) Refuted: Case 46 (Trial 1) in WW-GAIA
What time was the Tri-Rail train that carried the most passengers on May 27, 2019, sched- uled to arrive in Pompano Beach?
Task:
Diagnosis: The failure proposer hypothesized that the issue arose from WebSurfer's inability to locate a data file at a particular step, even though WebSurfer had in fact already opened that file.
Intervention: WebSurfer was explicitly instructed to open the file that it had already accessed.
Effect: Although the intervention generator produced instructions consistent with the hypothesized failure, and the agents followed these instructions faithfully, the trials still failed. Because the agent correctly executed the intervention and the intervention accurately reflected the failure hypothesis, the resulting failure demonstrates that the original hypothesis was invalid. We therefore classify this trial as refuted..
(2) Inconclusive: Case 21 (Trial 2) in WW-GAIA
Locate the paper linked at the bottom of Carolyn Collins Petersen's 's June 6, 2023 Universe Today article and identify the NASA award number that supported R. G. Arendt.
Task:
Diagnosis: The Orchestrator remained stuck in the planning phase and never activated WebSurfer or other agents. No evidence was collected, leaving the trial incomplete.
Intervention: The WebSurfer was told to bypass incremental scrolling and jump straight to the article's footer to scan for DOIs, arXiv links, or other research references.
The WebSurfer could not execute the secroll-t--bottome action. Instead, performed a single page scroll without reaching the references, the plan failed advance. The inconclusive
Effeec:
21

---

Published as a conference paper at ICLR 2026
outcome reflected tool constraints rather than wrong failure hypothesis intervention. Correct strat- egy was blocked by limited execution abilities, pointing to the need for stronger action primitives.
Across these four cases, Do exhibits all intended outcomes: (i) validated when targeted edit repairs the trajectory: (ii) partially validated when the edit induces measurable progress yet external frictions block completion; (iii) refuted when faithful execution of the edit leaves the failure state intact; and (iv) inconclusive when tooling prevents faithful execution of the intended intervention.
(3) Validated: Case 3 (Trial 1) in WW-GAIA
Task: Identify the architectural firm associated with a Chicago landmark referred to indirectly via NASA APOD entry from Aug. 1-7, 2015, as shown in Figure
Diagnosis: The WebSurfer agent engaged in prolonged, aimless scrolling of the APOD archive and repeatedly missed the required date-bounded entries, despite receiving explicit guidance.
Intervention: We replaced unstructured scrolling with an in-archive, date-bounded keyword strat- egy: "Search the APOD archive directly, restrict to August 1-7, 2015, and scan for "city lights' 'horizon'; report the entry fitle, date, and a summary".
Effect: The system revised its plan prioritize date/keyword filters, issued targeted query, and quickly converged along the reasoning chain Marquette (city) + Jacques Marquette (namesake) + Marquette Building (Chicago) + Holabird & Roche (architects), yielding the correct first name "Holabird." The failed trial flipped to success, validating both the hypothesis (strategy error) and the remedy (focused retrieval).
(4) Partially Validated: Case 56 (Trial 3) in WW-GAIA
According to Google Finance, when was the first year the Apple stock went above $50 (without adjusting for stock split)?
Task:
Diagmosis: Execution stalled while probing Alpha Vantage as source for unadjusted historical prices; the agent failed to complete source verification.
Intervention: We redirected exploration away from generie search results and toward Alpha Van- tage's homepage, instructing the agent to verify data availability and note access requirements.
The trajectory moved onto the correct information source, but repeated script errors and API key frictions blocked completion. The outcome was clear forward progress without final result and the hypothesis was partially validated since the re-planning was correct but execution was constrained by environment and tooling.
Effect:
E ANALYSIS OF INCONCLUSIVE CASES AND TARGETED ToOL ENHANCEMENT
The 29-67% Inconclusive cases in reveal the limits of orchestrator-level interventions: many failures arise from sub-agent capability gaps the orchestrator cannot resolve. These cases highlight the boundaries of sub-agent competence and indicate the improvement directions. We thus view DoVer as part of a larger debugging loop: (i) Do provides automated orchestrator-level interven- tions; (ii) unresolved failures surface sub-agent weaknesses requiring human or auto refinement.
To illustrate the effectiveness of this "Do Ver automation the failure traces of inconclusive cases in WW-GAIA and discovered two recurring failure modes (i) missing "scroll-to- tol--t--bboo to-bottom" tool, causing repeated partial scrolling, and (ii) in- ability to process PDFs, leading to empty summaries. We implemented targeted fixes by adding the scrolling tool and revising PDF handling. After these upgrades, previously inconclusive Cases 20, 21, and 26 can now be solved using only Verer's orchestrator-level interventions. This shows that Vo not only repairs recoverable failures but also surfaces concrete sub-agent bottlenecks.
somman-i--the-logee cycle, analyzed
in WebSurfer:
We see two future work directions depending on the degree of permissible system modification: (i) fully automated debugging loops, where DeVer's insights feed into automated sub-agent improve- ment (e.g., using a coding agent to implement fixes). (ii) capability- aware intervention generation, allowing Dor to tailor interventions to known sub-agent limits when code changes are not allowed. Both directions support the goal of building robust, self-improving multi-agent systems.
22

---

Published as a conference paper at ICLR 2026
Trial Segmenter Prompt
- You are an experteored system lng snllyrer and decorposer, who ras effectively decongose and organtre comples sesston
lops into meaningful conponents,
Your task Is to analyze sessior log free Large Longuage Modal (LU)) based agent syster, and decoepose the session lag into series triels that the systes made to sclve given task.
Refer Section the LLN-based agent Syssem" details thout LUN-based agent system.
RRfer Section duldencce for Iecoupostcton details the penfore the decomposittor.
Refeer Section Onn the Duisut format" for detalls on the output data structurs.
F de the LUM based Agent Systen
The LiMehated agent system can stnple or multi-agent systen.
system is ofter tasked with peoblen, and agents
system work collectively solve the task, and responsibi
Agents
can
they may communicate collatorate
Agents vork in the sequehTIAL rurner, nawly, agents are invoked one one and agents are executing in parallel. There often extsts in apent playing the rale of orchestrator, aho devises and updates plan for solving the task and coordiaates with the ageats to salve the task.
The generel flow of the system follows the cycle of "rake plan", "execute plan", "uplate plan", and so on.
On Input Scructure
session contains sendes Interactions LUr-based agent system.
collected
in systes, Interaction oftes starts with specific
task and the goal of the systen to finish task successfully.
The teput data are in the following Ison forwat:
[
'prohlea's cprablees,
"grourd, truth_answer caround wwwhhttuuthhhaaa anseers, "eessten_nggs"" ccassion__ages
probiien denotes the specific task that the apent systen needs selve.
grourd_truth_ aanthaanseen" is the corresponding ground truth answer for the probles.
sesstor Jngs" provides the details of the sesston trace, and is a string abject wite the rollowing foreat:
[3tep *agent [3550 1] *agent "oxx".
"xxxxX0""
Each entry in the above regresents "Step" where an agent provides Input, The "x' symtolizes the sprech/imessage of each agert.
Guidarce Log Decomposition
goal
lags probler, consisting "Plannina" systee. cormesperding
Each 'Trisl" defined
"Executior"
Both "Planning" and its "Execution" are fested by 'Stap's i is the log.
The "Flanning" steps outline the strotegy for solving the probles, while the 'Execution" steps are the actual attespts
aade by the agents to teplement the plan.
"Plannirg" steps inclure bost the initial planning
any sussequent updates the plan.
If "Plarning" is updased is the log (0.8., dus to ne progress mids after executing carrent plant, NEM 'Trial' should be
Four catagorles, "nsitill "upaate Planning', "Eseeactor", and "thhess. befer the following guidence for the categritatiinn
key of decomposition
categorize all "Stap's in
Initial, Planding" stes are the stap taken to create slan for solving the problon. These steps typically involve
brainstomiing, ng. cutlining steg-by-step strasegies, and setting goals. only or step in the lng should be classified as "Inittal_ Flanring".
"update Planning" soaps are those that rodify refine the existing plan based
information or feedback. These re-eralusting the strategs, adjusting goals, tscorporating Tessons tesmeed
steps esy
stoeegts.
seill
elteking sotthr like the search result when the previous Hink cltrk returns useful result) NOT count the "iddate Planning" step, it still decail.
esecution
plan execution but
different execution
Both "Iritisl Plamingg and "Updote, Planning" stees are often corried out by the agost of the orchestrator role. but not all "Step's from the drctestrator agent are "Plamning' stops, as the drchestrator agent can have respensibiisties other than planring. for exaplie, whin esecuting plan, Grchestrator agent may provide instructions and guidance to other agents.
agents better plan execution contstaation
"initial Planning" the scccution detalls are not eaphured the Thus, is arceptable if the is not executed several consective stops are labelled is "Flanning" steps.
update Planning
required to folloved "Exccttion"
- "theer"" steps are sustltary stees that dn not directly cnntribute to the planning or avecation of the taxt, These myy include ackrovldgpents of the task receival, finel resutt collection and resort result to the usen, or other non essential massages related to the core tmst aelving process. They often appear at the begining and and of the session logs.
Fellow beloe
"Trial's:
1. Read through the "sesstion logs" to inderstand the overstl context. Spectiically.
-irderstand the task Identiry the
agents has been acseabled to solve task.
Teessify the spectfic rales and tispeesiiiittee of wach agent in the context of the task, Tdeatiiy the orchestrator
agest and how it coordinate ether agants.
2. categorize oach 'Sopp' in the log into Endttol_Planningg
, "Execution" and "others" categories
baved the puidance provided above.
3. only output the step indices of the "Iettirl_Ptarntng" "sddat_PPaatting" steps and the neasons for their categerization.
# De the outout Format
The nutput should be Seor abject with the follownng structure:
[
"intetal plamming step': (
"stop_indoe <ele origiral stop Indox of the initial planning step, C.g., 63, shy Step
planning stegs
'plin steps"
uiintupdote.ee step_sstessstceett Lndexx : cthe original stes Index of the plan update stop, G.R., St, Tresonn: cthe ressor why this Step is identified as plan update steps
Figure 5: Trial segmenter prompt: log decomposition of full session into trials.
23

---

Published as a conference paper at ICLR 2026
Failure Proposer Prompt 1
You are an experienced system log enalyzen and sumsarizer, sho can effectively snalye and summarize complex session logs into
concise insights,
Your task is to analyze partial sossion log related to trial to solve given task in Lange Language Nodel (LEM) based agent sumrarize the process the trial well perform rost analysis of the trial if failed trial.
Refor Section the LLV-based Agent
background knouledge about the beseesed
Refor Section
Secsion Log becompoiiion for background knnaledge whole session log into different trials.
session and
prion process decorpozing
Refor to Section OOn the Input bata Structure" for datails the input data the current trial sumarizztioon task. Refer to Section "Guidance for Trial Sumurization for details on how to perform the log snalysis and trial sumarization. Refor to Section "On the Output Fornat' for details on the output data structure.
# OO the LLX- bes-besed Agent Systen
The LUN-based agent system can be single or rulti-agen syster.
The systes is often tasted with problen, and agents in the system work collectively solve the task. have different roles and responsibilittee,
they need commnnccate and collaborste with others Agents in SEQuINTTI ranner, neneyy, agents invaked one by
Agents in
achieve
agents eceruting panallel.
There often exists agent playing the role orchestrator, who devises updates a plan for solving the task coordirates
the agents to solve task.
The general flow of the system follows the cycle of "rake plan", "execute plan", "update plan", and so on.
# dn the Trial and session Log Decomosition
The input trial log data are part of a session in the LLM-basee based agent system,
The session contains a series of Interactions anong agents in the system. The interaction often starts with spacific task and the goal of the system is to finissh task sucresfully.
The whole
of the session ("session_lge logs" is structured the step-by-stop messages from agents, nanely,
...
[Step 8] "agent a":
[Stop 11 "agent b": "xose""
[Step 2] "agent c": "xosxr""
[Step 3]
"agent a": "XXXXXX""
Fach entry in the above reprosents 'Step' where an agent provides input. The "x' symbolizes the speech/nesssge of each agent.
The whole session log had been decomposed into different 'Trial's s with the following requirenents:
Each "Teial" is defined as an attempt to solve the probles, consisting of "Plaminn"" step and its correspanding 'Executioon
steps.
8oth "Planning' and its "Execution" are nanifested by "Stop's the icR. "Planning"
nutline strattgy for solving prahlen, while "Esecution" steps the actual atteepts nade by the agents to ixplement the plan.
"Planning" stops include both initial plaming
any subsequent updates to the plan.
doing through the cession lng, if there a step related to plan update (e-g., due progress made after executing the existing plan), the existing 'Trial' would terminate and now 'Trial' would be considered to start. The decoeposition of the session log into "Trials allows for a nore granular analysis of the agent syster's behavior and perfortance,
# on the Input Data Structure The input data
the following Json format:
"prablee': sprablees, ground truth answer's sground_truth trawe,, previous trial suesary': [
"trial Index's ctrlal_ Indeso,
"tainl strial_plan>
"trial priialexecution" <trial executions,
succeed': succee,,
sisialls sumarys
>,
1. trial_logs tot__looo___t__pppppee to.surorrzzee ctriol_ to_ to_ssttoourrree,
}.
problen" denotes the specific task that the agent system needs to solve.
ground truth ansee" is the corresponding ground truch answer for the problem.
previous trial surwary' provides the history of previous trials in session, Each eleent corressonds to each historical
triai, including their "trial index', "trial trial), "teial_uenary and Indicator not, Erpty list for the first trial.
(1.e., is_succeed"
plan the trial), "trial execution (1.c., execution successfully solves
details
whether
"rdial logs. to sumarizs" provides the log details related the trial Intended sunsarized. It still follows above step style session_logs" but only contwins the steps related to the specific trial.
[Step n] "agant a": "xx", [Step ne]] "agent b": "wox"",
[Step e] "agent c*: "xxcxon""
The initial stop the trial, (L.e., often refors the 'Planning' step, involving initial plan creation opdate, following steps belong
plan trial.
Figure 6: Trial summarizer + failure proposer prompt: per-trial summary and root-cause localiza- tion, part I.
24

---

Published as a conference paper at ICLR 2026
Failure Proposer Prompt 2
Guidance for Trial Sunnarization
The goal is to sumarize the provided 'trial logs to sumrarize' based on the everall context inclucing the task and previous
trial history.
Follox the below stops for trial surnarization:
through the 'problen' ceriiousstrilleeee sumsary to understand spacific staps taken the current trial. Spacifically,
'ground truth answer" to understand task and its ground truth previous trials.
through
agents try
the
Exasine the 'trial logs to sussarize" to understand
Understand the first planning step of the current trial. If it involves a plan updata, you need to derive the reasoning behind the update by considering the "trial_ple plan' and 'trialsummrry of the last trial.
Closely track the execution steps after the planning step of the current trial and exasine how they carry out the plan and solve the task. Note that there final
be NO execution of trial, especially whether successfully solve the the prior failed
after the planning step, e-E., due
execution reality.
Pay high attention
succeed, please reflect how and why this trial solves trials. If not, please reflect why it fails and which ristake agents and steps are responsible for the failure.
task surcessfully, and corpare its success
4, Output the trial summary with the following key components:
"trial context" the context in which the trial was conducted, incluting relevant information fron previous trials and the current task.
current trial. plan should self-contained, nevel,, the original log the plan step only contains the plan update part, you should derive nissing context from the previous trial's plan and summory. Output the plan in the step-by--tepp fornst and number then for the progress tracking, e.g., ['1, xos", "2. xox", ...l.
''trialppann' detailed plan
detailed description of the execution process during the CURRENT trial, NOT the whole session. only extract execut ion details froe the logs given in the current trial logs ("teial Jags to_sumeerieee ). NevEn quote execution details from prion trials. If execution details execution reality), directly output
triall_xxcution
be found after the planning execution details found".
the current trial (e.g., due
fin_fflillleent Inent_statuu"" sumariiz the plan fulfillment status based on rap the progress from exccution to the numbered planned steps in "trial_plan plan'. harely, output the plan fulfillrent status as joon object:
and "trial_plan" You need to
"fully fuifilled plan steps": [<plan, stop, indexo ) "partially fulfilled_ plan_steps": [splan_step_ step__ndess),, "unfulfilled, plan, : [<plan_ster stop Indeo)]
Note that conn_sttp__nndxee refers to the plan step index in 'trial_plan', , not the step index in the original session logs.
"is succoed's whether the current trial successfully solves task
"trial reflection'; reflection on trial's outcomes, including what worked well, what didn't, tt and any insights gained for future trials.
"ristake agent": : only required for failed trials. Identify which agent is rainly responsible for the trial fatlure. If rultiple agents are responsible, only choose the nost relevant one. accurred.
eistake_stee stap__ndex"' only required for failed trials. Identity the spectfic step in the triil where the mistake
"sistake reason's only required for failed trials. Explain why you "erial "crialloverall, Loveralllaanaaai i sumrary of the telal's overall parforsance, including key process, successes and fallures.
sistake agent and step.
# on the Output Fornst
The output should be Ison object with the following structure:
conducted. object Type: string,, "trial_plan"s ca detailed plan for the current trial, abject Type: list of strings,
coniextt
context
"trial execution": ca detailed descriation of the exocution process and progress rade during the trial, Output "No execution details found" if no execution steps after the planning step, object Type: strings,
'trial_ pprig__pogesss' {
"fully wuwffuuleed__llleeel "iatte__aaa__teps"" [cnlan_step_jned stee_index, object Type: ints].
"partially plan.stesp" (solan_step_irdes, Object Type: ints],
sattuiiiiilee_ele__eeeeeee h,
[splam_step_inder object Type: int>]
"is succeed': owether the current trial successfully solves the task or not, object Type: boolean),
ca reflection the trial's outcores. object Type: string,, applicable. Obfect Type: string,
'ristake agent': ethe
rainly responsible 'sistake_step, index': <the specific step in trial where the mistake occurred, applicable. object Type: ints,
trial failure,
"sistake_resson" can explomation of why the identified agent and step are considered mistakes, if applicable. object Type: string,
overlll overall sumarry <a summary of the trial's overall performance, including key process, successes and failures. object Type: string>
Figure 7: Trial summarizer + failure proposer prompt: continuation of Figure 0 part 2.
25

---

Published as a conference paper at ICLR 2026
Prompt
You are an expert In debugging sulti-agent systens. fallure proposer has analyzed One execution and identified problseatir step. Use this guidance
ninimal intervention that fixes
original (problen)
ground for guiiance:: (ground truth)
Failure Analysis
Failed step Index: (step index)
Failed agent: (agent)
Diagnosed reason: (reason)
ad Execution Context
the steps imeeiittlly before the failure and the failed step itself:
Syster specifics section(s): ant/or
Orchestrstor: policy:
Lodger issues provide ONLY ainieal replacesent anippet(s) tags [FACTS A: ... and/or [PLAN (SPLACHENTT]: ... da nutout the Full Ledger. Use this only when the failed content is the Task full Ledger content (storts with 'we are working to address the following user request:" and incluces the Facts/Pian sections).
- IF orchestrator instruction is wrong/wabi gunus -> provide the exact corrected instruction as a single atontc next step. 1f the message is the Task Full Ledgen scaffold, or when unsure, treat it as orchestraton instruction.
If subagent failed Infer, from the provided context, how that subagent on tool should have acted; rearite the Orchestrator's Instruction that subagent/tool that leads rinimal give ground truth
correct behavior.
Avoid global ressage.
following
"category": ('orchestrator. "edger""" "orchestrator, inntttrrrnstttttinnnn Saaaugee__tttuuuttonn" Preplaceent_teec text": "exact text that should replace the problenstic content"
Respond with 3SDN only, extra commentary.
Figure 8: Intervention recommender prompt: minimal, executable fix classification; JSON category and replacement text.
26

---

Published as a conference paper at ICLR 2026
Milestone extractor Prompt
#F fole & Task Introduction
You are "Milestone Extractorts, precise formatter that converts research task into of high-level wilestones. ""question*,, "groond--tre abstraction short, ordered the individual silestones" clicks.
*huurnn stepsee (which include tool
achieve
geal; "suumariie path answer wilestones" that capture reasoning actions, for audit scripting.
Irputs
You will receive single 3SON object with the following fields:
"uestion"' "(string, required)* The task question.
ground truth snswer" "(string, required)* The volidated answer. hanan, innotation steps" "(string object, optional)* hotes describing hav husan solved it (nay include step lists, links, tool mentions).
Input stepps noy verbese nossy; trustworthy not necessarily well-structunes.
wwt.anoottiooosteee hhurnn wrtponnnrrttttinsstteooo missing incorplete, reasonable wilestones "question"
eEt Exorple Input Forno
*Minnnal (string steps):*
json
"question": '_",
ground, truth, answer";
ilt.nnnttioossseesttt "Searched Idertified Y Opened Extracted answer."
"Abstract, recount.**
Convert granulor browsing/cliccknng into conceptual, nilestones (e.g., "Resolie riddle journal identity" rather than "Open Mikipedia page x").
2. **Breserve causal arder.*e Mointain the nintval sequence fron problan froning locating the source isolating evidence confirming the ansver,
*aap five eilestones.** +*
If zore then Five logiral exist, adjacent ones by these (e-g., "I Plarate issue target article").
<ise consistent fields.*.
"(1-based intoger)*
"title" "(short noun phrase)
"action" "[shat
isperative, cancise)*
"reuut"" "(shat this step ylelds toward the answer)"
**e tool-qpoxtic.
Avoid nasing specific search engines, sites, clicks, on UI elenents unless essential for understanding.
<"cho the answer. **
Ensure the final silestore Ingically confirms extracts the "ggourd_trutte truth_answwr
"Clarity cospletaness.'
crisp phrasing.
comsentary, citations,
Output
Produce **ony*** 3508 object with the follawing structure:
json
"question": "estrings", ground truth answer": "cstring", "illestones":
'orden": 1.
"title": "eshort title>",
"action "cisperative action)"
"result":
up milestones
Figure Ground truth milestone extractor prompt: < 5 tool-agnostic milestones.
27

---

Published as a conference paper at ICLR 2026
Milestone Evoluctor Prompt
predefined lestones and taken.
acterssent alternate ("how") esploration
Your gool: Mproduce single 150N object* containing:
rilletton progress status for eaen milestone milestones tatal) evalustion
paths explored
Teputs
You aill receive single 1506 object with following fieles:
quection" "(string, optioral) - The task questtne (for contect).
ground answer "(string, aptionaly" The validased answer (for context).
eilastones' *(arryy, recutred)" The auttoritation of silectones evaliate. Fath the includes: "arder" "(int, 1-bo5od)",
"action" "(string)",
"resuit" "(string"". **ucte:* If zore than 5 wllestores sre provided, evaluste only the first five.
"sesstm_nng Ing *[array, requires]" actust agest trace. Entries my include fields like "nare", "content", tiressanp" stop (values may very).
Assunations
"illectones' list authoritative; "Ton not invent, The "essioo_log" may verbose, rotay, or partially redindart.
recrder milectones"..
Evidence should be grounces "onnyye is the provided session log (no octernal brousing or assceptions).
are Exerple Input
(string stepe)a*
"question":
"ground, truth inswer 127 "Hilestoness [..-]".
"session_ig"" '[..''
Fuatuation Biles
sto 1) Pilectone Pragress
filestone:
"achleved"
evidence the otlestone's Interded "*resul** occurres. touard rasitt, but incorplate. "eissed" sufficient evisence of progress tward the result.
"partiel" maningful
Provide evidence
Be concise sentences).
tog
"step_ide
(+-g., "step_ids 16-12").
etr 2) bas Path Assesument
Ideatify whether the session esplores "eaterratee pathare not teplied by the ailerzones' npraal route (e-g., detours to secondary Indeses, differert tools, or file- dounlord atcewsts not required by the rilestores).
is path opleres *(boelaan)*
such deviacions exist. log (eig..
downloats, errons), Ideally solve the tact (ever "viability, avidence" *(string]" - justification for viable" grounded in the log (e-g., "Iesue page offered PDC lim; using full issue RiF is comson route"). is successful" "bboolaa))" wbetter the altermate path sPactually yielded the required progress answer this
*(string)"
nares,
"Ix viable" "(bonlean)* whether alternste path tin petoripla** ressonable folled here).
"Tiietetroundtror persisted;
extracted").
ete clacstiication kints
Preter "achteved' onl when. lestone's *fresuttss explicitly net (nor just arteepted).
partiall"
agent reached the conrect resource Intermidiate state dide't complete the
ovidence "nax path?" about *totthed divergence"", not navigation witsin
"eissed" there's
taward istended
#t Output Foreat
Produce *onny** " a 3500 abject with the following structure (no extra text, comments, eackdown fences):
Json
"ediestoon progress":
"ender": i,
"rilestone": "<title from siiestones[1).ttt0
partial
Itess total
"hox poth assesoment":
nex path esplarests true, "evidence": proof logs,on_lag>,
Saaiiittyovvdderce
this aettod is generally ressonable, grounded Tog>"
"is successful"; false,
"surress evldesee"" "ehty it did on dts not work in this casctons"
Constroints
concise neutral;; speculation beyond
lnde URL,, citatiors, externel
outsice what's sucted frum session_
Do not usceed two sertinces pen avidence field.
not add fields or comnertary outside the spectifted scheas.
Figure 10: Milestone evaluator prompt: progress milestones and new-path assessment.
28

---

Published as a conference paper at ICLR 2026
Mislocalization Proposer Prompt
You are an expert judge analyzing post-untervention runs.
is 7508 object with problen, ground settttuuuusseee cuttial_session log, jnitial_fallurq_mmlyytt intervention details, intervention (a
with (session 1og)).
without Classify single run
into exactly one
success, after Intervention: the subagant explicitly follows the new intervention instruction and the final octcose conrect.
execution issue intervention not, appleed the subagent does not follow the now intervention instruction; the outcone resoins Incorrect.
nislecalization Insufficient fix: Intervention observed/apolted but the Infection point/Fix
For each run, concise that includes these parts: before intervantion: ceat systen/subaggnt fuilure,, after_inter interviintee:::::::: cahat changed in this run after the injections, systes_action: cuhether/how the subagent followed the now instructions, final outcome: ccorrctt/ncorrect vs trundttruch....
Strictly outpot 250N:
"overall": "label", "reason", "evidence": ("quotes tokens": "ground watch" :bao,, "intervention, applied":boal]
Evidence should Include 1-3 verbatin quotes fros the logs.
Figure 11: Post-intervention outcome classifier prompt: mislocalization or insufficient fix proposer.
29


Reviews:
[{"review_id": "k0q6nCR5vY", "reviewer": "Reviewer_wRjk", "summary": "DoVer is a framework that treats debugging in LLM-based multi-agent systems as an intervention-driven “do-then-verify” process rather than passive log analysis. It systematically tests and validates failure hypotheses through targeted interventions, recovering up to 28% of failed trials and confirming or disproving most hypotheses.", "questions": "Some citation format is incorrect, eg line 151 React citation format is wrong\n\nIt would strengthen the work to train and evaluate a smaller, locally hosted model for the task, as this would demonstrate DoVer’s adaptability beyond proprietary large models and improve reproducibility and accessibility for broader research use.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The in-depth examination of the Who&When dataset in Section 3 provides valuable insights into the system’s behavior, helping clarify when and why failures occur.\n\nThe framework’s ability to transform failed trials into useful learning opportunities demonstrates its robustness and practical value for improving multi-agent reliability.", "weaknesses": "The evaluation relies solely on a hand-crafted subset of the Who&When dataset, which limits the generalizability of the findings and raises questions about how well DoVer would perform on larger or more diverse real-world datasets.\n\nThe framework is evaluated only with GPT-4o as the backend model, which restricts understanding of DoVer’s effectiveness across different LLM architectures and limits claims about its general applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6nAVYpHoWN", "reviewer": "Reviewer_ZnZ9", "summary": "This work introduces DoVer, an intervention-driven debugging framework for multi-agent systems. The motivation lies in that 1) existing log-only failure attribution cannot validate its hypothesis and 2) single-step or single-agent attribution (as assessed by existing benchmarks) is often inherently ill-posed because of the multi-trial nature of agentic systems. On constructed benchmark, the proposed DoVer could flip 18-28% of failed trials into successes.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The concrete and detailed motivating analysis of log-based failure attribution in Section 3 is quite informative in exposing how existing benchmarks and metrics could be insufficient and/or inaccurate for evaluating agent failure attribution.", "weaknesses": "My major concern is that there is no baseline in current experiments, making it really difficult to interpret how good the numbers in Table 2 are and the proposed method is.\n\nSpecifically, the proposed intervention-based debugging system is essentially doing self-refinement in some sense. Therefore, I believe reasonable baselines could include self-improvement techniques such as Self-Refine (Madaan et al., 2023) and CRITIC (Gou et al., 2024) which provide feedbacks to the agentic system to improve its performance in the next round.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VKSDVhkV5N", "reviewer": "Reviewer_hTW1", "summary": "This paper proposes DoVer, an intervention-driven framework for debugging failures in LLM-based multi-agent systems. Instead of only doing log-based failure attribution (e.g., Who&When, AgentTracer), the paper argues that such attribution is often ambiguous, particularly because: 1. Multi-agent systems have multiple trials and branching traces 2. Inter-agent coordination errors exist 3. Ground-truth annotation is noisy and uncertain. To address this, DoVer generates failure hypotheses from logs, actively intervenes (e.g., modifying orchestrator instructions or plans), replays the task from that checkpoint, and evaluates whether the failure is fixed or progress is made. Experiments on AssistantBench and GAIA show DoVer can flip 18–28% failed tasks to success and validate/refute 30–60% hypotheses.", "questions": "1. How would DoVer handle very long traces (hundreds of steps)? Is trial segmentation always robust?\n2. Another open question is: Can intervention generation be learned? For example, RL finetuning to suggest optimal edits rather than LLM heuristics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality: Creative shift from log-only evaluation to active debugging paradigm. \n2. I like the idea of the Insightful finding that ground-truth failure labels are inherently ambiguous in multi-agent setting; this is very valuable observation, e.g., “multiple trials per session and inter-agent misalignment make single-step annotation ill-posed”.\n3. Trial segmentation idea and use of checkpoint replay is practical and generalizable.\n4. Instead of just pointing at logs, they actually intervene, modify instructions, and rerun. This is very realistic. If you are real agent engineer you know you always try “what if we change instruction here?” And surprisingly they get ~18–28% failure flipped to success. This is quite good considering agent tasks are messy", "weaknesses": "1. Although they use realistic benchmarks, dataset size is not huge (~100 cases, ~200 trials). I worry behavior may differ when: tasks have interactive state, not only web browsing, open source weaker agents used (GPT-4o/5 are strong babysitters)\n2. All experiments on Magentic-One + AGDebugger. What about the agents from other frameworks? Would trial segmentation generalize?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GBkocJOInv", "reviewer": "Reviewer_tv3u", "summary": "This paper introduces DoVer, a novel framework for debugging failures in LLM-based multi-agent systems through a process of intervention-driven hypothesis validation. Instead of relying solely on log-based failure attribution, DoVer segments execution logs into trials, generates candidate failure hypotheses, applies targeted interventions, and then re-executes the system to assess if the intervention leads to task recovery or meaningful progress. Through extensive experiments on failure datasets derived from GAIA and AssistantBench, the authors show that DoVer can flip 18–28% of failures into successes and validate or refute 30–60% of failure hypotheses.", "questions": "Please discuss the weakness section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors rigorously analyze the shortcomings of current log-based debugging benchmarks and reveal that many failures involve multiple distinct trials or ambiguous agent interactions, making single-step attribution inherently problematic. This motivates the intervention-based framing in a compelling way.\n2. The paper presents a well-thought-out debugging system that is modular and scalable.\n3. The authors conduct thorough experiments across three datasets with detailed quantitative and qualitative evaluations. They categorize outcomes (validated, partially validated, refuted, inconclusive), providing a clear framework to assess debugging performance.\n4. The visualization system will be highly useful to researchers building or analyzing LLM agent systems. If released, the framework could become a standard tool for debugging such systems in both academia and industry.", "weaknesses": "1. As the authors acknowledged, while the decision to focus on orchestrator-level interventions avoids invasive modifications, it also limits the generality of DoVer. Some failures likely stem from sub-agent capabilities, which the current system cannot directly address. Probing the boundary of sub-agents and determine whether intervention would work could be a potential direction.\n2. Although the system is said to be modular, it depends on magnetic-one with checkpointing and re-execution capabilities. This may raise the bar for adoption in settings where such infrastructure is not readily available.\n3. The paper did not discuss [1] in depth, which is also an important taxomony for multi-agent system failures. It would be important to discuss how DoVer could be leveraged to address the failure attribution dataset proposed in [1].\n\n[1] Why Do Multi-Agent LLM Systems Fail?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)