AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
Authors: Guibin Zhang
Junhao Wang
Junjie Chen
Wangchunshu Zhou
Kun Wang
Shuicheng YAN
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agent
Agentic System
Failure Attribution
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Model (LLM)-based agentic systems, often comprising multiple models, complex tool invocations, and orchestration protocols, substantially outperform monolithic agents. Yet this very sophistication amplifies their fragility, making them more prone to system failure. Pinpointing the specific agent or step responsible for an error within long execution traces defines the task of \textbf{agentic system failure attribution}. Current state-of-the-art reasoning LLMs, however, remain strikingly inadequate for this challenge, with accuracy generally below $10\\%$.  To address this gap, we propose AgenTracer, the first automated framework for annotating failed multi-agent trajectories via counterfactual replay and programmed fault injection, producing the curated dataset TracerTraj. Leveraging this resource, we develop AgenTracer-8B, a lightweight failure tracer trained with multi-granular reinforcement learning, capable of efficiently diagnosing errors in verbose multi-agent interactions. On {Who\&When} benchmark, AgenTracer-8B outperforms giant proprietary LLMs like Gemini-2.5-Pro and Claude-4-Sonnet by up $18.18\\%$, setting a new standard in LLM agentic failure attribution. More importantly, AgenTracer-8B delivers actionable feedback to off-the-shelf multi-agent systems like MetaGPT and MaAS with $4.8\sim14.2\\%$ performance gains, empowering self-correcting and self-evolving agentic AI.

AgenTracer Tracer
AGENTRACER: WHO Is INDUCING FAILURE IN THE LLM AGENTIC SYSTEMS?
Guibin Zhang
Junhao Wang
Junjie Chen tt Wangchunshu Zhou
Kun Wang
Shuicheng Yan
NUS
CUHK
OPPO
V NTU
Main Contact:
gu.binzzouttookk.cooo
ABSTRACT
Large Language Model (LLM))based agentic systems, often comprising multiple models, complex tool invocations, and orchestration protocols, substantially out- perform monolithic agents. Yet this very sophistication amplifies their fragility, making them more prone to system failure. Pinpointing the specific agent or step responsible for an error within long execution traces defines the task of agentic system failure attribution. Current state-of-the-art reasoning LLMs, however, remain strikingly inadequate for this challenge, with accuracy generally below 10%. To address this gap, we propose AgenTracer, the first automated frame- work for annotating failed multi-agent trajectories via counterfactual replay and programmed fault injection, producing the curated dataset TracerTraj. Lever- aging this resource, we develop AgenTracer-88, lightweight failure tracer trained with multi-granula reinforcement learning, capable of efficiently diag- nosing errors in verbose multi-agent interactions. On Who&Wher benchmark, AgenTracer-8B outperforms giant proprietary LLMs like Gemini-2.5-PHt and Claude-4-Sonnet by up 18. 18%8. setting a new standard LLM agentic failure attribution. More importantly, AgenTracer-8B delivers actionable feedback off-the-shelf multi-agent systems like MetaGPT and MaAS with 4.8 ~ 14.2% per- formance gains, empowering self-correcting and self-evolving agentic AI. Codes are available at https: ggithub.... com/ bingreeky////eeeeoooo..e....yeea ArgenTacer..
Who&When
Who&When
TracerTraj-Code (agent-level)
TracerTraj-Man (step-level)
(handcraft; agent-level)
(handoraft; step-level)
72.11
06.02
55.0
$3.70
4620
$3.34
13:79
50-
Asotheai
Openit G01
Quedk adis-
Clect slmuu
onyy
Awchhook G0%
40%
Oieech
Figure 1: Benchmark performance comparison between AgenTracer-898 and leading industry providers.
INTRODUCTION
Large Language Model (LLM) -oweere agents have exhibited exceptional proficiency across a wide array of cognitive faculties, encompassing perception (Driess et al., 2023; Wang et al., 2024; Zheng al., 2023; Wei al., 2024), planning (Zhu et al., 2024; Huang al., 2024; Erdogan et al., 2025), reasoning (Putta et al., 2024; Masterman al., 2024), and action (Li et al., 2024; Yang al., 2024). As endeavors to address increasingly intricate challenges, such as iterative tool calling, complex document analysis, and multi-step web navigation, continue surge, the inherent limitations of relying on a single monolithic model have become increasingly conspicuous.
Consequently, numerous multi-agent frameworks, motivated by collective intelligence (Wang al., 2025a; Zhang et al., 2024a;b) and the society mind theory (Minsky, 1988; Li et al., 2023), have emerged, ensembling multiple LLM agents achieve refined subtask orchestration (Zhang al.,

---

AgenTracer
2025d; Hu et al., 2025), ultra-long context handling (Zhang al., 2024d), and broader environ- mental perception (Jiang et al., 2024). These systems have demonstrated superior performance over single-agent counterparts across range of complex real-world domains, including data sci- ence engineering (Hong et al., 2024), scientific discovery (Ghareeb et al., 2025; Ghafarollahi & Buehler, 2024), and software engineering (Wei et al., 2025). However, the integration of multiple autonomous agents alongside diverse auxiliary modules (e.g. external databases, tools, and mem- ory modules) inevitably exacerbates system fragility. As evidence, a recent empirical study by UC Berkeley (Cemri et al., 2025) reveals alarmingly high failure rates (reaching up to 86.7%, in popular multi-agent frameworks such as OpenHands (Wang al., 2025b) and MetaGPT (Hong et al., 2023), with failure modes ranging from improper task decomposition to role disobedience. Such a high incidence of failure casts a long shadow over the real-world reliability of multi-agent systems.
A natural response to this challenge is failure attribution, i.e., the precise identification of faulty components within the system upon task failure. Accurate failure attribution plays critical role across multiple dimensions: (I) system debugging, by enabling agentic systems to perform more effective self-debugging across iterative attempts, thereby enhancing performance; (II) data effi- ciency, by leveraging failed agent trajectories to construct more informative training data; (III) grounded self-improvement, by facilitating grounded self-correction through precise agent credit assignment. Despite its importance, this process is predominantly left as manual endeavor, requir- ing considerable human effort to analyze verbose trajectory logs.
While preliminary attempts have been made toward automation, their efficacy remains limited. For instance, Zhang et al. (2025c) employed state-of-the-arr models such as OpenAl-ol and DeepSeek- RI (DeepSeek-Al et al., 2024) perform failure attribution over trajectories from GAIA (Mialon et al., 2023), yet achieved accuracy below 10%. More critically, existing benchmarks for agentic failure attribution remain notably constrained. To the best of our knowledge, MAST (Cemri al., 2025) and Who&Wher (Zhang et al., 2025c) contain only 200 and 127 manually annotated multi- agent failure trajectories, respectively, offering limited scale for systematic evaluation. Accordingly, substantial research gaps remain along two critical axes: 1 training resources, concerning the automated construction of large-scale annotated multi-agent trajectories; and @ methodology, on developing swift and accurate multi-agent failure localizer, which we also refer to as tracer.
Present Framework. To address the aforementioned research gap, we present (1) AgenTracer, a fully automated pipeline for constructing rell-annotated multi-agent failure trajectories, and (2) AgenTracer-8B, a lightweight failure attributor for multi-agent systems. Methodologically, AgenTracer leverages + counterfactual replay, systematically replacing agent actions with oracle guidance to identify the decisive error step responsible for system failure. To enhance data diversity and ensure annotation precision, it further adopts a programmatic fault injection, synthetically generating failure instances by perturbing successful trajectories through targeted corruption.
Further, we fine-tune QWEX3-8B on the curated dataset to obtain AgenTracer-8B via rein- forcement learning (RL), enabling it to analyze long-horizon multi-agent collaboration traces. AgenTracer-8B takes as input the trajectory log and associated environmental feedback, and out- puts the identified decisive error step. A multi-granular reward is designed to supervise RL train- ing, emphasizing both step-level and agent-level attribution accuracy. At inference, AgenTracer-99l enables malfunctioning agentic systems to rapidly identify the critical failure step along with expla- nations, facilitating automated multi-agent debugging. Our contributions are as follows:
Automated Pipeline. We propose AgenTracer, the first automated pipeline for annotating multi-agent system failures, curating over 2,000 high-fidelity failure trajectories across seven datasets through counterfactual replay and programmatic fault injection.
Failure Tracer.
We develop AgenTracer-8B, a lightweight failure tracer dedicated for LLM agentic systems, trained via a multi-granular reinforcement learning to ensure accurate attribu- tion at both the step and agent levels, facilitating automatic agentic system debugging.
Empirical Evaluation. Experiments show that AgenTracer-8B facilitates (I) accurate attri- bution, outperforming giant LLMS like DREPSERK-H R1 by ~ 12.21% and 2EMINI-2.5-PMP by ~ 18. 18% on Who& When benchmark; and (II) self-evolution, enabling off-the-shelf agen- tic systems to improve performance by 4.8 14.2%.
2

---

Agen Tracer
2 RELATED WORKS
LLM-based Multi-Agent System. Contemporary multi-agent systems can be broadly categorized by their level of automation into three classes: Handcrafted, where the entire system configura- tion (e.g., LLM backbone. prompting strategies. and communication protocols) is manually speci- fied as in (Wu et al., 2023; Richards & et al., 2023; Li et al., 2023; Qian et al., 2023); Partially Automated, which automate specific system components: for example, AutoAgent (Chen et al., 2023a), LLMSelector (Chen et al., 2025), and MasRouter (Yue et al., 2025) automate agent role assignment: DsPy (Khattab et al., 2023) and TextGrad (Yuksekgonul al., 2024) optimize prompt design; GPTSwarm (Zhuge et al., 2024) and G-Designer (Zhang et al., 2024b) adaptively construct inter-agent topologies: Fully Automated, where all modules within the system are autonomously designed and evolved (Hu et al., 2024a; Zhang et al., 2024c: 2025b; Wu et al., 2025; Nie et al., 2025: Gao et al., 2025; Zhang et al.. 2025a). Our method is designed to provide precise error tracing across this entire spectrum of agentic systems. Accordingly, we curate trajectories from all three categories.
Failure Attribution for Agents. As multi-agent systems become increasingly intricate, incorpo- rating multiple intelligent agents (Wang et al., 2023; Chen et al., 2023b), tool integrations (Shen et al., 2024), and communication protocols Marro et al. (2024), the resulting high error rates and structural fragility have emerged as critical concerns. MAST (Cemri et al., 2025) was the first to alarmingly characterize this issue, identifying fourteen prevalent failure patterns ranging from task disobedience to reasoning-action mismatches. Building upon this, Who& When (Zhang al., 2025c) explored the feasibility of automating failure attribution, only to reveal that even giant reason- ing models like DeepSeek-R1 fail catastrophically. AgenTracer advances the field by introducing both a scalable data synthesis pipeline and a lightweight, high-accuracy failure attributor.
LLM-as-a-Judge & Credit Assignment. Two other research topics closely related to this work are: (1) J.LM-as-aJJudee (LaaJ), which leverages LLMs//geets as evaluators based on pre-defined criteria for tasks such as data annotation (Latif et al., 2025; OpenAI, 2022), value alignment (Ji al., 2025), reward modeling (Mahan et al., 2024; Lambert et al., 2024), and synthetic data gener- ation (Sengupta et al., 2025; Hu et al., 2025). However, when applied to multi-LLM systems, Laal has shown limited effectiveness, as demonstrated in Zhang et al. (2025c); (II) Credit Assignment is long-standing topic in multi-agent reinforcement learning (MARL), linking individual actions to long-term outcomes (Pignatelli et al., 2023; Arjona-Medina et al., 2019; Yin et al., 2023). Yet it remains largely unexplored in LLM agentic systems. The closest prior work, CollabUIAgents (He et al., 2025), uses LLMs to produce binary rewards per interaction, which are inherently unreliable. In contrast, AgenTracer achieves grounded credit assignment implicitly via principled attribution.
PRELIMINARY
In this section, we provide a general definition of LLM- based multi- systems and their opera- tional workflow, and then formally define the objective of the multi-agent failure attribution.
Notations. Consider LLMed multi-agent aulti- system M, tasked with collaboratively resolving user-issued query Q. The system consists of N agents, indexed by I = (1,2...., .N. operating in discrete time under turn-based protocol, i.e., only one agent is active each time step. Formally:
M M= (I,,,, pp),
(1)
where S denotes the set of system states; is the overall action space, with each agent E I having a local space A; C A: ju(t) E I specifies the agent scheduled to act at time t; V( | 8t, at n(t)) models the state transition dynamics given the current state 8t, action a; E AAuct)- and the acting agent. At each step t, the active agent p(t) selects an action a, E Ap(r) according its policy (y()). conditioned on the current state St, the query Q and subset of the prior interaction history He:
( H., 2), Hi S {ao, ai,..., (u.1.........
The structure of H4 is implementation dewpndddn..dddtd..t... In LLM Debate-style frameworks (Du et al., 2023), it comprises the rrior-round outputs from all agents; whereas in software development sys- tems (Qian et al., 2023; Hu et al., 2024b), tester agent may condition only on the latest code snippet submitted by a programmer agent. The full execution trajectory of the system is denoted by:
T (so, ao $1, .i....,.T))
(3)
where T' indicates the terminal step or stopping condition. The final response to the query Q is determined by the complete trajectory r, encapsulating the collaborative behavior of all agents.
3

---

AgenTracer Tracer
Data Source
Trajectory Collection
Automatic Construction
Multi-agent Systems
Successful Trajectory
Goali lacate the first fix of Sokit-Lears Jul 2017
[Tajoctoyy Repecscntation sq: actioo, Sy: octlin agent
Good
Ressoner Agent
Metoolt
AgentPrune
Agent
Web
ebullt ebullt injoction Injjct errers
Ie Accentiorfaciall ruplay
Orocle -idddd-ddddd correction
Planner Agent
Courdinate Agent
Code Agent
SmolAgents Workforce
OWL
if
trm
Failed Trajectory
Goali When the first the Apple stock went above $50?
P ((7,18'1
Diverse Benchmarks
Web Agent
4" enror agent F (decisivn) error
Resson Agent
Agent
2010
Training AgenTracer
Soffware
MBPP.
derelgmant
Agunt
w/ Google finance
Eivenn fuiled
Mulliple millutts (ressniing jodge)
gnaullr
Mulli-
Tocks
2303
with thi
mexard
Blackjack
HotporQA
GALA
Todginz
Judgmment:
Agunt
2000
emorr agent:
Shepe levels
Muthematicd
MATEL
Coordinate
Agent
Koyfin DB
FAIL
Reasoner
stops
Rule based reindorcement leerning
Figure 2: The overview of our proposed AgenTracer.
Objective Formulation. A trajectory may contain multiple suboptimal actions or minor devia- tions, but for effective, targeted debugging. it is crucial to distinguish these from the pivotal error that renders the final outcome incorrect. Following (Zhang et al., 2025c), we formalize this pivotal error as the decisive error, i.e., the earliest action in the trajectory whose correction is sufficient to steer the system from failure to success. Formally, let (2(T) E {0, 1} be binary evaluation function indicating failure (I2(T) = 0) or success (S2(T) 1), and R(T, t, af) be an oracle rectification oper- ator, which represents an idealized process where the original action is replaced by theoretically perfect, cracle-provide action a/. and all subsequent steps are re-simulated. Naturally, the set of all decisive agent-step pairs C(T) can be expressed as:
C(T) {(i, t) i vtt and ((T) 0 0 (R(T,,t, =);
from which we select the root cause by targeting the earliest error in time. The definitive failure- responsible agent i" and decisive error step f are therefore given by:
arg min t. (L.))E(())
(5)
Consequently, the goal of our failure tracer, AgenTracer, is to take a failed trajectory as input and output the failure-resposibbe agent i* and the decisive error step
METHODOLOGY
Figure illustrates the overall pipeline of AgenTracer. Specifically, AgenTracer samples multi- agent trajectories, and then applies programmatic fault injection (for successful trajectories) and counterfactual corrections (for failed ones) to identify the decisive error step, yielding 2,000+ tra- jectory-error step pairs, referred as TracerTraj-2.5k (D Section 4.1), with which we train dedicated failure attributor AgenTracer- 8B to accurately locate errors (I Section 4.2).
4.1 AgenTracer: AUTOMATIC TRAJECTORY ANNOTATION
Collection. AgenTracer begin by considering collection of ((= 6) distinct multi-agent sys- tems, [Mm) and corresponding set of queries for each system, D.,. (0, (m) }=m drawn from various task domains. For each query 0.m). .... executes and generates
trajectory 7 ((m) For the specific frameworks and datasets used, please refer Section 5.1. The trajectories from each system can be split into two sets, namely the successful (T(me) and failure (T (m) corpus:
T
(m) (0(()))) 1}, and T(m)
(7](m) | = 0).
sues
The overall successful and failure set can be expressed as Tance UM
M Tim and Trill = mml Till respectively, which serve as the raw input for the subsequent annotation stages.
4

---

AgenTraces
Locating Decisive Errors. To empirically identify the decisive error pair (i',t" .t') for each failed trajectory т E Tral we now detail our practical implementation of the rectification operator R introduced in Equation (4), which is is orchestrated by an analyzer agent aaaayze to conduct coun- terfactual intervention. Kanalyzer is provided with the full context of a failure: the trajectory т, envi- ronmental feedback FF (e.g., code execution errors, tool errors), and the ground-truth solution 9 for Q. For each step in the trajectory, the analyzer proposes minimally invasive, corrected action of designed to rectify the local error without revealing the complete solution:
df f Tanalyzer St. Ges He, F,9).
(7)
By sequentially applying this analyzer-driven intervention for E {0, 7 and evaluating the outcome (2(R(F,,, a,)), we can systematically search for the earliest step t that satisfies the condition in Equation (5). The agent active at this step, i* = fu(t), is labeled as the problematic agent, yielding a precise annotation (7, (i", .t)) ) for our dataset. This process (as presented in Lines to 8 of Algorithm 1), applied across the entire Tiil, forms D = {(T, (i*,t))) 7 E Thmb).
Utilizing Successful Trajectories. To further augment our dataset with high-precision annota- tions, we also leverage the corpus Tueee through programmatic fault injection. Its core principle is to take known-good trajectory and programmatically introduce a fault, thereby creating a synthetic failure instance where the decisive error is known by construction. Specifically, for each success- ful trajectory т E Tuuce, we select a step to serve as the injection point, at which a perturbation operator IT is utilized the original action ar generate corrupted action â, [[(a))) A new, syathetically-generall trajectory F is created by substituting this corrupted action:
(8) If this injection process successfully induces failure (i.e., (2()) 0), the pair (u((), 1) is, by definition, the decisive error for F. This allows us to generate a set of positive-sample datasets D+:
F = R,),,
D+ = ((F, ((") T E Tanes, 7 R(T, 4, (a.)), ?(() = 0, (i" (-) = (M(1),)).
(9)
The overall process is also elaborated in Lines 8 to 17 of Algorithm
Curated Dataset. By uniting the annotations derived from both failed trajectories and syn- thetically generated ones, we construct our final, comprehensive dataset, denoted as Pincer = D U DD+ which we also refer as TracerTraj-2.5KK comprising over 2,000 high-fideltty notated trajectory-error pairs. Detailed statistics on the dataset are placed Appendix B.
4.2 AgenTracer-SB: TRAINING AGENTIC FAILURE TRACERS
Having curated the Dracer dataset, we proceed to train our failure tracer, AgenTracer-88, whose base model is set QWEN3-8B. This phase aims to incentivize the model's ability to accurately pinpoint decisive errors within complex, long- nonghhorzoon trajectories. Since our approach is orthogonal the RL algorithm, conduct the experiments based on widely used online RL method, Group Relative Policy Optimization (GRPO) (Guo et al., 2025).
Online Reinforcement Learning. For each trajectory T sampled from Puncer. the current policy Told generates group of G candidate decisive error pairs, {Gi,, bx) G-) each of which is evaluated against the ground- -round-truth truth annotation (i",t") ,t)) using a mullti-rranlarr reward function Rg, which will be detailed below. Following previous practices that discover that KL divergence is not a necessity in RLVR training (Liu et al., 2025b; Yu et al., 2025), we omit the KI. divergence term, and introduce a dynamic clipping parameter Bs, which has been demonstrated to better balance exploration and exploitation throughout training (Liu et al., 2025a). The RL objective is thus formulated as:
G
CRL = -E,, (
min clip(pe. 1 Bs, 1 + (c,1+B.)An)
(10)
G
1=
where P& (ix, Ex). the policy ratio is ps (fcll) 1r)
[ar(pe]F] the estimated advantage is A (Rş.
mean({ [R ())((s(( (R))) + c) with = 1 10 being a small constant. The dynamic clipping parameter B, defined as B, max (0.2 Bo, Bo =nm),, with s as the current training step and Swast the total number of steps. This dynamic schedule intuitively encourages broader exploration in the initial stages of training and gradually shifts to more stable exploitation as the policy converges.
5

---

AgenTraces
Algorithm 1: Automated Trajectory Annotation Pipeline of AgenTracer
Input: Corpus of failed trajectories Taii Corpus successful trajectories Tawei Analyzer agent Tandyyeri perturbation operator II; Rectification operator
Output: The curated dataset Dircer
Initialize dataset: D + 0, D +
/- Part 1: Locate Decisive Errors via Counterfactual Intervention
for each trajectory т Tha do for to T do
a + (stalyzer de: He,F, Analyzer agent proposea corrected action R(T,t,)))/ Apply intervention to generate new trajectory
if ((') = then
+ u(t). L, D ((T,,(",
break // Found the earliest decisive error
/s Part 2: Generate Errors via Programmatic Fault Injection for each trajectory 7 Taces do
10
Let J {0, I.. be the set of possible injection points
11
Let Tampls f RandomSamplll K) Randomly select K steps for E Jampie do
[a;)) // Perturb original action create fault
14
# +- R(T, ,, Apply injection to generate a new trajectory
15
if (fF) = then
+ xtt),t" + Dt ((F
17
break // Stop after first successful injection for this trajectory return curated dataset Duncer = D UD+
Multi-Granular Reward Design. Regarding the implementation of advantage estimation in Equation (10), introduce multi-granular reward designed evaluate both the correctness of the attribution and the structural integrity of the output. The total reward R₁e for candidate predic- tion Pa is a gated combination content accuracy and format compliance:
Iroemat (> (A-rstep(fea (fs) + (1 - A) "aagen (oz))
R(pk)
(11)
whose components are further defined as follows:
Format Reward Tommat strict binary reward that equals and only if the model's output adheres to the required structure: reasoning must be enclosed within (think) (/think) tags, followed by a final answer within (answer) ... (/answer) tags. Furthermore, the answer itself must be formatted as (agentID) (stepID) for accurate extraction.
Agent-Level Reward Fagent is coarse grauseed binary reward that measures whether the tracer correctly identifies the failure-responsible agent i", defined ragent ( 1lfx = i*), where I((c) is binary indicator for the accuracy of located problematic agent.
Step-Level Reward step incentivizes temporal proximity to the true decisive error t* We use Gaussian kernel where the reward decays smoothly as the predicted step la moves away from t":
Tstep (1x) = exp (- (te (
(12)
20²
where a controls how sharply the reward penalizes distance from the correct step.
This muuli--ranular design creates smoother reward landscape for failure localization, as the partial credit from "stp stabilizes training. Simultaneously. the hard gating from Ffomaa ensures the model produces reliably parsable outputs. Through online RL with these designs, we obtain a reasoning- based multi-agent failure attributor AgenTracer-8B.
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Dataset Curation. For collecting TracerTraj-2.5K. we opt for six widely-used multi-agent sys- tems, comprehensively incorporating all automation levels: manually configured, including

---

AgenTracer
Table 1: Performance comparison on the Who& When benchmark. For each subset, evaluation is conducted at both the agent and step levels. Each cell reports two values: the left corresponds to the setting w/ & (the fallure tracer has access to ground truth trajectory), and the right corresponds to w/o G. The best and second-best results are bolded and underlined, respectively.
Who& When (handcraft)
Who& When (automated)
Model
Agent-level
Step-level
Agent-level
Step-level
QWEN3-8B
42.10739
1.7273.45
58.73/60.32
3,97/5.56
LLAMA-332-3B
37.93/50.00
1.72/3.45
37.30/45.23
2.38/8.73
QWEN3-32B
44.80/44.80
1.72/1.72
63.49/57.93
9.52/8.73
QWEN3-CODER
51.72/60.35
8.62/13.79
42.86/36.50
34.13/32.54
GPT-4.1
43.10/37..33
3,44/3.44
55.55/59.52
29.52/21.90
DREPSEEK-RI
56.90/53.44
13.29/6.90
66.67/65.08
31.32/29.52
GEMINI-2.5-PRO
51.72/51.72
9.72/6.90
61.11/57.14
29.52/25.86 86
-
56.90/50.00
17.24/18.97
57.93/51.11
40.65/38.83
AgenTracer
69.10/63.82
20.68/20.68
69.62/63.73
42.86/37.30
MetaGPT (Hong et al., 2023), AutoGen (Wu at., 2023) and Smolagents partially automated, including AgentPrune (Zhang et al., 2024a); fully automated, including AFlow (Zhang et al.,
2024c) and OWL-Workforce (Hu et al., 2025). Seven benchmarks include coding, MBPP+ (Liu et al., 2023), KodCode (Xu et al., 2025) and Blackjack (Hong et al., 2023); general agentic tasks, GAIA (Mialon et al., 2023) and HotpotQA (Yang et al., 2018); mathematical reasoning, MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021).
Environment. All experimental results are obtained on one server with 8 NVIDIA H100 (80 GB) GPUs. For RL training in Section 4.2, we use the verf training platform.
Model & Parameter Configuration. The analyzer agent Tamalyze in Equation (7) and perturbation operator in Equation (9) are both based on DEEPSEEK-RY (Guo et al., 2025) (see prompts in Appendix C). The coefficient À in Equation (11) is consistently set as 0.5, and the parameter in Equation (12) equals The LLM backbone used for training AgenTracer-8BB is QWEN3-8B. For RL traning in Section 4.2, set batch size to 32, rollout number 8, and learning rate 1 10-6
Benchmarks & Evaluation. To evaluate the failure attribution capability of AgenTracer-88, we adopt the Who& When benchmark (Zhang et al., 2025c), which comprises two subsets: hand- crafted set derived from Magnetic-One (Fourney et al.. 2024), and an automated set constructed from AG2 (Microsoft, 2024). Both subsets provide unseen trajectories with respect AgenTracer-885 In addition, we sample held-out test split from TracerTraj.2.55 using 9:1 ratio, yielding three evaluation subsets (devided by domains): TracerTraj-cole TracerTraj-mane and TracerTraj- agentic. Detailed statistics are reported in Appendix B. For evaluation, following (Zhang et al., 2025c), we adopt two primary metrics: agent-level accuracy and step-level accuracy. The former measures whether the attributor correctly identifies the faulty agent 1" within a trajectory, while the latter assesses whether the specific erroneous step t* is localized. We consider two evaluation set- tings: (i) w/ G, where the attributor has access to the ground truth G during failure attribution, and (i) w/o G without such access. The latter setting is harder and particularly valuable. We follow the "all-at-once" setting introduced in MAST, where the entire trajectory is provided the LLM in single pass, as Zhang et al. (2025c) has demonstrated this to be the most stable and effective.
Baselines. We compare AgenTracer agaist LLM baselines of varying scales, encompassing small-size models such as QWEN3-8B (Yang et al., 2025) and LLAMA-3.2-23 (Grattafiori et al., 2024); medium-size models, including QWEN3-32B and SWEN3-CODER-4400) INSTRUCT (QWEN3-CODER, (Yang et al., 2025); and large-size models, which primarily consist of state-of-the-art LLMs, such as GPT-4.1 (OpenAI, 2025), GFMINI-2.5-PRE (Comanici et al., 2025), CLAUDE-4-SONNET (Anthropic, 2025) and also DEEPSEEK-R R1 (Guo et al., 2025).
5.2
MAIN RESULTS
This section provides empirical evidence that AgenTracer-8B outperforms substantially larger models in failure attribution within complex agentic systems. Tables and 2 report results on Who& and TracerTraj subsets, respectively, presenting both agent-level and step-level at- tribution accuracy. Each table entry is divided into w/ ground truth G and w/o G during attribution.
'https:/ ://gi thub. www..hh..b..c.cceeeeeeeeteee cww.huggiihhcococcooooo........o ng face/ saaaesaman smo agents Phttps://4 // thub. wow/vol cengi ne/verl
7

---

Table Performance comparison on different subsets of TracerTraj. For each subset, accuracy is reported at the agent/step levels. Each cell reports two values: the left corresponds to the setting w/ 9, and the right w/o g. The best and second-bes results are bolded and underlined, respectively.
Code
MATH
Agentic
Model
Agent-level
Step-level
Agent-level
Step-level
Agent-level
Step-level
QWEN3-8B
45.35/32.99
2.36/1.15
31.74/33.58
9.52/T2.96
27.93/30.16
13.49/15.31
L1-MMA-32-3B
13.38/11.81
2.36/3.93
15.87/14.28
4.76/3.17
8.11/13.15
2.18/5.09
QWEN3-32B
62.99/63.78
2.36/1.75
17.46/17.46
4.76/7.93
30.55/30.55
18.60/15.31
QWEN3-CODER
69.29/66.92
14.96/14.17
28.57/33.58
11.11/22.22
40.67/43.12
24.99/25.69
GPT-4.1
59.84/53.54
12.59/11.02
39.55/35.18
35.81/26.63
43.61/40.67
24.99/25.69
DEFPSEEK-RI
11.81/11.81
10,23/10.23
42.68/38.58
29.52/18.51
45.12/46.11 19
27.13/21
GEMINI-2.5-PRO
70.07/66.92
11.02/6.29
58.79/58.79
32.22/27.40 40
37.16/32.18
18.60/17.04
CLAUDE-SONNEI4
65.98/63.78
15.51/11002
46.03/50.79
10/46.00
55.20/49.13
30.33/29.8 80
AgenTracer
72.95/72.21
18.85/18.85
59.32/66.10
57.63/57.63
53.28/50.61
36.17/35.55
MaAS GATA (Validation)
OWL GATA (Validation)
Trial
88
Trial
Accuracy (%)
30-
Trial
Accoracy (%)
Trinl
25
Trial
Trial
8
+AgenTracer
Seel--Refine
+CRITIC
+AgenTracer
Seel--Refine
+CRITIC
MetaGPT HumanEval-Plus
MaAS MATH-500
Trial
Accuracy (Xg
Trial
Acenracy (9)
66
Trial
Trial
Trial g
Trish
61
Trial 4
81
.66
+Self-Refine
+CRITIC
31
ApenTraacer
+Self-Reftne
-CRITIC
Figure 3: The multi-turn improvement performance brought by AgenTracer-88l compared with classical agent reflection baselines, Self-Refine, and CRITIC.
prevailing models are inadequate as failure attributors. As shown Ta- ble 1, smaller models like QWEN3-8B and -LAMA-3.2-3B fail deliver meaningful judgments, with step-level accuracy on Who& When (handcrafted) below 10%. Even larger models such as DEEPSEEK-RI 1 and GPT-4. perform poorly, achieving only 31.32% and 29.52% Who& When (automated) despite access to ground-trun truth G. Supplying G does not always help: on TracerTraj- math, CLAUDE-4-SONNET attains 46.03% (w/ ②) vs. 50. 79% (w/o 9), and on Who&Wher (hand- crafted), QWEN3-CODRR CoDER achieves 51.72% vs. 60.35%. This suggests that ground-truth supervision can sometimes mislead attribution, consistent with prior findings (Zhang et al., 2025c).
Observation I:
Observation @: AgenTracer consistently surpasses giant proprietary LLMs such as CLAUDE-
4-SONNET both agent- and step-level attribution. Under the w/ setting, as shown in Table 1, AgenTracer-9B outperforms GPT-4. 1 and CLAUDE-4-SONNET
Who& When (handcrafted) by 26.0% and 12.2% in agent-level accuracy, respectively. A similar trend is observed on TracerTraj (Table 2). where AgenTracer-SB improves step-level accuracy TracerTraj-agentee by 22.68% over its backbone QWEN3-8B, 3-B, while also surpassing DEEPSEEK- R (+9.04%) and GEMENI-2.5 PRO (+17.57%). More importantly, in the w/o & setting (arguably the more realistic scenario where ground truth unavailable), AgenTracer-88 remains robust: on tracerTraj-matt DEEPSEEK- RI suffers 9.21% drop without G, whereas AgenTracer-93 maintains 57.63%. This strongly substantiates the real-world deployability and practical significance of AgenTracer-9B.
5.3 BOOSTING MAINSTREAM MAS
Having established the accuracy of AgenTracer in failure attribution, a natural question arises: what practical value does it provide? The most direct answer is its potential supply actionable feedback to failing LLM-based agentic systems, thereby enabling swift self eef--improvement. To as- sess this capability, compare AgenTracer-SB with two classical self-refinement approaches,
8

---

AgenTracer Tracer
Oat Sqperentt suten from tae compary poital
Judgement from Error Qwen3-8B
Error Step:
Flons search quetartyy coles cass
Agent:
2
Garter
nave
The error occure Step G. The agant respontable the Codiing Agent The reesson vig
message
incoerect column name- "Toe-"Tr "Tri rop represents direct and veriiable breakdiown execution process. The agent was given a task and reiled to complete due to bug in segurence
Manager
Surfer
Manager
exitence
Cotracto Agerit The regin
due is Mookinn
Judgement from Error Mloude-4-Soneee Agent:
Error Step: 9
File
Manager
Coder
Step The aggnrt resperecible the Manager Tie reazcn Is that the Manager trcated the incoriect co umn nome as simple typo be floed.
The critizel mistake
emorr ccrnect
worpt
werning sign espacted fellure inselliiiee the setiemm the sorps, represents The truc point where the process falled
ying
24 kp indry
Repere "heee parkmrin
and roclwed paeching
name homel
MA
Manager
Coder
Manager
Judgement from Error Agent:
Error Step: 2
Tracer
Sorken
MuA
Conbrr Hostr
Ne check Step agent responsible last Surter -88-8-11 la conclusine ev disnnce that the aganni down kcaded a years-okd archived rsport Inalead of ourment This instial data acquistionn error invalidutad undire anaayosi flom
misttake is
the
anouer
Manager
File
Manager
Figure 4: Case study of failure attribution in long-chain document analysis task, comparing three models (QWEN3-8B, CLAUDE-4-SONNES and AgenTracer-88).
Self-Refine (Madaan et al., 2023) and CRITIC (Gou et al., 2024). Specifically, when an agentic system M completes problem-solving episode and produces failed trajectory we supply (wo 9) to either AgenTracer-88 or Self-Refine/CRITIC. Each method then generates reflective feedback the failure (for AgenTracer-8B this corresponds to the reasoning trace extracted from (think) ... (/think)). This feedback is subsequently injected into M during the next round of prob- lem solving, with the aim of leveraging external critique enhance its performance. We iterate this process for three rounds, and both Relf-Refine and CRITIC are instantiated using GPT-4.1.
Observation @: AgenTracer-Bal boosts off-the-shelf agentic systems by up to 14%. We com- bine AgenTracer with three representative agentic systems (MaAS (Zhang et al., 2025b), OWL Workforce, MetaGPT) and three benchmarks (GAIA, HumanEval+ (Liu al., 2023), MATH-500). As shown in Figure 3, conventional reflection-basee methods provide little insight for complex tra- jectories; even with GPT-4.1, CRITIC often degrades performance (e-g., CRITIC+MaAS+GAIA accuracy drops -4.9% at iteration-2 and 5.5% at iteration-3). In contrast, AgenTracer consis- tently improves outcomes. Notably, boosts OWL on GAIA by +4.8% and MaAS on MATH-500 by +14.21% surpassing both Self-Refine and CRITIC. Overall, AgenTracer provides reliable cor- rective feedback across diverse domains for complex agentic systems.
5.4
CASE STUDY & ABLATION STUDY
Case Study. Figure 4 compares QWEN3-8B, CLAUDE-4-SONNET, and AgenTracer-88 on a trajectory where the task is find the region with highest infant formula sales in Q1 2024. The final answer was incorrect: "North." " QWEN3-8B 3-B gives a superficial diagnosis, wrongly blaming code execution error at Step 6. CLAUDE-4-SONNET notes the Step error may have deeper causes. In contrast, AgenTracer-89 correctly attributes the root cause to Step 2, where the Web Surfer agent retrieved files with the wrong date, an error that only becomes apparent when analyzing ev- idence at Step 11. This highlights the intrinsic difficulty of failure attribution in agentic systems: errors are often subtle, originate early, and remain hidden behind seemingly correct outputs. We place other detailed attribution processes of AgenTracer 8B in Appendix E.
A Ablation Study. In Appendix D, we validate the effectiveness of both agent-level and step-level RL rewards training AgenTracer-83, as shown in Figures 5 and 6.
6 CONCLUSION
This work establishes a principled foundation for the study of agentic system failure attribution. By introducing AgenTracer, we provide the first automated framework capable of systematically gen- erating annotated failure trajectories, as well as AgenTracer-9B, lightweight yet effective failure tracer that leverages multi--ranulatt RL to achieve prevailing diagnostic accuracy. Empirical evalu- ation demonstrates that AgenTracer-SB not only surpasses state-of-the-an art proprictary LLMs like GEMENI-22 PRO and CLAUDE-4-SONe SONNET on the Who& benchmark but also yields consis- tent performance gains when deployed within real-world multi-agent frameworks. Beyond advanc-

---

AgenTracer
ing the state of failure attribution, approach paves the way for self coffecorret and self-evolving agentic systems, marking a step toward more resilient and autonomous collective intelligence.
ETHICS STATEMENT
This work studies failure attribution in LLM- agent systems through synthetic trajectory cura- tion and automated error annotation. All data is collected from standard academic benchmarks and simulated agentic frameworks, without involving human subjects or real-world becision-making. Consequently, our methods and experiments do not present direct ethical risks, and the generated resources are intended solely for research purposes.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we present an anonymous link in the abstract share our source code. Details on parameter settings and model configurations are elaborated in Section 5.1.
ACKNOWLEDGEMENTS
This research was supported in part by the NUS Start-up Grant (A- (A-0010106-00-00) and by the National Natural Science Foundation China under Grant No. 62320106007.
REFERENCES
Anthropic. Claude Sonnet htttpp///ww..artrrrgi......////. cessed 31 31-08-2025].
2025. [Ac-
Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brand- stetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019.
Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, al. Why do multi-agent Ilm systems fail? arXiv preprint arXiv:2503.1300 2025.
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. CoRR, abs/2309. 17288, 2023a. doi: 1044550/ARXIV22300.122 172888 URL https: //tp:///iii//ooo wtt.d/.....//.....5////..... 2309. 17288.
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Matei Zaharia, James Zou, and lon Stoica. Optimizing model selection for compound ai systems, 2025. URL httpt:htttaaatt
htrgaabs25502.144
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facili- tating multi-agent collaboration and exploring emergent behaviors in agents, 2023b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv prepring, abs/2110.141683 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
DeepSeck-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting
10

---

Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang. Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma. Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng. Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou. Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang. Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2024. URL https: /atp:i////rvvvv///a org/ abs 2412.19437.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied mul- timodal language model. 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305.14325, 2023.
Lutfi Eren Erdogan, Nicholas Lee, Schoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anu- manchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act Improving planning of agents for long-horizon tasks. arXiv preprint 20050225033 2025.
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang, Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. Magentic-one: A generalist multi-agent system for solving complex tasks, 2024. URL https: arxiv. org/ abs 244.///4114448.
Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, and Tianyu Pang. Flowreasoner: Reinforcing query-level meta-agents, 2025. URL https: //tpxxi..rggaas/////.
15257.
Alireza Ghafarollahi and Markus J. Buehler. Seiagents: Automating scientific discovery through org/ abs/2409/
multi-agent intelligent graph reasoning, 2024. URL
05556.
Ali Essam Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn J. Szostkiewicz, Jon M. Laurent, Muhammed T. Razzak, Andrew D. White, Michaela M. Hinks, and Samuel G. Rodriques. Robin: A multi-agent system for automating scientific discovery, 2025. URL https: / //arxiv.org/aata abs/ 2505 13400.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujuu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with inot-interective critiquing. In The Twelfth International Conference on Learning Representations, 2024. h /ttp///eepe/...e./// net/forue btt.rrmmmmmisx000xxxeeeeeee
11

---

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXi::240.221783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Zhitao He, Zijun Liu, Peng Li, Yi R Fung, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu, Advancing language multi-agent learning with credit re-assignment for interactive environment generaliza- tion, 2025. URL https: /tttpp:///iiiii/
wttp:o/pag......eeeeeee/ 1449.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework, August 01, 2023 2023.
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An Ilm agent for data science. arXiv preprint 2402. 18679, 2024.
Mengkang Hu. Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun. Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, Zeyu Zhang, Yifeng Wang. Qianshuo Ye, Bernard Ghanem, Ping Luo, and Guohao Li. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025. URL https:tpp/... www.hhttss//..iiieeeee http///axvvvv////// org/ abs /2005 272555228888 23885.
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408. 08435, 2024a.
Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-evolving multi-agent collaboration networks for software development, arXiv preprint arXiv:2410. 2024b.
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Ilm agents: A survey. arXiv preprint 402.:6240..02716, 2024.
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, Sirui Han, Yike Guo, and Yaodong Yang. Pku-saferlhf: Towards multi-level safety alignment for Imms with human preference, 2025. URL https://
wwxivv.rg//ee
hbtp:/bb//24000 ora.iv.org/abs/24055 15513.
Bowen Jiang, Zhijun Zhuang, Shreyas S. Shivakumar, Dan Roth, and Camillo J. Taylor. Multi-agent vqa: Exploring multi-agent foundation models in zero-shot visual question answering, 2024. URL
wwtpeiii.org///eeee aaa/////////00 14783.
Omar Khattab, Amav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.037144 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannameh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL
https:// /ttpps:////xvvv./// wwwpiiivorrg//ooo alos/ 13787.
Siddique Latif, Muhammad Usama, Muhammad Ibrahim Malik, and Bjöm W Schuller. Can large language models aid in annotating speech emotional data? uncovering new frontiers [research frontier]. IEEE Computational Intelligence Magazine, 20(1):66-77, 2025.
12

---

AgenTrace
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicative agents for "mind" exploration of large language model society. NeurlPS, 2023.
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking Ilms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428- 100534, 2024.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation large language models for code generation, 2023.
URL wwtph:tpp//.om /ttpvv/wvvvv///////a//ee./aaaavv .rrg abs /2305 01210.
Yue Liu, Shengfang Zhai, Mingzhe Du, Yulin Chen, Tri Cao, Hongcheng Gao, Cheng Wang. Xin- feng Li, Kun Wang, Junfeng Fang, et al. Guardreasoner-vl: Safeguarding vlms via reinforced reasoning. arXiv preprint arXiv:2505. /049, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: critical perspective, 2025b. URL https:
// //aaxiv.orggaaaea..... 2503 20783.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In NeurIPS, 2023. URL deeeffoo222eblb5a55500
nips. cttpsper_pppppppaeeeee -ttpsaaaeetttoooaaaaaatt-----o-o Conferent- html.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models, 2024. URL
https:
org/ abs /2.//2411... 148..2282........
Samuele Marro, Emanuele La Malfa, Jesse Wright, Guohao Li, Nigel Shadbolt, Michael Wooldridge, and Philip Torr. A scalable communication protocol for networks of large language models, 2024. URL https::t//// http//xxivv..///// http://///awvvooo/////////2//// 11905.
Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. arXiv preprint arXiv:2404. 11584, 2024.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Seialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Microsoft, GitHuu ag2ai/ag2: AG2 (formerly AutoGen): The Open-Source AgentOS. https: //github. caclb..com/ggaa/ haamm/g2aa//gga/ 2024. [Accessed 31-08-2025).
Marvin Minsky.
Society of mind.
Simon and Schuster, 1988. URL https:
/ //www. htt.rraaacccatttt..ccooooo/////s wwt.c///oooss/oioeet--------- Mwt.://///a.---Msss/......
9780671657130
Fan Nie, Lan Feng, Haotian Ye, Weixin Liang, Pan Lu, Huaxiu Yao, Alexandre Alahi, and James Zou. Weak-for-strong: Training weak meta-agent to harness strong executors, 2025. URL 04785.
https:/ http:s//aaxxv..rggg////
OpenAl. Chatgpt: Optimizing language models for dialogue, 2022. clog/chattpt/.
com/
OpenAl. Gpt-4.1 model card. 1, 2025. [Accessed 31-08-2025).
Perplexity, RL Training For Math Reasoning - perplexity.ai. https: http///ppppp.xxt......../pp
a ai/hhb/bbog//eee http://rttrraannnnr-eee for----oo-mriiioo rew.athihnr.nnnnng..... 2025. [Accessed 2025).
13

---

AgenTraces
Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment deep reinforcement learning. arXiv preprint arXhv:2312.01072, 2023.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.04 0ete:2408.0711999 199, 2024.
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, July 01, 2023 2023. 25 pages, figures, 2 tables.
Toran Bruce Richards and et al. Auto-gpt: An autonomous gpt-4 experiment. https://githat
com/ Si gni gnificant---e Gravitas AutttuutttaPa. 2023.
Saptarshi Sengupta, Harsh Vashistha, Kristal Curtis, Akshay Mallipeddi, Abhinav Mathur, Joseph Ross, and Liang Gou. Mag-v: A multi-agent framework for synthetic data generation and verifi- cation, 2025. URL https://///..... /ttt://arxxx aarpi.ivvv.v////v///./////avvavv www.v.owgg.aaaaaeooooo.o.oo abs / 2422. 04494.
Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small 1lms are weak tool learners: A multi-llm agent, 2024. URL https:
/ //arxiv. hww.voorg/abaa 2401. 07324.
Junlin Wang, Roy Xie, Shang Zhu, Jue Wang, Ben Athiwaratkun, Bhuwan Dhingra, Shuaiwen Leon Song, Ce Zhang, and James Zou. Improving model alignment through collective intelligence of open-source Ilms, 2025a. URL https: //apii/aaaavvv/aa org/ 2505 03059.
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533, 2024.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for ai software developers generalist agents, 2025b. URL https: /aaxiiv.vvv../////////aa////aaav org/ als 2407 16741.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self- collaboration, July 01, 2023 2023. work in progress.
Yuxi Wei, Zi Wang. Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang- Editable scene simulation for autonomous driving via collaborative llm-agents. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15077- 15087, 2024.
Yuxiang Wei, Olivier Duchenne, Jade Copet. Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing 1lm reasoning via re- inforcement learning on open software evolution, 2025. URL 2502. 18449.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen 1lm applications via multi- agent conversation framework, August 01, 2023 2023.
Shirley Wu, Parth Sarthi. Shiyu Zhao, Aaron Lee, Herumb Shandilya, Adrian Mladenic Grobelnik, Nurendra Choudhary, Eddie Huang, Karthik Subbian. Linjun Zhang, et al. Optimas: Optimizing compound systems with globally aligned local rewards. arXiv preprint arXiv:2507.03x. 2025.
Zhangchen Xu. Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025. URL https: httpss////xvv...///// org//bs/ /503. 02951.
14

---

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: /aaxiviviv..////.va/aaaa////aavv org/ abs /205 09388.
Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. Embodied Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 26275- 26285, 2024.
agent trained by Ilm from parallel textworld. In
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arviv preprint 1rXiv:1809.09600 2018.
Haiyan Yin, YAN Shuicheng, and Zhongwen Xu. Distributional meta- veeta-gradient reinforcement learn- ing. In The Eleventh International Conference on Learning Representations, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Guang- ming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei Ying Ma. Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingx- uan Wang. Dapo: An open-source I1m reinforcement learning system at scale, 2025. URL hatps:///axxvvcooo wwwp/ao./////oooooooooooooo.o.t/ atw./////./22./2/// 2503 14476.
Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning to route ulms for multi-agent systems. arXiv preprint arXiv:2502.1 // 1133, 2025.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text, 2024. URL https:: //rxxiv. org/abs/ /2006 07496.
Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506 2024a.
Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang. Tianlong Chen, and Dawei Cheng. G-designer: Architecting multi-agent communication topolo- gies via graph neural networks. arXiv preprint arXiv:2410.1178. 2024b.
Guibin Zhang, Kaijie Chen, Guancheng Wan, Heng Chang. Hong Cheng, Kun Wang, Shuyue Hu, and Lei Bai. Evoflow: Evolving diverse agentic workflows the fly. arXiv preprint arXiv:2502.07373. 2025a.
Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. Multi-agent architecture search via agentic supernet. arXiv preprint arXiv:2502.0418 04080, 80, 2025b.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong. Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating Agentic Workflow Generation, October 2024c. URL attp:://axiin wwt./org///b///oe///oo
10762. arXiv:2410.10762.
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al. Which agent causes task failures and when? on automated failure attribution of Ilm multi-agent systems. arXiv preprint arXiv:2505.00212, 2025c.
15

---

Agen Tracer
Wentao Zhang, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. Agentorchestra: A hierarchical multi-agent framework for
task solving, 2025d. URL https:
//aaxii..rrg/aaaaaaeeeea wttpor/gg/aa//ooeee/aeeee/////// /200. 12508.
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ô. Arik. Chain of agents: Large language models collaborating on long-context tasks, 2024d. URL https: //aaxxiv.rrg/////e/l.......ooo.t abs /240.. 02818.
Sipeng Zheng, Jiazheng Liu, Yicheng Feng, and Zongqing Lu. Steve-eye: Equipping llm-based embodied agents with visual perception in open worlds. arXiv preprint arXiv:2310.13255, 2023,
Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, and Ningyu Zhang. Knowagent: Knowledge-augmentt planning for Ilm-based agents. arXiv preprint arXiv:2403.03/11 2024.
Mingehen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs In Forty-first International Conference on Machine Learning, 2024.
16

---

AgenTracer
Table TracerTraj dataset statisties and distribution across three domains, including the associated multi-agent systems. For each domain, we list the included benchmarks, the number of curated trajectories, and the subset of trajectories annotated with error-step pairs (TracerTraj- 2.5K).
Metric/ / Domain
Coding
Mathematical Reasoning
General Agentic Tasks
MBPP+
MATH
GAIA
Benchmarks
KodCode
GSM8K
HotporQA
Blackjack
MctaGPT
AgentPrune
Smolagents OWL-Workforce
Multi-Agent Systems
AutoGen
AFlow AutoGen
AgentPrune
Collected Trajectories
2,170
1,185
1,300
(raw trajectories, annotated)
TracerTraj-2.02
1288
630
558
(including train test set)
Test set of TracerTraj-2.5K
147
63
56
A UTILIZATION OF LARGE LANGUAGE MODELS
In the development of this research, large language models (LLMs) utilized to refine the manuscript, conduct thorough literature reviews, and generate visualizations.
B DATASET DETAILS
Table 3 illustrates the detailed distribution of our AgenTracer.
C
PROMPT SET
Prompt for Analyzer Agent
promot You are software development team Lasked with diagnosing ing failed programming task. Your goal is to identify the critical error in the implementation.
Task Information:
ID: [task_id]
question: (question)
Ground Truth: truth
Model Prediction: (model_prediecionn)
Original Task Execution History: [hictory_str)
(history.conttraitt]]
vvalidation_inetruattii
Your diagnosis should
following JSON
"nistake_step"; eatep_number>, // The step number where the ezror occurred "mistaka_ogant" "etha agens that made the mistake>*, the agent. that made the mistake (e.g., "Rngineer", *Arch' "Arch'tect" " "Producttanageer", etc.) "reason"; seetaalledeeplans.lli wrong
Delailed explanation of why step
"suggested_ giix_gnide guidance
arrce,
complete solution
13
Important Guidelires:
17

---

1. Do NOT provide the complete solution in the suggested_fix. Only provide guidance on error.
2. Pocus identifying the reo: cause of the failure.
The 'nistake_step' should be a number corresperding to step in the implementation precess.
1. The mistake_agent should be the specifle agent that made the mistake le.g., anginees", "Architect", "EroductManages", "TeamLeades", "Datsknalyst"";
5. The 'raaonn should be detailed and explain why the current implamentation is Incorrect.
The "suggested_ should provide cleas guidance without giving away the complete solution.
Analyze the original task execution histery understand the context and identify where things went wrong.
cairICal:
histery your mistake_agan agent matches the agent that actually ly performed that step.
submitzing, verify your pistake_szep
the disgnosia result, NUST the Editor.create_fi command with the following format: rirst, the file using Editor.cst siitor.creato_ille command
IMPORTANT:
1:
"command_name" "Editor.craate_fille""
*rgge": If "Lllenam": ieleae_...epllaee/ /", 'L') diiggosssssssssse Json"
11
[]
Then command modi fy the
Please analyze the taak and provide your diagnosis in the apecified JSON format, The diagnosis result should
file named (sask_id.roplacs /*, (_")) _diagnosia, jaon" in the workspace directory, "**
Prompt for Analyzer Agent
prompt Eor"You an inselligent attack analysing succesalul execution Cor programming zask.
Your geal is to identify the most critical step in the exacution history that, if modified,
Lead incorrect final result.
Task Information:
Task ID: [task_id)
Question: sesccess_logl'eee (uuesllooogqqssioo"]]
Ground Truth: [success_lop] (uinosss_og""grruunddeeettt grounddrrrutt'
Model Prediction: (success_ log( model_predict
(previous_aree aaaatoov_attak_nnnt
Original Task Executies llssooy:: (history_str)
[history_comstraiitt)
Your sttack analysis should be in the following JSUN format:
[
"sttack_step": <ssep_number>, // The step number where the sttack should be applied
*m' stake_agent"; "<tha agens made mistake>*, The agent that made mistake
boriginal_conter "cbrief description description of the original step
this step does>", Brief "tttettconteet"; "<SPECIFIC, IMPLEMENTABLE attack instructions with concrete code changes>", // SPECIFIC attack instructions
*reason"; ledetailed explanation why this step is critical and how the attack Detailed explanation
1:
CRITICAL REQUIREMENTS FCR ATTACK XTTACK_CONTENT CONTENT:
The attack_conter MUST be SPECIFIC and IMPLENENTABLE Inciude
CODE CHANCES NODIFICATIONS
3. Provide SFFCTFIC instruction that can be directly executed FUNCTIONALITY CHANGES that
the program, fall
5. Examples of good s:tack_content: "Change return statement
tup (dct, to List(tup)
dct]**
"Nodify the funczion to return None instead of the tuple"
"Add bug: change 'return (dot, rreuurn the dictionary)"
(dct, dct)' (duplicate
18

---

AgenTracer
"Change the function zo ignore the dictionary: 'return tup"
vague instructions like *return incorrect type* "madify the
Inportant Guidelines:
Fozus identifying the
of patential failure, just any step.
2. The 'sttack_step' should be numbrr cozrespondiag to a step in the implemeatation precess.
3. The 'mistako_agent should he the agent that made the mistake.
should briefly descr what step does.
The "stlack_corten." MUST be SPECIPIC and IMPLENONTADLE with concrete changes. The 'reason' should be detailed and explain why this step is critical and how the attack would work.
Ana yse the original task execution history to understand the context and identiry where things could go wrong.
8. Focus on stepa that involve code generation, implementttoon, or key algorithnic decisions.
CRITICAL AEQURENENTS:
MUST
the file FIRST
Editor aitsoorrreata_fll.
2. You MUST write the content SECOND uaing Editor.write
MUST
exact filename: "itaak_.d.roppaes
'_() Lalattcck_nnllyiiise
4. You MUST NCT use the fandr command until both file operations are completed
You MUST provide the atzack analysis in valid JSON format
step-by-step process: Firat, the file:
10
command name"; "Editor. Seditor.creste_fille
*args*: [[
"fileroms": (taak_id.rapplass
11
rhen, write the atsack analysis ***json
1(
command_ramer "Bditor.write",
azgg": il
"path": "[task ideepaaed. L Lttl_aatcck_anallsit json",
"content"; "[["sttack_stop; "...", "origirall_onnante cooll_cootttt
atsack_content", "SPECTF"O cons, CHANGES HERE", "reason"; "J""
11
only after bath fi operations succesaful, end command: ***aann
1(
"command_rama"; "and*
}}
analyze provide your attack analysis. **.
D ABLATION STUDY
We perform an ablation study to investigate the contributions of agent-level and step-level rewards AgenTracer-8B under the w/o setting. shown Figure 5, removing the agent-level reward causes only a slight decrease in agent-level accuracy, e.g., from 72% to 68% TracerTraj- code, and from 50% to 48% on level supervision. In contrast, eliminating the step-level reward results in more substantial drop, particularly in the step-level accuracy (Figure 6), e.g., on TracerTraj-code from 18% to 12% and TracerTraj math from 35% 27%, demonstrating the critical role of fine- fine-grained step-level guidance. Overall, both reward components positively influence performance, and their removal consistently produces negative effect across domains.
indicating a moderate contribution of agent-
19

---

AgenTracer Tracer
AgentTraeer
72.2
w/o Agent-level Reward
70
68.9
69.2
w/o Step-level Reward
66.1
63.8
611
60
52.6
50.6
50
48.1
Code
MATH
Agentic
Figure 5: Ablation results the agent level under who 9 setting. Removing agent-level reward yields only a moderate performance decrease, indicating that agent-level supervision provides some but not dominant guidance in failure attribution.
80
AgentTracer
w/o Agent-level Reward
60
57.6
w/o Step-level Reward
48.2
40
35.5
33.2
27-7
20
18.9
18.9
Code
MATH
Agentic
Figure 6: Ablation results at the step level under w/o G. Removing step-level reward significantly decreases step-level accuracy, emphasizing the essential role of grainddi step-wise supervision for accurate error localization.
D.1 ANALYZER & PERTURBATION ANALYSIS
This section presents additional analysis of the analyzer agent Tanalyzer and the perturbation opera- tor II. We interpret "analyzer quality" as the capability of the backbone LLM used in Tanalyze- A stronger analyzer is expected to more reliably identify decisive errors during counterfactual correc- tion. To examine this effect, we evaluate multiple LLM backbones on 500 failed trajectories curated in Section 4.1 and report correction success rates in terms of pass @ 1-3, where pass@) denotes locating the decisive error in single pass. The results, summarized in Table 4, show that more capable analyzers consistently achieve higher success rates.
Table 4: Analyzer ablation across different analyzer LLM backbones.
Model
Pass@1 1
Pass Pass@2
DeepSeek- 13.1-Terminnn
54%
63%
68%
GPT-5-Mini
38%
43%
47%
Qwen3-Coder
46%
53%
58%
Qwen3-32B
26%
31%
39%
We further check the effect of the backbone LLM used in the perturbation operator II on program- matic fault injection. Using the same evaluation protocol, we evaluate several attacker backbones on 200 successful curated trajectories and report injection pass@1 1-3, where pass @ indicates that a single perturbed step successfully induces failure. Although the differences are milder than in the analyzer case, stronger attacker models still yield more effective perturbations. GPT-5-Miní excluded because its safety mechanisms systematically refuse adversarial perturbation prompts. Results summarized in Table 5.
20

---

Agen Tracer
Table 5: Ablation across attacker backbones used in the perturbation operator 1I (200 successful trajectories from Section 4.1).
Model
Pass@1
Pass@2
Pass@3
DeepSeek- Bateeekkk-..1111 -ermmnnss GPT-S-Mini
57%
63%
Qwen3-Coder
54%
58%
62%
Qwen3-32B
40%
45%
54%
D.2 TRAINING DATA ABLATION STUDY
To further substantiate the role of data generated through programmed fault injection, conduct an ablation analysis isolating its contribution relative to counterfactual correction data. Let D denote trajectories constructed via programmed fault injection and D denote trajectories derived from counterfactual correction. We compare three training configurations (using the full dataset, using only D and using only DD). Results both hand-crafted and automated variants of the Who&When benchmark are summarized in Table 6.
Table 6: Training data ablation study of AgenTracer. We opt for the "w/ 9 setting across the table.
Training Data
Who& When (handeraft) Agent-level
Who& When (automated) Agent-level
Step-level
Step-level
D and D
63.82
20.68
63.73
37.30
only D+
51.72
17.24
55.55
24.60
only D
58.62
17.24
60.31
34.12
These results yield key observations. First, counterfactual correction data exhibits higher intrin- sic value: models trained exclusively on 2 consistently outperform those trained solely on D+. Second, and equally important, both data sources contribute complementary signals. Removing D leads to measurable drop relative to training on the full dataset, indicating that fault-injected trajectories (despite being comparatively simpler) still provide useful supervision.
We emphasize that the relative ease of 'D+ is not necessarily drawback. Prior work in LLM RI. (e.g., the Perplexity technical report (Perplexity, 2025)) notes that blending data of varying difficulty can improve policy stability and overall learning efficiency. Our findings align with this perspective, suggesting that heterogeneous mixture error scenarios strengthens the model's robustness.
E FAILURE ATTRIBUTION EXAMPLE
Example of AgenTracer's failure attribution process
[response] <thinks>
Okay, let's Lzy to figure out where the mistake happened. The problem is about determining
interleaving of and using dynanic programming. previded code seens to be the correct approach, but maybe there's an error in the conversation sceps.
Looking at the corversation, the Team Leader assigned tasks to Alice, Bob, and Alex, Product Manager
the Architect designed the system, then Alex was supposed to implement the solution. flowever, in the conversation, there's step
actually writing sending the implementation task to Alex, but there's aubsaquent step where Alex provides the code.
mention
Mait, the vser provided the correct code in the iritial problen statement, Aut in the conversalion, steps The mistake might be that Alex didn't implement the code, but since the user's correct code, maybe the
Step22
Leader
answer
elsewhere.
Wait, Looking back, the user's answer is the correct code. sut the question is about the conversation steps. conversation steps show that the Team Leader sent the task to Alex, but there's no step where Alex actually
ifynni
21

---

writes the code, So maybe the mistake is that the code wasn't implemented, but the user's a answer provided code, Rowever, the probler here to idertify agent made an error in the conversation stepa leading to the wrong solution.
Alternatively, maybe the mistake is in the Architect's step where they
the system design. The system design meations a class diagram and sequeace disgraml but the function, not answer is function, which matches the PRD' requirement.
actual
Hovever, the provided
Mait, the code in the user's code but in the cenversa:ion steps. Let me check the steps again.
is corsect, perhaps the mistake isn't t in the
Tn Step17, the Architect wrote the aystem design, which includes a claas df dtagram with a method ia_interlesve.
actual funot not class. So maybe the Architact's system design was incorrect, leading to wrong implementat lon, However, the code provided by the user is function, which might natch the system design's class structure.
interleavingstrange
providad
the user's code correct, maybe the mistake in the code waa stiil writcen correctly. Alternativel maybe the miatake 1a in the Team Leader not following up Alex's implementation, Rosevar, since user's Inclides the correct code, perhapa the error i5 in the conversation stepa not showing Alex's implementation.
design, the
But the question is about the conversation steps leading to the wrong solution. since userfa
conversat ion steps don's Alex writing the code, maybe the mistake is that the code wasn't implemented, but the userfa answer the corract code. Fowaver, the problam to identify error the conversation scepa.
ALternatively, maybe the mistake is in the Architect's system. design. the system design mentions a class dingram with a class incerlaaiingstring. but the code is function,
Function correctly which matches the FRD. So naybe that's not an ecror.
is_interleave,
Rom, this is bit contusing. The usez's
correct, but the conversation stepa don't show Alex wrizing the code. Howevez, the question is about identifying an is
in the conversation steps. Since user's mistake la in the conversation ateps not showing the code being written. But the identity which agent
perhaps the
problem
the
steps.
ALternativelly maybe the mistake is in the Team Leader's a step where they aent the task to Alex, but there's a step where Alex actually writes the code. floevez, the user's a answer includes the code, so maybe the mistake is that the code wasn't implemented the conversation steps.
user's ansver but in the cenversazion steps not showing the implementacion. However, the question is about the conversation steps leading to the wzong solution.
correct code, so perhaps
code
The code initializes the dp table, checks the first rew and column, then sills the table, The condition for d3()) is (dp(i-1) and [1-i] 93(1-1-1] (dp( i11i-1] and #211-1] -- s3(1t)-1.). This seems correct.
wait, in the code, when checking the current character of 23, the index is itj-l. Since i+j characters, the choracter
a3(:i+]] correct.
i+1-1. That"
in the conversation scepa didn't show Alex implement it. flowever, the user' answer includes correcs code, perhaps mistake showing the implemestation. gut the question is about the conversation steps leading to wrong solut! on.
Therefore, maybe there'a
steps
I think the mistake to in the Architect's a system desigr. The system deaign mentions function,
diagram, but correct, so maybe that's not an ezror.
Hooever, the is
A Alternatively, maybe the mistake is in the iritial steps where the PRD was created, but the seems correct.
T'm a bit strck. let me ra-examine the conversation steps. The code provided by the show writing code. However, the user's answez is the correct code, so perhaps the mistake is in the conversation steps showing the conversation steps leading to the wrong solution.
correct,
steps
being written. sut question
22

---

alternatively, maybe the mistake is in the Architect's system design, which suggested claas-based approact, the code function, Roweve", the code correct, so that's not an error.
think the correct
is that the mistake is in the Architest's step 18tep17) wheze the system design was created, but the code is correct. llowever, since the user's answer correct, maybe there's no mistake. aut question requires identifying an error.
ALlernativeil maybe the mistake is in the Deam Leader's step where they aent the Lask to Alex, but there's ne step where Alex implemented the code. However, the user's includes the coda, perhops the mistake is the conversation. steps but in the code.
Wait, the user's coda la correct. maybe there's error in the conversation steps, but the question requires icentitying an error.
This is confusing. Maybe the mistake is in the Architect's system design, but I'm not Altarnatively, the mistake in the Leader not following up Alaxfa implementation.
But given the information, 11 have make best guess. The mistake might be in the Architect's syatem design, leading to wrong implementacion But since the code is
maybe
A ternativelyy
mistaka is in the initial steps ERC seems correct.
the
created, the
I think the correct answer is that the miatake 1a in the Architect's step 1Step1 but I'm not entirely sure. floweves, based on the conversation steps, the code wasn't implemented, but the user's
provided it. So perhaps the mistake is in the conversation steps not showing the cade, but the user's 5 answar is corract.
Given the confusion, I'll proceed with the Initial answec. </thinks
Agen: Name: Architect
Step Number: 17
Reason for Mistake: The Architact's system design incorreatly auggeated a claas-based approach
diagram aequence diagzam, which may confusicn or incorrect inplementation flowever, the actual solution requires cunetion-bacee approach, noт claas. Thia discrepancy could have caused misaligmment between the design and the final code, is correct. The mistake lies in the system design net accurately reflecting the required function-based implementation.
though the provided code
F MULTI-RUN RESULTS
This section reports multi-run evaluation results on the Whe& benchmark. In the main paper, all models were assessed under the official evaluation protocol of the benchmark, using temperature and top-k = decoding. Although prior work has shown that decoding with temperature typically exhibits low variance and produces highly stable outputs, independent runs for completeness. The following tables summarize the aggregated statistics (mean and standard deviation across three runs) for both the automated and handcrafted subsets, both the agent and step levels.
additionally conduct three
Table 7: Performance comparison on the Who& benchmark (w/o G). Each cell reports the accuracy + std, where the failure tracer has no access ground truth trajectory.
Who& (handcraft)
Who& When (automated)
Model
Agent-level
Step-level
Agent-Ievel
Step-level
QWEN3-8B
40.75 i 3.25
4.50 + 1.40
60.80
1.29
4.09 1 2.82
QWEN3-32B
46.90 1: 2.81
3.50 1: 1.40
57.10
3.06
6.88 1.87
QWEN3-CODER-PLUP
58.28 + 3.54
12.11:2.15
33.30 3.19
25.88 3.74
GPT-4.1
42.52 + 2.93
2.93
2.66 - 1.40
59.26 = 1.21
20.38 = 1.21
DEEPSEEK-R1-0528
57.28 + 5.07
10.32
65.87 5.99
26,18 + 2.10
GEMINI-2.5-PRO
52.87 2.63
4.59 2.63
58.20 2.42
24.14 +: 1.49
CLAUDE-SONNET-4
47.13 + 1.71
16.59 1.99
53.73 0.79
38,26 +: 0.46
AGENTRACER-SB
62.11 + 1.94
19.19 2.81
62.10 1.21
36.30 1.62
23


Reviews:
[{"review_id": "Zs8lSp2xOT", "reviewer": "Reviewer_1gdm", "summary": "This paper addresses the challenge of fragility in MAS by introducing (1) AgenTracer, an automated data generation technique for MAS failure attribution annotation datasets, (2) TracerTraj, a dataset of 2000 annotated trajectories with it, (3) RL-training an 8B model (AgenTracer-8B) to perform failure attribution given MAS execution trace. The authors demonstrate that AgenTracer-8B outperforms even larger proprietary models (like Gemini-2.5-Pro and Claude-4-Sonnet) on the Who&When attribution benchmark. Further, the authors demonstrate boosting the performance of MAS in-deployment using Self-refine and CRITIQUE with the AgenTracer-8B generated feedback.", "questions": "- Could the authors please elaborate on the limitations of the \"oracle guidance\" assumption? Are there ways this framework could be applied to more open-ended domains where a clear ground truth or oracle is not available? Is it possible, for example, to use a weaker, more readily available signal (e.g., final binary task success) to approximate the counterfactual replay? For example, in many domains, it is easier to verify if a task has been solved than to find the solution.\n- For the \"programmed fault injection\" component, could the authors provide more detail on the taxonomy of faults injected? How did they ensure these faults are representative of real-world failures in MAS, rather than just synthetic or easy-to-identify errors?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a highly relevant problem, of building models/systems to perform MAS error attribution\n- The paper closes the loop, It doesn't just present a diagnostic and error detection tool, but also demonstrates its practical value by demonstrating that using feedback from AgenTracer-8B, MAS can use Self-refine and CRITIQUE methods to achieve better performance.", "weaknesses": "- The primary weakness of the proposed framework is the AgenTracer pipeline's dependency on having access to ground truth / oracle guidance. The counterfactual replay method from the paper requires knowing the correct action at a given step to identify deviation as the decisive error.\n- The above weakness limits the general applicability of the data generation method to domains where ground truth is not accessible, which includes most complex and open ended domains where MAS are being deployed (for example, AI4Science, AI4Code, AI4Math, etc.). The paper needs to discuss how this method could be adapted to domains where such an oracle is unavailable.\n- This also potentially limits the applicability of the AgenTracer-8B model, as it has been trained with data facing the mentioned limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dmF89lKxGl", "reviewer": "Reviewer_oZRD", "summary": "This paper introduces AgenTracer, a system that automatically finds and explains why agent-based systems fail. It includes AgenTracer-8B, a smaller model that uses reinforcement learning and performs better than proprietary models. It also shows improvement on self-correcting and self-evolving agentic systems.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed automatic trajectory annotation framework appears to be both technically sound and conceptually novel. By systematically generating annotated failure trajectories, the authors establish a strong methodological foundation for analyzing and diagnosing agentic system behaviors. This approach could significantly improve the interpretability and reliability of complex multi-agent environments.\n\nThe performance of AgenTracer-8B is particularly impressive. The experiments demonstrate that this lightweight model achieves high diagnostic accuracy, outperforming even leading proprietary models. The use of multi-granular reinforcement learning is a well-motivated design choice that seems to balance efficiency and precision effectively.\n\nFinally, the empirical results showing that AgenTracer can enhance off-the-shelf agentic systems are especially promising.", "weaknesses": "This work proposed and combined different rewards for RL training, it would be goos to see ablation over different rewards", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iRNzdnwk2l", "reviewer": "Reviewer_VBFj", "summary": "This paper introduces AgenTracer, a framework for attributing failures in multi-agent LLM systems by identifying who (which agent) and when (which step) caused the decisive error. The core idea is to (i) generate labeled trajectories using counterfactual replay to locate the earliest decisive error and programmatic fault injection to synthesize failures from successes, producing a dataset (TracerTraj) with paired successful/failed runs; and (ii) train a specialized AgenTracer-8B model (Qwen-style backbone) with a multi-granular reward (agent-, step-, and format-level) using GRPO-like reinforcement learning. The intended scope is debugging and reliability for compound/agentic systems across coding, math, and general tasks. Reported outcomes include better Who&When attribution than strong general LLMs and modest multi-turn performance gains when AgenTracer is plugged into external agent frameworks.", "questions": "* How sensitive are results to reward weights, the dynamic clipping schedule, and the removal of KL? Please include a small grid or one-at-a-time sweeps.\n* How does performance change with weaker/stronger fault injection and with different analyzer qualities during counterfactual replay? An ablation here would help establish causal contributions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The decisive-error notion plus automated curation that combines counterfactual replay with programmatic fault injection is a useful addition to the toolset for diagnosing multi-agent failures, distinct from prior taxonomies that stop at coarse failure categories.\n* Training a compact attributor model with a multi-granular reward is practically valuable for plugging into agent stacks; the engineering is thoughtful and likely to see real use.\n* The plug-in demonstration is nice to see, and the integration studies show that better attribution can translate into gains in multi-turn settings, aligning with the intended way of using AgentTracer.", "weaknesses": "* The main tables appear to be single-run point estimates with no mean±std, confidence intervals, or hypothesis tests. The number of seeds is not stated. Without dispersion and tests (ideally paired where appropriate), it’s impossible to tell whether reported gains are robust or just noise.\n* The paper introduces several design choices (multi-granular reward, dynamic clipping, removal of KL, analyzer quality, injection strength), but sensitivity is not reported. Without targeted ablations and sweeps, we can’t separate genuine attribution skill from simpler effects (e.g., a format reward just enforcing response templates).\n* Much of the training signal comes from injected errors. There is no out-of-distribution evaluation on naturally occurring failures (e.g., tool/API drift, latency-induced desync, memory/context loss) or on different agent graphs/toolchains. As a result, it’s unclear how well the attributor generalizes beyond its own synthesis process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8hNGLOJogm", "reviewer": "Reviewer_uaBz", "summary": "This work features two contributions for failure attribution of multi-agent systems: 1. An automated pipeline, AgenTracer, that can annotate or create failure trajectories with two strategies (counterfactual replay and fault injection). 2. A model, AgenTracer-8B, which is trained with reinforcement learning upon the curated training data. Experiments on Who&When dataset demonstrate superior or competitive failure attribution performance compared with a range of open-source or proprietary models. It is also shown that applying the feedback from AgenTracer-8B can consistently benefit the self-improvement of agentic systems.", "questions": "Please see Weaknesses 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The two-fold strategy for annotating agent trajectories, i.e., counterfactual replay for identifying failures from failed trajectories and fault injection for injecting failures into successful trajectories, is sound and scalable.\n2. The experiments in Section 5.3, i.e., directly leveraging the failure attributor for self-improvement of MAS, are informative and valuable proxies that demonstrate real-world applicability.", "weaknesses": "1. The only complaint is that in Table 1, only one dataset (Who&When) is evaluated. Having at least one more benchmarks would make the evaluation more comprehensive.\n2. The reasoning or necessity behind the linear interpolation (via $\\lambda$) between the step-level and the agent-level reward in Equation (11) is not discussed or explained. Does it mean that there is an inherent trade-off between the step-level and agent-level failure attribution?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)