CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems

## Abstract

Abstract Multi-agent systems (MAS) are increasingly capable of tackling complex real-world tasks, yet their reliance on inter-agent coordination, tool use, and long-horizon reasoning makes error recognition particularly challenging. Minor errors can propagate across agents, escalating into task failures while producing long, intertwined execution trajectories that impose significant costs for both human developers and automated systems to debug and analyze. Our key insight is that, despite surface differences in failure trajectories (e.g., logs), MAS errors often recur with similar structural patterns. This paper presents CORRECT , the first lightweight, training-free framework that leverages an online cache of distilled error schemata to recognize and transfer knowledge of failure structures across new requests. This cache-based reuse allows LLMs to perform targeted error localization at inference time, avoiding the need for expensive retraining while adapting to dynamic MAS deployments in subseconds. To support rigorous study in this domain, we also introduce CORRECT -Error, a large-scale dataset of over 2,000 annotated trajectories collected through a novel error-injection pipeline guided by real-world distributions, and further validated through human evaluation to ensure alignment with natural failure patterns. Experiments across seven diverse MAS applications show that CORRECT improves step-level error localization up to 19.8% over existing advances while at near-zero overhead, substantially narrowing the gap between automated and human-level error recognition.

## 1 Introduction

Multi-agent systems (MAS) have emerged as a powerful paradigm for solving complex tasks that require diverse capabilities and collaborative problem-solving, with demonstrated success in various domains, including software development (Qian et al., 2023; Hong et al., 2023; Zhang et al., 2024), scientific research (Lu et al., 2024), web navigation (Zhou et al., 2023), and general-purpose task automation (Wu et al., 2024; Wang et al., 2025). By orchestrating multiple specialized agents, MAS can tackle challenges beyond the reach of single-agent systems (SAS) that rely on single LLMs for task solving, achieving human-level performance on benchmarks like GAIA (Mialon et al., 2023) and SWE-bench (Jimenez et al., 2023).

However, as MAS increasingly scale in both complexity (e.g., sophisticated interactions with other agents and tools) and deployments, decisive error recognition in MAS deployments becomes exceptionally challenging. Unlike SAS where failures can be traced to a single faulty output, MAS failures often emerge from cascading effects across agents: a single error-prone step by one agent can propagate through downstream interactions, ultimately causing task failure (Gao et al., 2025). We refer to this fundamental challenge as *decisive error recognition*: pinpointing the precise agent and step that first triggered the failure (Zhang et al., 2025). Efficient decisive error recognition is essential for reliable MAS deployments, sustaining service quality, and guiding operational management such as safe agent upgrades, targeted restarts, and service monitoring (Epperson et al., 2025).

Unfortunately, decisive error recognition in MAS remains open-ended due to three fundamental obstacles: (1) *Generality*: MAS span a wide spectrum of applications, and even within an application, requests can exhibit drastically different error patterns, making it difficult to design methods that generalize. Existing advances (Zhang et al., 2025) often resort to LLM-as-a-judge methods for generality, yet achieve $\leq$10% accuracy in pinpointing the exact error step, barely above the accuracy of random guessing (3%). Recent efforts (Ge et al., 2025) to improve accuracy through fine-tuning not only hurt generalization, but fall short in (2) *Data Efficiency*: obtaining labeled data in error recognition is notoriously expensive and inherently ambiguous. For example, in the Who&When dataset (Zhang et al., 2025), annotators spent over 30 expert hours labeling fewer than 200 trajectories, yet disagreement rates exceeded 50%. This makes any training-based approaches ineffective without voluminous data; and (3) *Computation Efficiency*: even with sufficient data, tuning LLMs for error recognition may be impractical to catch up with deployments where new error types arise continuously and sporadically, e.g., new attacks in cloud AIOps (Wang et al., 2025).

In this paper, we first notice that failures in MAS tend to recur with similar structures across requests (§[2.2](https://arxiv.org/html/2509.24088v1#S2.SS2)).
Because a MAS application often relies on the same role specifications, orchestration rules, tool APIs, and verification policies, diverse requests often funnel into common decision skeletons.
Our real-world analysis of Who&When dataset proposed in Zhang et al. (2025) shows that over 80% of failure trajectories have at least one counterpart with $\geq$0.8 semantic similarity in error logs. This suggests that error knowledge can be systematically *distilled, cached, and reused*.
However, naive approaches, such as in-context learning (ICL) that insert prior trajectories as in-context exemplars, quickly break down: logs can exceed 32,000 tokens (Yang et al., 2025; Dubey et al., 2024), contain low-entropy noise, and even underperform zero-shot baselines (§[2.2](https://arxiv.org/html/2509.24088v1#S2.SS2)).
These challenges demand a new approach: one that is general, data-efficient, and lightweight for practical deployments.

In this paper, we propose *CORRECT* (COndensed eRror RECognition via knowledge Transfer), a novel framework that formalizes decisive error recognition as a first-class problem for reliable MAS, and automatically distills prior error patterns into compact, reusable schemas that capture their essential signatures, triggering contexts, and propagation patterns.
During inference, *CORRECT* identifies relevant schemas and applies them to the new request, enabling accurate, lightweight, and training-free recognition in dynamic environments. Our contributions are summarized as follows:

- •
CORRECT: the first schema-guided detector with broad generality.
We introduce *CORRECT*, the first framework that distills recurrent MAS failures into compact error schemata and reuses them for decisive error recognition. Unlike LLM-as-a-judge approaches that trade accuracy for generality, or costly fine-tuning approaches, *CORRECT* leverages schemata to transfer knowledge across requests in real time. This design improves step-level localization accuracy up to 20 points over state-of-the-art designs (Zhang et al., 2025), while remaining training-free and computationally efficient, making it practical for deployment in dynamic MAS environments.
- •
*CORRECT*-Error: a versatile, large-scale, and high-fidelity dataset for MAS error recognition. To close the benchmarking gap, we construct *CORRECT*-Error, a large-scale collection of over 2,000 multi-agent trajectories with fine-grained, step-level error annotations. Unlike prior datasets that are small and noisy, *CORRECT*-Error is built using a novel error-injection pipeline guided by real-world failure distributions, yielding trajectories that balance controlled coverage with the realism of natural MAS failures. We conduct extensive human validation, confirming strong alignment between synthetic annotations and expert judgment. *CORRECT*-Error not only enables rigorous evaluation of *CORRECT*, but also establishes a reusable, extensible benchmark for the community to advance the development of MAS error recognition methods.

Together, these contributions advance the state of the art in MAS error recognition and establish a new foundation and datasets for making MAS more reliable, interpretable, and deployable at scale.

## 2 Background and Motivation

### 2.1 Decisive Error in Multi-Agent Systems

Task failures in MAS often arise from specific *decisive errors* that, once committed, render successful task completion impossible. We formalize this notion following prior advances (Zhang et al., 2025).
Consider a MAS executing a trajectory $\tau=\{(a_{1},s_{1}),(a_{2},s_{2}),\dots,(a_{T},s_{T})\}$, where agent $a_{i}$ performs step $s_{i}$. The outcome of the trajectory is denoted by $\mathcal{R}(\tau)\in\{0,1\}$, with $1$ for success and 0 for failure.
A step $(a_{k},s_{k})$ in a failed trajectory $\tau$ is a decisive error if replacing it with a correct alternative $\tilde{s}_{k}$ would change the outcome to success. Formally, the earliest decisive error is:
$(a^{*},s^{*})=\min_{k\in\mathcal{D}(\tau)}k$,
where $\mathcal{D}(\tau)=\{\,k:\mathcal{R}(\tau)=0\wedge\mathcal{R}(\tau_{[s_{k}\rightarrow\tilde{s}_{k}]})=1\,\}$, and $\tau_{[s_{k}\rightarrow\tilde{s}_{k}]}$ denotes the modified trajectory in which step $s_{k}$ is replaced by $\tilde{s}_{k}$.

Intuitively, a decisive error is the earliest step whose correction flips the trajectory outcome from failure to success. Identifying decisive errors is fundamental to reliable MAS. Unlike coarse-grained error recognition, which only flags failed trajectories, decisive error recognition pinpoints the exact *agent* and *step* responsible for initiating failure. This precision enables targeted interventions such as tuning role specifications, refining orchestration logic, or upgrading individual agents, without costly overhauls of the entire system.

### 2.2 Motivations of CORRECT

Automated decisive error recognition in MAS has primarily followed two directions: LLM-as-a-judge approaches (Zheng et al., 2023; Zhang et al., 2025) and fine-tuning specialized LLMs (Cemri et al., 2025). Both exhibit fundamental limitations in accuracy, generality, and efficiency.

Figure: Figure 1: MAS failure traces are complex and long, often exceeding the model capacity.
Refer to caption: x1.png

#### Limitations of Existing Advances.

LLM-as-a-judge methods were initially designed to rate the quality of LLM outputs, achieving up to 80% agreement with human preferences (Zheng et al., 2023). Zhang et al. (2025) extended this paradigm to MAS error attribution, introducing three variants: (i) all-at-once, which provides the full error log to the LLM and asks it to identify the responsible agent and error step; (ii) step-by-step, which incrementally reveals the trajectory and checks errors at each step; and (iii) binary search, which recursively partitions the log to localize the error. However, as MAS becomes more complex, which elongates the failure trajectory, these methods lose diagnostic precision (Figure [3](https://arxiv.org/html/2509.24088v1#S2.F3)): on the Who&When dataset (Zhang et al., 2025), Qwen-2.5-7B achieves only 3.5% step-level accuracy, far below practical deployment requirements.

Fine-tuning (FT)-based approaches, whether supervised or reinforcement learning–based, face substantial efficiency and expense challenges. Their success hinges on large-scale, high-quality labeled datasets. Yet annotating MAS failures is prohibitively expensive: annotators must disentangle long, interdependent interactions across agents and tools, taking 30 expert hours for annotating fewer than 200 trajectories. Worse, error trajectories vary across applications and requests, making FT-trained detectors brittle and undermining generalization for diverse deployments at scale.

#### Pervasive Error Similarity Yet Hard to Reuse.

Despite these challenges, our analysis of the Who&When dataset (Figure [3](https://arxiv.org/html/2509.24088v1#S2.F3)) reveals that more than 80% of failed requests share a semantic (cosine) similarity above 0.8, measured via BERT-based embeddings.
This reveals an underexploited opportunity: historical failures could be reused as exemplars for new requests.
A natural attempt is to adopt in-context learning (ICL), retrieving and appending similar trajectories to guide error recognition. However, our experiments (Figure [3](https://arxiv.org/html/2509.24088v1#S2.F3)) show that such a strawman approach even degrades recognition accuracy, due to two primary limitations: (i) extreme trajectory length: 17% of trajectories exceed 32K tokens, surpassing the context length of most LLMs (e.g., Qwen3 (Yang et al., 2025)); and (ii) low signal-to-noise ratio: execution trajectories interleave request-specific details and tool outputs with the true error-inducing steps, diluting critical information.

## 3 CORRECT: Condensed Error Recognition via Knowledge Transfer

Our observations call for a novel approach that can *systematically reuse structural error knowledge* without overwhelming context or succumbing to noise, while remaining general, data-efficient, and lightweight for real-time MAS deployment. To these ends,
we introduce *CORRECT* (COndensed eRror RECognition via knowledge Transfer), the first framework that distills past errors into compact error schemas and adaptively applies them for accurate decisive error recognition without any training, enabling efficient adaptation to diverse tasks and errors. As summarized in Algorithm [1](https://arxiv.org/html/2509.24088v1#algorithm1), CORRECT combines three interconnected phases: (1) *Offline schema extraction*, (2) *Online schema-guided error recognition*, and (3) *Dynamic schema management*.

### 3.1 Error Schema Extraction

Given an annotated error trajectory $\mathcal{T}=\{(a_{i},s_{i},r_{i})\}_{i=1}^{n}$, where $a_{i}$ denotes the agent at step $i$, $s_{i}$ represents the step content, and $r_{i}$ is the corresponding result, along with the identified error at step $s_{e}$ and error reason $r_{e}$, *CORRECT* generates an error schema $\mathcal{S}$ capturing (Figure [4](https://arxiv.org/html/2509.24088v1#S3.F4)): (1) *Error Signatures* $\Sigma$: Characteristic patterns such as agent actions, interaction sequences, and key behavioral markers; (2) *Error Context Analysis* $\mathcal{C}$: Detailed analysis of the conditions that led to the error, including agent states, task progress, and environmental factors; and (3) *Detection Heuristics* $\mathcal{H}$: Actionable rules and guidelines for identifying similar errors in new contexts.

To minimize human efforts, *CORRECT* leverages capable LLMs (e.g., GPT-5) to generate error schemas. We discuss how to ensure the quality of the schema in Section [3.2](https://arxiv.org/html/2509.24088v1#S3.SS2).

Figure: Figure 4: Example of an error schema generated on the Who&When dataset.
Refer to caption: x4.png

#### Clustered Schema Extraction.

Even with LLMs, generating a schema for every trajectory is still costly due to voluminous requests in practical MAS deployments. In fact, doing so is unnecessary because schema reuse often follows a long-tailed distribution: a small number of schemas are frequently reused, while most are rarely applied (§[5](https://arxiv.org/html/2509.24088v1#S5)). To exploit this, *CORRECT* performs the following offline procedure: (1) *Trajectory Clustering:* Failure trajectories are embedded semantically and clustered to group similar error patterns, and then (2) *Cluster-level Schema Generation:* One representative schema is generated per cluster, capturing the common error structure without redundant costs.

### 3.2 Schema-Guided Error Recognition Online

Once a failure request requires diagnosis, we start with its trajectory $\mathcal{T}_{\text{target}}$ and retrieve the top-k relevant schemas from the cache via semantic similarity search (e.g., cosine similarity of embeddings):

$$ $\text{sim}(\mathcal{T}_{\text{target}},\mathcal{S}_{i})=\cos(\text{embed}(\mathcal{T}_{\text{target}}),\text{embed}(\mathcal{S}_{i}))$ $$

With retrieved schemas, we prompt the LLM to perform error recognition by instantiating the schemas in the context of the target trajectory. These schemas, together with the trajectory and lightweight adaptation instructions, are passed to an LLM for diagnosis. Formally, the LLM receives $(\mathcal{T}_{\text{target}},\{\mathcal{S}_{j}\}_{j=1}^{k},\text{prompt}_{\text{detect}})$ as input prompt and produces the error recognition result:

$$ $\text{result}=\text{LLM}_{\text{detect}}\big(\mathcal{T}_{\text{target}},\{\mathcal{S}_{j}\}_{j=1}^{k},\text{prompt}_{\text{detect}}\big).$ $$

By leveraging schemas as condensed expert knowledge, this process guides the LLM to focus on salient failure patterns without the overhead of processing entire historical trajectories.

#### Adaptation with Schema Expansion and Distillation.

*CORRECT* maintains an effective schema cache through two complementary mechanisms.
First, *schema expansion*: when user feedback confirms successful recognition, *CORRECT* leverages the ground truth label from the user to generate and cache a new error schema following [3.1](https://arxiv.org/html/2509.24088v1#S3.SS1). Priority is given to trajectories with low similarity to existing schemas (i.e., $\text{sim}(\mathcal{T}_{\text{new}},\mathcal{S}_{i})<\delta$ for all cached schemas), ensuring the cache captures diverse error patterns rather than redundant ones.
Second, *schema distillation*: expansion alone may yield suboptimal quality, and frequently accessed error schemas (cache hits $>\theta_{\text{hot}}$) may benefit from further refinement. In such cases, *CORRECT* generates multiple candidate schemas and replays them against prior trajectories to select the discriminative one with the highest accuracy.

Together, expansion ensures coverage for novel errors online, while distillation preserves cache efficiency by retaining only high-quality, discriminative schemas.

Figure: Algorithm 1 CORRECT Framework

## 4 CORRECT -Error: A Large-Scale Error Detection Benchmark

Existing efforts for trajectory-level error analysis are limited in both scale and diversity, and human annotation is costly and difficult to scale (§[2.2](https://arxiv.org/html/2509.24088v1#S2.SS2)). To bridge this gap and evaluate the effectiveness of *CORRECT* (§[3](https://arxiv.org/html/2509.24088v1#S3)), we introduce *CORRECT*-Error, a large-scale benchmark that faithfully reflects the distribution of natural errors encountered in real-world MAS.

### 4.1 Bootstrap Error Synthesis Pipeline

In building *CORRECT*-Error, we develop a bootstrap methodology that uses a small set of human-annotated error trajectories as seeds for scalable error generation, blending realism with controllability. It follows a three-stage pipeline that distills human expertise into scalable synthetic data while preserving the structural and semantic integrity of real-world error patterns.

#### Stage 1: Diverse Trajectory Collection.

We begin by generating a large pool of successful multi-agent trajectories spanning heterogeneous tasks and domains. Alongside, we curate a smaller but high-quality set of human-annotated error trajectories. These serve as reference exemplars of realistic failure dynamics, capturing both localized mistakes and their downstream propagation.

#### Stage 2: Semantic Error Schema Matching.

Each successful trajectory is paired with its closest human-labeled error trajectory using semantic similarity measures that account for both high-level task goals and fine-grained agent interactions. This alignment ensures that the selected error schema is contextually aligned with the target trajectory, avoiding unrealistic mismatches. Then we use GPT-5 to devise an error injection strategy that specifies (i) where in the target trajectory to introduce the error, and (ii) how to adapt the error pattern while preserving its core semantics.

#### Stage 3: Contextual Error Injection.

Following the injection strategy, we prompt GPT-5 that generated the original successful trajectory to introduce an erroneous action at the designated point. This guarantees consistency in linguistic and behavioral style while embedding a realistic failure.

### 4.2 Human-Alignment Analysis

Following our bootstrap pipeline (§[4.1](https://arxiv.org/html/2509.24088v1#S4.SS1)), we synthesized over 2,000 trajectories across seven datasets (Figure [5](https://arxiv.org/html/2509.24088v1#S4.F5)), yielding 12.3$\times$ more data than Who&When with cost over 3 billion tokens using GPT-5 series models and GPT-4o series models based on Magnetic-One (Fourney et al., 2024) and AutoGen (Wu et al., 2024). The resulting benchmark spans diverse tasks, including multi-hop QA, common planning, mathematical reasoning, and scientific problem-solving. By leveraging limited

Figure: Figure 5: *CORRECT*-Error includes diverse tasks. The synthesized data preserves high realism, where human labelers frequently misclassified synthetic errors as genuine ones.
Refer to caption: x5.png

human annotations as seeds, our novel pipeline generates diverse error scenarios at scale.

To rigorously assess this authenticity, we conducted a human evaluation study involving three independent expert labelers over 120 hours. Each labeler was shown an equal mix of synthetic and human-labeled error trajectories, without disclosure of their origin. As shown in Figure [5](https://arxiv.org/html/2509.24088v1#S4.F5), labelers struggled to distinguish between the two: 47.1% of synthetic trajectories were misclassified as human-labeled, while 42.3% of genuine trajectories were correctly identified. This near-random classification accuracy (close to 50% in both cases) indicates that our synthetic errors are effectively indistinguishable from real ones. Moreover, we notice that a high inter-annotator agreement on authenticity: 94.4% of synthetic trajectories were judged as genuine by at least two out of three labelers, with 52.9% receiving unanimous consensus. We add detailed human-alignment analysis in Appendix [A.2](https://arxiv.org/html/2509.24088v1#A1.SS2).

Together, these results validate that our error injection pipeline faithfully captures the nuanced characteristics of real-world MAS failures. We emphasize that this methodology enables data generation that is cheap (automated and low-cost), abundant (scalable to millions of examples), precise (with unambiguous ground-truth labels), and realistic (closely matching natural error distributions derived from human annotations), building the foundation for training effective recognition models.

## 5 Experiments

We demonstrate that *CORRECT* significantly enhances error recognition, achieving improvements of up to 20% and an average gain of 28.7% across MAS tasks (§[5.1](https://arxiv.org/html/2509.24088v1#S5.SS1)), all at zero training costs. *CORRECT* remains robust under distribution shifts arising from model updates, dataset variations, schema cache size, and the number of stored schemata (§[5.2](https://arxiv.org/html/2509.24088v1#S5.SS2)).

#### Models and Tasks.

We evaluate *CORRECT* on both the human-annotated Who&When benchmark and our high-quality benchmark, Correct-Error, spans diverse tasks including multi-hop QA (HotpotQA (Yang et al., 2018), Musique (Trivedi et al., 2022), WikiMQA (Ho et al., 2020)), scientific reasoning (ARC (Clark et al., 2018), MMLU-Pro (Wang et al., 2024)), mathematical reasoning (Math500 (Lightman et al., 2023)), and general agentic tasks including planning (GAIA (Mialon et al., 2023)).
Who&When consists of two subsets: a Human-Crafted subset and an Algorithm-Generated subset. Experiments are conducted on both open- and closed-source models, including the Qwen (Yang et al., 2024; 2025), Llama (Dubey et al., 2024), GPT (Hurst et al., 2024; OpenAI, 2025), DeepSeek-R1 (Guo et al., 2025), and Gemini series (Comanici et al., 2025). We mask each trajectory itself and avoid receiving its own error schema for preventing data leakage. Additional experimental details are provided in Appendix [A.1](https://arxiv.org/html/2509.24088v1#A1.SS1).

#### Baselines.

We compare *CORRECT* against three state-of-the-art approaches:

- •
*LLM-as-a-Judge*, a zero-shot prompting strategy where an LLM directly inspects trajectories without auxiliary guidance (Zhang et al., 2025; Peng et al., 2023a);
- •
*Fine-tuning*, where an LLM is trained on the full trajectory dataset with cross-entropy loss to encode domain-specific failure patterns (Chen et al., 2025; Fu et al., 2025); and
- •
*Naive In-Context Learning*, which inserts complete error trajectories as few-shot exemplars, which may be constrained by limited context windows (Dong et al., 2022; Yu et al., 2025).

#### Metrics.

Following existing advances (Zhang et al., 2025), we report *step-level accuracy*, which provides actionable debugging signals. To account for the ambiguity of error attribution, we additionally report accuracy@k, where predictions within $k$ steps of the ground truth are treated as correct, e.g., Acc@0 requires identifying the exact erroneous step, while Acc@1 tolerates an offset of one step. This better reflects practical debugging scenarios, where approximate localization is often sufficient. We report the average performance over five independent runs.

### 5.1 End-to-End Performance

**Table 1: *CORRECT* achieves higher error recognition accuracy over existing advances on Who&When dataset. Acc@0 denotes exact-step accuracy (the model must pinpoint the precise error step). Acc@k denotes tolerant accuracy (a prediction is correct if it falls within $\pm k$ steps of the ground truth). For rows corresponding to *CORRECT*, numbers in parentheses indicate absolute improvements over the LLM-as-a-Judge baseline.**
| Method | Model | Human-Crafted | Algorithm-Generated |  |  |
| --- | --- | --- | --- | --- | --- |
| Acc@0 | Acc@1 | Acc@0 | Acc@1 |  |  |
| LLM-as-a-Judge | Qwen-2.5-7b | 3.5 | 8.6 | 19.1 | 42.9 |
| Qwen-3-30b | 1.7 | 5.2 | 15.1 | 42.9 |  |
| Qwen-3-80b | 6.9 | 8.6 | 21.4 | 47.6 |  |
| Llama-8b | 1.7 | 3.5 | 3.2 | 15.9 |  |
| DeepSeek-R1 | 3.5 | 17.2 | 23.8 | 54.0 |  |
| Gemini-2.5-flash | 5.2 | 13.8 | 31.8 | 56.0 |  |
| Gemini-2.5-pro | 3.5 | 8.6 | 25.4 | 50.0 |  |
| GPT-4o-mini | 3.5 | 12.5 | 12.7 | 39.7 |  |
| GPT-4o | 3.5 | 10.3 | 18.3 | 50.0 |  |
| GPT-5-nano | 1.7 | 5.2 | 19.1 | 41.3 |  |
|  | GPT-5 | 8.6 | 24.1 | 18.3 | 56.4 |
| Fine-tuned LLM | Qwen-2.5-7b | 3.5 | 11.9 | 18.9 | 42.9 |
| Naive ICL | Qwen-2.5-7b | 5.2 | 10.3 | 15.9 | 40.5 |
| *CORRECT* | Qwen-2.5-7b | 12.1 (+8.6) | 15.5 (+6.9) | 19.8 (+0.7) | 46.8 (+3.9) |
| Gemini-2.5-flash | 10.3 (+5.1) | 20.7 (+6.9) | 38.9 (+7.1) | 55.2 (-0.8) |  |
| Gemini-2.5-pro | 5.2 (+1.7) | 15.5 (+6.9) | 24.6 (-0.8) | 52.4 (+2.4) |  |
| GPT-5-nano | 6.9 (+5.2) | 17.2 (+12.0) | 24.6 (+5.5) | 44.4 (+3.1) |  |
| GPT-5 | 17.2 (+8.6) | 37.8 (+13.7) | 38.1 (+19.8) | 58.8 (+2.4) |  |

#### CORRECT achieves significant gains in error recognition accuracy ( Who&When dataset).

Table [1](https://arxiv.org/html/2509.24088v1#S5.T1) shows that *CORRECT* consistently surpasses existing advances across both human-crafted and algorithm-generated subsets.
On human-crafted data, *CORRECT* raises Qwen-2.5-7B’s exact-step accuracy from 3.5% to 12.1% (a 3.5$\times$ improvement), and improves GPT-5 from 8.6% to 17.2%. These gains extend to tolerant metrics as well, with GPT-5 + *CORRECT* achieving 37.8% at Acc@1 versus 24.1% for the baseline. By contrast, fine-tuning (3.5%) and naive ICL (5.2%) offer only marginal improvements, suggesting that standard supervised learning and raw in-context trajectories fail to capture complex error patterns.
On algorithm-generated data, *CORRECT* maintains clear advantages, with Gemini-2.5-Flash improving from 31.8% to 38.9% (+7.1 points) and GPT-5 exhibiting the largest gain (+19.8 points).
These consistent improvements across model families (Qwen, Gemini, GPT) and scales (7B to GPT-5) highlight the effectiveness of condensed error schemas.

**Table 2: Performance comparison across multiple datasets. All numbers are accuracy (%).**
| Method | Tolerance | Dataset | Avg.<br>Improv. |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gaia | HotpotQA | Musique | WikiMQA | Arc | Math500 | MMLU-Pro |  |  |  |
| Synthesized by GPT-4o-mini |  |  |  |  |  |  |  |  |  |
| LLM-as-a-Judge | Acc@1 | 28.6 | 34.8 | 27.8 | 14.7 | 64.0 | 10.2 | 58.3 | - |
| Acc@3 | 42.9 | 59.4 | 77.8 | 55.9 | 75.0 | 23.4 | 62.5 | - |  |
| Acc@5 | 50.0 | 63.8 | 77.8 | 64.7 | 78.0 | 35.6 | 66.7 | - |  |
| CORRECT | Acc@1 | 28.6 | 60.9 | 38.9 | 44.1 | 80.4 | 57.1 | 69.1 | +20.1 |
| Acc@3 | 50.0 | 94.2 | 88.9 | 88.2 | 88.2 | 87.8 | 88.2 | +27.6 |  |
| Acc@5 | 64.3 | 95.7 | 88.9 | 94.1 | 91.2 | 95.9 | 94.1 | +28.7 |  |
| Synthesized by GPT-5-Nano |  |  |  |  |  |  |  |  |  |
| Baseline | Acc@1 | 16.7 | 14.7 | 11.9 | 6.44 | 62.8 | 41.8 | 50.0 | - |
| Acc@3 | 27.8 | 48.9 | 43.9 | 35.6 | 69.6 | 57.1 | 64.7 | - |  |
| Acc@5 | 38.9 | 65.8 | 58.8 | 56.1 | 71.1 | 64.3 | 70.59 | - |  |
| CORRECT | Acc@1 | 30.6 | 35.8 | 32.7 | 16.9 | 80.4 | 57.1 | 69.1 | +16.8 |
| Acc@3 | 44.4 | 72.7 | 61.2 | 49.9 | 88.2 | 87.8 | 88.2 | +20.1 |  |
| Acc@5 | 52.8 | 84.3 | 77.2 | 69.0 | 91.2 | 95.9 | 94.1 | +18.9 |  |

CORRECT delivers 17–28% average improvements (Correct-Error benchmark).
Table [2](https://arxiv.org/html/2509.24088v1#S5.T2) highlights *CORRECT*’s strong generalization across seven datasets. For GPT-4o-mini subset, *CORRECT* improves average accuracy by +20.1%, +27.6%, and +28.7% at Acc@1, Acc@3, and Acc@5, respectively using Qwen-2.5-7b. Gains are especially pronounced on knowledge-intensive tasks: HotpotQA (+26.1 points), WikiMQA (+29.4 points), and Math500 (+46.9 points). At higher tolerances, performance gaps widen further: *CORRECT* reaches 94.2%, 91.2%, and 95.9% at Acc@5, compared to baseline scores of 63.8%, 78.0%, and 35.6%. *CORRECT* exhibits a similar trend in GPT-5-nano subset, with average improvements of +16.8%, +20.1%, and +18.9% across tolerance levels using Qwen-2.5-7b. Even on the challenging GAIA benchmark, *CORRECT* raises accuracy from 28.6% to 30.6%, while achieving near-perfect scores on several datasets at Acc@5.

#### Strong Schema Transferability across Datasets and Models.

Figure [8](https://arxiv.org/html/2509.24088v1#S5.F8) shows that schemas distilled from human-crafted trajectories transfer effectively to algorithm-generated data. Across GPT-5-nano, Gemini-2.5-Flash, and Qwen-7B, *CORRECT* with transferred schemas consistently outperforms baselines, with Gemini-2.5-Flash improving from 31.8% to 36.5%. This cross-domain transferability indicates that distilled schemas capture fundamental error patterns.

Moreover, Figure [8](https://arxiv.org/html/2509.24088v1#S5.F8) demonstrates that *CORRECT* benefits directly from model upgrades: using GPT-5 instead of Qwen-72B as the schema generator raises detection accuracy from 8.6% to 12.2% for GPT-5-nano and from 10.3% to 12.2% for Qwen-2.5-7B. These adaptive gains confirm that better models yield higher-quality schemas that immediately enhance downstream performance. Together, these results establish *CORRECT* as a flexible, upgrade-compatible framework that leverages both existing schema libraries and future model advances without architectural changes.

Figure: Figure 6: *CORRECT* delivers consistent improvements on Algorithm-Generated, with schemata generated and transferred on Hand-Crafted dataset.
Refer to caption: x6.png

### 5.2 Ablation Studies

#### Impact of Schema Repository Size.

Figure [8](https://arxiv.org/html/2509.24088v1#S5.F8) shows *CORRECT*’s robustness to cold-start scenarios and varying cache sizes. On Correct-Error, even with only 10% of the schema library, *CORRECT* achieves 69.1% and 35.4% Acc@1 on MMLU_pro and HotpotQA, substantially outperforming baselines. Performance improves steadily with larger caches but plateaus beyond 50% (tens of schemas), suggesting that a relatively small set of diverse schemas captures most common error patterns. This logarithmic growth pattern validates our clustering-based extraction strategy: error patterns are finite and reusable across trajectories. Importantly, *CORRECT* retains $\sim$90% of peak performance with only 30% of the cache, highlighting efficiency for practical deployments.

#### Impact of Number of Schemas in Online Error Recognition.

Figure [11](https://arxiv.org/html/2509.24088v1#S5.F11) demonstrates that retrieving and using a single schema already greatly improves baseline Acc@1 (12.1% vs. 8.6%). Accuracy increases as more schemas are added (13.8% with 5 schemas, 15.5% with 10), though gains diminish (1$\rightarrow$5 adds +1.7%; 5$\rightarrow$10 adds only +1.7%). At higher tolerances (Acc@5), settings converge to $\sim$48%. These results show that a small set of well-matched schemas already efficiently captures most critical patterns, while additional schemas offer limited incremental value.

#### Comparison with Oracle Error Schema.

To estimate the upper bound of our framework, we compare *CORRECT* against an oracle configuration where each trajectory uses its own ground-truth schema. As shown in Figure [11](https://arxiv.org/html/2509.24088v1#S5.F11), the oracle achieves superior step-level accuracy, while *CORRECT* with 5 retrieved schemas reaches 71.5%—104% of oracle performance. This narrow gap shows that our semantic retrieval strategy effectively identifies schemas encoding near-equivalent knowledge to trajectory-specific patterns. The convergence toward oracle performance validates that diverse MAS error patterns often share structural regularities, which can be captured via a finite schema library.

Figure: Figure 9: Performance of Correct with different number of error schemata on Handcrafted subset of Who&When.
Refer to caption: x9.png

#### LLMs can’t Recognize Errors During Execution

Figure [11](https://arxiv.org/html/2509.24088v1#S5.F11) shows that LLMs have limited metacognitive ability to detect their own errors during execution. When asked to identify injected errors at the exact step, they achieve only 21% accuracy on *flipped* trajectories (where errors alter the final answer) and 17–18% on *non-flipped* trajectories (where errors do not affect the outcome). This reveals a fundamental limitation: *agents lack the self-awareness to recognize their own mistakes, regardless of downstream task success.* These findings underscore the necessity of external error-detection mechanisms like *CORRECT*, as relying on self-monitoring leaves MAS vulnerable to undetected and compounding failures.

## 6 Conclusion

We introduced CORRECT, the first schema-guided framework that distills recurrent MAS failures into compact, reusable schemata, enabling accurate, lightweight, and training-free identification of decisive errors in new runs. Complementing this, we release CORRECT-Error, a large-scale, high-fidelity benchmark capturing realistic error patterns. Across diverse tasks, models, and deployment scenarios, CORRECT significantly outperforms existing advances, offering a practical, generalizable path toward reliable, interpretable, and scalable MAS deployment.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Cemri et al. (2025)
Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al.
Why do multi-agent llm systems fail?
*arXiv preprint arXiv:2503.13657*, 2025.
- Chen et al. (2025)
Jack Chen, Fazhong Liu, Naruto Liu, Yuhan Luo, Erqu Qin, Harry Zheng, Tian Dong, Haojin Zhu, Yan Meng, and Xiao Wang.
Step-wise adaptive integration of supervised fine-tuning and reinforcement learning for task-specific llms.
*arXiv preprint arXiv:2505.13026*, 2025.
- Chen et al. (2024)
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu.
Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024.
- Clark et al. (2018)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
*arXiv:1803.05457v1*, 2018.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Dong et al. (2022)
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al.
A survey on in-context learning.
*arXiv preprint arXiv:2301.00234*, 2022.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv e-prints*, pp. arXiv–2407, 2024.
- Epperson et al. (2025)
Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi.
Interactive debugging and steering of multi-agent ai systems.
In *Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems*, pp. 1–15, 2025.
- Fourney et al. (2024)
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al.
Magentic-one: A generalist multi-agent system for solving complex tasks.
*arXiv preprint arXiv:2411.04468*, 2024.
- Fu et al. (2025)
Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao.
Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning.
*arXiv preprint arXiv:2506.19767*, 2025.
- Gao et al. (2025)
Mingyan Gao, Yanzi Li, Banruo Liu, Yifan Yu, Phillip Wang, Ching-Yu Lin, and Fan Lai.
Single-agent or multi-agent systems? why not both?
*arXiv preprint arXiv:2505.18286*, 2025.
- Ge et al. (2025)
Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang.
Who is introducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis.
*arXiv preprint arXiv:2509.13782*, 2025.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Ho et al. (2020)
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa.
Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.
In *Proceedings of the 28th International Conference on Computational Linguistics*, pp. 6609–6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
URL [https://www.aclweb.org/anthology/2020.coling-main.580](https://www.aclweb.org/anthology/2020.coling-main.580).
- Hong et al. (2023)
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al.
Metagpt: Meta programming for a multi-agent collaborative framework.
In *The Twelfth International Conference on Learning Representations*, 2023.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Jimenez et al. (2023)
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.
Swe-bench: Can language models resolve real-world github issues?
*arXiv preprint arXiv:2310.06770*, 2023.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.
- Lightman et al. (2023)
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
*arXiv preprint arXiv:2305.20050*, 2023.
- Lu et al. (2024)
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha.
The ai scientist: Towards fully automated open-ended scientific discovery.
*arXiv preprint arXiv:2408.06292*, 2024.
- Mialon et al. (2023)
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom.
Gaia: a benchmark for general ai assistants.
In *The Twelfth International Conference on Learning Representations*, 2023.
- OpenAI (2025)
OpenAI.
Gpt-5.
[https://openai.com](https://openai.com), 2025.
Large language model, accessed via ChatGPT.
- Peng et al. (2023a)
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.
Instruction tuning with gpt-4.
*arXiv preprint arXiv:2304.03277*, 2023a.
- Peng et al. (2023b)
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.
Yarn: Efficient context window extension of large language models.
*arXiv preprint arXiv:2309.00071*, 2023b.
- Qian et al. (2023)
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al.
Chatdev: Communicative agents for software development.
*arXiv preprint arXiv:2307.07924*, 2023.
- Su et al. (2024)
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
*Neurocomputing*, 568:127063, 2024.
- Trivedi et al. (2022)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.
Musique: Multihop questions via single-hop question composition.
*Transactions of the Association for Computational Linguistics*, 10:539–554, 2022.
- Wang et al. (2024)
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al.
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.
*Advances in Neural Information Processing Systems*, 37:95266–95290, 2024.
- Wang et al. (2025)
Zhaodong Wang, Samuel Lin, Guanqing Yan, Soudeh Ghorbani, Minlan Yu, Jiawei Zhou, Nathan Hu, Lopa Baruah, Sam Peters, Srikanth Kamath, Jerry Yang, and Ying Zhang.
Intent-driven network management with multi-agent llms: The confucius framework.
In *SIGCOMM*, 2025.
- Wu et al. (2024)
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al.
Autogen: Enabling next-gen llm applications via multi-agent conversations.
In *First Conference on Language Modeling*, 2024.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yang et al. (2024)
Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang.
Qwen2.5 technical report.
*ArXiv*, abs/2412.15115, 2024.
URL [https://api.semanticscholar.org/CorpusID:274859421](https://api.semanticscholar.org/CorpusID:274859421).
- Yang et al. (2018)
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.
HotpotQA: A dataset for diverse, explainable multi-hop question answering.
In *Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 2018.
- Yu et al. (2025)
Yifan Yu, Yu Gan, Lillian Tsai, Nikhil Sarda, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Henry M Levy, and David Culler.
Echolm: Accelerating llm serving with real-time knowledge distillation.
*arXiv preprint arXiv:2501.12689*, 2025.
- Zhang et al. (2025)
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al.
Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems.
*arXiv preprint arXiv:2505.00212*, 2025.
- Zhang et al. (2024)
Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jignesh M Patel.
Reactable: enhancing react for table question answering.
*Proceedings of the VLDB Endowment*, 17(8):1981–1994, 2024.
- Zheng et al. (2023)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
*Advances in neural information processing systems*, 36:46595–46623, 2023.
- Zhou et al. (2023)
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al.
Webarena: A realistic web environment for building autonomous agents.
*arXiv preprint arXiv:2307.13854*, 2023.

## Appendix A Appendix

### A.1 Specifics of experimental settings

We implement our evaluation pipeline based on Zhang et al. (2025). We host open-source models using vLLM(Kwon et al., 2023) and access GPT-series models(Achiam et al., 2023) via the OpenAI API. To handle long contexts exceeding standard model limits for Qwen models(Yang et al., 2025), we employ RoPE(Su et al., 2024) scaling with 4× length extension using the ”yarn”(Peng et al., 2023b) scaling type. To simulate realistic deployment scenarios where ground truth is

Figure: Figure 12: Percentage of human labelers to believe the trajectory is not synthesized.
Refer to caption: x12.png

unknown, we exclude the correct answer from evaluation prompts.
For our method, we first generate all the error schemata using GPT-5 model. We then derive a similarity mapping to assign error schemata based on the semantic embedding decoded by BAAI-BPE-M3 model(Chen et al., 2024). To avoid the data leakage, we mask each trajectory itself and avoid receiving its own error schema. We decide the number of error schemata from the experiments using Qwen-2.5-7b models on Hand-Crafted dataset, Algorithm-Generated dataset, and HotpotQA dataset of CORRECT-Error.
We use the same number of error schemata across all models and all datasets in CORRECT-Error. Specifically, we use 1 error schema for all experiments on the Algorithm-Generated dataset, 10 error schemata for all experiments on the Hand-Crafted dataset, and 5 error schemata for all experiments on CORRECT-Error.

### A.2 More details of the CORRECT-Error

We implemented a variant of Magentic-One (Fourney et al., 2024) using selector group based workflow control using AutoGen (Wu et al., 2024) to generate CORRECT-Error.

Apart from the figures we showed in section [4](https://arxiv.org/html/2509.24088v1#S4), we observed strong inter-annotator consensus: 94.4% of synthetic trajectories fooled at least two labelers, while 52.9% were unanimously mistaken for genuine errors. We show the distribution in Figure [12](https://arxiv.org/html/2509.24088v1#A1.F12)

### A.3 Prompts for Offline Error Schema Generation

We show the prompts we used for offline schema generation in Fig [A.3](https://arxiv.org/html/2509.24088v1#A1.SS3)

### A.4 Prompts for online schema-guided generation

We show the prompts we used for online schema-guided generation in Fig [A.4](https://arxiv.org/html/2509.24088v1#A1.SS4)

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could the author's provide examples of schema generated by CORRECT in offline and online phases? While one of the emphasis of CORRECT is inference-time schema inference, could the authors' discuss how is the offline mode different from the application of a static human curated taxonomy like MAST?
- Could the authors' address the impact of starting from successful trajectories, what about domains with high failure rates?
soundness: 3
strengths: - The paper proposes a technique that is initialized with offline analysis of failure trajectories, but can also update the schemata/cache online, to identify new failure modes as they occur during deployment
- The paper shows evidence of human alignment with bootstrapped trajectories
- Evaluation across diverse benchmarks and models, demonstrating accuracy improvements in MAS with the intervention
summary: The paper addresses the challenge of error recognition in MAS execution, finding the root cause of failure in a long trajectory. The paper builds upon the insight that while the specifics of the failure trajectory appear unique, they stem from recurring structurally similar patterns.

With this, the authors' propose CORRECT, a training-free framework, that builds an "error schemata" by offline analysis of failure trajectories, identifying the signature pattern and context of errors. During online inference, it retrieves the most relevant schemata from a cache, providing them to LLM-as-judge, enabling targeted and accurate error localization without fine-tuning.

The paper also introduced CORRECT-Error, a large-scale benchmark of annotated failure trajectories.
weaknesses: - The primary baselines are naive ICL, zero-shot LLM-as-judge and finetuning. Since the proposed approach is about context intervention through a cache, could the authors comment on prompt optimization techniques like MIPRO as a baseline?
- The dataset is built by corrupting successful trajectories through error injection. Could the authors' discuss the effect of sampling bias by only selecting from task instances for which an existing MAS is able to successfully bootstrap a valid solution. What about domains with close to 100% failure rates?

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness
soundness: 3
strengths: This paper presents a large-scale, human-verified benchmark specifically designed for multi-agent system (MAS) error attribution. The benchmark captures realistic and diverse error patterns observed in MAS deployments, providing a valuable resource for evaluating and comparing model robustness and interpretability. Its human validation ensures high fidelity and reliability, making it a strong contribution that can serve as a foundation for future research in this area

The authors propose a schema-guided framework for MAS error attribution that leverages a schema and schema-cache mechanism to systematically identify and interpret recurrent error patterns. The framework operates on a retrieval-augmented principle, where previously learned error schemata are reused to analyze new failures efficiently without additional training.

The paper supports its contributions through extensive experiments conducted across a wide range of tasks, models, and deployment scenarios. The empirical results demonstrate that the proposed CORRECT framework significantly outperforms existing baselines in accuracy, generalization, and computational efficiency.
summary: The paper introduces CORRECT, a schema-guided framework that identifies key recurrent multi-agent system (MAS) failures without requiring additional training. It also presents CORRECT-ERROR, a large-scale benchmark that models realistic error patterns for evaluating MAS reliability. Together, these tools improve the accuracy, interpretability, and scalability of MAS deployment across diverse tasks and systems.
weaknesses: While the proposed framework demonstrates strong performance across static evaluation settings, a notable limitation is the absence of experiments in a streaming or online deployment scenario. In real-world MAS applications, systems often operate in a dynamic environment where the schema cache starts empty and must adapt incrementally as new error cases emerge. Evaluating CORRECT under such a cold-start or streaming setting would provide deeper insights into its practical robustness, adaptability, and scalability over time. Without this, it remains unclear how efficiently the framework can learn and reuse schemata in continuously evolving contexts.

Another limitation is that the framework relies on user confirmation or supervision during the cache-building process. While this human-in-the-loop component ensures higher accuracy in schema construction, it also introduces additional overhead and dependency on expert input, which may limit scalability in fully automated or large-scale deployments. Future work could explore strategies for automated or semi-supervised schema validation to reduce reliance on manual confirmation while maintaining interpretability and reliability.

Also note that the idea of using human-verified cache to improve model exists in the literature, eg, EcoAssistant.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. Regarding the cache management, what are the empirical values of $\delta$ , $\theta_{hot}$, and how sensitive are results to them?
2. Have you observed cases where an irrelevant but high-similarity schema misleads the detector? How is this mitigated?
soundness: 3
strengths: 1. The paper addresses a timely issue in the filed of MAS. The proposed idea of distilling reusable failure patterns into short templates is novel and can avoid costly re-training.
2. The empirical evaluation demonstrates strong performance gain.
summary: The paper introduces CORRECT, a training-free framework that detects the earliest decisive error in long multi-agent trajectories by caching and reusing compact “error schemata.” A clustering step extracts representative schemas offline. At inference they are retrieved by semantic similarity and supplied to an LLM that pinpoints the responsible agent-step. Experiments on WHO&WHEN and CORRECT-Error show step-level accuracy gains up to 20 points over baselines while incurring no additional training cost.
weaknesses: 1. My major concerns about the paper lie in the presentation, specifically many techical details are left out:
- The fine-tuning baseline is only named. The paper never explains how the instruction-tuning dataset was built, whether the model was trained to output just the failure-step index or a reasoning trace accompanied, and whether supervised fine-tuning or RLHF was used. Hyper-parameters are missing, even from the appendix.
- In Stage 2 of Section 4.1, the text says GPT-5 “adapts the error pattern while preserving core semantics,”. But the paper provides no prompt, no definition of “error pattern,” and no description of the source.
- Many trajectories exceed 32k tokens, but the method relies on BERT embeddings, whose context window is far shorter. The paper does not mention how embedding is done to handle the length. 

2. Algorithm 1 is under-explained. The pseudo-code drives the whole framework, but the main text never walks through its lines. Key variables (e.g., $\delta$ , $\theta_{hot}$, cache replacement policy) and their interaction with Sec. 3 concepts are left implicit. I could not map each step to Sections 3.1/3.2 without guessing. A major rewrite is needed for clarity.
3. The method invoke proprietary models at test time, the api cost and wall-clock time are not reported.
4. There is limited discussion of failure cases. Where does CORRECT mis-fire? Examples of erroneous schema matches or degenerate clusters would strengthen the paper.
5. The paper does not discuss [1], which is another concurrent work with Who&When. How the method transfers to the error recoginition dataset proposed in [1] requires further discussion.

[1] Why Do Multi-Agent LLM Systems Fail?

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. How exactly are decisive-step labels produced in both WHO&WHEN (human-crafted/algorithm-generated) and CORRECT-Error? Do you perform counterfactual replay that replaces step k and re-executes to confirm flip-to-success, or rely on annotator judgment?

2. What happens when top-k retrieval deliberately includes partially mismatched schemas (e.g., same tool family but different failure signature)? Could you report accuracy as a function of retrieval noise or add a hard negative experiment?

3. Have you tried a causal-replay baseline that toggles a small set of suspect steps (e.g., tool-call deltas) to find the earliest flip-to-success point? Even if expensive, a smaller-scale study would clarify whether schemas offer gains over more explicit verification.

4. The paper claims the framework is "training-free", yet it relies heavily on LLMs for both schema generation and final diagnosis. While it avoids fine-tuning, it is not independent of trained models. Could you clarify this claim or rephrase it to more accurately reflect that the method is "fine-tuning-free" but dependent on powerful pre-trained models?

5. Could you elaborate on the computational cost and practical implementation of the "schema distillation" process? How expensive is it to replay schema candidates on prior trajectories, and how does this scale as the cache grows?
soundness: 3
strengths: 1. The discussed topic is important and timely. The problem of error attribution in increasingly complex and long-horizon MAS is both critical for reliability and under-explored.

2. The central idea of abstracting away from noisy, full-length trajectories into condensed "error schemata" is insightful and novel. This schema-guided approach is a clever way to perform knowledge transfer without the prohibitive context length requirements or computational costs of naive ICL or fine-tuning.

3. Extensive Experimental Evaluation: The authors conduct a thorough evaluation across two major benchmarks (WHO&WHEN and their new CORRECT-Error), seven diverse sub-tasks (e.g., HotpotQA, GAIA, Math500), and a wide range of open-source and proprietary models. The ablation studies on cache size and the number of retrieved schemata provide valuable insights into the framework's behavior.
summary: This paper addresses the important and challenging problem of "decisive error recognition" in Multi-Agent Systems (MAS). The authors propose CORRECT, a novel, training-free framework that leverages knowledge transfer to identify the root cause of task failures. The core idea is to distill lengthy and noisy failure trajectories into compact, reusable "error schemata." These schemata are then retrieved at inference time to guide an LLM in diagnosing new, similar failures. To facilitate research in this area, the authors also introduce CORRECT-Error, a large-scale benchmark of over 2,000 annotated failure trajectories created via a novel error-injection pipeline. Experiments conducted on the new benchmark and the existing WHO&WHEN dataset show that CORRECT improves error localization accuracy over several baselines.
weaknesses: 1. A major weakness of the proposed framework is its critical dependence on a highly capable and proprietary LLM (GPT-5 is mentioned) for the offline schema extraction phase. The quality of the entire system hinges on the ability of this "teacher" model to generate high-quality schemata.

2. The dynamic schema management system (expansion and distillation) is described at a high level, but practical details are sparse. The process of "replaying" candidate schemas against prior trajectories to select the best one  sounds computationally intensive, especially as the number of trajectories and schemas grows.

3. Baselines center on generic LLM-as-a-judge and naive ICL. The paper does not pit CORRECT against specialized verification pipelines which could also reduce long-context noise without relying on learned schemas.

4. The schema idea is also central to the benchmark generation (semantic matching + schema-like error patterns). While the human study is encouraging, the pipeline may implicitly bias toward the kinds of failures that schemas capture well, potentially inflating CORRECT’s advantage. The paper does not quantify how often decisive errors in natural logs deviate from the assumed, reusable “pattern” structure.


Decision: Reject