Can Agent Fix Agent Issues?
Authors: Alfin Wijaya Rahardja
Junwei Liu
Weitong Chen
Zhenpeng Chen
Yiling Lou
Venue: NeurIPS 2025 poster ()
Keywords: LLM-based agents
SE agents
issue resolution
Primary area: evaluation

Abstract: LLM-based agent systems are emerging as a new software paradigm and have been widely adopted across diverse domains such as medicine, robotics, and programming. However, maintaining these systems requires substantial effort, as they are inevitably prone to bugs and continually evolve to meet changing external requirements. Therefore, automatically resolving agent issues (i.e.,bug reports or feature requests) is a crucial and challenging task.  While recent software engineering (SE) agents (e.g., SWE-agent) have shown promise in addressing issues in traditional software systems, it remains unclear how effectively they can resolve real-world issues in agent systems, which differ significantly from traditional software. To fill this gap, we first manually analyze 201 real-world agent issues and identify common categories of agent issues. We then spend 500 person-hours constructing AgentIssue-bench, a reproducible benchmark comprising 50 agent issue resolution tasks (each with an executable environment and failure-triggering tests). We further evaluate state-of-the-art SE agents on AgentIssue-bench and reveal their limited effectiveness (\.e., with only 0.67% - 4.67% resolution rates). These results underscore the unique challenges of maintaining agent systems compared to traditional software, highlighting the need for further research to develop advanced SE agents for resolving agent issues.

Can Agents Fix Agent Issues?
Alfin Wijaya Rahardja"
Junwei Liu" Fudan University jwliu24@m. Fudan.edu. edu.cn
Fudan University 24212010055e2 Fudan.edu.cm
Weitong Chen Fudan University 21307 30392èm. fudan.edu.cm edu.cn cn
Zhenpeng Chen Nanyang Technological University zhenpeng, chen@ntu. eww.eettuuuuuu...... sg
Yiling Lou University Illinois Urbana-Champaign yilingleillinois..
Abstract
LLM-based agent systems are emerging new software paradigm and have been widely adopted across diverse domains such as medicine, robotics, and program- ming. However, maintaining these systems requires substantial effort, as they are inevitably prone to bugs and continually evolve to meet changing external require- ments. Therefore, automatically resolving agent issues (i.e., bug reports or feature requests) is a crucial and challenging task. While recent software engineering (SE) agents (e.g., SWE-agent) have shown promise addressing issues in traditional software systems, it remains unclear how effectively they can resolve real-world issues in agent systems, which differ significantly from traditional software. To fill this gap, we first manually analyze 201 real-world agent issues and identify common categories of agent issues. We then spend 500 person-hours constructing AGENTISSUE- BBANTISSUU-BENCHH BENCH, reproducible benchmark comprising 50 agent issue res- olution tasks (each with an executable environment and failure fallure-triggering tests). We further evaluate sfate-of-the-art SE agents AGENTISSUE- BEENC and veal their limited effectiveness (i.e., with only 0.67% 4.67% resolution rates). These results underscore the unique challenges of maintaining agent systems com- pared to traditional software, highlighting the need for further research to develop advanced SE agents for resolving agent issues. Data and code available wtt.c/ocoaaaa/aa
1 Introduction
LLM-based agent systems have seen widespread adoption across diverse domains, such as
medicine [38] (56). programming [18 58 63 61], robotics 65), psychology [47] 60]. and general-
purpose personal assistants [6. 77. Driven by rapid advancements, agent systems are emerging as
new software paradigm, playing increasingly pervasive role in shaping and supporting the full spectrum of human activities.
As products of human intellectual labor, similar as traditional software systems, agent systems are also inevitably prone to quality issues. Recent work (34) has shown that multi-agent systems exhibit diverse failure modes during operation. Moreover, agent systems are continuously evolving to meet changing external requirements, making their maintenance both crucial and labor-intensive. For
Equaal contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

instance, by May 2025, the agent system MetaGPT [26] had accumulated over 800 GitHub issues (an issue typically a bug report or a feature request), highlighting the substantial maintenance workload associated with agent systems.
Automating the issue resolution process has been important and challenging direction with substantial dedicated research effort. In particular, with the recent advances in agent systems, there is a growing trend toward developing software engineering agents [52 66
58
[8] 28 27 (referred
to as SE agents in this paper), which can automatically resolve real-world software issues. Recent SE agents have demonstrated strong potential in resolving issues in traditional software systems. For
instance, Agentless [52] correctly resolves 50.80% of issues on SWE-bench [3 real-world issue resolution benchmark for traditional Python software.
Although SE agents have shown promise in resolving issues in traditional software systems, it remains unclear how effectively they perform agent systems, which is a new software paradigm that differs significantly from traditional software. Therefore, in this work, we aim to answer the central question: can SE agents fix issues in agent systems?
To understand issues in agent systems, we first perform an empirical study to analyze and catalog real-world agent issues. In particular, we collect 201 real-world GitHub issues along with developer- committed patches from 18 widely-used agent systems. We further build a taxonomy of agent issues with human annotators via grounded theory, resulting in 6 categories and 20 sub-categories of common agent issues. Our taxonomy reveals that real-world agent systems exhibit a diverse range of issues, many of which possess unique characteristics not typically found in traditional software systems. The findings highlight the large engineering effort for maintaining agent systems, confirming that automated issue resolution for agent systems is challenging and critical problem.
We then build AGENTISSUE- BENCH,, the first reproducible benchmark for agent issue resolution. Reproducing agent issues is particularly more challenging compared to traditional software issues, largely due to the nondeterminism of LLMs and the volatility of external resources (e.g.. tools) that agents interact with. As a result, from the 201 issues analyzed, we invested 500 person-hour hours to successfully reproduce 50 agent issues. Each issue resolution task in AGENTISSUE- -AANNESSSU--EENCC is packaged within executable Docker environment, along with failure- triggerin tests, user-reported issue descriptions, the buggy version, and the developer-committed patched version of the codehase.
We further evaluate multiple state-of-tee of-the-rrt SE agents (i.e., Agentless (52, AutoCodeRover [66]] and SWE-agent (58) with both GPT-4o [ and Claude-3.5 [15] on AGENTISSUE-BENCC.
We find that all of the existing SE agents exhibit limited capabilities in resolving agent issues. For instance, only 0.67% to 4.67% of agent issues are correctly resolved, which is significantly lower than the resolution rates achieved when these SE agents are applied to traditional software (e.g. 23.20% 50.80% resolution rate [33). We further conduct a qualitative analysis to break down the resolution capabilities of SE agents across different categories. Notably, the majority of resolved issues pertain to utility or dependency issues, while the most of LLM-related issues (e.g., compatibility with LLM providers LLM operation issues) remain unsolved. Overall, our analysis reveals the limitations of current SE agents in resolving agent issues, underscoring the need for building advanced SE agents tailored to the maintenance of agent systems.
In summary, this work makes the following contributions:
Taxonomy, We present the first taxonomy of issues in agent systems, derived from extensive manual analysis, which summarizes the common maintenance demands encountered during agent system evolution.
Reprodueible benchmark AGENTISSUE-BENC BENCH. We manually construct the first issue resolution benchmark of real-world agent issues. Each task is packed into an executable Docker environment, including issue descriptions, failure-triggering tests, and both buggy and patched versions of the codebase, enabling easy reproduction and validation through one-click execution.
Evaluation. We evaluate state-of-the--ar art SE agents on AGENTISSUE- BENCH with both quan- titative and qualitative analysis, and find their limited capabilities in solving agent issues. Our findings highlight the unique challenges of maintaining agent systems, underscoring the need to develop more powerful SE agents for resolving agent issues.
2

---

2 Background and Related Work
2.1 LLM-based Agent Systems
LLM-based agent systems are emerging as new software paradigm, which have been widely
applied across various fields (e.g., medicine [8 56), programming [18, 58) robotics [4] 65
psychology 47 60), and general-purpose personal assistants ( with remarkable abilities. An
LLM-based agent system [51 [5) typically consists of: (i) an LLM-controlled brain that decomposes and schedules tasks (i.e., planning) and records the historical behaviors (i.e, memory); (ii) a perception component that receives information from the environment; and (iii) an action component that interacts with the environment by invoking external tools. In addition, single-agent systems can collaborate to form multi-agent systems, which can tackle more complex tasks with better flexibility and effectiveness.
Quality problems in LLM-inteerated systems. Given the widespread adoption of LLMs, recent work has been looking into quality problems (e.g, bugs or runtime failures) in LLM-integrated systems. For example, Shao et al. [49] catalog the integration bugs in LLM and RAG systems. Different from their work, our work specifically focuses on LLM agent systems. This scope distinction leads to notable differences in taxonomies, as our taxonomy is framed from agent architecture perspective (e-g.. featuring broader coverage of tool-related issues, finer-grained categorization of memory issues, API and model binding issues). Along this direction, Cemri el al. (4 build taxonomy of failure modes in multi-agent systems. While their work focuses on runtime failure symptoms by analyzing failure trajectories, our taxonomy centers on agent issue resolution by analyzing both real- worldw user-reported issues and developer-commitee
patches. Therefore, our work complements existing efforts by providing perspective on maintaining agent systems, encompassing a broader scope that includes not only bug fixes but also feature requests. Moreover, our work is further different from existing work by introducing the first reproducible benchmark for agent issue resolution and empirically evaluating saate-of-the-ae SE agents on their ability to resolve agent issues.
2.2 Software Engineering Agents
Software Engineering (SE) agents are a category of agent systems specifically designed to tackle SE tasks 42
In particular, there is growing trend in both industry and academia toward developing
9 46 62). which can support end-to-end software maintenance by automatically resolving user-reported issues (e.g., bug fixes or feature requests). For instance, Devin [18] is one of the first SE agents capable of resolving software issues by invoking file editors, terminals, and search tools. More recently, SWE-agent (58) interacts with the code repository Interface (ACI). capable of performing actions such
SE agents 18
28 52 66 27 58
environment through a custom as manipulating files and executing bash commands; AutoCodeRover [66] incorporates a suite of code search tools that iteratively retrieve relevant code contexts to navigate the repository and localize issue locations; Moatless 27 equips agents with code search and retrieval tools to identify the issue locations; Agentless 522 optimizes the agent workflow with human expertise, incorporating hierarchical localization and regression testing to improve issue resolution rates. In this work, we evaluate the effectiveness of existing SE agents in resolving issues in agent systems.
Benchmarking issue resolution capabilities of SE agents. With the rise of SE agents, an increasing number of benchmarks have been developed to evaluate their capabilities in addressing real-world issue resolution tasks. For instance. Jimenez et al. [40] build ben-benchh from GitHub issues of 12 Python libraries. Based on SWE-bench, researchers further propose a series of benchmarks, e.g.. SWE-bench Lite [40], SWE-bench verified B and SWE-bench Lite-S [52]. which are refined versions of SWE-bench with additional quality checking. While the SWE- series only includes issues of Python software, Zan al. [64 44 further propose SWE--bech Java, an issue resolution benchmark for Java software, and Yang et al. (59 build SWE-hench Multimodal, comprising frontend issue resolutions tasks from open-source JavaScript libraries. More recently, OpenAl releases SWE- Lancer Diamond (24, an issue resolution benchmark with end-to-end tests for Expensify [20) software. While existing benchmarks focus exclusively on issue resolution in traditional software systems, our work introduces the first reproducible benchmark targeting issues in agent systems, an emerging software paradigm with features distinct from traditional software. Using this benchmark, we find that current SE agents are still unable to resolve the majority of issue resolution tasks in agent systems.
3

---

3
Agent Issue Taxonomy
To understand issues during agent system maintenance, we first manually analyze and categorize real-world GitHub issues in widely-used agent systems.
3.1
Methodology
Figure Saareeiillnee
Dillustrates our methodology of systematically collecting and analyzing agent issues.
3.1.1
Data Collection
Agent system collection. select diverse and representative agent systems, we first use the GitHub search API to obtain 50 repositories with keywords "Al agents" by Feb 2025. We then manually go through each repository to keep the ones that are LLM--based agent systems (filter out the unrelated ones like paper lists tutorials); to focus agent systems with active maintenance, we only keep the ones with more than Ik stars and 30 issues. In this way, we collect 18 agent systems, such as MetaGPT (26), AutoGen (I]) GPT-engineer 21 and CrewAl []]. The full list of our analyzed agent systems is Appendix
MetaGPT
Filtering criteria
171 open coding
Taxonomy
CrewAl
Clear description
construction
GitHub
AutoGen
Patch
Single problem
201 o issues
Independent label
Taxonomy Evaluation
6660
GPT-engineer
Closed
issues
Agent system collection
Agent issue extraction
Taxonomy construction
Figure 1: Methodology of agent issue taxonomy construction.
Agent issue extraction. For each studied agent system, we adopt the following inclusion criteria extract high-quality issues. (i) The issue has been closed with ceweloper-commiitee patch to address the issue, as patches can serve ground truth for understanding root causes of agent issues; (ii) The issue has clear descriptions without misleading information (e.g., exact patches or misleading patches in the problem description). This criteria has been widely used in constructing high-quality issue resolution benchmarks for traditional software systems [52 40 ; (iii) The issue should only
report one problem instead of mixing multiple problems. In the end, we obtain 201 issues in total.
3.1.2 Manual Labeling
We randomly separate our collected 201 agent issues into (i) 171 issues (85%) for building the taxonomy and (ii) 30 issues (15%) for evaluating constructed taxonomy.
Taxonomy construction. We manually catalog the 171 agent issues with grounded theory B77 In particular, three human annotators with extensive software development and machine learning experience apply open coding [35] 36, [6) 48 to annotate each issue based on the issue description and the developer-commiited patch. They break down each issue into segments and label them with descriptive codes. Then they organize the open codes into structured categories by merging and linking relevant ones. All the annotators further discuss and review the taxonomy until reaching consensus.
Taxonomy evaluation. We further evaluate our taxonomy on the remaining 30 agent issues. Two annotators independently label each issue. Their annotation reaches high agreement ratio (Cohen's Kappa 0.849); meanwhile there emerge new categories in addition to our taxonomy during their annotation.
3.2 Taxonomy
Table
I presents our taxonomy agent issues, mainly covering categories. Appendix F presents detailed examples for each sub-category. In addition to the "utility issues" category which may also occur in traditional software systems, the remaining five categories are uniquely tied key agent system components (e.g., tools and memory), making them distinctive to agent systems.

---

Incompatibility with LLM providers. Most agent systems incorporate existing LLMs from LLM providers (e.g.. OpenAl 22, DeepSeek (17), and Anthropic [TO), and improper usage of providers' interfaces impairs agent functionality. Such issues often stem from missing dependencies or incorrect Table 1: Taxonomy of agent issues.
Category
Sub-category
Description
Incompati-
Incompatible dependencies
Miss or misuse the libraries of LLM providers.
bility with
(1.49%)
LLM
Unsupported models (2.99%)
Lack the support of recent LLMs.
providers
Incompatible parameters
Use undefined parameters or miss parameters of LLM query interfaces.
(7.46%)
LLM providers (2.99%)
Tool-
Tool dependency issues
Miss or misuse libraries for running agent- -enok-n tools.
related
(3.48%)
issues
Tool configuration issues
Misconfigure the settings of agent-invoked tools.
(19.90%)
(3.47%)
Tool implementation errors
Incorrect implementation of self-developed agent-invoked tools.
(8.46%)
Misuse tool interfaces (4,48%
Incorrect tool invocation due to missing/wre /wrong parameters.
Memory- related
Memory initialization issues
Incomplete inconsistent memory states duc database ini- tialization or workspace resetting issues.
(2.49%)
issues
Memory
content errors
Incorrect message attributes, miseadiigg content, or content loss caused by faulty storage logic.
(14.43%)
(10.95%)
Memory dependency issues
Incorrect internal module dependencies or external libraries required by memory operations.
(1.00%)
Model access misconfigura-
Model access errors caused by misconfiguraation like incorrect model binding or authentication credentials (e.g, API keys). LLM token management issucs such as incorrect limits or pricing.
LLM operation
tion (6.97%)
Token usage misconfiguration (3.48%)
issucs
(31.34%)
Incorrect model output han-
Incorrect parsing logic for model output or miss handlers for un- expected model behaviors like empty exceptional responses. Missing/incompatible libraries related model operation such as tokenization or transformer dependency conflicts. Truncated outputs caused by exceeding context limits or mis- calculating context length.
dlers (8.46%)
Model dependency issues (2.99%)
Context length issues (4.98%
Prompt-related issues (4.48%)
Suboptimll prompt content or prompt management issues (e.g..
fail to set/update prompts).
Workflow issues (6.47%
Abuormal agent workflows like hanging or repeated loops. Implementation errors in LLM-unrelated components (e.g..
Uulity
Uility implementation issues
issues
(8.96%)
U/DDccerr/ogging))
(20.40%)
Utility dependency issues (4.48%)
Miss/incompatible libraries circular internal dependencies required by general utilities (e.g.. lesting or file operations). External component misconfiguration (e.g., I/O paths, network settings).
Utility configuration issues
(6.97%)
invocations of provider APIs. Moreover, due to the rapid evolution of LLMs, users frequently request new feature support newly-released LLMs.
Tool-related issues. The versatility of agent systems partly stems from their proficiency in utilizing tools to interact with the environment. As result, many agent-related issues arise during tool invocation, including missing tool-dependent libraries, misconfigurations, or incorrect use of tool interfaces. In addition to external tools, agents may also rely on internal tools (e.g,, custon-developed functions), where implementation flaws can trigger unintended behaviors during tool execution.
Memory-related issues. The memory mechanism in agents tracks the trajectory of agent operation, and most memory-related issues arise from incorrect memory content. For example, agents may pollute memory with irrelevant information when they mistakenly extract unrelated attributes from the current context, or memory entries may be missing or incomplete due to failures in storing data.
Workflow issues. Due to the autonomy and flexibility of agent systems, unexpected behaviors can emerge along the agent workflow, such repeated actions or hanging states. Although it is difficult to completely eliminate such issues, developers commonly mitigate them by incorporating status checkers to monitor and regulate the agent workflow.
5

---

LLM operation issues. A large portion (31.34%) of agent-related issues occur during LLM operation. For example, proper configuration model access and token usage critical, and misconfiguration in these areas can disrupt agent functionality. Additionally, many issues stem from incorrect handling of model outputs, including: (i) flawed parsing implementations, or (ii) missing handlers for unexpected model responses. Beyond the suboptimal prompt content (e.g., unclear model instructions), prompt management can also introduce risks: as agent systems often maintain a large and evolving pool of prompts, failures in prompt updates configuration can result in models being queried with incorrect or outdated instructions.
Summary. Our taxonomy reveals that real-world agent systems exhibit diverse range of issues, many of which possess unique characteristics not typically found in traditional software systems. In particular, developing and maintaining agent systems demands substantial engineering effort, as developers must manage correct dependencies, configurations, and implementations across multiple components (e.g-, model providers, L.LM operations, memory mechanisms, and tools). Therefore, we believe that automatically resolving issues in agent systems represents a challenging and increasingly vital research direction in the era of LLMs.
AGENTISSUE-BENE Benchmark
We then manually build AGENTIsSUE-BENCH the first reproducible issue resolution benchmark of real-world agent issues. AGENTISSUE-BENCH can used to evaluate the efficacy of state-of-the-art SE agents in solving issues in agent systems.
4.1 Benchmark Construction
We construct AGENTISSUE- BENCH out of the 201 GitHub agent issues we collected in Section 3 In particular, we try to reproduce each issue according to the following procedure.
Step 1: Failure reproduction. For each issue, we pull its corresponding buggy commit and set up the agent system. particular, we manually write test script (i.e., failure- triggering test) to reproduce the problematic behaviors according to the issue descriptions. In this step, we filter out the issues where we cannot observe the same buggy behavior as issue descriptions.
Step 2: Patch reproduction. We then pull the corresponding patched commit and execute the triggre-t test on it. this step, we only keep the issues where the patched version can pass the failure- teilurr-iigggerin tests (i.e., problematic behaviors disappear on the patched version).
Step 3: Non-flakiness verification. Given the nondeterminism of LLMs, repeat the previous two steps three times for each issue so as to eliminate the test flakiness. In this step, we filter out issues where there are inconsistent behaviors on executing one failure-trigerrin test.
Through such multi-step filtering process, the original 201 agent issues are narrowed down to 50 reproducible issue resolution tasks, collectively forming AGENTISSUE-BENGE reproducing issues in agent systems is significantly more challenging than traditional software systems, agent issues are associated with diverse internal and external components and resources. In particular, most agent issues fail to reproduce for the following reasons. (i) The nondeterminism of LLMs leads to unstable model outputs, which hinders the reproduction of agent issues such as workflow errors; (ii) External resources (e.g., agent-invoked tools, dependent libraries, or LLM providers) may have changed since the issue was reported, making it impossible to reproduce the same failure; (iii) Issue descriptions lack sufficient details or steps on how to reproduce the problematic behaviors; (iv) Agent systems cannot be correctly set up and exhibit unexpected failure behaviors that are different from the issue descriptions. Overall, the entire reproduction process takes huge manual effort (approximately 500 person-hours).
We find that
4.2 Benchmark Details
Benchmark statistics. Figure shows the distribution of ACENTISSSUR-BENCC across different issue categories. Overall, we can observe that the 50 reproduced agent issues in AGENTISSUE- BENCH cover all the main categories identified in our taxonomy of agent issues, indicating that MEANTISSUE-BENCL is representative of real-world agent issue distribution. Moreover, issues in AGENTISSUE- SEATTIISSE-EENCCH involve patches of different scales (Detailed statistics are Table

---

Memoor
Misuse tool interfaces (6.0%)
content errors (10.0%)
Too implementation errors (6.0%)
Memory initialization issues (0.0%)
Tool configuration (6.0%)
Model access
16.0%
misconfiguration (4.
24.0%
output
dependency issues (6.0%)
h'andlers (2.0%)
Model dependency
22.0%
4.0%
incompatible parameters to LLM providers (4.0%)
Issues (8.0%)
reller-
LLM operation
Context length issues (4.0%)
10.0%
24.0%
issues (8.0%)
Utility implementation
issues
Workflow
Prompt-related issues (4.0%)
Ubliiy issues
incompatibility
Utility configuration issues (12.0%)
providers
Workflow issues (10.0%)
Toal-no leddd
Utility dependency issues (4.0%)
issues
Figure 2: Distribution of AGENTISSUE-BENG
consists of the following components: (i) Issue description: a user-reported textual description of the problem; (ii) Buggy version of the agent system: the buggy commit of the agent code repository in which the issue occurs; (iii) Developer- committed patch: the code changes between the buggy and correct versions, serving as the ground truth for issue resolution; (iv) Failure-triggering tests: test scripts that reproduce the issue on the buggy version but pass the patched version; (v) Docker environment: a container with all necessary dependencies and configurations execute the agent system.
Each issue resolution instance in AGENTISSUE-BiET
Issue Description
SE Agents
Generated Patch
Kickoff hangs when LLM call fails. Minmum example reproduce the issue...
SWE-aperi
AutoCodeRovee
agent.py
Agantless
agents
crew www.eeeeeeeeerrrrrpppppp
Code Repository
ublities
agents
uttiitis
I+1
Patch Verification
llm_utils.pp py
agent.py
Irn.py
Fallure-tfggering Tests X
llm.py
Figure A task example in AGENTISSUE-
Task formulation.
The agent issue resolution task can be formulated as follows: (i) Input: the issue description and the buggy codebase of the agent system; (ii) Output: a patch (i.e., code edit to the buggy codebase) that aims resolve the issue. Figure shows the task example in AGENTISSUE-BI -ENCH.
Evaluation metries. To evaluate how technique tackles the agent issue resolution task, adopt the following metrics to evaluate the patches output by the technique (i.e., SE agents in our experiments). (i) Localization accuracy: if the generated patch modifies the same location as the developer-committee patch, we consider to have accurately localized the issue. We then compute the percentage of issues for which the generated patches can achieve accurate localization. (ii) Plausible resolution rate: if the generated patch makes the cuilure-triggerin applied, we consider compute the percentage of issues for which the generated patches are plausible patches. (iii) Correct resolution rate: the generated plausible patch is further camaanticlll--quiialleee committed patch, we consider it to correctly resolve the issue (i.e., denoted as a correct patch). In particular, given the insufficiency of tests in practice, patches are not necessarily correct patches but are just overfitting to the failure-triggering tests. Therefore, only reporting the plausible resolution rate can overestimate the effectiveness issue resolution techniques. Following the common practice in the program repair area [55]
tests pass after being plausibly resolve the issue (i.e., denoted as a plausible patch). We then
the developer-
common 455 57 4 that plausible
54 53
39
7

---

Table 2: Overall results of SE agents AGENTISSUE- BENCH
Plausibly resolved %
Correctly resolved %
Localization % Function-level
Avg.
SE Agent
LLM
File-level
SCost
GPT-4o
12.00
3,33
27.82
12.99
0.65
Agentless
Claude-3.5-S
12.00
4.00
27.35
17.50
0.33
GPT-40
7.33
1.33
22.07
14.77
0.23
AutoCodeRover
Claude-3.5-S
12.67
4.67
25.81
19.18
0.05
GPT-40
0.67
0.67
11.67
4.22
1.15
SWE-agent
Claude-3.5-S
2.00
2.00
9.52
6.78
0.57
further involve human annotators to manually check whether the plausible patches are semantically patches. In particular, two participants independently review each plausible patch by comparing it to the golden patch (i.e., developer-committed) focusing on whether the semantics the patch fully resolve the underlying issue as intended and do not introduce other functional or semantic errors. If both reviewers agree that the patch is semantically equivalent and correctly resolved the issue, it is labeled as correct. there is a disagreement between the two reviewers, a third participant would be involved an adjudicator. The final label is determined only after all three reviewers reach a consensus. We then compute the percentage issues for which the generated patches are correct patches.
equivalent to developer-committe
Experiments
In this section, investigate how state-of-the-art SE agents automatically resolve real-world issues in agent systems by evaluating their efficacy on AGENTISSUE- BEANNCH
5.1 Experimental Setup
Studied SE agents. We include three state-of-the-art SE agents, including SWE-agent (58) Au- toCodeRover (66), and Agentless (52). These agents are selected given that they are fully open- sourced and achieve superior effectiveness resolving issues for traditional software systems [33 We directly adopt their released implementation with the original hyperparameter settings.
Backbone LLMs. Based on the recent SWE leaderboard [33] state-of-the-ar SE agents achieve higher fixing rate general software issues when equipped with backbone LLMs GPT-4o MI and Claude-3.5 Sonnet [15]. Therefore, in our experiments, we mainly study how effective SE agents are in resolving agent issues with these two backbone LLMs (temperature 0).
Evaluation pipelines. We apply studied SE agents on AGENTISSUE- and collect their generated patches for each issue resolution task. We then calculate the metrics of fault localization accuracy, plausible and correct resolution rates for each studied SE agent. To eliminate the randomness from LLMs, we repeat all experiments three times and present the average results. In particular, our major metric (average resolution rate over three runs) is essentially average pass@) over three runs. Table 8 in Appendix G further presents the pass @ and pass@3 3 over one run.
5.2 Quantitative Results
Overall resolution effectiveness. Table ssows the results of the studied SE agents AGENTISSUE- BENCH. In general, state-of-the-ar SE agents can only correctly resolve a small number (i.e., 0.67% 4.67%) of agent issues. In addition. in most cases, SE agents even fail to correctly identify the location (i.e., files or functions) for resolving the issue, e.g., file-level/funccioo-eeyy accuracy is less than 28%/20%. Such observations reveal the limited capabilities of state-of-the-an SE agents in understanding and resolving the issues in agent systems.
localization
In addition, Figure compares the correct resolution rate of SE agents agent issues (on benchmark SRRENTISSUE-BENCCE versus on traditional software issues (results from SWE-bench Lite (33] ) As there is previous data of ouvo with Claude-3.5-$ leave it as blank. Overall, SE agents demonstrate significantly lower resolution rates on agent issues compared traditional software issues. These findings highlight the unique challenges posed by agent systems and underscore the need for developing SE agents specifically tailored to maintain agent systems, which is an emerging and distinctive software paradigm.
SWE-bench, we

---

Table 3: Breakdown of resolved agent issues (unresolved categories presented).
Category
Resolved%
Sub-category
Resolved%
Tool-related issues
2/12 (16.67%)
Tool dependency issues
2/3 (66.67%)
LLM operation issucs
1/11 (9.09%)
Prompt-related issues
1/2 (50.00%E)
Utility issues
2/12 (16.67%)
Utility configuration issues
2/6 (33.339)
Comparison among SE agents and backbone LLMs. As shown in Table 2 SE agents with Claude-3.5-$ achieve better effectiveness than with GPT-4o in terms of plausible resolution, correct resolution, and localization accuracy. In particular, AutoCodeRover with Claude-3.5-4 achieves the highest resolution rate (i.e., 4.67%) and the highest function-leve localization accuracy (i.e., 19. 18%). Overall, observe larger potential of Claude-3.5-S in understanding agent issues than GPT-4o.
Figure 3 shows the unique and overlapped agent issues that are correctly resolved by each SE agent. An issue is counted as correctly resolved by agent if it was solved in at least one of the three experimental runs. We could observe that each SE agent can uniquely fix resolved by any other SE agents. In addition, there is agent issue that can be fixed by all SE agents. In other words, existing SE agents exhibit complementary capabilities to resolve agent issues.
bugs that cannot be
00.67
Agent Issues JAGENT BSSUE-BENCHI
Traditional Software Issues 15WE bench Lit)
30.67
Agentless
Agentless
Retotves
30-
23.00
18:33
1
4.67
NAA
2.67
2,00
Agenttess (o)146)
(Cluide-3 5
Agentiess
RunCodeRover
(GPT 46)
AutsCodeRRvv
(Eladde 2.555
SlSwE agont (EPT (EPT.40)
SHE-aget Cluuleee
Auto Code
SWE-Agent
Auto Code
SWE-Agent
Rover
Figure 4: Resolution rate of agent issues v.s. tradi- tional software issues.
Claude-3.5 Sonnet
GPT-4o
Figure 5: Venn diagrams of resolved issues.
Costs. As shown Table 2 the average costs of applying SE agents to agent issue are controllable, ranging from $0.05 to $1.15. The cost range is similar as applying these SE agents to resolve traditional software issues (e.g., $0.45 $2.53 [52)
5.3 Qualitative Results
In this section, we further break down the issues that SE agents can and cannot resolve, aiming to better understand their strengths and limitations in resolving agent issues. Table 3 presents the issue categories that can be resolved by at least one studied SE agent.
Resolved agent issues. Overall, the majority of agent issues resolved by SE agents are still related to utility (e.g., log/file operation/UTI, which actually share high commonality with traditional software systems. As result, SE agents are inherently able to resolve issues of this category in agent systems. Moreover, besides common utility issues, some of the dependency issues on agent-specifer components (e.g., tool) can also be resolved by SE agents. The reason why SE agents can handle such agent issues might that the dependency issues often contain explicit error messages (e.g., missing libraries or incompatible variables/interfacee agent components (e.g., tool), they can still be similar to dependency issues in other general software components, which are straightforward and informative to resolve.
As result, even if the dependencies are unique to
Unresolved agent issues. Overall, the majority of agent-specific issues cannot be resolved by any SE agent. For example, SE agents resolve a very few (or even none) issues LLM provider incompatibility, memory, or LLM operation. The reason might be that the exchanges with LLM providers are unique features in agent systems and agent systems are emerging in the recent period, which thus are less covered in the LLM training data. In addition, the autonomous and flexible nature of agent systems stemming from LLMSs makes it challenging to identify the root causes of LLM operation issues. Figure and Figure 7 in Appendix F show two unresolved issues for which all SE agents cannot even correctly localize the buggy files.

---

In summary, our analysis further confirms the limitations of existing SE agents in resolving the agent issues which are particularly related agent-specific features, highlighting the necessity of building more advanced SE agents for maintaining agent systems.
Limitation and Future work
While AGENTISSUE-BENE is representative of real-world agent issues by covering wide range of different categories, the generality of our findings can still be restricted due to the data source and the benchmark scale. First, our work only focuses on reactive agent issues (i.e., fist-expose-then-fix as our data source (i.e., uspr-reported Github issues) inherently captures problems reported by agent users. This scope intentionally excludes other maintenance aspects such as preventative strategies proactive LLM monitoring) and performance optimization, which are typically observed from the developers' perspective using internal logs. Second. the current benchmark size is limited as reproducing issues in agent systems is significantly more challenging than in traditional software systems. Due the nondeterminism of LLMs and changeable external resources (e.g., tools and LLM providers) interacted with agent systems, only a small number of agent issues (50 out of 201 issues) can be successfully reproduced. Moreover, huge manual effort (approximately 500 person-hours) are dedicated to preparing the Docker environment, configuring agent systems, and writing failure- triggering tests. In the future, we plan continuously maintain and extend our benchmark to support future research on agent system maintenance. The continuous work of our benchmark is available at our website [6]. In particular, the benchmark has been extended with 20 more reproducible issues since the paper submission time. Similar trends (i.e., poor resolution rate) can be observed in those additional issues. Detailed results are presented in Appendix H
Discussion. Based on our findings, we further discuss implications for future research towards building more effective SE agents for resolving agent issues. (i) Adding a knowledge base agent- needed external resources. Our findings show that existing SE agents struggle with issues related to external resources. A promising direction is to augment agents with evolving knowledge base built from API documentation, release notes, and historical issues. Integrating this knowledge could empower SE agents to better reason about and diagnose the issues related to external resources. (ii) Training SE agents with instances and trajectories collected from AGENTIsSUE-BENCC Our benchmark and study provide training data specifically on the emerging agent systems. As our work provides executable environments and tests of buggy/fixed agent systems, future work can collect instances and trajectories (c.g., agent-enviromment/ool interactive trajectories) for fine-tuning more powerful SE agents that specifically targets at agent issue resolution. (iii) Adding a dynamic analysis component in SE agents. Our results highlight the limited localization accuracy of current agents, suggesting a large gap between an issue description and its root cause. To address this, future SE agent architectures could move beyond static analysis and incorporate a dynamic analysis component. By utilizing runtime information like execution trajectories and tool outputs, the agent can gather richer signals for more accurate bug localization and patch generation.
7 Conclusion
In this work, we analyze 201 GitHub issues from 18 real-world agent systems and construct the first taxonomy of agent issues. We further build WENTIISUE-BENCCC the first reproducible benchmark of 50 high-quality agent issue resolution tasks. Experiments on state-of-the-art SE agents demonstrate their limited effectiveness in addressing agent issues (with resolution rates ranging from 0.67% to 4.67%), highlighting the unique challenges in maintaining agent systems and the pressing need for more advanced SE agents tailored to this emerging software paradigm.
References
[1] Gpt-4o, 2024.
[2] Openai, 2024. https://chat.openai.coo
[3]
Swe-bench wwwprrinnnsswe-bbbee-eieerriaeee
verified, 2024.
https://openai..com/inn
[4] agent-squad, 2025. https///itthbbboom///wwsaab//aas
10

---

[5] Agentissue-bench, 2025.
httpss//anooyooss.opee.......ss
Bgent
[6]
2025.
[7] Agixt, htto::///github.ccoom///o//11177
[8] ai, 2025.
htt.ss///githubb.co////eee////
[9] Aider, 2025. https://aider.chat/
[10] Anthropic, 2025,
https:/www.anttrrpicc.comm/
[11] autogen, 2025.
https://gtthho.cc////rrroffffrat
[12] babyagi, 2025.
https://ssttuuuooo//////eeean/nr
[13] camel, 2025,
httpss//ggthbb.ommmmmmmaaaaaa/aa
[14] Chatdev, 2025. https://gghhub..co//////es//tta
[15] Claude 3.5 sonnet, 2025.025..1
sonnset
[16] Crewai, 2025. htt.:s//ggstho..ccmm/ceeee//////
[17] Deepseek, 2025. https///ww.ddeesseek.ccmm
[18] Devin, 2025. https://devin.ai:
[19] evo.ninja, 2025. http::/ggthbb..oo/eggccccioooooo
[20] Expensify website, 2025. https://ww.exxensstt.com///
[21] gpt-engineer, 2025.
[22] gpt-researcher, 2025, http:://gttto.ccoossssssooo///ee
[23] haystack, 2025. http:://ggtthbbcco///eeeettttt//
[24] Introducing the www.h/ccccboccoonookoooo///////o
lancer/
[25] lagent, 2025.
httpss//gtthhhooo///////////////
[26] metagpt, 2025.
[27] Moatless, 2025.
httpp://ggthho.coo//oowo//////ss
[28] Opendevin, 2025.
https://ggictbbb.ccm///eeeeeeiin
[29] pythagora, 2025.
[30] Rugaai-catalyst, 2025.
[31] superagent, 2025.
[32] Swe-agent, 2025. httpss//ggthuoooooo/////gaggggg/
[33] Swe leaderboards, 2025, https::/www.wwebbttt com.
[34] M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. G. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and Stoica. Why do multi-agent LLM systems fail? CoRR, abs/2503. 13657. 2025.
[35] Z. Chen, Cao, Y. Liu, Wang, T. Xie, and X. Liu. A comprehensive study on challenges in deploying deep learning based software. In 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/ESE 2020, pages 750-762, 2020.

---

[36] Z. Chen, H. Yao, Y. Lou, Y. Cao. Y. Liu, H. Wang, and X. Liu. An empirical study on deployment faults of deep learning based mobile applications. 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, pages 674-685, 2021.
[37] B. Glaser and A. Strauss. Discovery of grounded theory: Strategies for qualitative research. Routledge, 2017.
[38] J. Gottweis, W. Weng, A. N. Daryin, T. Tu, A. Palepu. P. Sirkovic, A. Myaskovsky, F. Weis- senberger, K. Rong. R. Tanno, K. Saab, D. Popovici, J. Blum, F. Zhang, K. Chou, A. Hassidim, B. Gokturk, A. Vahdat, P. Kohli, Y. Matias, A. Carroll, K. Kulkarni, N. Tomasev, Y. Guan, Dhillon, D. Vaishnav, B. Lee, T. R. D. Costa, R. Penadés, G. Peltz, Y. Xu, Pawlosky,
A. Karthikesalingam, and V. Natarajan. Towards an Al co-scientist. CoRR, 2025.
[39] N. Jiang, T. Lutellier, Y. Lou, L. Tan, D. Goldwasser, and X. Zhang. KNOD: domain knowledge distilled tree decoder for automated program repair. In 45th IEEE/ACM International Conference on Software Engineering. ICSE 2023, Melbourne, Australia, May 14-20, 2023, pages 1251-1263. IEFE, 2023.
[40] C. E. Jimenez, Yang, A. Wettig, S. Yao. K. Pei, O. Press, and K. R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[41] S. S. Kannan, V. L. N. Venkatesh, and B. Min. SMART-LLM: smart multi-agent robot task planning using large language models. In IEEE/RSJ International Conference Intelligent Robots and Systems. IROS 2024, Abu Dhabi, United Arab Emirates, October 14-18, 2024, pages 12140-122 12147. IEEE, 2024.
[42] J. Liu, K. Wang, Y. Chen, X. Peng, Z. Chen, L. Zhang, and Y. Lou. Large language model-based agents for software engineering: A survey. CoRR, abs/2409.02977 2024.
[43] F. Long and M. Rinard. An analysis of the search spaces for generate and validate patch genera- tion systems. In Proceedings of the 38th International Conference on Software Engineering, ICSE 16, page 702-713, New York, NY, USA, 2016. Association for Computing Machinery.
[44] S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke. Swe-lancer: Can frontier Ilms earn $1 million from real-world freelance software engineering? CoRR, 12115, 2025.
[45] J. Petke, M. Martinez, M. Kechagia, A. Aleti, and F. Sarro. The patch overfitting problem in automated program repair: Practical magnitude and a baseline for realistic benchmarking. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, page 452 -56, New York, NY, USA, 2024. Association for Computing Machinery.
[46] Y. Qin, S. Wang, Y. Lou, J. Dong, K. Wang, X. Li, and X. Mao. Agentfl: Scaling llm-based fault localization to project-level context. CORR, abs/2403.16662. 2024.
[47] H. Qiu and Z. Lan. Interactive agents: Simulating counselor-client psychological counseling via pole-playing lIm-to-llm interactions. arXiv preprint arXiv:2408. 1587:220...7777, 2024.
[48] C. B. Seaman. Qualitative methods in empirical studies of software engineering, 1EEE Transactions on software engineering, 25(4):557-572, 1999.
[49] Y. Shao, Y. Huang. J. Shen, L. Ma. T. Su, and C. Wan. Are Ilms correctly integrated into software systems? In 2025 TEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 741-741. IEEE Computer Society, 2025.
[50] L. Wang, C. Ma, X. Feng. Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and Wen. survey on large language model based autonomous agents. Frontiers Comput. Sci., 18(6):186345, 2024.
12

---

[51] Z. Xi, W. Chen, X. Guo. W. He, Y. Ding, B. Hong, M. Zhang. J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan. X. Wang. L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin.
S. Dou, R. Weng, W. Qin, Y. Zheng, X. Qiu, X. Huang, Q. Zhang, and T. Gui. The rise and potential of large language model based agents: survey. Sci. China Inf. Sci., 68(2), 2025.
[52] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying llm-based software engineering agents. CoRR, abs/2407.0144 2024.
[53] C. Xia, Y. Ding, and L. Zhang. The plastic surgery hypothesis in the era of large language models. In 38th IEEF/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023, pages 522-534. IEEE, 2023.
[54] C. Xia and L. Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In A. Roychoudhury, C. Cadar, and M. Kim, editors, Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022. pages 959-971, ACM, 2022.
[55] C. S. Xia and L.. Zhang. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. In M. Christakis and M. Pradel, editors, Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, pages 819-831. ACM, 2024.
[56] Y. Xiao, J. Huang, R. He, J. Xiao, M. R. Mousavi, Y. Liu, K. Li, Z. Chen, and J. M. Zhang. AMQA: an adversarial dataset for benchmarking bias of Ilms in medicine and healthcare. CoRR, abs/2505. 19562,, 2025.
[57] B. Yang and J. Yang. Exploring the differences between plausible and correct patches at fine-grained level. In 2020 IEEE 2nd International Workshop on Intelligent Bug Fixing (IBF), pages 1-8, 2020.
[58] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering. CoRR, 157993 2024.
[59] Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. R. Narasimhan, D. Yang. S. Wang, and O. Press. Swe-bench multimodal: Do Al systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
[60] Q. Yang. Z. Wang. H. Chen, S. Wang, Y. Pu, X. Gao, W. Huang, S. Song, and G. Huang. Psychogat: A novel psychological measurement paradigm through interactive fiction games with LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, August 11 1-16, 2024, pages 14470-14505, 2024.
[61] Z. Yuan, W. Chen, H. Wang, K. Yu, X. Peng, and Y. Lou. TRANSAGENT: an Ilmebased multi-agent system for code translation. CoRR, 198994 2024.
[62] Z. Yuan, W. Chen, H. Wang, K. Yu, X. Peng, and Y. Lou. TRANSAGENT: an Ilm-based multi-agent system for code translation. CoRR, abs/2409, 19894, 2024.
[63] Z. Yuan, M. Liu, S. Ding, K. Wang, Y. Chen, X. Peng, and Y. Lou. Evaluating and improving chatgpt for unit test generation. Proc. ACM Softw. Eng., ((FSE):1703-17)
2024.
[64] D. Zan, Z. Huang, A. Yu, S. Lin, Y. Shi, W. Liu, D. Chen, Z. Qi. H. Yu, L. Yu, D. Ran, M. Zeng, B. Shen, P. Bian, G. Liang, B. Guan, P. Huang, T. Xie, Y. Wang, and Q. Wang. Swe-bench-java:
A github issue resolving benchmark for java. CoRR, abs/2408. 14354, 2024.
[65] F. Zeng, W. Gan, Y. Wang, N. Liu, and P. S. Yu. Large language models for robotics: A survey. CoRR. abs/2311 12t/2311.07222 1.07226, 2023.
[66] Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury. Autocoderover: Autonomous program improvement. In M. Christakis and M. Pradel, editors, Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20. 2024, pages 1592-1604. ACM, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize contributions and scope at the end of introduction section (Section D and the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6 discusses the limitation of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper mainly focuses empirically evaluating the agent issue resolution
capabilittis of SE agents.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our replication package (including both the benchmark and the code) is available at 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our replication package 5 includes the data and code for reproducing the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see NeurIPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips wttpccccpppcccccecccccccccccccct
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have specified the data splits Section 3.1.2 and the hyperparameters in and Section and Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have presented the mean values along with their corresponding two- sigma(#22r) errors for the main experimental results in Table and explained the detailed calculation method in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Table 2 reports the average token/money needed run experiments. Our experiments solely on online LLMs and thus do not impose strict requirements on computational resources
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://///eerrss.../p///////ttts
Answer: [Yes]
Justification:
The paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work empirically evaluating SE agents in solving agent issues, which does not involve any societal impact.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper mainly focuses on an empirical study.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited and respected the license of our analyzed agent systems in Table
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, (paper swierswir thcode. com coom/aaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces new issue resolution benchmark AGENTISSUE- Its statistics are described in Section 4.2 and in our replication package [5].
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Appendix shows the details in human-invol tasks, including the instruc- tions and compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Human-invol involving tasks were approved by the Institutional Review Board (IRB) at our institution (as mentioned in Appendix C).
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The paper empirically studies LLM agents and describes the usage of LLMs in Se.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy http::/s//merriossccoooooooeeeee for what should or should not be described.
20

---

A Analyzed agent systems
Table presents the full list of analyzed agent systems and their statistics.
Table 4: Statistics of analyzed agent systems
Agent
#stars
#Loc
Creation time
License
agent-squad
5.4k
109,019
07/23/2024
Apache-24 License
AGiXT
111,946
04/04/2023
MIT License
SDK
365,300
05/23/2023
Apache-2.0 License
autogen
44.2k
197,969
08/18/2023
CC-BY-4.0, MIT License
camel 3
12.4k
206,152
03/17/2023
Apache-2.0 2.0 License
babyagi 12
21.4k
8,800
04/03/2023
MIT License
CrewAl 16
31.3k
171,395
10/27/2023
MIT License
Haystack [23
22.9k
180,500
11/14/2019
Apache-2.0 License
Lageett 225
2.1k
13,075
08/20/2023
Apache-2.0) License
MetaGPT 26
55.4k
90,709
06/30/2023
MIT License
RagaAl-Catalyst
30
16.2k
47,252
10/01/2024
Apache-2.0 License
ChatDev [4
26.8k
40,478
11/04/2023
Apache-2.0 License
gpt-engincer 21
54.1k
17,460
04/29/2023
MIT License
Pythagora 29
1.8k
5,859
01/21/2023
Apache-2 2.0 License
SWH-agent
32
15.7k
63,388
04/02/2024
MIT License
evo.ninja 19
1.1k
31,862
08/18/2023
MIT License
Superagent [1
5.8k
58,602
05/10/2023
MIT License
gpl-researcher
22
21.3k
168,849
05/12/2023
Apache-2.0 License
B Patch Scales of AGENTISSUE-
Table presents the statistics of the patch scales in AGENTISSUE-
C Human Participation
All nlmmnn-nnollved tasks in our experiments (including taxonomy construction, taxonomy evaluation, and issue reproduction) were approved by the Institutional Review Board (IRB) our institution. Additionally, all participants were compensated at rate of $15 per hour.
In taxonomy evaluation, each human annotator is provided with the following instructions: "Given the taxonomy of agent issues (along with the definition of each agent issue category), please label each agent issue with any category the taxonomy. If there are no applicable categories in the given taxonomy, please return the label as non-applicable."
D Experiment Statistical Significance
Table o presents the mean results of SE agents on AGENTISSUE BEENTISSUE-BNNCI, BENCII, along with their corre- sponding two-sigma (#:2r) errors. To calculate the two-sigma errors, we conducted the experiment on AGENTISSUE-BENT BENCH three times, computed the standard deviation for the results and multiplied by 2, shown in Equation [
20 2
Σ x 1)2
(I)
where N is the number of experimental runs, x; is the result of the i-th run, and F is the mean result.
E Examples of Unresolved Issues
In this section, provide two issue examples that all SE agents fail to localize the buggy files and generate correct patches.
For the example depicted in Figure 6 the agent lacks up-to-date knowledge regarding which LLMs currently support the "stop" parameter. As a result, the agent incorrectly passed the "stop" parameter to LLMs that do not support (i.e., ol or-rreveew and ol minin,,, ultimately aggravating the issue.
21

---

Table 5: Mean and maximum values for various patch attributes in studied agents
Attribute
Mean
Max
Lines edited
66.05
355
Files edited
3.58
34
Functions edited
6.79
54
Table 6: Mcan results with 2-sigma( (22) errors of SE agents on AGENTISSUE- BENNHH
Plausibly resolved %
Correctly resolved %
Localization %
SE Agent
LLM
File-level
Function-level
GPT-4o
12.00 (=4.00)
3.33 (#4.62)
27.82 (4:5.51)
12,99 (=2.78)
Agentless
Claude-3.5-S
12,00 (=0.00)
4.00 4.00 (=0.00)
27.35 (+0.00)
17.50 (=0.00)
GPT-4o
7.33 (12.31)
1.33 (=2.31)
22.07 (#7.21)
14.77 (=2.62)
AutoCodeRover
Claudo-3.5-S
12.67 (+6.11)
4.67 (+2.31)
25.81 (+11.43)
19.18 (+5.54)
GPT-4o
(+2.31)
0.67 (+2.31)
11.67 (+5.177
4.22 (+44.07)
SWF-agent
Claude-3.5-S
2,00 (100.00
2.00 (=0.00)
9.52 (15.244
6.78 (42.59)
For the example depicted in Figure the agent fails identify the root cause of the KeyError, Le., a conflict arising from generating multiple diffs for a single file. This issue is specific to the agent system, as it involves the handling of model outputs. However, instead of performing deeper analysis, the agent merely prints an error message, resulting an unsuccessful patch.
Issue
Repository: CrowA)
Issue 1323
ID: 1322
defined the model to s1-preview or ø1-mini and temperature to 1, get following error:
Desoription:
400 f'error: ('message:: "Unsupported parameter: 'stop is not supported with this model.", 'type 'invalid_requuee certid_reuue_eereee 'param "stop 'code "unsupported parameter ']]
Errrr code:
Apparently stop parameter used, way for the crew to no this parameter.
supported. didn't
Golden Patch
Generated Patch (Agentiesss Cloude 3.6-6)
ww/lrewwtiegennss/coo agant, esacutor py
arojerewal/prjjec//erwu base.py
use stop _ro__stop_oddd: words: bool,
def s.ould eew.eexc_eeeeeee parpmpprr
omit unrelated
(self) bool:
self seLf.use_stae uus.uuusppppp. words
hasattr(
use sttpsstoortre words
self. Lim, 'model_nane
# omit unrelated Lines
):
answer LLMC
model name self,
self.lim.
model_ mane__nmee. Lower()
stop-self.stor
return any(nane in model_ name
self use utt_tuss_stpp_eee words else None,
name [' d'al-preview', mini'])
callbacks-self. cabbacks.
J.cal[(seL..messaees
Figure 6: This unresolved issue arises because not all LLMs support the 'stop' parameter, requiring users to control its use (e.g., via use_stop_wone words in the Golden Patch). The agent- agent-generated patch aggravated the issue by passing 'stop" to unsupported models (i.e., o1-preview and ol-mini).
Examples of Issues in Different Categories
In this section, we provide detailed issue examples of each sub-category in Table For each issue, we provide the repository name, the user-pu -rovvdedd issue description and the summarization of the developer-commet patch (along with the link the original issue and Pull Request (PR) pages).
F.1 Incompatibility with LLM providers
Incompatible dependencies
Repository: gpt-reseurcher
Link to the Issue: http://hhhhooossssddoeegerreeert
22

---

Issue
Repository:
gpt-engineer
Issue ID: 1197
PR ID: 1198
create Java Spring project with gpt-engineer first, and want to separate applicationyyaml two env config files Error:
Description:
File "gpt. inginner/corn//dffulttsstt
line 318, _improve
files, files_dict, errors salvage, horrect.hunkkmmese
files_dict,
files. dict, memory)
"gpt_enge wwwp_onnnne/oo///eeeaatttaeeeeee fio_to_lines,
py", 351, salvage__orreed htrple_corre__hheeee hunkss too_ines_dictiiills dictifiles, diiitfcifffillenaaae fiwennneeeeeppe
KeyError: "ecmmaliresoooeeeepplllioooensee
Golden Patch
Generated Patch (Agentiess+ Claude 3.6-8)
onginearrloom//haa.. to py
unrelated Lines diff
Only validate the file
exists in iiles_dict
farse_diff_ block(ditt block)
diff.filename_prs in files #tles_dict:
for filename, diff_ob] in
difft.item()::
omit unrelated lines
if filename not in diffs:
else:
diffs[tilename]] diff_obj
erroz_men lrss__essaggeseee appeedd
else:
f"Warning: File
print(f" print(e""Ahiiltplle
(diff. filename, pre]] repesitory"
diffs found for (filenams).
Only the first k)
Figure 7: This unresolved issue stems from the LLM generating multiple diffs for single file, resulting in conflicts. The Golden Patch resolves this by retaining only the first diff. contrast, the agent-generated patch fails to investigate the root cause and simply logs an enror message.
Link to the PR: httpps//whhb..oooo/sssooo////ees
Issue Description:
Testing revealed that the invocation method for token_counter and related
functions in Claude has changed, requiring verification.
Fix Strategy: Update the version of anthropic library and use the latest released APIs.
2. Unsupported models
Repository: ChatDev
Link to the Issue: http://ggttti..ooomoo/o///////ls
Link to the PR: https///gghbbbbbommoopemm/////Ca
Issue Description: Can't 't do anything with 3.5 turbo. The code it makes is brutal. Can it be possible to add GPT 4 Turbo?
Fix Strategy: Update the version of openai library and add support for GPT-4 Turbo.
3. Incompatible parameters to LLM providers
Repository: CrewAl
Link to the PR: https://gghhbboooommmrwww//lll//
Link to the Issue:
Issue Description: defined the model ol -preview or ol-mini and temperature to 1, and
get the following error. Unsupported parameter: "stop" is not supported with this model. Apparently the stop parameter is used, and is not supported. didn't 't find a way for the crew no use this parameter.
Fix Strategy: Added the option uss_stop_woods to allow users to configure whether to use the stop parameter.
F.2 Tool-related issues
Tool dependency issues
Repository: lagent
Link to the Issue: https//ggthhb.oom//mmmmm///////s
Link to the PR: http:///ttiiccco//mmmmm////a//uu
23

---

Issue Description: When running the agent with web search capabilities, an error occurred. ModuleNotFoumdEror No module named "tenacity"
Fix Strategy: Add tenacity in the dependency configuration file.
2. Tool configuration issues
Repository:
gpt-researcher
Link to the PR: http://iihinnoooosssddodogooeorr
Link to the Issue:
Issue
Description:
It looks
like
cannot
set
RETRIVER solely exception about
to
duckduckgo or others.
It
always throws
an
Exception: Tavily API key not found. Please set the TAVILY API KEY environment variable.
Fix Strategy: Ensure the retriever set up according to the user's configuration specified via environment variables.
3. Tool implementation issues
Repository:
SWE-agent
Link to the PR: http::/gghhiaaooooSSSSegeeeeeees
Link to the Issue:
Issue Description: If the agent tries to cat out the content of word file (.docx), then line
buffer.eecode( fails, and the program crashes.
Fix Strategy: Replace buffer.decode() with buffer.decode""utt ("ttff8""""""""" uutt-u,,erros==bbacckaiiii crros=""acckklahrrplaec"" that the program will not crash when reading non-utf8 encoded bytes.
4. Misuse tool interfaces
Repository: camel
Link to the Issue: wttp:ghhuubb.oooo/mmmaaaaaaaaaaa
Link to the PR:
Issue Description: When "KAUST" is the entity word to be searched, the returned result by
wikipedia API (wikipedia.summary) is the summary about KAIST.
Fix Strategy: Set the auto sugget parameter to False when invoking wikipedia.summarry)
that does not change the search word (such as KAUST -> KAIST)
F3
Memory-related issues
Memory initialization issues
Repository:
CrewAl
Link to the Issue: http://sttttlccoocccccrww/llll/s
Link to the PR: http::/gthhhbboooomcccwwlllllled
Issue Description:
Looks
like
reset-memories
is throwing an error on -a
An unexpected error occurred: No crew found.
Fix Strategy: Fix the get_crew method to obtain the correct crew instance, and ensure that memory is only reset when it None
Memory content issues
Repository: camel
Link to the Issue: http:///w....aa/a//a////////////
Link to the PR:
Issue Description: Current (memory storage) logic checks the content in the chunk, the content is None then the message would be appended, but for some API like SambaNova. there may include many None content in chunks in the middle of response. We need change the logic, checking choice.finish reason then append the message would be better.
Fix Strategy: Update the logic for storing messages. Determine whether a chunked message has been fully stored by checking if the finish reason attribute is not None

---

3. Memory dependency issues
Repository:
anlogen
Link to the Issue: http://sttthl.oooo/ooooos/ssusgs
Link to the PR: http::/ghhhhb.ccuchccoooo////agr
Issue
Description:
Running autogenstudio ui -port 8081
fails
with
ImportError:
cannot import name "InnerMessage"" from "autogen_agentchat. messagges
Fix Strategy: Since 'InnerMessage' has been renamed to 'AgentMessage', all references to "InnerMessage" are renamed to 'AgentMessage'.
E.4
LLM operation issues
Model access misconfiguration
Repository: camel
Link to the Issue: http://giiiiinooommmmmmma/a/i/mi
Link to the PR: http:://ghtt...coooooom/////////
Issue Description: The decorator api weckkkyysssuuuiieeeeeee currently only supports setting the
value of "DUMMY_TOKEN)" in the environment variable, but does not support directly calling
DummyCCasstpi__kg==axxxx
Fix Strategy: Refactor the api_keys_required( decorator and make it compatible with the method of directly setting API key.
2. Token usage misconfiguration
Repository: camel
Link to the Issue: http::/ggiiaa.cooommmmmmmml////p
Link to the PR: http:s//ghhabbooommmmmmlmi/a///p
Issue Description: For LIM API served by OpenAll-oopptible providers, if the max_ max_tokens
is not provided then it would use NOT_GIVEN from openai, which will lead to
TypeError: '<=" not supported between instances of 'int' and 'NotGiven'
Fix Strategy:
Unify the default loken_limin property in the base model class to make sure provided for different models.
3. Incorrect model output handlers
Repository: MetaGPT
Link to the Issue: http:://ttttt..ooocccccaaa/a///t
Link to the PR: http://dshhlbbcoomeeeeuu///a/a/t
Issue Description: When running python3 debate.py "Talk about Artificial General Intelligence"
an error occurs: ValueError: The response.tex quick accessor only works for simple (single-Part) text responses
Fix Strategy: The root cause that the Gemini model flags the request as poten-
tially involving sensitive or harmful content and blocks it. To address this scenario, the BlockedPromplExcppion has been added to catch exceptions triggered by blocked prompts.
Model dependency issues
Repository:
lagent
Link to the Issue: http:///ghhhhhooomm/mmmnn////iis
Link to the PR: http://ggthbbcommmmnmn./////gggh
Issue Description: Encounter AttributeError: 'GenerationConfig object has no attribute "_cos_ coken tensor'
when running code in the transformers library.
Fix Strategy: Update the version constraint of transformers avoid conflict.
5. Context length issues
Repository: resereserrrher
25

---

Link to the Issue: http:y/hhhicooooosssddoooorrrtrr
Link to the PR: http::/gghhhb.ooo.aaaaaiiiiieeed
Issue Description: Exceed maximum context length error in encerate_rporte
Expected a string with maximum length 1048576, but got a string with length 1304783 instead.
Fix Strategy: Limit context to 25k words (with safety margin) by trimming older records while keeping recent and relevant ones.
Prompt-related issues
Repository: gpt-researcher
Link to the PR: http://hhh...ooooissdddooopopoop
Link to the Issue:
Issue Description: "Introduction" and "Conclusion" sections remain in English even when
LANGUAGE is set to a different language (e.g., "japanese") in the configuration.
Fix Strategy: Update the prompts for "Introduction" and "Conclusion" generation include language specification instructions.
F5
Workflow issues
Workflow issues
Repository:
CrewAl
Link to the Issue: http:///tttlbooooocccc.wl///////
Link to the PR: http://gdtbb..coom/eeee////lllee
Issue Description: Execution fails for steps with multiple preceding parallel steps.
Fix Strategy: Modify the asynchronous listening mechanism ensure that subsequent steps can proceed smoothly after the preceding parallel steps are completed.
F6
Utility issues
Utility implementation issues
Repository:
SWE-agent
Link to the Issue: http:///hhii.ioo.S.///eeeeee///a
Link to the PR: http://giibbbocooooaaaaaaa//////
Issue Description: In the React frontend framework, the default value of a textbox is supposed
update based another selected dropdown item, but this dynamic binding is not functioning
as intended.
Fix Strategy: Use useEffect to monitor the change of the dropdown item, and when it changes, reset the textbox display the default value.
Utility dependency issues
Repository: autogen
Link to the Issue: https//gggttlooooommmcoooo//ttgs
Link to the PR: httpss//ginab..ommmmccooo//////p
Issue Description: Running pytest with the latest version 8.0.0 released an hour ago is not working.
Fix Strategy: Limit version of pytest to under 8.0.0
Utility configuration issues
Repository: CrewAl
Link to the Issue: https//stthbboooocccccvl//leeees
Link to the PR: httpss//gghaabo.coom/crwwwlllllr
Issue Description: When loading a variable from the yaml environment, the crewai library tries load in gbk encoding
Chinese is present in it, but my yaml file is utf-8. You want to add a measure to detect file encoding when loading yaml files.
Fix Strategy: Explicitly specify UTF-8 encoding when reading the YAML file.
26

---

Table 7: Overall results of SE agents AGENTISSUE-BENCE with task difficulty
Plausibly resolved %
Correctly resolved %
Localization
SE Agent
LLM
File-level Function-level
GPT-4o
11.46
3.51
27.25
12.08
Agentless
Claude-3.5-S
11.32
3.77
26.82
16.88
GPT-40
6.84
1.40
21.62
14.37
AutoCodeRover
Claude-3.5-S
11.58
4.33
25.21
18.46
GPT-4o 0.70
0.70
11.26
3.85
SWE-agent
Claude-3.5-S
2.11
2.11
9.15
6.40
Table 8: Pass vs. @ass@3 AGENTISSUE-BENT BENCH
GPT-44o (%) AutoCodeRover
Claude-3.5-S (%) RutoCodeRover
Pass@k
Agentless
SWE-agent
Agentless
SWE-agent
Pass@1 6.00
2,00
2.00
4.00
6.00
2.00
Pass@3
6.00
2.00
2.00
4.00
6.00
2.00
G Comparison among SE agents with various metrics
further evaluate the fixing capabilities of SE agents by considering task difficulties. In particular, we consider the utility errors as of low severity/difficulty while the other agent-specific issues as of high severity/difficulty Table presents the weighted resolution rate (0.2 for utility errors and 0.8 for other agent-specific errors) among SE agents. Overall, could observe similar findings between weighting and non-weightn noigwhigt results, including (1) limited capabilities of agents in agent issue resolution, (2) the superiority of AutoCodeRover with Claude-3.5-5 and (3) outperformance of Claude-3.5-S over GPT-4o.
Table g presents the pass and pass @3 over one run. Overall, we could observe consistent trends on these metrics our current findings, including (1) overall limited capabilities of agents in agent issue resolution, (2) the superiority of AutoCodeRover with Claude-3.5-S, and (3) outperformance of Claude-3.5-S over GPT-4o.
H Extended AGENTISSUE-BENT
Table 2 presents the results of SE agents on the 20 more issues 6 that are additionally reproduced after the paper submission time. Overall, could observe similar trends these additional issues that SE agents exhibit limited capabilities of resolving agent issues (i.c., up to 5% correct resolution rate).
Table 9; Results of SE agents additional issues
Plausibly resolved %
Correctly resolved %
Localization File-level Function-level
SE Agent
LLM
GPT-4o
5.00
0.00
10.00
6.67
Agentless
Claude-3.5-S
5.00
0.00
12.50
6.67
GPT-4o
5.00
0.00
12.50
5.83
AutoCodeRover
Claude-3.5-S
5.00
5.00
20.83
11.25
GPT-40
0.00
0.00
0.00
0.00
SWE-agent
Claudc-3.5-S
0.00
0.00
0.00
0.00
27


Reviews:
[{"review_id": "t3Fzcj0dCK", "reviewer": "Reviewer_61d1", "summary": "In this paper, the authors  investigates whether state-of-the-art software engineering (SE) agents can effectively resolve issues in LLM-based agent systems. Specifically, the authors manually analyze 201 real-world agent issues from 16 widely-used agent systems and categorize them into a taxonomy with 6 main categories and 20 sub-categories. deriving a taxonomy of agent issues with 6 categories and 20 subcategories. The authors find that key challenges unique to agent systems include LLM operation errors, tool dependencies, memory-related issues, and workflow anomalies. The authors also construct AGENTISSUE-BENCH , a benchmark of 50 agent issue-resolution tasks with Docker environments, failure-triggering tests, and ground-truth patches. The study uncover the need for specialized SE agents tailored to the complexities of LLM-based systems, such as handling non-deterministic LLM outputs, dynamic tool interactions, and intricate memory mechanisms.", "questions": "See weakness 1 and 2", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "strengths:\n\n1. Comprehensive empirical study: The authors conduct an empirical study of 201 real-world GitHub issues from 16 widely used agent systems (e.g., MetaGPT, AutoGen)\n\n2. Comprehensive category: The first systematic categorization of agent issues, derived from 201 manually analyzed real-world examples, provides valuable insights into the unique challenges\n\n3. Benchmark is available: AGENTISSUE-BENCH addresses a major barrier in agent system research by offering a standardized, executable evaluation framework with Dockerized environments and test cases.\n\n\n4. Comprehensive SOTA evaluation on the benchmark: Evaluates multiple SE agents (Agentless, SWE-agent, AutoCodeRover) with advanced LLM backbones (GPT-4o, Claude-3.5-Sonnet), demonstrating consistent limitations across tools.\n\n5. Provide insightful thoughts towards multi-agent system design. The paper highlights the urgent need for specialized SE agents capable of addressing the interplay between LLMs, external tools, and system workflows. It provides open access to data, code, and benchmarks, fostering transparency and reproducibilit\n\n\nweakness:\n\n1. While the paper emphasizes issue diagnosis and resolution, it does not explore complementary aspects of agent system maintenance, such as preventative strategies (e.g., proactive LLM monitoring) or performance optimization (e.g., reducing tool dependency overhead). A broader perspective could offer a more holistic view of challenges and solutions.\n\n\n2. Limited Discussion of Solutions to Identified Challenges: Can the authors provides some discussions about how to solve the challenges that they uncover? The paper excels at diagnosing problems and more concrete solutions are also expected.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the recognition and constructive feedback for our work. \n\n**1: \"While the paper emphasizes issue diagnosis and resolution, it does not explore complementary aspects of agent system maintenance, such as preventative strategies (e.g., proactive LLM monitoring) or performance optimization (e.g., reducing tool dependency overhead).  A broader perspective could offer a more holistic view of challenges and solutions.”** \n \nGreat point! We’d like to clarify that our current work primarily focuses on reactive issues (i.e., first-expose-then-fix), as our data source is *user-reported GitHub issues*, which inherently reflect problems that have already been encountered and reported by agent users. As a result, preventative strategies (e.g., proactive LLM monitoring) and performance optimization are not typically observable in our dataset, since these aspects are difficult to capture from *a user’s perspective*. Rather, they are more often observed from the developer’s perspective. To investigate such issues, a more suitable data source would be *internal logs from agent development and operations*, which are generally available only to industry practitioners or professional agent developers — thus beyond the scope and access of our current study. That said, we agree that these issues are important and represent promising directions for future work. We will clarify the scope and coverage of our study in the paper.\n\n**2.“Provide some discussions about how to solve the challenges.”**\n\nGreat point. Based on our study, we believe there can be several architectural improvements towards building more effective SE agents for diagnosing and resolving issues in LLM-based agent systems:\n* **Adding a knowledge base on agent-needed external resources.** As our study reveals that existing SE agents fall short in fixing external resource-related (e.g., tools, dependencies, or LLM providers) issues, it indicates that existing SE agents need to be enhanced with more up-to-date and evolving knowledge on how to use these agent-needed external resources. Therefore, one architectural improvement is to add and maintain an evolving knowledge base on all agent-related external resources (e.g., built on the API documentation, release notes, or historical issues), and then integrating with retrieval-augmented generation (RAG) could help SE agents better reason and diagnose the agent issues related to external resources. \n\n* **Training better SE agents with instances and trajectories collected from AgentIssue-bench.** Recent work[1] has shown that task instances and agent trajectories can help train more powerful SE agents for *traditional* software issue resolution (e.g., SWE-bench). Our benchmark and study further fill the gap by providing training data specifically on the emerging software system – agent systems. Specifically, as our work provides executable environments and tests of buggy/fixed agent systems, we can collect massive instances and trajectories (e.g., agent-environment/tool interactive trajectories) for fine-tuning more powerful SE agents that specifically targets at agent issue resolution.\n\n* **Adding a dynamic analysis component in SE agents.** As our study reveals the limited *localization accuracy* of existing SE agents in fixing agent issues, it indicates that one big  challenge for agent issue resolution lies in the huge gap between the issue description and root causes. However, existing SE agents only analyze the *static information* (i.e., error symptoms in  issue descriptions) for issue resolution, leaving massive important runtime information untapped. To tackle this limitation, SE agents should add a dynamic analysis component, which can utilize the runtime and intermediate outputs during agent operation (e.g., trajectories, fine-grained outputs of each key step like tool invocation, memory updating). Such a dynamic analysis can provide more informative hints for accurate issue localization and patch generation.\n\nWe believe these topics above would  provide actionable guidelines for future work towards building more powerful SE agents. We’ll include these discussions  into our paper.\n\n---------------\nReference: \n\n[1] Training Software Engineering Agents and Verifiers with SWE-Gym. Pan et al.  ICML 2025."}, {"review_id": "xAyRkviRVJ", "reviewer": "Reviewer_WCGa", "summary": "The paper derives a six‐category taxonomy from 201 real GitHub issues in LLM-based agents, then builds AgentIssue-Bench, a 50-task Dockerized benchmark of reproducible bugs and patches.\nThe authors evaluated three SE agents (Agentless, AutoCodeRover, SWE-agent) with GPT-4 and Claude, finding only 3.33–12.67% successful fixes.", "questions": "How do the authors plan to grow AgentIssue-Bench beyond these 50 scenarios? Could parts of the Docker/test setup be automated, or could the community help contribute new tasks?\n\nWhat concrete steps do you take to keep each task reproducible as LLM outputs and external tools change? Do you pin random seeds, snapshot dependencies, or filter outputs?\n\nHave the authors thought about varying task difficulty or severity? For instance, would an “LLM operation” bug count differently than a simple utility error, and would weighting by complexity alter your evaluation?\n\nIt would be beneficial if the authors can provide thoughts on why this manuscript fit better to the main NeurIPS conference, instead of the benchmark and dataset track of NeurIPS.", "limitations": "Yes there is an explicit limitation section in the manuscript.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe manuscript is well-written and easy to follow.\n\nThe authors mine over 200 real-world GitHub issues to produce the first comprehensive taxonomy of error modes unique to LLM-based agent systems.\n\nThe AgentIssue-Bench provides a robust, one-click platform for evaluating agent debugging performance.\n\nThe evaluation has both quantitative success rates and qualitative breakdowns by issue category.\n\n---\n\nWeaknesses\n\nWith only 50 tasks selected from 201 identified issues, the benchmark’s limited scale may not fully represent the diversity of agent failures or support strong statistical claims.\n\nBuilding each scenario demanded 500 person-hours of manual effort. The manuscript lacks a clear pathway for automating or streamlining this process as the benchmark grows.\n\nAlthough the authors acknowledge challenges around LLM nondeterminism and evolving external dependencies, they didn't offer concrete strategies to ensure long-term reproducibility.", "was_revised": true, "final_justification": "My questions are majorly addressed by the authors' responses.", "rebuttal": "We thank the reviewer for the recognition and constructive feedback for our work.\n\n**1.“How do the authors plan to grow AgentIssue-Bench beyond these 50 scenarios? Could parts of the Docker/test setup be automated, or could the community help contribute new tasks?”**\n\n- *What steps can be automated.* The following table summarizes the key steps for agent issue reproduction and to what extent they can be automated. As mentioned by the reviewer, indeed some key steps can be fully/partially automated, except the step *“failure-triggering test generation”*  which stands as the major bottleneck for automating the agent issue reproduction pipeline.  In fact, failure reproduction has been an open challenge in software engineering, and even for traditional software, recent agents could only reproduce <13% failure tests[1].    \n\n  |Step| Content| Automation| Challenges|\n  |----|-----|-----|--------|\n  (1) Agent set-up| Configure and run agents. | Medium | Although it's feasible to automatically configure and run agents via scripts, there are often some challenging cases on setting missing/incorrect dependencies or user-privacy information (e.g., API keys) that require additional manual fixing effort.|\n  (2) Failure-triggering test generation| Generate test code to trigger the same failure as the issue description. | Low | Automated failure test reproduction has been an open challenge and is far beyond practical usage.|\n  (3) Reproduction verification| Check if the failure-triggering tests fail on the buggy version but pass on the correct version.| Full | This step is fully automated via scripts.|\n  (4) Dockerization| Create and push final Docker images to repository. | Full |  This step is fully automated via scripts.|\n\n\n- *Continuous and long-term extension.* To continually grow AgentIssue-Bench, a practical solution is to *standardize a semi-automated pipeline* to balance the manual effort and benchmark quality, including (1) fully automated steps (e.g.,  reproduction verification and dockerization), (2) human-dominated steps (e.g., failure-triggering test generation), and (3) human-enhanced steps (e.g., agent set-up). Actually, we’ve already been working on implementing such a pipeline, so as to support the long-term extension of our benchmark. Based on the semi-automated pipeline, *we have dedicated continuous effort to expand AgentIssue-bench by additionally reproducing 31 more issues* since the paper submission DDL; and on the additional issues we observe similar trends that SE agents have limited capabilities of resolving agent issues (e.g., the best SE agent AutoCodeRover only fixes 2 issues). \n\n* *How the community helps to contribute.* As an open-source benchmark, we welcome all the pull requests from users if they report new agent issues that satisfy our quality requirements. In addition, we’re open to integrating the automatic techniques/tools from the community for certain steps (e.g., failure-triggering test generation) to optimize our benchmark construction pipeline.  \n\n**2.“What concrete steps do you take to keep each task reproducible as LLM outputs and external tools change? Do you pin random seeds, snapshot dependencies, or filter outputs?”**\n\nWe’ve adopted the following steps to ensure reproducibility: \n* *Fixed randomness parameter and multi-round verification mechanism.* To mitigate nondeterminism introduced by LLMs, we fix parameters that control the randomness of LLM output (e.g., temperature = 0)  to reduce variability in outputs; additionally,  we run each task multiple times to filter out the cases with inconsistent LLM behaviors (as mentioned in line 218).\n* *Snapshot external dependencies in Docker.* We archive the external dependencies of each issue into an executable Docker,  which serves a fixed execution environment (i.e., including repository state and third-party dependencies), enabling long-term reproducibility. \n\n* *Long-term maintenance for our benchmark.* While with the above steps, there might still be a small number of cases becoming non-reproducible given the changes in LLM provider APIs (which cannot be snapshot and controlled). Therefore, we plan to long-term maintain our benchmark by regularly automatically running each issue to check their reproducibility – we will remove those no longer reproducible ones and continuously include new reproducible issues into the benchmark via our semi-automated pipeline.  \n\nWe’ll include the discussion above in our paper.\n\n **3.“Have the authors thought about varying task difficulty or severity? For instance, would an “LLM operation” bug count differently than a simple utility error, and would weighting by complexity alter your evaluation? ”** \n\nGreat point! Following the reviewer’s suggestion, we consider utility errors as of low severity/difficulty while the other agent-specific issues can be considered as of high severity/difficulty. The following table presents the weighted resolution rate (0.2 for easy tasks and 0.8 for hard tasks).  Overall, we could observe similar findings between weighting and non-weighting results, including (1) overall limited capabilities of agents in agent issue resolution, (2) the superiority of AutoCodeRover with Claude-3.5, and  (3) outperformance of Claude-3.5 over GPT-4o. We’ll include these discussions into our paper. \n\n\n|                      | w/Gpt-4o                        |                             |                           | w/Claude-3-5-sonnet              |                             |                             |\n|----------------------|----------------------------------|-----------------------------|---------------------------|----------------------------------|-----------------------------|-----------------------------|\n|                      | **Agentless** | **SWE-Agent** | **AutoCodeRover** | **Agentless** | **SWE-Agent** | **AutoCodeRover** |\n| **Correct Resolved**  | 5.34%     | 2.64%     | 3.86%         | 8.04%     | 6.35%     | 11.43%         |\n| **Plausible Resolved**| 17.83%    | 4.86%     | 11.96%        | 11.69%    | 6.35%     | 15.56%         |\n\n**4.”It would be beneficial if the authors can provide thoughts on why this manuscript fits better to the main NeurIPS conference, instead of the benchmark and dataset track of NeurIPS.”**\n\nWe believe the manuscript fits better in the main NeurIPS conference because it goes beyond mere benchmarks but also presents both the taxonomy of agent issues and important findings on LLM/agent capabilities: \n\n* **New problem for automated agent issue resolution**: This work makes the first attempt to define and formalize the automated agent issue resolution problem. Given the rapid evolution and increasing adoption of agent systems, we believe it is critical to bring attention to the issue resolution problem of this emerging software category.\n* **Empirically grounded taxonomy of agent issues**:  We perform a rigorous open-coding analysis to develop a comprehensive taxonomy of common agent issues. This taxonomy enhances the community’s conceptual understanding of the problem space and provides a structured foundation for future research on agent system maintenance.\n* **Novel findings revealing limited capabilities of SOTA agents in solving agent issues:** We present an extensive empirical study to reveal the limited capabilities of state-of-the-art SE agents in agent issue resolution, with diverse experimental settings and analysis. \nWe believe that such empirical studies on exploring the capabilities of LLMs/agents should be of broad interest to the NeurIPS community, as there have been many similar empirical studies in past main NeurIPS conferences[2-6].\n\n**5.”With only 50 tasks selected from 201 identified issues, the benchmark’s limited scale may not fully represent the diversity of agent failures or support strong statistical claims.”**\n\nWhile given the difficulty and huge manual effort  in reproducing agent issues, we’re only able to reproduce 50 issues as the first version of our benchmark by the submission time. However, we plan to and are also currently working to extend our benchmark in the long term. In fact, we have dedicated continuous effort since our initial submission to expand AgentIssue-Bench by additionally including 31 more issues;  and on the additional issues we observe similar trends that SE agents have very limited capabilities of resolving agent issues (e.g., the best SE agent AutoCodeRover only fixes 2 issues).  We'll discuss the results in our paper and also build a leaderboard website for continuous update.\n\n------------\nReference:\n\n[1] “AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions” Wang et al, arXiv 2024. \n\n[2] Deng, Yinlin, et al. \"Can LLMs implicitly learn numeric parameter constraints in data science APIs?.\" Neurips main conference (2024): 54205-54238.\n\n[3] Ma, Yiwei, et al. \"I2ebench: A comprehensive benchmark for instruction-based image editing.\" Neurips main conference (2024): 41494-41516.\n\n[4] Zeng, Zhongshen, et al. \"Mr-ben: A meta-reasoning benchmark for evaluating system-2 thinking in llms.\" Neurips main conference (2024): 119466-119546.\n\n[5] Ho, Sy-Tuyen, et al. \"Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights.\" Neurips main conference (2024): 84197-84245.\n\n[6] Duan, Jinhao, et al. \"Gtbench: Uncovering the strategic reasoning capabilities of llms via game-theoretic evaluations.\" Neurips main conference (2024): 28219-28253."}, {"review_id": "yHGr9DylxR", "reviewer": "Reviewer_pzHu", "summary": "The paper highlights the emerging software paradigm of \"agent systems\", and perform an empirical study to taxonomize issues observed in agent software repositories, highlighting that the issues are different from traditional software issues and therefore motivating the need to study automated issue resolution for this subdomain of software engineering.\n\nNext, the paper proposes AgentIssue-Bench, a (N=50) benchmark for software-engineering agents, consisting of tasks to solve issues occuring in agent systems. Each task instance in the benchmark consists of executable environment packaged as docker images, along with unit tests and manually implemented and verified failure-to-pass tests. It is demonstrated that current SE agents with leading LLMs achieve a resolution rate of 3.33% to 12.67% highlighting the need for improvement in this subdomain. The authors highlight that majority issues solved pertain to dependency related issues, whereas most tasks related to agent capabilities remain unsolved.", "questions": "Can the authors please clarify how the semantic equivalence was checked to differentiate \"plausible correct\" and \"correct\" issue resolutions?\n\nCan the authors please clarify why is the function level localization percentage lower than plausibly resolved percentage? Does this indicate that there are issues where the LLM generated solution was able to solve the testcases even without correctly localizing the bugs? If so, does this indicate issues with the quality of unit tests (For example, SWEBench-Verified creates a rubric to identify good qualities in unit-tests for testing SE agent capabilities).", "limitations": "yes: Specifically, the authors highlight the dataset size (n=50), as well as being restricted to 16 agentic systems", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "I really appreciate the overall problem identified by the paper, which is about the emerging class of LLM-integrated software that handles non-determinism, yet aiming to provide a coherent experience to its users. The taxonomy study applied to this specific subdomain is useful and valuable. Further, the constructed benchmark has tests manually verified by human annotators, which is great for benchmark quality. The authors also make a distinction between \"plausible correct resolution\" and \"correct resolution\", which is a step in the right direction for SE agent benchmarking.\n\nHowever, I have the following concerns with the benchmark:\nAs the authors highlight in the paper, reproducing issues in agent repositories is difficult, because, \"External resources (e.g., agent-invoked tools, dependent libraries, or LLM 228 providers) may have changed since the issue was reported ...\".  Precisely because of this reason, the core contribution of the paper, which is the benchmark and its unit-tests, may not hold in a few months time. As an example, the authors highlight in Figure 6, that \"o1-preview\", and \"o1-mini\" do not support the stop keyword. While the fix is pragmatic (to include a parameter), the issue itself may become irrelevant: \"o1\" model, from the same series does support the parameter. This implies that the issues included in the benchmark are very time dependent on the model providers behaviours (which is not fixed, and can change at any time without notice), and therefore, not capture the performance/reasoning ability of the models, but rather their time-specific awareness of the current state of LLM APIs.\n2. While the paper is well motivated, and the benchmark curation approach is sound, the authors claim through the taxonomy that the type of issues occuring in agentic repositories are such that they typically do not apply to traditional software. However, going through Appendix F, where the authors list an example issue from every category, I present the following annotation (from my understanding) of the underlying issue captured in the highlighted example:\n\n```\nF.1.1: Dependency\nF.1.2: Dependency\nF.1.3: Incompatible parameters to external LLM providers\nF.2.1: Dependency\nF.2.2: Environment variable related configuration\nF.2.3: File read/encoding\nF.2.4: Information about how to call wikipedia API\nF.3.1: Software configuration issue\nF.3.2: SambaNova API specific\nF.3.3: Dependency changed the public API and those changes not migrated\nF.4.1: Environment variable related configuration\nF.4.2: TypeError\nF.4.3: Gemini's content policy rejects specific prompts. Add new exception to fix it.\nF.4.4: Dependency\nF.4.5: Context Window Management\nF.4.6: Changing model behaviour using prompt\nF.5.1: Control flow issue\nF.6.1: Frontend/UI\nF.6.2: Dependency\nF.6.3: File encoding\n```\n\nOut of the 20 highlighted, 6/20 pertain to dependency upgrades, 4/20 are related to reading files and environment variables, 1/20 is a UI issue, and so on. These are very typical of traditional software. So I believe that while the stated goal of the paper is to build a benchmark to test SE agent's capabilities in solving agent-specific issues, most of the issues present in the benchmark do not actually capture this capability. From my reading, only 4/20 of the highlighted issues actually pertain to LLM-specific problems.\n3. Benchmark size: The benchmark size is 50, out of which 20 are described in the appendix itself and have issues as highlighted above.", "was_revised": true, "final_justification": "The authors have clarified the discrepancy in function-level localization and plausibly resolved percentage, highlighted that to tackle the time-dependency issue I raised, they have maintained docker containers checkpointing repository states. Further, they have highlighted the collection of more tasks, along with the intent to continuously update the tasks, addressing my concern about the time-dependency of the benchmark.", "rebuttal": "We thank the reviewer for the constructive feedback. \n\n**1.“clarify how the semantic equivalence was checked to differentiate ‘plausible correct’ and ‘correct’  issue resolutions?”**\n\nWe conduct a structured manual evaluation process to check semantic equivalence. Two participants independently review each plausible patch by comparing it to the golden patch (i.e., developer-committed), focusing on whether the semantics of the patch fully resolve the underlying issue as intended and do not introduce other functional or semantic errors. If both reviewers agree that the patch was semantically equivalent and correctly resolved the issue, it is labeled as correct. If there was a disagreement between the two reviewers, we brought in a third participant as an adjudicator. The final label was determined only after all three reviewers reached a consensus. This multi-step adjudication process ensured consistency and rigor in our evaluation of patch correctness.\nOne example is autogen-1174, AutoCodeRover’s generated patch successfully passed the failure-triggering test, qualifying it as plausibly resolving the issue. However, during the manual review, we found that the generated patch incorrectly discarded a “tool_responses”: tool_returns key-value pair that the ground truth patch explicitly preserves. The developer’s ground truth patch is designed to return “role” key, “tool_responses” key, and “content” key. The generated patch alters this by removing the “tool_responses” key, returning only the “role” and the “content”. Following our methodology, this failure to replicate the full informational output of the ground truth patch means the two solutions are not semantically equivalent. Therefore, the generated patch was classified as a plausible correct but not correct. We will include the description of our check process in our paper. \n\n**2: “Why the function-level localization percentage is lower than the plausibly resolved percentage? ”**\n\nGreat point. Function-level localization percentage is lower, as we exclude the issues that are not localized within a function. For example, some dependency-related issues are fixed by modifying configuration files (e.g., requirements/runtime.txt), where there is no function at all. These cases are not included in calculating function-level localization accuracy. If we also exclude these cases for calculating plausible fix rate, the plausible fix rate would be lower than function-level localization rate. \nSuch a problem does not exist for file-level localization. Therefore, all the file-level localization percentage is higher than plausible fix rate, indicating the quality of our tests. We’ll clarify this in our paper.  \n\n**3.“...issues included in the benchmark are very time dependent on the model providers behaviour (which is not fixed, and can change at any time without notice), and therefore, not capture the performance/reasoning ability of the models, but rather their time-specific awareness of the current state of LLM APIs. ”**\n\n* First, we’ve specifically snapshotted all the external dependencies into dockers, therefore, the majority of the external resources-related issues can stay with long-term reproducibility. Moreover, we plan to continuously maintain and monitor our benchmark so that the evolution of provider APIs would not invalidate our benchmark. For example, for the issues that are unrelated to provider APIs, they can always stay reproducible by using the latest APIs (e.g., termly running tests can automatically detect unreproducible issues and then alert us to fix it accordingly). The only exception is for provider API-related issues, although we cannot control their reproducibility, such issues only account for a small portion  (e.g., 2.49% for incompatible parameters to LLM providers) in our benchmark, thus not threatening the overall usability of our benchmark. If the reviewer has any good suggestions to maintain the long-term reproducibility of provider API-related issues, please let us know and we’re happy to incorporate it to our benchmark. \n\n* As our benchmark is mined from real-world agents, we consider it reflects the actual distribution and realistic problems developers can encounter in practice. Therefore, although provider API-related issues focus on capturing the time-specific awareness of the current state of APIs, we still believe it is an important and real-world problem that should not be ignored for building agent maintenance techniques. In fact, for traditional software, recent research efforts have been dedicated to analyzing LLMs’ awareness in using deprecated external APIs[1]. Therefore, we consider to keep this issue category in our benchmark to reflect the real-world distribution of agent issues. \n\n* In addition to such provider API-related issues, our benchmark also includes many other issues that directly capture the performance/reasoning capabilities of models, such as incorrect model output handlers (capture cases where LLMs produce unexpected or ill-formed outputs), prompt-related issues (reveal instances where the model fails to correctly interpret task-specific inputs), and memory content errors (illustrate scenarios where the model's internal context handling leads to degraded performance). \n\nIn summary, we consider our benchmark as of overall long-term reproducibility.\n\n**4:“...However, going through Appendix F, I present the following annotation of the underlying issue... These are very typical of traditional software. So I believe that while the stated goal of the paper is to build a benchmark to test SE agent's capabilities in solving agent-specific issues, most of the issues present in the benchmark do not actually capture this capability.\"**\n\nThanks a lot for such detailed feedback on our taxonomy and we really appreciate it! However, based on the annotation, there might be some misunderstanding for which we’d like further to clarify. \n\nOur taxonomy is derived by considering both the issue symptoms and the fix strategies. While some issues may appear similar to those found in traditional software (such as dependency updates), they pose unique challenges within agentic systems. Take F.1.1 and F.1.2 as examples. Although the fix strategy is broadly described as \"updating the version of the OpenAI/Anthropic library,\" this task is far from trivial for an SE agent. The issue descriptions often only describe the symptom, and the agent must first infer that it stems from a library version mismatch. For instance, GPT-4 Turbo is only supported in newer versions of the OpenAI library. Beyond updating the dependency, the agent must also integrate the new model into the agent system's existing framework, including designing interfaces, updating dispatch logic, and ensuring backward compatibility with existing models. This demands a deep understanding of the agent system's architecture, its abstraction layers, and the invocation patterns of various LLMs. In this context, these issues are deeply agent-specific, not simply traditional dependency problems.\n\nSimilarly, F.4.2 is not a trivial TypeError. While the symptom relates to OpenAI’s NOT_GIVEN sentinel value causing a comparison failure, the patch involves a broader architectural change, moving the token_limit property from the base_model to a unified_model_type abstraction class. This reflects a deeper system-level understanding of how token limits vary across LLMs and how such properties should be organized within the agent system to support extensibility and correctness. An SE agent would need to comprehend these architectural decisions to produce a correct and sustainable fix.\n\nAdditionally, beyond the appendix examples, our paper provides a quantitative breakdown of issue types in Figure 2. Notably, 72% of the benchmarked issues involve core agent components, such as LLM orchestration, memory management, and tool integration. These challenges are specific to agentic systems, arising from their tight coupling with evolving LLM APIs, dynamic tool usage, and memory mechanisms. As such, resolving these issues requires an SE agent not only to fix code but also to reason about LLM behavior, understand architectural abstractions, and ensure system-wide consistency across heterogeneous components.\n\nIn summary, while some issues may resemble those found in traditional software at a surface level, the underlying resolution process reflects challenges that are uniquely agent-specific. This directly aligns with the stated goal to evaluate the SE agent’s ability to operate in the context of agentic systems.\n\n**5: “The benchmark size is 50, out of which 20 are described in the appendix itself and have issues as highlighted above.”**\n\nWhile given the difficulty and huge manual effort  in reproducing agent issues, we’re only able to reproduce 50 issues as the first version of our benchmark by the submission time. However, we plan to and are also currently working to extend our benchmark in the long term. In fact, we have dedicated effort since our initial submission to expand AgentIssue-Bench by additionally including 31 more issues, most of which are agent-specific and not utility-related ones; and on the additional issues we observe similar trends that SE agents have limited capabilities of resolving agent issues (e.g., the best agent AutoCodeRover only fixes 2 issues).  We'll discuss the results in our paper and also build a leaderboard website for continuous update.\n\n----------\nReference:\n\n[1] LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion. Wang et al. ICSE 2025"}, {"review_id": "eqRfQL2IPZ", "reviewer": "Reviewer_o2xH", "summary": "This paper presents a comprehensive study on the challenges of maintaining and resolving issues in LLM-based agent systems. The authors first conduct an empirical study by manually analyzing 201 real-world GitHub issues from 16 popular agent systems, resulting in the first taxonomy of agent issues. This taxonomy categorizes problems into six main areas: Incompatibility with LLM providers, Tool-related issues, Memory-related issues, LLM operation issues, Workflow issues, and Utility issues, revealing that many issues are unique to the agent paradigm and distinct from those in traditional software.\n\nBuilding on this analysis, the paper introduces AGENTISSUE-BENCH, a new benchmark for evaluating the automated resolution of agent-related issues. The benchmark consists of 50 reproducible issue resolution tasks, each packaged in a Docker environment with failure-triggering tests. The authors then evaluate three state-of-the-art Software Engineering (SE) agents (Agentless, AutoCodeRover, and SWE-agent) on this benchmark. The results show that these agents have limited effectiveness, with correct resolution rates ranging from only 3.33% to 12.67%. This performance is significantly lower than their success rates on traditional software bugs, highlighting the unique challenges posed by agent systems and underscoring the need for specialized SE agents.", "questions": "1. Your taxonomy of agent issues shares conceptual overlap with prior work on bugs in LLM-integrated systems, such as [43]. Could you elaborate on the key distinctions and novel insights your taxonomy provides by offering a more direct comparison with the defect classes identified in previous studies?\n\n2. In your evaluation, you report the average resolution rate over three runs. Could you justify this metric choice over a more standard one like Pass@k, which is common for stochastic tasks, and clarify how an issue was determined to be \"resolved\" for the Venn diagrams in Figure 5?\n\n3. Given that your findings reveal current SE agents are ill-equipped to handle agent-specific issues, what novel methods or architectural changes do you hypothesize are needed to build agents that can effectively tackle the complex LLM operation, memory, and workflow problems identified in your benchmark?", "limitations": "This paper provides a valuable and timely investigation into the maintenance challenges of the rapidly emerging paradigm of LLM-based agent systems. The primary strength of this work is the creation and public release of AGENTISSUE-BENCH, the first reproducible benchmark specifically designed for agent issue resolution. The amount of effort invested—500 person-hours to create 50 reproducible tasks—is substantial. This benchmark, complete with executable environments and failure-triggering tests, represents a significant resource for the community. Furthermore, the empirical study and the resulting taxonomy of agent issues provide a structured understanding of the types of bugs and feature requests that are common in these systems, which is a valuable contribution in itself.\n\nDespite its strengths, the paper has a few weaknesses that detract from its overall impact, particularly concerning its framing as a main track research paper, the novelty of its taxonomy, and certain aspects of its evaluation methodology.\n\n## Low Significance for main research track\nThe paper has low research significance for a main research track, and is better suited for a dataset/benchmark track. While the authors spend a significant amount of time and effort curating this dataset, there is not much novelty in the core methodology. The authors investigate a specific set of bugs, they reproduce existing techniques on these bugs, and do not introduce any significant novelty in terms of new algorithms or approaches for fixing these agent-specific issues. The main contribution is the benchmark and the empirical findings from applying existing tools. While this is a very valuable contribution to the field, it is primarily a resource paper. A main research track typically expects novel techniques or fundamental theoretical insights, whereas this work's core is an empirical study and the creation of an evaluation suite.\n\n## Taxonomy comparison with existing ones\nThe presented taxonomy shares some commonalities with existing ones that focus on the integration of LLMs within software systems. As a matter of fact, the main distinction of these bugs with other traditional software systems is the presence of LLMs. The paper mentions a related work [43] by Shao et al. which also catalogs bugs in LLM-integrated systems. However, the authors do not provide a detailed comparison. For instance, the \"Incompatibility with LLM providers\" and \"LLM operation issues\" categories in this paper appear to overlap significantly with the defect classes identified in [43]. A more thorough discussion comparing and contrasting the two taxonomies would have better highlighted the unique contributions of this work. Without this comparison, the novelty of the taxonomy is somewhat reduced, as it's unclear how much it extends beyond previously identified integration problems.\n\n## Evaluation\nThe evaluation methodology could be strengthened.Using the average of three runs to report resolution rates is a reasonable approach to handle the non-determinism of LLMs, but a metric like Pass@k, which is standard in code generation and related fields, would probably represent a better metric. It is also unclear how the Venn diagrams in Figure 5 were produced given the three runs they performed. Are any issues solved in any of the three runs counted? For an issue to be counted as \"solved\" by an agent in the diagram, does it need to be solved in all three runs, or just one? This ambiguity makes it difficult to interpret the complementary strengths of the different SE agents. Clarifying the criterion for inclusion in the Venn diagrams is essential for the reproducibility and clear interpretation of these results.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "# Strengths\n- The paper introduces AGENTISSUE-BENCH, a high-quality and reproducible benchmark for agent issue resolution.\n- It presents the first taxonomy of agent issues, which is based on an extensive manual analysis of real-world examples.\n- The work provides significant results that reveal the limitations of current SE agents in handling agent-specific problems.\n\n# Weaknesses\n- The paper's primary contribution is a benchmark, which lacks the significant methodological novelty for a main research track.\n- Its taxonomy is not sufficiently compared to existing work, which diminishes its perceived novelty.\n- The evaluation relies on non-standard metrics, and the reporting of some results lacks clarity.", "was_revised": true, "final_justification": "The authors addressed my concerns:\n- discussed in more details the comparison with related work\n- Clarified the metrics used\n- Discussed the future works", "rebuttal": "**1:\"Taxonomy comparison with [43]”**\n\n[43] only centers on LLM+RAG systems whereas we target a broader range of LLM-based agents.  Such distinctions in scope and focus that lead to notable differences in our taxonomies:\n\n* **1. Broader Coverage of Tool-Related Issues:**   [43] only includes issues on retrieval tools, due to its focus on LLM+RAG systems. However, agents often incorporate a wider range of tools beyond RAG, such as file I/O tools, command-line interfaces, and web APIs. Our taxonomy captures a more comprehensive set of tool-related issues, including tool dependency, configuration, and implementation, which are NOT covered in [43] but frequently occur in real-world agents.\n\n* **2. Finer-Grained Categorization of Memory Issues:** [43] includes only a single category for inefficient memory problems, whereas our taxonomy provides a more detailed breakdown of memory-related issues. We additionally cover issues on memory initialization, content errors, and memory dependencies, which are critical in agents but missing in [43]. \n\n* **3. Inclusion of API and Model Binding Issues:** Our taxonomy explicitly includes issues caused by dependencies on external APIs or model bindings, e.g., agents often support switching between LLMs, which can introduce binding or invocation failures. The entire issue category is common in practice but missing in [43]. \n\n* **4. Broader Workflow Issue Categories:** [43] only identifies a single workflow-related issue: \"Out-of-sync LLM downstream tasks\". However, workflow failures in agent systems can also include unintended suspensions, incorrect execution sequences, and broken workflows. Our taxonomy reflects this broader set of workflow concerns.\n\n* **5. Component-Oriented vs. Cause-Oriented Organization:** Our taxonomy is organized by system components (e.g., LLM, tool, memory), enabling clearer mapping of issues to the corresponding agent parts. In contrast, [43] uses a cause-based structure, which can lead to overlapping or ambiguous categories  (e.g., “Unclear context in prompt” vs. “Lacking restrictions in prompt”).\n\n**2: “Justify the metric choice? clarify how an issue was determined to be \"resolved\" for Figure 5?”**\n\n- *Why use average resolution rate*. First, we’d like to clarify that our used metric (average resolution rate over three runs) is essentially pass@1, but we present average pass@1 over three runs to mitigate the randomness from LLMs. In fact, all the existing work[1-4]  on automated issue resolution reports pass@1, as in practice developers often check only one patch. Therefore, our metric is actually in line with the common practice of existing work, but we perform more rigorous evaluation by repeating our experiment three times. \n\n  Moreover, based on reviewers’ suggestions, we further present the pass@1 and pass@3 over one run in the following table. Overall, we could observe consistent trends on these metrics as our current findings, including  (1) overall limited capabilities of agents in agent issue resolution, (2) the superiority of AutoCoveRover with Claude-3.5, and  (3) outperformance of Claude-3.5 over GPT-4o. We’ll include these discussions into our paper. \n \n|  | w\\Gpt-4o    |  |  | w\\Claude-3-5-sonnet |   |  |\n|---|---|---|---|---|---|---|\n|  | **Agentless** | **SWE-Agent** | **AutoCodeRover** | **Agentless** | **SWE-Agent** | **AutoCodeRover** |\n| **Pass@1**  | 6.00%  | 2.00% | 2.00%   | 10.00%  | 8.00%| 14.00%  |\n| **Pass@3**  | 10.00%  | 6.00% | 8.00% | 10.00%  | 8.00% | 14.00% |\n\n\n- *How an issue was determined to be \"resolved\" for Venn diagram Figure 5?*\n For an issue to be included in the Venn diagram in Figure 5, it needed to be solved by the agent in at least one of the three experimental runs. The reason behind this “best-of-three” approach was to capture the agent’s capability and effectively visualize the complementary strengths and overlaps of each SE agent. We’ll clarify this in our paper. \n\n**3. “What novel methods or architectural changes are needed?”**\n\nBased on our study, there can be several architectural improvements towards building more effective SE agents for diagnosing and resolving issues in LLM-based agent systems:\n\n* **Adding a knowledge base on agent-needed external resources.** As our study shows that SE agents fail to fix external resource-related (e.g., tools, dependencies, or LLM providers) issues, it indicates that SE agents need to be enhanced with more up-to-date and evolving knowledge on how to use these agent-needed external resources. Thus, one architectural improvement is to add and maintain an evolving knowledge base on all agent-related external resources (e.g., built on the API documentation, release notes, or historical issues), and then integrating with retrieval-augmented generation could help SE agents better reason and diagnose the agent issues related to external resources. \n\n* **Training better SE agents with instances and trajectories collected from AgentIssue-bench.**  Recent work[10] has shown that task instances and agent trajectories can train more powerful SE agents for *traditional* software issue resolution (e.g., SWE-bench). Our benchmark further fills the gap by providing training data specifically on the emerging software system – agent systems. Specifically, as our work provides executable environments and tests of buggy/fixed agents, we can collect massive instances and trajectories (e.g., agent-environment/tool interaction trajectories) for fine-tuning more powerful SE agents specifically for agent issue resolution.\n\n* **Adding a dynamic analysis component in SE agents.** As our study shows the limited *localization accuracy* of SE agents in agent issue resolution, it indicates that one big challenge for agent issue resolution lies in the huge gap between the issue description and root causes. However, existing SE agents only analyze *static information* (i.e., error symptoms in  issue descriptions) for issue resolution, leaving massive important runtime information untapped. To tackle this limitation, SE agents should add a dynamic analysis component, to utilize the runtime and intermediate outputs during agent operation (e.g., trajectories, fine-grained outputs of each key step like tool invocation, memory updating). Such a dynamic analysis can provide more informative hints for accurate issue localization and patch generation.\n\nWe believe these topics above would  provide actionable guidelines for future work towards building more powerful SE agents. We’ll include these discussions  into our paper.\n\n**4. “ While this is a very valuable contribution to the field, a main research track typically expects novel techniques, whereas this work's core is an empirical study...”**\n\nWe appreciate your recognition of the value of the benchmark and empirical findings.  We believe the manuscript fits better in the main NeurIPS conference because it goes beyond mere benchmarks but also presents both the taxonomy of agent issues and important findings： \n\n* **New problem for automated agent issue resolution**: This work makes the first attempt to formalize and define the automated agent issue resolution problem. Given the rapid evolution and increasing adoption of agent systems, we believe it is critical to bring attention to the issue resolution problem of this emerging software category.\n\n* **Empirically grounded taxonomy of agent issues**:  We perform a rigorous open-coding analysis to develop a comprehensive taxonomy of common agent issues. As responded in the first comment, we believe our taxonomy provides novel and different scope from existing work. We believe our taxonomy enhances the community’s conceptual understanding of the problem space and provides a structured foundation for future research on agent system maintenance. \n\n* **Novel findings revealing limited capabilities of SOTA agents in solving agent issues:** We present an extensive empirical study to reveal the limited capabilities of SOTA  SE agents in agent issue resolution, with diverse experimental settings and analysis. As responded in the third comment, these findings further indicate practical guidelines to build more powerful SE agents.\n\nIn summary, this paper highlights a novel challenge: issue resolution in agent-based systems by SE agents. It provides a clear taxonomy of unique failure scenarios and exposes key challenges faced by SE agents.  These insights go beyond traditional benchmarking and offer broader implications for the design and evaluation of agent systems, which will be of broad interest to the NeurIPS community. In fact, the past editions of main Neurips conferences have also been accepting empirical studies on exploring the capabilities of LLMs/agents[5-9].\n\n----\nReference\n\n[1] Tao, Wei, et al. \"Magis: Llm-based multi-agent framework for github issue resolution.\" Neurips 2024.\n\n[2] Zhang, et al. \"Autocoderover: Autonomous program improvement.\" ISSTA 2024.\n\n[3] Yang et al. \"Swe-agent: Agent-computer interfaces enable automated software engineering.\" Neurips 2024\n\n[4] Ma  et al. \"Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration.\" arXiv 2024.\n\n[5] Deng, Yinlin, et al. \"Can LLMs implicitly learn numeric parameter constraints in data science APIs?.\" Neurips main conference (2024)\n\n[6] Ma, Yiwei, et al. \"I2ebench: A comprehensive benchmark for instruction-based image editing.\" Neurips main conference (2024)\n\n[7] Ho, Sy-Tuyen, et al. \"Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights.\" Neurips main conference (2024)\n\n[8] Duan et al. \"Gtbench: Uncovering the strategic reasoning capabilities of llms via game-theoretic evaluations.\" Neurips main conference (2024)\n\n[9] Zeng et al. \"Mr-ben: A meta-reasoning benchmark for evaluating system-2 thinking in llms.\" Neurips main conference (2024)\n\n[10]  Pan et al.  Training Software Engineering Agents and Verifiers with SWE-Gym. ICML 2025."}]

Author rebuttal:

Meta-review: This research provides a valuable empirical foundation by analyzing 201 real-world GitHub issues from 16 systems to create the first taxonomy of agent-specific issues—comprising 6 categories and 20 subcategories—and introduces AGENTISSUE-BENCH, a Docker-packaged benchmark with 50 reproducible tasks for evaluating issue resolution. Tests of three leading software engineering agents (Agentless, AutoCodeRover, SWE-agent) using GPT-4 and Claude revealed critically low success rates (3.33–12.67%), exposing their inadequacy in handling challenges such as non-deterministic outputs and dynamic tool interactions. While the paper is well-written and methodologically rigorous with a technically sound benchmark, it could be strengthened by deeper exploration of preventative measures, performance optimizations, and more substantive discussion of solutions, in addition to improving metric clarity and standardization. Despite these limitations, reviewers ultimately lean toward acceptance due to the significant empirical contribution and practical utility of the benchmark.
Decision: Accept (poster)