OrcaLoca: An LLM Agent Framework for Software Issue Localization
Authors: Zhongming Yu
Hejia Zhang
Yujie Zhao
Hanxian Huang
Matrix Yao
Ke Ding
Jishen Zhao
Venue: ICML 2025 poster (2025)
Keywords: LLM
LLM Agent
Software Engineering
Primary area: applications

Abstract: Recent developments in Large Language Model (LLM) agents are revolutionizing Autonomous Software Engineering (ASE), enabling automated coding, problem fixes, and feature improvements. However, localization -- precisely identifying software problems by navigating to relevant code sections -- remains a significant challenge. Current approaches often yield suboptimal results due to a lack of effective integration between LLM agents and precise code search mechanisms. This paper introduces OrcaLoca, an LLM agent framework that improves accuracy for software issue localization by integrating priority-based scheduling for LLM-guided action, action decomposition with relevance scoring, and distance-aware context pruning. Experimental results demonstrate that OrcaLoca becomes the new open-source state-of-the-art (SOTA) in function match rate (65.33%) on SWE-bench Lite. It also improves the final resolved rate of an open-source framework by 6.33 percentage points through its patch generation integration.

OrcaLoca: An LLM A gent
F Framework for Software Issue Localization
Zhongming Yu
Hejia Zhang Yujie Zhao
Hanxian Huang
Matrix Yao?
Ke Ding
Jishen Zhao
Abstract
ZZZ % Resolved
72.08%
KZZZ
F Function
Recent developments in Large Language Model (LLM) agents are revolutionizing Autonomous Software Engineering (ASE), enabling automated coding. problem fixes, and feature improvements. However, localization precisely identifying software problems by navigating relevant code sections remains significant challenge. Current approaches often yield suboptimal results due to lack of effective integration between L.M agents and precise code search mechanisms. This paper introduces ORCALOCA an LLM agent frame- work that improves accuracy for software issue localization by integrating priority- -aase schedul- ing for LLLM-uided action, action decomposition with relevance scoring, and distance-avee context pruning. Experimental results demonstrate that ORCALOCA becomes the new open-source state- of-the-art (SOTA) in function match rate (65.33%) on SWE-bench Lite. also improves the final resolved rate of an open-source framework by 6.33 percentage points through its patch generation integration. ORCALOCA is available at https: / Igiituubccmm fi whminggu//rccacccooo
70
% File
Percentage
60-
Mean
53.50%
40-
33.10%
30-
% Resolved
% Function
% File
Figure 1. Distribution and average of file function match rate and resolved rate on SWE-Beneh Lite LeaderBoard.
leverage capabilities such as fault localization, action plan- ning, and trogram-buildiin unit tests. Among these abilities, localization the ability precisely identify and navigate to relevant code for resolving software engineering problems remains crucial yet underexplored challenge in ASE.
Localization is well-recognized as a critical yet challenging step (Yang et al., 2024b; Xia et al., 2024) in ASE. As shown in Figure/ on average, only 53.5% of issues achieve correct function match across all submitted agents solutions (Jimenez al., 2025). Localization is challenging due an inherent complexity of software repositories. For instance, the average codebase of SWE-bench (Jimenez et al., 2024) consists of 3,010 files with around 438K lines of code. Worse yet, user requirements are often expressed in imprecise natural language, making it even more challenging to extract relevant code from a large repository based on the user's issue input. In particular, identify three key challenges of LLM agent- based localization:
1. Introduction
Large Language Models (LL.Ms) have advanced rapidly, driving intelligent agents across diverse domains. In Autonomous Software Engineering (ASE) (Devin, 2024), LLM-driven agents enable automatic code generation, pro- gram repair, and feature enhancement. Incorporating LLMs into software development processes has been demonstrated promising by tools such as GitHub Copilot (Microsoft, 2023) and LLM- agents like AutoCodeRover (Zhang al., 2024b) and SWE-agent (Yang et al., 2024b). To navigate repositories, create patches, and fix problems, these agents
1) How to explore the codebase with strategic action plan- ning and precise navigation? Prior works on based software localization encounter two key limitations: (i) action planning inefficiencies arise as certain methods rely solely on LLMs for guidance (Zhang al., 2024a), resulting in unstable and redundant search behaviors; (ii) graph-based scheduling (Ma al., 2024b) limits flexibility by enforcing preprocessed traversal routes that confine searches to neighboring nodes.
Eqqual contribution Univeessity of California, San Diego, USA "Intel Corporation. Correspondence to: Jishen Zhao <jzhao@ucsd.cduu
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
2) How to achieve both context conciseness and search space completeness? Concise context, such as code skeletons,

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
reduces noise and keeps the context manageable but risks omitting critical details for precise localization. Conversely, fully detailed search space ensures completeness but introduces overwhelming noise, redundancy. and irrelevant exploration paths. Achieving both conciseness and com- pleteness simultaneously is challenging, as existing methods often optimize for one at the expense of the other, leaving an open gap in effective localization.
approaches like FLUCCS (Sohn & Yoo, 2017), DeepFL (Li et al., 2019), and TRANSFER (Meng al., 2022). However, effective fault localization in large-scale software systems remains challenging due to the vast size of codebases and the overwhelming volume of error messages, which often exceed the capabilities of standalone learning models.
Since the advanced code and natural language understanding capabilities of LLMs, Recent studies (Yang et al., 2024a; Wu et al., 2023; Li et al., 2024; Hossain et al., 2024; Kang et al., 2023; Qin et al., 2024; Wang et al., 2024c) have proposed LLM-based FL methods. These methods incorporate agents and tools to address the challenges of large-scale systems. AUTOFL (Kang et al., 2023) enhances standalone LLMs with tool invocations, such as repository retrieval tools, for more effective exploration of code repositories. RCAgent (Wang et al., 2024c) integrates four tools (code analysis, log analysis, memory retrieval, and information collection) to support decision-making AgentFL. (Qin et al., 2024) scales LLM-based fault localization to project-level contexts by combining multiple agents with static analysis tools like Tree-sitter.
3) How effectively manage context during exploration? Large repositories often introduce noise due to ambiguities, such as function overrides and inherited classes. As the exploration process progresses, irrelevant information can accumulate, misleading the LLM and resulting in incorrect identification of bug locations. Existing frameworks (Zhang et al., 2024a; Wang et al., 2024b), merely concatenate all search results into the context, which is insufficient to manage the expanding complexity of large-scale exploration.
To address these challenges, we propose an agent system consisting of three key components:
Priority-Based Scheduling for LLM-Guided Actions: To address challenge 1), we design a dynamic action scheduling system that incorporates priority queues and LLM- action generation for codebase exploration. The priority queue dynamically reorders actions based on their contextual relevance and urgency, solving the shortcomings of previous systems that lacked effective action management.
However, effectively and robustly exploring the codebase while balancing the trade-off between context granularity and search space remains a significant challenge. In contrast to existing techmiques, ORCALOCA introduces dynamic ac- tion scheduling exploration system and mechanisms to score decomposed actions, addressing these limitations effectively.
Action Decomposition with Relevance Scoring: To re- solve challenge 2), we introduce method that decomposes high- actions, such as class skeletons file skeletons, into finer-grained sub-actions. These sub-actions are eval- uated and ranked according to their relevance to the issue using multi-agent workflow, ensuring comprehensive exploration while avoiding noise and redundancy.
2.2. LLM-- for Software Engineering
LLMs have recently demonstrated remarkable capabilities in achieving human-level performance across a wide range of tasks, significantly advancing the field of ASE. Unlike traditional function-level or file-level coding tasks like Humeneval(Chen al., 2021), ASE requires not only basic coding proficiency but also advanced skills in managing and interacting with code repositories. To solve such more complex tasks, LLM-based agents enhance project-level software engincering tasks by iteratively and autonomously performing actions, observing feedback, and planning future steps (Hong et al., 2023; Kong et al., 2024; Wang al., 2024a; Yang al., 2024b; Xia et al., 2024; Ouyang et al., 2024; Zhang et al., 2024b).
Distance-Aware Searched Context Pruning: To address challenge 3), we design a context manager that dynamically prunes the searched context. The pruning algorithm leverages node distance heuristic within the graph-oriented codebase. By filtering out irrelevant data, the context manager ensures that exploration stays focused and aligned with the bug localization.
2. Related Work
OpenHands (Wang et al., 2024b) is community driven platform integrating widely used agent systems to explore end-to-end LLM-based agent solutions for handling complex SE tasks. AutoCodeRover (Zhang et al., 2024b) introduces LLM agents with specialized code search methods to iteratively retrieve code context and locate bugs using test cases. Agentless (Xia et al., 2024) proposes a two-stage bug-fixing system based on a streamlined workflow approach. Repounderstander (Ma et al., 2024a) empowers agents to comprehensively understand the whole repositories by a
2.1. Fault Localization Algorithms and Systems
Fault localization (FL) aims to identify suspicious locations (e.g.. statements or methods) in source code that are associated with bugs. Prior to the advent of LLMs, fault localization had been extensively studied, with techniques such as spectrum- based fault localization (SBFL) (Jones & Harrold, 2005), mutation- based fault localization (MBFL) (Papadakis & Le Traon, 2015), and learning-based
2

---

OrcaLoca: LLM Agent Framework for Software Issue Localization
Issue
Potential Bugs/PB)
follwing modells Py migrations CreateModel Missing import statement in genereled migralion... think bay the module ojango. db migratatsinnss writer, but i'm not sure.
found Django...
Class Decomposition
TypeSerializer
MigrationWWitte
TypuSeridlizer seesSeridizeersselllizee
seriaize at_string
Distanca-Aware
Abbreviated Problem
Score & Rank
basedir
TypeSertellarsseriillia
Context Pruning
TypeSerializer
Closer PB Tager
beriaizer__facter
operations
LLM Guided Action
search, calaableeaerioliee
Deconatructoble.
modelspy
12
OperationiWiifer
ASQ status
Due to LLM's amphasis
d
CreateModel
modelspy
R changes to 3 and
action the priorily of
MigrahonwWiieeserriliee
writerpy
serializer_ Factory
which
surpasses that of
CreateModel
MigrationWriter
serialize
File Related Action to ASQ
(d) Context Manager Detail
as__sti
T
Report
OperationWriter
examining sevisiization
File Action Decomposition
the
serializerpy
seridlizeryy
django/db//mgrente httpo/////mirrroooo/////ooo cooos//erraazzenn The spacial caso fimodtots Modal, 'models Modsll Di) explicaly sets en empty import for models Model, which couses the missing inport statement the generated migration nic..
TypeSarialized serialize] mothod
Factory
TypeSerializer
Deconstruct..
TypeSerializer
Score & Rank
FloctSerializer
m
seriolize
Top-K
BeseSeriolizer
DaconstructalleSeriiiieee
TypeSerializer serialzzss
(b) CodeGraph and Exploration Sequence
(c) Action Planning Queue Detail
(e) Final Output
Figure 2. An overview of ORCALOCA using demonstrating example from issue djjanoo 1 14580. (a) shows an abbreviated version of the issue's problem statement, where the user emphasizes Creat eMode and MigrationWritet onWriter: (b) presents exploration sequence agent over part of the whole CodeGraph. (c) provides details of the Action Scheduler Queue (ASQ). Specifically, action decomposition applied from to and from to as discussed in Scction 3.3. Additionally, techniques described Section 3.2 are used to handle steps from agent's 's final output. Please note this is demonstration, experiments may use different configuration.
CreateMode
and to (d) illustrates the distance-awe context pruning process, elaborated in Section 3.4. Finally, (c) shows the
code knowledge graph for repositories and a Monte Carlo tree search-based repository exploration strategy.
is containment, which represents hierarchical relationships, such as methods within classes or classes within files. 62 is the reference that represents relationships such as function calls between entities. The entities include functions, classes, methods, and files. Each code entity E in the CodeGraph is assigned with unique identifier (UID) using the format file_patt (: cls) (: ::methoo) For example, in standalone functions, the UID is simply file aall_path: moot These identifiers encode the containment hierarchy directly, with :: representing the "containment" relationship. To enhance compatibility with the CodeGraph, we redeveloped the API from AutoCodeRover (Zhang al., 2024a) to provide better support for CodeGraph- based searches (See Appendix A).
However, existing approaches remain limited as their search processes rely entirely on the LLM manage and guide actions, often resulting in unstable and ineffective search performance. Meanwhile, current systems, such (Zhang et al., 2024a; Xia et al., 2024), directly incorporate all search results as context, which is inefficient and can mislead the LLM. contrast, ORCALOCA employs a Priority- Priseity-Based Action Scheduling System for I.LM-gudde actions and Distance-Aware Context Pruning mechanism, significantly improving both efficiency and robustness.
3. Methodology
Building upon the ideas of Chain of Thought (CoT) (Wei et al., 2022) and ReACT (Yao et al., 2022), ORCALOCA follows a reason-and and-act workflow with a constrained action space. We design a custom- -essondd LLM prompt, which will generate Observation (O), Potential Bug Locations (PB), and Search Actions (SA) in each step. Here, for- mulate PB set of entities Upg: PB= {Upale (up e V}. To better illustrate the agent workflow, we formulate it as a tuple M, where M = (S,C,A,P, A.C.A...o). Here, S means the
3.1. Search System Setup and Agent Workflow
Our search system is inspired by prior works such as (Ma et al., 2024a; Ouyang et al., 2024), which employ graph databases for indexing code repositories. Similarly, we graph-based representation (V,E), to facilitate indexing and
construct a CodeGraph, of the codebase searching code entities. As illustrated in Figure 2. (b), the CodeGraph G contains two primary edge types e1,02 E
3

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
state space, including previous observations, potential bug locations, and retrieved search results. A stands for action space, which is restricted by our search APIs. In A. each action E A represents query for retrieving relevant code snippets, generating a feedback as Search Result (S Rt. VSR with UID,SR = USR E V. The context space C' means for the environment, which contains the repository structure formulated by CodeGraph.
Algorithm ORCALOCA Agent Core Algorithm
1: Initialize state So f- problem statement
2: Initialize ASQ + 0
3: while ASQ not empty and not converged do
4: Generate PB,, +
5: for all a4 E S.A. do
if aq redundant then
Skip og
For the evolution of the agent state after action, we denote the transition function as P x Ax - (S). In our agent, LLM plays the key role of state transition, in which the next state S141 is formed by adding new search results and refining potential bug locations. The agent follows policy π xCC A (AA), which is co-managed by I.M and Action Scheduler Queue (ASQ). The policy determines the next action exe- cute based priority, where we have detailed description in Section 3.2. At step the action a will also generated by the decomposition mechanism, which is described in Section 3.3.
8:
else if ag previously seen then Increment counter Cog and update priority
10:
else
11:
Add dg to ASQ end if
12:
13:
end for
14: Select top-priority a from ASQ
15:
Execute 0, to get SR, if USR E Valass vysite then
16:
17:
Generate at by relevance scoring via sub-agent Add af to ASQ with higher priority
18:
The agent begins from the initial state so. which consists
19:
end if
of the problem statement (See Figure 2. (a)) and the reproducer information from the issue (See Appendix C), if available. Please note that these details are concatenated in our system prompt (See Appendix D) and will be provided to LLM at each subsequent step. During the exploration, LLM agent will generate O,, PB,, and SA, in every step I. In specific, the state transition would be O ~ P(O,...... .S. H.C..) indicating the generated O and PB are dependent on all previous generated states. Here, SRCM, is the pruned set of search results managed by the Context Manager (CM), see Section 3.4. The process terminates when ASQ is empty or follows the convergence condition (See Appendix E). In the end, the conclusion step produces only the conclusion (Oconclusion) and the bug locations (B), summarizing the identified issues and their locations after all exploration steps are completed, see V..
20:
Pretetch SR's UID to check validity
21:
Prune SR using CM based on distance to PB,
22:
Update St41 t ,(,
23:
end while
24: Generate Ocanclusion: B PLI.(OO.,,BBI...SIIII
queue to manage the LLM-generated actions, offering more comprehensive and effective method for action planning.
To achieve a thorough reasoning COT, our agent limits each step to only processing one action. However, for SA gen- erated by LLM, it may have multiple action candidates based the given context. To address this, we design policy that uses dynamic action scheduler queue (ASQ) on top of LLM L-M-enerrtted actions. The ASQ has priority management which is implemented on top of a heap data structure.
Figure 2. (e). Here B (PPgaaX(PBBOaii)))
Unlike traditional reinforcement learning, where the goal is to maximize cumulative rewards, our agent is designed to converge to the correct bug location effectively. The evaluation target is elaborated in Section 4.1.4.
In ORCALOCA, action priorities are dynamically adaptable across different levels. The default priority for action a E SA is 1. However, this priority can be elevated based contextual relevance and strong relationships. For instance, in Figure 2. (c), the step from to shows how the action involving the file serializer, py is assigned a higher priority due to its strong connection with seria zeer_ ffe..f.oorr.... The same principle is set for action decomposition, which is discussed in Section 3.3.
To have a better understanding of Figure 2, we provide a core algorithm pseudocode in Algorithm 1. It summarizes the essential components discussed in Sections 3.2, 3.3, and 3.4.
For implementation details such as ASQ intial actions guided by reproducer, top-k output mode, batch action execution, please refer to our discussion in Section 5.
To account for urgency, we also keep a counter Co2 for each unique action ag. When the LLM generates the same action repeatedly, the counter Ca, grows, indicating the LLM''s focus on checking the content. The counter Cop replaces the original priority value and adjusts the position of ag in the queue. This system ensures that the most important actions are carried out first. For example in Figure 1. (c), the step
3.2. Priority-Based Scheduling for LLM-Guided Actions
To solve challenge 1) we discussed in Section 1, ORCALOCA provides more robust framework, which leverages a priority

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
2024; Zhang et al., 2024a) frequently employed skeletal techniques for huge classes files, returning solely the class and methods signature. However, brutal traversal over all the methods could lead to noisy context and redundant actions. To overcome this challenge, we propose action decomposition with relevance scoring.
Action Search Database
The agent want to search about class Compller, and found mulliple metches the Inverted Inclex
The MadeeChelccfiedd
sjangffommmmooi py::
ModeelCholceFedd
Action
Disembiguation Info
Multple matched dassess found about dass:
SQUCompier-
Stored
DB
Possible Location f:
File
Specifically, if the search result SR of an action aj corresponds to a class Usg E Voluse, we employ a score and rank sub-agent to evaluate the relevance of each method in the class Nurioes = {v - ydlass E E1 to the problem statement. The sub-agent (implemented by another LLM agent) will select the top-k most relevant methods, which are recomposed as new search actions, denoted as af. These decomposed actions at are assigned a higher priority (e.g. 2), and pushed the ASQ for execution. In this way, the main agent could work with the scoring sub-agent in multi- -agent workflow. Moreover, we extend this decomposition principle to handle large files. For file that triggers skeleton mode, we collect code entities within the file, like functions and classes, and treat them as individual units for the sub-agent. We have shown the illustrated example in Figure 2. (c).
djaang hes.kknddnsslllooolllii pyy
Possible Location Fele
Hit
Action
django http:///mmmollooooolooooooo colppoooo
search cless
(NNdee@hhocceiied.
@engo/tommmmmddllpyy py)
Search Action For Loc 1
Search Action
For Loc
(a) Redundant Action Elimination
(b) Example Disambiguation
Figure 3. Detailed examples for ORCALOCA solving redundancy and disambiguation problem.
from 6 to shows that seri awwiaelfacctttoooo ory would
come to the next step due to its counter has accumulated to 3, which even surpasses the file related action models py corresponding to Creat
Additionally, to address the unpredictability and hallu- cinations of LLMs, we set up redundancy climination mechanism to improve action scheduling. This mechanism ensures that redundant actions are avoided, enhancing efficiency and preventing unnecessary exploration.
In addition to enhancing granularity, our method addresses ambiguities, which commonly appear in large software repos- itories such as function overrides, and inherited classes. To resolve these issues, we implement a robust disambiguation mechanism within our decomposition strategy. We first con- structed an inverted index that stores only the callable indices that exhibit ambiguities. The value of the index encloses the exact location, including the file, path, and relevant class, if applicable. As shown in Figure 3. (b), when our API finds query with ambiguities, it will locate itself in the inverted index, enabling us to gather all the possible locations to form a disambiguation message for the L.MM agent. Additionally, we will split the potential locations and fine- treinedii push back the related search actions the action queue.
Consider the previous agent API used by systems like (Zhang et al,, 2024b; Maet al., 2024a). When it comes search class content, it has two different APIs aaarrhhccasss... (cls) and search.cllas..inffiii
(cls, f) which will target at class searching. Initially, the LLM may lack precise information about the location of the target class, which leads to the use of the general method search class (ModelChoiceFiee)
However, af- ter analyzing the returned content, the LLM will learn the file path and generate a subsequent, more specific action, such (Model Choice ceFiel d,
as searchhclass.in.fff.. django/ forms hwt///oooessspoeeommoo py)) Without careful han- dling of API ambiguities in scheduling, even a unique class like Model 1 wwwhhcheeeeelleecccccc could result in duplicate actions and redundant content searches.
3.4. Distance-Aware Searched Context Pruning
To prune the irrelevant context and keep LLM focusing on useful information, we developed distance-aware context pruming method, which we call as the Context Manager (CM). The CM is designed to maintain concise and relevant set of search results (SR) by evaluating their relationship to the potential bug locations (PB).
To mitigate this, as illustrated in Figure 3 (a), we maintain an action search database. Before an action is passed to the agent's iss chain-of-thouge (COT) reasoning, we prefetch its UID from CodeGraph and register its unique identifier (UID) in this database. This prefetching process ensures that each action is checked against previously executed actions, preventing duplicates and enabling more efficient scheduling.
First of all, to enhance relevance, the CM retains only SR entries linked to valid search query UIDs. Disambiguation messages (See Figure 3. (b)) and skeleton messages, typ- ically used for large files and classes, are explicitly excluded to prevent irrelevant data from polluting the context.
3.3. Action Decomposition with Relevance Scoring
The pruning process is guided by CodeGraph G, where each search result SR is mapped to a unique graph node USA E V. The CM evaluates each SR based on
Achieving both conciseness and completeness simulta- neously is challenging. Previous solutions (Xia et al.,
5

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
its distance to the potential bug locations PB, which also represented nodes in the graph. Specifically, the CM computes the average shortest path distance between each node USR and the candidate nodes in PB: d(SR, PB) = PP EvEpB min (ddrssn.)),d(vuusss)) where d(vsR. o) represents the shortest path from USa to u in the directed CodeGraph, and UsR) represents the reverse shortest path. The final distance metric for pruning is defined as the minimum of these two values.
studies using SWE-bench Common.
4.1.2. BASELINES
We compare ORCALOCA against 17 different approaches listed on the public leaderboard (Jimenez et al., 2025) of SWE-bench Lite. These approaches are categorized into 2 groups: (1) closed-source source solutions, such as Alibaba Lingma (Ma et al., 2024b); (2) open-source solutions, including OpenHands (Wang et al., 2024b), AutoCodeRover (Zhang et al., 2024b), Agentless (Xia et al., 2024), RepoGraph (Ouyang et al., 2024), HyperAgent (Phan et al., 2024), and SWE-Agent (Yang et al., 2024b).
Once distances are calculated, the CM prioritizes the most relevant results. selects the top-k candidates based on the calculated average distance, ensuring that LLM bypass those irrelevant code blocks. As shown in Figure 2. (d), in the last step, the context will filter out the irrelevant info like Operat conttiiiiiiii eeerr CreateModel, which will make the conclusion step have a stable and correct bug location output. Importantly, the CM is applied to every step during the exploration phase.
The SWE-bench Lite leaderboard mandates that each submission include the generated patches for addressing the given issues. This requirement enables the computation and comparison of broader range of metries beyond the resolved rate. In addition to analyzing the leaderboard data, we reproduced the Agentless-1.5 model for direct comparison with ORCALOCA, as its editor component is integrated into our system.
By aligning SR entries with the structural relationships within the CodeGraph, the CM helps the system focus areas most likely to contain the bug. This approach not only streamlines the input context but also improves the accuracy and efficiency of the search process.
4.1.3. IMPLEMENTATION
ORCALOCA is built on the Llamalndex framework (Liu, 2022), which supports various foundation models. For our experiments, we used Claude-3.5 3.5- (Anthropic, 2024) as the underlying model, with sampling temperature set to 0.1 to prioritize deterministic results.
4. Evaluation
4.1. S.1.Setup
4. DATASETS
For the top-k values used in action decomposition (Sec- tion 3.3), we set & = for class decomposition and k = 2 for file decomposition. In the context pruning (Section 3.4), the context window size is configured to retain 12 entries (top-k). Our framework also supports wide range of customizable configurations, enabling users to fine-tune their agent workflows. These settings include parameters such as class decomposition, file decomposition, disambiguation decomposition, priority adjustment, and the ability to enable or customize priority levels. This flexibility allows users to tailor their agent's 'ss behavior to specific use cases, enhancing both exploration and fine- tne-tuning capabilities. The cost of searching is about $0.87 per instance.
SWE-bench (Jimenez al., 2023) is widely used dataset for evaluating the ability of LLM systems to address real-world software engineering challenges. It comprises 2,294 task instances derived from 12 popular Python repositories, where each task requires a patch to resolve the issue described in its corresponding GitHub issue.
To reduce evaluation costs and complexity, the SWE-bench team introduced two refined subsets:
SWE-bench Lite contains 300 instances filtered using heuristics, such as removing tasks with images, external hyperlinks, or short descriptions. Each task includes func- tional tests to validate the correctness of submitted patches. SWE-bbenh Verified, developed in collaboration with OpenAI, includes 500 instances manually validated by professional annotators, providing greater reliability.
To evaluate the contribution of ORCALOCA to the final Resolved Rate on SWE-bench Lite, we integrated the Repair, Patch Validation, and Patch Selection components of Agentless-1.5 (Xia et al., 2024) by converting the output of ORCALOCA into Agentless format. Inspired by Repograph (Ouyang et al., 2024), the dependencies of the output code are also added. We largely adhered to the experimental setup outlined in the Agentless public repository, using the same LLM model, Claude-3.5- 1.55-Soone--222 (oinet--0241000 1022. For the repair process, we generated 40 patches (1 at temperature of and the rest 0.8) with the str replaecee fo roratt t. argument
To further optimize costs for repeated experiments, we defined a smaller subset, SWE-bench Common, consisting of 93 instances that form the intersection of SWE-bench Lite and SWE-bench Verified. Its compact size and high reliability make it ideal for tasks such as ablation studies.
In our experiments, we evaluate the performance of ORCALOCA using SWE-bench Lite and conduct ablation
6

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
Table Performance and ranking on suhmissions of SWFi-hench-Lae Lite (See Appendix G for submission details). Cutoff: 01/13/2025. * indicates a tie in ranking. A indicated the agent is souuce. The best results for each metric are bolded and labeled as w.. The best open- source ones are underlined and labeled as *k.
The reported results for AutoooodeRover-v2.. were obtained from their latest submission to SWE-bench, as they did not submit to SWE-bench Lite. To ensure alignment, manually filtered their results to match the SWE-bench Lite subset.
+ The reported results Agentless- are our reproduction based on open- source code they provided. The discrepancy between this result and the one they submitted to the leaderboard could be attributed to the outdated reproduction script shared in their repository.
Resolved
Function Match
File Match
LLM Agent
LLM
Rate (Count)
Rank
Rate (Count)
Rank
Rate (Count)
Rank
Blackbox AI A
N/A
49.00% (147)
63.33% (190) S
81.33% (244)
6
Gru (2024-12-0$
N/A
48.67% (146)
61.67% (185)
83.33% (250)
Globant Code Fixer
N/A
48.33% (145)
67.33% % (202)
W
84,00% (252)
2
devlo A
N/A
47.33% (142)
4
66.67% (200)
2
84.67% (254)
1 W
OpenCSG Starship i
@ GPT-4o
39.67% (119)
10
49.00% (147)
70.67% (212)
16
Bytedance MarsCode
N/A
39.33% (118)
11
56.33% (169)
13
79.67% (239)
7"
Alibaba Lingma
N/A
33.00% (99)
15
57.33% (172)
11
75.00% (225)
13
Kodu-v1
Claude 3.5 Sonnet
44.67% (134)
5 *
52.00% (156)
15
65.00% (195)
19
OpenHands + CodeAct v2.1
Claude 3.5 Sonnet
41.67% (125)
6
63.67% (191)
4
81.67% (245)
5
PatchKitty-0.9
Claude 3.5 Sonnet
41.33% (124)
59.67% (179)
75.33% (226)
12
Composio SWE-Kit
Claude 3.5 Sonnet + @ o1mini
41.00% (123)
8
61.00% (183)
7
79.67% (239)
7
Moadess Tools
Claude 3.5 Sonnet
39.00% (117)
12
59.33% (178)
79.33% (238)
9
DeepSeck V3
30.67% (92)
16
54.33% (163)
74.33% (223)
14
AutoCodeRover-v2.0
@ GPT-4o
37.33% (112)
13
57.00% (171)
12
77.67% (233)
11
Agentless- 1.5t
Claude 3.5 Sonnet
34.67% (104)
58.67% (176)
78.67% (236)
RepoGraph
X GPT-4o
29.67% (89)
17
47.67% (143)
18
70.33% (211)
17
HyperAgent
Claude 3,5 Sonnet
25.33% (76)
18
47.67% (143)
18
67.67% (203)
18
SWF-agem
Claude 3.5 Sonnet
23.00% (69)
19
51.67% (155)
16
71.67% (215)
15
GPT-4o
18.33% (55)
20
42.00% (126)
21
57.67% (173)
21
@ GPT-4
18.00% (54)
21
43.67% (131)
20
61.00% (183)
Claude 3 Opus
11.67% (35)
22
33.67% (101)
22
47.67% (143)
22
ORCALOCA
Claude 3.5 Sonnet
41.00% (123)
8*
65.33% (196)
*k
83.33% (250)
set. During patch validation, we employed both regression and reproduction tests. Regression tests were filtered with a temperature of O, while reproduction tests were generated using 40 samples (1 at a temperature of 0 and the rest at 0.8). Finally, the results of selected regression and reproduction tests were used to identify the most effective patch among the 40 candidates. The cost of editing is about $0.90 per instance.
Function Match Rate and File Match Rate assess the localization accuracy of ORCALOCA by calculating the percentage of Match in instances. These metrics, inspired by prior works such as Agentless (Xia et al., 2024) and Repograph (Ouyang et al., 2024), evaluate how well the agent's 's outputs align with the golden patch. (To align with these works, we use the term function general term that includes functions and methods).
4.1.4. METRICS
To determine Function Match, we define the golden and agent-generated localization function results for each instance i as sets: Pl.goldem: Biune C V. following definitions in Section 3.1. A match is registered the golden set a subset of the agent's prediction: Blune C Biuue For File Match, we consider the subset of file nodes in the graph g, denoted as: Vills According the definition of our graph, every node v V is either a file node or has an ancestor by containment edge that is file node. Thus, we define a mapping function: fileOf: V - ffle. which returns the file containing node v. The File Match is then determined as: B golden C B ptlle agent
To evaluate the performance of ORCALOCA, we utilized four metrics: Resolved Rate, Function Match Rate, File Match Rate, and Function Match Precision. Each metric is designed to provide unique insights into the effectiveness and quality of the agent.
Resolved Rate is metric originally proposed by the SWE-bench benchmark (Jimenez et al., 2024), which we adopted for our evaluation. The benchmark assesses whether an issue is resolved by constructing a Docker container for each instance, applying the user-submitted patch, running regression tests within the container, and analyzing the test results. The final metric is the percentage of the instances that are resolved.
where Biile = (filef(()) W B,).
Function Match Precision metric proposed by us
7

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
to assess the quality of localization results. For instance, localization output that includes every function in the repository would always ensure function match but would be practically useless. To solve this problem, the Function Match Precision is computed for each instance as FMP; = [ n B. // | mgeell and the final metric is the average of FMP; per instances.
Orcal oca
Agentless OpenHandd
Orcal Loca
Agentless
AutoCodaRover
AutoCodeRoveer
OpenHands
171
176
112
104
194
123
2
3
8
1
6
8
16
113
65
25
4.2. Results
4.2.1. PERFORMANCE ON LEADERBOARD
Func Match
Resolved
As shown in Table I, our ORCALOCA sets a new open-source State-Of Ofee-OF-The-Art (SOTA) with a Function Match Rate of 65.33% (196 out of 300) and File Match Rate of 83.33% (250 out of 300). These results demonstrate the effectiveness of our proposed localization methodology.
Figure 4. Unique localizations and solutions of open source agents.
of Locs approach filters functions more preciscly at the cost of match rate.
In both cases, ORCALOCA achieves +6.67 percentage points improvement in Function Match Rate and +4.62 percentage points increase in Function Match Precision compared to Agentless-1.5 1.5, demonstrating the effectiveness of our localization methodology. Crucially, the +6.33 percentage points gain in Resolved Rate confirms that our enhanced localization directly translates to better patch resolution.
Moreover, ORCALOCA demonstrates strong performance on the Resolved Rate metric, successfully resolving 41 .00% (123 out of 300) issues in the SWE-bench Lite dataset. By integrating the editing capabilities of Agentless-1.55 we achieved 6.67 percentage points improvement in function match rate and 6.33 percentage points increase in the final resolved rate over its performance. These results establish ORCALOCA as significant milestone in the research community's efforts toward developing more robust autonomous software engineering solutions.
4.2.3. UNIQUE LOCALIZATIONS AND SOLUTIONS
We analyze the unique issues localized and resolved by ORCALOCA compared to other open-source agents including Agentless (Xia et al., 2024), AutoCodeRover (Zhang et al., 2024b) and OpenHands (Wang et al., 2024b). As shown in Figure 4, ORCALOCA uniquely localized 6 issues, demonstrating the effectiveness of our approach. Additionally, it resolved 8 unique issues, emphasizing the impact of accurate localization in ASE. These results highlight ORCALOCA 's capability to other systems, even if they are developed with significantly larger resources (like OpenHands).
4.2.2. IMPACT OF LOCALIZATION RESOLVED RATE
To evaluate how ORCALOCA 's improved localization enhances the final patch resolved rate, we fully reproduced Agentless-I.5 1.5 (Xia et al., 2024) on SWE-bench Lite as baseline. As shown in Table 2, ORCALOCA outperforms Agentless- 1.5 across all three key metrics: Resolved Rate, Function Match Rate and Function Match Precision.
strong complement
Agentless-1.5 reports two sets of localization metrics due to its multi-sampling approach (four localization attempts per instance in the official reproduction). Patch generation then evenly distributes these samples, producing 10 patches per localization result (40 in total, as per Section 4. 1.3). To fairly evaluate localization performance under this setting, we compute metrics using two aggregation methods:
4.2.4. ABLATION STUDIES
We conducted our ablation study on SWE-bench Common, smaller subset of SWE-bench Lite, to evaluate the contribu- tions of each proposed method. As shown in Table 3, remov- ing any of these methods caused a noticeable performance drop of approximately 3-5 percentage points. Specifically;
Union of Locs: Merges function sets from all localization attempts into single aggregated union set per instance before computing metrics. This typically results in higher Function Match Rate but a lower Function Match Precision, as more functions are included.
Priority Scheduling (Section 3.2): Eliminating scheduler priority weakened ORCALOCA heuristic planning ability, making it more susceptible to distractions from less important content.
Mean of Locs: Computes metrics separately for each localization attempt and reports the average. This method generally yields a higher Function Match Precision but
File & Class Disambiguation Decomposition (Section 3.3): Removing the decomposition approach restricted ORCALOCA 's ability to explore a broader search space, thereby reducing overall performance. Notice here through the experiment we prove the LLM hard to locate with correct info by only getting the disambiguation
lower Function Match Rate.
As expected, the Union of Loes method captures more correct functions but also increases noise, whereas the Mean
8

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
Table 2. Impact of localization on resolved rate. UL. stands for Union of Locations; stands for Mean of Locations.
between efficiency and precision.
System Overhead and Cost Analysis. We note the system overhead majorly introduced by dynamic code search and LLM serving. In particular, shortest-pale distances on the CodeGraph are computed on the fly, and the graph itself is reconstructed per repository and commit ID to ensure semantic precision. In future work, we plan to adopt a caching mechanism to reduce redundant graph construction for frequently queried repositories. For LLM part, since it can be measured by token cost, we analyze a detailed token usage in Appendix F.
Function Match
Agent
% Resolved
Rate
Precision
OrcaLoca
41.00%
65.33% %
38.34% %
Agentless (UL)
58.67%
29.01%
34.67%
Agentless (MI.
47.33%
33.72%
Table Ablation study results. Experiment completed on SWE- bench Common dataset.
Methods
Func. Match Rate
Model Generalization. Our experiments primarily use Claude due to its strong code reasoning capability, However, our framework is model-agnostic. Future work includes supporting open-source models such Qwen (Yang et al., 2025) and LLaMA (Touvron et al., 2023), especially when fine-tuned and deployed locally. This extension will substan- tially reduce the cost of repo-level benchmark evaluations and make our approach more accessible to the community.
ORCALOCA
76.34% (71)
w/o. priority scheduling
73.12% (68)
w/o. file & class decom.
72.04% (67)
- w/o. disambiguation decom.
70.97% (66)
w/o. context pruning
72.04% (67)
info (See Figure 3. (b)).
Multi-Langguaee and Cross-Language Support. Our current implementation focuses on Python repositories, as it leverages Python-specific syntax parsing and. Supporting other languages would require integrating language-specific parsers and handling syntax and semantic differences, which introduces extra engineering overhead. Still, our general framework is on structural connections instead of language semantics. Cross- Cross-language linking presents the primary difficulty for multi-language repositories- such as Python/C++ hybrids common in ML systems. We are currently working index to capture inter-language relationships and enable, which will require more static analysis and more complex lross-lan coordination methods.
Distance-Aware Context Pruning (Section 3.4):
Without distance-aware context pruning, ORCALOCA was forced to handle a larger and noisier context, making significantly more difficult focus on the most relevant code snippet. Thus the noise will degrade the final bug localization accuracy.
in principle since it works
5. Discussion
We introduce several practical extensions to our system that enhance performance, ficxibility, and efficiency beyond the core workflow. In addition, we highlight current limitations and outline promising directions for future exploration.
Initial Actions from Reproducer. As described in Ap- pendix C, we extract the calling stack from issue reproducers and construct first actions. This warms up the agent with more relevant context, avoiding the cold-start issue caused by relying exclusively on the problem statement and search API.
6. Conclusion
We presented ORCALOCA, framework designed hance software issue localization by incorporating innovative methodologies such as priority- -based scheduling for LLM- generated actions, action decomposition with relevance scoring, and distance- aware context pruning to streamline the search process and improve localization accuracy. On the SWE-bench Lite benchmark, ORCALOCA achieved a 65. 45.33% function match rate, establishing a new open-source state-of-the-ar (SOTA) for software issue localization. Furthermore, by integrating the patch generation component from another open-source framework, ORCALOCA attained a final resolution rate of 41.00%, achieving a 6.33 percentage points improvement over the original framework. These contributions not only advance the field of ASE but also pro- vide a modular framework that may inspire future research in integrating LLMs with automated debugging systems.
Top-k Retrieval Output Mode. To allow customizable result granularity, we provide a top-k retrieval mode, other than directly generated by agent (see Section 3.1). In this mode, the final bug location choices are chosen from the top-k relevant search results RS)) stored by the Context Manager, allowing users to adjust the precision-recall tradeoff for various evaluation settings.
Batch Action Execution. To reduce reasoning length and token consumption, our system supports a batch mode where multiple top-priority actions can be executed in one step. In our experiments, we adopt conservative batch size of 1 to maintain stable accuracy. Larger batch sizes (e.g., 2) are supported but may slightly affect the final accuracy. Thus, the batch size can be tuned depending on the desired balance

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
Acknowledgment
repair. Proceedings of the ACM on Software Engineering. I(FSH):1471-1493 2024.
This research was partially conducted using computational resources provided by the Google Cloud Platform (GCP) Credits Award.
Jimenez, C.E., Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve world github issues? arXiv preprint arXiv:2310.06770, 2023.
We sincerely appreciate the valuable suggestions paper writing provided by Yun Joon Soh and Haolan Liu from the STABLE Lab at UC San Diego
Jimenez, C. E.,, Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE--bench Can language models resolve real-world github is- sues? In The Twelfth International Conference on Learning Representations, 2024. URL https: / / /poenreview net/ forum? seerrmmid-vTFRS d=FTFFNG
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench leaderboard. https: /www. swebench. com. 2025.
References
Anthropic. Introducing claude 3.5 sonnet. https::/// //op::://ww. anthropic. www.coo/////eeeoooe hwwm//www/////e//m/ cclaude-e311 3-5 1-55-ooee --n-nssn 2024.
Jones, A. and Harrold, M. J. Empirical evaluation the
tarantula automatic naut-localization technique. In Pro- ceedings of the 20th IEEE/ACM international Conference Automated software engineering, 2p3 273-282, 2005.
Blackbox. Blackbox ai. https://www/ www.:wwwwwlcc....oooo....ooo ai/.2024. 2024.
Kang. S., An, G., and Yoo, S. A preliminary evalua- tion of Ilm-based fault localization. arXiv preprint arXiv:2308.052: 05488,0587.00 2023.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. H.D.D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377. 2021.
Kodu-AL. Kodu-v1. https: /ww.. kodu ai/, 2024.
Composio. Empower your ai agents with compo- sio a platform for managing and integrating tools with Iims and agents using function https:// composicccciiio dev/ 2024.
Kong, J., Cheng, M., Xie, X., Liu, S., Du, X., and Guo, Q. Contrastrepair: Enhancing conversation- -based automated program repair via contrastive case pairs. arXiv preprint
calling. introductiion///ttrro/vveerr
Li, H., Hao, Y., Zhai, Y., and Qian, Z. Enhancing static analysis for practical bug detection: An Ilm Ilm-integrated approach. Proceedings of the ACM Programming Languages, 8/00PSLA1);4744999 2024.
Devin. Devin, ai software engineer. https:/ //www/ cognition cognition.ai/int..coo. htt.://www......oooonnnneiiiiiia 2024.
devlo. devlo.
Gao, T. Viztracer, 2025. URL https: /gthhb...
Li, X., Li, W., Zhang, Y., and Zhang. L. Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, pp. 169-180, 2019.
com/ com/gaooe /ttp////oootaaaii///tttttaaaaaat 2025-01-24.
Accessed:
Globant. Globant code fixer. https://ail www.aaigggbaa......o com/us-en/,20 2024.
Liu, J. Llamalndex, 11 2022. / / //gittuu..../////// com/ (eerryjliu/llane index.
URL https:
Gru. Gru. wwt.thtp///....... //re:.////ruu.e 2024
Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, 7., Zhou, I.., et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint ar3iv:2.08.. 003502 2023.
Liu, Y., Gao, P., Wang, X., Peng. C., and Zhang, Z. Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:2409.0083 2024.
Hossain, S. B., Jiang, N., Zhou, Q., Li, X., Chiang, W.-H., Lyu, Y., Nguyen, H., and Tripp, O. A deep dive into large language models for automated bug localization and
Ma, Y., Yang, Q., Cao, R., Li, B., Huang, F., and Y. How to understand whole software repository? arXiv preprint arXiv:2406.01422 2024a.
10

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
Ma, Y., Yang, Q.. Cao, R., Li, B., Huang, F., and Li, Y. How to understand whole software repository? arXiv preprint arXiv:2406.011 422.0122, 2024b.
Wang, Z.,, Liu, Z., Zhang, Y., Zhong, A., Wang, J., Yin, F, Fan, I.., Wu, L.., and Wen, Q. Reagent: Cloud root cause analysis by autonomous agents with tool-augmented large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 4966-4974, 2024c.
Meng, X., Wang, X., Zhang, H., Sun, H., and Liu, X. Improving fault localization and program repair with deep semantic features and transferred knowledge. In Proceedings of the 44th International Conference on Software Engineering, pp. 1169-1180, 2022.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V. Zhou, D., al. Chain-of of-thoughtttt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Microsoft. GitHub Copilot Your Al pair programmer. https://githun com/ wtt.ttteeeooop.lll 2023.
Wu, Y., Li, Z.., Zhang, J. M., Papadakis, M., Harman, M., and Liu, Y. Large language models in fault localisation. arXiv preprint S.KV:22308.12777 2023.
Moatless. Moatless tools. htttp::/qithub.co. com/
aorwall/moat htssmmottesss-oolll s, 2024.
OpenCSG. Opencsg starship agentic coder. https:://penccst com/ starssip,pp... 2024.
Xia, C. S., Deng, Y., Dumn, S., and Zhang, L. Agentless: Demystifying IIm-based software engineering agents. arXiv preprint arXiv:2407.01488 2024.
Ouyang, S., Yu, W., Ma, K., Xiao, Z., Zhang, Z., Jia, M., Han, J., Zhang, H., and Yu, D. Repograph: Enhancing ai software engineering with repository-Ievee code graph. arXiv preprint arXiv:2410. /4684, 2024.
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C.,, Lv, C.,, et al. Qwen3 technical report. arXiv preprini arXiv:2505. 09388, 2025.
Yang, A. Z., Le Goues, C., Martins, R., and Hellendoom, V. Large language models for test-free fault localization. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-12, 2024a.
Papadakis, M. and Le Traon, Y. Metallaxis-fl: mutation- based fault localization. Software Testing, Verification and Reliability, 25(5-7):605-622 2015.
Phan, H. N., Nguyen, T. N., Nguyen, X., and Bui, N. D. Hyperagent: Generalist software engineering agents solve coding tasks at scale. arXiv preprint arXiv:2409. /6299, 2024.
Yang, J., Jimenez, E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent-computer interfaces enable automated software engineering, arXiv preprint
2024b.
Qin, Y., Wang, S., Lou, Y., Dong, J., Wang, K., Li, X., and Mao, X. Agentfl: Scaling Ilm-based fault localization to project-level context. arXiv preprint 2xXiv:2403.16366 2024.
Yao, S., Zhao, Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03662, 2022.
Zhang, Y., Ruan, H., Fan, Z., and Roychoudhury, A. Au- tocoderover: Autonomous program improvement, 2024a.
Sohn, and Yoo, S. Fluces: Using code and change metrics to improve fault localization. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 273 283, 2017.
Zhang. Y., Ruan, H., Fan, 7., and Roychoudhury, A. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp- 1592-1604, 2024b.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal. N., Hambro, E., Azhar, F., al. Illama: Open and efficient foundation language models. arXiv preprint 27X,112302.133977, 2023.
Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable code actions elicit better Ilm agents. arXiv preprint arXiv:2402.010333 2024a.
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407. 6641, 2024b.

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
A. Code Graph Details
A.1. Graph Construction Process
The CodeGraph represents the structural and semantic relationships within a codebase by integrating containment and reference relationships. It is constructed using Abstract Syntax Tree (AST) analysis and additional directory-based hierarchical relationships.
A.2. Containment Graph Construction
The containment graph models the lexical and structural hierarchy of the codebase. We extract entities by analyzing each file in the repository using AST, identifying: Classes: yeluss Functions: Junction Methods; method files: ylile
A containment edge e1 is added to represent hierarchical relationships: y y. c
EE1,
y fuunction + + E1
Although directories are not code entities, we explicitly include them in the CodeGraph to preserve structural context. The directory structure is modeled as follows:
Files within the same directory are connected via containment edges.
A directory node is linked to its subdirectories.
The root directory (" ") connects to all I-depth subdirectories and files, forming the top-level hierarchy:
This could be summarized as a formula ellle + ydirectoryy E El,, ensures that file relationships and directory nesting are explicitly represented in the CodeGraph.
e ddlrrcttory + suubiirector E Edi,
ddirectory + o100r E Ed,, which
A.3. Reference Graph Construction
The reference graph captures execution dependencies between code entities, including function calls, variable references, and module imports. Using function call analysis from the AST, we add reference edges: ycaller - yealles E 62, where 62 represents function call. We didn't 't use static analysis to get references like the method used in another function B, which we think is future direction for better ASE.
A.4. Heterogeneous Graph Representation
Our CodeGraph a heterogeneous graph, integrating both containment relationships (e1) and reference relationships (e2). We efficiently apply Depth First Search (DFS) for code entity search during the agent exploration.
B. Search API
We follow the design principle of AutoCodeRover's search API while implementing merged design with default fi le.ppath For example, in search.cl ass we have default f file.path h argument equal to None. In this scenario, we leverage LLM to decide whether it needs to add file-path argument or not based on the given context. To guide the agent, we provide the doestrings of the search APIs as part of the system prompt. The detailed API definition and docstring are attached below.
def search fwlefleceeeellltee contents
self, file_name: str, directory-patth str None None
str:
м"НАРТ to search the file skeleton
If you want to see the structure of the file, including class and function * signatures.
class class and search_methon in class to get the detailed Be sure to call search 4 information.
Args:
file name (str): The file name to search. Usage:
search_fill filefffeteeettts. (exxm)ple.py") Do not include the path, only ly the file name.
12

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
directory_path (str) The directory path search. Usage: search_file_com
("example.py) "path/to/dirretorry"
Returns:
str: If file contents exceed 200 lines, we will return the file skeleton, string that contains the file path and the file skeleton. Otherwise, we will return the file path and the file contents.
def search_class (self, class_namd str, file_path: str None) -> str: """A to search the class in the given repo.
Args:
ciass_name name (str): The class name to search.
File_path (str): The file path to search. If you could make sure the file
- path, please provide it to avoid ambiguity.
Leave it as None if you are not sure about the file path.
Usage: class ("ModelChoiceFiee 'django/forms http:m/mmmmee..p.....
or search_class ("ModelChoiceFieldd)
Returns:
str: The file path and the class content. If the content exceeds 100 lines, - we will use class skeleton.
If not found, return the error message, If multiple classes are found, return + the disambiquation message.
Please call search__ethoo__in in get detailed information of the method after skeleton search.
If the methods don 't have docstrings, please make sure use search_methor in class to get the method signature.
def search search_method__eeee
self, class name: str, method name: str, file path: str
-> str:
"""API to search the method of the class in the given repo.
Don't try to use this API until you have already tried barch_clase to get the
4 class info.
Args:
class_name (str): The class name to search. nethod_name (str): The method name within the class.
file_path (str): The file path to search. If you could make the file path, please provide it to avoid ambiguity.
Leave it None if you are not sure about the file path.
Usage: mearch_method..oe inniiin_illll class ("#odelChoiceFielde "to_python") or in ("ModelChoi wtt.:ffrmmmmmmmlsp.....
-
"to_python",
-
Returns:
str: The file path and the method code snippet. If not found, return the error message.
IF multiple methods are found, return the di disambiguate message.
def search_ (self, query_namer str,
str None) -> str: ""*API to search the callable definition in the given repo
If you are not sure about the query type, please use this API. The query can be function, class, method global variable.
Args:
13

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
query_name (str): The query search. The format should be only the name. file_path (str): The file path to search. If you could make sure the file path, please provide it to avoid ambiguity.
Leave it as None if you are not sure about the file path. or "django/
Usage: search_callable
CroodelChoiceFieee search ("#odelChoicefiielll
Returns:
str:
The file path and the code snippet. If not found, return the error
message.
If multiple matches are found, return the disambiguation message.
def Search_sourre,. _oceee_coeeee (self, file_path: str, source_codes str) -> str: """API to search the source code in the file. If you want to search the code 4 snippet in the file.
Args:
file_path (str): The file path to search. source_code (str): The source code to search.
Returns:
str: The file path and the related function/clase code snippet. If not found, return the error message.
a m
C. Reproducer Agent
Although the ORCALOCA search agent can inspect and explore the code repository statically, is unable to collect runtime information. To supplement this, we developed an auxiliary reproducer agent that attempts to reproduce reported issues and capture execution traces. Because successful reproduction is inherently limited (Only 38.0% of issues can be successfully re- produced in our experiment), this agent serves as a complementary analysis step rather than core element of our search design.
As illustrated in Figure the reproducer agent proceeds three stages:
Identifies suspicious functions and files from plain text sources, including tracebacks, code snippets, logs, and natural language descriptions;
Reproduces the issue by generating and executing snippet, then judges reproduction result and retries if failed;
Extracts key information from the trace through filtering and re-ranking.
C.1. Plain Text Parser
Extracting relevant data from execution traces is challenging due to their tremendous size. To narrow the search space, we identify initial suspicious keywords from the problem description.
We first segment the description into multiple patterns tracebacks, code snippets, and natural language. Each segment is then processed using tailored prompts to extract relevant keywords with higher accuracy.
C.2. Reproduction Snippet Generator
To reproduce the issue, we set up conda environment inside a Docker container following the methodology in SWE-Agent (Yang et al., 2024b). We then generate and execute reproduction snippet using an L.MM and record its execution trace with VizTracer (Gao, 2025).
The snippet's 's output sent to an LLM judge, which determines whether the issue was successfully reproduced. successful, the reproduction log and code are forwarded the plain text parser for further analysis.
14

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
Summarize & NL Parse
Slice
Traceback Parse
Has traceback?
Reproduce Succeeded?
Tracer Stack Filter
To Search Agent
Reproduce Judge
Reproduce Log Parse
Reproduce Code Parse
Has source code?
Reproduce Failed?
Source Code
Parse
Figure 5. Internal structure of reproducer agent. Appendix C.I contents are labeled in blue, C.2 in red and C.3 in purple.
C.3. Stack Trace Selector
Once trace data is collected, we apply filtering strategies based on empirical observations. Our case study indicates that the root cause bug is often:
Located in the same file as suspicious keyword;
A close descendant of a suspicious keyword the trace;
Near the root trace tree.
Using these heuristics, we assign priorities to trace entries and filter the top K 25 candidates.
For finer-gae aner-grained ranking, we compute a relevance score for each candidate by feeding its code context into LLM. The final ranking is determined using weighted sum of the LLM-generated score and the initial keyword- based priority. We retain candidates that exceed predefined absolute score threshold and rank within the top 5.
D. Key Contents in Framework Prompts
Extractor Agent Prompt
Common System Prompt:
You are an expert python developer, mastering at summarizing and extracting from Github issues.
Slice Sub-agent:
Your task is to slice strings from human reported github issue. Every slice shculdn't overlap with another slice. Non-existanct slice should be set
Your output should strictly follow the format below. (output_ format)
NOT REDUNDANT WORDS (like 'json', 'output', etc.)
15

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
The meanings of each field are:
An example given below: (example)
Below is the real task for you to solve: <repo_name> (repo_nmmm))ee httpeppooaaneerroooooooooe_ </repo_nam>>> (input_description)
Parse Sub-agent:
Your task is to extract python code keywords and the filepath that belong to (if exist from human reported github issue. Non-existanct filepath should be set tc
Your output should strictly follow the format below, (ntpuutformat)
NOT
REDUNDANT WORDS (like 'json', 'output', etc.)
The meanings of each field are: (
An example given below: (example)
Below is the real task for you to sclve: <repo_name> trepo_aamee</rrpoonnn <input sraput_descriptionn craput_descriptionn </input_dde descriptissiin
Judge Sub-agent:
Your task is to judge whether input GitHub issue is successfully reproduced, based on the reproducer_ioe generated by reproducer snippet; If the reproduce didn't succeed, try to generate a fixed reproduced snippet..
Some examples of judgment include:
1. sUCCESS (the exact same error message) from descriptso is found in reproducer_logy
2. FAILURE if the error message from input_description is different or irrelevant from the one found in reproducer_log;
3. SUCCESS if (the reproducer_lag:
printed output) from input_ descripttin is found in
4. FAILURE if the reproducer in input_description is expected to have output (erroz or printed log) but reproducer_ies is empty;
5. FAILURE if the reproducer in input_description is expected to raise error, but error is found from reproducer_log:
6. FAILURE if the reproducer in input_ depuu_ddscription is not expected to raise any errors, but or more errors are found from eeproducer_logy
7. FATLURE if the Inpat_descripe probl aroblematic behavior, but the reproducer_log matches with the expected one;
describes di fferent output for expected and
Your output should strictly follow the format below. loutput }ormat DO NOT SPEAK ANY REDUNDANT WORDS (like 'json', 'output', etc.)
16

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
The meanings of each field are: (output_fiillss)
Below the real task you solve:
<repo <repo__ame> name> (repe_ (repo_name| nww. </repo name?
seaput_descrition Sleput_descriptiee </input_@escriptionn
<reprooducer____pppt>>>>>>
[raproducee__nnlpot]]
</rsproducer__miipptt
<reproducer_log>
(reproducer_log]]
</reproducer_Log
Summarize Sub-agent:
Your task is to summarize a remon-reported GitHub issue natural language.
Your output should strictly follow the format below.
loutput format)
DO NOT SPEAK ANY REDUNDANT WORDS (like 'json', 'output', etc.)
The meanings of each field are: (output_fields)
An example given below: (example)
Below is the issue for you to summarize: <repo_names retp__ameee/rrppoo...eeee (input_descri en s/inpur_description
Code Scorer Sub-agent:
You Python coding expert. Your job is score how likely piece of code will need to be modified to solve a GitHub issue. The issue description will be presented in 'problem_sstatement
sproblen_statement: [problem_statement: statement siproblem_statenee
Ploase score how likely this picce of code will need to be modi fied to solve a GitHub issue. Please score the likeliness with an integer between 0 and 100, the higher more likely. Your output will be processed by program instcad human, so please ONLY output single integer.
Searcher Agent Prompt
You are professional software engineer who uses API calls to report bug code snippets from text into ison format.
You need to extract where are the bug locations analyzing the text. The given contains the problem statement and the code snippets.
17

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
There are some API calls that you can use to extract the information. The API calls include: (toal_desc)
<TASKS>
Every time you will do the following things:
1. Provide the observation based given input:
Every time will provide new search result in tag <New Info>.
It may contain the disambiguation info if the search action is rolated to multiple classes or methods.
Also, previous search results will be provided in the tag <Search Result>. You need to analyze the
search result based on the previous one and provide the observation based on the whole context.
2. Think about where the bug might be in the code by the whole given context (including all Search Result), and provide the potential bug Locations. The potential here means the most possible locati ions up to the current context.
3. Check whether it contains any class, method, or function you need to further search. Especially, if di disambiguate specific fic class or method.
info provided, you need to search for the
Plan the new_ sew_search__c swwrreeeaattiiieeeeaeeee actions based on the current context. You can use the giver API
calls search for the bug locations.
You can put multiple actions in the new www__ssarc__cciion list. Be sure to use argurents
in the tool description.
If you make sure the context is enough to answer the question, you can keep the new_scarch_action list empty.
final standalone step to provide the final bug locations when nothing ng else to search. Please keep in mind to follow the instructicn "Now let's come to a conclusion. ".
The conclusion
</TASKS>
<OUTPUT FORMAT>
1. Regular Step Format:
Provide your answer in clear JSON structure like this, (step_format)
Make sure each API call is written as a valid Python expression and code_ snippet is a valid Python string.
In potential_bug_locttion method name.
should provide the file path, class name, and
It's not the final answer, just hint for possible bug locations. If the method does not belong to any class, set the class to an empty string. You can provide multiple actions in the rew_search_actionss Do NOT add any title or description.
2. Conelusion Format:
After no input actions the search queue, provide the final bug locations JSON structure like this.
(bug_locations)
DO NOT generate observation or new_search_actions the conclusion step. DO NOT mix it with any title or description. If the methed does not belong to any class, set the class to an empty string. </OUTPUT FORMAT>
E. Convergence Configuration
Farly Stop Convergence Mode In most cases, agent naturally converges when there are no remaining actions in ASQ. However, in scenarios where the action sequence is lengthy and requires multiple execution steps, we introduce an early stop convergence mode to optimize efficiency.
This mode is controlled by a BERT embedding model, which evaluates the similarity between consecutive observations at
18

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
each step. Specifically, for observations, O2 and Oz+1, we compute their cosine similarity using their BERT embeddings:
cos (BERT (0 (0++1)) [EETT(O))] (ERT(O++1)
the similarity score exceeds 0.97, the two observations are considered equivalent.
To ensure stability in the decision-making process, we apply sliding window mechanism over consecutive observations. Specifically, require that the similarity condition holds for K 15 consecutive steps before triggering convergence:
1(cosf: 1oosff;00.999 0.997 =K
Once this condition is met, the agent terminates execution and reaches conclusion.
F. Cost Breakdown Analysis
We chose token cost as our primary metric because I.L.M inference dominates the overall time and monetary expenses of our system. For runtime analysis, since our implementation primarily leverages API services from external model providers. inference time can be considered approximately proportional to token usage.
In Table 4, we summarize the average per-instance token cost across different agents:
Table 4. Average token cost per instance for different agents.
Agent
Cost
OpenHands
1.14
SWB-Agent
1.62
AutoCodeRover
1.30
Agentless- 1.5
1.05
OrcaLoca
1.77
Orcal OrcalLoca-battee baa-batch(=2)
1.48
Notably, over half of OrcaLoca's token cost attributed the editing phase (0.90 1.77), which is primarily contributed by the edit component from Agentless- 1.5, adopt their editing mechanism in implementation. Although this paper primarily targets performance and accuracy, the reduced cost observed in PreaLoca-batee highlights large optimization potential for improving efficiency in the localization phase.
In OrcaLoca-batch, we implemented batched action execution during localization, extracting the top-priority actions in groups from the scheduler (See Section 5. Table presents a comparison of old and new token costs across ten sampled issues from SWEBench-Lite. The ratio (New Cost/ / Old Cost) reflects the cost improvement:
19

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
Table 5. Token cost comparison before and after hatched action optimization.
Instance ID
Old Cost
New Cost
Ratio
django-13551
0.30
0.26
0.87
django-1581
1.44
0.97
0.67
django-16255
0.17
0.18
1.06
pylint-7228
0.71
0.66
0.93
pytest-8906
1.93
0.87
0,45
scikit-leam-134433
0.31
0.21
0.68
sympy-14774
0.53
0.15
0.28
sympy-15011
1.14
0.64
0.56
sympy-16792
1.05
0.64
0.61
sympy-24213
0.55
0.20
0.36
Due to budget constraints, sampled 10 issues with varied token profiles. Using weighted averages cost bins, estimate that per-instance localization cost was reduced by an average of 34% (from 0.87 to 0.58) without negatively impacting localization correctness.
We are committed to further optimizing Orcal Loca and plan to explore additional efficiency improvements in future work, such as integrating kv cache techniques during inference.
G. Other Competing Methods
Blackbox AI Agent (Blackbox, 2024) is building coding agent to transform the way we build software.
Gru(2024-12-08) (Gru, 2024) builds different agents to solve different software engineering problems. But all Grus are built with the same principles: Clear Problem Domain, Dedicated Tools and Direct Value Delivery.
Globant Code Fixer Agent (Globant, 2024) is independent and intelligent software entities designed to transform business operations.
devlo (devlo, 2024) boosts user's productivity by handling development tasks, freeing to focus on innovation and ship products faster.
OpenCSG Starship Agentic Coder (OpenCSG, 2024) is a multi agent collaborative and scalable environment to empower user in building the next generation of intelligent applications.
Bytedance MarsCode Agent (Liu al., 2024) novel framework that leverages LLMs automatically identify and repair bugs in software code.
Alibaba Lingma Agent eM al., 2024b) understands the whole software repository to achieving automatic software engineering.
Kodu-v1 (Kodu-AI, 2024) implements a VS Code extension that adapts to user's s skill level, helping user bring ideas to life faster than ever before.
OpenHands + CodeAct v2.1 (Wang et al., 2024b) is a platform for the development of powerful and flexible Al agents that interact with the world in similar ways those of a human developer: by writing code, interacting with command line, and browsing the web.
PatchKitty-0.. It may have been developed concurrently with our work and is reportedly designed by researchers from UC Santa Barbara. While it was claimed to be open-source in its SWE-bench Lite submission, repository or related links have been released yet.
Composio SWE-Kit (2024-10-30) (Composio, 2024) helps user connect Al agents to external tools like Gmail, GitHub, Salesforce, etc. It's like a bridge between user's AI and the tools needs to get work done.
20

---

OrcaLoca: An LLM Agent Framework for Software Issue Localization
Moatless Tools (Moatless, 2024) is hobby project where the authors experiment with some ideas they have about how LLMs can be used edit code in large existing codebases. They believe that rather than relying on agent to reason its way to solution, it is crucial build good tools to insert the right context into the prompt and handle the response.
AutoCodeRover-v2.. (Zhang al., 2024b) is automated approach for solving Github issues autonomously achieve program improvement, where LLMs combined with sophisticated code search capabilities, ultimately leading program modification patch.
Agentless-1.3 (Xia et al., 2024) is an agentless approach to automatically resolve software development issues. Compared the verbose and complex setup of agent-based approaches, employs simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools.
RepoGraph (Ouyang et al., 2024) is plug-in module that manages repository-level structure for modern AI software engineering solutions.
HyperAgent (Phan et al., 2024) is novel generalist multi-agent system that addresses broad spectrum of SE tasks across multiple programming languages by emulating the workflows of human developers.
SWE-agent (Yang al., 2024b): a system that facilitates LM agents to autonomously use computers solve software engineering tasks. SWE-agent's custom agent-computer interface (ACT) significantly enhances agent's ability create and edit code files, navigate entire repositories, and execute tests and other programs.
21


Reviews:
[{"review_id": "4mDjxc3Ynm", "reviewer": "Reviewer_LnLT", "summary": "OrcaLoca is a novel framework that leverages LLM agents to improve software issue localization by precisely identifying the problematic sections within large codebases. The paper introduces three key innovations: priority-based scheduling for dynamically managing LLM-guided actions, action decomposition with relevance scoring to break down high-level queries into more granular sub-actions, and distance-aware context pruning that filters out irrelevant code to maintain focus. These techniques address longstanding challenges in automated bug localization, including imprecise navigation and excessive context noise, by integrating a structured code graph representation with targeted search strategies. Empirical evaluations on the SWE-bench Lite dataset demonstrate that OrcaLoca not only achieves SOTA performance with a 65.33% function match rate and 41.00% resolution rate.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and for highlighting several key strengths of our work. For the weak points, we list our responses part by part, as follows:\n\n**Weakness 1:** Heavy reliance on LLM outputs can lead to unpredictability and occasional hallucinations, affecting consistency in localization results. For example, different LLMs or even different LLM versions might yield different results.\n\n**Response:**\nThank you for this valuable point. We acknowledge that LLM-based agents may suffer from unpredictability or hallucinations. However, in our system, we have incorporated several design choices to mitigate these issues and improve reliability.\n\nAs illustrated in Figure 3, we introduce redundant action elimination to prevent the LLM from repeatedly generating previously executed (and potentially hallucinatory) actions. Furthermore, our agent adopts a context management framework, which enables pruning out noise and irrelevant data. Our agent has a self-consistency mechanism, where the LLM evaluates and refines its own intermediate search result to guide convergence, improving both stability and correctness. To evaluate consistency, we conducted a controlled experiment by selecting five representative instances from SWE-bench ([django__django-13551], [django__django-16255], [scikit-learn__scikit-learn-13439], [sympy__sympy-14774], [sympy__sympy-15011]). For each, we ran the agent five times and consistently observed the same function-level localization results under the same LLM configuration, demonstrating strong intra-model stability.\n\nFor inter-model consistency, we agree that different models varying a lot due to their reasonability and model weight discrepancy. To solve this problem, we would focus on improving LLM consistency by exploring consistency-aware prompting, robust fine-tuning, and more powerful code index and embedding in the future.\n\n**Weakness 2:**\nThe evaluation primarily focuses on SWE-bench datasets.\n\n**Response:**\nCurrently, we rely on SWE-Bench, which is the state-of-the-art benchmark so far for evaluating software agents. We recently discovered additional benchmarks such as SWE-Gym and LocBench, however they were released so lately that we do not have time to review the findings before submission. In the future, we plan to expand our method to include more benchmarks and experiment with other approaches.\n\nAnother reason is that the benchmark uses repo-based software, which increases the cost significantly. Experiments for 300 instances on SWE-bench lite will cost between `$500` and `$600`, not considering debugging and reproduction for different baselines. To address this issue, we will continue to work on efficient models (such as small model distillation and finetuning) to better preserve tokens.\n\n**Weakness 3:** The paper provides limited discussion on computational overhead and scalability, leaving questions about resource requirements for practical deployment.\n\n**Response:**\n\nThank you for bringing this up. Our initial experiments primarily focused on accuracy and localization performance, so we did not perform extensive optimization for computational overhead. Currently, our system runs using API-based access to LLMs, which means that the local infrastructure requirement is minimal—a standard CPU server is sufficient to orchestrate the agent pipeline.\n\nHowever, we acknowledge that scalability and cost-efficiency are crucial for practical deployment. In future work, we plan to serve our own models (e.g., fine-tuned variants) using lightweight LLM-serving frameworks such as Sglang, which would likely require GPU resources (e.g., A100 or H100) depending on the model size.\n\nFor additional details regarding cost analysis, including token usage and runtime cost on SWE-bench Lite, please refer to our response to Reviewer 3 (due to character limitations here). Thank you for your understanding!", "claims_and_evidence": "1. The claim regarding a 6.33% improvement in the final resolved rate due to integration with a patch generation component is quantitatively supported. However, this improvement is context-dependent, relying on the successful integration of another system, which might not generalize across different systems.", "methods_and_evaluation_criteria": "Both the methods and the evaluation criteria are thoughtfully chosen and make strong sense for the application.", "theoretical_claims": "The paper primarily focuses on presenting a novel framework with empirical supports.", "experimental_designs_or_analyses": "1. Although the experiments use a low-temperature setting (0.1) to promote deterministic behavior, the inherent variability of LLM outputs might still introduce some fluctuations in performance, which are not deeply discussed.\n\n2. The performance improvements, particularly the increase in the final resolved rate, partly depend on the integration with an external patch generation component. This dependency may affect the reproducibility of the results if similar integration conditions are not met.", "relation_to_broader_scientific_literature": "The paper extends existing fault localization and automated debugging techniques by integrating LLM-based methods with graph representations and hierarchical planning, drawing on ideas like CoT reasoning and repository-level code graphs. By benchmarking on SWE-bench datasets and comparing with systems such as AutoCodeRover and Agentless, the work demonstrates how its contributions address limitations in prior research while paving the way for more precise LLM-driven code search and context management.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The introduction of a priority-based scheduling mechanism for LLM-guided actions enhances the strategic exploration of large codebases.\n2. Decomposing high-level actions with relevance scoring allows for more precise localization by breaking down complex tasks into manageable sub-actions.\n3. The distance-aware context pruning method minimizes noise by filtering out irrelevant code, thereby maintaining focus on pertinent sections.\n\n\nWeaknesses:\n1. Heavy reliance on LLM outputs can lead to unpredictability and occasional hallucinations, affecting consistency in localization results. For example, different LLMs or even different LLM versions might yield different results.\n2. The evaluation primarily focuses on SWE-bench datasets.\n3. The paper provides limited discussion on computational overhead and scalability, leaving questions about resource requirements for practical deployment.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eDKmtLmst7", "reviewer": "Reviewer_BLXs", "summary": "The authors presented a new agentic framework called OrcaLoca to identify relevant code and resolve issues in software engineering problems. They introduce 3 new approaches: giving each action a relevancy score, maintaining a priority queue of actions where they are dynamically reordered based on contextual relevance and a context manager which prunes the search context.\nThese methods ensure that actions closely aligned with the issue are executed first resulting in more focused search. The context manager further helps in actively filtering out irrelevant context, reducing noise. Overall, this framework offers a precise approach to software issue localization compare, making it valualbe for finding and fixing issues in large scale codebase.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful and constructive feedback, as well as for highlighting several key strengths of our work. We also thank you for raising this important point regarding the discrepancy between Function/File Match performance and the Resolved rate in baselines.\n\nGenerally speaking, higher localization performance usually offers a better background for the stage of downstream editing (Fig.1). However, since the editors are different across different implementations, the final resolved rate is not guaranteed to be high by providing a higher localization rate. For instance, although HyperAgent and RepoGraph agents scored exactly the same in the Function Match in Table.1, HyperAgent's inadequate editing capability results in a final resolved rate less than that of HyperAgent.\n\nAnother reason is caused by agent workflow and algorithm stage plan. In modular systems where localization is seen as a separate step, localization provides an exact starting point that can greatly improve the quality of the final changes, which is shown in Table 2. However, not all agents follow a strictly modular localization → edit pipeline. Some approaches, such as swe-search[1], interleave editing, and localization stages, where editing feedback can influence and refine the localization in an iterative loop. In such designs, the final Resolved rate may align more directly with localization performance because the two components are tightly coupled. Kodu-v1 may also have particular algorithms to set up a tighter link between localization and edit stages. \n\n[1] SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement\nhttps://arxiv.org/abs/2410.20285\n\nNote: We discovered that we omitted to mention this study since it was under the Moatless tool and not reported on SWE-Bench. Later on, we will include this in our revised version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Most agentic workflows involve adding reasoning, few shot examples and improving retrieval for improving performance. However, they actions are usually static, and thus explorations can suffer from noise accumulation. OrcaLoca improves on that idea by introducing \n relevancy score to actions, priority scheduling mechanism and context pruning techniques - improving overall search.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths\n- The paper presents novel approaches to address the problems caused by static action prediction from LLMs, especially when search space is large. This could be translated to other agentic framework such as searching from large unstructured databases.\n\nWeaknesses\n- Although their appraoch leads to SOTA performance on Function and File Match, i.e., software issue localization problem, these improvements do not translate into the highest Resolved rate. The authors should comment on this discrepancy. It would be valuable to understand why method like Kodu-v1 achieved higher Resolved rate despite having significantly lower Function Match rate.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Nupw8Ej2L", "reviewer": "Reviewer_BMAE", "summary": "This paper introduces OrcaLoca, an LLM agent framework that improves software issue localization by integrating three key components: priority-based scheduling for LLM-guided actions, action decomposition with relevance scoring, and distance-aware context pruning. Experimental results demonstrate that OrcaLoca achieves good performance on the SWE-bench Lite benchmark with a function match rate of 65.33% and a file match rate of 83.33%. Additionally, by integrating patch generation capabilities, OrcaLoca improves the resolved rate by 6.33 percentage points compared to existing frameworks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful and constructive feedback, as well as for highlighting several key strengths of our work. For the responses for weakness, we listed our analysis part by part below:\n\n**Weakness 1:** The complexity of the approach\n\n**Response:** \n\nThank you for your suggestions. We are actively exploring strategies to simplify the workflow while maintaining localization performance. One promising direction involves using a reward model to efficiently pre-score relevant code indices in a single pass, followed by a more powerful LLM to process only the high-value candidates. However, these methods are still under experiment. The current area is relatively new so there exists a big design space to explore and have adventure. We are also very interested to see if there exists a more elegant and efficient localization method.\n\n\n**Weakness 2:** Computational overhead\n\n**Response:**\n\nThank you for highlighting the importance of a detailed computational cost analysis. We agree that comparing resource usage across approaches is valuable for readers.\n\nWe chose token cost as our metric because LLM inference dominates our system's overall time and money expenses. For time analysis, since we are majorly leveraging the API service from model providers, it could seen as a metric proportional to the token count.\nIn our revised manuscript, we will include a table summarizing the token cost for each agent:\n\n| Agent          | Cost |\n| ---|---|\n|      OpenHands | 1.14 |\n|      SWE-Agent | 1.62 |\n|  AutoCodeRover |  1.3 |\n|      Agentless | 1.05 |\n|       OrcaLoca | 1.77 |\n| OrcaLoca-batch | 1.48 |\n\nNotably, over half of OrcaLoca's cost is attributed to the editing phase (0.90 out of 1.77), which is influenced by the specific edit tool used.\nAlthough we majorly target performance and accuracy issues in our paper, localization has a large potential for efficiency optimization shown by OrcaLoca-batch. \n\nIn OrcaLoca-batch, we implemented a batched actions optimization for the localization process, where we extracted the top batch steps from the priority queue. \nThe table below summarizes the old and new token costs per instance, with the ratio (New Cost / Old Cost) indicating the improvement:\n\n| Inst ID            | Old Cost | New Cost | Ratio |\n| ---|---|---|---|\n|       django-13551 |      0.3 |     0.26 |  0.87 |\n|       django-15814 |     1.44 |     0.97 |  0.67 |\n|       django-16255 |     0.17 |     0.18 |  1.06 |\n|        pylint-7228 |     0.71 |     0.66 |  0.93 |\n|        pytest-8906 |     1.93 |     0.87 |  0.45 |\n| scikit-learn-13439 |     0.31 |     0.21 |  0.68 |\n|        sympy-14774 |     0.53 |     0.15 |  0.28 |\n|        sympy-15011 |     1.14 |     0.64 |  0.56 |\n|        sympy-16792 |     1.05 |     0.64 |  0.61 |\n|        sympy-24213 |     0.55 |      0.2 |  0.36 |\n\nDue to the cost budget of experiments, we sampled 10 issues from the original SWEBench-Lite with different levels of token cost. Using weighted averages based on sampled instances in each cost bin, we estimate that the per-instance cost for localization was reduced by an average of 34% (from 0.87 to 0.58) with no adverse effect on localization correctness.\n\nWe are also committed to further refining OrcaLoca and will explore additional optimizations (e.g., implementing kv-cache techniques) in future work.\n\n**Weakness 3:** Model dependency\n\n**Response:**\n\nWe agree that model generalization is an important concern. In our experiments, we have actually tested with models like GPT-4o and Gemini 2.0, and they worked well in our test samples, proving our agent framework is model-agnostic.  However due to budget constraints, we could not exhaustively evaluate across a wider range of LLMs. We majorly tested Claude because it was the best model with code reasonability at that time. In future work, we plan to include open-source models such as Qwen, LLaMA, and others—particularly once we fine-tune and serve our models locally, which will largely save cost for the repo-level benchmark evaluation.", "claims_and_evidence": "The claims in the paper are well-supported by comprehensive experimental evidence. The authors provide detailed performance metrics on the SWE-bench Lite benchmark, comparing OrcaLoca against 17 different approaches.", "methods_and_evaluation_criteria": "The evaluation methodology is sound and appropriate for the problem. The authors use established benchmarks (SWE-bench Lite and SWE-bench Verified) and metrics (Resolved Rate, Function Match Rate and File Match Rate) that are standard in the field.", "theoretical_claims": "The paper does not make formal theoretical claims requiring proof verification.", "experimental_designs_or_analyses": "I carefully examined the experimental designs and analyses presented in the OrcaLoca paper. Overall, the experimental methodology appears sound, with appropriate benchmarks, metrics, and comparative analyses.\n\nHowever, **there are also some limitations**: more in-depth analysis of ablation experiments, evaluation on multiple models, and evaluation of costs compared to other methods.", "relation_to_broader_scientific_literature": "The paper thoroughly discusses related work in LLM-based fault localization, positioning OrcaLoca's contributions in the context of existing approaches. The authors clearly identify limitations in previous systems such as Zhang et al. (2024a), Wang et al. (2024b), and Agentless (Xia et al., 2024), explaining how OrcaLoca addresses these limitations.", "essential_references_not_discussed": "The paper covers most essential references in the field.", "other_strengths_and_weaknesses": "**Strengths**\n- The paper addresses a significant challenge in Autonomous Software Engineering (ASE) with a novel approach that demonstrably outperforms existing methods.\n- The framework design is comprehensive, tackling multiple identified challenges in LLM-based localization.\n- The analysis of unique localizations (Figure 4) demonstrates OrcaLoca's ability to solve issues that other systems cannot.\n\n**Weaknesses**\n\n- **Complexity of the approach**: While the ablation studies show that each component contributes to performance, the overall design is complex with multiple interacting parts. The results show that removing individual components reduces performance by 3-5 percentage points, which raises questions about whether a simpler design might achieve comparable results with less complexity.\n\n- **Computational overhead**: The paper doesn't thoroughly discuss the computational costs associated with OrcaLoca's approach. While Section 4.1.3 mentions that \"the cost of searching is about 0.87, the cost of editing is about 0.90 per instance,\" a more detailed analysis of time and resource requirements compared to other approaches would strengthen the paper.\n\n- **Model dependency**: The experiments primarily use Claude-3.5-Sonnet-20241022, and it's unclear how the approach would perform with other widely used models like GPT-4o or open-source models. Testing with a broader range of LLMs would demonstrate the generalizability of the approach.", "other_comments_or_suggestions": "A more in-depth analysis of **complex components** would be helpful to improve this paper.", "questions_for_authors": "See about \"Other Strengths And Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PeCBkWwr6i", "reviewer": "Reviewer_9wos", "summary": "This paper proposes a novel framework for automated software issue localization tasks. Issue localization is a critical component of autonomous software engineering, making the problem addressed in this paper well-motivated. The proposed technique combines several strategies to improve localization accuracy and efficiency:\n- Use of a priority queue to identify higher-relevance or more urgent actions first.\n- Decomposition of search actions for finer-grained and more relevant operations.\n- Preservation of the most relevant code snippets as the agent iteratively retrieves them using a shortest-path heuristic on the code graph (a graph-based representation of the codebase that facilitates indexing and efficient search while capturing both code containment and references).\n\nExperiments are conducted on the widely used SWE-bench Lite benchmark for Python projects. The authors report:\n- A new open-source state-of-the-art Function Match Rate of 65.33% (an approximately 2% improvement over the previous open-source best).\n- A File Match Rate of 83.33%.\n- A final Resolved Rate (percentage of issues successfully fixed by the generated patches) of 41.00%.\n\nAdditionally, an ablation study is conducted on a smaller subset of SWE-bench, demonstrating that each component of the proposed approach positively contributes to the overall improvements.\n\n## update after rebuttal\nThank you for your response to my questions. I don’t have any further questions at this time. After considering all of the discussions here, I’ve decided to keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and for highlighting several key strengths of our work. We listed our responses for weaknesses and questions below:\n\n**Weakness 1:** While the overall approach is innovative, parts of the paper are difficult to parse... Ensuring the paper is self-contained and introducing the agent framework more gradually would significantly improve readability.\n\n**Response:**\nThank you for your useful advice. We agree that Section 3.1 contains a lot of technical information in a small space, which may hinder readability. In the revision, we will summarize the APIs utilized in previous systems (Zhang et al., 2024b; Ma et al., 2024a) to make the paper more self-contained and accessible to a wider audience.\n\n**Weakness 2:** The approach is tested entirely on Python issues. It’s unclear how well the code-graph approach generalizes to Java or cross-language settings.\n\n**Response:**\nThis is an important point. Currently, we rely on SWE-Bench, which is the state-of-the-art benchmark so far for evaluating software agents and is Python-based. We recently found more new benchmarks like SWE-Gym and LocBench, but these are also Python-based. We recognize the need for broader generalization and are actively working on developing a cross-language benchmark to evaluate agent performance in more diverse environments. Our graph-based framework could be improved further to support other languages with appropriate parsing and indexing backends.\n\n**Q1:** How do you envision OrcaLoca’s performance scaling for extremely large repositories (e.g., tens of thousands of files, multi-million-line codebases)?\n\n**Response:**\nThis is an excellent question. Scaling to extremely large repositories such as Linux or Android codebases is an exciting but challenging direction. Key bottlenecks include efficient index management and the context management of CoT (Chain-of-Thought) reasoning during retrieval. In future work, we aim to incorporate an efficient Process Reward Model (PRM) to compress reasoning chains while reducing overall token consumption. We are also exploring hierarchical indexing and chunking strategies to handle large codebases more efficiently.\n\n**Q2:** Do you anticipate any major challenges when applying OrcaLoca to other programming languages or multi-language repositories?\n\n**Response:**\nApplying OrcaLoca to other single-language repositories generally presents no fundamental issues, as our system operates on the code structure and relationships. However, languages with strong type systems (e.g., C++, TypeScript) offer opportunities for type inference integration, which could enhance LLM reasoning by leveraging compiler-level information. This could reduce reasoning effort and improve localization accuracy.\n\nFor multi-language repositories—especially common in ML systems (e.g., Python + C++ via PyBind)—the main challenge lies in cross-language linking. We are currently extending our index to infer connections across language boundaries and enhance reasoning with cross-language semantic context. This will require both richer static analysis and more advanced reasoning strategies.\n\n**Q3:** Your approach relies on the shortest paths in a code graph. Are these distances cached or computed dynamically?\n\n**Response:**\nCurrently, shortest-path distances are computed dynamically, as our current system serves primarily as a proof of concept. That said, we recognize the performance bottleneck this can introduce. In future iterations, we plan to add a caching layer and incremental computation mechanisms to improve efficiency, especially for large or frequently accessed repositories.", "claims_and_evidence": "Overall, the paper’s main claims are well-supported by experimental results. Specifically, it provides detailed evidence for the following:\n- OrcaLoca achieves a new state-of-the-art function match rate among open-source systems.\n- OrcaLoca improves the final resolved rate by 6.33 percentage points over a baseline system.\n- Priority-based scheduling and action decomposition enhance relevance and reduce noise in context.", "methods_and_evaluation_criteria": "The authors use standard SWE-bench metrics (Resolved Rate, Function Match Rate, and File Match Rate) along with a new metric, Function Match Precision, to measure localization accuracy. This approach is reasonable and aligns with established practices in the emerging field of LLM-based bug fixing.", "theoretical_claims": "The paper does not present any formal theoretical claims or proofs.", "experimental_designs_or_analyses": "OrcaLoca is tested on the SWE-bench Lite, a widely used curated set of 300 real-world GitHub issues. The authors also conduct an ablation study on a smaller subset, SWE-bench Common. They compare with 17 solutions (both closed-source and open-source). I think this is fairly comprehensive evaluation of their technique. The paper also systematically removes each of the major components to gauge its importance on localization performance.", "relation_to_broader_scientific_literature": "The related works section of the paper is fairly comprehensive and includes sufficient details on prior LLM-based fault localization and debugging frameworks (including Agentless, AutoCodeRover, Repograph, as well as spectrum-based and mutation-based approaches for debugging). I agree with the authors that the main novelty of this paper is the application of multi-step agent designs to codebase-level tasks. The approach successfully bridges the gap between large LLM text-generation capabilities and more “surgical” code exploration techniques.", "essential_references_not_discussed": "In general, the references from the emerging field of LLM-based bug fixing are fairly comprehensive.", "other_strengths_and_weaknesses": "Strengths:\n- The three major components (priority scheduling, decomposition, distance-based pruning) are distinct yet cohesive ideas. I think the paper is giving future researchers a clear blueprint for adaptation or replication. \n- Strong empirical results: SWE-bench Lite is a practical, recognized dataset, making the results fairly credible. The jump in function match rate and final resolved rate, plus ablation tests, provide strong evidence that each piece is adding value. \n\nWeaknesses:\n- While the overall approach is innovative, parts of the paper are difficult to parse. Section 3.1, in particular, introduces multiple details in quick succession. More introductory context would help guide readers through the agent workflow. Additionally, the paper references external APIs from prior systems (e.g., “Consider the previous agent API used by systems like (Zhang et al., 2024b; Ma et al., 2024a)”) without summarizing them, which may confuse less-informed readers. I encourage the authors to use available space more effectively to provide sufficient grounding before diving into technical details. Ensuring the paper is self-contained and introducing the agent framework more gradually would significantly improve readability.\n- The approach is tested entirely on Python issues. It’s unclear how well the code-graph approach generalizes to, say, Java projects or cross-language scenarios.", "other_comments_or_suggestions": "It would be interesting to see a break-down of performance across different types of issues (e.g., small vs. large repositories, etc.).", "questions_for_authors": "- Q1: How do you envision OrcaLoca’s performance scaling for extremely large repositories (e.g., tens of thousands of files, multi-million-line codebases)?\n- Q2: Do you anticipate any major challenges when applying OrcaLoca to other programming languages or multi-language repositories?\n- Q3: Your approach relies on shortest paths in a code graph. For large repositories, do you cache or approximate these distances, or are they computed dynamically each time?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript proposes a new framework for software issue localization -- one of the long-standing challenges in software engineering. The framework is well designed and presented with rigorous empirical evaluation on some of the most challenging repository-level issue resolution benchmarks. All the reviewers call out the technical merits of the work, and I concur with the reviewers. Using ReAct style frameworks -- that are fairly general and extend to pretty much any new domain --- is the most common strategy in software agents and in particular for code issue resolution that involves navigation of large codebases, invocation of search tools, as well as reasoning over extracted code snippets. It's great to a see a well-thought out framework designed especially for the localization problem, which is a significant component to the success of coding agents. 

The paper makes clear technical contributions and I recommend acceptance. There a few minor issues the reviewers raise (about in-depth analysis, ablations, cost analysis, etc.) which the authors do address in their rebuttals. I encourage the authors to revise the final version to include these analyses.
Decision: Accept (poster)