Nemotron-CORTEXA: Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Authors: Atefeh Sohrabizadeh
Jialin Song
Mingjie Liu
Rajarshi Roy
Chankyu Lee
Jonathan Raiman
Bryan Catanzaro
Venue: ICML 2025 poster (2025)
Keywords: Coding assistant
code embedding model
localization agent
SWE-bench
prompt design
contextual information
multi-LLM
multi-prompt
Primary area: deep_learning->large_language_models
TL;DR: More accurate localization and diverse solutions enhances LLM software agents

Abstract: Large Language Models (LLMs) have demonstrated significant potential in code generation by following natural language instructions.  Unfortunately, crucial real-world software engineering tasks, such as debugging or repository-level feature implementation, involve processing extensive contexts beyond current LLM context sizes and performing complex reasoning that is brittle using standard autoregressive decoding. Enhancing LLMs' performance in these scenarios requires careful consideration of the contextual information provided to the model, optimizing how the model leverages that, and identifying tools that enable more effective navigation of the development environment.
To address these challenges, we introduce Nemotron-CORTEXA, an agentic system built on a predefined scaffold that enhances LLMs' ability to navigate and reason efficiently in complex software engineering contexts. Specifically, we develop a novel code embedding model that retrieves the most relevant files with greater precision, along with a localization agent that refines the granularity of the retrieval process. Additionally, we demonstrate that providing diverse contextual information and utilizing different prompt formats enable the model to identify and resolve issues more efficiently. We evaluate Nemotron-CORTEXA using SWE-bench, a benchmark derived from real-world GitHub issues. Compared to the widely used Agentless framework, Nemotron-CORTEXA achieves a higher issue resolution rate at a lower cost, highlighting its practical impact in addressing real-world software engineering challenges.

NEMOTRON-CORTIE
Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Atefeh Sohrabizadeh" *
Jialin Song"
Mingjie Liu'
Rajarshi Roy' Chankyu Lee! Jonathan Raiman"
Bryan Catanzaro
Abstract
1. Introduction
Large Language Models (LLMs) have demon- strated significant potential code generation by following natural language instructions. Unfor- tunately, crucial real-world software engineering tasks, such debugging repository-leve fea- ture implementation, involve processing exten- sive contexts beyond current LLM context sizes and performing complex reasoning that is brit- tle using standard autoregressive decoding. En- hancing LLMss performance in these scenarios requires careful consideration of the contextual information provided to the model, optimizing how the model leverages that, and identifying tools that enable more effective navigation of the development environment. To address these chal- lenges, we introduce agentic system built on a predefined scaffold that enhances LLMs' ability to navigate and reason efficiently in complex software engineering con- texts. Specifically, we develop novel code em- bedding model that retrieves the most relevant files with greater precision, along with a localiza- tion agent that refines the granularity of the re- trieval process. Additionally, we demonstrate that providing diverse contextual information and uti- lizing different prompt formats enable the model to identify and resolve issues more efficiently. We evaluate NEMOTRON CORREEA using SWE- bench (Jimenez et al., 2023). a benchmark derived from real-world GitHub issues. Compared to the widely used Agentless framework (Xia et al., 2024), TEMOTRON-CORTEXA achieves a higher issue resolution rate at lower cost, highlight- ing its practical impact in addressing real-world software engineering challenges.
Recent advancements in Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, ranging from creative writing to analytical reason- ing (Wei et al., 2022; Yuan et al., 2022; Gomez-Rodriguee & Williams, 2023; Ahn et al., 2024). The influence of LLMs on software engineering has been especially widespread and recognized. Early models showcased their proficiency in generating syntactically correct code and solving interview- style programming questions (Chen al., 2021; Roziere et al., 2023; Li et al., 2023; Guo et al., 2024). Building on this progress, recent efforts in LLM research on program- ming have shifted from snippet and function-level gener- ation towards developing LLM coding agents capable of tackling entire real-world software engineering tasks (Yang et al., 2024; Xia al., 2024; Wang et al., 2024a; Antoni- ades al., 2024; Liu et al., 2024b; Xie et al., 2025). These coding agents are equipped to utilize tools commonly used by human engineers, including Command Line Interfaces (CLI) and Integrated Development Environments (IDE), to perform critical tasks such as debugging and applying patches in a repository. A key benchmark for evaluating the effectiveness of these LLM agents is SWE-bench (Jimenez et al., 2023), which features real-world software engineering problems derived from GitHub issues, aimed at generating patches that successfully resolve them. The rapid improve- ments on the SWE--bench leaderboard highlight the growing interest in building powerful LLM software agents and un- derscore their potential to automate software engineering tasks. In this work, we propose several ways to simultane- ously enhance the efficiency and accuracy of LLM agents within this context, and use SWE-bench
MAMMOTRON-CORTER
an
benchmark.
System design decisions play a crucial role in the end-to-end performance of an LLM software agent, since worl-world software engineering tasks often involve processing a large code repository and reasoning about relations among mod- ules. practical example of the impact of system design can be found in the coding agents of OpenHands (Wang et al., 2024a) and SWE-Agent (Yang et al., 2024): the au- thors allow an agent to perform free-form action sequences guided by system prompts that outline the available action
Equal contribution 'NVIDIA, Santa Clara, CA, USA. Correspondence to: Atefeh Sohrabizadeh <asohra- bizad@@nvidia.com Jialin Song <jialins @oviia.@nniii..............
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
types. While such free-form flows can be effective, the large action space often results in many costly iterations and makes debugging difficult. To mitigate this, Agentless (Xia et al., 2024) employs more structured approach. guiding the L.LM through a predetermined sequence of steps. De- spite these advances, challenges remain to accurately locate the problem and generate correct repairs.
We propose diverse solution generation method that leverages different contextual information and varied prompt formats, significantly enhancing sample effi- ciency.
Experimental results demonstrate that NEMOTRON- CORTEXA outperforms Agentless, OpenAl's go-to ap- proach for showcasing their real-world coding perfor- mance, while being more cost-effective.
(Code Optimization for Repository Tasks with EXecution Agents), coding agent designed to strengthen issue localization and repair genera- tion. First, we develop a dedicated code embedding model that optimizes the retrieval of relevant files and a localiza- tion agent that refines the granularity of issue localization. Second, we equip the localization agent with programming tools based on Abstract Syntax Tree (AST) and Language Server Protocol (LSP) that enable compact code naviga- tion by taking advantage of multi-step reasoning. Third, we adopt an ensemble of LLMs approach that improves localization accuracy and helps identify additional relevant code elements that can improve the repair process. The inspiration for ensembling was drawn from our observation that localization agents instantiated with different LLMs perform particularly well on different problems, as also noted in (Zhang et al., 2024a). We can explain this bene- fit by observing that even when models fail to accurately identify the exact entity (function or class) causing an is- sue, they nonetheless often identify closely related entities. Our optimizations here improve the localization precision over Agentless, a competitive approach on SWE-bench that is widely used in model benchmarking (Liu et al., 2024a; Jacch et al., 2024), by 18.08% and recall by 11.32%, average. In addition, we show that the introduction of diver- sity in the repair stage can further enhance performance. To induce diversity, we develop a simple and efficient method that leverages different contextual information and multiple prompt formats during patch generation. This approach outperforms standard temperature sampling (Brown al., 2024; Xia et al., 2024) and allows NEMOTRON- REMOTRON-CORTEXA to achieve higher issue resolution rate while minimizing the number of inference calls for patch generation by 4.4x outper-
We present NEMOTRON-CORTEE
2. Related Works
2.1. LLMs for Code Generation
LLM progress in code generation has been driven by re- search efforts to specialize them through continued pre- training on code data (Guo et al., 2024; Bai et al., 2023; Roziere et al., 2023; DeepSeck-Ale ct al., 2024) sourced from open repositories (Kocetkov et al., 2022; Lozhkov et al., 2024) and commit histories (Muennighoff et al., 2023). Fur- ther enhancements come from reinforcement learning (Le et al., 2022) and, more commonly, instruction fine-tuning. Instruction fine-tuning incorporates techniques to solve com- plex coding challenges (Luo et al., 2024), broadens model capabilitics by leveraging unlabeled open source code (Wei et al., 2023; Yu al., 2024; Wu et al., 2024), ensures solu- tion accuracy with self-generated tests (Chen et al., 2022), and improves code validation and debugging via interactions with LLM agents (Lei et al., 2024).
2.2. LLM Agents for Software Engineering
Real-world software engineering tasks are more complex than solving interview questions evidenced by the devel- opment of SWE-bende (Jimenez et al., 2023). Recent rapid progress in designing LLM agents highlights the importance of this application. SWE-agent (Yang ct al., 2024) enables an LLM agent to interact with the development environment for tasks like search and edit. AutoCodeRover (Zhang et al., 2024b) builds more sophisticated code search tool based on the abstract syntax tree. Multi-agent systems perform divide-and sonide-and-conuuee strategy by developing specialized agents for each sub-task (Liu et al., 2024b; Wang et al., 2024a). Agentless (Xia et al., 2024) takes a different ap- proach by designing a multi-step pipeline where LLMs com- plete each step. Iterative approaches and self-reflections, such as in (Chen et al., 2023; Shinn et al., 2024) improve coding performance, with SWE-Search (Antoniades et al., 2024) integrating self-feedback within a Monte-Carlo Tree Search framework to refine strategies.
compared to Agentless. NEMOTRON-CORTES forms Agentless by achieving an issue resolution rate of 42.00% on the SWE-bench Lite and 52.60% on SWE-bench Verified, while costing only $0.51 per instance.
In summary, main contributions are follows:
We develop a code embedding model specialized in re- trieving relevant code chunks a given bug, achieving thte-off-the-ar file retrieval accuracies on the SWE-bench benchmark.
2.3. Code Retrieval
We design a localization agent that integrates advanced programming tools and leverages an ensemble of LLMs to deliver more precise and granular issue localization.
Code retrieval is a critical task in domains like code expla- nation, code summarization, and documentation lookup, as
2

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
shown by recent benchmarks (Li et al., 2024; Wang al., 2024b). Early efforts to create code-specific embedding models include BERT-based (Devlin, 2018) models such as CodeBERT (Feng et al., 2020) and UniXcoder (Guo et al., 2022). While there has been a growing interest in develop- ing decoder-only LLM-based embedding models for text re- trieval tasks (Neelakantan et al., 2022; Moreira et al., 2024b; Meng et al., 2024; Lee et al., 2024). few models target code retrieval. Notable exceptions include the models developed by Voyage AI (Voyage AI, 2024) and OpenAl (Neelakan- tan et al., 2022). More recently, 2024c) emerged as a family of code embedding models, excelling in many code retrieval tasks. However, limited focus has been given to code retrieval for identifying buggy code based on issue descriptions.
tions: the LLM is expected to make decisions based solely file names, which can lower its accuracy, or depend on prior knowledge of these repositories. On the other hand, processing the entire codebase and passing the full contents of each file to it would be prohibitively expensive. To tackle this, we develop a coding embedding model specifically trained to retrieve the most relevant files based on the is- sue description. Our model, called NV-EmbedCode! is fine-tuned from an existing text-based embedding model that uses bidirectional attention followed by average pool- ing (Muennighoff et al., 2024; Lee et al., 2024; Moreira et al., 2024a).
(Liu et al.,
The retrieval task involves identifying the "oracle" file- the file requiring edits to resolve the issue - based on the issue description. Since files can be large, we chunk them and prefix each chunk with meta- meta-informan information, such as the full file path, to help the model maintain context about the chunk's location. We curate dataset specifically for this purpose, mapping issues to code edits. The relevant pas- sages for retrieval are the chunks from the oracle files. Once chunk is retrieved, the oracle file is located by referencing its meta-information. The issue descriptions often may have redundant information, such as package version details, or insufficient details, like ambiguous buggy code with little explanation. To mitigate this problem, we prompt an LLM to generate a concise summary that includes all relevant information, such as filenames, functions, error messages, stack traces, and other technical details. We then create a second dataset containing these summaries, while selecting the positive passages the same way as before.
3. NEMOTRON-CORTES
3.1. Overview of the Pipeline
Since an entire codebase frequently exceeds the context win- dow of present LLMs, we are obligated to subsample the relevant files to provide a context that fits yet is sufficient for resolving the issue. Consider that issue descriptions are typi- cally under 1000 tokens, while repository codebases contain orders of magnitude more tokens. For example, SWE- instances have on average 195 words per issue for 438k lines of code (Jimenez et al., 2023). Our approach is therefore to use the issue description to retrieve relevant files that form a context that fits and is likely to be sufficiently informative to resolve the issue. This retrieval not only ensures the model has the right context but also helps filter out irrelevant or confusing information. Therefore, similar to Agentless (Xia pipeline is divided
To enhance the embedding model's understanding of code data, we augment our training set with additional publicly available datasets for text-to-code (retrieving code documents based on a textual query), code-to-code (retrieving code documents based on a code query), and hybrid code (retrieving a combination of code and text documents based on hybrid query) retrieval tasks, as categorized by (Li et al., 2024). We also extend the hybrid code category by generating a synthetic dataset where an LLM creates code in response to a query. To help the model differentiate between these datasets, we prefix the queries with dataset-speciffe instructions following the approach in prior works (Wang et al., 2023; Lee et al., 2024) with the template "Instruct: { cdataset-instructt Similar to NV-Embed (Lee et al., 2024), we mask out the instruction tokens during average pooling.
et al., 2024), the NEMOTRON-COREE into two main stages: localization and repair, as shown in Figure 1. The localization stage involves two steps: first, identifying the most relevant files (Section 3.2), and then refining the granularity of the retrieval to focus on specific functions, classes, or methods (Section 3.3). During the repair stage, we combine different contexts and prompts to increase patch diversity (Section 3.4), followed by filtering process to select the most promising patch (Section 3.5).
3.2. File Localization with Code Embedding Model
Accurately localizing the issue to the relevant files is es- sential for the success of the pipeline. If this step fails, the model cannot produce an effective patch, making the remain- der of the pipeline irrelevant. To tackle this, Agentless cre- ates a concise repository representation, similar to the Linux tree command, and provides it to the LLM identify the top N suspicious files. It also experiments with using gen- eral embedding model but finds that the embedding- based approach performs worse than the prompt-based method. However, the prompt- based approach has its own limita-
Query: {query)".
We train the embedding model with a contrastive learning objective (Gao et al., 2021) to maximize the similarity be- tween the query and its relevant (positive) passage, while
Available at https:: /bui 1 aaniiiii www.coooo/iioo.e hwn//vidaal nv- embedcode- 7b-v1
3

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
pargstermi/nigeh... der int. bett meeat cEe
Hanoodterrd F2_ TV VERSONNOUO Ro
6. File Localization
Aanosftmmal ogein po! cims fet FR roef modile oefreds prspeted but applizzoon)
PC SefeR DN Media mender 30
(b) Exiity Locaizzatonn
shhunkkng
Teat Codl
pet an Cangourrinn opplication
Reccotin modes
Mangeeeeress detreds pespetyy up pk 17
Order bo Siniin
Text Cell Randia
The publo inteface
Selction
L salbbrrahhiggg oo! cat cofresser josen ferioul lgangeoon der get rrag applictennt:
pe srreng decenceryl E magt application
Certon Autionin
LLM Summmary
feom sroven
Prcll hun.
Generation
Patch
tser para
Oenesto corplee kes
2. (b) Patch
Selection
83 43 or _M
Petch Fitter with Teats
Vote
BRAC
bnd
by:
Figure 1. Overview of NEMOTRON- NEMOTRON-CORTEXA From the issue description, REMOTRON-CORTEXA first retrieves relevant files using NV-EmbedCode, our specialized code embedding model, then refines issue locations to granular entities with a localization agent. We use novel strategy generate diverse candidate patches and select based on L-L--genertee tests and majority voting.
minimizing the similarity to irrelevant (negative) passages. -warr We adopt the positive- aware hard negativee mining technique from NV-Retriever (Moreira et al., 2024b) to minimize the likelihood of including false negatives in the data. Since the base model may not fully understand code data, we fine-tune it and subsequently mine new hard negatives from the refined model to improve the quality of the hard nega- tives. Each batch includes samples from all datasets, but we avoid using in-batch negatives to prevent conflicts between positive and negative samples. This problem occurs when chunking the oracle files to create positive passages, as single query may correspond to multiple positive passages. If these query-positiv pairs appear in the same batch, it can lead to misleading negative and positive samples.
rale: sepurable
cur sepanabiiityy transfore:
Fandas
separabli watrrb _ewppratcaaaaac.............. www.naccctrrraattt trassfore
Cable
the
Caran
uperadayy
transfers.
'saft"
tranufore r_oospaa
def jeeued sarllaaa pas, eaurp)
(a) Code Example
(b) Code Graph
Figure 2. Example code and its graph representation.
and tracing call stacks. To facilitate structured navigation of the repository, we start by building its graph representa- tion. The nodes represent code files and entities functions, classes, and class methods - extracted using AST. Fach node stores the corresponding code text and its position metadata. We establish two types of directed edges: contain and use. The contain edges establish hierarchical relationships, con- necting files to their functions and classes, and classes to their methods, thereby providing agents with structural view of the repository. Complementarily, use edges capture functional dependencies, including function invocations and class instantiations. For example, in Figure 2, the graph contains node for the file separable py and a node x,
3.3. Entity Localization with Localization Agent
With the code embedding model, we can obtain short list of files relevant to the issue. However, the full texts of these files often still exceed a model's context window. We need to narrow down the issue locations further. A code file consists of high-level entities, such classes and functions, the next step of localization is to identify can- didate entities from few top-ranked files. Agentless uses a direct prompting approach, providing a skeleton format of each file containing only entity headers and asking an LLM to select the relevant ones. However, without access to the code content, the model lacks sufficient context to respond accurately. Inspired by information foraging theory in software debugging (Lawrance al., 2010), we develop localization agent that iteratively explores the repository by performing common actions such as looking up definitions
each for the three functions separability_mate The file node has
sesaarrbllll and mndd__oor_________ edge pointing to each function with the contain type. separabi libbiiiityyy. tty_matri x node has an edge pointing to separable with the use type, as does ssparaableeeeeeeeee to petordxaatri..
We equip the localization agent with three tools to navigate

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
through the graph. They are reminiscent of features in popular IDEs such Go To Definition and Call Hierarchy. We use an LSP server implement these tools.
even with all other factors held constant.
To explore this fürther, we studied the effect of two spe- cific edit formats on the resolved instances of SWE-bench. These formats are illustrated in Figure 1. The first format, referred to as the sea rechrrepllle raplace format, consists of search component that includes the original code snippet to be replaced and a replace component that specifies the replacement content. The second one an edit_fi fflt API that specifies the file path, the starting and ending line numbers of the code to be edited, and the content to be in- serted or modified. Additionally, we evaluated the effect of two different contexts within the code section of the prompt: the entities identified in Section 3.3 and the top-retrieved file from Section 3.2.
get definition(file name, entity _name): returns the code definition of the specified entity in the requested file.
get incoming incoming.dependenee dépendeney(tle name, entity _amee: re- turns the definition codes of the specified entity's source nodes using the list of its incoming edges.
get wwwooottggggeeeoeoooo dependency(flle _ame,,, entity name): re- turns the definition codes of the specified entity's destina- tion nodes using the list of outgoing edges.
Concurrent to our work, MarsCode Agent (Liu et al., 2024b) also builds a code knowledge graph from a repository with AST and LSP. Our version differs by focusing only on high- level entities and providing additional tools built on top of standard LSP functions.
11.0%
2.0%
The localization agent leverages the file ranking results from the code embedding model, focusing the top-ranked ones. For these files, we create concise representation containing only definition signatures of high-level entities. These signatures, combined with the issue description, are then fed into an L.M to identify a few entities relevant to the issue description. This step is similar to Agentless's s direct prompting approach after identifying suspicious files (Section 3.1.2 in (Xia et al., 2024)). Next, the localization agent examines the full code of the chosen entities and utilizes tools to inquire about related entities. Each tool call returns a collection of new code snippets that the agent stores in its context for future queries. The agent iterates through multiple interactions with the repository to gather sufficient context. Since excessive tool calls can lead to context overload, the agent is tasked to filter the context and keep only the relevant ones in memory buffer. After a few iterations, the localization agent is prompted to select a final set of entities requiring modifications to fix the issue.
2.0%
2.1%
4,0%
Contant: entity Frorept: searthheplace Tamp: 6.8 (panepi)
Content: entity
Frorept: ediiffs
0.0 (cessi33)
Content: -Pranpt: editffle
TTorp:: (pensg))
Figure 3. Venn diagram showing the distribution of the percent- age of resolved instances across context and edit format changes for SWE-bench Lite. Different combinations of context and edit formats demonstrate varying strengths, here ranging from 19.3% to 32%, and cover different instances which, can contribute to higher overall pass rate of 40.3%.
Figure 3 depicts the Venn diagram of the resolved instances when we vary these choices of contexts and edit formats. The numbers in each section indicate the percentage of re- solved instances in SWE--bechh Lite that fall into that subset. While it is expected that changes in context would result in different instances being resolved, the diagram also high- lights that the final fix is highly sensitive to the choice of the edit format. Different combinations of context and edit format exhibit varying strengths in resolving the instances and cover very different instances. Existing approaches focus on identifying the single most effective choice and then rely temperature sampling to generate diverse solu- tions. In contrast, we propose changing both the context and the edit format, in addition to temperature sampling. This method increases solution diversity, which in turn increases the number of resolved issues while reducing the number of inference calls needed for patch generation.
3.4. Diverse Patch Generations
After localizing the issue and narrowing it down to specific suspicious entities, the next step is to generate a patch that resolves it. In this step, the L.LM is prompted to generate a patch based on the provided code context. Instead of producing the entire code, the LLM typically is prompted to generate only the edit, either as a search/replace format (Xia et al., 2024; Aide, 2024) or git diff format (Jimenez al., 2023; Yang et al., 2024). Our observations indicate that the choice of context and prompt format significantly impacts the correctness of the generated patches, leading to varied sets of resolved instances. Specifically, LLMs are highly sensitive the format of the requested edits, often yielding different solutions when only the patch format is changed,
3.5. Patch Selection
After generating a diverse set of patches, we apply a series of filtering steps identify the most promising patch as the
5

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
final solution. First, we remove solutions that fail to produce valid edit instructions or result in syntax errors. Next, we normalize the edited codes by removing comments, doe- strings, and empty lines, as well as standardizing variable, function, and class names. Using these normalized versions, we record the frequency of the repeated solutions. We then use regression tests and the generated reproduction tests to filter out solutions that fail these tests. Our approach closely follows the methodology outlined in Agentless (Xia al., 2024) and directly leverages their released artifacts?. In this process, Agentless first executes all existing tests in the repository to identify a subset of passing tests that success- fully run on the original codebase. The passing tests are then analyzed by an LLM, which identifies any test that is not suitable verify whether the issue has been correctly fixed. The remaining tests are designated as regression tests. For reproduction tests, Agentless generates a complete testing file designed to both reproduce the original issue described and verify whether the issue has been successfully resolved. These tests are executed on the original repository to filter out any tests that fail to detect the issue. We retain all valid reproduction tests at this stage. In cases where no solutions remain after filtering, we re-include all solutions available apply majority voting
Table I. Overall comparison over SWF--ench Lite and Verified.
Claude refers to the Soaudde3,5-Sommee model.
% Resolved
Avg.
Method
LLM
Lite
Verified
$ Cost
125
265
OpenHands
Claude
(41.67%)
(53.00 %)
(Wang et al., 2024a)
78
1.10
(26.00%)
122
254
Agentless
Claude
(40.67%)
(50.80%)
(Xia ct al., 2024)
96
GPT-4o
0.70
(32.00%)
Moatless Tools (Team, 2024a)
115
Claude
(38.33%)
AutoCodeRover (Zhang et al., 2024b)
92
192
GPT-4o
(30.67%)
(38,40%)
SWE-agent
69
168
Claude
(Yang et al., 2024)
(23.00%)
(33.60%)
126
263
NEMOTRON- CORTEXA
Mix
0.51
(42.00 %)
(52.60%)
et al., 2024). Additionally, we synthetically generate a dataset of code produced from coding queries in Code- Feedback (Zheng et al., 2024) by prompting DeepSeek- v2.5 (DeepSeck-AL, 2024). The LLM summaries mentioned in Section 3.2 are generated using Llama-3.1 405B (Dubey et al., 2024). The final dataset contains approximately 534k pairs of query-positive documents, as detailed in Ap- pendix A.1
before applying the test. Finally, on the remaining solutions based on the frequency of their repetition to select the final patch.
4. Experiments
We build the code graphs using tree-sitter' to extract entities from code files. We implement the three code navigation tools in Section 3.3 with pylsp". The localization agent starts with the top-6 files according to our code embedding model and performs a maximal of 5 rounds of entity local- ization with tools. The agent can issue multiple tool calls at each round. We instantiate the localization agent with four different LLMs: DeepSeek-v3 (Liu al., 2024a), GPT- 4o (OpenAI, 2024), Llama-3.3 70B (Dubey et al., 2024), and Qwen2.5-72BB IInttrcctt (Team, 2024b; Bai et al., 2023). The final localization result is an ensemble of them. During localization, we use greedy sampling to generate responses. The patches are generated using Claude 3.5 Sonnet.
4.1. Experiment Setup
To train the embedding model, we use a Parameter-Efficiee Fine-Tuning (PEFT) technique called Low-Rank Adapta- tion (LoRA) (Hu et al., 2021). Specificallyy text embedding model with contrastive loss, applying LoRA with a rank of 16, alpha of 32, and a dropout rate of 0.1. Our base model is NV-EimbedQA--Mistraa--a (NV-EmbedQA-v2 for short) (Moreira et al., 2024a), a com- mercially available version of NV-Embed-v2 (Lee et al., 2024). We employ the Adam optimizer (Kingma, 2014) with 200 warm-up steps, learning rate of le-6, and linear decay. The training is conducted using Bfloat with maximum sequence length of 512 tokens. The model is fine-tuned with a batch size of 64, where each batch consists of a query, one positive document, hard negatives, and we do not use in-batch negatives, as explained Section 3.2.
fine-tune
4.2. End-to-end Issue Resolution Results
We validate bench (Jimenez et al., 2023) Lite and Verified, which consist of 300 and 500 instances, respectively. Table I summarizes the comparison to five representative open- source baselines: OpenHands (Wang al., 2024a), Moatless Tools (Team, 2024a), AutoCodeRover (Zhang et al., 2024b), SWE-agent (Yang et al., 2024), and Agentless (Xia et al., 2024). NEMOTRON-CORTEXA
on SWE-
The training data is curated from the SWE-hench train- ing set (Jimenez et al., 2023) and portion of the ColR training set (Li et al., 2024), which includes data from APPS (Hendrycks et al., 2021), CoSQA (Huang et al., 2021), CodeTransOcean (Yan et al., 2023), StackOver- NowQA (Overflow, 2021), and SyntheticTTxx12SQL. (Meyer
6

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
achieves the highest resolution rate of 42% (126 / 300) on Lite and the second highest rate 52.60% (263 / 500) on Verified. Notably, NEMOTRON- CORTEXA achieves its performance at an average cost of only 10.51 per issue. In comparison, OpenHands reports a 26% resolution rate on SWE-bench Lite at $1,10/issu,, and Agentless reports 32% at $0.70/is Further scaling test-time compute in MMMOTRON-CORTESS can yield higher resolution rates CINOOTROO-CORTE....
Table 2. Recallåk in retrieval for SWE- -rEche Lite and Verified. where k is the number of oracle files, using either the original issue description or its L..M-generated summary as input.
Method
Model / Approach
Issue
Lite
Verified
Original
42.33%
40.67%
Lexical
BM25
Summary
38.00%
41.75%
Prompt*
GPT-4o
Original
63.00%
65.55%
here. Our contributions drive strong performances at minimal cost: our code embedding model and localization agent provide more accurate and informative contexts for LLMs, while our novel diverse solution generation strategy significantly reduces the number of candidate generations needed. We will break down their impacts in the next sections.
Original
60.33%
61.00%
EiReeddedd 2.R
Summary
55.33%
55.46%
Original
56.00%
59.90%
NV-Embed-v2
Summary
57.67%
59.62%
Embedding
Original 61.33%
62.81%
NV-EmbedQA-v2
Summary
55.67%
59.20%
Original
67.67%
68.37%
NV-EmbedCode
Summary
70.33%
71.95%
4.3. Embedding Model Analysis
Refers to Agentless results
We rely on the golden solutions provided in SWE-bench as the ground truth for identifying the oracle files. While it is possible address the issue by editing other parts of the codebase, this ground truth offers a reliable base for compar- ing different approaches. We evaluate three approaches for this localization task: lexical-based (i.e., BM25), prompt- based, and sideeddding-beedd methods, as detailed in Sec- tion 3.2. The prompt-based results are taken from Agentless logs. For the embedding- files to 450 tokens with no overlap. This choice stems from our experiment with existing embedding models, where we tested token lengths ranging from 450 to 4096 and overlaps of O or half the context window, and saw that the mod- els perform better with smaller context windows (see Ap- pendix 1). This may be attributed to two factors: first, these models are often trained on short documents, and sec- ond, a smaller window allows the model to attend more to individual code chunks and focus on their semantics rather than normalizing the vector embedding across the entire file. Each chunk is further prefixed with the complete file path as additional metadata.
and prefer the original descriptions. This is likely due to differences in linguistic styles and vocabulary of these sum- maries compared to the natural queries the models were originally trained on. The decrease in the performance of BM25 with these summaries for SWE-bench Lite sug- gests a reduction in exact term matches and the frequency of keywords. By developing our code-specific embedding model, NV-EmbedCode achieves significant improvements. Not only does it outperform existing embedding models with original issue descriptions, but also achieves signifi- cantly better results with L.LM-generate summaries. These summaries enhance the issue descriptions by eliminating redundancy and removing irrelevant details, which can im- prove their effectiveness for retrieval. The improvement in NV-EmbedCode is due to the inclusion of diverse code and synthetic data in our training set and the use of instruction templates. which enable the model to recognize and adapt to different linguistic styles. Although the prompt-base based method can outperform embedding bbaedd ones when paired with existing embedding models, it falls short compared to our model.
approach, we chunk the
Table 2
presents a comparison of these methods on SWE- bench Lite and Verified sets. We evaluate Eodemddd against state-of-the-an models, including 3FR-Embeddiing- 2 R (Meng et al., 2024), NV-Embed-v2 (Lee et al., 2024), and NV-EmbedQA-v2 (Moreira et al., 2024a). The evalu- ation metric is recall@k, where k is the number of oracle files for a given instance. This metric measures the pro- portion of the oracle files correctly identified within the top results. The results indicate that existing embedding models struggle to perform well with the LLM summaries
4.4. Entity Localization Analysis
Accurate entity localization provides focused context, in- creasing the likelihood of an LLM generating correct patches. NEMOTRON-CORTEX employs localization agents to identify relevant entities. At the end of an agentic search trajectory, we combine its first iteration and last itera- tion responses to extract entities that balance the breadth and depth of localization. We refer to the first iteration results as Direct Prompting (DP), and the last iteration results as Localization Agent (LA). We compare our entity localiza- tion accuracy to Agentless (Xia et al., 2024), using results extracted from their logs, as their method includes a dedi- cated entity localization step. We use the golden solutions in
'The costs for more recent, nigher-performing submissions have not been reported. However, they are likely substantially higher as Openllands now allows more inference calls per issue and Agentless uses Claude 3.5 Sonnet, which is priced higher than GPT-4o.
7

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
4.5. Repair Generation
Table 3. Precision and recall metries in retrieving oracle entities for SWE-bench Lite and Veritied. DP denotes direct prompting entities and LA refers localization agent results.
As discussed in Section 3.4, changing the context and edit format leads to different instances being resolved. One ap- proach here is to identify the best-performing combination of context and edit format and then generate multiple solu- tions using temperature sampling. For example, Agentless generates 40 patches per instance by sampling 4 edit loca- tions and creating 10 patches in the search/ rep 1 lace format for each (1 using greedy sampling and using tem- perature sampling). They report that the number of resolved instances increases until 40 patches but plateaus beyond that point.
Lite
Verified
Method
Precision
Recall
Precision
Recall
Agentless
15.90%
59.94%
17,54%
58.37%
NEMOTRON-
35.83%
60.56%
16%
57.46%
CORTEXA (DP)
NEMOTRON-
38,91 %
71.17%
39.62% %
64.04%
CORTEXA (LA)
NEMOTRON-
35.84%
74.72%
34.51%
67.62%
CORTEXA (LA+DP)
Our observations indicate that we can achieve similar or better performance with fewer patches by increasing the diversity in how these patches are generated. Figure 4 lustrates the number of passed instances (out of 300 total) for SWE-bench Lite across different contexts, edit formats, and temperature settings, We consider three contexts here: 1) File, the top-1 I file retrieved by NV-EmbedCode, 2) LA, and 3) DP. With greedy sampling (temperature 0.0), the LA context, and the search/ rechhrreplace ace format, we can already achieve a high pass rate of 96 instances, highlighting the effectiveneess of our localization step. Adding a patch generated via temperature sampling increases the number of correctly resolved instances to 107 (pass @2). 2). Interestingly, while these two strategies have the highest pass rates indi- vidually, their combination does not introduce many new resolved instances compared to other alternatives. For exam- ple, changing the context to File and the prompt format to edit f. le results in 115 resolved instances overall, even though this new combination individually has the lowest pass rate. This pattern persists across other contexts and edit format combinations as well (Section 4.7), indicating that varying these factors increases solution diversity, ultimately leading to a higher total number of correct solutions while reducing the number of required inference calls.
Table 4. Localization agent's precision and recall metrics in re- trieving oracle entities for SWE--bench Lite and Verified set when instantiated with different models. Ensemble of all four agents yiclds substantial gains in the recall.
Lite
Verified
Model
Precision
Recall
Precision
Recall
Qwen2.5-72B
34.40%
40.33%
36.58%
41.71%
Llama3.3-70B
33.44%
40.40%
36.21% %
39.74%
Deepseek-v3
35.30%
45.94%
40.39 %
46.94%
GPT-44
34.25%
44.61%
36.67%
45.20%
Ensemble of Above
35.84%
74.72%
34.51%
67.62%
SWE-bench to extract ground truth entities (oracle entities).
Table 3 shows mean precision and recall results for Agent- less and NEMOTRON-CORTERR ponents. Overall, NEMOTRON COROOEA achieves higher precision and recall than Agentless on both SWE-bench Lite and Verified. Its localization stage also costs less: S0. 11 per instance for NEMOTRON-CORTES per instance for Agentless. The superior LA results demon- strate the advantage of combining specialized tools with multi-step reasoning. Adding DP results further boosts the recall by capturing relevant entities that might be overlooked during the agent's 's later, more focused iterations. Addition- ally, the dual approach of DP and LA also provides diverse high-quality contexts for repair generation and enables more efficient repair generations, which we discuss in the next section. MIMUUTRON-CRTEES employs an ensemble of four localization agents, each instantiated with a different LLM. Table compares the ensemble results with those of the individual agent. The recall increases by 28. 78%78% and 20.68% over the best individuals for the Lite and Ver- ified sets, respectively, thus retrieving significantly more relevant entities. The precision also increases for the Lite set because the ensemble benefit from models' strengths in solving different problem subsets. This result points to an alternative axis for inference-ting timeeeeeeteeeeeeee scaling by employing multiple models.
together with its two com-
compared to $0.15
While temperature sampling is less effective than chang- ing other factors, it remains useful strategy for gen- crating new patches. In our final patch generation pro- cess, we consider 4 contexts (File, LA, DP, and LA+DP, where LA and DP outputs are merged), two edit formats (sear enearch/repp ace and edit file), and two tempera- ture settings (0.0 for greedy sampling and 0.8 for temper- ature sampling). We first generate 4 solutions with greedy sampling by using the File and LA context, and both our edit formats. We then generate single additional solution by greedy sampling the LA+DP context and edit_file prompt. Finally, we generate 4 more solutions via temperature sampling, covering all four contexts, with two using the search/ search/replace replace format and two using the edit file format. These 9 solutions can achieve pass@1 performance (after applying the filtering steps as
8

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Table Precision and recall metries in retrieving oracle entities for SWE-bench Verified. The localization agent starts with the top-6 files retrieved by different embedding models.
Table 6. Pass@9 and Pass@1 results across different generation combinations for SWE-bench Verified with 500 instances. The primary diversification method is temperature sampling. We fur- ther evaluate the impact of introducing diversity by varying the edit format as described in Section 3.4 and using the four contexts mentioned in Section 4,5,
Embedding Model
Precision
Recall
NV-EmbedQA--2 (LA+DP)
27.62%
65.03%
NV-EmbedCodee (LA+DP) 34.51%
67.62%
Context
Edit Format
Pass@9
Pass @1
detailed in Appendix A.3) of 126 for the Lite set and 263 for the Verified set, surpassing Agentless results of 122 and 254 with 40 patches. This demonstrates how diversifying the solutions can lead to higher number of correct patches.
File, LA, DP, LA+DP
Both
307
263
LA
search/replace
270
237
LA
Both
284
245
DP
Both
261
220
LA+DP
Both
286
239
120
File
Both
257
223
16 Ruritt fin
110
DRVD Duddi_ Fie Rio 10. ecorctimppace
107
105
109
1100
entities as context and the search/ search/replace replace edit format-
UA-O ooit file
as it has shown to be the best single policy-while applying temperature sampling for diversity. We then extended this experiment by employing both edit formats while keeping the context fixed, and repeated the process across all four context types discussed in Section 4.5. Table 6 presents comparison of the pass@9 and pass 1 (after applying the filtering steps outlined in Section 3.5) metric for each combination. These results reinforce our claim that diversi- fying both the edit format and context increases the number of correct patches, as each configuration brings distinct strengths and weaknesses. By combining them, we are able to leverage their complementary advantages.
eassed Instamces
scurchroplace
64
70
60
Figure 4. #Passed instances of SWE-bench Lite under different contexts File, Localization Agent (LA), and Direct Prompting (DP) with varying edit format and temperature settings. Two- segment bars show pass @2 when both input solutions are used.
4.6. Ablation Localization
5. Conclusion & Future Work
In this section, we provide ablation studies on SWE-bench Verified for our two contributions to localization: 1) our fine- tuned VEMOTRON-CORTEXA embedding model; 2) the lo- calization agent for entity localization. First, Table demon- strates that as NV-EmbedCode achieves higher recall @k as shown Table 2, the localization agent produces more accu- rate entity localization results. This improvement indicates that starting with more accurate, relevant files enhances the agent's is success in identifying oracle entities.
CORTTRO demonstrates that improved local- ization and diverse solutions are effective avenues to en- hance LLM software agents for real-worle tasks. Our com- prehensive analyses validate three key contributions: 1) our code embedding delivers oftee-of-thh-ar file retrieval accuracies here; 2) our localization agent enables precise issue location identification; 3) our diverse solution genera- tion method, leveraging different contexts and edit formats, exhibits superior sample efficiency. MAMOORRN--CRTEER outperforms Agentless on SWE-bench at lower cost, show- casing its effectiveness.
Second, to study the impact of entity localization on final resolution, we compare using entity results (LA+DP) and the top-1 retrieved file (File) as context for patch generation. We generate 9 patches with both edit formats and temper- ature sampling and apply the filtering steps described in Section 3.5. As shown in Table 6, using entity localization results as contexts significantly improves the resolution rate, highlighting the importance of entity localization.
We recognize that NEMOTRON-CORTEDR be further improved by refining the selection process and/or scaling the number of generated patches. Currently, ma- jority voting the patch selection process is suboptimal, especially since we diversify the solutions generated, and correct solution may not appear multiple times. In fact, by using LLM--bsed voting instead of majority voting, gen- erating more accurate reproduction tests, and employing reasoning models during patch generation, NEMOTRON- CORTEXA'S 's resolution on the Verified set can reach 68. 68.4% while costing $3.28/issue. These optimizations are com- plementary to our approach, and a detailed analysis of the impact of each component is left for future work.
resolution can
4.7. Ablation on Solution Diversity
In Section 4.5, we demonstrated that diversifying the solu- tion generation strategies can increase the resolution rate. To further investigate it, we conducted an additional ex- periment in which, instead of the 9 diversified patches as described in Section 4.5, we generated patches using LA

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Acknowledgments
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023.
The authors would like to acknowledge the constructive feedback from anonymous reviewers and the work of ICML 2025 program chairs and area chairs.
DeepSeek-AL. DeepSeek- V2: A Strong, Economical, and Efficient Mixture-of- Language Model, 2024.
Impact Statement
DeepSeck-AL, Zhu, Q. Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., Zeng, W., Bi, X., Gu, Z., Xu, H., Dai, D., Dong, K., Zhang, L., Piao, Y., Gou, Z., Xie, Z., Hao, Z., Wang, B. Song, J., Chen, D., Xie, X., Guan, K., You, Y., Liu, A., Du, Q. Gao, W., Lu, X., Chen, Q., Wang, Y., Deng, C., Li, J., Zhao, C., Ruan, C., Luo, F., and Liang, W. DeepSeek- Coderr--- Breaking the Barrier of Closed-Source Models in Code Intelligence, 2024, URL https: /tapss//arxiv... arxiv.
Reliable and capable LLM software agents have the po- tential to automate routine programming tasks, such as de- bugging, and boost developer efficiency by freeing them to work on more innovative projects and explore more ef- ficient solutions. By providing natural language interfaces to programmming tasks, these agents can also enable people without prior programming experience to use code in their work. This potential can obviously be misused to do harm by malicious actors, for example, to generate malware or automate cyber attacks. Thus it is important to build robust safety mechanisms in LLM agents minimize their nega- tive impacts. By carefully considering these factors, we can work to maximize the benefits of LLM software agents.
org/ abs www.ass//0 2406 11931.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407221783, 2024.
References
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W. Large language models for mathematical reasoning: Pro- gresses and challenges. arXiv preprint arXiv:2402.00157, 2024.
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M. Shou, L., Qin, B., Liu, T., Jiang, D., al. Code- bert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
Aide, S. K. P. SOTA swebench-veriffed: (re)learning the bitter lesson, 2024. URL httpps:/aaie dev/ blog/sota batt//ttettbttrrreeee lesson. Accessed: 2025-01- 25.
Gao, T., Yao, X., and Chen, D. Simese: Simple con- trastive learning of sentence embeddings. arXiv preprint D.Xiv:2104.0882) 2021.
Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., and Wang, W. SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement. arXiv preprint arXiv:2410.20285 2024.
Kömez-Rodríguez C. and Williams, P. A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14504- 14528. Association for Computational Linguistics, 2023.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, Fan, Y., Ge, W., Han, Y., Huang, F, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., and Yin, J. Unixcoder: Unified moss--odall pre- for code representation. arXiv preprint arXiv:2203.0385. 2022.
Brown, B., Juravsky. J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787 2024.
Guo, D., Zhu, Q. Yang. D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., al. DeepSeek- Coder: When the Large Language Model Meets Programming The Rise of Code Intelligence. arXiv preprint arXiv:2401.14/1 14196, 2024.
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., and Chen, W. CodeT: Code Generation with Gener- ated Tests, 2022. URL https: http///axvvvvr///eeeeee org/ abs/ 2207. 10397.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. O.D Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv: 107. 03374444 2021.
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E.,, Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. Measuring Coding Challenge Competence With APPS. NeurIPS, 2021.
10

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 106509 2021.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Huang, J., Tang, D., Shou, L., Gong, M., Xu, K., Jiang, D., Zhou, M., and Duan, N. Cosqa: 20,000+ web queries for code search and question answering. arXiv preprint 1rXiv:2105.13239, 2021.
Liu, Y., Gao, P., Wang, X., Liu, J., Shi, Y., Zhang, Z., and Peng, C. MarsCode Agent: Al-native Automated Bug Fixing. arXiv preprint arXiv:2409. 00899, 2024b.
Liu, Y., Meng, R., Jot, S., Savarese, S., Xiong, C., Zhou, Y., and Yavuz, S. CodeXEmbed: A Generalist Embed- ding Model Family for Multiligual and Multi-task Code Retrieval. arXiv preprint 2rXiv:2411. 2024c.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai ol system card. arXiv preprint arXiv:2412.16770 2024.
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poiriee J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402. 19173, 2024.
Jimenez, C. E.,, Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world GitHub issues? arXiv preprint arXiv:23 10.067300 2023.
Luo, Z., Xu, C., Zhao, P., Sun, Q.. Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: openreview.eeeeeeeee net/ forum? Man-ddddUnnUUISH
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1ee..1112.66. 226900, 2014.
Kocetkov, D., Li, R., Allal, I.. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., Bahdanau, D., von Werra, L., and de Vries, H. The Stack: TB of permissively licensed source code, 2022. URL https: /arxivv.vvv./////////aaa///aavva org/ abs/ / 22112111112 15533.
Meng, R., Liu, Y., Joty, S. R., Xiong, C., Zhou, Y., and Yavuz, S. SFR-Embedding--: Advanced Text Embedding with Multi-stage Training, 2024. URL htttt:://huugingge hta.fface.co/allsforreeeeeeee Salesforce/ SFR- FER-Embedding-2_F
Lawrance, J., Bogart, C., Burnett, M., Bellamy, R., Rector, K., and Fleming, S. D. How programmers debug, revis- ited: An information foraging theory perspective. JEEE Transactions on Software Engineering, 39(2):197-215, 2010.
co/
Meyer, Y., Emadi, M., Nathawani, D., Ramaswamy, L., Boyd, K., Van Segbroeck, M., Grossman, M., Mlocek, P., and Newberry, D. Senthetic-Text-To--eee dataset for training language models to generate SQL queries from natural language prompts, 2024.
A synthetic
Le, H., Wang. Y., Gotmare, A. D., Savarese, S., and Hoi, S. H. CodeRL: Mastering Code Generation through Pre-
trained Models and Deep Reinforcement Learning, 2022.
URL https: // ////axivv... wttp:/o////bb///////// 01220000017000 780.
Moreira, G. d. S. P., Ak, R., Schifferer, B., Xu, M., Os- mulski, R., and Oldridge, E. Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine- tuning and deploying Rerankers for RAG. arXiv preprint 20f1::2409006691, 2024a.
Lec, C., Roy, R., Xu, M., Raiman, J., Shocybi, M., Catan- zaro, B., and Ping, W. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv preprint or 2405: 2024.
Moreira, G. d. S. P., Osmulski, R., Xu, M., Ak, R., Schif- ferer. B., and Oldridge, E. NV-Retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.1583) 2024b.
Lei, B., Li, Y., and Chen, Q. AutoCoder: Enhancing Code Large Language Model with AIEV-INSTRUCT, 2024. URL https:t///// wwwiaaivivv.... wrw. http///s/2244... 14900
14906.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. Starcoder: may the source be with you! arXiv preprint 1611 2023.
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang. X., von Werra, L., and Longpre, S. OctoPack: Instruction Tuning Code Large Language Models. arXiv preprint arXiv:2308.07122 2023.
Li, X., Dong, K., Lee, Y. Q. Xia, W., Yin, Y., Zhang, H., Liu, Y., Wang, Y., and Tang, R. Coir: A comprehensive benchmark for code information retrieval models. arXiv preprint arXiv:2407.0288 2024.
Muennighoff, N., Su, H., Wang, L., Yang, N., Wei, F., Yu, T., Singh, A., and Kiela, D. Generative representational in- struction tuning. arXiv preprint arXiv:2402 009906, 2024.

---

Code LLMs Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Neelakantan, A., Xu, T., Puri, R.,, Radford, A., Han, J. M., Tworek, J., Yuan, Q.. Tezak, N., Kim, J. W., Hallacy, C., et al. Text and code embeddings by contrastive pre- training, arXiv preprint arXi::2201. 2022.
Wei, Y., Wang, Z.,, Liu, J., Ding, Y., and Zhang, L. Magi- coder: Source Code Is All You Need. arXiv preprint arXiv:2312.02120, 2023.
Wu, Y., Huang, D., Shi, W., Wang, W., Gao, L., Liu, S., Nan, Z., Yuan, K., Zhang, R., Zhang, X., Du, Z., Guo, Q., Pu, Y., Yin, D., Hu, X., and Chen, Y. InverseCoder: Unleashing the Power of Instruction- with Instruct, 2024. URL org/ /b///b//// 2407 05700.
OpenAl. GPT-4o, 2024. URL https: //tptt:///..a..mmmmmm
openai cwm/ docs 2025-01-27.
Accessed:
Overflow, S. Stacksample: 10% of stack overflow q&a, 2021. URL
http://wwwwgg........ com/ datasets/ Satasetssstackons http///cckooerlloo//takaaaaaaaae cessed: 2025-01-26.
Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. Agentless: De- mystifying Ilm-based software engineering agents. arXiv preprint arXiv:2407.0/ 40440....44,,, 2024.
Ac-
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv preprint ar arXiv:2308. /2950, 2023.
Xie, C., Li, B., Gao, C., Du, H., Lam, W., Zou, D., and Chen, K. SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution. arXiv preprint arXiv:2501.05000 2025.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Yan, W., Tian, Y., Li. Y., Chen, Q., and Wang. W. Code- transocean: A comprehensive multilingual benchmark for code translation. arXiv preprint trXiv:2310.0995. 2023.
Team, M. T. URL
Moatless tools, November 2024a. https://githule http:///tttu....oo.aaooooaa com/acrwall/ moatless-tools
Yang, J., Jimenez, E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engineer- ing, arXiv preprint arXiv:2405.15799 2024.
Team, Q. Qwen2.5: A Party of Foundation Models, Septem- ber 2024b. URL https:://// qwenlm.eemmmmmmmmmmm...mmmmmmm.m hoannlm...iiuut github. / blog/qwen2.5/.
Yu, Z., Zhang, X., Shang, N., Huang, Y., Xu, C., Zhao, Y., Hu, W., and Yin. Q. WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning, 2024. https::::// /arxiv.
Voyage AI. Voyage-code-3: more accurate code retrieval with lower dimensional, quantized embeddings, Decem- ber 4 2024. URL / voyageai... com/ 2024 / 12 04
org/ / abs 22312.112222 14187.
Yuan, A., Coenen, A., Reif, E., and Ippolito, D. Wordcraft: story writing with large language models. In Proceedings of the 27th International Conference Intelligent User Interfaces, pp. 841-852, 2022.
Wang. L., Yang, N., Huang, X., Yang, L.,, Majumder, R., and Wei, Improving text embeddings with large language models. arXiv preprint arXiv:2401.00366 2023.
Zhang, K., Yao, W., Liu, Z., Feng, Y., Liu, Z., Murthy, R., Lan, T., Li, L.,, Lou, R., Xu, J., et al. Diversity empowers intelligence: Integrating expertise of software engineering agents. arXiv preprint M.Xiv:2408.077 2024a.
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh. J., et al. Openhands: An open platform for AI software developers as generalist agents. arXiv preprint vrXiv:2407.16741, 2024a.
Zhang, Y., Ruan, H., Fan, Z., and Roychoudhury, A. Au- tocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1592- 1604, 2024b.
Wang, Z. Z., Asai, A., Yu, X. V., Xu, F. F., Xie, Y., Neubig, G., and Fried, D. GodeRAG-Benchi Can Re- trieval Augment Code Generation? arXiv preprint arXiv:2406.1 14497, 2024b.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q V., Zhou, D., et al. Chain-of-thougle prompting elicits reasoning in large language models. Advances in neural information processing systems, 1122824-248811 2022.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeinterppeter:: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024.
12

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
A. Appendix.
A.I. No-embedCode Training Details
We employ three types of instructions for the various retrieval datasets used in training NV-EmbedCode:
1. Bug-to-File:
Applied to the dataset derived from the SWE-bench training set, which maps issue descriptions to their respective "oracle" files files requiring edits to resolve the issue. These oracle files are identified based on the pull request (PR) that resolved the issue successfully.
2. BugSummary-to-Finn Also based on the SWE-bench training set, except that here the original issue descriptions are replaced with summaries generated by an LLM - Llama-3.1 405B.
3. General: Used for all other retrieval datasets, which include a diverse set of tasks, including text-to-code code-to-text, code-to-code, and hybrid retrieval.
Table 7 provides an overview of these instruction templates along with the number of samples for each. Each training sample consists of one query-post query-positive document pair and 7 hard negative documents.
Table 7. Instruction template and number of samples used for each category of NVV-EmeedCode training dataset. Instruction Template
Dataset Type
Number of Samples
Bug-to-File
Given bug description, retrieve codes that need to be edited to resolve it Given summary of bug description generated by an I.I.M, retrieve codes that need to he edited to resolve it
177k
BugSummary-to- File
92k
General
Retrieve code or text based on user query
265k
The SWE-bench training set consists of 19,008 issues extracted from pull requests across 35 non-test repositories. We filter out instances in which the corresponding golden solution creates new file or modifies non-Python files, resulting in a curated subset of 13,922 issues. For each remaining instance, the oracle files are designated as positive documents. As described in Section 3.2, all documents are segmented into chunks; in this setting, we use a chunk size of 512 tokens. We also include a portion of the ColR training set (Lii et al., 2024), which includes data from APPS (Hendrycks et al., 2021), CoSQA (Huang al., 2021), CodeTranSOcean (Yan al., 2023), StackOverflow (Overflow, 2021), and SyntheticTex12SQL. (Meyer et al., 2024). Additionally, we synthetically generate dataset of code produced from coding queries in CodeFeedback (Zheng et al., 2024) by prompting DeepSeek-v2.5 (DeepSeek-Al, 2024). Table 8 summarizes the number of samples for each dataset.
Table Number of samples for cach training datasct of NV-EmbedCode.
Dataset Name
Main Retrieval Task Dataset Type
Number of Samples
SWE-bench Train Set (Original issue)
Issue-to-Orace File
Bug-to-File
177k
SWE-bench Train Set (LLM summary issue) APPS (Hendrycks et al., 2021)
Issue-to-Oracle-File
BugSummary-to-FFit
92k
Text-to-Code
General
5k
CoSQA (Huang et al., 2021)
Text-to-Code
General
20k
SyntheticTex122QL. (Meyer et al., 2024)
Text-to-Code
General
100k
CodeTransOccan (Yan et al., 2023)
Code-to-Code
General
1k
StackOverflow@A (Overflow, 2021)
Hybrid Code
General
14k
CodeFeedback (Zheng et al.. 2024)
Hybrid Code
General
125k
DeepSeek-v2 v2.5 (DeepSeek-AL. 2024)
As described in Section 4.3, we evaluated range of chunk sizes from 450 to 4096 tokens using the Febe--mddddiinMiii model determine the optimal chunking of code files. Our experiments on the SWE-bench Lite set revealed that a chunk size of 450 tokens yields the highest retrieval accuracy, as illustrated in Figure 5. Based on these findings, we adopted chunk size of 512 tokens for training and 450 tokens for inference with our model. Consistent with the behavior observed in the SFR- FeRe-Emdedding-W Mistral model, NV-EmbedCode also exhibits a decline in retrieval accuracy as chunk size increases, as shown in Figure 5. This can be attributed to two factors: first, a more localized view of the files enables the model to more effectively determine if the code snippet is related to the issue; second, larger chunk sizes will be disttiibbii,,,,,,, as both our current training set and the training data used for the base model consist primarily of shorter documents.
13

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Embedding
70.00
NV EmbedCodie Lite
NV-EmbedCode Verified
Recallak (%)
55.00
50.00
45.00
40.00
35.00
150
1k
2k
4k
Chunk size (#tokens)
Figure 5. @ecall@k @ in file retrieval for SWE-bench when changing the maximum number of tokens per each chunk. k is the number of oracle files based on golden solution that resolved the issue.
Table NEMOTRON REMOTRON-CORTEX. CORTEXA file retrieval accuracy compared to free-form agentic retrieval baselines for SWE-bench Verified. As baselines retrieve arbitrary number of files, report recall @ co. For NEMOTRON-CORTEXA, we use recall @ 10, since other methods retrieved least 15 files on average.
Method
Recall @o
OpenHands (Wang et al., 2024a)
85.60%
AutoCodeRover--2 (Zhang al., 2024b)
88.20%
MarsCode (Liu al., 2024b)
87.20%
SWE-agent-vl (Yang et al., 2024)
76.80%
NEMOTRON-CORTEM.
94.00 % (recall @ 10)
A.2. Additional Results for Localization
Compare with agentic file retrieval methods. We compare file retrieval accuracy of NV-EmbedCode against four representative free-form agentic approaches: OpenHands (Wang et al., 2024a), AutoCodeRover-v2 (Zhang et al., 2024b), MarsCode Agent (Liu et al., 2024b), and SWE-agent-vl (Yang et al., 2024). For each agent, we extract files from their official trajectory logs in the SWE--bench leaderboard. Since these methods can retrieve an arbitrary number of files, we denote the accuracy metric with recall@oo.. oc. In this comparison, we use recall@ 10 for SEMOORRO--ORTEE. as other baseline methods retrieved at least 15 files on average. Table shows that our code embedding model, NV-EmbedCode, enables NEMOTRON- CORREOX to achieve significantly higher file recall accuracy compared agentic baselines, demonstrating the effectiveness of finetuning for the specialized task of file retrieval from issue descriptions.
Entity localization with different ensemble methods. Section 4.4, we discussed how NEMOTRON-CORTEE utilizes an ensemble of four localization agents, each instantiated with a different model. Here, we explore two alternative ensemble strategies. The first uses temperature sampling with a single model to generate diverse outputs. The second aggregates entity results across all steps from single agent. We use DeepSeek-v3 as it is individually the strongest in entity localization. For the temperature sampling ensemble, run entity localization four times; with temperature 0.8. When aggregating all steps, we use greedy sampling (temperature 0). Table 10 demonstrates that the model- based ensemble achieves higher entity recall than both alternatives, while the temperature sampling baseline obtains a higher precision by retrieving fewer entities. These results highlight that model diversity helps improve the recall accuracy of entity localization the most among the three ensemble methods.
with temperature 0 and three times
Entity Localization Accuracy and Resolution Status. To further assess the impact of localization on the final resolution rate, we divide the instances into two sets, resolved and unresolved, based on whether any of the nine patches successfully resolve the issue We then calculate the entity localization accuracy for each set. As shown in Table 11, we observe that resolved cases exhibit consistently higher precision and recall in localization compared to unresolved ones, indicating a strong correlation between localization accuracy and successful patch generation.
14

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Table 10. NEMOTRON-CORTEEXA3 entity ensemble achieves higher recall accuracy compared other ensemble baselines.
Method
Precision
Recall
Temperature Sampling
49.93% %
57.03%
Aggregating Results of All Agent Iterations
35.47%
53,47%
Ensemble of Models (LA DP)
34.51%
67.62 %
Table 11. Average entity precision and recall accuracy as broken down by resolution results. Resolved instances have higher accuracies.
Lite
Verified
Result
Precision
Recall
Precision
Recall
Resolved
44.54%
86.87%
46.05%
90.91%
Unresolved
27.58%
63.20%
32.75%
64.06%
A.3. Solution Filtering
After generating patches for each instance as described in Section 4.3, we have the correct patch for 146 instances of SWE-bench Lite and 307 instances of SWE-bench Verified. However, the PASS.to.PASS and FAIL.to PASS tests provided by SWE-bench cannot be used identify the correct patches. Table 12 summarizes the average number of patches remaining after each filtering step described in Section 3.5, as well the number of resolved instances achieved by applying majority voting at each step (pass@) 1). The results demonstrate the effectiveness of these filtering steps but also indicate areas for improvement, which we leave for future work. Specifically, the regression and reproduction tests, generated using an LLM,, may have inaccuracies that could result in removing correct solutions or failing to prune the wrong solutions. Furthermore, majority voting is not an ideal heuristic here, as certain incorrect solutions may repeat multiple times, whereas some correct solutions may only be generated once or fewer times than the incorrect ones. Additionally, in cases of ties between solutions based on their frequency, choose one of them randomly. A more robust scoring mechanism can help mitigate these limitations.
Table 12. The impact of filtering techniques on the average number of remaining patches and the number of resolved instances when majority voting is applied after filtering (pass@11 1).
Lite
Verified
Filtering
avg.
% Resolved
avg.
% Resolved
Syntax check
7.7
7.9
Normalization
5.7 107 (35.67%)
5.5 225 (45.00%)
Regression test
4.6 116 (38.67%)
4.5 242 (48.40%)
Reproduction test
4.0 126 (42.00%)
4.0 263 (52.60%)
A.4. Cost Analysis
For measuring costs, we use the following prices for the models we employ:
Claude 3.5 Sonnet API: $3/M input tokens and $15/M output tokens.
GPT-4o API: S2M5/M input tokens and $10/M output tokens.
DeepSeek-V33 V3: $0.9/M input tokens and $1.1/M output tokens.
Llama 3.3-70B: $0.59/M input tokens and 80.73/M output tokens.
Qwen2.5-72B: $0.4/M input tokens and $0.75/M output tokens.
NV-EmbedCode: As this a Mistral 7B-based model, we use Mistral pricing of $0.11/M input tokens. We cache the chunks and query our vector database for repeated ones.
Table 13 summarizes breakdown of the cost for each step of our pipeline. SWE-bench Lite and Verified have total of 800 instances, but 93 instances are common between them. We consider the common ones only once in the cost calculation.
15

---

Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Table 13. Breakdown of the for running NEMOTRON-CORTEXA the 707 unique instances of SWE-bench Lite and Verified,
considering the 93 instances common between them only once.
#Input Tokens
#Output Tokens
Total Cost
Avg. Cost / Instance
Stage
Model
M
USD
USD
File Localization
NV-EmbedCode
217.95
23.97 (6.69%)
(6.69%)
Entity Localization
Mix
42.66
1.74
53.68 (15.00%)
0.08 (15.00%)
Generating Tests
GPT-4o
13.00
5.74
89.90 (25.11%)
0.13 (25. 11%))
Patch Generation
Baude-3.5-Somme
47.63
3.17
190.44 (53.20%)
0.27 (53 533
Total
321.24
10.65
358.00 (100%)
0.51 (100%)
16


Reviews:
[{"review_id": "PDuinDMrF4", "reviewer": "Reviewer_WZSG", "summary": "The work proposes an agentic system around LLM to solve GitHub issues (Swe-bench tasks).\nThey mainly proposed a code embedding model used for file retrieval and built a localization, diverse patch generation and filtering mechanism around it. \nFinally, they demonstrated good performance of Cortexa while being cost-effective.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback. We appreciate your thoughtful comments.\n\n**Comparison to SOTA:** In response to your comment on Cortexa's performance compared to other works, we would like to emphasize that our primary goal was to identify ways to increase the efficiency of coding agents. We observed that while free-form agentic flows are effective here, their vast pool of actions often leads to numerous costly iterations. Similar to Agentless, we aim to develop a more structured approach by identifying and improving key stages crucial for the final task, ultimately increasing efficiency and performance simultaneously. This approach also simplifies debugging and human intervention.\n\nFor instance, while OpenHands achieves a 26% resolve rate on SWE-bench Lite at `$1.1/instance` (with the cost for other submissions unreported), Cortexa achieves a 42% resolve rate at `$0.51/instance`. We recognize that Cortexa's performance can be further improved by refining the selection process and/or scaling the number of generated patches. These optimizations are orthogonal to our approach and are left as a future work. Currently, the majority voting in the patch selection process is suboptimal, especially since we diversify the solutions generated, and a correct solution may not appear multiple times. Indeed, we observe that Cortexa’s performance can be increased by adopting a better patch selection approach - we’ve switched from a majority vote to using an LLM for selection, and pass@1 on the Lite set increases to 42.67% and on the Verified set to 54.8%. Additionally, another approach that brings our results in line and beyond most open approaches is scaling the number of generated patches. For instance, increasing it to 32 (16 by Claude-3.5-Sonnet and 16 by DeepSeek-V3) increases the performance on the Verified set to 58.8%\n\n**Confidence interval:** We used a temperature setting of 0 for all LLM calls, except for 4 out of 9 generated patches during the repair stage. Each of these 4 patches utilized different combinations of context and edit formats. To address the concern about stochasticity of LLM calls, we generated two additional patches for each combination (temperature sampling), resulting in 81 variations of the final 9 patches. The table below shows the peak performance statistics of these 9 patches on the Verified set. As mentioned in Appendix A.1, the patches used in the paper achieved a peak performance of 307 instances (61.4%) which falls within the confidence interval. We will include these results in the paper.\n\n| Target                          | Min  | Max  | Mean   | Std  |\n|---------------------------------|------|------|--------|------|\n| Peak perf (pass@9)     | 299  | 311  | 304.52 | 2.71 |\n| #valid patches                  | 7.75 | 7.88 | 7.79   | 0.02 |\n\n**Ensemble increases precision:**\nIndeed, this is a surprising effect we observe. In Table 4 we reported precision/recall averaged across instances. To illustrate why ensembling is helpful consider the following example retrieval task over 5 documents (A, B, C, D, E) and two queries and relevant documents {Q1, (A,B)} and {Q2, (C,D)}. Say method 1 returns (A, C) for Q1, and (A, B) for Q2; method 2 returns (C, D) for Q1, and (B, C) for Q2. Method 1’s mean precision is (1/2 + 0/2) / 2 = 1/4. Method 2’s mean precision is (0/2 + 1/2) / 2 = 1/4. The union for both methods results in (A, C, D) for Q1 and (A, B, C) for Q2. The ensemble mean precision is (1/3 + 1/3) / 2 = 1/3 greater than either methods’ precision (1/4). For each question instance, the union ensemble has a lower precision (1/3) than the highest individual precision (1/2). This phenomenon explains why our ensemble method benefits from models’ strengths in solving different problem subsets.", "claims_and_evidence": "yeah, the claims are clear. The evidence presented is somewhat okay but can be more convincing by reporting confidence intervals.", "methods_and_evaluation_criteria": "The method and evaluation criteria (resolve rate, precision and recall) seem reasonable.\nHowever they reported one number but the LLM's are quite stochastic and ideally should be run multiple times to report a confidence interval.", "theoretical_claims": "Experimental paper --> no theoretical claims.", "experimental_designs_or_analyses": "Yes.\nI think experiment design (or dataset) looks alright for this work. They evaluated on SWEbench which consists of real world github issues.", "relation_to_broader_scientific_literature": "There is already some work in code agentic systems focusing on code retrieval, AST graphs, patch selection, etc. \nThis work shows the benefits of a fine-tuned retrieval model used in loop with coding agents to solve the github issues.\nAlso, there are works which achieve more than 60% in pass rate and the work doesn't connect or comment on them.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n- Paper is clearly written and easy to understand\n- Novel code embedding model and its utilization to resolve the GitHub issue by using it within agentic systems\n\nWeakness:\n- Performance is weaker as compared to other SOTA methods\n- Doesn't report confidence interval, so currently uncertain if we can use the results to derive conclusions", "other_comments_or_suggestions": "38: Identify what?\n\nWhy does localization via ensemble increase precision? It should reduce precision due to the stochasticity of different LLMs combined together.\n\nCan you comment on why the performance of open hands is better than cortexa?", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kfp2zB14Lo", "reviewer": "Reviewer_KBKm", "summary": "- This paper introduces CORTEXA, a software agent that involves training a model specifically for localizing the right files, building a localization agent to identify the right entities within a file, and a workflow for diverse patch generation and selection.\n- CORTEXA outperforms Agentless and achieves similar performance to OpenHands on SWE-Bench. The paper also individually analyzes each component of the agent.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work and for offering constructive feedback to help enhance our paper. Regarding the comparison to the SOTA works, please refer to our response to Reviewer WZSG. Additionally, our sample efficiency is demonstrated by achieving higher resolution rates than Agentless with only 9 patches, compared to Agentless's 40 patches.\n\n**Importance of different components:** One of the main messages of our work is that improved localization of the issue is key to increasing resolution. Our initial experiments demonstrated that when GPT-4-turbo was given the ground truth file plus additional relevant files to fill the context window, it solved only 4 Lite set instances with 1 greedy patch. Providing only the oracle file increased this to 53 instances, and narrowing to ground truth functions/classes reached 72 instances. The dramatic increase in instance resolution from narrowing the context suggests that models are severely impacted by superfluous context and can benefit from a more localized edit location. This observation is the reason why we focused our work on better localization.\n\nTo verify if this phenomenon holds in our final end-to-end results, we ran new experiments by generating 9 patches with the results of the file localization vs the entity localization, as shown in response to Reviewer sboj. Using the entities retrieved by the localization agent (LA) as context, we get to pass@1 of 245, while the pass@1 with File retrieval results is 223. This is consistent with our early observations.\n\nOur second key contribution concerns generating diverse candidate patches. We induce diversity by using the variety of different contexts obtained throughout the localization process, in addition to requesting different edit formats as outputs from the language models. Using the standard temperature sampling with the LA results and search/replace edit format, we obtain a pass@9 of 270 and pass@1 of 237 after the filtering steps mentioned in Section 3.5. Using our approach to maximize patch diversity we increase the pass@9 to 307 and the pass@1 to 263. A further advantage of this patch diversity method is that it is straightforward to adopt and makes more effective use of the existing sampling budget by leveraging the strengths of different contexts. We believe this technique has been overlooked by previous works and could easily be adopted independent of Cortexa.\n\nAs demonstrated in Table 2, our embedding model significantly influences file localization. To further evaluate its effect on entity localization, we performed new experiments on the Verified set by running the entity localization step with file retrieval results from the base model. The localization agent takes a list of files from the file localization step as the starting point and navigates the repository as defined in Section 3.3. Table below indicates that starting with more accurate relevant files enhances the agent’s success in identifying oracle entities.\n| Embedding Model  | Entity Precision | Entity Recall |\n|---|--|--|\n| NV-Embed-QA    | 27.62%          | 65.03%       |\n| Cortexa        | 36.22%          | 68.09%       |\n\n**Localization model:** We chose NV-Embed-QA as our pretrained model as it is the strongest commercially-available text embedding model in our tests. Its commercial availability aligns with our goal of releasing Cortexa and its associated retrieval model. Our dataset includes 290k query-positive pairs from 19k issues across 35 non-test repositories with positive passages from oracle files in the golden patch. Additionally, we generated 125k coding question-answer pairs using DeepSeek-v2.5. We also included data from APPS, CoSQA, Text2SQL, CodeTransOcean, and StackoverflowQA. We will add a detailed table of all data to the paper. While we acknowledge the potential of the embedding model for broader coding tasks, its purpose here was to help with retrieving relevant files to an issue description. A more comprehensive analysis of the embedding model on diverse datasets falls out of the scope of this paper and will be addressed in a future work.\n\n**Claude for Table 2:** We agree that several strong LLMs exist that may surpass GPT-4o here such as Claude, however we use Agentless as our baseline and thus wanted to facilitate comparisons by keeping the same LLM. Another important motivation in our choice of embedding models over prompt-based approaches is because many LLMs including GPT-4o and Claude have been trained on these exact repositories, allowing them to identify relevant files by name alone. By contrast, our embedding model does not suffer from this same test leakage and thus ensures that the gains cannot be attributed to memorization from a specific LLM. Furthermore, reading the full contents of the files with these LLMs is prohibitively costly, as shown in CodeMonkeys (Ehlrich et al. 2025), whereas our embedding model achieves equal or better accuracy using only a fraction of the cost.", "claims_and_evidence": "The paper's main claims are: \n- We develop a code embedding model specialized in retrieving relevant code chunks to a given bug, achieving state-of-the-art file retrieval accuracies on SWE-bench: this is supported by Table 2, however additional baselines could be considered. \n- We propose a diverse solution generation method by leveraging different contextual information and varied prompt formats, significantly enhancing sample efficiency: I was unsure where the sample efficiency results could be found.\n- We design a localization agent that integrates advanced programming tools and leverages an ensemble of LLMs to deliver more precise and granular issue localization. Experimental results demonstrate that CORTEXA outperforms Agentless [...] while being more cost-effective: CORTEXA does outperform Agentless, but this result does not mention other baselines like OpenHands.", "methods_and_evaluation_criteria": "- Details on how the localization and repair stages interact with each other are unclear.\n- Why did the authors select this particular model (NV-Embed-QA)? Additional justification would be helpful.\n- The details about the localization models are somewhat short and not expanded on in the Appendix. For example, the authors mentions \"The final dataset contains approximately 534k pairs of query-positive documents\" but do not provide details about the specific distribution.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Evaluation is only limited to SWE-Bench. In contrast, Wang et al. 2024a evaluates on 15 benchmarks. I raise this point particularly because the performance on SWE-Bench alone is not that different from prior work, so a comparison on additional benchmarks could help distinguish the performance of CORTEXA. An additional point is that because the localization model is trained on SWE-Bench data, it would be interesting to evaluate on more diverse datasets.\n- Additional experiments ablating the performance of the 3 components identified as key contributions would be helpful. Which ones affect CORTEXA performance the most? While I appreciate the individual evaluation of each component, I believe the broader community would be more interested in understanding which component to focus on when building new software agents.\n- In Table 2, why do you only consider GPT-4o for Prompt? Claude would be a much stronger baseline to compare against.", "relation_to_broader_scientific_literature": "The various components of CORTEXA aim to address various pain points that have been discussed in the broader literature regarding software agents. Section 4.3-4.5 provide interesting insight into how to design these different agent components.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Could you please address the aforementioned questions about CORTEXA methodology and evaluation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xEZY0ycp0n", "reviewer": "Reviewer_sboJ", "summary": "This paper proposes CORTEXA, a coding agent for bug-fixing based on Agentless v1.5. CORTEXA differs from Agentless in two key ways: 1) it uses an embedding-based file-retrieval (using a finetuned embedding model) combined with an agentic entity retrieval approach instead of direct prompting for localization, 2) it increases patch diversity by combining different temperatures, contexts, and prompt formats when sampling repair patches. Combined this yields slightly better performance at (probably) lower cost.\n\n### Update After Rebuttal\nI thank the authors for their detailed response and would like to encourage them to include the additional ablation results in a revised version for the paper. These additional results convince me of the effectiveness of the proposed approach beyond Agentless and agentic retrieval approaches. I have updated my score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the time and effort you have put into reviewing our work and are grateful for your insightful comments that help improve our paper. For comparisons to SOTA works, please refer to our response to Reviewer WZSG. We would like to highlight that Cortexa outperforms Agentless in several key areas:\n1. Patch Generation Efficiency: Our reported performance is with only 9 patches, whereas Agentless requires 40 patches. When generating a comparable number of patches as mentioned in response to Reviewer WZSG (32 patches), Cortexa significantly outperforms Agentless, achieving 58.8% on the Verified set (p-value = 0.006).\n2. Localization Performance: While Agentless uses a combination of prompt-based and embedding-based approaches, our specialized embedding model achieves higher retrieval accuracies. From Table 2 in the Agentless paper, its localization cost is `$0.15/instance`, while Cortexa’s localization cost is `$0.11/instance` (including file and entity localization) as shown in Appendix A.2.\n3. Recall in Oracle Entity Identification in Table 3: Our recall rate in identifying the oracle entity is statistically superior to Agentless, with a p-value of 0.0007.\n\nWe will release our code including all the prompts we used, full logs, and our embedding model upon the paper’s publication.\n\n**Additional baselines on retrieval:** Here are the new baseline results on the Verified set:\n\n**1. Temperature sampling and ensembling across multiple stages instead of ensembling multiple models:** We use DeepSeek-v3 here as it is individually the strongest in entity localization. Neither method can match Cortexa’s recall. The temperature sampling approach obtains a higher precision since it retrieves fewer entities. However, recall matters more for patch generation to provide contextual information. The results show that model diversity is the most effective approach here.\n|Model|Precision|Recall|\n|-|-|-|\n|Temp Sampling (deepseek-v3)|49.93%|57.03%|\n|All stages (deepseek-v3)|35.47%|53.47%|\n|Cortexa (DP+LA)|36.22%|68.09%|\n\n**2. Agentic retrieval baselines:** We select 4 agentic approaches here: OpenHands, AutoCodeRover-v2, MarsCode Agent, and SWE-agent-v1. For each agent we extract files from their official trajectory logs in the SWE-bench leaderboard. Since these methods can retrieve an arbitrary number of files, we denote the accuracy metric with recall@inf. In this comparison, we use recall@10 for Cortexa as other baseline methods retrieved at least 15 files on average.\n\nFor entity retrieval accuracy, these agentic approaches often do not explicitly retrieve entities. Moreover, there is no exact way to measure entity retrieval results from their logs as there is no fixed pattern. Thanks to its localization agent step, Cortexa provides a concrete way to measure the more granular entity retrieval accuracy, which proved helpful to iterate and improve the agent.\n|Model|Recall@inf|\n|-|-|\n|OpenHands|85.6%|\n|AutoCodeRover-v2|88.2%|\n|MarsCode|87.2%|\n|SWE-agent-v1|76.8%|\n|Cortexa|94.0% (recall@10)|\n\n**Impact of diversifying generations:** To address this comment, we run the following experiments on the Verified set: \n1. Taking the context (localization agent, LA) and the edit format (search/replace) that give us the highest pass rate with temperature 0 and generating 9 patches for it (1 greedy sampling and 8 temperature sampling).\n2. Changing the edit format to use both edit formats.\n3. Changing the context to other choices of DP (direct prompt), LA+DP, and File that uses the results of the embedding model.\n\nTable below summarizes the pass@9 and pass@1 (after applying the filtering steps outlined in the paper) results for each combination. These results support our claim that diversifying both the edit format and context increases the number of correct patches, as each combination has its own strengths and weaknesses, and diversification allows us to leverage all of them.\n|Context |Edit Format|pass@9|pass@1|\n|-|-|-|-|\n|All|Both|307|263|\n|LA|Search/replace|270|237|\n|LA|Both|284|245|\n|DP|Both|261|220|\n|LA+DP|Both|286|239|\n|File|Both|257|223|\n\n**Context filtering:** In our prompts, we ask the agent to filter irrelevant context by issuing a special `<keep>` command after consuming a list of code context. We will provide these prompts upon code release.\n\n**Chunking strategy:** We experimented with the SFR-Embedding-Mistral model starting with a chunk size of 4k and gradually reducing it. We observed a recall@k of 43.67% at a chunk size of 4k on the Lite set and it peaked at a chunk size of 450 with recall@k of 57.33%. Based on these results, we selected a chunk size of 450 for both training our model and final inference. Similar to the SFR-Embedding-Mistral model, our model's retrieval accuracy decreases with larger chunk sizes. For instance, with a chunk size of 4k, the retrieval accuracy on the Lite set is 35.67%, compared to 70.33% with a chunk size of 450. We will include the full experimental results in the paper.", "claims_and_evidence": "Key claims:\n* Their embedding-based file retrieval approach outperforms a purely prompt-based approach => Well supported in Table 2, although additional ablations would strengthen this result.\n* Their agentic retrieval approach combining different models and retrieval stages improves entity retrieval recall over Agentless => Well supported in Tables 3 (when combined with ensembling across models) although interesting baselines such as using the same model at temperature or combining different retrieval stages from an individual model.\n* Their diversity-focused patch generation approach performs better than simply sampling at temperature => Well supported for pass@2 and two samples in Figure 4, with no ablations for the full end-to-end fixes with 9 generations.\n* Combining these components leads to a notably stronger method than Agentless => limited support with actual performance difference to the underlying Agentless being minimal <2% and no statistical significance reported (t-test for SWE-Lite gives only p=34%)", "methods_and_evaluation_criteria": "The dataset choice (SWE-Bench), evaluation criteria, and conducted ablation studies are mostly appropriate. Comparisons based on cost with unclear API prices for cited reference are hard to evaluate. Comparing at equal number of pre-filtering patches with Agentless may have yielded statistically significant improvements.", "theoretical_claims": "None", "experimental_designs_or_analyses": "I checked the soundness of all experiments reported in the main paper and they seem consistent and reasonable. However, more baselines and other methods could significantly strengthen the conclusions:\nIn Table 2, an additional embedding baseline of generating descriptions of to-be-retrieved code (and similar for the embedded code) and direct agentic retrieval like SWE-Agent or AutoCodeRover.\nIn Table 3, a comparison to other agentic retrieval approaches (e.g. AutoCodeRover v2, or even SWE-Agent v1).\nIt should perhaps be made more explicit when CORTEXA (and Agentless) use an ensemble (e.g. Table 2 (Prompt only) and Table 3).", "relation_to_broader_scientific_literature": "Comparison to other methods in Table 1 is quite limited with many approaches being relatively old and e.g. the closely related MarsCode Agent (published September 2024) not being included. A more detailed comparison of retrieval performance to e.g. MarsCode Agent, and AutoCodeRover (using a similar agentic approach and published in April 2024) should be included. While the authors claim MarsCodeAgent to be contemporary it was released more than 4 months before this submission.", "essential_references_not_discussed": "No critical works missing, although a broader overview of current code agents could be provided, see e.g. the 26 different agents compared to in Agentless.", "other_strengths_and_weaknesses": "### Strengths\n* Focus on efficiency via diversity in ensembling and as a result matching/improving performance at a lower cost is an important and interesting direction\n* Demonstrates the effectiveness of FineTuning embedding models specifically for code retrieval in the bug-fixing setting\n* Confirms the effectiveness of agentic entity retrieval and combining results across retrieval stages\n* Shows the effectiveness of increasing diversity before patch aggregation using different edit formats and contexts.\n\n### Weaknesses\n* In multiple places relevant baselines are not considered, limiting conclusions to be relative to Agentless.\n* Important details are not described (e.g. how is the context filtered (Section 3.3)? What prompts are used?)\n* Without these details and the trained embedding model (or more details on its training process), the results are not reproducible.\n* Variability and statistical significance of results are not measured/discussed but p-values computed with the available data show no statistically significant improvement over Agentless, despite a significantly more complex approach.", "other_comments_or_suggestions": "* I would suggest a table format with fewer (especially vertical) lines to improve readability.\n* Computing p-values using paired sample tests could strengthen claims.", "questions_for_authors": "1) Will the finetuned model, the generated datasets, or the agent traces be released?\n2) Ho was the context filtered by the code agent (Section 3.3)\n3) How does localization cost compare to Agentless? How would an Agentless approach perform at a matched token cost in Table 3?\n4) Can you discuss the ablation results on different chunking strategies mentioned around line 335 (left)?\n5) Can you compare to other localization methods in Tables 2 and 3 (see above)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CORTEXA, an agentic system designed to improve Large Language Models on software engineering tasks by enhancing localization and solution diversity. Reviewers acknowledged the clarity of the presentation (WZSG), the novelty of combining a fine-tuned embedding model with an agentic approach for localization (WZSG, sboJ), and the interesting direction of improving efficiency via diverse patch generation (sboJ, KBKm). However, initial concerns were raised regarding the performance compared to state-of-the-art methods (WZSG, KBKm), the lack of statistical significance testing or confidence intervals (WZSG, sboJ), the limited evaluation scope and missing baselines (KBKm, sboJ), and insufficient methodological details impacting reproducibility (KBKm, sboJ).

The authors provided a detailed rebuttal including substantial new experimental results and clarifications. Concerns about performance were addressed by presenting results from scaled-up experiments and improved selection mechanisms, demonstrating significant gains over the primary baseline (Agentless) and competitiveness with other relevant methods, often at lower cost. Statistical significance was established for key claims through p-value reporting and confidence interval analysis derived from additional runs. The authors also included new ablations clarifying the impact of the localization and diversity components, added comparisons to more contemporary agentic retrieval baselines, provided further methodological details, and committed to releasing code and models, addressing reproducibility concerns.

Overall, the initial reviews highlighted valid points requiring clarification, while the extensive rebuttal and promised changes effectively mitigated the major weaknesses. The strengthened empirical evidence, particularly the demonstrated performance gains achieved through improved localization and patch diversity with a clear focus on efficiency, now supports the paper's contributions. The work presents valuable insights and techniques for developing more effective LLM-based software engineering agents, justifying its inclusion in the conference program.
Decision: Accept (poster)