SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Authors: John Yang
Carlos E Jimenez
Alexander Wettig
Kilian Lieret
Shunyu Yao
Karthik R Narasimhan
Ofir Press
Venue: NeurIPS 2024 poster (2024)
Keywords: Language models
Natural language processing
Software engineering
Primary area: natural_language_processing
TL;DR: We introduce the concept of agent-computer interfaces to enhance LM agent performance on software engineering tasks.

Abstract: Language model agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that language model agents represent a new category of end users with their own needs and abilities, and would benefit from specially built interfaces to the software they use. We investigate how the role of interface design affects the performance of language model agents. As a result of this exploration, we introduce SWE-agent: a system that facilitates language model agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively, far exceeding the previous state-of-the-art achieved with non-interactive language models. Finally, we provide insight on how the design of the agent-computer interface can impact agents' behavior and performance.

SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
John Yang"
Carlos E. Jimenez'
Alexander Wettig
Kilian Lieret
Shunyu Yao
Karthik Narasimhan
Ofir Press
Princcton Language and Intelligence. Princeton University
Abstract
Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent new category of end users with their own needs and abilities, and would benefit from specially-buil interfaces to the software they use. We investigate how interface design affects the performance of language model agents. As result of this exploration, introduce SWE-agent: a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's 's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanFvalfix, achieving state-of-the-art performance on both with pass @1 1 rate of 12.5% and 87.7%, respectively, far exceeding the previous state-of-te saae--oftthe-ar achieved with non-interactive LMs. Finally, we provide insight on how the design of the ACI can impact agents' behavior and performance.
Introduction
Recent work has demonstrated the efficacy of L.M agents for code generation with execution feed- back [39]. However, applying agents to more complex code tasks like software engineering remains unexplored. To solve programming tasks, LM agents are typically designed to use existing applica- tions, such as the Linux shell or Python interpreter [53, 57, 59]. However, to perform more complex programming tasks such as software engineering [20], human engineers benefit from sophisticated ap- plications like VSCode with powerful tools and extensions. Inspired by human-computer interaction (HCI) studies on the efficacy of user interfaces for humans [7], we investigate whether LM agents could similarly benefit from better-designed interfaces for performing software engineering tasks.
SWE-agent
Ogent-Computer Interface
Computer
LM-friendly commands Navigate repo PSeorch tils Eelt lines
Terminal
View flle
Filesystem
LM Agent
skleam/
examples/
LM Mi-frrindlly environment feedback
README.rst
Figure 1:
SWE-agent is an LM interacting with a computer through an agent-compuct interface (ACI), which includes the commands the agent uses and the format of the feedback from the computer.
"Equal contribation. Correspondence to johnby stan ford edd, carlose pew.iiietooonnnnnnn Data, code, and leaderboard at swe-agent.com
ncetonn edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Consider the simple setting of an agent interacting directly with a Linux shell [59]. In practice, we find that LM agents can struggle to reliably take actions in this environment. For example, it fails to provide simple commands to edit small file segment, and does not provide any feedback if the user makes an invalid edit. These deficits substantially hamper performance, motivating the need for an agent-computer interface (ACI), i.e., an abstraction layer between the LM agent and computer, to enhance the LM agent's abilities in computer environments (Figure 1).
From this effort, we introduce SWE-agent, an agent composed of an LM and ACI, that can interact with a computer to solve challenging real-world software engineering problems, such as those proposed in SWE-bench [20]. In contrast to the Linux Shell's ' granular. highly configurable action space, SWE-agent's ACI instead offers a small set of simple actions for viewing, searching through and editing files. The ACI uses guardrails to prevent common mistakes, and an agent receives specific, concise feedback about command's 's effects at every turn. We show that ACIs tailored specifically for LMs outperform existing user interfaces (UIs) designed for human users, such as the Linux shell.
Using GPT-4 Turbo as a base LM, SWE-agent solves 12.47% of the 2,294 SWE-bench test tasks, substantially outperforming the previous best resolve rate of 3.8% by a non-interactive, retrieval- augmented system [20]. We perform an ablation study on subset of 300 SWE-bench test instances (SWE-bench Lite) to analyze our ACI design choices. The results show that SWE-agent solves 10.7 percentage points more instances than the baseline agent, which uses only the default Linux shell. Although our ACI was developed for GPT-4 Turbo, we show that it is portable to a different LM; SWE-agent with Claude Opus can solve 10.5% of the benchmark tasks.
Our contributions are twofold. First, we introduce the concept of the agent-computer interface (ACT) and demonstrate how careful ACI design can substantially improve LM agent performance without modifying the underlying LM's weights. Second, we build, evaluate, and open-source SWE-agent, a system that provides LMs an ACI for solving real-world software engineering tasks. Unlike prior works that independently explore the merits of tool use, prompting techniques, and code execution in interactive settings, our approach unifies these factors within the ACI framework. We show that crafting LM-centric interactive components has meaningful effects on downstream task performance.
2 The Agent-Computer Interface
An LM acts as an agent when it interacts with an environment by iteratively taking actions and receiving feedback [42, 62). Typically, the environment has hard constraints, as in robotics, where agents control actuators in the physical world. On the other hand, digital environments can be molded by abstractions in the form of application programming interfaces and user interfaces for software and humans respectively. Naturally, existing interfaces have been designed with one of these users in mind. We argue that LM agents represent a new category of end user, with their own needs and abilities. We refer to the interface LM agents use to interact with computers the agent-computer interface (ACI). Figure 2 illustrates how ACIs provide LM agents with important functionality to interface with computers, similar to how code editors also help humans use computers more effectively.
f Computer
Computer U
ACI
File Viewer
SB LM Agent
Human
PC
File Editor
Code Search
Figure 2: Specialized applications like IDEs (e.g., VSCode, PyCharm) make scientists and software engineers more efficient and effective at computer tasks. Similarly, ACI design aims to create a suitable interface that makes LM agents more effective digital work such as software engineering.
Disparities in humans' and LMs' abilities and limitations motivates different interface design guide- lines. For instance, the current generation of LMs lack the visual understanding abilities to directly operate GUI-based applications with rich visual components and signals. However, many of the features provided by these applications, such as syntax checking and navigation tools, could be useful to LM agents they were presented in a suitable manner. Additionally, humans can flexibly ignore unnecessary information, whereas all content has fixed cost in memory and computation for LMs

---

and distracting context can harm performance [27]. Therefore. LM agents may be more effective at interacting with computers when provided an interface that was built informed by these differences.
Ultimately, a well-designed ACI should help the LM agent understand the state of the application given previous changes, manage history avoid unnecessary context from prior observations, and provide actions that models can use efficiently and reliably. The ACI specifies both the commands available to the LM and how the environment state is communicated back the LM. It also tracks the history of all previous commands and observations and, at each step, manages how these should be formatted and combined with high-level instructions into a single input for the LMM
In this paper, we assume a fixed LM and focus on designing the ACI to improve its performance. This means that we shape the actions, their documentation, and enviromment feedback to complement an LM's limitations and abilities. We draw inspiration from the field of HCI, where user studies elicit insights about how compatible different interfaces are with respect to human intuition and performance [7]. We use two approaches to enhance performance on a development set: (1) manually inspect agent behavior to identify difficulties and propose improvements, and (2) run grid search to select the best ACI configuration.
Taking these two actions resulted in several insights about design principles that seem especially important for building effective ACIs:
Actions should be simple and easy to understand for agents. Many bash commands have documentation that includes dozens of options. Simple commands with few options and concise documentation are easier for agents to use, reducing the need for demonstrations or fine-tuning. This is defining principle for all SWE-agent commands that we describe in Section 3.
Actions should be compact and efficient. Important operations (e.g., file navigation, editing) should be consolidated into as few actions as possible. Efficient actions help agents make meaningful progress towards a goal in a single step. poor design would therefore have many simple actions that must be composed across multiple turns for a higher order operation to take effect. We show this idea in action in the Editing and Search interface analyses in Section 5.
3. Environment feedback should be informative but concise. High quality feedback should provide the agent with substantive information about the current environment state (and the effect of the agent's ss recent actions) without unnecessary details. For instance, when editing a file, updating the agent about revised content is helpful. Figures 3a, 3b and Table show this.
4. Guardrails mitigate error propagation and hasten recovery. Like humans, LMs make mistakes when editing or searching and can struggle recover from these errors. Building in guardrails, such as code syntax checker that automatically detects mistakes, can help agents recognize and quickly correct errors. We show the effect of editing guardrails in Table 3.
Analysis and ablation studies in Section 5 demonstrate how alternative ACIs affect LM performance. Our studies shows how these principles appear recurrently across actions, feedback, and workflows.
3 SWE-agent: Designing an ACI for Software Engineering
Here we describe how SWE-agent provides an ACI for LMs to act as software engineering agents, enabling them to effectively search, navigate, edit, and execute code commands. The ACI com- prises several principal components, including search/navigation, file viewer, file editor, and context management. At each step, SWE-agent generates thought and a command, then incorporates the feedback from the command's i execution in the environment (ReAct; Yao ct al. [62]). Built atop the Linux shell, SWE-agent also allows access to common Linux commands and utilities when needed.
Search and navigation. Navigating codebases requires finding the relevant file and content. A common strategy to do this involves looking up terms that might be useful, e.g., files, functions, or class definitions mentioned in an issue. We introduce the special commands find ffffiie file, search_fiie and hearch_dize which output a summary of search results when searching for filenames and strings within files or directories. Figure 10 shows examples of these search result formats. The find_file command searches for filenames in the repository. while the sea bearch_file and search_di locates strings in a file(s) of subdirectory. Our interface encourages efficient searches by suppressing verbose results. The search commands return at most 50
3

---

Observation (File Viewer)
Response (Thought Edit Action)
[File: solvers/die solvers/diophantine.p (3093 lines total]]
We need to add check to ensure that all powers are the same. Let's edit the code to include this check.
(480
Lines above)
401:
elif total_degree = 3:
402:
diop__yye "cubic_thue"
403:
edit 404;407
405:
484:
elif (total degree 3):
if [coeffof[ diop_type "genenal_sum_ove""
elif (total deggree and len(set isp 1):
406:
107:
if all[coeff[k] -- 1):
408:
if diop_.type is not None:
diop_type gemerral_su_evee
409:
neturn var, coeff, diop_tyye
end_of_edit
(2684 Lines below)
(a) Observation from the file viewer.
(b) Action using the edit interfacc.
Figure 3: The file viewer and edit command are integrated. (a) The file viewer shows the agent the open file's content with line numbers. (b) The agent invokes the edit function replace lines 404-407 in the open file. After the edit, the file viewer shows the agent the now updated version of the file.
results for each search query; if a search exceeds this number, we do not report the results and instead suggest that the agent write more specific query.
File viewer. After finding a file they want to view, agents use the interactive file viewer by calling the command open on the relevant file path. The file viewer presents a window of at most 100 lines of the file at a time. The agent can move this window with the commands scroll doww and scroll 1 up or access a specific line with the goto command. To facilitate in-file navigation and code localization, we display: the full path of the open file, the total number of lines in the file, the number of lines omitted before and after the current window, and the line number (prepended each visible line). Figure 3a shows an example of this interface.
File editor. We provide few commands that let LMs create and edit files. The edit command works in conjunction with the file viewer, allowing agents to replace a specific range of lines in the open file. This command takes 3 required arguments: the start line, end line, and replacement text. In a single step, agents can replace all lines between the start and end lines with the replacement text, as shown in Figure 3b, After edits are applied, the file viewer automatically displays the updated content, helping the agent observe the effects of its edit immediately without invoking additional commands. Figure 3b shows example agent response, including a file edit.
Similar to how humans can use tools like syntax highlighting to help them notice format errors when editing files in an IDE, we integrate code linter into the edi t function to alert the agent of mistakes may have introduced when editing file. Select errors from the linter are shown to the agent along with snippet of the file contents before/after the error was introduced. Invalid edits are discarded, and the agent asked to try editing the file again.
Context management. The SWE-agent system uses informative prompts, eтror messages, and history processors to keep agent context concise and informative. Agents receive instructions, documentation, and demonstrations on the correct use of bash and ACI commands. At each step, the system instructs them to generate both thought and an action [62]. Malformed generations trigger an eтrоr response, shown in Figure 32, asking the agent to try again, which is repeated until valid generation is received. Once received, all past error messages except the first are omitted.
The agent's environment responses display computer output using the template shown in Figure 30; however, if no output is generated, a specific message Y"Your command ran successfully and did not produce any output") is included to enhance clarity. To further improve context relevance, observations preceding the last 5 are each collapsed into a single line, shown in Figure 31. By removing most content from prior observations, we maintain essential information about the plan and action history while reducing unnecessary context, which allows for more interaction cycles and avoids showing outdated file information. ŠA provides further implementation details.
4 Experimental Setup
Datasets. We primarily evaluate on the SWE-bench dataset, which includes 2,294 task instances from different repositories of popular Python packages [20]. We report our main agent results on the full SWE-bench test set and ablations and analysis on the SWE-bench Lite test set, unless

---

otherwise specified. SWE-bench Lite is a canonical subset of 300 instances from SWE-bench that focus on evaluating self-contained functional bug fixes. We also test SWE-agent's s basic code editing abilities with HumanFvvlFixx, a short-form code debugging benchmark [32].
Models. All results, ablations, and analyses are based on two leading LMs, GPT-4 Turbo (gpt 1opt-4-1106 -pee-iieeee [34] and Claude 3 Opus (claude-33 --puss 2o0puss202422999 [6]. We experimented with number of additional closed and open source models, including Llama 3 and DeepSeek Coder [14], but found their performance in the agent setting to be subpar. Many LMs' context window too small, such as Llama 3's context window of 8k. GPT-4 Turbo and Claude 3 Opus have 128k and 200k token context windows, respectively, which provides sufficient room for the LM interact for several turns after being fed the system prompt, issue description, and optionally, demonstration.
Baselines. We compare SWE-agent to two baselines. The first setting is the non-interactive, retrieval- augmented generation (RAG) baselines established in Jimenez et al. [20]. Here, a BM225 retrieval system retrieves the most relevant codebase files using the issue as the query; given these files, the model is asked to directly generate a patch file that resolves the issue.
The second setting, called Shell-only, is adapted from the interactive coding framework introduced in Yang et al. [59]. Following the InterCode environment, this baseline system asks the LM to resolve the issue by interacting with shell process on Linux. Like SWE-agent, model prediction is generated automatically based on the final state of the codebase after interaction.
Metrics. We report % Resolved or pass@1 as the main metric, which is the proportion of instances for which all tests pass successfully after the model generated patch is applied to the repository [20]. We also report the $ Avg. Cost metric, the API inference cost incurred by SWE--gent averaged over all successfully resolved instances. Due to budget constraints, we set the per-instance budget to $4; run exceeded this budget, existing edits were submitted automatically.
Configuration search. During the design process of SWE-agent, we arrived the final ACI design through qualitative analysis of system behavior on a small set of sank-p examples from the development split of SWE-bench. For the remaining hyperparameter choices, we performed a sweep over the window size, history processing, and decoding temperature, shown in $B.1.
Results
Across all systems, SWE-agent w/ GPT-4 Turbo achieves the best performance all-around, success- fully solving 12.47% 2286/2294) of the full SWE-bench test set and 18.00% (54/300) of the Lite split. As shown in Table 1, compared RAG on Lite, SWE-agent 8-13x more costly but yields a 6.7-fold improved % Resolved rate. An LM-friendly ACI's value is confirmed by SWE-agent's 's 64% relative increase compared to Shell-only, both with GPT-4 Turbo.
In Table 2, SWE-agent yields strong performance on HumanEvalFix with 88.3% pass@1 rate. Figure reveals that average performance variance is relatively low, but per-instance resolution can change considerably. More results are given in the appendix: $B.2 shows that the success rate is uncorrelated to the issue age (controlling for possible test pollution), B.5 presents more details on performance variance and pass@k, and B.7 discusses extra evaluation details.
5.1 Analysis of ACI Design
We perform several ablations of the SWE-agent interface, specifically with respect to the SWE-agent w/ GPT-4 configuration, summarized Table 3. Our case studies shed light on interesting agent behavior along with the impact of different ACI designs.
Human user interfaces are not always suitable agent-computer interfaces. Current LMs vulnerable to a number of pitfalls when searching for relevant content in a Linux shell environment. Some exploration patterns (e.g., chains of cd, 1s, cat) are extremely inefficient. grep fi nd look ups can perform better but occasionally produce many lines of irrelevant results. We hypothesize that better localization is possible with faster navigation and more informative search interface.
http:://ithu..comm// Token counts for different models are not directly comparable since they different tokenizers.
5

---

Table Main results for SWE-agent performance on the full and Lite splits of the SWE-bench test set. We benchmark models in the SWE-agent, Basic CLI, and Retrieval Augmented Generation (RAG) settings established in SWE-bench [20].
SWE-bench
SWE-bench Lite
Model
% Resolved
$ Avg. Cost
% Resolved
$ Avg. Cost
RAG
w/ GPT-4 Turbo
1.31
0.13 0.25
2.67 4.33
0.13
w/ Claude 3 Opus
3.79
0.25
Shell-only agent
w/ GPT-4 Turbo
11.00
1.46
w/o Demonstration
7.33
0.79
SWE-agent
w/ GPT-4 Turbo
12.47 10.46
1.59
18.00
1.67
w/ Claude 3 Opus
2.59
13.00
2.18
35
% Resolved
Table Pass@1 results HumanBvalFix [32]. Except for SWE-agent, we use scores as reported in Yu et al. [65].
30
25
Model
Python
JS
Java
20
CodeLLaMa-instrucce 13B
29.2
19.5
32.3
15
2
5
GPT-4
47.0
48.2
50.0
k
DeepseckCoder--CodAAppc...
49.4
51.8
45.1
Figure 4: SWE-agent w/ GPT-4 Turbo Pass@k performance across 6 runs on SWE-bench Lite.
WaveCoder-DS-6.7B SWE-agent w/ GPT-4 Turbo
57.9
52.4
57.3
87.7
89.7
87.9
Table 3: SWE-bench Lite performance under ablations to the SWE-agent interface, which is denoted by t. We consider different approaches to searching and editing (see Figures 5 and 6, respectively). We also verify how varying the file viewer window size affects performance, and ablate the effect of different context management approaches.
Editor
Search
File Viewer
Context
edit action 15.0 1.3.0
Summarized
18.0
30 lines
14.3 3.3
Last 5 Obs. 18.0
w/ linting
18.0
Iterative
12.0 6.60
100 lines *
18.0
Full history 15.0 s.
No edit
10.3 1.7.
No search
15.7 2/3
Full file
12.7 u.3
w/o demo.
16.3 1.7
Figure compares the Shell-only setting two different search interfaces. Iterative search, directly inspired by traditional user interfaces for search, e.g., Vim or VSCode, shows results one by one via the file viewer. Agents can look through results using next and prev actions. Each result displays the matching line along with surrounding lines of context. An advantage is that an agent can begin editing directly after seeing the relevant code its search. However, when given large number of search results, agents tend to look through every match exhaustively, calling next until each result has been inspected. This inefficient behavior can exhaust an agent's cost budget or context window, leading to even worse performance than the not having additional search tools at all (15.7% for No search vs. 12.0% 1 with Iterative search).
Compact, efficiint file editing is critical to performance. SWE-agent's file editor and viewer are designed to consolidate the editing process into a single command that enables easy multi-line edits with consistent feedback and automatically updates the agent's view of the file after editing. In the No edit t setting. editing options are restrictive and prone errors; the primary methods available are either replacing entire files through redirection and overwriting or using utilities like sed for single-line or search-and-replace edits. Both methods have significant drawbacks. Redirection involves copying and rewriting entire files for even minor changes, which is both inefficient and error-prone. Although sed can facilitate specific edits, executing multi-line edits cumbersome and can lead to unintended consequences that are challenging to detect. Moreover, both strategies

---

p No Search
p Iterative Search
P Summarized Search
Agent periorma manusl search with 1a cd. or usme grep but atruggles enterpoet the mesu!
Actions show naxt/ paev search reault are repasted many times until resutt are rohausted
(1) Show resutss sngle putput
(2) agent to retry' too many mestts
search PuSycamn"
cadebase may contain deftiitioo PVttycomm
TVSystemm
seazch "PVSyater"
modty the constructor..
(Resultt (Line 78)]
pearch "class Präystes"
boncheackk//
sctup.cf
[Flle: wwllb/vvysste
Found natches for *cleza PVSyatarr in /pvlib python: thon: /pvisb...Pvibb pythoo//pwiib/ pvayster. py (1. matchee) End of natches
docs/
wwc.iunee +P README mnd
22:4 of zequized pa.... 23: 23 PARASS
pvlss//
retup- Py
'aape'
(97 more Aned cmiitec)
Itle most licely the sourcw pubb/ folder Let's change directory.
PVSystem
It's located pattlppwyeeemms
pultb/
nect
open pvl pvl:b/pvaystew.pp
Figure 5: Three different Search interfaces for task instance pvlib wilib-pythhnn-o 1224. In Shell-only, an agent performs localization using only standard bash commands and utilities. Compared to Iterative search, Summarized search shows an exhaustive list of search results and provides guidance on refining under-specifted queries.
No edit
@ edit w/o Linting
edit w/ Linting
Fallare Modia:
causes uncertainty
edit conceliiates necossary stops 1 loction but roctesory from self thour red emerss oiffcullt
"M--conn'""" edi1 (0) Show He before & ather edit (2) Block edit with syntax comaile omrors
(2) cat floods contact window (5) Ma must many states (eg line the pre/post-ectt)
M use out d'spley symoy! parang/mathematiag
Let' edit the _otomm conversiios desiinaryy to ind the mapping..
edit the _atto__crreniirs Some as Editing achio)
cus parsing //ea thee Jea..p
(Fle Content (-1079 Lines)
end_of_ _dii
1065:1059 [Code Snippot)
Year propocee HdLt hos Incrunased
$290 Indernatto Wirres
systte
1053:
Funetiort_parse...
leaked
To fix the
modify
1254.
(Pitle Viseer Balt loellies
the Mattemattccaaaoe
sed 128 retglt Va (taee
1066:
KJ57
"t":
"P1"1
[Fille Vineer Originnl Cuan)
"oar changes have beer azalled.
(No Duspptt
10st:
"g";
seers thami's misunderatarding.
Syntax Erroe!
1960
1069:
"A": vy
introdicus untra T.I update my commenc to fix this...
write Python
add the nuies..
$ echo (30 more (inee omvitte)
X302:
for........
(ooss snippei] undudd
24102. Editing with bash commands requires several actions to successfully modify file. The Editing component defines an edit t command that leverages the File Viewer component to replace the bash style of editing workflow with single command. Linting is beneficial for stymieing cascading errors that often start with an error-introducing edit by the agent.
Figure 6: Three different Edit interfaces for task instance sympy sympy-24
lack immediate feedback about file updates, making these silent operations potentially confusing for models interpret and increasing the risk of errors. Without SWE-agent's file editor interface, performance drops to (10.3% 1 7.7). We also find that agents are sensitive to the number of lines the file viewer displays. Either too little content (30 lines, 14.3% 43.7) too much (entire file, 12.7% 4. J.5.3) lowers performance.
Guardrails can improve error recovery. A prominent failure mode occurs when models repeatedly edit the same code snippet. The usual suspect for this behavior is an agent introducing a syntax error (e.g., incorrect indentation, extra parenthesis) via errant edit. As discussed in Section 3, we add an intervention to the edit logic that lets a modification apply only if it does not produce major errors. We compare this interface with the No edit and edit w/o linting alternatives in Figure 6. This intervention improves performance considerably (without linting, 15.0% 330))
5.2 Analysis of Agent Behavior
Recurring Sobblem-solving patterns emerge when LMs are equipped with a useful, intuitive ACI. We describe several model behaviors and moblemm-sllving patterns that can be discerned from model performance and each model's 's corresponding trajectories.
Reproduction and/or localization is the first step. SWE-agent usually begins with either writing reproduction code and/or localizing the issue's cause to specific lines of code. As shown Figure 7, all trajectories begin with either create (reproduction) or find__t fedffile/see file//errch di (localization). To reproduce, models will create te a new file, add reproduction code to it with an ed t,, then run with python: hon; this is the most popular triple of actions in Table 8. Using this feedback along with file
7

---

search dir
search file
scroll_ up
250
Find file
scroll down
39,9
find
gota
200
create
pytho3
Frequency
Pyrhon
12.1
150
exit_ coos
pytest
subenit
4.8
100
23.4
50
Incurret Implomentatio
Failed Find Relevant
Overly Specilic Implementaiio
Gave Up Premoturely
0
15
27
30
33
36
Failed to Recover from Edit
Can't Reprodice
Turn
Fiiled to Find Edit Location
Ran Out of Time
Figure 7: The frequency with which actions are invoked at each turn by SWE-agent w/ GPT-4 for task instances that it solved on the SWE-bench full test set (286 trajectorics).
Figure 8: Failure mode distribution for SWE- agent w/ GPT-4 Turbo trajectories of unresolved instances. Each instance is labeled automatically using an LM with the categories from Table 9.
names and symbols in the issue description, agent will start with broad, directory-level keyword search, before then zooming into specific files and lines. This is reflected in Figure 22, where the most likely actions following localization sequences like (python, find file) and (search dir, open) are search_ file and goto, indicative of how an agent "zooms in" on a bug. Extensive analysis correlations between different groups of actions are discussed in $B.3.3
Remaining turns are mostly "edit, then execute" loops. As exhibited in Figure 7, from turn 5 onwards, the most frequent two actions for all turns are edit and python. Captured as high probability next actions following (edit, python)
Figure 22, additional localization operations are often interspersed across these later turns, where agents might look at more in-file code with search_ffi file, scroll _a___ppppoo,,,,, or other files altogether with search_ _iir find__ Biidd__ill file. This behavior usually arises in response to new information from re-running the reproduction script. Submissions are distributed normally from turn 10 onwards, although resolved task instances correlate more with earlier submits (see $B.3.1). 1. A walk-through of common trajectory phases is in $B.3.2.
Editing remains challenging for agents. A non-trivial minority of edit actions raise a linting error; out of 2,294 task instances, 1,185 (51.7%) ) of SWE-agent w/ GPT-4 Turbo trajectories have 1+ failed edits. While agents generally recover more often than not from failed edits, the odds of recovery decrease as the agent accumulates more failed edits. Recovery refers to sequence of consecutive failed edits followed immediately by a successful edit. Any attempt at editing has 90.5% chance of eventually being successful. This probability drops off to 57. 7.22% after single failed edit. More editing phenomena are discussed in $B.3.3, and data about agents' generated fixes are in $B.6.
Agents succeed quickly and fail slowly. We find that runs submitted relatively early are much more likely to be successful compared those submitted after a larger number of steps or cost. We show in Table 15 the distribution of resolved and unresolved instances, including only instances that did not exhaust their budget. We observe that successful runs complete earlier and at a cheaper cost than unsuccessful ones. In general, successful instances solved by SWE--aget w/ GPT 4 finish with median cost of $1.21 and 12 steps compared to a mean $2.52 and steps for unsuccessful ones. Furthermore, we find that 93.0% of resolved instances are submitted before exhausting their cost budget, compared to 69.0% of instances overall. For these reasons, we suspect that increasing the maximum budget token limit are unlikely to substantially increase performance. More statistics about how trajectories typically conclude are in $B.9.
Most failures are incorrect implementations. We use GPT-4o to automatically categorize unre- solved trajectories (SWE-agent w/ GPT-4 Turbo on SWE-bench Lite, n =248) into one 9 manually defined categories described in Table 9, On hand-labeled validation set, the LM's judgment agrees with the authors' on 87% of instances. From Figure 8, about half (52.0%) of unresolved instances fall into the Incorrect Implementation or Overly Specific Implementation categories, suggesting that agents' proposed solutions often simply fail functionally address the issue or are insufficiently general solutions. Cascading failed edits make up another 23.4% of failures. More details in $B.4.
8

---

Related Work
6.1 Software Engineering Benchmarks
Code generation benchmarks, which evaluate models on the task of synthesizing code from natural language descriptions, have served as a long-standing bellwether for measuring LM performance [5,
1, 15, 30J. Subsequent works have built upon the code generation task formulation to contribute new benchmarks that translate problems to different (programming) languages [3. 49], incorporate third-party libraries [25, 29], introduce derivative code completion tasks [18, 32], increase test coverage [26], change the edit scope [8, 9. 64], and add robustness to dataset contamination [19]. Code generation problems are largely self-contained, with short problem descriptions (~100 lines) and corresponding solutions that are similarly brief. requiring nothing more complex than basic language primitives. Tests are either handwritten or generated synthetically via fuzz testing. In recent months, the rapid development of LMs has begun to saturate many of these benchmarks. For instance, the top method solves 94.4% of HumanEval [70].
Gauging future trends with the code generation task paradigm can be limited by the simplicity of this setting and cost of human-in-thed -n-te-----o---ppp problem creation. In response, recent efforts have demonstrated that software engineering (SE) can serve as a diverse, challenging testbed for LM evaluation [68, 20, 28]. Repository-leve code editing introduces many reasoning challenges grounded in real SE subtasks, such as spotting errant code and identifying cross-file relationships and understanding codebase- specific symbols and conventions. As a field, SE has generally studied tasks in more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23).
We use SWE-bench because it unites many separate SE tasks, such as automated program repair [10, 40, 55]. bug localization [4, 58]. and testing [22, 46, 56] under a single task formulation that faithfully mirrors practical SE. Furthermore, SWE-bench task instances are diverse, having been automatically collected from real GitHub issues across 12 different repositories. addition, SWE- bench performance is based on rigorous, execution based evaluation with human human-written unit tests.
6.2 Language Models as Agents
The co-emergence of stronger LMs, increasingly challenging benchmarks, and practical cases have together motivated a paradigm shift in LMs' inference setting. Instead of traditional zero/few- shot generation, LM agents [17, 42, 47, 54] that interact with real/virtual world have proliferated as the default setting for web navigation [24, 33, 36, 41, 45, 61, 62, 71], computer control [35, 53, 57], and code generation tasks [16, 50, 63].
Interaction and code generation are increasingly used together, with code as the modality of choice for actions [48, 59], tool construction [13, 51, 69], and reasoning [39, 66, 67]. Coding agents have also been applied to offensive security 37, 60), theorem proving [44], and clinical tasks [38, 43, 52]. To the best of our knowledge, SWE-agent is the first work explore language agents for end-to-end software engineering (SE).
7 Discussion
We introduce SWE-agent, an agent composed of an LM and ACI capable of autonomously solving software engineering tasks. Through our design methodology, results, and analysis, we demonstrate the value of ACIs tailored to leverage LMs' strengths and mitigate their weaknesses. Beyond empirical applications, we hope the further study of ACIS can also make principled use of and contribute to our understanding of language models and agents, analogous to the synergy between human-computer interaction (HCI) and psychology [2]. Humans and LMs have different characteristics, training objectives, specialities, and limitations [12, 31], and the interaction design processes can be seen as systematic behavioral experimentation that could reveal more insights into these differences towards establishing a comparative understanding of human and artificial intelligence.

---

Acknowledgements
We thank Austin W. Hanjie, Sam Ainsworth, Xindi Wu, Yuhan Liu, Mengzhou Xia, Dan Friedman, Tianyu Gao, Adithya Bhaskar, Aatmik Gupta, Louisa Nyhus, Alisa Liu, Ori Yoran and Richard Zhu for their valuable feedback and advice. We would also like to thank the broader Princeton Language and Intelligence community for supporting our work. We acknowledge support from an Oracle Collaborative Research award and the National Science Foundation under Grant No. 2239363. Any opinions, findings, conclusions, recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views the National Science Foundation
References
[1] Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021.
[2] J. M. Carroll. Human-computer interaction: psychology science of design. Annual review of psychology, 48(1):61-83, 1997.
[3] F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, I.. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. 7i, C. J. Anderson, M. Q. Feldman, Guha, M. Greenberg, and A. Jangda. Multipl-e: A scalable and extensible approach benchmarking neural code generation, 2022.
[4] S. Chakraborty, Y. Li, M. Irvine, R. Saha, and B. Ray. Entropy guided spectrum based bug localization using statistical language model. arXiv preprint arXiv:1802.06947. 2018.
[5] M. Chen, J. Tworek, H. Jun, Q Yuan, H. de Oliveira Pinto, and K. ct. al. Evaluating large language models trained on code, 2021.
[6] W.-L. Chiang, L. Zheng, Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
[7] A. Cooper, R. Reimann, and D. Cronin. About 3: the essentials of interaction design. John Wiley & Sons, Inc., USA, 2007. ISBN 9780470084113.
[8] Y. Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, Bhatia, D. Roth, and B. Xiang. Crosscodeeval: diverse and multilingual benchmark for cross- file code completion. In Shirth- Conference Neural Information Processing Sys- tems Datasets and Benchmarks Track, 2023. URL https: / wwwp//ppee.eweeeeeeeee net/ forum? id=wgDcbBMS
[9] X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou. Classeval: A manually-crafted benchmark for evaluating 1lms class-level code generation, 2023.
[10] Z. Fan, X. Gao, M. Mirchev, Roychoudhury, and H. Tan, Automated repair of programs from large language models, 2023.
[11] R. Fang, R. Bindu, Gupta, Q. Zhan, and Kang. Lim agents can autonomously hack websites, 2024.
[12] T. Griffiths. Understanding human intelligence through human limitations. Trends in Cognitive Sciences, 24(11):873 883, 2020.
[13] Y. Gu, Y. Shu, H. Yu. X. Liu, Dong, Tang, J. Srinivasa. Latapie and Y. Su. Middleware for Ils: Tools are instrumental for language agents in complex environments, 2024.
[14] D. Guo, Q. Zhu, Yang, Xie, K. Dong, W. Zhang, G. Chen, Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence. CoRR, abs/2401. 14196, 2024. URL https: //arxiv. org/ab......... /20022411000 14196.
10

---

[15] D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps, 2021.
[16] S. Holt, M. R. Luyten, and M. van der Schaar. L2MAC: Large language model automatic computer for unbounded code generation. In The Twelfth International Conference Learning Representations, 2024. URL https: /ttp:r////eeeviieeeeeeeeeee
net/ / forum? ? d=EhrzQwsVA4
[17] S. Hong, M. Zhuge, J. Chen, X. Zheng. Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, Xiao, C. Wu, and Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework, 2023.
[18] Q. Huang, J. Vora, P. Liang, and J. Leskovee. Mlagentbench: Evaluating language agents machine learning experimentation, 2024.
[19] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Colar-Lezama K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024.
[20] C. Jimenez, Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. SWE-hench: Can language models resolve real-world github issues? In The Twelfth International Con- ference on Learning Representations, 2024. URL https:/ /oppnreeeeeew..ee///////ee net forum? LL-VTF&yNQMG
[21] R. Just, D. Jalali, and M. D. Ernst. Defects4J: A Database of existing faults to enable controlled testing studies for Java programs. ISSTA 2014, Proceedings of the 2014 International Symposium Software Testing and Analysis, pages 437-440, San Jose, CA, USA, July 2014. Tool demo.
[22] S. Kang, J. Yoon, and S. Yoo. Large language models are few-shot testers: Exploring Ilm-based general bug reproduction, 2023.
[23] R.-M. Karampatsis and C. Sutton. How often do sinleestatemmett bugs occur? the manysstubs4j dataset. 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pages 573-577, 2019. URL https: /apii.pppi// semanticscholeee org/CorpusID: 173188438.
[24] Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P.-Y. Huang, G. Neubig, Zhou, Salakhut- dinov, and D. Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024.
[25] Y. Lai, C. Li, Y. Wang. T. Zhang, R. Zhong, L. Zettlemoyer. S. W. tau Yih, D. Fried, S. Wang. and T. Yu. Ds-1000 A natural and reliable benchmark for data science code generation, 2022.
[26] J. Liu, C. Xia, Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigor- ous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01220, 2023.
[27] N. Liu, K. Lin, Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts, 2023.
[28] T. Liu, C. Xu, and J. McAuley. Repobench: Benehmarking repository-leve code auto- completion systems. In The Tweifth International Conference on Learning Representations, 2024. URL https: : // //opperrvii ew, net. forum? d=pP j IOQQU.
[29] Y. Liu, X. Tang, Z. Cai, J. Lu, Y. Zhang, Y. Shao, Z. Deng, H. Hu, K. An, R. Huang, S. Si, S. Chen, H. Zhao. L. Chen, Y. Wang, T. Liu, Z. Jiang, B. Chang, Y. Qin, W. Zhou, Y. Zhao, A. Cohan, and M. Gerstein. MI-bench: Evaluating large language models for code generation in repository-level machine learning tasks, 2024.
[30] Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L.. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sun- daresan, S. K. Deng, S. Fu, and Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation, 2021.

---

[31] R. T. McCoy, S. Yao, D. Friedman, M. Hardy, and T. L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained solve. arXiv preprint arXiv:2309. 13638, 2023.
[32] N. Muennighoff, Q. Liu, A. R. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, S. Singh, X. Tang, L. V. Werra, and S. Longpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: / //openreview..o new. perum?id-mwlPae
[33] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022.
[34] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, Babuschkin, S. Balaji, V. Balcom, P. Bal- tescu, H. Bao. M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Sermadett-Shapiee C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage. K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung. D. Cummings. J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou,
D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He. M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton. K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang. R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn. H. Jun, T. Kaftan, Lukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick. J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Lukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. Mc- Grew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O' OeKeefe, J. Pachocki, A. Paino. J. Palermo, A. Pantuliano, G. Parascandolo. J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto. Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, Power, F. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever,
Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, Wei, C. Weinmamn, A. Welihinda, P. Welinder, J. Weng. L. Weng. M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph. Gpt-4 technical report, 2023.
[35] C. Packer, S. Wooders, K. Lin, V. Fang, G. Patil, I. Stoica, and J. E. Gonzalez. Memgpt: Towards Ilms as operating systems, 2024.
[36] O. Press, M. Zhang, S. Min, L. Schmidt, N. Smith, and M. Lewis. Measuring and narrowing the compositionality gap in language models. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687-5711, Singapore, Dec. 2023. Association for Computational Linguisties. doi: 10. htt.://ww//22..nnnnnnsocee 378. URL https: / /aclaaccooooooooo ogy. org/2023. f. findi ngg -emiingssmmmmlmmmmmmm 378.
[37] M. Shao, B. Chen, S. Jancheska, B. Dolan-Gavitt, S. Garg, R. Karri, and M. Shafique. An empirical evaluation of Ilm for solving offensive security challenges, 2024.
12

---

[38] W. Shi, R. Xu, Y. Zhuang, Y. Yu, J. Zhang, H. Wu, Y. Zhu, J. Ho, C. Yang, and M. D. Wang. Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records, 2024.
[39] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
[40] D. Sobania, M. Briesch, C. Hanna, and Petke. An analysis of the automatic bug fixing performance of chatgpt, 2023.
[41] A. Sridhar, Lo, F. F. Xu, H. Zhu, and S. Zhou. Hierarchical prompting assists large language model on web navigation, 2023.
[42] T. Sumers, S. Yao, Narasimhan, and T. L. Griffiths. Cognitive architectures for language agents, 2023.
[43] X. Tang, A. Zou, Z. Zhang, Z. Li, Y. Zhao, X. Zhang, A. Cohan, and M. Gerstein. Medagents: Large language models collaborators for zero-shot medical reasoning, 2024.
[44] A. Thakur, G. Tsoukalas, Y. Wen, J. Xin, and Chaudhuri. An in-context learning agent for formal (ecorem-proving 2024.
[45] R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou, C.-C. Chang, I. Krivokon, W. Rusch, M. Pickett, P. Srinivasan, L. Man, K. Meier-Hellstern,
D. Lepikhin,
M. R. Morris, T. Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera- Arcas, C. Cui, M. Croak, E. Chi, and Q Le. Lamda: Language models for dialog applications. 2022.
[46] Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q Wang. Software testing with large language model: Survey, landscape, and vision, 2023.
[47] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang. Z. Chen, J. Tang, X. Chen, Lin, W. X. Zhao, Z. Wei, and J. Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), Mar. 2024. ISSN 2095-2236. doi:
URL http: //xx. dxx doi. dow. wttp://....00//////.... 000.000000/1110///00 01170/111144440 40231-1.
[48] X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji. Executable code actions elicit better Ilm agents, 2024.
[49] Z. Wang, G. Cuenca, S. Zhou, F. Xu, and G. Neubig. Mconala: benchmark for code generation from multiple natural languages, 2023.
[50] Z. Wang, S. Zhou, D. Fried, and G. Neubig. Execution-based evaluation for open-domain code generation, 2023.
[51] Z. Wang, D. Fried, and G. Neubig. Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks, 2024.
[52] M. Wornow, A. Narayan, K. Opsahl-Ong. Q. McIntyre, N. H. Shah and C. Re. Automating the enterprise with foundation models, 2024.
[53] Z. Wu. C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong. Os-copilot: Towards generalist computer agents with self-improvement, 2024.
[54] Z. Xi, Chen, X. Guo, W. He, Ding, B. Hong, M. Zhang, Wang. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, and Gui. The rise and potential of large language model based agents: A survey, 2023.
13

---

[55] C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 959-91 1591971, 2022.
[56] C. S. Xia, M. Paltenghi, L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models. In Proceedings of the 46th IEEF/ACM International Conference on Software Engineering, 2023.
[57] T. Xie, D. Zhang, J. Chen, X. Li, Zhao, R. Cao, T. J. Hua. Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024.
[58] A. Z. H. Yang, C. Le Goues, R. Martins, and V. Hellendoorn. Large language models for test-free fault localization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400702174. doi: 10.1 10.11455257750333223422 URL https://doi. org 15. 01111115555500 3623342.
[59] Yang, A. Prabhakar, K. R. Narasimhan, and S. Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: httpp:///ppeereiew... net tat.
[60] Yang, A. Prabhakar, S. Yao, K. Pei, and K. R. Narasimhan. Language agents hackers: Evaluating cybersecurity skills with capture the flag, In Multi-Agent Security Workshop@ NeurlPS'2 22, 2023.
[61] S. Yao, H. Chen, J. Yang, and K. Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023.
[62] S. Yao, J. Zhao, D. Yu, N. Du, Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, 2023. URL https:/ /owpnreveeeee..e//oo//o .nt / forum? i d=WE_vluYUL-
[63] P. Yin, W.DD Li, K. Xiao, A. Rao, Y. Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski, A. Polozov, and C. Sutton. Natural language to code generation in inter- active data science notebooks, 2022.
[64] H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, T. Xie, and Q. Wang. Codereval: A benchmark of pragmatic code generation with generative pre-trained mod- els. In International Conference on Software Engineering, 2023. URL https://api. semant cschol ar org/ CorpusTD: (D:.:u59:::55555113
[65] Z. Yu, X. Zhang, N. Shang, Y. Huang, C. Xu, Y. Zhao W. Hu, and Q. Yin. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. arXiv preprint arXiv:2312. 14187, 2023.
[66] E. Zelikman, Q Huang, G. Poesia, N. D. Goodman, and N. Haber. Parsel: Algorithmic reasoning with language models by composing decompositions, 2022. URL https: UtLht:ts:://aat http:pps////.x....raa../o abs/2212 11261...156...
[67] E. Zelikman, E. Lorch, L. Mackey, and A. T. Kalai. Self-taught optimizer (stop): Recursively self oeff-improving code generation, 2024.
[68] Zhang, B. Chen, Y. Zhang, Keung, Liu, D. Zan, Y. Mao, J.-G. Lou, and W. Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: / openreview..w.eceoc/eeeee net/ forum? iirrummmiig09yTY Cqh..
[69] Zhang, Zhang, J. Liu, I.. Song, C. Wang, R. Krishna, and Q Wu. Training language model agents without modifying language models, 2024.
14

---

[70] A. Zhou, K. Yan, M. Shlapentokh- Rolapentokh-Rothman. H. Wang, and Y.-X. Wang. Language agent tree search unifies reasoning acting and planning language models, 2023.
[71] Zhou, F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, Y. Bisk, D. Fried, U. Alon, and G. Neubig. Webarena: A realistic web environment for building autonomous agents, 2023.
15

---

Appendix
In the appendix, we provide additional analyses and more extensive discussions about SWE-agent, agent-computer interface (ACT) design, and model performance various evaluation benchmarks. We also provide several thorough case studies of SWE-agent behavior on select task instances. Data, code, and leaderboard at swe-agent.com
A SWE-agent Design
In this section, we go into greater discussion about the design methodology, appearance, and implementation of each of the SWE-agent components. As described in Section 3, the SWE-agent interface consists of several components that enable agents to accomplish key sub- tasks that are fundamental to solving software engineering problems. These are generally the following:
System Prompt
Describe envirenment and commands
Specify response format
Demonstration
Full trajectory of a successful example
Localization: Identify file(s)line(s) causing the issue.
Issue statement
2. Editing: Generate fixes addressing the given issue.
Give reported issue description
3. Testing: Write new scripts or modify existing test files to reproduce the issue and/or verify if fixes are correct.
Instructions to resolve issue
High-level strategy tips
To enable LM-based agents efficiently carry these individual functions and progress towards the overarch- ing goal of resolving codebase issue, we provide file viewer, file editor, search navigation system, and con- text management system. In Section A.I, we provide thorough breakdown of each of these components. In Sec- tion A.2, we discuss the technical design decisions and challenges of building SWE-agent. In Section A.3, we discuss how SWE-agent configured to support the final interface, along with how SWE-agent is built to enable easy extensibility and customization to alter the interface.
Thought & Action
Environment Response (collapsed)
Thought & Action
Environment Response (collapsed)
Thought & Action
Environment Response
Submit
Patch File
diff -ggit affrc/stt/luff//ute//// /tsr://stfturt//reee//00.. httec///tttutt/rrrte////de. b/stc/sslltuuff/ooee//66///
A.1 ACI Design
Figure 9: An overview over the structure of trajectory: We first present the sys- tem prompt, demonstration (optional), and issue statement. The agent then in- teracts in turn with the environment. Past observations may be collapsed, i.e. we truncate any long output, as described in Section 3.
In this section, we revisit each component discussed in Section 3. Per section, we first briefly review the compo- nent. We then discuss the underlying motivation for the component with respect to existing software tools. Finally, we note any additional thoughts that influenced the design process of the component with some occasional discus- sion of what aspects of the component heavily influence language model behavior.
For quick, text-free overview, comprehensive documen-
lation for all commands, their usage, and docstrings are included in Table 4. Figure 9 visualizes the message history for SWE-agent. Each prompt template is discussed thoroughly in Section C.
File viewer. As discussed in Section 3, the File Viewer is fundamental to a language agent's ability to understand file content and understand how different programmatic entities relate to one another. The File Viewer refers to interface that consists of the four commands, as shown in Table 4, and customized standard output for displaying n lines of a file at a time. Using the file viewer, an agent can look at lines of file at time and jump around the file. The File Viewer enables agents to perform fine- -raigrdi localization steps and also understand relationships between intra-file entities.
First, we discuss why existing software systems and graphical user interfaces are sub-optimal for LM use. Shell-only setting, there are several commands that can used to inspect file content. However, out of the box command line tools are sub-optimal or limiting for language agents for
16

---

Table 4: In additional the standard Linux Bash commands, we provide SWE-agent with specialized tools, including an interactive file viewer, search functionalities, and edit tools for the open file. Required arguments are enclosed <> and optional arguments are in []. The last column shows the documentation presented to the L.M
Category
Command
Documentation
File
open <path> <<line_nmberz
Opens the file at the given path in the editor. iine_number provided, the window will move to include that line.
viewer
goto -line_number
Moves the window to show mem_nu
scroll_down
Moves the window up 100 lines.
scroll_up up
Moves the window down 100 lines.
Search
search file <search <search_term> term> [<file>]
Searches for search. term in file. If file not provided. searches in the current open file.
tools
search_dit dir <search_terny [<dir>]
Searches for search_ters in all files in dir. If dir not provided, searches the current directory.
find file <file_name _aml>_name>>>> [<dir>]
Finds all files with the given name in dir. If dir is not provided, searches in the current directory.
File editing
edit <Π>:<m>
Replaces lines n through m (inclusive) with the given text in the open file. All of the replacement _eextttt_tt___ will be entered, so make sure your indentation is formatted properly. Python files will be checked for syntax errors after the edit. If error found, the edit will not be executed. Reading the error message and modifying your command is recommended as issuing the same command will return the same error.
<replacement_texx
end of edit
create <filename>
Creates and opens a new file with the given name.
Task
submit
Generates and submits the patch from all previ- ous edits and closes the shell.
several reasons. First, commands that print files to standard output (e.g. cat, print can easily flood language agent's context window with too much file content, the majority of which is usually irrelevant the issue. Enabling language agent to filter out distractions and focus on relevant code snippets is crucial to generating effective edits. While commands like head and tail 1 reduce length to the first/last n lines, it not intuitive use bash commands perform in-file navigation. It is either impossible or requires a long list of arguments to show specific file lines. Furthermore, since such Bash commands are stateless, "scrolling" up/down relative the current file position typically requires regenerating the same lengthy command with minor changes. Interactive tools like more and less accommodate this, but (1) representing navigation actions (multiple key up/down clicks) is intuitive for humans, but is verbose and costly for language agents, and (2) even if jumping to specific line number allowed, it is not possible to quickly identify what clases/methooss/ymbblk are declared in a file and then immediately go their definitions.
There are couple features of the File Viewer interface that make it friendlier and more operable than the Shell-only setting. First, the File Viewer standard output contextualizes code snippets with prepended line numbers and indicators of the number of lines above/below the current region. These details give more focused view of a file without compromising easy viewing of other parts of the codebase. This kind of file presentation also makes precise and consistent editing commands possible, discuss more thoroughly in the following section.
17

---

File Viewer
Find Files
open ateosphero.py
find_file atnosphoze.pp py
(255
more Lines above)
Found for
/pilib pulib pythen//vlp//
256:def
257:
puwyeareee_pettepp.or relatiioe
/pvlib...pvlie wwwppplyylooooo/ooeeeo
236:
Calculates prsciptiable water (en) from anssent sir terperature [c)
and relatively hunldity (t) using
Search Directory
apptosinataly otherates.
281:
search_dis dir APPARENT _ENITH_MODELS
2821
284:
205:
203:
The rodal wad developed sxpanding Eq. In [3]_:
wath::
python: httlll/prrhhondooo///////////eee
Found watches for *APPARETT SAOMENTT.NNTTHUPPPE 2O is /pclio...vlib
txt (1
Yeho.v
ratchea)
/pelih_..vlin wwllib-yythn/pp111 ptthen/ppllbb/tttppphr matches]
/pelib...wiibr httpi/ppttoo/ppp///////
(76 lines omitted)
Ennd of python
for DPPARBTTTENNTTI. HODELS" in pvittb_..ppiibb
9481
346:
345: def bied. www.tutttttetooooomo ood, tod_d6(000 ood630):
Search File
546:
Apuroxiauto broadhand corosol optioal depth.
search.file APPARENT NENITHLLODELLS
Bird and Helatros developed correlation breatband aecosol
Found astches for "APPARENT LEDEEE ppvit
390:
degth (400) using two mavelergtha, 500 and se0
python//wllb//tee wtt.ll//atosphseeppoo py:
351:
Line 32:APPARENT TEDECIILOOOLLLL + ('sirple', 'kasten1380"
952:
Pacoreters
Hastenyoungg900
353:
13:A199635 MODELS APPARENT ZENETH MOOELS
PRUF_7ENETHH_MOO "MP"ARENT MODELS" pwvlibb...viiie..
[undtless]
lines belon)
python/ httpn//vlt////e pwist//ttoospher.ppp
Figure 10: The File Viewer and Search components of the SWE-agent interface. The corresponding commands for each component are shown in blue. These examples are copied from trajectories generated by SWE-agent w/ GPT-4 Turbo on the pvlib
1 1603 task instance.
Another advantage of the File Viewer is that the commands are designed to be complementary and grounded in the File Viewer standard output. This saves the model from having to do repetitive or additional actions that unnecessarily increase the potential for error. As a concrete example, if an agent used a sed command to view the first 100 lines of a file and wants to look at the next 100 lines, it will have to recalculate parameters such as the start line and end line and reflect these updates correctly in the subsequent generation. As a rule of thumb, reducing the need for models to do this arithmetic by constructing actions and standard output that complement one another and build upon the effects of prior actions is highly preferable.
File editor. The File Editor, working in conjunction with the File Viewer, primarily refers to the edit command and the guardrails it enforces protect models against self-incurred cascading edit errors. Editing and testing are crucial to language agents' success on programming tasks, and a well-designed interface directly influences how well an agent's capabilities can be elicited. In other words, a bad interface undermines model performance.
As discussed in Section 3, editing can be very difficult in Shell-only setting. Built in commands (c.g., sed) often require a lengthy list of arguments, and the mis-specificattin of an argument can easily throw model off track as it attempts to correct self-incurred errors. We also observe that when agents use such commands directly, they struggle with the arithmetic skills required to generate an edit. Details such including the correct indentation level, inserting delimiters at specific points in line, and adhering to stylistic preferences of the codebase all require some amount of planning or calculation. Similar to the Shell-only file viewing process, file editing may also require repeating many commands. For instance, performing a multi-line edit can only be represented as multiple sed calls with requisite, delicate tweaks to the arguments for every turn. Furthermore, as referenced in Section 5.1, editing in Shell-only is usually a "silent" procedure. Confirming whether an edit succeeded and viewing its effects requires additional steps that can bloat the editing process with extra, needless commands.
The edit command, documented in Table 4, addresses the Shell-only failure modes by being grounded in the File Viewer standard output. The line numbers argument eliminates the need for any additional arithmetic, and the find-and-replace edit mechanism is format that existing models are more used to. With this functionality, agents can also perform multi-line edits in a single action.
Finally, as mentioned in Section 5.2, an important feature of the edit t command is that it does not apply changes which incur a linting error. fair and verified assumption we make when considering this feature is that the original codebase associated with each task instance is well-formed. In other
18

---

words, we assume that codebase maintainers will only push syntactically sound code that can be compiled successfully. When an agent issues an edit, is applied the codebase. Then, we run the following linting command (CURRENT_FTIA refers to the file that is currently open):
flake8 --isolated --seleect= F822, FB31, E1 11 E.1 E1 F113, E999, E902 "$CURRENT_FILE" 2>&1
The arguments for select are enror codes that refer syntax issues such as indentation. F821 and F822 indicate undefined names/symbols. F831 indicates a duplicate argument function definition. E111, E112, E113 are indentation errors. E999 denotes syntax error and E902 occurs if flake8 cannot read the source file.
If the edit does not introduce any of these errors, this command will produce output. The edit kept and the updated file content is shown using the File Viewer centered around the lines the edit occurred. If however the linting command produces output, which indicates the edit introduces syntax error, the edit is reverted. place of the file viewer, a message shown in Figure is displayed to the agent which shows the error that was caused, what the edit would have looked like, and the original file content. During the development process, we experimented with variations to this message, including the omission of one or more parts. Our takeaway was that having all three messages is helpful. Without the error type, the agent might misdiagnose what the mistake Without snippet of the changed file content, the agent will re-issue the same command more frequently. Without a snippet of the original file content, the agent has to attend to the same content from several turns ago; agents also sometimes generate ed t's with respect wrong, errant file content because is from a more recent turn.
Linting Error Message
Your proposed edit has introduced new syntax error (s) Please + understand the fixes and retry your edit commaand.
ERRORS:
F821 undefined name "orientactoonssrraeee
F821 undefined name vorientation_strate;
This is how edit would have looked if applied
[File: /pvlib pwt.bpppttiiiappppp.////aaaaaaa.
(1890 lines total)
(64 more lines above)
65:def basic_chain (times, latitude, longitude,
66:
module_parameters, temperaturee inverter_parameters,,
67:
68
irradiance-None, weather-None,
69:
surface_::1tt-None
10:
71:
solar
colax_positionn_ettteee numpy
numpy',
72:
cirmass_model= aat___/eeeeaaattttturuul1111""1
73:
altitude-None, pressure-None, **wargg))
74:
(1816
more 1ines below)
This is the original code before your edit
[File: /pvlib http:pippwwiibpppppppppaaapaapea
www.b//pwp..ttooo///////aaaaaaaa (1891 1ines total)
(64 more lines above)
65:def basic_char (times, latitude, longitude,
66:
module_parameters, remperanaeteeeeaeeeeaeeeeeeeeess
67:
inverter_parameterrs,
68:
irradlance-NNone, weather=Kone,
69:
surface sirfaceetil:=None, surface
surface.azimuthhee azimuth=None,
70:
orientation stt.aaiiioo_trrrteeeeeooooo t
71:
19

---

72:
sola solar__pssitionn_mtttddderrrr _umpp'', airmass
73:
74:
altitude-None, pressure=None, **kwargs):
75:
(1816 more lines below)
Your changes have NOT been applied. Please fix your edit command and " try again.
You either need to 1) Specify the correct start/end line arguments or Correct your edit code.
Do NOT re-run the same failed edit command. Running it again will Lead to the same error.
Figure 11: A linting error message. This is emitted if a model generates an edit command that introduces syntax error into the codebase. The error message shows the before and after of the proposed edit along with what eπrr messages were thrown. The problem with this edit is that it omits the orientat ctunnatioon. ser__siratttggoeeeeeeeeeseee field in its edit of the basic_chain chain method definition.
The editing guardrail has a drawback. To a certain degree, forces some edits to be done a particular order. For instance, in Figure 11, if the model's intention in fact to remove the argument, due to the SWE-agent editing guardrails, would have to remove all references from the function implementation either the same time in single action, or before removing from the method header split into two separate actions. For this particular scenario, the latter necessary because the file snippet is not large enough show the entirety of the basic chain implementation. This example highlights the trade-offs between the flexibility and guardrails of command. Deciding whether to introduce guardrail depends on how well it reduces common model errors compared to whether such restrictions hamper models' preferred workflows.
orientat connattio_streen wwtiiiinntttteeeee
basic_chain
Search & navigation. The File Viewer and File Editor together allow agents to make edits, write tests, and perform localization at file level. The Search & navigation module complements these capabilities by giving agents the tools to perform keyword-driven localization at both a directory level and file level.
As discussed, the main struggles with using built in Shell-only search commands such as grep and find are (1) given a general enough term, they are prone to producing too many search results that can consume an inordinate amount of space in the context window, and (2) they are highly configurable, making search result outcomes potentially inconsistent in appearance, The alternative to these search utilities is to navigate the file system directly with ed and look at what's 's in each folder with variations of 1s and cat: this kind of approach can take a large number of turns without yielding any particularly useful information.
Figure 10 visualizes the standard output for the three different search commands. The search_dil and find_ff fi ee helps agents perform directory level searches. The reason we provide two com- mands is due the kinds of keywords that are present in issue description (e.g.. class references, file names). The search rch_ficheeee file command allows agents to search for terms at file-level, which is helpful for efficient fine tre-ggraied localization. Taking a step back, the goal of these search commands is to make it easy for the agent to utilize any signal (c.g., line number, stack trace, natural language) about the root cause of an issue that may be present in the issue description or codebase. Once again, simpler command usage patterns with consistent output formats are easier for agents to use and reduces the chance for mistakes irrelevant outputs.
The main guardrail in place for all three search commands is curbing the number of search results 50 fewer. The downside that reporting an error forces the model generate another search query which can be an expensive operation. This reflects a trade-off' between keeping observations concise and making additional calls the base LM.
A.2 Implementation
The SWF-agent codebase generally composed of three modules: the environment, the agent, and the logging mechanism for saving task episodes into trajectories and patch generations.
20

---

Environment. The SWE-agent environment is heavily influenced by the InterCode library [59]. For the general pipeline of agent interactions with the environment, our work directly adopts InterCode's interactive coding task formulation. The environment integrates large parts of the interaction handling environment, which essentially the Shell-only setting referenced in
logic from the InterCode- the paper. As part of this adoption, SWE-agent also uses Docker containers to ensure reproducible and safe execution. Because of this, SWE-agent's 's infrastructure makes it easy for a user to swap out the Dockerfile (a domain specific language for defining container) to support other codebases and programming languages beyond the scope SWE-bench task instances. One difference that SWE-agent makes minor adjustments the underlying communication logic that transfers actions and observations between the Docker container and agent entity.
Agent Beyond serving as an agentic wrapper for facilitating multi-turn queries from an LM, the agent module defines the functions that render the ACI (e.g., context management, commands, interface logic, input/output format) and supports inference for closed/open, APl-based/local language models. The main workflow is to define an interface as class and/or of commands, which can then be specified via configuration file, discussed more thoroughly in Section A.3, The commands for the top performing SWE-agent with GPT configuration are shown in Table 4.
Logging. For each task episode, the main artifacts produced are the trajectory, which contains a history the interactions between the agent and environment, and the final patch generation, which can represents a summary of the changes proposed by the agent during the interaction. The patch generation can be used directly for SWE-bench [20] evaluation.
A.3
Configuration
The SWE-agent system is instantiated by three components: an LM, a SWE-bench style dataset or GitHub issue, and a configuration file. The configuration file serves to specify the design of the ACI. Iteratively refining the configuration file is the main way we achieved better agent performance and carried out different analyses for the main paper. In this section, we will present thorough review of what a SWE-agent configuration file looks like.
An agent-computer interface is generally made up of four categories of configurable components:
1. Prompt templates:
These prompt templates are used to inform the language model of the task setting, show the list of available commands, augment environment responses with the values of state variables, and provide the initial task setting.
2. Command files: These files contain the source code of bash or Python functions and scripts. Commands are easily modified, added, and removed through manipulating these files' code contents directly. Documentation added in these files can also be injected into prompts to inform the model the available commands.
3. Control flow: Methods for parsing model responses and processing history can be specified through these configuration arguments.
4. Environment variables: Initial values variables that may interact with commands and the shell can also be specified in the configuration.
In the following Figure 12, we include an annotated example of the contents of configuration file.
Configuration (yaml)
# Prompt Teaplates: Control how observations of environment are shown to agent
systen_template:
First "system" message shown to agent
Instance_template. |-
Instance prompt, contains task instance-speci content next next_step_tet wwwpstteeppppppe teep_teeplate:
Format zemplate of per-turn observation (Contains standard output * from agent's action)
coxt_sttpp_ooooopppttttooopppppp |-
21

---

Format template of observation when there standard output 4 from the agent's action
1-
Format template of error message (Used when agent 's action causes error)
demonstracioontemplaaae
Format template for showing a demonstration to the agent
demonstrations:
folder>/s.tra)
File is demonstration of how to solve task. This could agent generated trajectory,
You can include 1+ demonstracions
+ Rnyment States: Define Features of the SWEEnv envi ronment env mnv_variables:
+ Default variables for SWEEnV at the beginning of each instance (URRENT_FIIL 0
CURRENT_ LINE:
OVERLAP:
SEARCH_FILES:
SEARCH_INDEX:
SEARCH_RESULIS:
LANDOW_SI2E,
SPART_INDEL.
END END_INDEX:
START CURSORC
SND_CUROSR:
START_ CURS__CCRRRRS.C MARK:
END_CURSOR MARK:
staate_comand:
#
allows you to update state variables to reflect any aspect of the environment (e.g, current working directory)
code:
state() I echo ("pwd": "'SPWD!")"
$ Action Interface: Define how agent interacts with the SWEEnV
+ cavironment
command_files:
path/to/bash_file.et
Each file contains list of commands implemented in bash
You include 1+ command siles
parse_conmante Reference to functionality for aetining command documentation
+
history_processor: Reference to functionality for controlling agent 's message history
parse_function. Farser run on agent output
Figure 12: An example the configuration file that defines the SWE-agent ACI. A configuration is represented as single yaml file, allowing you to define the commands that agents may use, write prompts shown to the agent over the course of a single trajectory, and control the input/output interface that sits between the agent and environment.
The prompt templates are explained in detail in Section C. The environment variables and command files work in tandem; environment variables make the interfaces stateful, and when commands are invoked, the corresponding states are updated to reflect the changes the environment and interface. The parse_command, parse fone_ffuccctee ion, and implementations declared within the agent module. The parse_commant file describes how command documentation should be presented to the agent. The the input/output formats for the agent. The h cittor__processee points to the logie for controlling and modifying the message history enforeed each turn throughout single task episode.
all reference
i in is what enforces
22

---

The configuration- based workflow of SWE-agent makes it easy to test new ACIs by incorporating novel commands, input/output formats, context managers, and more into the existing codebase. In the following subsections, we showcase existing implementations of several of these components and discuss how they can be extended.
Commands. We describe how to implement your own commands for the SWE-agent ACI. As shown in the above Figure 12, commands are declared as list of one or more file paths in the command_ files argument. Individual commands must be declared as separate functions in PY or sh files. Every command subscribes to the following skeleton code in Figure 13.
Command Skeleton Code
$ @yaml
4 signature: [command] [argument (s)
$ docstring: [Brief description of what your command does,
4 argumenta:
4 [argument name]:
type: [type (i.e. integer, string)]
4
description: [Brief description of this argument)
# required: [true false]
# [azgument name]:
[cmmandd] () {
A Implementation here
The skeleton code for defining command that can be accessed in the SWE-agent ACI. The function definition includes both the underlying implementation along with several arguments that describe how to use the command, which is compiled into the System template's 's command documentation run time.
Figure 13:
The choice of Python or Bash based implementations of commands means they can be written to do wide variety of actions, and the use of Docker means that the commands and system can be co-designed. Here is list of guidelines around how to implement commands correctly.
Command arguments can be referenced via positional parameters notation (i.e. $1).
If there are arguments, omit the arguments section.
The implementation for your command is unconstrained. There are no limitations on the form of the underlying command code.
The minimal documentation requirements are signature and docstring.
Global variables can be used make stateful changes to the environment. For instance, for the commands associated with the File Viewer, you'll ll see we define the CURRENT_L.IN LINE variable for the file viewer. This variable is modified across multiple commands, including
open, goto, scroll_in
scroll. seroll_down, and edit.
Third party libraries can be freely imported and used by commands (e.g., flake8).
To show effects of command, print to standard output (e.g., with echo). The command parsing logic implemented such that does not look for return value.
Once the file path containing the command is added to command_docs command immediately available use subsequent task episodes. Including a that uses more complicated commands can be helpful to showcase proper use and may increase the frequency with which the agent uses the command.
an argument, the demonstration
Input/output format. The input/output format defines what a correctly formatted response for an agent should look like. Selecting suitable format greatly affects how well agents can interact with the environment. The methods for communicating and enforcing the input/output format are separated across several arguments. In Figure 12, the value of parse arse_functt_ funct ion should point class definition that enforces the format and actually parses the agent's responses. Informing the agent of expectations around the input/output format should take place system_temp) ate, and the agent can be reminded of these standards via the format_ers _eroor serror_temp) ate.. New input/output formats can be easily devised and enforced by updating these arguments to point to new class or display different natural language instructions.
23

---

Context management. Context management is implemented as a class within the agent module. The history pww.eessoppcssrrrrooooooo argument allows one specify which context manager use via the configuration file. Underneath the hood, the context manager is invoked per turn of the interactive loop. From the entire recorded history of the agent's interactions so far, the context manager constructs the literal history to be fed to the agent to invoke the next response. The general design of wwtooryyppocssrrrroo allows for easy experimentation towards more sophisticated strategies for managing history.
24

---

B Extended Results
In this section, we provide additional results, including performance marginalized against differ- ent dimensions, patch generation statistics, and problem solving patterns reflected by SWE-agent trajectories. Per analysis, we provide numerical or qualitative evidence that supports our findings, describe our takeaways from each finding, and discuss both the strengths of SWE- relative to prior baselines along with future directions based on improving common failure modes.
B.1 Hyperparameter Sweep
We performed hyperparameter sweep using a subset of 37 instances sampled randomly from the dev split of SWE-bench. We present the results in Table where we perform the sweeps for both the GPT-4 Turbo and Claude 3 Opus models. For GPT-4 Turbo the best configuration has % Resolved rate of 15.1%, with a temperature of 0.0, window length of 100 and history set to last five observations (described in $3). There is three way tie for Claude 3 Opus between the aforementioned configuration along with two additional settings Ceemperature/Window/HHte 0. 0.2/100/Last- and 0. 0.2/200/Full). We elect to run inference of both models on the SWE-bench test sets (both full and Lite splits) using the 10/100/Laas configuration.
of
Table 5: Hyper parameter sweep results on a subset of the SWE-bench dev split. % Resolved shows the mean score across 5 samples.
Model
Temperature
Window
History
% Resolved
GPT-4 Turbo
0.0
100
Full
14.1
GPT-4 Turbo
0.0
100
Last Obs.
15.1
GPT-4 Turbo
0.0
200
Full
9.2
GPT-4 Turbo
0.0
200
Last Obs.
10.8
GPT-4 Turbo
0.2
100
Full
10.8
GPT-4 Turbo
0.2
100
Last Obs.
12.4
GPT-4 Turbo
0.2
200
Full
8.7
GPT-4 Turbo
0.2
200
Last Obs.
10.8
Claude 3 Opus
0.0
100
Full
5.4
Claude 3 Opus
0.0
100
Last Obs.
8.1
Claude 3 Opus
0.0
200
Full
7.0
Claude Opus
0.0
200
Last Obs.
7.1
Claude 3 Opus
0.2
100
Full
7.4
Claude 3 Opus
0.2
100
Last Obs.
8.1
Claude 3 Opus
0.2
200
Full
8.1
Claude 3 Opus
0.2
200
Last Obs.
B.2 Model Performance
We present analyses of model performance marginalized across different dimensions and categories.
Performance by Repository. We include breakdown of model performance by repository the SWE-bench Lite dataset in Table 6. We also include and adjust the performance of Claude on SWE-bench, inherited from the baseline performances established in the original work. As presented above, SWF-agent performance is superior to prior approaches, solving not only a higher percentage of problems across repositories, but also resolving problems in repositories that previously nearly or completely unsolved by prior retrieval augmented generation baselines used in the original SWE-bench work (e.g. matplotlib, sympylsympy).
Temporal Analysis. In Table 7, we provide a temporal breakdown that shows the % Resolved statistics for task instances from different years. There is clear correlation between task instance's
http://hhhh...ommmmmmmooliaaaaaa https///ithub..oo///my/y/yy/
25

---

Table 6: Resolved performance across repositories represented in the SWE-bench Lite dataset.
Each row corresponds to repository while each column is the model's 's performance for that repository. The numbers in parentheses in the "Repo" column is the number of task instances SWE-bbenh Lite that are from the corresponding repository.
SWE-agent
RAG
Repo
GPT
Claude 3 Opus
GPT
Claude Opus
Claude
astropy/astropp (6)
16.67%
33.33%
0.00%
0.00%
0.00%
django/django (114)
26.32%
16.67%
4.39%
6.14%
5.26%
mupplotiib/matpllttib (23)
13.04%
13.04%
0.00%
0.00%
0.00%
mwaskom/seabom (4)
25.00%
0.00%
25.00%
25.00%
0.00%
pallets/flask (3)
0.00%
0.00%
0.00%
0.00%
0.00%
psf/requests (6)
33.33%
16.67%
0.00%
0.00%
0.00%
pydata/xarray (5)
0.00%
0.00%
20.00%
20.00%
0.00%
pylint-dew/pylint (6)
16.67%
0.00%
0.00%
0.00%
0.00%
pytest-dew/pytest (17)
17.65%
5.88%
0.00%
5.88%
5.88%
scikitlleam/sciiii-llamm (23)
17.39%
17.39%
0.00%
4.35%
8.70%
sphinx-doc/sphinx (16)
6.25%
6.25%
0.00%
0.00%
0.00%
sympylsympy (77)
10.39%
5.19%
1.30%
2.60%
0.00%
Table 7: % Resolved performance for task instances from different years represented in the SWE- bench Lite dataset. Each row corresponds to a year while each column is the model's performance for task instances with created_as timestamp from that year. The numbers in parentheses in the Year column the number of task instances in SWE-bench Lite from that corresponding year.
SWE-agent
RAG
Year
GPT 4
Claude 3 Opus
GPT 4
Claude 3 Opus
Claude 2
2023 (30)
23.33%
13.33%
3.33%
3.33%
0.0%
2022 (57)
21.05%
17.54%
5.26%
7.02%
1.75%
2021 (42)
23.81%
11.90%
2.38%
4.76%
2.38%
2020 (66)
10.61%
7.58%
3,03%
1.52%
1.52%
Before 2020 (105)
17.14%
10.48%
0.95%
4.76%
5.71%
creation year and its resolution rate across either models setting, For instance, while the SWE-agent w/ GPT-4 approach solves the highest percentage of problems from 2021, while the RAG w/ GPT-4 and SWE-agent w/ Claude 3 Opus approaches perform better task instances from 2022.
B.3 Trajectory Analysis
We present additional characterizations of trajectories corresponding to task instances that were successfully resolved by SWE-agent w/ GPT-4 Turbo (unless otherwise specified).
B.3.1 Turns to Resolution
Figure 14 visualizes the distribution of the number of turns SWE-agent needed to complete task instances that were successfully resolved. On the full SWE-bench test set, SWE-agent w/ GPT-4 takes an average of 14.71 turns to finish trajectory, with median of 12 turns and 75% trajectories being completed within 18 turns. On the Lite split of the SWE-bench test set, SWE-agent w/ Claude 3 Opus takes an average of 12.71 turns to finish a trajectory. with median of 13 turns and 75% of trajectories being completed within 15 turns. From the distribution, is evident that across models and SWE-bende bench splits, the majority of task instances are typically solved and completed comfortably within the allotted budget.
This also points to a general area of improvement for language agent systems - if language agent's initial problem solving approach, typically reflected in the first 10 to 20 turns, does not yield good
26

---

Resolved by Turn (GPT 4, Full)
Resolved by Turn (Claude Opus, Full)
30-
25-
40-
20.
Count
15
20-
10
10 15
20
25
30
35
40
10
15 Tum
20
25
Turn
Figure 14: Distribution of the number of turns for interactive trajectories corresponding solved task instances on SWE-bench. The left histogram shows this distribution for SWE-agent w/ GPT 4 on the full SWE-bench test set (286 trajectories). The right histogram is the performance of SWE-agent w/ Claude 3 Opus on the Lite split of the SWE-bench test set (35 trajectories).
Submitted Prediction Total Steps by Resolution
Submitted Prediction Cost by Resolution
Romhaz
The
Ttc
100
Fiequency
Frequency
000
30
08.5
1.00
Total Steps
Cost (USD)
Figure 15: The distribution of agent trajectories by total steps (left) and (right) for SWE-agent with GPT-4 Turbo on SWE-bench. The distributions of resolved instances are shown in orange and unresolved are shown in blue. Resolved instances clearly display an earlier mean and fewer proportion of trajectories with many steps or that cost near the maximum budget of $4.00.
solution, struggles to make use later turns that build upon past mistakes. To remedy this issue and induce stronger eиrоr recovery capabilities in language agents, future directions could consider improving either the model, the ACI, or both.
B.3.2 Walkthrough of Trajectory Phases
We deseribe what happens in different phases of agent's problem solving trajectory. To support our observations, we present several tables and distributions that help highlight consistent trends.
Initial reproduction, localization steps. First, the initial steps that SWE-agent usually takes is heavily dominated by Localization and Reproduction operations. The most commonly occurring pattern in general is the create, edi t, python triplet. Across these commands, agent creates an empty python file, adds an executable code snippet via edit, and then attempts to run it. As an alternative, the agent also sometimes decides to start off instead with Localization, identifying the files/lines causing the issue. Depending on how informative the issue description and results for initial search queries are, agents will run additional search queries with finer grained search tools to zoom in on the target problematic code area (e.g., search_daaaeeeeee open, search_faleeseeeeeeeeeeeeaeeseee www.ffeerelllddoooooooo
These trends are also reflected in Figure 16, which shows distribution of patterns across turns according to the categories defined in Table 8. The three leftmost bars reflect that Reproduction followed by Localization constitutes the lion's share of operations that occur in the early phases of trajectory. For more thorough breakdown, we also include Figure 17, which shows an estimated distribution of each action with respect different turns, normalized across the total number of times
27

---

Table 8: We present a table of the most frequently occurring action patterns each turn ("frequently means 2 times) trajectories task instances resolved by SWE-agent w/ GPT-4. For instance, the pattern create,ed t.python appears 156 times at the first to third turns. In addition, we also manually assign each entry a category (Reproduction, Localization (File), Localization (Line), Editing, Submission) that generally captures the underlying purpose of such pattern. "Reproduction" refers to the sub-task of recreating the error or request described by the issue. "Localization refers to the sub-task of identifying the code that is the cause of the issue.
Turns
Pattern
Count
Category
1-3
create, edit. python
156
Reproduction
1-3
search _irh open, search file
21
Localization (File)
1-3
search_die _irc open, scroll_down
12
Localization (Line)
1-3
create, edit, edit
11
Reproduction
1-3
search_di open, edit
10
Localization (Line)
2-4
edit, python, find_file
71
Localization (File)
2-4
edit, edit edit
37
Reproduction
2-4
edit,python search_dir
26
Localization (File)
2-4
open
15
Localization (File)
2-4
open, edit, edit
13
Editing
24
open, edit, create
13
Editing
2-4
open, scroll_ scroll_down
9
Localization (Line)
244
open, scroll_ edit
5
Editing
244
open, edit, submitt
5
Submission
3-5
python, find file, open
61
Localization (File)
3-5
python, edit, python
25
Editing
3-5
search_file, goto, edit
24
Localization (Line)
3-5
python, search_dir open
23
Localization (File)
3-5
edit, create, edit python, edit.
13
Editing
3-5
11
Editing
3-5
python, open. edit
7
Editing
3-5
python, find_file, file_file
7
Localization (File)
3-5
edit, edit, submit
4
Submission
3-5
edit, edit, create
4
Editing
4-6
find_file, open, edit
28
Editing
46
find_file, open, search file
19
Localization (Line)
4-6
edit, pythn edit, edit find_file, open, goto
11
Reproduction
4-6
8
Editing
4-6
8
Localization (Line)
4-6
goto, edit, submit
7
Submission
4-6
goto, edit, create
7
Editing
4-6
find_file, open, scroll down
6
Localization (Line)
46
scroll_down, scrol__ddwn edit
5
Localization (Line)
4-6
find file, open
5
Localization (File)
5-7
open, beearhh_file, goto
29
Localization (Line)
5-7
open, python
20
Editing
5-7
open, goto, edit
7
Editing
5-7
scroll_down, edit, submit
4
Submission
6-8
scroll_down (x3)
6
Localization (Line)
6-8
search_file, goto, scroll scrool__wwoo
4
Localization (Line)
7-9
edit, pcito,ython.
20
Editing
7-9
goto, python
12
Editing
8-10
python, rm, submit
19
Submission
8-10
search_ goto, search_ file
4
Localization (File)
9-11
edit (x3)
18
Editing
9-11
edit, open, edit
Editing
9-11
goto, search_faiiieeeeeeeeeeeeeeeeeses file goto
4
Localization (Line)
28

---

Freq. of Pattern Categories per Turn
Localization (Line)
200
Localization (File)
Frequency
150
Editing
Reproduction
100
Submission
50
1-3
2-4
3-5
4-6
5-7
6-8
7-9
8-10
9.11
10-12
11-13
12-14
Turns
Figure 16: We assign each pattern to one of five categories (as presented in Table 8) and present histogram the turns at which patterns from specific categories show up frequently.
the command occurs across all turns, From these graphs, we can see that create is invoked much more frequently the very first turn than any other turn. The search r and search_ file distributions are roughly bi-modal, with peak of occurrences for both actions showing up in Turn (if the agent decides to do Localization immediately) and the Turn 4 (if the agent decides to do Localization after Reproduction). We also present Figure 18, which communicates similar information as Figure 17, but presented instead as a stacked bar chart with more commands. The chart is created directly from Figure 7, with the frequency of actions at each turn normalized across the total number of trajectories with a length greater than or equal to n turns.
Density Plots of Actions across Turns
0.25
search dir
search file find_file
0.20
create
edit
Density
0.15
exit_cost
submit
python
0.10
0.05
0.00
5
10
20
25
30
35
Turn
Figure 17: This density plot shows a normalized distribution of actions across different turns of trajectory. exit_cose refers to when the token budget cost was exhausted and the episode's changes are automatically submitted (contrary to an intentional submit invoked by the agent).
Cycle of edit, then evaluate. From the fifth turn onwards, the distribution of actions per turn can be generally described as alternating edi t and pyt hython/yytett actions. After reproducing the issue and localizing the file(s) responsible for the problem, agents will typically make edits to the
29

---

file, then run the reproduction script existing tests to check whether the proposed edits resolve the original issue and maintain existing desirable behavior. This pair of actions will often repeat for several turns, as initial edit usually does not successfully resolve the given issue. Multiple rounds of editing that are supplemented by execution feedhack from prior turns are conducive to more well-formed, successful subsequent edits. As reflected in Table 8, for turn 4 onwards, the most popular pattern that begins at each turn usually falls under the Editing category. This is also made obvious by Figure 18, where the edit command is the most popular command for Turns 5 to 31, with only one exception (Turn 30). From Figure 17, it is also notably that the distributions of the edit and python commands quite similar, they typically follow one another.
Distribution of Actions per Turn (Normalized)
1.0
seurch dir
find
0.8
find
create
Frequency
edit
0.6
exit_cost
suhmit
open
scroll
scroll
goto
python3
python
0.0
pylest
Il
12
13
21
22-
23
25
27
28-
29
30
31
92
33
14
35
36
Tum
Figure 18: A normalized view of Figure The distributions for turn n are normalized across the number of trajectories that have length of at least more turns.
Interspersed across these later turns are additional Localization operations for inspecting other parts of the current file (e.g., scroll down, scroll_us __pp) or opening other files (e.g., open, search_dir/Ell dir/file, find file). These minor trend lines reflect the tasks that involve multi- line or multi-file edits. Figure 18 displays a steady presence of such actions from Turn onwards. Agents will invoke such actions to read different parts (e.g., documentation, implementation) of long function, especially when it does not entirely within the file viewer's number of lines. After editing one function rumning the reproduction script will occasionally propagate eπror in different function B, where function B invokes A. This is a common reason for the additional directory and file level navigation that occurs in the later stages of trajectory.
Concluding submission turns. There a consistent proportion of submit actions per turn, with a relative peak around Turn 10, as shown in Figure 17. As mentioned in Section 5.2 and above, the majority of resolved task instances end with an intentional s submit command. As suggested by both Figure 15 and Figure 18, submissions are concentrated between Turns 10 and 20, becoming less frequent for each turn beyond this range. This trend reflects how agents struggle to use later turns to their advantage, particularly when the original problem solving approach fails, which is fairly evident by Turn 20. Effectively utilizing later turns to either remedy multiple prior errors or pivot to different problem solving approach are all viable strategies given the 20+ turns that remain. However, due to overwhelming context or greedy tendencies, agents do not reflect such dynamic behavior, instead opting to focus on continued local editing rather than additional exploration.
Finally, there is sharp cut off of exit_ cost actions scattered throughout Turns to 40; this reflects that the $4 cost limit we impose on runs roughly corresponds to this number of turns. The discrepancies mainly comes from variations in the size of observations, with trajectories containing multiple observations that have high number of tokens corresponding ones that terminate relatively earlier. Increasing the cost allowance per task episode would directly increase the maximum number of the turns per episode.
30

---

B.3.3 Breakdowns of Action Sequences
In this part, we include more granular examinations of patterns of actions that emerge frequently in trajectories. We also identify consistent associations between groups of actions and how their effects build off one another across several turns.
Editing Trends. Editing is core facet of agents' ability to reproduce issues and propose fixes effectively. It is also the action that models typically struggle with the most. Here, we list several trends we were able to discern about how agents edit.
First, across the full SWE-bench meaning the edit invocation raises linting error. Going forwards, we refer to such an occurrence as a failed edit. Out of 2, 2,294 task instances, 1,185 (51.7%) have at least one turn with an failed edit. Of these trajectories, there is a median of 3 failed edits per trajectory, with a max of 33. The rate of failed edits is smaller for resolved task instances. Out of 286 resolved instances, 113 (31.5%) have at least one turn with an failed edit, with a median/mean/man /mx of 2 failed edits per trajectory, with max of 26. Figure 19 shows corresponding distributions.
set, a non-trivial minority of edit actions are unsuccessful,
Failed Edits per Trajectory (All)
Failed Edits per Trajectory (Resolved)
175
1000
150
800
125
Count
Count
600
100
75
400
50
200
0
0
0
5
10
15
20
25
30
35
40
0
5
10
15
20
25
30 35
40
Num. Failed Edits
Num. Failed Edits
Figure 19: Distribution of the number of failed edit actions per trajectory across all (left) and resolved (right) task instances by SWE-agent with GPT-4 Turbo. "failed" edit refers an edit action that raised a linting error. The left-most bar for both graphs corresponds to the number of trajectories with no failed edits.
Second, with linting enabled editing, agents "recover" more often than not from failed edits. To understand whether and how effectively agents use linting error feedback construct a subsequent, well-formed edit action, we define two terms. Recovery refers to a sequence of failed edits followed immediately by successful edit, suggesting the agent used linting feedback to make well-formatted edit. An unsuccessful recovery is consecutive failed edits followed immediately by non-edit action.
Chance of Successful Edit Given 2 Failed Edits
Probability
0.2
0.0
10
19
of Consccutive Failed Edits
Figure 20: Probability of successful edit after n failed edits. The likelihood of recovery decreases as increases.
31

---

Across trajectories corresponding to resolved task instances, there are 135 occurrences of 1+ failed edit attempts. Out of these, the agent recovers successfully 104 times. The number of consecutive failed edit attempts before a successful versus failed recovery is also vastly different. Successful recoveries are usually preceded by 2.03 edit attempts, less than the average 4.22 failed edit attempts of unsuccessful recoveries. Across all task instances, the relative rate of unsuccessful recoveries increases, with 810 successful recoveries versus 555 unsuccessful ones. While the number of consecutive failed edit attempts resulting in a recovery remains steady (2.2), it increases significantly for unsuccessful recoveries (5.59).
Third, the odds of recovery decreases as the agent accumulates more failed edit attempts. Figure 20 displays a line plot of the probability of successful edit given n failed edit attempts a row. The leftmost data point of n = 0 means that any attempt at editing has a 90.5% chance of eventually being successful. This value drops off once the agent incurs single failed edit; there is only a 57. 57.2% chance the edit is ultimately successful. In other words, there is a 42.8% chance the agent never recovers upon encountering edit error.
Action sequence analysis. We calculate the transition probabilities showing the likelihood of the next action given the previous n actions. To perform this analysis, we first determine the 15 most commonly occurring sequences of actions, for n E {1, 2, 3, 43. We then count how frequently each command appears after this sequence and finally normalize the counts across the total number of occurrences of the sequence to get a likelihood of the "Next Action" with respect to the preceding sequence of actions.
We show these transition probability heatmaps, with n = in Figure 21, n = 2 in Figure 22, n = 3 in Figure 23, and 4 in Figure 24. From these graphs, it is immediately obvious that several action sequences emerge consistently across many task instances. The high likelihood cells in these heatmaps suggest that SWE-agent uses common problem solving patterns which correspond higher order operations such as reproducing an issue, localizing buggy code, and proposing/verifying edits.
In Figure 21, we see direct associations between pairs of actions. There several obvious trends. All trajectories begin with create, find_ foddd_ilee, file, search _dirr, and end on either a submit t or exit_ __osst. The most popular next action is edit; it is the most likely action to follow create, edit, goto, pytest, and python. Scroll (e.g., scrol 1 ddddw/dowppppp up) and search (e.g. find file, sea seaarch_dit actions tend to be repeated.
Other interesting correlations are also present. The edit/evaluate pattern is reflected in the correlation between the edit and python pair. A variety of localization patterns are also conspicuous. Sometimes, searching for file turns out to be less fruitful than searching for a keyword, and visa pair. The invocation of open is
versa. This is reflected in the find file and search_ representative of an agent honing in on a specific file to then continue localizing (search file 0.35, scroll_ ddow 0.18, goto 0.09) begin editing (edit 0.25).
As the number of prior actions considered increases. more complex operations carried across multiple commands become apparent, echoing the observations from Table 8. In Figure 23, reproduction (e.g. [create, edi tt python]) is typically followed by adjustments to the script (edit it 0.39) or localization fiind_fieee 0.31, search_de dir 0.22). Fruitful localization patterns are once again reflected by ffind_ffle / deir, open, search_filee are followed by goto. In Figure 24, the most popular 4-grams are related to reproduction or editing. The [edit, python, rm, submit] t pattern is a popular way for trajectories to finish. Common failure modes are also apparent; repeated actions like edit (4x) and scroll down (4x) typically continues cascading.
B.4 Failure Modes
In this section, we provide insight on categorizing common agent failure modes. We perform an automated analysis of the unresolved trajectories (n 248) from the SWE-bench Lite split with default configuration. We first create a list of possible failure categories based on model behavior analyzed in Sections B.3.2, which are described in full detail in Table 9. A validation set of 15 instances are then sampled from the 248 instances left marked unresolved by SWE-agent and the authors hand-label them according to these categories. Finally, we combine the agent's trajectory with the patch generated by its changes and the gold patch for reference and use an LM to categorize
We use 4002044-2220---0 from OpenAL.
32

---

Transition Probabilities Heatmap
START>
0.65
0.00
0.00
0.16
0.00
0.01
0.00
0.00
0.00 0.00 0.19 0.00 0.00
0.00
1.0
create
0.01
0.98
0.00
0.00
0.00
0.00
0.00
0.00
0.00 0.00 0.00 0.00
0.00
0.00
1959
edit
0.03
0.02
0.01
0.01
0.03
0.00
0.00
0.01
0.01
0.00
4745
0.8
exit
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
2851
find file
0.01
0.01
0.00
0.21
0.00
0.62
0.00
0.00
0.00
0.00
0.13
0.00
0.00
0.00
3335
Current Action
goto
0.01
0.39
0.02
0.02
0.13
0.05
0.00
0.00
0.17
0.04
0.01
0.14
0.01
0.00
15341
0.6
open
0.01
0.01
0.03
0.06
0.00
0.01
0.18
0.00
0:35
0.00
0.00
pytest
0.11
0.30
0.04
0.12
0.05
0.14
0.05
0.02
0.00
0.00
0.11
0.04
0.02
0.00
4417
python
0.02
0.01
0.15
0.13
0.00
0.01
0.00
0.00
0.11
0.01
0.04
0.00
678
0.4
scroll down
0.01
0.02
0.02
0.04
0.00
0.00
0:61
0.02
0.02
0.04
0.01
0.00
1587
scroll
0.00
0.16
0.02
0.01
0.11
0.01
0.00
0.00
0.08
0.53
0,02
0.05
0.01
0.00
23.58
0.2
search dirch__iir-0
0.01
0.01
0.00
0.09
0.00
0.73
0.00
0.00
0.00
0.00
0.16
0.01
0.00
0.00
610
scarch filc
0.00
0.01
0.05
0.06
0.00
0.01
0.12
3888
submit
-ommitt-.000
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
56
-0.0
create
edit
exit, cost
find file
golo
open
Pytest
python
scroll down
screll up
search dir
search file
submit
<END>
Next Action
Figure 21: Heatmap displaying the relative frequency of different actions being invoked after the
most popular actions SWE-agent w/ GPT-4 Turbo trajectories across all task instances.
Transition Probabilities Heatmap
START>> create 0.00
1384
1.0
create, edit
1658
edit (2x)
0.00
0.02
0.83
0.00
0.91
0.02
0.80
0.15
0.01
0.01
1880
edit, python
0.00
0.02
0.48
0.01
0.15
0.03
0.12
0.00
0.01
0.00
0.00
0.12
0.01
0.04
1278
0.8
find file, open
0.00
0.02
0.19
0.00
0.04
0.08
0.04
0.00
0.00
0.18
0.01
0.01
0.42
0.00
7777
Previous 2 Actions
goto, edit.
0.00
0.06
0.38
0.01
0.01
0.05
0.05
0.00
0.51
0.01
0.01
0.01
0.02
0.10
1312
-0.6
open, edit
0.00
0.06
0.32
0.01
0.02
0.01
0.08
0.01
0.41
0.00
0.00
0.02
0.03
0.06
1438
open, scroll down
0.00
0.00
0.19
0.01
0.02
0.05
0.07
0.00
0.00
0.57
0.02
0.02
0.03
0.00
22.17
open, search file
0.00
0.00
0.02
0.01
0.02
0.74
0.05
0.00
0.00
0.01
0.00
0.02
0.11
0.00
-4298
-0.4
python, edit
0.00
0.00
0.34
0.01
0.91
0.90
0.05
0.88
0.57
0.00
0.00
0.01
0.00
0.01
2074
python, find file
0.00
0.00
0.00
0.18
0.00
0.00
0.00
0.00
0.00
0.09
0.00
0.00
825
submit
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
-1199
0.2
scroll down (2x)
0.00 0.01
0.02
0.05
0.00
0.00
0.02
0.04
-662
search dir, open
600
search file, goto
0.00
2334
-0.0
EED
create
edit
exit cost
find file
golo
open
pytest
python
scroll down
search dit
scarch file
submit
scroll.
Next Action
Figure 22: Heatmap displaying the relative frequency different actions being invoked after the most popular pairs of actions in SWE-agent w/ GPT4 Turbo trajectories across all task instances.
33

---

Transition Probabilities Hcatmap
<START>, create, edit
0.04
0.11
0,00
0.01
0.00
0.00
0.00
0.82
0.00
0.00
0.01
0.00
0.00
729
create, edit, python
0.03
0.36
0.00
0.28
0.00
0.09
0.01
0.01
0.00
0.00
0.20
0.00
0.02
1223
edit (2x), python
0.57
0.01
0.09
0.03
0.11
0.00
0.00
0.00
0.00
0.08
0.02
0.05
edit (3x)
0.01
0.82
0.04
0.00
0.01
0.01
0.00
0.07
0.00
0.00
0.00
0.00
0.02
5614
edit, python, edit-
556
Previous 3 Actions
edit, python, find file-
1401
0.6
edit, python, open 0.00
0:44
0.02
0.02
0.24
0.06
0.00
0.00
0.11
0.01
0.01
0.08
0.01
971
edit, python, rm 0.00
0.00
0.01
0.00
0.00
0.02
0.00
0.01
0.00
0.00
0.00
0.00
0.95
2048
find file, open, scarch file- 0.00
0.01
0.001
0.05
0.71
0.02
0.00
0.00
0.02
0.00
0.02
0.15
0.00
703
0.4
open, search_file goto 0.01
0.39
0.01
0.02
0.09
0.06
0.00
0.00
0.17
0.04
0.02
0.17
0.01
1513
python, edit (2x) 0.01
0.36
00.01
0.01
0..00
0.05
0.00
0.57
0.00
0.00
0.00
0.00
0.00
1071
python, edit, python
0.01
0.63
0.01
0.12
0.01
0.07
0.00
0.00
0.00
0.00
0.10
0.00
0.03
651
0.2
scroll down (3x)-
0.04
0.02
0.03
0.00
0.01
0.03
0.01
546
search_di dir, open, search_ file
532
search file, goto, edit-
0.01
0.04
0.00
1518
0.0
create
edit
exit cost
fmd file
goto
open
Pytest
python
scroll down
scrull up
search dir
search file
submit
Next Action
Figure 23:
Heatmap displaying the relative frequency of different actions being invoked after the most popular triplets actions SWF-agent w/ GPT-4 Turbo trajectories across task instances.
Transition Probabilities Heatmap
<START>, create, edit, python 0.00
0.02
0.32
0.00
0.36
0.00
0.06
0.01
0.00
0.00
0.00
0.24
0.00
0.00
581
1.0
create, edit, python, edit 0.00
0.00
0.35
0.00
0.01
0.00
0.00
0.00
8.60
0.00
0.00
0.01
0.00
0.00
474
create, edit, python, find file 0.00
0.00
0.01
0.00
0.17
0.00
0:74
0.00
0.00
0.00
0.00
0.08
0.00
0.00
547
edit (2x), python, edit 0.00
0.01
0.45
0.02
0.01
0.00
0.04
0.00
0.44
0.00
0.00
0.01
0.00
0.02
693
0.8
edit (3x), python 0.00
0.02
0.58
0.02
0.06
0.04
0.11
0.00
0.00
0.01
0.01
0.07
0.04
0.04
1123
Previous 4 Actions
edit (4x)-
0.00
0.05
.399
0.6
edit. python, edit (2x)
1059
edit, python, edit, python
679
edit, python, find file, open
457
0.4
edit, python, m,, submit
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
419
open, search file, goto, edit 0.00
0.00
0.03
0.04
0.00
0.27
0.01
0.00
0.01
0.02
Q.12
$20
python, edit (2x), python
0.02
0.01
0.12
0.01
0.06
0.00
0.01
0.00
0.00
0.09
0.01
0.08
363
0.2
python, edit, python, edit
0.00
0.00
0.27
0.01
0.00
0.00
0.02
0.00
0.68
0.00
0.00
0.00
0.00
0.00
336
scroll down soowll_downn (4x)
0.00
0.01
0.07
0.04
0.02
0.04
0.02
0.00
0.00
0.72
0.01
0.03
0.03
0.01
1034
search dir, open, scarch file, goto
0.00
0.01
0.41
0.01
0.01
0.09
0.06
0.00
0.00
0.22
0.04
0.03
0.12
0.01
4564
0.0
<END>
create
edit
exit cost
find file
goto
open
Pylest
python
scroll down
scroll up
search dir
search file
submit
Next Action
Figure 24: Heatmap displaying the relative frequency of different actions being invoked after the most popular quadruplets of actions in SWE-agent w/ GPT-4 Turbo trajectories across all task instances.
34

---

each trajectory. In Figure 8, we show the results of this automated categorization. Evaluated on our validation set, the LM generated labels agree with the authors' labels on 87% of instances.
We find that about half (52.0%) of the unresolved instances fall into the Incorrect Implementation or Overly Specific Implementation categories, suggesting that agents' proposed solutions often simply fail to functionally address the issue or are insufficiently general solutions. Another significant category the Failed Edit Recovery category, making up 23.4% of instances, which happens when models fail to generate well-formed edits to files, which can seriously inhibit their performance. The remaining failure modes make up less than 25% of instances, but highlight different aspects of the challenges faced by the agent in the problem-solving process.
Table 9: Descriptions of failure mode categories.
Category
Description
The agent tried but was not able to successfully reproduce the problem in the issue.
Failed to Reproduce
Failed to Find Relevant File
The agent never opened or saw the correct file.
The agent opened and viewed the correct file but didn't find or edit a relevant location.
Failed to Find Edit Location
The agent made relevant change but its solution was not sufficiently general; in this case it might solve the very specific issue suggested but does so in way that might change the hehavior of the code other, more general, cases.
Overly Specific
Implementation
The agent made a change to a reasonable area but their solution didn't correctly address the issue.
Incorrect Implementation
The agent seemed to be on the right track to a solution, but the episode ended before they could complete their changes.
Ran Out of Budget
The agent went into an edit loop, making recurrent failing edits without recovering.
Failed Edit Recovery
The agent decides to stop solving the problem after encountering some difficulty.
Gave Up Prematurely
There was some other problem that prevented the agent from resolving this issuc.
Other
B.5 Performance Variance and @ass@k k Rate
Since running SWE-agent on SWE-bench can rather expensive, we perform, all results, unless otherwise stated, are reported using pass@1 1 metric (% Resolved). However, we also test our main SWE-agent configuration for higher number of runs to test the variance and pass@kk performance for k E (3, 6). These results are shown in Table 10, suggesting that average performance variance is relatively low, though per-instance resolution can change considerably.
Table 10: Performance for 6 separate runs of SWE-agent with GPT-4 on SWE-hench Lite. The % Resolved rate for each individual run is shown in the first table, and the pass@k rate in the second.
SWE-bench Lite
Run
Run 2
Run
Run 4
Run 5
Run
Avg.
Resolve %
17.33
18.00
18.00
18.67
17.33
18.33
17.94p.19
Pass
Pass @2
Pass @
Pass
P
Pass@6
Pass@k
17.94
23.89
27.35
29.67
31.33
32.67
35

---

B.6 Patch Generations
In this section, we present some statistics and analysis around the edits generated by SWE-agent. At the end of a task episode, the edits made by SWE--gget are aggregated and saved as a single .patch file, the canonical representation for code changes of pull request GitHub. From these patch representations, we can quantitatively characterize an agent's 's generations and see how they compare to the original solutions written by human codebase maintainers.
Table 11 presents a summary four basic statistics about the model generations. Lines added and lines removed refer to the total number of lines that were added or deleted in the patch, an indicator of the size of the modification. The number of hunks and files is more indicative of how many "regions" of the codebase were modified. A higher number of hunks and files suggests that there are more distinct, separate places in the codebase where the patch made changes. For both "Resolved" and "All" categories of task instances, models tend to generate "larger" edits (e.g., more lines added, hunks, and files) than the corresponding gold solution. Prior RAG baselines Jimenez et al. [20] typically produce smaller edits on average. The source of this increase for sentt-generated solutions can largely be attributed to additional reproduction code.
Table 11: We show the (median) / (mean) value for several statistics characterizing patch generations. We calculate these statistics across two dimensions. First, the "Resolved" / "All" labels denote whether the patch resolved the issue. Second, for the task instances specific to each model, we calculate the same statistics across the gold patches. To diminish the effect of outliers, we calculate these statistics based on values falling within within the 90th percentile of the distribution.
Model
Outcome
Lines +
Lines
Hunks
Files
SWE-agent w/ GPT-4 Turbo
Resolved
5.7 12.0/ 16.58
1.0/ 1.32
1.0/1.52 2.0/1.83
1.0/ 1.22
Any
1.0/ 1.35
1.0 1.53
Gold
Resolved
2.0 3.58
1.09
1.3
1.0/
Any
7.0/ 11.67
4.0/4.05
2.45
1.0/ 1.24
SWE-agent
Resolved
3.09
1.0/
1.0 1.0/
w/ Claude 3 Opus
Any
11.0/ 15.25
2.0/ 2.0/2.14 1.79
Gold
Resolved
3.91
1.0/1.94
1.0
1
Any
6.0 10.68
2.0/3.61
2.0
1.0/1.13
When comparing the "Resolved" and "All" categories, we see that successfully resolved edits are relatively smaller than the original distribution. This trend is consistent with the RAG based solutions; issues that require multiple edits across codebase remains challenging for agents.
B.7 HumanEvalFix Evaluation
In this section, we include further discussion about our evaluation of SWE-agent on HumanEvalFix. We choose to evaluate on the HvliFEvalFix task because focuses on code editing and debugging. which was empirically demonstrated in Muennighoff et al. [32] to be more difficult task for LMs (as reported in their work, GPT scores 78.3% on HumanEval, compared 47.8% on Humxx. The code editing task can also be thought of as "subtask" in SWE-bench; being able to identify and fix bugs is a major part of software enginecring.
We adopt the HamanEvalFix dataset (164 problems per language) to be compatible with the SWE- agent setting. Following the documentation in Muennighoft al. [32], SWE-agent is initialized in a directory with single file containing a buggy code snippet and example test(s) if available. It is then asked edit the code and verify its fixes. The configuration file is identical the one used for SWE-bench, with the exception of a language-specific demonstration. For this task, localization and navigating a large codebase are not necessary; the main focus on generating the correct edit. SWE-agent achieves the best performance on the HumanEvalFix benchmark for three of the languages we evaluate on, as shown in Table 2. Figure 25 also suggests that the large majority of task instances are solved within the first 10 turns.
36

---

HumanEvalFix-js
HumanEyaaFiixjaaaa
Human
80
Frequency
Frequency
Frequency
60
40
20
0
0
10
20
30
0
10
20
30
10
20
30
Turn
Tum
Turn
Figure 25: Similar Figure 14, show the distribution of the number of turns for trajectories corersponding to solved task instances from the HumanEvalFix dataset.
B.8 Dataset Information
In the following Table 12, we provide descriptions of the two datasets that we use for evaluation: SWE-bench [20] and HumanEvalFix [32]. Both datasets have been released under permissive software licenses that allow for evaluation use, and can used in proprietary systems.
Table 12: Information about each of the datasets that evaluate SWE-agent
Dataset
Released
License
Splits
Count
Languages
GitHub Repo
SWE-bench
10/10/2023
MIT
Test 2294
Python
princeton-nly -lpp/ SWE-bench
Lite 300
Dev 225
HumanEvalFix
07/23/2023
MIT
Test
164
Python, JS, Go Java, C++, Rust
bigcode-project/ octopack
B.9 Miscellaneous
In this section, we include additional minor analyses around agent behavior and their generations.
Agents are better at localizing files than BM25. The interactive setting also enables agents to identify the correct file(s) to edit more often compared to the RAG baselines in Jimenez et al. [20]. To measure this, we calculate the Fl score between the set of [edited, removed] files by the agent's prediction versus the gold patch. SWE-agent w/ GPT-4 Turbo achieves an F1 score of 59.05%, while BM25 w/ Claude 3 Opus produces Fl score of just 45.47%.
Most resolved task instances are intentionally submitted. There are four ways task episode ends.
"Submit" refers task episode that ends when the agent generates the submi i command.
"Exit Cost (Submit)" refers the scenario where the episode ends because the cost limit was hit, and the changes so far are gathered and submitted as an edit.
"Exit Cost (No Submit)" refers to when the cost limit was hit and no edit's t's were made, so there was nothing to submit. In this scenario, the instance is guaranteed to unresolved.
"Early Exit" refers to when the task episode terminates because an agent issued too many Any changes submitted an edit.
malformed responses in
Table 13 shows the counts for the number of trajectories that ended on these four different outcomes, categorized across the agent, SWE-bench split, and whether or not that task instance was resolved. For SWE-agent with GPT-4 Turbo, the majority of "All" task instances are submit t. For the trajectories corresponding to"All" task instances by SWE-agent with Claude 3 Opus, slightly less than 50% of task instances are submitted, while the slight majority auto- sutt-submitted when the cost limit is hit.
37

---

Table 13: This table showcases the counts for the four ways ("Submit", "Exit Cost (Submit)", "Exit
Cost (No Submit)", "Early Exit") task episode could conclude.
Submit
Exit Cost (Submit)
Exit Cost (No Submit)
Early Exit
Model
Split
Outcome
SWE-agent w/ GPT-4 Turbo
Full
Resolved
266
20
0
All
1589
630
48
1
Lite Resolved
50
0
All
203
95
0
SWE-agent
Full Resolved
206
35
0
0
w/ Claude 3 Opus
882
1048
Lite
Resolved
32
0
All
133
156
0
However, these trends do not hold for "Resolved" task instances. For SWE-agent with both models, the large majority of these task instances are submi t. Reiterating the conclusion in Section 5.2 and prior visualizations in Section B.3, we see here again that resolved task instances often imply that the agent is able produce and verify an edit within the allotted number of turns. The SWE-agent ACI is also effective at eliciting well-formed thoughts and actions from agents. Across all runs, there are only two "Early Exit" occurrences, where the episode terminated because the agent generated too many malformed responses in row.
Finally, Table 13 also upholds expected trend. Task instances that finish with submi t action more likely to be resolved than those that are cutoff by cost. For instance, for SWE-agent with GPT-4 Turbo full SWE-hench, 14.3% of task instances that end with submit t resolved, which is much higher than 3. % for those finishing on exi cost.
38

---

C Prompts
In this section, we go through the prompt templates that make up the agent's history, discussing them in the order of presentation to SWE-agent. Per template, we describe its purpose, walk through its content, and note any additional motivations that influenced how wrote the template. The companion figures of template content are all drawn from our default configuration, using SWE-agent w/ GPT-4.
The template content can and should adapted slightly to fit the agent's intended use case. The purpose of this section is to describe our thought process for how we designed each template for these tasks to serve as reference for future work. Across templates, we find that providing tips which tell agents to not make specific mistakes, avoid common pitfalls, and use helpful execution signals are effective for cliciting more successful problem solving.
Prompt Workflow. We present Figure 26 which shows the order in which different prompt templates are invoked. This flow of prompts reflects the logic that generates trajectories similar the one that is visualized in Figure 9.
Terminates Action is subnit, or
Cost exceaded,
Turns (Till 'submit action)
3. 2+ Consecutive Format Errors
Next Step
Initial Messages
Show output of mode's action prompt action
(1) System
Turn #1
Thought Action
Next Step (No Output)
(2) Demonstration
Language Model
Execute Action Env.
Show massage stating action suaceeded quietly
(3) Instance
Inference
Format Error indicate prior action not be parsed, model must retry
Task Episode Begins
The flow of prompt templates throughout single SWE-agent task instance episode. The system, demonstration, and issue templates are shown all together at the beginning of the task episode, followed by tur--ppecffic prompts that shown depending whether the agent response is well-formatted and whether the action has standard output.
Figure 26:
System Template. The system template describes the interactive task setting, the commands at the agent's disposal, and the expected response format. not change in content across task instances, and is not removed or collapsed at any point from the message history. The agent is told of the general task setting, which command line that comes with special file viewer interface. After this, the agent is presented the command documentation which shows a usage example and docstring for every custom command, mirroring the content of Figure 27. As discussed before, from manual observation, we find that agents need lot of support to make effective use the edit command.
the first message for any episode, does
System Prompt
SETTING:
You are an autonomous programmer, and you't 're working directly in the command line with special interface.
The special interface consists of a file editor that shows you 100 lines of a file at time. In addition to typical bash commands, you can also use the following commands help you navigate and edit files.
COMMANDS: [documentation)
Please note that THE EDIT COMMAND REQUIRES PROPER INDENTATION, If you'd like to add the
line print(x)' you must fully write that out, with all those spaces before the code! Indentation is important code that not indented correctly will fail and require fixing before can be run.
RESPONSE FORMAT:
Your shell prompt is formatted as follows:
(Open file: <path>) <ewd> $
You need to format your output using two fields; discussion and command. Your output should always include one discussion one command field EXACTLY the following example:
39

---

DISCUSSION
First I'll start by using what files are in the current directory. Then maybe look relevant files to see what they look like.
Is -a
You should only include SINGLE command in the command section and then wait for response from the shell before continuing with more discussion and commands. Everything you include the DISCUSSION section will be saved for future reference. If you's d like to issue two commands at once, PLEASE DO NOT DO THAT! Please instead first submit just the first command, and then after receiving a response you Il be able to issue the second command. You't re free use any other bash commands you want (e.g. find, grep, cat, cd) in addition the special commands listed ahove. However, the environment does NOT support interactive session commands (e.g. python, vim), so please do not invoke them.
Figure 27: The system prompt for SWE-agent describes the environment. The document atinon
field is populated with brief description of all enabled commands, similar to Table 4.
An agent will occasionally generate an edit t with either the wrong level indentation or incorrectly specified line numbers. Because of this, proper indentation. Finally, the system prompt describes what the agent's response should look like, communicated with example (e.g. JSON format, XML delimiters) followed by a paragraph reinforcing the importance of issuing a single thought/action pair per turn. Because of the constraints imposed by Docker containers, we include one last point about the command line environment not supporting any interactive session commands, such as vi or python. The system template does not introduce any task instance specific information.
include note telling the agent to pay attention to
Demonstration Template. If provided, the demonstration template immediately follows the system template the second message showing the agent trajectory which resulted in the successful resolution of a task instance from the development set. As confirmed by the ablation in Table 3, including demonstration slightly helps agents with understanding proper command usage and reduces the rate of errant responses. Rather than being written out entirely like the system template, the demonstration template put together based discussed in Section A.3. First, the demonstran rat ons argument points at 1+ traj file(s), each containing multi-turn, SWE-agent style history interactions.
fields shown in the configuration file
and poa__demee in rn__isstory arguments control how the demonstration is represented in the message history. If the put__emee ____in_hha_ history argument is set True, every turn of the demonstration trajectory used as a separate message in the history. The alternative is write out the entire demonstration a natural language instruction indicating it is demonstration (this is the method that is displayed in Figure 28). The effectiveness of each method varies by model. The first method of adding demonstration as separate messages can be preferable because it gives an agent the impression that has already solved task instance and is working the one. However, for different model, it can be confusing to have the contexts from two tasks in a single trajectory, which is why the second method of explicitly telling the model that it's being given a demonstration can be better.
Second, the demonstrate aantinnttooooo demonstraatioon_.eepp..e.
single message, wrapped in
Demonstration
Here is demonstration of how correctly accomplish this task. It is included to show you how to correctly use the interface. You do not need to follow exactly what is done in the demonstration. DEMONSTRATION :INSTANCE PROMPT))
The code has been updated to use the "round" function which should fix the rounding issue. Before submitting the changes, it would be prudent to run the reproduce. py code again to ensure that this change has the desired effect. I see that "Current directory" is still /maashmalloo-oooe marshma 1low", so we can just run 'zeproduce. directly.
40

---

python reproduce.py
345
(Open file: http://hhhi...oooooooaoooooooooa /et. (Current directory: / bash-$
The output has changed from 344 to 345, which suggests that the rounding issue has been fixed. Let's remove the reproduce.py file since it is longer needed.
rm reproduce.py
Your command successfully and did not produce any output. wowallllolfflleeeoeeeeeeeeeeelee
(Open file: 1orrhmallox-codee aeshmalllo-cccie__aassaaaae mas___aarshaallaaa www.mmaaaaaa/srrrro/o/ hatpma/l//////////a///a/aa (Current directory: m low)
bash-$
rm doesn't have
output when it deletes successsully, that must have worked. Now that have fixed the issue, should submit our changes to the code base. Let's submit changes using the "submit" command.
submit
END OF DEMONSTRATION
Figure 28: simplified demonstration template showing how demonstrations are provided to the model single message. Here show only the final 3 turns in the demonstration for brevity.
We are unsure demonstrations actually help agents understand the nuances of domain specific problem solving. Because of the diversity of software engineering issues, we think the role the demonstration plays is primarily help the agent learn to issue properly formatted commands. Prior work has demonstrated that fine tuning may have the potential to imbue agents with a certain degree of expertise around how to adaptively solve task instances that may vary in terms of what strategy most successful.
Instance Template. The instance template introduces the agent to the task instance. The problem statement is shown, followed by brief set of instructions that reiterate important points from the system template. These points are the one thought/action per-turn requirement, mentioning the lack of support for interactive shell commands, and reminder of the importance of editing indentation. Finally, a notably effective part of the instance template is the inclusion of tips which serve as an additional guidelines for how to operate successfully in the bash environment, shown in Figure 29, These tips were developed manually and iteratively; after running SWE-agent with a particular configuration on the development set, manually looked at the trajectories for failure modes. The tips were born out of these failures, and through repeated inspection, we found that such tips did reduce the frequency of errant problem solving strategies that they are meant to address. While our manual approach to writing tips certainly does not scale, representing feedback for common mistakes tips is surprisingly effective. Developing better methods for this process of identifying failure modes and writing natural language instructions that describe the correct alternative behavior could be avenue better performance for future SWE-agent based systems. Finally, the end of the message, the agent is presented with a command line prompt indicating that the task has begun and that the agent should issue its first command.
Instance Message
We We're currently solving the following issue within repository. Here's the issue text: ISSUE:
(issue)
41

---

INSTRUCTIONS:
Now, you're going to solve this issue on your own. Your terminal
+ session has started and you're in the repositozy's s root disectory. You can to help you. Edit all the files you need to and run any checks ONLY ENTER
any bash commancs or the special interface
tests that you want. Remember, YOU + A TIME. You should always wait for feedback after every command. ~ When you're re satisfied with all of the changes you've 've made, you changes the code base by simply running the any interactive
COMMAND
can submit submit command. Note however that you cannot this environment, but you
session commands (c.g. python, vim)
can write scripts and run them. E.g, you can write a python script and then run it with "python <script_name>.p
NOTR ABOUT THE EDIT COMMAND; Indentation really matters! When editing + a file, make sure to insert appropriate indentation before each line!
IMPORTANT TIPS:
Always start by trying to replicate bug that the issues
+ discusses. If the issue includes code for reproducing the bug, we
+ zecommend that you ze-implement that in your environment and run it to make sure you can reproduce the bug. Then start trying to the bug 4 reproduction script to make sure that the bug has indeed been fixed.
fix it. when you think you've fixed the bug,
2. If you + command, command that not work ence will not work the + second time unless you modi it!
command and it doesn't work, try running different
3. If you open file and need get to area around specific line that is not in the first 100 lines, say 1ine 583, don't just
> use the scroll down command multiple times. Instead, use the gote - 533 command. It's much quicker.
4. If the bug reproduction script requires inputting/roading
4 specific file, such buggy-input.png, and you'd
4 understand how to input that file, conduct a search in the existing repo code, to see whether someone else has already done that. Do this by running the command: find_file "buggy-ingut.pm the linux 'fina' command.
If that doesn't work,
5. Always make sure to Look at the currently open sile and the current working directory (which appears right after the currently open file). The currently open file might be in different directory than the working directory! Note that some commands, such as 'create', open files, so they might change the current open file.
6. When editing files, it is casy to accidentally specify wrong line number or to write code with incorrect indentation. Always check the code after you issue an edit to make sure that it reflects what you wanted to accomplish. If it didn't, issue
another command to fix it.
(Open file: loper filel)
(open_file))
(Curgent directory: (working dirl)
bash-$
42

---

Figure 29:
The instance template. This message shows the task instance's problem statement (referenced by the (issue) field), shows additional task instance-specific information, and provides set of tips suggesting recommended problem solving approaches and pitfalls to look out for.
Next Step Template. Assuming an agent's response is well formed and contains an action, there are two simple templates used to present the corresponding output of the agent's 's action, shown in Figure 30. If an agent's action produces some standard output, the agent simply shown this output with command line prompt indicate that the agent should respond with the next action.
Next Step Template
(ORSERVATION)
(Open file: ppathhtoooppe//iile (Current directory: /path/to/crt cwd) bash-$
The environment's "next step" template. This is emitted after each observation to inform the model of the current state of the shell and programs.
Figure 30:
However, an agent's action runs silently and produces no output (e.g. rm abc. py, touch abc. .y), found that simple showing output along with prompt can be confusing for agents to interpret, and it will often run additional, unnecessary commands to determine the effect of the prior action. To guard against this situation, the agent is informed verbosely that the command ran successfully and did not produce output. While the System, Demonstration, and Instances template only used single time, the next step template is used repeatedly. In the SWE-agent configuration described in this work, the next step templates are fairly simple, as they essentially just add the command line prompt the end of the execution standard output. We have not explored other variations to this style.
Collapsed Observation Template. As shown in Figure 9 and discussed in Section 2, old observations are collapsed; meaning that the structure and order of the agent's 's interaction history preserved, but the content of old observations are replaced with one-line placeholder. This summary simply states that the observation omitted with the number of lines that were removed, as shown in Figure 31.
Environment Response (collapsed) Template
old output omitted (101 lines)
Figure 31: A collapsed environment response. The content old observations are replaced with this template.
The purpose of collapsing old observations twofold. First, simply reduces the number of tokens needing to be processed at each interaction step. Second, by removing old observations' content, it can also reduce the presence of outdated or duplicate information, such as directory and file contents or command output. We think that both of these purposes can serve to improve agent performance.
Error Message Template. An agent's response is not always well formed. As discussed, an agent is generally expected to respond with single thought and action per turn. In our work, we experimented with asking the agent generate one of three different styles of responses for communicating one thought and one action (same type of response across any single episode).
1. A string where the action is wrapped markdown-style code block ("").
2. A JSON style dictionary with "thought" and "action" keys.
3. An XML style generation with "<<though>>" and "<actionn>" delimiters.
On occasion, agent may generate response that doesn't conform the requested format. If this occurs, we show an error message to the agent indicating that the prior message was malformed and to issue another response that does make the same mistake, presented in Figure 32. If a model generates 3 malformed responses in a row, the episode will terminate early.
43

---

Error Message
Your output was not Formatted correctly. You must always include one discussion and one command as part of yeur response. Make sure you do not have muitiple ALscussion/commarc tags. Please make sure your output precisely matches the following format: DISCUSSION
Discuss here with yourself about what your planning and what you're going to do in this step.
command that you'l re going to run
Figure 32: The environment's s error message. This is emitted if model generation doesn't 't conform to the thought-action format suggested.
Another context management trick is that if models generate a malformed response, but then subse- quently respond with a valid onc, the message history is modified such that the action and response correspond to the malformed generation is removed. This kind of de-noising reduces unnecessary context and helps prevent future malformed generations. Each well-formatted response becomes an additional in-context demonstration of how to interact with the environment correctly; this "momen- tum" of correct responses is effective for helping agents continue to issue correct actions at later turns trajectories when there is lot of information in the message history.
44

---

D
Qualitative Analysis
We choose successful and unsuccessful task instances from SWE-bench Lite perform full qualitative analysis, based on the trajectories generated by the default configuration using GPT-4 Turbo.
Analysis for pst/requests-2217
Issue Summary. The agent was tasked with resolving problem in the requests library where the method variable, bytes object (e.g., b GET' ), was being incorretlly converted to a literal string ("b' GET' ") by the builtin. stst function. This caused the requests library to raise a 404 Not Found response due to the incorrect method string. The issue was observed in Python 3.4 and in Python 2.6, indicating version-specific difference in handling bytes.
Interaction Summary
1. Locate the Target File:
The agent began by finding the sessions py file in the requests library directory. correctly identified that the relevant file was the requests directory rather than the build directory.
2. Search for the Issue: opened sess tons., py and used the search,
command
locate the builtin st. function call. It found the relevant line (line 428) where the method variable was being converted.
3. Edit the Code:
The agent edited the sessions. py file to check if the method was bytes object and, if so, decoded it to ASCII before applying the builtin_stz function.
4. Test Changes: To verify the fix, the agent created new file, reproduce sacuuce_issure issue py, which attempted to send an HTTP request using binary method string. The script printed the response status code to check the request was handled correctly. It ran the reproduce_ _isee_sssse___eeeeeeeeeeseesseee py py script, which executed successfully and received 200 OK response, indicating that the issue was resolved.
5. Clean Up and Submit: Finally, the agent deleted the reproduction script and submitted the changes to the code basc.
Upon evaluation, the agent's proposed changes successfully pass the instance's unit tests. When comparing the agent's generation to the 'gold patch', we notice that the gold patch uses the to_natt ve weessting function, defined in eequests/utils.pyttt the agent's proposed changes. Ultimately, while the agent successfully resolves the issue, it fails to use existing utility functions, which could lead to less maintainable or portable code in production.
Evaluation.
which behaves identically to
Figure 33: Trajectory for instance psf requests-2311..
Issue
psf req_reequsst223
We're currently solving the following issue within our repository. Here's the issue text:
ISSUE:
method = builtin str(method) problem
In requests/sessions.pp py command:
method builtin str(method)
Converts method from b b'GET"
"b'GET""
Which the literal string, no longer a binary string. When requests tries use method "'GE""" gets 404 Not Found response. I am using python3.4 and nython-neutronclient (2.3.9) with requests (2.4.3). neutronclient is broken because uses this "args binary string, including method.
citlsssaf_eeecoee list(args)" command which converts all the values to
I'm not sure if this is bug with neutronclient or a bug with requests, but I'm starting here. Seems requests handled the method value being a binary string, we wouldn't have any problem. Also, I tried in 2ython2.6 and this bug doesn't 't exist there. Some difference between 2.6 and 3.4 makes this work right.
45

---

INSTRUCTIONS: (31 lines)
(Open file: n/a)
(Current directory: /psł requests) bash-
SWE-agent
psf requests-2311
To address the issue, we first need to replicate the behavior described. The issue mentions that the 'huiltin_str ustr"rssrrr function in 'requestv/sssioos..yy is converting binary string method like "b" D'GET" into the literal string "b'GET", which is incorrect. We should start by locating the "sessions.py file within the 'reguests" library examine the code around the 'huiltin str' function call. Let's begin by finding "sessions.py py the current repository.
find file "sessions.py"
Observation
psf uss_requustt-221
Found 2 matches for "sessions. py" in /psf requests: /psf http://wwe////ll/l//////t/////// http://cccceetttssssoo..... /psf_ sss__ooggcctrrreeess///sssssppp
(Open file: n/a) (Current directory: req_ess))sts) bash-$
SWE-agent
psf requests-2317
We have found two matches for "sessions.py" The one in the 'build/lib/requeess" directory is likely build artifact, and the one we're interested probably '/psf rewpeesssrrrrreusssssssssssssspp py". Let's open this file inspect the code around the 'builtin str" function call.
open requests/ sestttsssessiieo wessiins.sss.......ooo.....s.... oy
Observation
psf requests-2317
[Fi 1e:
(671 1ines tetal)]
1:# cocing: ut
2:
3:"m
prrquests.sessi..
5: 5:-.*..******...................
6:
7:This module provides a Session object to manage and persist settings across
r:requests (cookies, auth, proxies).
9:
10:"**
11:import 68
fro collections ons import Mapping
13:from datetime import datetime
14:
1rof .auth import basic auth_str
16:from .compat import cookielib, OrderedDict, urljoin, urlparse, builtin _st
46

---

17:from cookies import
18: cookiejar_irom_die extract_cookies_to_r
RequestsCokkeeaar merge_ (oge__cokies)
19: mooees import Request, PreparedRequest,
DEPAULT_REDIE DEPAUI_RBIRRTTLLNIT
from hooks import default_hooks, dispatch_hood 21:from .utils import to to_key_val list, default headers_
s__natvv_striie
22:from .exceptions import
23: TooManyRedirects, Invalicschema, ChunkednncodnngErre. ContentDeooingErroot
24: from .structures import CascInsenoitiveDictt
25:
26: from adapters import HTTPAdapter
27:
28: from utils import {
29: requote_urk get hnaviron_proxiees get_netro, auth
should
should_bypass bypass byppssssssssppiesss...pppppsss
30:
get_ auth Srom_ur: url
31:)
32:
from .status_codes import codes
34:
35:# formerly defined here, reexposed here for backward
compatibility
from .models import REDIRECT_STANE
37:
38:
39:def berre_ssetinn((rrquee_eeeee session_setting - dict dict_elass-Oderrdbiie
40:
41: Determines appropriate setting for given request, taking into account the
42: explicit setting on that request, and the setting the
- session. If a
43: setting is dictionary, they will be merged together using "dict_class"
44: ***
45:
46:
if session_setting is None:
47:
return request_setting
48: request_setting None:
49:
50:
return session_settimg
51:
52:
Bypass if not dictionary (e.g. verify)
53: if not
54:
isinstance
Mapping) and
55:
isinstance (request_setting Mapping)
56:
):
57:
return roquest_setting
58:
59: merged_setting
dict_class (to_key) jee__eey_vlll val laal_lisss list (ewessicnssettieeee
60: merged_setting updeee I key_val_list ist trequest_setting
61:
62:
Remove keys that are set to Nene. for (k, v) in
63:
setting.items( (te
64: it is None:
65:
del merged_setting(X)
66:
67:
merged_setting diet ( (k, v) for (k, v) in
() if v is not None)
47

---

68:
69:
return merged
70:
71:
72:def merge_hooks (request_hooks, session_hooks,
diet_cllas--rdeeebbbittt
73:
74:
Properly merges both requests and session hooks.
75:
76: This is necessary because when request_hooks ('response's
[}), the
77: merge breaks Session hooks entirely.
78:
79: if session_hooks is None session_hooks.ges ('response))
[]:
80:
return request_hooks _ooks
81:
82: if request_hooks is None or hooks. get rreepensse)
[]:
83:
return session_hooks
84:
85: return merge_setting (recussthkkkk,,,,kkkkk session_hooks, dict_class)
86:
87:
88: class ScssionRedrrettiikk (objjet
89:
def resolve_redirecte (self, resp, reg, stream-False, timeout=None,
90:
fyrify- cert=None, prexies-None)
91:
"""Recevees Response. Returns a generator of Responses.
92:
93:
94:
hist [] # keep track of history
95:
96:
while resp. rosp.is_redirees redrr rrquest cop.
97:
98:
99:
> 0:
100:
# Update history and keep track of reditects.
(571
more lines below)
(Open file: n/a)
(Current directory: requests))
bash-$
SWE-agent
psf requests-2311
The 'sessions.py' file is open, but the 'builtin str' function call visible in the first 100 lines file. Since we don't have exact line number where 'builtin_str str' function is called, we'll need to search within the file for it. Let's use the 'search file' command to find the usage of 'builtin str' within 'sessions.py'.
search_filh file "builtin_str"
48

---

Observation
psf requests-2317
Found 2 matches for "builtin str" in
/psf_ peeueets//rqqussss//ssssssl
Line 16:from .compat import cookielib, OrderedDict, urljcin, urlparse, builtin_str
Line 428:
method = builtin (method)
End of matches for souiltin_str"" in
(Open file: /psf_ pewpsssss//uuueet////sssooooo (Current directory: /psf requests) bash-$
SWE-agent
ref__rqqustts-211
have found two occurrences of 'builtin str' "sessions.py" " The second occurrence on 428 where the method is being converted to string. We should inspect this line to understand the context and see how address the issuc.
Let's navigate line 428 in 'sessions.pp py' see the relevant code.
goto 428
Observation
psf requests-2317
[File: /pst____e______
weeggees/reuusst//eesiiitt (671 lines total)
(412 more lines above)
413:
data before giving up, float, ("connect
timeout, read
414:
timeout <user/advanced, html Ptimoouts>' _] tuple.
415:
:type timeout: float or tuple :param allow_redirecte (optional) Set to True by
416:
default.
417:
:type allow sellowredirects: bool
418:
:param proxies: (opticnal) Dictionary mapping protocol to the URL of
419:
the proxy.
420:
:param stream: (optional) whether to immediatell Nownload the response
421:
content. Uefaults to "Falss"".
422:
:param verify: (optional) if "True", the SSL, cert will be verified.
423:
A BUNDLE path can also be provided. :paran cert: (optional) if String, path to ssl client cert file (.pem).
424:
425:
If Tuple, ('cert', 'key') pair.
426:
"""
427:
428:
method builtin_str str (method)
429:
430:
Create the Request. req Request (
431:
432:
method methoa. upper(),
433:
url url, headers headers, files files, data data or (),
434:
435:
436:
437:
json = json, parans params or (h,
438:
49

---

439:
auth auth,
440:
cookies cookies, hooks hooks,
441:
442:
443:
prep self pat..preprre_request (req)
444:
445:
proxies proxies or (}
446:
447:
settings = self.merge wtt.emerrgeeeerrrrreeeeee wew.eeennentt....eeemme _ntttennntsttngs
448:
prep.url, proxics, stream, verify, cert
449:
450:
451:
Send the request.
452:
send send_kwargs, -
453:
'timeout': timeout,
454:
aallowrreirectte al allov_redi rects,
455:
456:
send kwargs. update (settings)
457:
reap self. (prep, **send kwargs)
458:
459:
return resp
460:
461:
def get (self, url, *kwargs)
462:
"""Sends GET request. Returns :class: "Response"
object.
463:
464:
ppran url: URL for the new :class: "Rquess" object. :paran Vrlkwargs: Opticnal arguments that "irequest"
465:
takes.
466:
"""
467:
468:
kwargs setrgggsetdefuult ('allew_reel reddrectts,,, True) return self. request ('GET', url, **kwargs)
469:
470:
471:
def options (self, url, **kwargs):
472:
"""Sends a OPTIONS request. Returns iclass: "Respense"
object.
473:
474:
ppram url: DRL for the new :class: "Rquuest" object. :param (*/akwargs: Opticnal arguments that "'rqqest""
475:
takes.
476:
**"
477:
478:
kwargs.setdeffn setdefault ("allow_redireats" True) return self. request ('OPTIONS', url, **warggs)
479:
480:
481:
def (sed,, url, *kwargs)
482:
"""Sends a HEAD request. Returns rclass: 'Response"
object.
483:
484:
paran url: URL for the :class: 'Request" object. :param \*A*wargs: optional arguments that "'request"
485:
takes.
486:
487:
488:
kwargs setrggssetdefuult ('allow rew.leeteddrret' False) return self request ('HEAD', uri, **kwargs)
489:
490:
491:
def post (self, url, data=Nene, ison=None, **kwargs)
492:
"#*Sends POST request. Returns :class: "Response"
object.
493:
494:
:arran url: URL for the new :class: 'Request" object.
50

---

495:
:paran data: (optional) Dictionary, pytes, file-like object to send the body of the :class: RRquess",
496:
:paran json: (optional) json to send in the body of the :class: "laass"Kequess" Request",
497:
:param Wilkwargs: Opticnal arguments that "rrqqest"
takes.
498:
"""
499:
500:
return self. request ('POST', url, data=data, jsen=ison,
**kwargs)
501:
502:
def put (self, url, data=None, **kwargs):
503:
"""Sends a PUT request. Returns :olass: "elsses"Respense "Response"
object.
504:
505:
param url: URL, for the new :class: "Rqquest" object. :paran data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class: RRquest".
506:
507:
:paran brlekwargs: Opticnal arguments that "request"
takes.
508:
"""
509:
510:
return self. request ('PUT', url, data-data, 4kkwargs)
511:
512:
def patch (self, url, data-None, **kwargs): (159 more lines below)
(Open file: /psi requstts/rqueets/eesst (Current rirectry:: request)) bash-$
SWE-agent
psf requests-223
We are now looking at the section the 'sessions.py file where the 'builtin_str function is used convert the 'method' variable. According to the issuc, this conversion is causing a problem when the 'method' is binary string because it's being converted to a literal string that includes the binary prefix 'b'...'"
To address this issue, need ensure that the 'method' variable is correctly converted string without the binary prefix. This might involve checking type of 'method' variable and handling binary strings differently from regular strings. Let's edit the line to add check that ensures binary strings are decoded to regular strings before being passed to 'builtin_str'.
edit 428:428
# Ensure method is string, decode if it's binary string if isinstance (method, bytes)
method method. decode ( ''ascii') method builtin_str (method)
ond_of_edit
Observation
psf requests-2317
[File:
(674 lines total)
(377 more lines above)
378:
def request (self, method, url, params-None,
379:
380:
data=None,
381:
headers=None,
382:
cookies-None,
51

---

383:
files=None,
384:
auth-None,
385:
timeout:None,
386:
allowwrred1reetts=111ee
387:
proxies-None,
388:
hooks-None,
389: strean-None,
390:
verišy=None,
391:
cert-None,
392:
json=None):
393:
"""Construets :olass: "Request <Request>", prepares 1 and sends it.
394:
Returns :class: Response <Respense>" object.
395:
396:
:param method: method for the :class: Request"
:class:"kequest
object.
397:
:paran url: URL for the new :class: Request" object. :paran params: (optional) Dictionary or bytes te be sent in the query string for the :class: 'Request"
398:
399:
400:
:param data: (optional) Dictionary or bytes to send in the body of the
401:
:class: "Rquesst."
402:
:param json: (optional) json send in the body of the :class: "Request"..
403:
404:
:paran headers: (opticnal) Dictionary of HTTP Headers to send with the
405:
iclass: 'Requess"
406:
:param cookies: (optional) Dict CookieJar object send with the
407:
:class: "Rquestt.
408:
:paran files: (optional) Dictionary of "fileemme' file-like-objer
409:
for multipart encoding upload.
410:
:param auth: (optional) Auth tuple or callable to
enable
411:
Basic/Diget/Custonn НТTР Auth. :param timsout: (opticnal) How long wait for the
412:
4 server to send
413:
data before giving as float, or a ("cennect
timeout, read
414:
timeout <user/ acvanced. htm #timouusmt tuple. :type timeout: float or tuple
415:
416:
:param allow_redirects: (optional) Set to True by
default.
417:
:type allow_redirects: bool
418:
:paran proxies: (epticnal) Dictionary mapping protocol to the URL of
419:
the proxy.
420:
:paran stream: (optional) whether to immediately download the response
421:
content. Defaults "False"".
422:
:param verify: (optional) "True"", the SSL cert will be verified.
423:
A CA_RUNDLE path can also be provided. :paran cert: (optional) it String, path to ssi client cert file (.pem).
424:
425:
I= Tuple, ('cert', 'key') pair.
426:
427:
428:
Ensure method is string, decode if it's binary if isinstance (method, bytes):
4 string
429:
52

---

430:
method methoa. aecode ascii') method builtin str (method)
431:
432:
433:
Create the Request.
434:
req Request (
435:
method method. upper
436:
url url, headers = headers,
437:
438:
Eiles files,
439:
data data (h,
440:
json json, params params or (h,
441:
442:
auth = auth, cookies - cookies, hooks hooks,
443:
444:
445:
446:
request peep = prepare request (req)
447:
448:
proxies proxies
449:
450:
settings - self. www.geeeeirrmmeeneeeeeee
451:
prep.url, proxies, stream, verify, cert
452:
453:
454:
Send the request.
455:
send_kwarge =
456:
'timeout'; timeout,
457:
'allow_redirects" alLow_redirests,
458:
459:
htt__kwaarsuuuaatt (settings)
460:
resp self. (prep, **send_kwargs)
461:
462:
return resp
463:
464:
def get (self, url, **kwargs)
465:
"""Sends a GET request. Returns :class: Response"
object.
466:
467:
ppram url: DRL for the new :class: "Rquuest" object. :param (*/akwargs: Opticnal arguments that "'requet""
468:
takes.
469:
**"
4700
471:
kwargs.setde setdefault ("alloxrreirretts" True) return self. request ('GET', url, **kwargs)
472:
473:
474:
def (self, url, *kwargs):
475:
"""Sends a OPTIONS request. Returns rolass:" Respense"
object.
476:
477:
paran url: URL for the :class: 'Request" object. (197 more lines below)
File updated. Please review the changes and make sure they are correct (correct indentation, no duplicate lines, etc), Edit the file again if necessary.
(Open file: /ps requests/requests/eeeete (Current directory: /psf requests) bash-$
53

---

SWE-agent
psf requests-2317
The edit has been made to decode binary strings before converting them to string with 'builtin str'. This should prevent the issue where binary string like "b'GET" incorrectly converted the literal string ""b' "ET"".
To verify that this change resolves the issue, we should create a script that replicates the problem as described. We'll write script that uses the "requests" library to make request with a binary method and check if it now handles the method correctly. Let's create
file called 'reproduce_ jsssueepysy test the fix.
create reproduce_issue.
Observation
psf requests-uuse
[File: /psf /ps__eegust//sppoooeuessssuuuu lines total)
1:
(Open file: http/pppssssaaaaeeeeeeaaappsasse respsssssseeeeesstsssssssssssan http:/sssssssssspp..o..o ons. py)
(Current director:: request)) bash-$
SWE-agent
psf requests-2317
Now that we have created the 'reproduce_iss pypruduce_issue.py file, let's write script that attempts to send a request with binary method string. which should be handled correctly due to our fix 'sessions.py We'll simple script that imports the 'requests' library, sets a binary method, and tries make request to a test URI.
edit 1:1 import requests
Define a binary method string method b'G
Test URL (cxample) url 'https://ht
Attempt send request with binary method response requests. request (method, url)
Print the response status code print ('Response status code:', response.status_codt
Print a success message if the script completes print ("Script completed successfully, no errors. ") ond_of_edis
Observation
psf requests-2317
[File: /pst_ wwwpeeeesss/ppppcccesssssss (16 lines total) 1: requests
2:
3:# Define binary method string
4:method b'GET
5:
6:# Test URL (example)
7:url = htttps///httbbin.or/eee
54

---

8:
9:# Attempt to send a request with a binary method
r0:response requests . request (method, url)
11:
12:# Print the response status code
13:print ('Response status code:', respense. status_code)
14:
15:# Print succeas message if the script completes
16:print ("Script completed successfully,
errors.")
File updated. Please review the changes and make sure they correct (correct indentation, no duplicate lines, etc). Edit the file again if necessary.
(Open file: /psf_ (Current rictorry: request)) bash-$
SWE-agent
psf requests-2317
The script has been written and should attempt send request using binary method string. This will help us verify if issue been resolved by the changes we made to "sessions.py Let's run the script to see if it executes successfully without encountering the issue described.
python reproduce_isste issue.
Observation
psf requessssssst
Response status code: 200
Script completed successfully, no errors.
(Open file: /pst_ http://pprrcceessseseeeeeeeeeeee (Current directory: /psł requests) bash-$
SWE-agent
psf requests-2311
The script successfully and received 200 OK response, which indicates that the issue with handling binary method strings been resolved. The 'requests' library now correctly processing binary string 'b' GET" as a valid HTTP method.
Since the fix appears to be working, we can clean by removing the pprroduc__isue.ppe script, as it is longer needed. After that, we submit our changes the code base. Let's remove the 'reproduce_isseet jusuucppsssuppppp file.
exproduce_issuee
Observation
psf requests-2317
(Open file: /psf_ rewpsstts/reprrcuue__esssse (Current rirectory: request) bash-$
55

---

SWE-agent
psf requests-2317
The 'reproduce serrodduce_isuuepppee issue.py file has been successfully removed. Now that have verified the fix and cleaned up the test script, can proceed submit our changes the code base. Let's submit changes using the 'submit' command.
submit
Observation
psf requests-2317
diff --git a/requests/eessions.. index . 100644 a/requests/sessiaee arregussts/eessions ee -425, +425, class Session (Bssssieeiiie))))))tt))i)) If Tuple, ('cert', 'key') pair.
b/Eequets//sssions..p
$ Ensure method a string, decode it's binary
string
if isinstance (method, bytes): method = decoed (' aasccill) method builtin_str (methed)
+ Create the Request.
(Open file: /ps reuuesttreprrouce..co issuue.yyssepyy (Current directory: request)) bash-\$
Gold Patch
psf requests-2317
diff --git a/requesta/sessiion..py
prregpests/sesssiosss..
a/requests/sessiinn
+++ prrequests/eessionne @e -13,7 +13,7 @e
from datetime import datetime
from auth import _basis_auth_stz -from compat import cookiclib, OrderedDict, urljoin, urlparse, builtin_str
+from .compat import cookielib, OrderedDict, urljoin, urlparse from coooiie import { bookiejjar_fro_diict extract_cookies_ttg
RequestsCookieJrr merge_cookies) from .models import Request, FreparedRequest, DEFAULT REDIRECT LIMIT ee -425,7 7 +425,7 7 e0 def request (self, method, url, If Tuple, ('cert', 'key') pair.
method builtin str str (methed)
method = to_native_string (method)
$ Create the Request. reg Request
56

---

Analysis for pr11mtt-devpyy1im---599
Issue Summary. The issue involves the -notes option in pylint, which fails to report fixme warnings (W0511) for note tags that are entirely punctuation. Specifically, when rumning py 1 lint with the -notes=" "os==EE "YEs, ???" option, only the "YES" tag is recognized, and the "???" tag ignored. The expected behavior is to recognize the punctuation-only tag normally.
Interaction Summary
Reproducing the Issue:
The agent started by creating test Py file and added comments to replicate the issue described. This was followed by running pylint to confirm the issue, which only reported the "YES" note and not the "???" note.
2. Searching for Relevant Code:
The agent attempted to search for the handling of the -notes option using the search_dir command. After some initial confusion with the command options, successfully identified the pylint/checkers /misc.cc..... py file containing the relevant code.
3. Editing the Code:
The agent located the regular expression responsible for matching note tags within the open: method the Encod i nncchccke class. It modified the regular expression to handle punctuation-only note tags by replacing the word boundary \b with a lookahead assertion
(? [W $)
4. Test Changes with Reproduction Code:
After making the modification, the agent re-ran pylint on the test. py file. The output confirmed that both note tags ("YES" and "???") were correctly identified.
Clean Up and Submit:
py Finally, the agent removed the test py file and submitted the changes.
Evaluation The agent effectively replicated the issue, identified and navigated to the relevant code, made a targeted modification to the regular expression, and verified the fix through testing. While the agent's solution successfully passes the necessary tests, there are slight differences in behavior between the agent's solution and the gold patch. In particular, the agent replaced the Vo word boundary in the regular expression to handle note tags with the regex pattern ('WI mark the tag boundary. Comparing this to the gold patch solution which uses the pattern (: \s \2) in the same way, it may be that the agent's 's solution causes other unintended consequences that aren't captured by the existing tests.
Figure 34: Trajectory for instance pylint serlinttdevv__ee dev -ev dew__yylint-555
Issue
pylint-dev_pylint-5583 pylint-5859
We're currently solving the following issue within our repository. Here's the issue text:
ISSUE:
"-notes" option ignores note tags that are entirely punctuation
Bug description
If a note tag specified with the "-notes" option is entirely punctuation, pylint won't report a fixme warning (W0511).
YES: yes ???: no
pylint test py -notes="YEs, ???" will return a fixme warning (W0511) for the first line, but not the second. Confiuuration ini
Defauit
Command used
shell
57

---

pylint test.py -notes= -notes="YES,?77
### Pylint output
shell
Module test
test. py:l:l: w0511: YES: yes (fixme)
Expected behavior
*************. Module test
post.pp:l::1 N0511: YES: yes (fixme)
te:t.p w0511: ???: no (fixme)
Pylint version
shell
pylint 2.12.2
astroid 2.9.0
Python 3.10.2
(main, Feb 2022, 05:51:25) [Clang 13.0.0 (clang-1 1300.0.29.3) ### OS Envi ronment.
macOS 11.6.1
UP Additional dependencies
No response_
INSTRUCTIONS: (31 lines)
(Open file: n/a)
(Current directory: /pylint) bash-S
SWE-agent
pylint-dev -aiiiidde___plint5555555
To begin addressing the issue, we should first try replicate the bug described. We can do this by creating new Python script that includes the comments with the note tags "YES" and "???" and then running pylint with the n-ootes' option. Let's start by creating file called 'test.py" and adding relevant comments to it.
create test-py
Observation
pylint-5859
[File: 1:
(1 lines total)]
(Open file: n/a)
(Current directory: /pylint-dev p bash-$
58

---

SWE-agent
pylint-dev pylint-5859
Now let's add the example comments from the issue into "lest.py'.
edit 1:1 YES: yes ???: no end of edit
Observation
pylint-dev pylint-5859
[File: hytpt-ddey__lllltttttttttts (2 lines total)]
1:# YES: yes
2:# ???: no
File updated. Please review the changes and make sure they are
correct (correct indentation, no duplicate lines, etc). Edit the file again if necessary.
(Open file: ipylinc-dey___ee balidd_s___llntttttppt (Current directory: /pylint-ddv__y111 bash-$
SWE-agent
Siiintddev__plintttte
The "test.py" file has been updated with the example comments. Next, we should run pylint with the '-notes" option see if we replicate the issue where the second line with punctuation-only tags is ignored.
pylint test py -onte=="Bss ???"
Observation
pylint-dev. pylint-5859
Module test
test cest.py:l:1: N0511: YES: (fixme)
(Open file: /pylint-dev peyn-dddyyyiiitttttt..pp
(Current directory: bash-$
SWE-agent
pylint-dev_pylint-55555
output confirms issue: pylint only reports fixme warning for first line with the "YES" tag and ignores the second line with the "???" tag. This matches the behavior described the issue. The next step is to locate the part of the pylint code that handles the '-notes' option and determine why it's ignoring
tags that entirely punctuation. We should start by finding the file that contains the relevant code. Since we don't know the exact file name, we can search directory for files that might contain string "-notes' to find where the option is handled.
search_dir "--notes"
59

---

Observation
pylint-dev _ylintder__pxlint-58859 pylint-5859
grep: unrecognized option '--notes'
Usage: grep [OPTION]... PATTERNS [FILE]... Try 'grep --help' for more information.
grep: unrecognized option !--notes'
Usage: grep [OPTION].... PATTERNS [FILE]...
Try 'grep --help' for more information
No matches found for "--notes" in /pylint-dev_py
(Open file: (Current directory: bash-$
SWE-agent
pylint-dev_ppyint---ss
scems that the 'search_dis dirr command did not work as expected because "-notes' was interpreted as an option for 'grep' rather than scarch term. To avoid this issuc, we can enclose the search term in quoles to ensure it's treated as literal string, Let's searching again using the correct format for the search term.
search_dd dir "--notes"
Observation
pylint-dev. 0ylint-dev_pylint--5559 pylint-5859
grep: unrecognized option '--notes' Usage: grep [OPTION]... PATTERNS [FTLE]... Try 'grep --help' for more information.
grep: unrecognized option '--notes'
Usage: grep [OPTION]... PATTERNS [FILE]
Iry 'grep --help' for more information.
No matches found for "--notes" in /pylint-dev_
(Open file: (Current directory: bash-$
SWE-agent
pylint-dor_ppylit---8s
appears that the issue persists because search term is still being interpreted as an option. To circumvent this, we can use different search term that is less likely to be confused with command-line option. Let's try searching for just "notes" instead, which should still lead us to the relevant part of the code.
search dir "notes"
Observation
pylint-de__pyliit--553 pylint-5859
Found 24 matches for "notes" in /pylint-dev
hpp11nntdeer_ppp11m////nggiiui (2 matches)
rst (1. matches)
matches)
http::/ww.....i.../////////////a (2 matches)
hpwppitteeee_pppii///pppppiiipia p:i.ar//g.lln//nneeeeesssssssppp (9 matches)
/pylint-dev
pyylittrder.ppphooo///iiitttte (2 matches)
60

---

/pylint-dey http://wwwwaiacaaaaa/aro//aaaaaa /py http://wwwwwaa///e//////////a/aa (2 matches)
matches)
http:///////////////u////a/////a (1 matches)
(2 matches)
/pylint-dev http://www..ppooo///ss/////oooos
(1 matches)
End of matches for "notes" in /pylinn-ddw___ylln
(Open file: jpylint-dew__pplinng (Current directory: jpylint-dev__pylll bash-$
SWE-agent
pylint-dev pylint-5859
The search results indicate that there are several matches for "notes" in the repository. but the most likely candidate for the code need to examine is in pylint/checkers/miss.pp it contains the most matches. This file probably where miscellaneous checkers, including the fixme warning (W0511), are implemented.
Let's open the 'misc.py" file to investigate further.
open pylintt/hhecerrs//iss.
Observation
pylint-der_ppylin---e
[File: htgp://rr.eeee....p////////////e (203 lines
total)]
1:#
Copyright (c) 2006, 2009-2013 LOGILAB S.A. (Paris, FRANCE)
<contact&Loçilab..f
2:# Copyright
2012-2014 Google, Inc.
3:1 Copyright (c) 2014-2020 Claudiu Popa <pcmanticoreégmail.
4:1 Copyright (c) 2014 Brett Cannon <brett@python.oom org>
5:1 Copyright (c) 2014 Alexandru Coman <fcoman@bitde fender.comm
6:# Copyright (c) 2014 Arun Persaud Carunenubati.net.
7:# Copyright
2015 Ionel Cristian Maries <contact@iinnemee..
8: Copyright (c) 2016 lukasz Rogalski <rogalski. waa.skki...ggmai...mmm com>
9:# Copyright (c) 2016 glegoux <gillcs. het.l....eggueemmal..oooom
10:# Copyright (c) 2017-2020 hippo91
com>
11:# Copyright (c) 2017 Mikhail Fesenko <prosggaRgmall.com.
12:# Copyright (c) 2018 Rogalski, lukasz
4 <lukasz.
13:# Copyright (c) 2018 lucas Cimen <lucas. wiaasssccimmmmmmm.cmmm
14:# Copyright (c) 2018 ville skyttä skille.skytta ff
15:# Copyright (c) 2019-2021 Pierre Sassoulas
<plerze.sasoolaeemaa..
16:# Copyright (c) 2020 wtracy kafishionado@gmail. com>
17:1 Copyright (c) 2020 Anthony Scttile <asott vasottile@umico weriiieeemmmiiom
edu> edu
18:# Copyright (c) 2020 Benny <benny mmcllerr91gmail com>
19:4 Copyright (c) 2021 Dani&l van Ncord
(1655331bamillkodddeee. noreply.
seareply.gittub.coom com>
20: Copyright (c) 2021 Nick Drozd <nicholasdrozddgmail comm
21:# Copyright (c) 2021 Mars Mueller
<301303/1+edee@pessee www.eeepesees...rppyy.......
.noreply. wwtreepyygiihuu....m
22: Copyright (c) 2021 Konstantina Saketou
4 <56111305-ksaketouuueeee wtt..e.....orpp...i...uuu...
23:
24:# Licensed under the GPL:
25: For details: https: http///gttttttteeeeee
26:
27:
61

---

return 28: "C"hec code is ascii only has encoding declaration (PEP 263) ***
29:
30: re
31: :1:import tokenize
3ro typing import TYPE_CHBCKING, List, Optional
33:
34:from astroid import nodes
35:
36:from pylint.checkee import BaseChecker
37:from pylint. interfaces import IRawChecker, ITokenChecker
38:m pylint. import ManagedMessage
from pylint. www.ils.pragaapppo PragmaParserfrrro, parse_pragma
import OPTION_FO,
40:
41:if TYPR_CHRCKING:
42:
from pylint.lime import PyLinter
43:
44:
45:class byylalannggedeesssseccheherrrrr (BaseChecker)
46:
47: """Chss for messages that are enabled or disabled by id
instead of symbol.
48:
49: implements IRawChecker
50: name "miscellaneous"
51: mags
52:
"I0023":
53:
"$s", ic-esyya
54:
55:
"Used when message is enabled or disabled by id.",
56:
57:
58:
options ()
59:
60:
def _y (sclf)
(self) -> None:
61:
iyl...Lnttr..b___cmmmcceee clear
62:
63:
def ht__by_id__mnaged__sgg (self) -> List [ManagedMessagee]]
64:
65:
66:
def process module (self, node: nodes Module) None: """Inspect the source sile to find messages activated or deactivated by "nn
67:
68:
managed_msgs - self i
69:
for (mod_name, msgid, symbcl, lineno, is is_disabled) in Basaged_msgsr
70:
is mod name verb = "disable" if is is_disabled else "enable"
name:name:
71:
72:
txt f" (msgidd) is cryptic: use '# pylint:
(verb)= instead"
73:
adlf.add_message hwg.aaaawoecccooooooooooooaaaaaa
line=lineno, args-txt)
74:
self._clear_rr id Baa__iddmaaaee__aaaa_ _sgs
75:
76:
77:class EncoaingChecker (BaseChecker)
78:
79:
"**Checks for:
80:
warning notes in the code like FIXME, XXX
81:
encoding issues.
82:
83:
62

---

84:
implements_
(RawCheecerr, ITokenChecker)
85:
86:
configuration section name
87:
name
88:
msgs
89:
"w0511":
90:
"$s",
91:
"Sixme",
92:
"Used when warning note FIXME or XXX is
detected."",
93:
94:
95:
96:
options
97:
98:
"notes",
99:
100:
"type": "csv",
(103
lines below)
(Open file: /pylint-det wyw____int/ttsttt_e_.eee
(Current directory: /pylint-dov_pplll bash-$
SWE-agent
pylint-dev pylint-dev_ppyiit-55555 pylint-5859
We have opened the 'misc.py" file which likely contains the code for the fixme warning (W051 related to the '-notes" option. Since the file is 203 lines long and currently viewing the first 100 lines, need seroll down the "goto" command find the relevant section that handles the '-notes' option.
Let's scroll down to see more the file.
scroll_down
Observation
pylint-de__pyllit--5553 pylint-5859
[File: /pylint-dev_ htg.:u/www...ppiioo/////////eeee (203 1ines
total)]
(98 more lines above)
99:
100:
"type": "csv",
101:
"metavar": "<comna separated values>", "default"; ("FIXME", "Xxx", "TODO"), "help":
102:
103:
104:
"List of note tags to take in
consideration,
105:
"separated by comma."
106:
107:
b
108:
),
109:
110:
"notes-rgx",
111:
112:
"type": "string",
113:
"metavar": "<regexp>",
114:
"help": "Regular expression tags to take in consideration."
115:
116:
63

---

117:
118:
119:
def open (sel));
120:
super (). ()oppe
121:
122:
notes "I" join (re.escape (note) for note in
cconfig.noter.
123:
if self. cal...cofig.notesssoo rgx:
124:
regex_string =
=f" ((notes) (serfc (Bel www.lll..oo.ig.ont...c.cooo
Ab"
125:
clse: segex_string = =f"#\s* ([notes)) \o"
126:
127:
128:
felf___ime_paatter - cemcomlle (regex_stringg re.
129:
130:
def
131:
seli, lineno: int, line: bytes, sile_encoding str
132:
Optionallstr:
133:
try:
134:
return line.decode (file_encoding except UnicodeDecodett1or:
135:
136:
pass
137:
except LookupError:
138:
is
139:
Line.startswiee (b**")
140:
and "coding" in str str (line)
141:
and file_encoding in str (line)
142:
143:
msg #*Cannot decode using encoding
'[file_encoding ', bad encoding"
144:
self add message ("syntax-error", line-lineno,
args-mag)
145:
None
146:
147:
def grocess_modulle (self, nede: modes.Module -> None:
148:
"""Inspect the source file to find encoding preblen. encoding node, node.filc file_cncoding if node. file_cnecding
149:
else "ascii"
150:
151:
with sede.stream/ () as stream:
152:
for lineno, line in enumerate (stream)
153:
sheck enceding [innno 1, line,
encoding)
154:
155:
def process_tokens (self, tckens):
156:
"""Inspect the source to find fixme problems."" "H* """ if not self. return
157:
158:
159:
comments =
160:
token_info for tofon_info in tokens if token info.type == COMMENTT.COMMENT
161:
162:
for comment in comments:
163:
comment_text comment. comment.sttingg htt.mentt..rinngl.......... # trim 1 and whitespaces
164:
165:
handle pylint eisable clauses
166:
disable_option match
OPTION PO. search (comment _tmemeen___ex)t)___ text)
167:
if :
168:
try:
169:
values []
170:
try:
64

---

171:
for pragma repr in p_rep for p_rep in
172:
173:
parse_pragma (disable_opt www.pppwwoommtaoooop...... (2))
174:
if p_rep.action "disable"
175:
176:
values.extenis (pragma wwwpgaa_repraa______
wew.aar.rr......ee.s...e.. messages)
177:
except PragmaParserError:
178:
# Printing useful information dealing
with this error is done in the lint package
179:
pass if set (values) & set (self. cee.f..onfig.ntte..... continue
180:
181:
182:
except Valuekrror:
183:
sel ftlffadd_mesaapee bbad-inline-optinn
184:
185:
args=disable_ option line=ccmment start
strppp (),
186:
187:
188:
continue
189:
190:
# emit warnings if necessary
191:
match
(**"
comment_textt ())
192:
1s match:
193:
self.add_
194:
"fixme",
195:
col col_offeet-coment.star 1, a args=comment_texx,
196:
197:
iine=comennt.saad [0],
198:
more lines below)
(Open file: httpr-oow_..liiioggg////aaaaaatr
(Current directory:
bash-$
SWE-agent
pylint-der_pyylin---t
We found relevant section the 'misc.py' that handles the '-notes" option. The regular expression pattern used to match note tags is constructed in lines 122-128. It seems that the issue might be related to how the regular expression is constructed or how it is used match comments in the source code.
The regular expression is built from the 'notes' configuration, which list comma-son separattd values, and an optional 'notes-rgx' regular expression. The pattern is then compiled and used in the 'process_tokens ttkeets method to search for matching comments.
The issue might be that the regular expression pattern does not account for note tags that consist entirely of punctuation characters. To fix this, we may need to adjust the regular expression pattern ensure matches these tags correctly.
Before we make any changes, let's examine the regular expression pattern construction in more detail. We should look at the 'open' method where the pattern is compiled, specifically lines 122-128.
goto 122
65

---

Observation
pylint-dev pylintdeww_pylint-5589
[File: wttp://pww..tttyyyt///////ttt/aa (203 lines
t
(103 more lines above)
104:
"List note tags to take in
consideration,
105:
"separated by comma."
106:
107:
108:
)
109:
110:
"notes-rgx",
111:
112:
"type": "string",
113:
"metavar": "<regexp>",
114:
"help": "Regular expression of note tags to take in consideration.",
115:
116:
)
117:
118:
119:
def open (self):
120:
super () oppen
121:
122:
notes = "I". "j"njoin (re.escape (note) for note in
www.iioonniii..o........oeoo...o notes)
123:
if www.coiignnoot.s.e..
124:
regex_string
4
rf"
s* ((notes) (self config notes__ggx))
( \b"
125:
else:
126:
regex string = rf" ss ( \o"
127:
128:
self. _els._fixm fixme pattern paterr re. cempilll (regex (regex_string re.I)
129:
130:
def _wchccceehcccinnnnnnee
131:
self, lineno: int, line: bytes, file_encoding: Optional
132:
133:
try:
134:
return line.decode (file_encoding) except UnicodeDecooeerrro:::
135:
136:
pass
137:
except LookupError:
138:
if
139:
line.startswine (b**")
140:
and "coding" in str str (line)
141:
and file sele_encoding in str str(nine)
142:
143:
msg = f"Cannot decode using encoding
'(file_encoding] ', bad encoding"
144:
self add message ("syntax-orror", line=lineno,
args=mag)
145:
return None
146:
147:
def soteesssmodulesseee (self, node: Modulle -> None:
148:
."" Inspect the file to find encoding ng problem. ... encoding node.file encoding if file encodingg
149:
else "ascii"
150:
151:
with stream) stream:
152:
for lineno, line enumerate (stream) self._checc_encodii (1 (lineno + 1, line,
153:
encoding)
66

---

154:
155:
def srocess_tokens (self, tckens)
156:
"*"Inspect the source to find fixme problems. if not return
157:
158:
159:
comments
160:
token_info for token info in tokens 6 token info.type = tokenize.
161:
)
162:
for comment in comments:
163:
comment_text comment. string (sstrip()))))) trim 't' and whitespaces
164:
165:
# handle pylint disable clauses disable_option_me
166:
OPTION_PO search (comment_text)
167:
is desable_pptionnneee match:
168:
try:
169:
values
170:
try:
171:
for pragma_repr in p_rep
172:
173:
for p_rep in www.mmttttcccup/////m
parse_pragma (disaaleeeeis
(2))
174:
if p_rep.
"disable"
175:
):
176:
values. exieess (pragma repar (eepr.esssges)
177:
except PragmaParserErree Printing useful information dealing
178:
4 with this error is done in the lint package
179:
pass if set (values) & set (self. (ssl...conig..netes)
180:
181:
continue except ValueError:
182:
183:
self, aelf.add_message
184:
"bad-Inline-gption",
185:
argg=-disable__ptionn www.maatccccc.cccc http:/abe....pppooo.maoooooooole
186:
line=comment. start [0],
187:
188:
continue
189:
190:
emit warnings it necessary
191:
match semfs__fimm_pattee..seee (*4" +
comment_text. lower
192:
if match:
193:
self.add message
194:
"fixme",
195:
sal_offset=ccoment.sstttt [1] + 1,
196:
args=commcnt_text,
197:
Iineccoment.starrll0)
198:
199:
200:
201:def register (linter: "pyliineer") None:
202: Linter.reiister_cheeeeee checker (Encoding(hecker (linter))
203:
(linter)
(Open file: boylinttdew__pyliittttte pwintt/pgginn/hhheeee////ec..
(Current directory: /pylint-dev_
bash-$
67

---

SWE-agent
pylint-dev pylint-dev_pyyint---33 pylint-5859
We are now looking at the section of the 'misc.py" file where the regular expression for the "-notes' option is constructed. The regular expression is built from the 'notes' configuration option, which a list note tags. These tags are escaped using "re.escape" and joined with a pipe to form an alternation pattern. This pattern then used to create a regular expression that matches comments starting with one of the note tags followed by word boundary 'lb'.
The issue might be with the word boundary 'lb', which does not match if the tag is followed by punctuation. Since the tag "???" is entirely punctuation, the word boundary may not be recognized correctly, causing To fix this issue, we can modify the regular expression to ensure that it matches note tags that are entirely punctuation. We can do this by removing the word boundary and allowing the tag to be followed by any non-word character the end of the string. Let's edit the regular expression construction fix the issue.
tag not to match.
edit 124:126
regex_string
r ((notes) | (self, config. ) (?-\W15)
else:
regex_atring sf* ((notes)) (?-\W1S)*
end_of edit
Observation
pylint-der_ppylitt--3
[File: http://www.........///////////aa (203 lines
total)]
(73 more lines above)
74:
self. by id managed_msgss msgs()
75:
76:
classass EncodingChecker (BaseChecker)
78:
79:
Checks for:
80: warning notes in the code like FIXME, XXX
81: encoding issues.
82:
83:
84:
implements_ (IRawChecker, ITokenChecker)
85:
86: configuration section name
87: name "mi accelaneouus"
88: msgs
89:
"w0511":
90:
"$s"
91:
"Sixme",
92:
"Used when warning note as FIXME or XXX is
detected.",
93:
94:
95:
96:
options
97:
98:
"notes",
99:
100:
"type": "csv",
101:
"metavar": "<comma separated values>", "default"; ("FIXME", "xxx", "TODO"), "help":
102:
103:
104:
"List of note tags to take in
consideration,
68

---

105:
"separated by comma." "
106:
107:
108:
)
109:
110:
"notes-rgx",
111:
112:
"type": "string",
113:
"metavar": "<regexp>",
114:
"help": "Regular expression of note tags to take in consideration.""
115:
b
116:
)
117:
118:
119:
def open (self):
120:
super () open
121:
122:
notes "I" join (re.escape (note) for note in self.con notes)
123:
if self.configg... sels.config.notes_Pe
124:
regex_string
+s* (isictes//sll..ooooigg....
(?=\W1S)"
125:
else:
126:
regex_string rf" s** ((notes)) (?=WW1S)
127:
128:
self _ iixme_pattern = re. compile (regex_string re.I)
129:
130:
def chcckhckkeec _check_encodingl
131:
self, lineno: int, line: bytes, file_encoding: str Optional try:
132:
133:
134:
line. wiileeeeeoodiieeeeeeeeeeeeeeeeee except UnicodeDecodeError: pass
135:
136:
137:
except LookupError:
138:
if
139:
Line.startswith (b"#")
140:
and "coding" str str (line)
141:
and file_encoding in str (line)
142:
):
143:
msg t"Cannot decode using encoding
/ffile_encoding/ ', bad encocing"
144:
self.ad add mered_messeeeeeee ("syntax-error", line=lineno,
args-mag)
145:
return None
146:
147:
def process_module (self, nede: nodes. Module) -> Nene: """Inspect the source file to find encoding problem. encoding = node. file_cncoding if node. filc_enceding else "ascii"
148:
149:
150:
151:
with soree.strem//. () as stream:
152:
for lineno, line in enumerate rstm) self._check_ente bececcceeecdinggeeee (nnnno + 1, line,
153:
encoding)
154:
155:
tokens (self, tckens) def """Inspect the source to find fixme problems, ***
156:
157:
if not self. celf.coofig.notes.. notes:: return comments =
158:
159:
69

---

160:
token_infs info for token info in tokens ir ceseeizeecoMMEN.
token inpo.type
161:
)
162:
for comment in comments:
163:
comment_text comment. (triin:n.ttring::)..... '4' and whitespaces
trim
164:
165:
# handle pylint disable clauses
166:
disable_option matp__matchh_ -atte_pottooonccchhhhe
OFTION_FO, search (comment_text)
167:
if dieable_optionmmttch:: try:
168:
169:
values []
170:
try:
171:
for pragma_repr in P_rep
172:
173:
for p_rep in
parse parse_pragmaa_aaeeaaapaa pragma ttdisable_pptions match. orop (2))
(30 more 1ines below)
File updated. Please review the changes make sure they are
correct (correct indentation, no duplicate lines, etc). Edit
the file again if necessary.
(Open file: /pylint-dev. http://ppph..eeeeuep///////////p hwwp:////wyiiim///httttt///sssap (Current directory: pey bash-$
SWE-agent
The edit to the regular expression has been made successfully. We replaced the word boundary "bb' with lookahead assertion '(==WIS)" that matches either non-word character the end of the string after the tag. This should allow the regular expression match tags that consist entirely of punctuation.
Before submitting the changes, we should test to ensure that the fix works as expected. We can do this by running pylint the 'test.py' file again with the "-notes" option to see if it now correctly identifies the "???" note tag.
Let's run pylint "test.py" with the updated code.
pylint test --notes= "YES, ???"
Observation
Seiittdder__ylin---eee
Module test
test post.py:::l1 W0511: YES: yes (fixme)
test :1:2:: W0511: ???: no (fixme)
(Open file: ipylint-der int_ddyyyyyy/yyyyllll wttpiiityylyttteeeeeeeeeeeeeeeee (Current directory: dey bash-$
SWE-agent
pylint-dew__pylin--58533
The pylint output now shows that it correctly identifies both the "YES" and "???" note tags, which means the fix has resolved the issue. With the bug fixed, we can proceed clean up any test files and then submit changes the code base. First, let's remove the "test.py" file since was only needed for reproducing the issue.
70

---

test.py
Observation
pylint-dev -ylii-dee__
(Open file:
(Current directory: eyylint-de__pyllie bash-$
SWE-agent
The 'test.py" file has been successfully removed. Now that we have fixed the issue and cleaned up the test files, can submit our changes the code base. Let's submit the changes using the 'submit" command.
submit
Observation
pylint-dev_pyllingg pylint-5859
diff --git index a/pyLint/chhcceer///lse/ http:/int/checcess/iisss..
100644
ee -121, +121, 9 00 class EncocingChecker (BaseChecker))
notes "I". (rre.eap (note) for note in colf.config.nottes)
if sclf.coofig.notee..er
regex_string
rf"#\s.
\b"
regex_string
rf" ss* (saf:e])]sa...coffi.nottttt. (?=\W1S)"
eise:
zegex_string = ([notes)) \b"
zegex_string = ([nctes)) (?=\W1$)
self self.__ixme_patter compile (regex_string, re.1)
Gold Patch
pylint-dev pylint-5859
diff --git a/pyyiinc/chcccerr//icc.. s/pplimt//ccccens//llc..ppp +++ //ppylant/cheeckrs///isss
b/pptiint//heccees//ssssc..
ee -121,9 +121, 9 00 def open
notes "I" (rr.join (note) for note in 4 sel: cols.coonig it sels.config.notes__er
regex_string
rf" rf* (ftcceeellls....ooofggoooooeeee
\b"
regex_string
If" s4 ((ntes)) oee..eeallsssl....ooiiinnnnees [gxi) (?=(:)
else:
regex_string
rf* ((nctes) \b"
71

---

regex_string ((notes)) (?= (: N
_itt___ixmee pattern = re compile reeee__string re II
Analysis for sympy/sympy-2166
Issue Summary. The issue is reporting an UnedinedKind type returned when printing sympy Derivative kind attribute. They expect something like a Der werrivitiieeiiii instead of UndefinedKind
Interaction Summary
1. Reproduce the Issue:
The agent began by creating Python script reeproducejissee.e py) reproduce the issue, using the reproduction code provided in the issue report. It ran the script and successfully reproduced the issue.
2. Searching for Derivative Code: The agent begins to search for code where the Derivative class is defined. It makes several attempts with different strategies to find the relevant code. After a few unsuccessful approaches, looking for particular filenames, searches simply for "class Derivative", which provides single search result. It continues to read the Derivative class definition, which spans many hundreds of lines.
3. Exhausting Budget: It continuously scrolls through the funct py file where Derivative defined until is exhausts its budget and terminates the episode without submitting.
Evaluation The agent makes few mistakes using the search function effectively. The agent should have made more efficient searches by using more commonsensical patterns (c.g. searching for "class Derivative") early on, instead of more speculative search attempts. In this particular setting, the gold patch shows that the updated code involves overriding an inherited property kind from the sympy.basic object. This fix would require more thorough understanding of the codebase and inheritence structure of Derivat ive to be effective. An agent attempting to resolve this issue would need have more targeted and efficient search approaches improve its ability identify this problem.
Figure 35: Trajectory for instance sympy_sympps - sympy-21614.
Issue
sympy sompy_sympy-2261
We're currently solving the following issue within our repository. Here's the issue text:
ISSUE:
Wrong Derivative kind attribute
I'm playing around with the 'kind' attribute.
The following correct:
from sympy import Integrai, Derivative
from sympy import MatrixSymbol
from sympy. abc import Matrixsymbol ('A', 2, Integral (A, x) i.kind
# MatrixKind (NumberKind)
This wrong: d Derivative (A, x)
d.kind UndefinedKind
INSTRUCTIONS: (31 lines)
(Open file: n/a)
(Current directory: /sympy sympy) bash-$
72

---

SWE-agent
sympy sympy-21614
First, I'll create new Python script to reproduce the issue as described. This script will attempt to create 'Derivative' of 'MatrixSymbol" and print out the 'kind' attribute verify if it indeed 'UndefinedKind..
create reproduce issue.su... .py
Observation
sympy sympy__mmp
[File: /sympy sympy/reproduce issue py (1 1ines total)
1:
(Open file: n/a)
(Current directory: /sympy_syyyyg bash-$
SWE-agent
sympy sympy-21614
Now let's paste in the example code from the issue that demonstrates the incorrect behavior for the 'kind' attribute of a 'Derivative' of 'MaarixSymbol'.
edit 1:1
from sympy import Integral, Derivative from sympy import MatrixSymbol from sympy.abc import
A MatrixSymbol ( ('A', 2, 2)
= Derivative (A, x)
print (d.kind)
Expected: Matrixkind (NumberKind), but getting Undefi nedk end_of_ed
Observation
sompy__smmy-2261
[File: gasmm/__syyprrrrrodcceiiiss py (7 lines total)]
1:from sympy import Integral, Derivative
2:from sympy import MatrixSymbol
3:from sympy.abe import x
4:A MatrixSymbol ('A',
2)
5:d Derivative (A, x)
6:print (d.kind)
7:4 Expected: MatrixKind (NumberKind), but getting UndefinedKind
File updated. Please review the changes and make sure they are (correct indentation, duplicate lines, etc). Edit the file again necessary.
(Open file: /sympy
(Current directory: /sympy sympy) bash-$
SWE-agent
sympy sympy-21614
The reproduction script has been created and updated with the code that demonstrates the issue. Now, let's run the script to confirm that the 'kind" attribute the 'Derivative' is indeed UnndffiedKind'.
73

---

python reproduce issue.s.... py
Observation
sympy sympy-21614
UndefinedKind
(Open file: (esmppp_s/anyyreprccoooluuuuuu (Current directory: /sympy_yynpy) sympy) bash-$
SWE-agent
sympy sympy-21614
The output confirms that the 'kind" attribute of the 'Derivative' is 'UndefinedKind'' which is not the expected behavior. The next step locate the part of the code that handles the 'kind' attribute for the "Derivative' class and make the necessary corrections.
Since the 'Derivative' class is likely to be defined a file related to calculus or diffcrentiation, will scarch for the file that contains the 'Derivative' class definition.
find find_file file "derivative.py"
Observation
sympy Hompy_sympy-21611
No matches found for "derivative.py" in /sympy sympy
(Open file: /sympysyyyyypeeeeeepeee hrmpy___ssmyyrprroducessssss
(Current directory: _ympp) bash-$
SWE-agent
sympy sympy__ymp-221114
seems that the file name I searched for does match the actual file name where the 'Derivative' class defined. 'Derivative' class might defined in file with a different name, possibly something more general related to symbolic calculus in SymPy.
Let's try broader search for files that might contain the 'Derivative' class definition. Since the class name 'Derivative, it's likely that the file name contains "calculus" or similar keyword. I'll use the "scarch_dir" command to look for the "Derivative" within the Python files in the current directory.
search_dis "Derivative"
Observation
sympy sympy_2ym
More than 182 files matched for "Derivative" / /sympy sympy. Please narrow your search.
(Open file:
(Current directory: /sympy_sympy sympy) bash-$
SWE-agent
sympy Syppy___mmpy2111444
The search for "Derivative" returned too many results, need to narrow it down. Since the 'Derivative' class is core part of SymPy, it's likely to be in module. A common convention is to have such
74

---

core classes in file named after the module, so I will search for a file that might be named 'core.py' or similar.
file "core.py"
Observation
sympy sympy_ssmmphl Sympy-2111
Found matches for "core. py" in /sympy sympy:
/sympy_ /wwp://ymmp//uunttt///err...//
/sympy hytp://wmpp/stttaaaeeeeeeraaaaaa /sympy swmr://smt///ttaateess//rrrrrr
(Open file: //tppy/_s/////rpppppuuuuussssoo (Current directory: /sympy_symygygpppp
bash-$
SWE-agent
sympy sympy-2100 21614
The scarch has returned several 'core.py" files, but the most likely candidate for containing the 'Deriva- tive' class definition within the main 'sympy/core' directory. Let's open the 'core.py" file the 'sympy/core' directory to investigate further.
open sympy/core/core.pp
Observation
sympy sympy-21614
[File:
(104 lines total)
1: 1nn The core's core.
2:
3:# used for canonical ordering of symbolic sequences
4:# via _emp_ method:
5:# FIXME this is *s0* irrelevant and outdated!
Seordering_of_classee
7: singleton numbers
8: 'Zero', 'One', 'Half', 'Infinity', 'NaN', 'Negati veOne''
'Negatives Songativinfinityy, numbers
9:
10:
'Integer', 'Rational', 'Flcat', singleton symbols
11:
12:
'Expl', 'Pi', 'Inaginarynnit', symbols
13:
14:
'Symbol', 'wild', 'Temporary', arithmetic operations "Po'', 'Mul', 'Add',
15:
16:
17:
function values "erivattve', 'Integral', defined singleton functions 'Abs', 'Sign', 'Sqrt', 'Fioor', 'Ceiling', "Re', 'im', 'Arg', 'Conjugate', 'Exp", 'Log', 'Sin', 'Cos', 'Tan', 'Cot', 'ASin', 'ACos', 'ATan', 'ACot',
18:
19:
20:
21:
22:
23:
24:
25:
26:
"Sinh', 'Cosh', 'Tanh', 'Coth', 'ASinh', 'ACosh', 'Aaanh',
'ACoth',
75

---

27: "BisingFactorial', 'FalLinggactoriall,
28:
"factorial', 'binomial' "Gamma', "LowerGamma', 'UpperGanna', 'PolyGamma', 'Brf',
29:
30:
31:
special polynomials
32: "Chebyshev', 'Chebyshev2',
33: undefined functions
34: 'Function', 'MildFunction',
35: anonymous Eunctions
36: 'Lambda',
37: Landau symbol
38: 'Order',
39: relational operations
40:
'Rquality', 'Unequality', 'stricttreaterrThan"
'Stricttlessthan',
41:
'GreaterThan', 'lessThan',
42:1
43:
44:
45:class Registry:
46:
47:
Base class for registry objects.
48:
49:
Registries map
object using attribute nctation.
4 Registry
50: classes behave singletonically: all their instances share
the same state,
51: which is stored in the class object.
52:
53: All subclasses should set slets_ ()
54:
55: slots_ = ()
56:
57: def setattr (self, name, obj):
58:
(sel.
name, obj)
59:
60:
def deelaatrrssslllle name):
61:
delattr (self. (self.__lllleeeee..e__ class name)
62:
63:#A set containing all sympy class objects
ss:all_clases se
65:
66:
class NasicMeta (type)
68:
69:
def init (cls, *args, **ww)
70:
all add cls. cls._sypp sympy___p__..... property (lambda self: True)
71:
72:
73:
def cmp (cls, other):
74:
If the other object is not Basic subclass, then are not equal
75:
it.
76:
if not isinstanc
RasicMeta):
77:
return -1 ml cls. name
78:
19:
n2 other. name
80:
if nl == n2:
81:
return
82:
83:
UNKNOWN (orreeringgg__ccclsssss)))
84:
try:
85:
il = ordering.of_classe index (n1)
76

---

86:
except ValueError:
81:
UNKNOWN
88:
try:
89:
12 ordering_of_classee index (n2)
90:
except Valustrror:
91:
UNKNOWN
92:
UNKNOWN and i2 UNKNOWN:
93:
return (nl n2) (nl n2)
94:
return (il i2) (il 12)
95:
96:
def lt_ (cls, other):
97:
if cls
(other) -1:
98:
return True
99:
return False
100:
(4 more lines below)
(Open file: /sympy sympy/reproane issue.py) (Current directory: /sympy sympy) bash-$
SWE-agent
sympy sympy-21614
The 'core.py" file in the 'sympy/core' directory does the 'Derivative' class. It's more likely that the implementation is in another file within the 'sympy/core" directory. Since 'Derivative' class is fundamental part of symbolic calculus, it might named something like 'function.py' or 'operations.py" Let's try to find a file that specifically contains the implementation of the 'Derivative' class within the 'sympy/core' directory.
contain the implementation details of
find file "function.p py" sympy/core
Observation
sympy sympy-21614
Found 1 matches for "function.py" py" in hatp:p/s_ssyy//yyyyyooooaoaa
(Open file: /sympy /wwppaaasypyy///aaaaaaaaaaaaaaat http://////////////////or///////
(Current directory: /sympy_sympy) sympy) bash-$
SWE-agent
sympy sympy-21614
Great, we have found the 'function.py" the 'sympy/core' directory, which likely to contain the implementation of the 'Derivative' class. Let's open this file to investigate and make the necessary changes.
open sympy/core//unntioo ion.p
Observation
sympy sympy-21614
[File: //mppyssssws///s/rsr///////neeee (3389 lines tctal)] ] 1nn
There are three types of functions implemented in SymPy:
3:
77

---

4: detined functions (in the that they be evaluated) like
5: exp or sin; they have a name and a body:
6:
f exp
7:
2) undefined function which have name but no body.
Undefined
8:
functions can be defined using a Function class as
Follows:
9:
f Function('F) ('E')
10:
(the result will be Function instance)
11:
3) anonymous function (or lambda function) which have body (defined
12:
with dunny variables) but have name:
13:
f Lambda (x, (xp(x)+x)
14:
f Lambda (x,, y), (x)
15:
The fourth type of functions are composites, like (sin + cos) (x); these work in
16:
SymPy core, but are not yet part of SymPy.
17:
18:
Examples
19:
20:
21:
>>> import sympy
22:
= sympy. ("f")
23:
>>> from sympy.abs import
24:
>>> (x)
25:
f(x)
26:
print (sympy, (f (x) ffnc))
27:
ionctionn'E:
28:
>>> fxx)
29:
(x,)
30:
31:"""
32:
33:from typing import Any, Dict as tDict, Optional, Set as tSet, Tuple as tTuple, Union
34:
35:from aadd import Add
36:from assumptions import ManagedProperttis
37:from bbssic import Basic, _atomic
38:from cache import cacheit
39:from
.ompatibiiitt import iterable, is is_sequence, int, 4 orderea, Iterable
40:from
.decorators import -sympifyit
41:from exxpr import Expr, Atomi cExpr
42:from
numbers import Rational, Float
43:from
.operations import Lattice0p
44:from
.rules import Transform
45:from
.singleton import
46:from
.sympify import sympify
47:
f8:m sympy.core
wot..ycocrccooinerr........ containers import Tuple, Dict from sympy.core.
import global_parametee 50: from sympy.core. import fuzzy_and, fuzzy_or, fuzzy_not, PuzzyHool
51:from uympy.utilities import default_ soruttt_sort_tte key
52: from wtw.:uttiiiiies.e.ettiiiie import Symtybporccttaalaannnnn
www.uutilities..teeaaaaoo import has__ups dups, sift 54: from sympy.utilities.ntte misc import filldedent
53 : from
55:
56: import mpmath
57: import mpmath. 1ibmp as mlib
58:
78

---

59: import inspect
fr:from collections import Counter
61:
62:def _coeff_isnegle
63:
***Reuurn True if the leading Number is negative.
64:
65:
Examples
66:
67:
68: >>> from sympy. core. function import _cceff_finneeec
69:
from sympy import s, Symbol, co, pi >>> (cocff_isneg (-3+pi)
70:
71:
Truc
72:
>>> _coef _coff__iseee )3))
73:
False
74:
>>> _coff ssee (-0o)
75:
True
76: >>> _coff isneg (Symbol ('n', negative=Truel) # coeff is 1
77:
False
78:
79:
For matrix expressions:
80:
81:
>>> from sympy import Matrixsymbol, sqrt
82: >>> MatrixSymbol ("A", 3,
83: >>> __coff__sneeeee (-gart (2) =A)
84: True
85: >>> _coef=_ (sqrt (2) +A)
86: False
87: =**
88:
89: if a. a.is_MatMul:
90:
= [0]
91:
if a.is_MMll Mul:
92:
args [0]
93:
return a. N.mberumber and a. a.ie_cxteeded_negativv
94:
95:
96:class PoleError (Exception)
97:
pass
98:
99:
100:class ArgumentIndexfen (ValueError):
(3289 more lines below)
(Open file: httpa//ssssmm//yyyoo//oo/////rr/ (Current directory: /sympy_sympy bash-$
SWE-agent
sympy Sympy-2111
The 'function.py" file is now open, and it's quite large with 3389 lines. The 'Derivative' class is likely to he defined in this file, but need find the exact location. Since the file too long scroll through manually, I'll use the 'search_file command to find the 'Derivative' class definition within file.
search file "class Derivative"
79

---

Observation
sympy syp__symp-22111
Found 1 matches for "class Derivative" in /sympy_yyypeeepp hynoyyssyyy//yyy///orrrre function.pe
Line 1053:class Derivative (Expr):
End of matches for "class Derivative"
(Open file: /sympy /ttppaaaaapyy///yaaaaaaaa smmp//smmyy/occe//auttioo...pp (Current directory: sympyysympy) bash-$
SWE-agent
sympy sympy-21614
We have found that the 'Derivative' class is defined line 1053 of 'function.py' Let's navigate that line inspect the implementation and determine how correct the 'kind' attribute issue.
goto 1053
Observation
sympy sympy-21614
[File: http://ww////////////////tccaca/ (3389 lincs tctal) ]
(1037
more lines above)
1038:
nargs ias_int (nargs),) nargs FiniteSet (*nargs)
1039:
1040:
nargs nargs
1041:
1042:
def matches (self, expr, repl dict=l), old=faise):
1043:
if not isinstance (expr, (AppliedUndef, Function))
1044:
if return None
1045:
not in self.nargs:
1046:
return None
1047:
1048:
repl_dict repl_ repl_dict.copy repl_dict [self] expr return repl_dict
1049:
1050:
1051:
1052:
1053:class
Derivation ive (Expr)
1054:
1055:
Carries differentiation of the given expressicn with
respect to symbols.
1056:
1057:
Examples
1058:
1059:
1060:
from sympy import Derivative, Function, symbels, Subs >>> from sympy. import x, y
1061:
1062:
>>> f, g symbols g', cls-Function)
1063:
1064:
>>> Derivative x, evaluate=True)
1065:
2+x
1066:
1067:
Denesting of derivatives retains the ordering of
variables:
1068:
1069:
>>> Derivative (Derivative y), y), x)
1070:
Derivative (f (x, y), Y, x)
1071:
80

---

1072:
Contiguously identical symbols merged into tuple
giving
1073:
order of the variables of differentiation will be made the symbol and the count:
1074:
1075:
>>> (erivative(f 8, (erivative (x, 2), Yr x)
1076:
1077:
1078: If the derivative cannot be performed, and evaluate is
True, the
1079:
canonical:
1080:
1081:
>>> Derivative (x, y), y, evaluate=True) (f y),
1082:
1083:
1084:
Derivatives with respect to undefined function can be
calculated:
1085:
1086:
Derivative +22, (xx), evaluate=True) (x)
1087:
1088:
1089:
Such derivatives will show up when the chain rule is used
to evalulate derivative:
1090:
1091:
1092:
>>> ((g(()) (xff
1093:
Derivative (f (g (x)), g ) +Derivative (g (x), x)
1094:
1095:
Substitution used represent derivatives of functions
with arguments that are not symbols or functions:
1096:
1097:
1098:
12*x (x) 2*Subs (f (y) .diff y, 2*x
3)
1099:
True
1100:
1101:
Notes
1102:
1103:
1104:
simplification of high-order derivatives:
1105:
1106: Because there be significant amount of
simplitication that can be
1107:
done when multiple differentiations are performed, results
will be
1108:
automatically simplified in fairly conservative fashion unless the
1109: keyword "simmlify"" set to False.
1110:
1111:
>>> from sympy import sqrt, diff, Function, symbols abe abe import y,
1112:
>>> from sympy
1113:
g symbols ('f,g', cls-Function)
1114:
1115:
>>> e sqrt ((x + 1)++2 x)
1116:
>>> diff (e, (x, 5), fy count_eps
1117:
136
1118:
>>> diff (x, 5) co).count_ops()
1119:
30
1120:
1121:
Ordering of variables:
1122:
1123:
If evaluate is set to True and the expression cannct be
evaluated, the
81

---

1124:
1ist differentiation symbols will be sorted, that is, the expression is
1125:
assumed to have continuous derivatives up to the order
asked.
1126:
1127:
Derivative wrt non-Symbols:
1128:
1129: For the most part, one may not differentiate wrt non-symbols.
1130:
For example, do not allew differentiation wrt "xxy"
because
1131:
there are multiple ways of structurally defining where x*y
appears
1132:
in expression: very strict definition would make
1133:
(xxysz) diff (xxy) 0. Derivatives wrt defined functions
4 (like
1134:
cos cos(x)) are not allowed, either:
1135:
1136:
(x*y*2) diff (xxy)
1137:
Traceback (most recent last):
(2252 more lines below)
(Open file: /sympy swmp://ymnyy/coee//ucccooo..pp
(Current directory: /sympy_sympy) bash-$
SWE-agent
sympy serp_ssmmpy2111111
We the beginning of the 'Derivative' class definition. To address the issue with the 'kind' attribute, we need where 'kind' property defined or handled within this class, Since file is large, I will scroll down to find the relevant section where the 'kind' property is likely to be implemented.
scroll_dow down
Observation
sympy sympy22111
[File: hwtmm///wwwm////yy/////////nntno
(3389 1ines tctal)]
(1135 more lines above)
1136:
>>> (xxy*2) .diff (xxy)
1137:
Traceback (most recent call last):
1138:
1139:
ValueError: Can't calculate derivative wrt xxy.
1140:
1141:
To make it easier to work with variational calculus,
however,
1142:
derivatives wrt ApplicdUndef and Derivatives are allowed.
1143:
For example, the Lagrane method one may write
1144:
F(t, u, v) where u = f(t) and v (t) These variables
can be
1145:
written explicitly
functions of time::
1146:
1147:
>>> from sympy.ad abe import t
1148:
>>> F' Function ('F')
1149:
>>> U f(t)
1150:
U.diff(t)
1151:
1152:
The derivative wrt f(t) can be obtained directly:
1153:
82

---

1154:
>>> direct (t, V) ddifff
1155:
1156:
When differentiation wrt non Symbol is attempted, the
non- MonnSymbol
1157:
is temporarily converted to Symbol while the
differentiation
1158:
is performed and the same answer is obtained:
1159:
1160:
>>> indirect (t, U, V) subs (U, .diif(()) subs (x,
1161:
assert direct indirect
1162:
1163:
The implication of this nen-symbol replacement is that all
1164: functions are treated as independent of other functions
and the
1165:
symbols are independent the functions that contain
them::
1166:
1167:
(E)
1168:
0
1169:
>>> (x)) (x))
1170:
0
1171:
1172:
It also that derivatives assumed depend only
1173:
on the variables cifferentiationn, not on anything
contained
1174:
within the expression being differentiated::
1175:
1176:
f(x)
1177:
>>> Fx (x)iff
1178:
>>> (x.diff()) $ derivative depends on x, net F
1179:
0
1180:
Fxx (x.diff(x)
1181:
>>> Fxx. derivative depends on x, not Fx
1182:
0
1183:
1184:
The last example be made explicit by showing the
replacement
1185:
of Fx Fxx with y:
1186:
1187:
>>> Fxx, subs (Fx, y)
1188:
Derivative (y,
1189:
1190:
Since that in itself will evaluate to zero, differentiating
1191:
wrt Fx will also be zero:
1192:
1193:
>>> ()it
1194:
0
1195:
1196:
Replacing undefined functions with conerete expressions
1197:
1198:
One must be careful to replace undefined functions with
expressions
1199:
that contain variables consistem with the function definition and
1200:
the variables of differentiation or else insconsistent
result will be obtained, Consider the following example:
1201:
1202:
1203:
eq xg)
1204:
>>> eq. subs x*y) (iff doiti ()
1205:
(yDer)vaativeegl))), y) + gly)
83

---

1206:
eq (x, subs (x),,, doit
1207:
y=Derivative y)
1208:
1209:
The results differ because (x(x" was replaced with an expression
1210:
that involved both variables differentiation. the
abstract
1211:
case, differentiation of (xx)" by "y" is 0₃ in the concrete case,
1212:
the presence of "y" made that derivative nonvanishing and
produced
1213:
the extra "g(y)" term.
1214:
1215:
Defining di fiferentiatin for object
1216:
1217: An object must define Lowal_derival (symbo1) method
that returns
1218: the aifferentiation result. This function only needs to
consider the
1219: non-trivial case where expr contains symbol and should 4 call the diff
1220: method internally (not _eval_derivative)) Derivative
should be the only
1221:
call _eval_derivatiie.
1222:
1223:
Any class can allow derivatives to be taken with respect
to
1224:
itself (while indicating its scalar nature). See the
1225:
docstring of dxpr._diff_rrt.
1226:
1227:
See Also
1228:
====s=s=
1229:
sort_varriable_cout
1230:
***
1231:
1232:
is Derivaave True
1233:
1234:
@property
1235:
def _diff_wrd wrt (self):
(2154 more lines below)
(Open file: /sympy swwp///wmpp//crre/tuucco... (Current directory: /sympy_sympy) sympy) bash-$
SWE-agent
sympy sympy__ympy--aaeeeeepmyyy sympy-21614
We have scrolled down through the 'function.py file, but have not yet found the implementation of the 'kind' property for the 'Derivative' class. We need to continue scrolling to find the relevant scction.
scroll 1_owwn
Observation
Hompy__ymmyy22611
[File: /sympy__ayyeeppppeepp htwp:///ssy///////////teetiooooe (3389 lines tctal) (1233 more lines above) [File contents omitted for brevityl (2056 more lines below)
84

---

(Open file: /sympy swmp://sppy//orrr//auccooooooy (Current directory: sympy) bash-$
SWE-agent
sympy Sypp__2ymm
We have scrolled further down the 'function.py" file, but we still haven't I found the implementation of the 'kind' property for the 'Derivative' class. We need continue scrolling to find the relevant section where the 'kind' property likely to be implemented overridden.
scroll down
Observation
sympy sympy_21112
[File: /aympyy_syayyyyla//aaaaaaaaaapaa
symp://swmp//orrr//uuciii,,, (3389 lines tctal) (1331 more lines above)
[File contents omitted for brevityl (1958 more 1ines below)
(Open file: /swmpy__yapp//syypgpgppp. wwwp://sggg//cree///aaacioi.oo (Current directory: /sympy_sympy) bash-$
SWE-agent
sympy sympy_sympy-22 sympy-21614
We have scrolled further down the 'function.py file, but still haven't found the implementation of 'kind' property the 'Derivative' class. We need to continue scrolling to find the relevant section where the 'kind' property is likely to be implemented or overridden.
scroll down
Observation
sympy sympy-21614
[File: /awppy__sappp//e/eeeaeeaaaaaaaee
swmpy//saay/orrr/ttctcioo.. (3389 lines total) ]
(1429 more lines above)
[File contents omitted for brevityl (1860 more lines below)
(Open file: hatpe//_ssm///mmg/c/////////oee (Current directory: /sympy_sympy) bash-$
SWE-agent
sympy sompy_sympy-21111
have scrolled further down 'function.py" file, but we still haven't found implementation the "kind' property for the 'Derivative' class. We need to continue scrolling to find the relevant section where the 'kind' property is likely to be implemented or overridden.
scroll_dom down
85

---

Observation
sympy sympy-21614
[File: /aympy__yyaay//aa_aaaaaaa_aaa (1527 more lines above)
hwwpy//yapy//ooe//untttttt (3389 lines total)
[File contents omitted for brevityl (1762 more lines below)
(Open file: /sympy /wnyy//ymmy//orrr//ttttoo.....p (Current directory: Sayypp__ymppp. bash-$
SWE-agent
sympy sympy-21614
We have scrolled further down the 'function.py file, but we still haven't I found the implementation of the "kind" property for the 'Derivative' class. We need to continue scrolling to find the relevant section where the 'kind" property is likely to be implemented overridden.
scroll_down
Observation
sympy sympy-21111
[File: h/mmp/__swmy////p////r/////reee
(3389 1ines tctal)]
(1625 more lines above)
[File contents omitted for brevityl (1664 more 1ines below)
(Open file: /sympy wttp:////gg///////////ooooooooo/ (Current directory: bash-$
SWE-agent
sympy sympy_sympy-21611
We have scrolled further down the 'function.py" file, but still haven't 't found the implementation of 'kind' property the 'Derivative' class. We need to continue scrolling to find the relevant section where the 'kind' property is likely to be implemented or overridden.
scroll
Observation
sympy sympy22111
[File: http://wswwy//y/////////////tttt
(3389 lines tctal) ]
(1723 more línes above)
[File contents emitted for brevityl (1566 more lines below)
(Open file: /sympy swtp://smmy//cree///utti....o (Current directory: /sympy bash-$
SWE-agent
sympy_ Sympy__5m
We have scrolled further down the 'function.py" file, but we still haven't t found the implementation of the 'kind' property for the 'Derivative' class. We need to continue scrolling to find the relevant section where the 'kind' property is likely be implemented overridden.
86

---

scroll_down down
Observation
sympy sympy-21614
[File: /sympy__yyy htwp://wwww/////////////ooooooo. (3389 lines tctal) (1821 more lines above)
[File contents omitted for brevity] (1468 more lines below)
(Open file: /sympy wwwppeeaaapy/y/aeeeeaaeaaaaaaaae wwwp:///pp//occc///tttooooooop (Current dircctory: syympy_syppy. bash-$
SWE-agent
sympy_Sympy-2261
We have scrolled further down the 'function.py" file, but we still haven't it found the implementation of the 'kind' property for the 'Derivative' class. We need continue scrolling find the relevant section where the 'kind' property likely to he implemented or overridden.
scroll down
Observation
sympy sympy-21614
/sympy__ap.p hwwp:///yaay/orrr/tuiii... (3389 lines total) ] (1919 more lines above)
[File:
[File contents omitted for brevityl (1370 more lines below)
(Open file: /sympy /ympy//smm//cce///atttooo... (Current directory: Saympp__ympppp bash-$
SWE-agent
sympy sympy-21614
We have scrolled further down the 'function.py' file, but we still haven't i found the implementation of the 'kind' property for the 'Derivative' class. We need to continue scrolling to find the relevant section where the 'kind' property is likely to be implemented overridden.
scroll_dww down
Observation
sympy sympy__ympy-21111
[File:
/sympy_ https//w/////m/////rr/////tiiioa (3389 lines tctal) ]
(2017 more lines above)
[File contents omitted for brevityl (1272 more lines below)
(Open file: /sympy syap://smmmy/ooee//uuutt.o..oo (Current directory: /sympy_syypa bash-$
87

---

SWE-agent
sympy sympy-21614
We have scrolled further down the 'function.py" file, but we still haven't I found the implementation of the 'kind' property for the 'Derivative' class. We need continue scrolling to find the relevant section where the 'kind' property likely to be implemented or overridden.
scroll down
Observation
sympy sympy-21614
[File:
/sympy_ http://swwyyyym////////////ttttt (3389 lines tctal) ]
(2115 more 1ines above)
[File contents omitted for brevityl (1174 lines below)
(Open file: hwwp:///w/////////////////tioooo (Current directory: bash-$
SWE-agent
sympy sympy-21614
We have scrolled further down the 'function.py' file, but we still haven't found the implementation of the 'kind" property for the 'Derivative' class. We need to continue scrolling to find the relevant section where the 'kind' property is likely to be implemented overridden.
down
Observation
sympy sympy-21614
[File: /sympy_ http://sww/////mp/////////utttoo (3389 lines tctal) ] (2213 more lines above)
[File contents omitted for brevity] (1076 more lines below)
(Open file: /aympy_sysppyllllllaaaaaaaaaaaat swwp://ymmm///rree//uttiio..oo (Current directory: /yympy_eympy) bash-$
SWE-agent
sympy sympy-21614
We scrolled further down 'function.py file, but we still haven't found implementation of the 'kind' property for the 'Derivative' class. We need to continue scrolling to find the relevant section where the 'kind' property is likely to be implemented or overridden.
scroll 1oool_ddwn
Observation
sympy Hompy__ymmyy21611
[File: /sympy__sppppppp..ppyyp http://///////////////////iiiioa (3389 lines tctal) (2311 more lines above) [File contents omisted for brevityl (978 more lines below)
88

---

(Open file: /sympy wwmp///sepy//orrr//acccoo.o.. (Current directory: /sympy_syppy) bash-$
SWE-agent
sympy sympy-21614
We have scrolled further down the 'function.py" file, but we still haven't tt found the implementation of the 'kind' property for the 'Derivative" class. We need to continue scrolling to find the relevant section where the 'kind' property is likely to he implemented overridden.
scroll down
Observation
sympy sympyy2111
[File: //wppyy_.ypyyy/////e.eee.eaa
wwtp://ywwp//orr///tttioo.oo.o (3389 1ines tctal) ]
(2409 more nines above)
[File contents omitted for brevityl (880 more lines below)
(Open file: /sympy s/mp://smpy/ccce///utttiiooo (Current directory: bash-$
sympy)
SWE-agent
sympy Sympy__ymmyy2111111
We have scrolled further down the 'function.pe ppyiionnppp"oo file, and we have now found the 'diff' function, which is related to differentiation. However, still need to find the implementation of the 'kind' property for the 'Derivative' class. Since we the section related differentiation, should be close to the relevant
Let's continue scrolling to find the 'Derivative" class implementation.
seroll
Observation
sympy sympy-21614
[File: /sympy_ httpo//wwww///p///////////icicca (3389 lines tctal) ] (2507 more lines above) [File contents omitted for brevity] (782 lines below)
(Open file: htwppaaaaypyy///yaaaaaaaaaaaeaae (Current directory: sympy) bash-$
SWE-agent
sympy sympy__smpyy-22 sympy-21614
We have scrolled further down the 'function.py file, and we have found the 'diff' function, which related differentiation. However, we still need to find the implementation of 'kind' property for the 'Derivative' class. Since we are in the section related to differentiation, we should he close to the relevant code. Let's continue scrolling find the 'Derivative' class implementation.
scroll_down
89

---

Observation
sympy sympy-21614
[File: /aympy__ayapp/_aa__aaaaa (2605 more lines above)
hwwpy//yypy//ooe/ttncttt. (3389 lines total)
[File contents omitted for brevityl (684 more lines below)
(Open file: /sympy sympy//ymooy/ooreettuutttooo (Current directory: Saympy_symppp. bash-$
SWE-agent
sympy sympy-2m0o -26114
We scrolled further down the 'function.py related to differentiation. However, we still need to find the implementation of 'kind' property for the 'Derivative' class. Since we are in the section related to differentiation, we should be close to the relevant codc. Let's continue scrolling find the 'Derivative" class implementation.
we have now found the 'diff' function, which
scroll_down
Observation
sympy sympy-21111
[File: /sympy_yappppppeeepeppp. sympp//srny//orre//utttioo.. (3389 lines tctal) (2703 more lines above) [File contents omitted for brevityl (586 more lines below)
(Open file: /sympy smmp://smpy/ocree//uccto.....o (Current directory: sympy) bash-$
SWE-agent
Sympy21111
We have scrolled further down the 'function.py file, and we have now found the 'expand" function, which is related to expression expansion. However, still need to find the implementation of the 'kind' property for the 'Derivative' class. Since in the section related expression manipulation, should be close to the relevant code. Let's continue scrolling to find the 'Derivative" class implementation.
scroll_doms
Observation
sympy sympy-21614
[File:
/sympy_ http_///////////o/////ttttttiiie (3389 lines tctal) ]
(2801 more lines above) [File contents omitted for brevityl (488 lines below)
(Open file: hswppysssappy/ssaeeaeaaaaaa swwp://smmgy/ccrrr/tuuuco.o.ooo (Current directory: /sympy_sympy bash-$
90

---

SWE-agent
sympy sympy-21614
We have scrolled further down the 'function.py' file, and we have now found various expansion functions and wrappers, but we still need to find the implementation of the 'kind' property for the 'Derivative" class. Since we are in the section related expression manipulation, we should close to the relevant code.
Let's continue scrolling to find the 'Derivative' class implementation.
scroll_dom
Observation
sympy sympy__syppy- Sympy-2111
[File: /sympy__spppppp wwwpp//ywwp///rrrr///niiooooooo (3389 lines tctal)) ] (2899 more 1ines above)
[File contents omitted for brevity] (390 more 1ines bolow)
(Open file: /sympy /swppa_syypppllllleeeeeeaaaaaaat hwwp://wymy//cooc////tt...oooo (Current directory: syymp__smppy. bash-$
SWE-agent
sympy sympy-21614
We have scrolled further down the 'function.py' file, and we have now found various expansion functions and wrappers, but we still need to find the implementation of the 'kind' property for the 'Derivative' class. Since we are in section related expression manipulation, we should close to the relevant code.
Let's continue scrolling to find the 'Derivative' class implementation.
scroll_dow
Observation
sympy syppy__ymm
[File:
/sympy http://wwwwwsss/////////////ccia (3389 1ines tctal)
(2997 more lines above) [File contents omitted for brevityl (292 more lines below)
(Open file: /sympy wttpppaasspyyyaaeeeaaaaaaaaaaaae wwwp://mmy//crre//accciiooooooo (Current directory: syympy_sympy bash-$
SWE-agent
sympy Sympy__5mm
We have scrolled further down the 'function.py file, and we have now found various expansion functions and wrappers, but still need to find the implementation of the 'kind' property for the 'Derivative' class. Since are in section related expression manipulation, we should close to the relevant code.
Let's continue scrolling to find the 'Derivative' class implementation.
acroll_down
91

---

Observation
sympy sympy-21614
[File: /sympy__saapy/////aa.aaa. (3095 more lines above)
(3389 lines total) |
[File contents omitted for brevityl (194 more lines below)
(Open file: /sympy sympy//smoy//ccre///ttiiio (Current directory: Saympy__ympgp. bash-$
SWE-agent
sympy sympy-21614
Exit due to cost limit
exit_cost
Observation
sympy sympy-21614
Exited
Gold Patch
sympy sympy-21614
diff --git
a/symyy/core/ffuctionn.2 a/symmy//cord//uuut.oo..yy +++ b/sympy/core///uutiii
b/aympy/coree/futtion
@e -1707, +1707, 60 def free_ sreebsybols (self) f)
retaup (count. free_symbols)
return ret
@property def (selE):
return self. aelf.args[]] kind
def Leval_subs old, new)
* The substitution (old, new) cannot be done inside
+ Derivative (expr, vars) for variety of reasens
Analysis for django/djjago-14411
Issue Summary. The issue involves the interface. The label element for this widget has 'for' attribute that points to lab-labelblle element. Since the ReadCnlyP SerCCnnyyasswordii wwwwwodiasshhiieeee renders text rather than an input element, the label should not have a 'orr' attribute.
in the Django admin
Interaction Summary
Locate the Relevant File: The agent began by searching for the definition using the find_file command, but found no results. It then used the search_dix command to find references to widget within the repository and located relevant matches in several files. The agent chose to first examine the he helpers. py file, but found it only contained a reference to the widget and did not include the label rendering code.
2. Explore Related Files: Next, the agent opened the forms .p directory, which de-
file in the djjago/cont.ri//ette fined the RecdbulyyPasworrdlsssttigggg and its associated
92

---

It identified that the widget used template (read_enta cerid_nnll_ie mallypassword hash html) for rendering, but the template did not include relevant information relating to the Widget's for attribute.
feaddhlyyaassoorcllahhhffa
3. Edit the Class Definition: Returning to forms PY, the agent found the UserChangeForm which used the aascomlyasswordisssiiiie
reasoned that Django's form rendering system likely added the 'for' attribute automatically. To address the issue, the agent attempted ield to exclude the 'for' attribute. The agent made several
write a label_tas method for the
aasaassswdaastt y edits this effect, importing the necessary utilities from django ut 1s. html and adding custom label_tag method.
4. Submit Without Testing: After the desired changes were made, the agent proceeds to submit without testing assumes that Django server cannot be run and tested this environment.
Evaluation The agent correctly identified the need to customize the label rendering for the SeaddOnyPaaseee wwwassswordaaahiiee.
class. However, its solution failed solve the issue since it did not understand how labels were defined for Widget object. In actuality, the model needed to override the id_for_labe property defined the Widget class change this behavior properly. Similar functionality can be seen in other parts of the codebase.
Figure 36: Trajectory for instance d
Issue
django django-14411
We're re currently solving the following issue within our repository. Here's 's the issue text:
ISSUE:
Label for ReadOnlyyPasswordHaae Widget points non-labelable element.
Description
(last modified by David Sanders)
In the admin, the label element for the SaaddonlyPassworddl Widget widget has a 'for' attribute which points non-labelable element, since the widget just renders text, not input. There's 's no labelable element for the widget, the label shouldn't 't have 'for' attribute.
INSTRUCTIONS: (31 lines)
(Open file: n/a) (Current directory: /ajango__ajannoo bash-$
SWE-agent
django_
django-14411
To address the issue, we need locate contains the Pa Widget widget's rendering code. Since this likely part of the Django admin, we should start by searching for the file that defines file within the repository.
Pasworddss Widget". Let's 's use the 'find file* command to locate this
find file "FcudonlyPaassordiatllllccce
Observation
django__django- django-14411
No matches found for "headon.yylasswerrdahhhhhhiggg in djjano
(Open file: n/a)
(Current directory: /django__ ajange) bash-$
93

---

SWE-agent
django django-14411
It seems that the 'find file' command did not yield any results for "ReadOOnlyPssswodlllal Widger""...""...... This could be because the file name does not exactly match the widget or widgets in the admin app. Let's try a broader search using the ssearch_dir command to look for the term "ReadOnlyPasswerr HashWidged within the files of the repository. This should help us find the file where the widget is defined used.
Since the widget is likely
part of the Django admin, might located a related
search_die
"Meedinllyaessworchhhhi...g
Observation
django_
django-14411 django__djanoo14411
Found matches for
Lajjango_djjage
httpa///ohuuuggaoo/ooaoo//aa//ar (1 matches)
/django
http:///aaaaaaa//ooooo/ttt/tttta (2 matches)
/django_ httpa/////a///////aaaaa///////ta
matches)
(4 matches)
/Ajjang___canno/reettttaa_______ htt.://aiiettaaaaatt...aeeeeeeee (1 matches)
End of matches for "Reasoulyyasswerrccssiiidd in /django__aaa. d'ango
(Open file: n/a)
(Current directory: /ajango_djanna bash-$
SWE-agent
Lhango__djang---i
The search results indicate that there matches for "ReadOnly"s wawswooddassssHahhhhha Widgee in several files, but the most likely places where the rendering code would be defined are in 'helpers.py" and 'fommspy' within cjjano/coottib/admin
django/contrib/auth directories, respectively. Since the issue is related to the admin interface, let's first check the 'helpers.py' file in the 'django/con- trib/admin' directory. We'll Ill open this file and look for the relevant code.
open djang//connttbb//thihhbbbbooo
Observation
django_ django-14411
[File: http://www..agggg//gh/o/////t/ta (424 1ines
total)
1:import json
2:
from django import forms
4:from clango.contrib www..r..aa.ii.nnnnn....mmma..aea admin.utils import
5:
display_for_fiell fielsees,, holp
6:
fielup_field, quote,
7:)
8:from django. core. exceptions import ObjeetDoesNotExitt
9:from django. models. www.dd.reeaaeee.....eeeeeeeeeeee import
10:
ForeignObjectReL, ManyToManyRel, OneToOneField,
11:)
import flatatt 13:from django. .ewplatte.eeaaulfflll........... linebreaksbr
12:from django. fongo.forms..
import capfirst,
14:from django. bjango.urls .rrl import NoReverseMatch, reverse htmll
15:from django.. uu illilssa html import conditional_eccaate format_hta html
94

---

16:from django wtw.uttiiss.................... safestrinng impert mark_
17:from django its. trans import gettext, gettext Jazy as 18:
19:ACTION_CHECCBSS
Leleecte__action
20:
21:
22:olass ActionForm Form)
23: action = forms worms. ChoiceField Laaaall(ebaataaaaaaa_a ('Action:'))
24:
across = forms. BelleafFilll
25:
label=',,
26:
rcquired-False,
27:
initial=0,
28:
widget=forms HiddenInput (( ('claas'; 'select-across')
29:
30:
31:
c2eckbxx forms, CheckboxInput (('clase claass' 'action-select I,
lambda value: False)
33:
34:
35:class AcminForm:
36:
def iniii_ form, fieldsets, frepopulatcd_
seadonly_fields-tte model_ =do=
37:
self. self.fielcses form, fieldsets
38: delff.preppulatee_ffiel.
39:
'field' form[field_name]]
40:
'dependencies': [form[f] for f in dependencies]
41:
for name, dependencies in
prepopulates sttpppplattt__eeddsttteeee ())
42:
self felffoede__ammii model_admin
43:
if readonly_fields is None: readonly_fields
44:
45:
self.readonly_fiell.. fields readonly fields
46:
47:
def (itter(selff)
48:
for name, options in self.fieldsets: yield Fieldset
49:
50:
self.form name, readonly_fielés-se htw.ffee@@eeel...rllolllff..liie readonly_fields, model_admin model _addin_ *xoptions
51:
52:
53:
54:
55:
56:
@property
57:
def errors(self),
58:
return felf.form.errors
59:
60:
@property
61:
def son_field _ield_eeroorr errors (self):
62:
return self htw..foooooonffclldoorooooo
63:
64:
@property
65:
def media (self):
66:
media = self. form.mediia
67:
for fs in self: media media fs.media
68:
69:
return media
70:
71:
72:class Fieldset:
73:
def init__ form, name-None, readonly_fields=())
fields= (),, classes=(),
74:
description=Kene, model [odel_aamii==Noe))
95

---

75:
serf. form
76:
seli.name, self.fielas nane, fields
77:
seli.classes = join (classes)
78:
self self.description - description
79:
self.model_admin model_admin
80:
relf.reedonlyffelless readenly_fields
81:
82:
@property
83:
def [edia(selE))
84:
if 'collapse' in self.classes:
85:
return forms Media (oocia()==''amin))) ftmin//sscollapec 1)
86:
return forms. (edia()
87:
88:
def _iter_(selff)
89:
for field in self.fields:
90:
yield Fieldline (self. field,
self www.:rreacnyy..ill..ll model admin=self. model admin)
91:
92:
93:class Fieldline:
94:
def initt_seee, form, field, readonly_fields-ten
@aoe__aamiin-Nonee
95:
forf.form = form # A django. forms. Form instance if not (field, "_iter_"" isinstance (field,
96:
str):
97:
self.field [field]
98:
else:
99:
filldi field
100:
salf.has_viiibe__fie not
(324
more lines below)
(Open file: n/a)
(Current directory: /django (jjano)__jjanc)) django) bash-$
SWE-agent
django 1jango-14411
The 'helpers.py' file is now open, but we need to find the specific part of the code that deals with the Since the file quite long, should use the 'search_fits file" command locate the tern within this file.
search file "Beadbalypassoordissiii.gee
Observation
django_ddaango- django-14411
Found matches for WRead0nnyyaasssrrrrssallllgggg
Line 215:
$ Beadbmulylaswworrdsssllaeeee
End of matches for "keada..yaasseccccssshiiig""""""
in
(Open file: /btp:a/w..aangguggggn///iia/aaaa
(Current directory: /django__jjange
bash-$
96

---

SWE-agent
django django-14411
We have found reference to Let's navigate that line examine the context and how the label is being rendered.
on line 215 of the "helpers.py file.
goto 215
Observation
Bjango__dangg-o
[File: http://hhh.gggaa.o//o/////////aa (424 lines
total)]
(199 more lines above)
200:
return format_ ( <'<a bref="()*> url,
remote_cbj)
201:
except NoReverseMatch:
202:
return str (remote_cbj)
203:
204:
def contents (self) (sel):
205:
from django.conten contrib. www..ot.........te.............e edmihh.emmplatetass..ciii list
import _boollan icon
206:
field, obj, model_admin self. filldiiill
4 self. form.
hol.f.fom.insteaee instance, self colf.mocel_admin
207:
try:
208:
E, attr, value facku__iiel(ffield, obj,
model _ammin)
209:
except (AttributeRrror, ValueRrror, ObjectDoeafo:Existt)
210:
result_reph = self.empty_ value dissslll
211:
else:
212:
if field self. form. fields:
213:
widget self. [forlf ffafeeddd.fild.iii.............. widet I This isn't elegant but suffices for
214:
contrib.
215:
Resdon
216:
if getattr (widget, head_only', False) return widget.render (field, value) 1s is None:
217:
218:
219:
if getattr (attr, 'boolean', False); result_repr _bonlean_icon (value)
220:
221:
else:
222:
if hasattr (value, "_htm]") result_repr value
223:
224:
else:
225:
result_ repr linebreaksbr (value)
226:
else:
227:
if isinstance remoteefffil,,, ManyToManyRel)
and value is not None:
228:
result_repr ", jojoin (map (str,
value.all ()))
229:
elif
230:
isinstance (f. remote_field, (ForeignObjectRel, OneToOneField) and
231:
value not Nene
232:
": result_repr self www.eeeetniinnnneemeeeeam (f. remote_field, value)
233:
234:
else:
235:
result_reph repr display for for fir_d f,
self wetfffepttyvvllle value {
236:
result_repr = linebreaksbr (result_repr)
97

---

231:
return conditional result_repa
238:
239:
240:class Inl ineiddinnooomeee
241:
242:
wrapper around an inline formset for use in the admin system.
243:
244:
def init_ (self, inline, formset, fieldsets,
prepopulatee__iilldd--oe readonly_fieldeeNenn, model acnennacmi,,,
245:
- has
246:
bas__hangg-eecmlssoorrrvxue
has delete__errissiinnTnni
247:
has_viek_permis on
248:
self.opts inline
249:
self. = formset
250:
sel sildssielssets = fieldsets
251:
melf.model__ddiin model admin
252:
if readonly_fieldd is None:
253:
readonly_fields= ()
254:
self.readonly_fiele:: - readonly_fields
255:
if prepopulatte__iieles is None:
256:
prepogulated_fields ()
257:
eeld.prrpoollate__fielees prepopulated_
258:
self.classes
"ooin (oaaiiiinnnnnllali. classess) if
else
259:
sels.nas_aad_pemmisstt has_add ped_iission
260:
sel
sel_.has_change_eemmiiss has_change_permiss
261:
self_haa_delate_permi self.has_view__prmm
has_delete_permiis has_view_permissine permission
262:
263:
264:
def iter (self):
265:
if selé. hels.has_change_peee permission: shnn
266:
raadolly_fielee_foooddttttt = self readenlldlyy clse:
267:
268:
roadoly_ficcld_fooocdddtt
self.rodeely_ffeldd:
flatten_fieldeet (tlelldetttselF..ieeldssstt.
269:
for form, original in zielself. 270: www.set..et...ueeeeee..........e ()):
self self.
271:
view_on_site_us
celif.ops..o peotss.ee_vvew__ooe sorigiin
272:
yield InlineAdminform!
273:
form.formset, form, self.fieldsed
self www.pppppllaees_fileee...
274:
original, readonly_fielde for editing,
model www.:aaainissee...........
275:
276: for form in self. www.foorsssteeecem extmscteextraaaorm:::: yield InlinchaminFozml
277:
278:
279:
self, formset, form, self,fieldsed self f. wwt.rrppppppaeed_feeeeeeee.
280:
None, self readenlyyffly,,ds,, model acmin=self.opts
281
282:
if selé.has add permissicn: yield InlineRaminForm/
283:
284:
self. formset, wow.f...rrmet.mmtty...mmm.m self. fellssees,,,,, relf..rreopulatcc_fielldd None, selffreadonly. model_edmiinseeff.ppss
285:
286:
287:
98

---

288:
289:
def fields (
290:
ik getattr (set...osstt......t...... "ik", None)
291:
empty_form w.w. war..forrreet..mtty.ooo. meta_labels empty_ 293: meta_help_texts ompyy_Comm._eetaahhlppooooos or ()
292:
or {
294:
for i, field name
< enumerate Eellaaten_fielddeeee (self.fieldsete.
295:
i Ek and name = field_name
296:
continue 1s not self.har
297:
- self. reli.rraadonly_feeld::: htw.taaacchageeeceisssiiii or neme
298:
yield
299:
'name': field_name,
(125
lines below)
(Open file: /ajango httaaaaaaaaaaaaaaaaaaaaaaaaaaaae
(Current directory: /django djange) bash-$
SWE-agent
django django__djangolll django-14411
navigated to the relevant section the 'helpers.py where ReadOnlyPasswordlee Widget' mentioned. The comment on line 215 suggests that there is special case handling for the The code on line 216 checks if the widget has 'read_ont _olloonlll attribute and, if renders the widget directly.
Passwwodaas
address the issue, we need to ensure that label for the 'ReadOnIyPasseeee PasswordHash Widget' does have 'orr' attribute, as it points to a non-labelable element. We should look for the part of the code where the label is heing rendered with the 'for' attribute and make the necessary changes.
Since the code snippet are looking at does not show the rendering of the label, we need find where the label is being rendered. It might the template files somewhere else the code. Let's scarch for term "label" in current file find where labels are being rendered.
search_ file "label"
Observation
Found 23 matches for "1abel"
line 5: diappla__or__ieldd, Tlatten_fieldsett
help text www_texx_t____eee sier_ label for field,
Line 23:
action forms. Cht.cceieeeeealaaaallll... ('atttdaaaaa__aaaiitttt ('Astion:"))
Line 25:
label='', def label_tag (self);
Line 131:
Line 133:
contents
(ww.ffffeldd....oooo..oo.o.o.lle label)
Line 142:
checkboxes should net have label suffix as the
+ checkbox appears
Line 143:
# to the left of the label.
Line 144:
return self, label.l Jabel_euffi==' if holf.is_checkban else None, is form._meta.
Line 146:
Line 163:
and class_name
form._meta. meta..abeass
Line 164:
label form__meta. haa._ettalaallllclss___ssoo
Line 166:
label (abel_for_fielld
form._meta.. m
model, model_acming form-form)
Line 175:
'label': label,
Line 186:
def label (self):
Line 190: label self field| llabel')
99

---

Line 313: Line 191:
return format_ htm_ (ttaataannnaaaaallllaelill ('taaaabbbb(()))111/1////b
flatatt (attrs), capfirst (label), fel...oom..a.e... suti
Line 195:
remote remote_fieicaeeeeeeee meta_labels - empty__orm.__nea..looo or (}
Line 292:
Line 300:
'label's meta_labels (field_name)
or label_for_
Line 312:
label form Field.label if label is None:
Line 314:
label label_forr laael__or_ffeldddee___eee_ (f helf.opts.modee self opts, form=onpty_form)
Line 317: End of matches for "label" in /bba://.ddgg///oaagg////////////
'label'; label,
(Open file: httpaa/ogaaaaaoooooooooooottttta pers.p (Current directory: Laango_jjanoo bash-$
SWE-agent
django django-14411
search results show several instances where labels are being handled the 'helpers.py py file. The most promising lines for our issue seem to be around line 191, where a label tag is being formatted with a 'for" attribute. Let's navigate to that line to examine the context and see if this is where we need to make changes.
goto 191
Observation
django
django- 14411
[File: httph//wwwwgha/g//a///////ttat/a
(424 lines
total)]
(175 more lines above)
176:
'help_text's help_text,
177:
'filld field,
178:
179:
sclf.form form model_ admin self self. is_first
180:
181:
182:
self.ia_checkbox False sel self.is_readonny - True
183:
184:
model www.aaamm....eeeaecc......eeeeee value wilppyessaleee_spplaaaaaap ()
185:
186:
def label (sell)
187:
attrs - ()
188:
if not self.is_first:
189:
attrs
"inline"
190:
label sel
191:
return format_html
flatatt (attra), capfirst (label), self.form labellaubelliii
192:
193:
def uet_aamin_uri1se14 remote_field, remote_obj):
194:
uri_name = : Seddiin:s_$sscchaae "s_anggeee' $
195:
mmote__feldd model wptp___aappppaaaaaaaeaeaappa app_label,
196:
Remote_field model m_t.m masttaamodd__aeaeee
197:
198:
try:
199:
url = reverse _ur__nme,,
args= (quote (remote_obe __tee__ojppppp_
100

---

200:
format_ html <'<a bref=" (>()</>>, url,
remote_cbj)
201:
except NoReverseMatch:
202:
return str (remote_obj)
203:
204:
def contents (self)
205:
from contrib. tww.anteeplattttsss... list
import _bolllannneee _icn
206:
field, obj, self www..forrmmtttt. instance,,,, self.model_admix
= self. fieldd
207:
2, attr, value lookup_field(ficld,, obj, try:
208:
model_admia
209:
except (AttributeRrror, ValueError, 0bjjectioesNotHxiist)
210:
result_repr
211:
else:
212:
is field in for.
213:
widget self. wtw.llieedddldldddaaoo.l.lell This isn't elegant but suffices for
214:
4 contribbauthe
215:
I Readonlyasssorrrhhhhiii
216:
if getattr (widget, tred_oonyy False) return widget. (field, value)
217:
218:
is is None:
219:
if getattr (attr, 'boolean', False): result_repr = _conlean_icon (value)
220:
221:
else:
222:
if hasattr (value, "_html"") result_repr value
223:
224:
else:
225:
result_ repr = linebreaksbr (value)
226:
else:
227:
if isinstance rww.teeeeeeeeiiieeeeeeeeeeeeeeee field, ManyToManyRel)
4 and value is not None:
228:
result_repr = ", jojoin (map (str,
valuc.all()))
229:
elif (
230:
isinstance (f. remote_field, RForeigrobjecttel,, One(ooneFieed) and
231:
value is net Nene
232:
233:
resuit_repr (f. remote_fieedd value) else:
234:
235:
result_repr display__or_fiell (value,
self www.eeepptttuuue display)
236:
result repr Linebreaksb (resultrrerr
237:
return conditional escape (result_repr)
238:
239:
240:class InLincAdminFormStt:
241: **"
242:
A wrapper around an inline formset for use in the admin
system.
243: =*.
244: def init (self, inline, fornset, fieldsets,
245:
faardonnlyyilddddeeeooee admin-Non has maa_add_pemmisiionTnne
246:
has_change-pemmlssssorrrr
- has bes__@@@lee__pmmmssiiorrrr
247:
ras_viee_permissionee
101

---

248:
sel2.opts inline
249:
sormeet formset
250:
seli.Sieldsets = fieldsets
251:
selffmodel_adminn model_admin
252:
if readonly_fields is None:
253:
readonly_fieldss ()
254:
sels.readonly_fieelds readonly_fields
255:
if sippppulateddfieldss is None:
256:
frepopullats__fieelds ()
257:
self cal..prroooulltcooffeleee
frepopulated_ficldd
258:
sclf.classes join www.iinnnnnnine.e classess)
- inline.
inline.classen clse
260: se15.bas_add__emmisstt = has has_add_permission
259:
sel sel - hes_change_pemmision
261:
sel se1f_haa_dalete_pemmi sel
262:
has_view_pewt perewweeeei_____ ssicn
263:
264:
def
iter (self):
265:
if selé.has wele.has_change__ppee permission: pemssseinnnne..msssnnnn
266:
reabolly_fiel@__fooodddiiii else:
relf.readenly_fieelld
267:
268:
readnly_fieedd_foooeddiie - relf.readenlyffiedd: +
SeatttenFfilddeetee (sel
269:
270:
for form, original in zip(self, for.set.toooo.......... intttttliiiiailaorr.ssoo
4 self www.:felfffni.eee wtw.:ott.ttte...ueeeeeeeeeeeeee. ()):
271:
view_on_site_ur/
self www.opttteeeeeeeeeeeeeeeeeeeeeee
url (origiinal)))
272:
yield inlineAdminnFrmm
273:
self.forme form, fell.fiel,sets,
sel self. att.ppppppllate_fiiellll.ee.
274:
original, readcnly_fields for for_editing,
model acw.imseeieee.oot......
275:
view_on on site url=view httprvieewo__stt__rr______ sn__itee_rl_
(149
more lines below)
(Open file: hagpa///..ounooggoooooo//o/////s
(Current directory: /django django) bash-$
SWE-agent
django 1jjngo-_aj
We now looking at the section of the 'helpers.py py" file where label tag being generated. However, this label tag does not include a "orr' attribute, which means this is not the part of the code responsible for the issue described.
The issue states that the label for the 'ReadOniyPasswondHlaa Widget' has a 'for' attribute pointing to non-labelable element. Since the 'ReadOnly ReadOnilyasswordHaas Hass Widget' likely custom widget used the Django admin, the rendering Python code.
label might be handled a template the widget's 's own
Given that the RRaadOnlyPaswwrrHHsshhet into the 'forms.py' file in the "jjango/contri//uut" directory, as it might contain the definition of the widget and the rendering of label. Let's open the "forms.py
is part of the 'django.contrib.awe contrib.auth app. we should look
the "djangolcontri//auth directory to investigate further.
open cjangg//contrib/uutte..
Pyy
102

---

Observation
django @jangoo-14411
[File: /twjango_aaaag///aaaaaaaaaa http://wwwaaaaaaa////////a///tta httph//g...aaaaa////////o//tttt/ (440 lines
total)]
1:import unicodedata
2:
3rrrom django Import forms
4:rrom crangi.contrib.... import
5:
authenticate, [re_user_model, hassword_validatiin
6:)
7:from BUCCABBE_PASSSOORDDPREEEE identify_hasher,
hathhherrss import
auth. hasherss
8:
9:)
10:from
blaango.cocrrib.uuth
import User
11:from
django. toke import default__toee_geeerraaao
12:from
django.contrib.siteen shortcuts import get_current_aite
13:from
<jango.coxe.exceptiia. import ValidationError
14:from
cijngo.core.msil import SmaiiMuultAAAeerratiiie
15:from
import loader
16:from
django.utils.ence import urlsafe__asee6__nnee
import ferce_bytes
17:from
18:from
import capfirst import gettext, gettext_ as ()
f9:
20:
21: UserModel
22:
23:
24:def
(s1, 92):
25: ***
26: Perform case- iase-insessitive comparison of two identi fiers,
using the
27: recommended algorithm from Unicode Technical Report 36, section
28:
2B) (2). 29: ***
30: return unicodedata. normaaannoomaaaaaaaa lic ('NEKC', s1). caseffl()) ==
unicododata..nrmallle.. ('NFKC', s2) casefool())
31:
32:
33:class Reaconnyyasswordisshhhiiii (igrss
34: template_ _aaeee_aaeeea__ = aaub///lggetsrreddonnll htw.reaww...oalooaoooooooo.daee. htm1
html
35: reao_only True
36:
37:
def get_contexx (self, name, value, attrs):
38:
context super .eet_context (name, value, attrs) 39: summary []
40: if not value (UNSSABLB_PASSWORR PASSWORD (ASSSCRD_PREFIX)
value... startswith (UNUSABLE,
41:
cummary.appente (( 'label': gettext ("No password
set. 1)
42:
clse:
43:
try:
44:
hasher identify_hasher (valuc)
45:
except ValueError:
46:
summary. append (('1abel': gettext ("Invalia
password format or unknown hashing algorithm.
47:
else:
48:
for key, value in
fusher.safie sunnary (value) .items ()
49:
summary append label'; gettext (key),
'value'; value_))
50:
context suuumaayy) summary
51:
return context
103

---

52:
53:
54:class
widget = ReadouulPPssvorraaiiiieeeettttt :
55:
56:
57:
def _niit_ *args, **wwargs):
58:
swargs..ettdefaulee ("required", False)
39:
awarga.setdefaule ('disabled) True)
60:
super init (vag__ **kwargs)
61:
62:
63:class
64:
def to_python (self, value)
65:
return unicodedata. norma ize ('NFKC), super () to__ythhn (value))
66:
67:
def aidget_attre (self, wicget):
68:
return
69:
**super wwidget attrs (widget), "autocapitalize": 'none', 'autocomplete' 'username',
70:
71:
72:
73:
74:
75:class UserCreationFon poreeorrmfffrsa (
76:
77:
form that creates user, with no privileges, from the given username and password.
78:
79:
80:
error_messages -
81:
'password mismatch': ['The two password fields didn't match.'),
82:
83: passwordi forms. CharFieldd
84:
label=_("Pa) ("Passs""",,"
85:
strip=false,
86:
widget=forms. Passwordinnut (attrs=
''autocompltte': (atteocceeeptttttttttttttttttte
'new password')),
87:
[ext_html() (),
88:
89:
password2 forms CharField
90:
label ("Password confirmation")
91:
widget=forms, Password (attrs- l'autocomplete': 'new- plew-password'),
92:
strip=False,
93:
help_text=_f ("Enter the same password as before, for verification.").
94:
95:
96: class Meta:
97: model User
98: fields = ("username",)
99:
field_classes - ('username!: UsernamePield)
100:
(340
more lines below)
(Open file: /django /btarg//chungccccttt//////////ss
(Current directory: fdjango_djanne bash-$
104

---

SWE-agent
django django-1441 14411
We have opened the 'forms.py" file in the "django/contri/aauth directory and found the definition starting line 33. The widget
of the RRaa@OnlyPasswendaaaiiiiiig 'read _allswordd hash.html' for rendering. as specified line
template named
Since the issue is related to the label's 'for' attribute in the rendered HTML, we should look at the template file 'read_op
hash.html to see how the label is being rendered. Let's use the "find file' command locate this template file within the repository.
file
html"
Observation
django 1aggoodddano--441111
Found matches for "read_only_paswwrd_her htm1 in
/django /django_ddangge /djjngo__daanoo///oooo Ly password hash.html
(Open file: /django //wgg//chiggggcaaaataa//tt///taa (Current directory: /django_ (jagcc) bash-$
SWE-agent
django_ django-14411
We have located the template file 'read only password hash in the djaagoloontribaatt//een plates/auth/widgets' directory. Let's open this file to inspect the HTML and see how the label is being rendered.
open http://www..aaaaaaaa///////////a _ashh. html
Observation
Bhango__django--i
[File: http:///wwhaooooooot////////////
read read_onny wtw.loolssssooddaalalaal hash html (5 lines
total)]
1:<div($ include
$1>
2:18 for entry in summary $)
3;<strong>({ entry, label }}</strong> ($ if entry, value $): (1
entry.val value () endif $)
4:18 endfor $} 5:</div>
(Open file: /ajango //tpg///aaaagggccotttttttttttttt py) (Current directory: /django (django__jjngo))) bash-$
SWE-agent
django django-14411
template file 'read_only_ oaly_passwoot hash.html' does show a 'label' element 'for' attribute, which means the issue might be elsewhere. The issue description mentions that the label element for the RcaddOnlyPaswordHHatt Widget has a 'for' attribute pointing to non-lahclable element. Since we don't 't see 'label' element here, the 'for' attribute likely added in the Python code that generates the HTML for the widget diffcrent part of the template.
105

---

Given that the RReddOnlyPassee PasswordHHashWddeee is a custom widget, the rendering of the label with the 'for' attribute might be handled in the Python code where the widget used in form, rather than in the widget's template itself.
Let's go back the forms.py" file and look for the usage of "ReadOnlyPasswordee Widget in form where label might be rendered with 'for' attribute. We should look for form that includes and inspect how the label is being generated.
open cjango/contrrib/autt/ Forp://///tttttttoooooooo
Observation
_jango_ django-14411
[File: hbthh/gww.caaaaggooo/a/////a/o/t htrp://uttt//o....../. py (440 lines
- total)
1: 1:import unicodedata
2:
3rrrom django import forms
frro
auth import
3: authenticate, get_user_mocess fessword_validatee
6:)
7:from django. contrib. auth, hashers import
8:
UNDSABLE_PASSWORD
identify_hasher,
9:)
10:from django.contrile auth. mocels import User
auth
11:from cow.geo.ccorrb..uth......
import befault_token_generee
12:from django. saango.conrrib.ittes.cco who..ttttstssstutsss........s... import get_current_siee
13:from
import Val validationRrror
14:from
django.core.mal import EmaiMMultiAlteraatiivv
15:from
tjapgo.template import Loader
16:from
utlagg..uills.encciin import force bytes
17:from
django.utils.hhe import urlsafe__aseed_enneee
18:from
import capfirst
fr
lanng.utils.transe..
import gettext, gettext_lazy as
20:
21: UserMMdll get_user_mode _
22:
23:
24:def _ww_cooe__i_comppaae (s1, 82)
25:
26: Perform case-insensite comparison of two identi fiers,
+ using the
27: recommended algorithm from Unicode Technical Report 36, section
28:
2.11.2 (2).
29:
30: return www.dedaaa..oomallee............ (e ('NEKC', sl). caasefold)) == wat.ooeeaaaa.nmmmallaa......eaae ('NFKC', s2) casefold(()
31:
32:
33:class ReadonllyPasswrrssssttieee. (forms. Midget):
34: template_name =
http:oollyypsssxr____ea html
35: read_only = True
36:
37:
def get_context (self, name, value, attrs):
38:
context super hetllleee__ctttteeee (name, value, attrs) 39: summary []
40: if not value or
value. stl.esstartswihh (UNUSABL8_ (UNUSABUUSPAASSWOH__ BASSSSL___SSSRRD_____
41:
commary.appente (( 'label's gettext ("No password
set. 1)
42:
else:
106

---

43:
try:
44:
hasher sdentity_hasheee (value)
45:
except ValueError:
46:
summary. appendd (('label': gettext (" Invalid
password format or unknown hashing ")]
47:
else:
48:
for key, value_ in
hasher. sateerssaafeeeee www.ssauaeaarrraaaaaaaaaaaaaaaaa (value).items .items() :):
49:
summary.append ( label': gettext (key),
'value'; value_))
50:
context ('rttsssumna s'summaay'] summary
51:
return context
52:
53:
54:class ReadOn1 i (formss
55:
widget Readonlypasssorinahhii
56:
57:
def init (self, +args, **kwargs):
58: kwargs.setdefault ("required", False)
59: kwargs setdefalt ('disabled' True)
60:
super in_ iniiiintirggss **wargg))
61:
62:
63:class UsernameField
(forms (formm (tforms.ChaFFeeddd 64: def to_python (self, value)
65:
return aalcodedata.normall. 'NEKC', super toooppyoooo (valuee)) (value))
66:
67:
def widget_attrs (self, widget):
68:
return {
69:
**super () wi widget_attrs (widget), 'autocapitalize': 'none', 'autocomplete": 'username',
70:
71:
72:
73:
74:
75:class UserCreationForm (forms. ModclForm):
76:
***
77:
form that creates a user, with no privileges, from the given username and
78: password.
79:
80: error mecor_message "password_mismattee ("The two password fields didn't match.'),
81:
82:
83: passwordl forms. CharFielld
84:
label=_ ("Pes==(PPassoraa))
85:
strip=False,
86:
vidget=forms.Pe Passsoocin (attrs=( (
'new-password'))
87: oat.apaeetaaeoootaaaaaaaaaaaaaaa
text_htmi (),
88:
89:
password2 forms. CharField
90:
label= ("Password contirmation"), Soasstt=borssPPsswoooooonn widget=forms PassworaInput (attrs= 'autocomplete'; 'new-passwone pow-password'1,
91:
92:
strip=False,
93:
help_text=_ ("Enter the same password before, for verification.
94:
95:
107

---

96:
class Meta:
97:
model User
98:
fields = ("username",)
99:
field_classes - ('username': Usernamerield)
100:
(340
more lines below)
(Open file: /ajango htt.g//gwgaaagoooorr///////////a gets/read only ) (Current directory: aalann__jjange bash-$
SWE-agent
django _jango__django-144 django-14411
We are back in the 'forms.py' file where the 'ReadOnlyPassvve lash is defined. Since the issue is with the label's 'for' attribute, need find where the "ReadChiyyPaswoodHssshhdc is used a form and how the label is being rendered there. The "ReadOnlyPaseee Pashollaasssorosshisiihheee and it's likely used in the admin forms for the User model. We should search for the usage of PasdonlyPsswwwdHaashhh
is a custom field that uses the "ReadOniyyPsswoordHssiWWWWi
within this file to find where it's included in form.
Let's use the 'scarch_file file' command search for PeaaChniyaasswoddsssiiieett the current file.
search file "MeadoalyPPaswoocHaaahaiaaaal
Observation
django__django114 django-14411
Found matches for "ReadOoilyfasserrdsssiiiiia"""""
/django /djaago__djaaagoo//ooooao Line 54:class Readou/jesssocccsssnttdooooorood
Line 136:
password Readonlyyaasswrrddaahhiiildddd
End of matches for "#e:dbnllPaaserrddeeheiii.ddd""" in
(Open file: /django__aaae http:///a/aaaaa/a///////////a//a (Current directory: [ajjano__django_ bash-$
SWE-agent
django django_ django-14411
We have found two references reference line 54 the class definition itself, and the second reference line 136 is likely where the field used a form.
in the 'forms.py' file. The first
Let's navigate to line 136 to see the context in which RRaaChilyPaswwoodassaFeddd is used and how the label is heing rendered.
goto 136
Observation
django django-14411
[File: /ttp:ooww...aaooognno/oo////tttr (440 lines
total)]
(120 more lines above)
121:
if password:
122:
try:
108

---

123:
'user's password, but you can change the password password.valicaatoon wwwiaate htw.d/aatttppsoodddpppppppoopeee (password,,
self. instance)
124:
except ValidationError as error:
125:
(alf.cdderror((paasoorr error)
126:
127:
def save (self, commit-True):
128:
user super save (commit-False)
129:
user.set_paaskkeee (self. data ["paswword"") if commit:
130:
131:
user.save.
132:
return user
133:
134:
135:class UserChangeForm (forms Model
136:
password ReadonlyPasworrdhahh (
137:
1abel=_ label=_("Passwii ("Password"),
138:
@elp_text=_0
139:
'Raw passwords are not stored, so there is no way
this
140:
4 using
141:
'ka bref="()">thie form</a>.'
142:
143:
144:
145:
class Meta:
146:
model = User
147:
fields all
148:
field_classes ('username's Usernamerield)
149:
150:
def Wniiiit__ innit_issss arrgs, **wwargs)
151:
super init (.args, **kwargs)
152:
password self felf.fields.get. ('password))
153:
if password:
154:
paasword.holg_text
password. hhassorddholp.ccxtffomm.. ('/assssoo...////////
155:
uscr_permissions self fields get (' ('user_pormissioons))
156: if user_permissions::
157:
user wtt.pemmissions.quereeee
user jssr_perrissiion.quueee serponnssssssssss.e..errrreellld
('content_type)
158:
159:
160:ciass Authent ietiieenticaaiooooooooooo (tormooooooottoooFoooooooo
161: **"
162:
Base class for authenticating users. Extend this te get
form that accepts
163:
username/password logins.
164:
.**
165:
username =
UsernamcFicld ford Text Input (attrs=( l'autofcuus' Truc)))
166:
password forms.CharField
167:
label=_ [abel=_("Passoor"")
168:
strip-False,
169:
widget=forms. PasswordInpll (attrs- (attrs=((autocomplll laautccoppeete
"curzent-password")
170:
171:
172:
error messages
173:
'invalid_login':
174:
"please enter correct $ (username) s and password.
Note that both
175:
"Fields may be case-sensitive.""
109

---

176:
177:
'inactive's ("This account is inactive.")
178:
179:
180:
def _init_ (self, request=None, Margs, *karggs)
181:
"""
182:
The 'request' parameter is set for custom auth use by subclasses.
183:
The form data comes in via the standard 'data' kwarg. "*"
184:
185:
self request request
186:
self.users welfuuuse_ccchhe = None
187:
super () _nitiiitttt__ (+args, **kwargs)
188:
189:
the length label for the "username"
field.
190:
sel:. Geli.username_Faee field
UserMocel. peoocell_mmtt..om met.etta.geee field (UserModel U
USERNAME FIELD)
191:
lenghh self. uslnnuse field. length
254
192:
self.fields (aralidds'ussaaaaee ('username' ] _mm .max_length
@ssrrnae__ax__ennaeee
193:
self.fielde ('username| ] widget.attre 'mmxlennthh]]
194: if sel fields ('username') ] label None:
195:
self.fields "elf.fieedds'uusennnnee 'username' label
capfirst (sels.username_fiooe wew.olldvvvbboeeeeaaaea
196:
197:
def clean (self):
198:
username self www.ceeee_ddaa.. get ('username') password - self.cleaned_ee get ('password')
199:
200:
201:
if username is not None and password:
202:
self. self.user_cache cache authenticate (self. request, 4 username-usernane, password-password)
203:
if
is None:
204:
raise self.get_Imaaldllooonnoeeee
205:
else:
206:
self sel....ofiimmmoiiinnllleeeee (self. (se.....urr_ccchee
207:
208:
return walf.cleaned_ca..
209:
210:
def consirm allowea
Loo user):
211:
.*"
212:
Controls whether the given User may log in. This is policy setting,
213:
independent of end-user authentication, This default behavior is to
214:
allow login by active users, and reject login by inactive users.
215:
216:
If the given user cannet leg in, this method should raise "VValddtiinErree
217:
218:
219:
If the given user may leg in, this method should return
None.
220:
(220 more lines below)
(Open file: lctaago__ajaanoo//aaooooo http//////////////////////////// formss.yyrr/ (Current directory: _ajango__janoo) bash-$
110

---

SWE-agent
django (jango-14411
We now looking the "UserChangeForm class where the The field is instantiated with label on line 136, and the help text includes HTML anchor tag for changing password.
is used.
The issue reported is that the label element for the ReadOniyyPaswworHHshh Widget' has a "for' attribute which points to a non-late labelabbbbbe element. However, in the "UserChangeForm the label is defined in the Python code and not in the HTML template. This suggests that the 'for' attribute is likely added automatically by Django's form rendering.
To resolve issue, we need to ensure that when form rendered, the label for the 'ReadOnly- PasswordHash does not include a 'for' attribute. Since the rendering of form fields to HTMI. handled by Django's form rendering system, may need to customize the rendering for this specific field exclude the 'for' attribute.
One way to this override the 'label tag" method the PaeadOOnly to ensure does not render a 'for' attribute. Let's edit the PReadChlyyPusswodHHaffiddd class to include this customization.
We will add custom 'labelttag" method the 'ReadOniyyasswoddssaFiid class that returns label without the 'orr' attribute.
edit 54:60
class Reacoouspssssorriillllllllllllc (forms. Field)) widget Readool//Bassorcchheiiieeee
def inin__(el1t *args, <*kwargs);
xwargs. setdefault ("required", False)
waarg.setdefault ('disabled', True) **wargs)
super() s)
def label_ (sel_,, contents-Nene, attrs=None, sabel_suffix=None))
contents contents or self.
if Label_auffix is not None and label_suffix not in contents:
contents +- label_suffix
widget self.
id_ widget. ('id') or return format_htmn (=[(aaal()(((/Laabella
format html _tiit_oiiin""" []="[1"', sorted (attrs. (attrs. items )
if attrs else ()), contents)
end_of_edit
Observation
django__django-ll
Your proposed edit has intuode new syntax error Please understand the fixes and retry your edit commmand.
ERRORS:
FF21 undefined name 'format_html'
F821 undefined name "Format_html_jes
This is how your edit would have lccked if applied
[File: total)
(448 lines
(47 more lines above)
48:
for key, value_ in
hasher. sacherssafe_aee sawf_suumaayyyyyaaaaaaaaaaaaayya (value) ittms ():
49:
summary. append (I label': gettext (key),
ceumaary.ppped..
'value': value_1)
50:
context suumaayy')
summary

---

51:
return context
52:
53:
54:class keaaonnlyyassoordiashhiiiil..ll (forms (forms.Fiell)
55:
widget Readmnlyssssorrciahhiiiio
56:
57:
def init init_ *args, **wwargs):
58:
twargs.set:defaule (tt..attddeeaultteeerrereeeee False)
59:
kwargs. setdefautt (t ('disabled', True) super () init__ **kwargs)
60:
61:
62: def (abel_tag contents=None, attrs=None,
Label_auffix-None)
63: contents contents or self.
64:
label_suffit is not None and label_suffis
contents:
65:
contents += label__sufie
66: widget = self.
67: id widget. ('id') "
68: format_html_joinn/
return format_html (=<label))((<<aaeee))))
()="()"', catter itemtsssssittmmti.mm items ()) if attrs
else ()), contents)
69:
70:
71:class UsernameField (forms. CharField):
72: def to_python (self, value)
(376 more lines below)
This is the original code before your edit
[File: /django__aaa htwp:///iiiii///ooottt/tttttttt/ (440 lines total)
(48 more lines above)
49:
summary.append (( 'label': gettext (key),
'value'; value_))
50:
context ('smmmar'") summary
51:
context
52:
53:
54:class Raabouljfassorrriiiiieeettttotr. :ield))
55: widget Readoul/fasswrccteeeiddteett
56:
57:
def initt_ +args, **kwargs):
58: kwargs.ettdefault (""reuurrerr False)
59:
('disabled', aetreal (diabll True) super init (*args, **kwargs)
60:
61:
62:
63:class UscrnameFicld (forms. (
64: def to_python (self, value)
65:
return unicododata. iotpoeatta.nommaaaaa.oa.a nww.a 'NEKC', super () tt..tt__phhtt (to__ythho(vvalue) (375 more lines below)
Your changes have and try again.
been applied Please fix your edit command
You either need to 1) Specify the correct start/end line arguments
or 2) Correct your edit code. DO NOT re-run the same failed edit command. Running it again will lead to the same error.
(Open file: hstpangiihaaannogaooooo/oottttts
112

---

(Current directory: bash-$
/django a jango)
SWE-agent
jjango Gjango-14411
The message indicates that the names 'format htmll and 'format html joinn' undefined. These functions are utility functions provided by Django for safely creating HTML output. use them, we need to import them from "django.utils.hie Let's correct the edit by adding the necessary imports and retrying the edit command.
edit 54:60 From ejango.ut: 11 1s.htmm html import format_html, Format_htm1_3 _iiin
class fradllulPssssoodllaciefeeettttta
widget Reudaulyfassworciacntii.ee
def Unit__self,, *args, +*kwargs);
kwargs setdefault (e ("required", False)
saargsssetdffalle ('disabled', True) iatl___nnni__________ *kwargs)
def (sbel_ contents-None, attrs=None, >abel_suffix<None))
contents contents or seli.
it Label_suffix is not None and Jabei_ not in contents:
contents +- label_suffix
return format_html ("clabel>=)<//abbb]) contents)
ond_of_ede edit
Observation
django__daango django-14411
[File: httpang..cchhaaa/aam//////t/tttt (448 lines
total)]
(3 more lines above)
4:from contgi....rrbb.com import
5: authenticate, get_user_model, paasword_walldattionn
6:)
7:rrom django.contrib. hashers import
hathhe
8: UNUSAHHI__PASSWOOD_RRYYY
9:)
10:from cjango.contrib. wutheatrii.a.ut.mmm....... auth. import User
11:from www.pp..co.rbb.uu....ooom
import default token generator
12:from sjaang..cottribsstees. shortcute import cat_cuurentt... site
13:from craaggo.ore..exception import ValidationError
14:from
django.core.ne mail import Email
15:from
tempaattemplate import loader
16:from
chango.ut:ils.enoon import urlsafa_base6d_encee
import force force_bytes
17:from
18:from
import capfirst
19:from
jjango.utils. tta.g.uuttil..taalaaaii.... import gettext, gettext_
20:
21: USerrMMeee = get ussr__sodeee ()
22:
23:
24:def wunicooe_eiiccoppreee 82):
25: -**
26: Perform case-insensitive comparison of two identifiers,
using the
113

---

27:
recommended algorithm from Unicode Technical Report 36, section
28:
2.11.2 (B) (2)
29:
30:
return unicodedata wwn.oollllll. ize iee ('NEKC', sl). casefold unicodedata.normaa
('NFKC', s2) casefold()
31:
32:
33:clase a3:class Recchl//aasswrrdhssMlic..g (fomss :idget))
34: template_nai hatth///.ggttrrraaaboyaassseece. html 35: read_only True
36:
37:
def get_context (self, name, value, attrs):
38:
context super get_context (name, value, attrs) summary not value or value... startswith (ONDSA318_PASS90OR) PREFIX)
39:
40:
41:
Sumnary.appenee (1 'label: gettext ("No password else:
set.") 1)
42:
43:
try:
44:
hasher identify_hasher (value) except ValueError:
45:
46:
summary. sunmary.append (('label'; gettext ("Invalid password format or unknown hashing algorithm. ")
47:
else:
48:
for key, value_ in
hasher. watterrssffeeee sunnary (value) items ()
49:
summary-appeno append (( ('label'; gettext (key),
'value': value_))
50:
context [ ('summary'] summary
51:
return context
52:
53:
54:from django.
import format_html, format_html_jcin
55:
56:class caaoon//pesseoddlaaaaiillolooooo
57: widget Readonlyyssswoorcsslliddeeee
58:
59:
def _iiii_ (self, *args, **kwargs):
60:
wwargs.setdefane ("required", False) set ('aisabled', True) super() init init_ (+args, **wargg)
61:
62:
63:
64:
def label_tag contents-None, attrs=None,, labei_suffix-None)
65:
contents contents
66:
label_ is not None and label_suffi. net
contents:
67:
contents += label_suffix
68:
return Cormat_html ("1labcl> contents)
69:
70:
71:class UsernameField (formsshaaafiilld)
72:
def to_python (self, value)
73: return www.odeeaaa...mmaa.aaaooaoao (NEEKC,,
super () python (yttu((v (value))
74:
75:
def widget_attrs attrs (self, widget):
76:
return
77:
**super widget_attrs (widget))
78:
'autocapitalize" 'none', 'autocomplate": 'username',
79:
114

---

80:
81:
82:
83:class UserCreation"'rr (ontmoommffrmmmm (forms. (forms.McdelForm)
84:
85:
form that creates user, with no privileges, from given username and password.
86:
87:
88: error mcror__cssages-oo
89:
password_mismatchh ("The two password fields didn't match. )
90:
91: passwordl - Forms.charField
92:
label [ooel=_(Passwwrr"
93:
strip-False,
94:
xidget=torms (attrs=| 'autocomplete': oatp://cceettttaaootooooooooooe.
'new password'l),
95:
help_tex teppp-ssttss.ddd www.:eaaaiaalliaiiiiii.ooo.omoo. www.woord......dd.ooooooooo..oo. wtwpssororvaliddtrraaaaooao _elppp text_html (),
96:
97:
password2 - forms. CharField
98:
label= label=_("Passwil ("Password confi confirmatinn),
99:
widget=forms PasswordInput (attrs= "autocomplete':
'new- (Lex-password"))
100:
strip=False,
101:
Delp_text=_("Enter the same password as before, for verification.")
102:
103:
(345 more lines below)
File updated. Please review the changes and make sure they are correct (correct indentation, duplicate lines, etc). Edit the file again if necessary.
(Opon file: haapa///aaa//gaaggg//o/o//////// http://auutt/roo..yy/// (Current directory: /django djangc) bash-$
SWE-agent
django django__django-I 1jango-14411
The changes to the import of 'format html' and 'format_hrae atml_join",, and the addition custom 'label_tag" method that returns a label without the for' attribute.
class have been successfully applied, including the
Before submitting the changes, would be prudent to test that the changes have the desired effect. However, since we cannot run Django server render forms in this enviromment, we will proceed with submitting the changes.
Let's submit the changes to the code base using the 'submit' command.
submit
Observation
django__djang- django-14411
diff --git a/Changg/cootrrri//ttt//ffmssss b/cjang//comtrii///tt/ffooommp
index 011.922279...12299938989 100644
++ //cjaamg/ccootrrb/atttt//osssss
115

---

ee -51, 6 +51, 8 @e class RecbbnlyPaswercchhhiiiieee (forms. Widget) return context
+from django.utile utwossttils......... html import format_heml, format_htmi_join
class htw.oo///aasswdddssssssiioooee.o widget teedonllyfassooordaashhiiigg
ee -59, +61, 12 ee class hecoou/ppasserrcdleeeelddllol... kwargs sctdefault ('disabled', True)
super in ( (+args, **kwargs)
def label_tag sontents-None, attrs-None, 1abel_suffix=Nonee)
contents contents or self.labe
it label_ suffix is not None and label_suffix not in
contents:
contents label suffix
return format_html ( contents)
class UsernameFielle (formm (ta def to_python (self, value)
Gold Patch
django 1ijango-14411
diff --git
a/chango/coattii///ttt//om.mm b/tbaagg//canttt////tttt/tmmm//
ee 60, +50, 9 ee def get_context (self, name, value, attrs) context ['sumnary'] = summary return context
def lo_ (sclf, id_): return None
clas ReadOn ReadoOnyPasssunn ylasswordHashhiell (forms.Fiell) dget.
wi dget
116

---

E
Statements
E.1 Ethics & Broader Impacts
Prior works have established how the growing usage of LMs that can code also carries a certain amount of risk. We identify three main risks that could arise when building and using system like SWE-agent, then discuss how we incorporates measures that mitigate such problems.
First is the security risks that come with executing LM-generated code on device. When evaluating on software engineering tasks that feature an incredibly diverse number of issue descriptions, run- ning code generations on personal computing device can have negative side effects, such as the unintentional removal of digital assets (e.g., rm -rf asset/). To defend against this, we design SWE-agent to ephemeral containers for both inference and evaluation. SWE-agent's execution environment and the SWE-bench evaluation framework are both carried out in sand-boxed code environments, which is made possible with Docker. Executing code in Docker container ensures that its effects are mostly isolated from the rest of the system. While not considered as secure as virtualized hardware isolation, the namespace isolation provided by Docker containers is deemed sufficient for code that is not deliberately engineered to exploit recent container escape vulnerabilities. More details are discussion in $A.2.
Second, if the wider community develops interest for SWE-agent and builds upon it, it is also possible that illegitimate evaluation datasets infrastructure can be used to inject testing devices with malicious code or instructions to generate malicious code. For instance, an unofficial repository claiming to host inferencecovalluaiin harness for SWE-agen/bennhh could include task instance with an issue description that tells the LM agent to build key logging functionality and store it in hidden folder. To eliminate confusion and reduce the possibility of such an event, provide clear guidelines listed on our GitHub repositories, data stores, and websites indicating the official repositories and channels that we actively maintain. We also encourage third parties incorporate any improvements into our codebase and help with integrating such contributions.
Lastly are the consequences of software engineering agents being deployed in the real world. Prior works have conceptualized and put forth prototypes of agents that can carry out offensive security measures. It is also not difficult imagine that system like SWE-agent can be incorporated into pipelines resulting in the production of malicious code. SWE-agent's 's strong performance on SWE-bench implies that future AI systems will likely be increasingly adept in the aforementioned use cases. Releasing SWE-agent as an open source tool can support research towards designing sound, effective constraints for what software engineering agents are permitted to do. It can also serve system that legal experts and policy-making entities can experiment with to shape the future of what Al-driven end end software engineering could look like.
E.2 Reproducibility
To help the greater community reproduce the results presented in this paper and build on the SWE- agent platform, we open source all of our resources that were created for this project. The source code for the interactive pipeline, context management logic, command implementations, interface design, and everything else is entirely available in GitHub repository. We provide extensive text and video documentation describing how to run and modify different parts of the codebase. Practitioners should be able to easily recover our findings by running the agent with simple scripts. We also open source all inference and evaluation artifacts (e.g.. trajectories, code generations, evaluation execution traces, analysis notebooks). The results presented in the main and supplementary parts of this paper can be fully rendered from the data. Finally, we also maintain an active online help forum to assist with any reproduction problems questions about how to build ACI design and SWF-agent.
E.3
Limitations & Future Work
The final SWE-agent configuration has small toolkit, albeit highly effective. With SWE-agent's highly extensible design, we re excited by the prospect of adding more tools, such as web browsing or static analysis, that can leverage more signals from an issue description and codebase to improve the % Resolved performance. Many tools trialed by prior works from software engineering and language model agents, such as static/dynamic analysis, spectrum based fault localization, or test generation via fuzzing could prove useful.
117

---

Second, in this work, the ACI development process and case studies are done manually. Many components of SWE-agent were crafted from observations of recurring behavior within single trajectory across multiple trajectories. Automating part or all of this process could not only accelerate work built on top of SWE-agent, but also provide greater insights into developing ACI principles for agentic software engineering. Contemporary works have explored automated prompting to improve performance on traditional sequence to sequence tasks, supplanting the need for manual prompt design. Thinking about automating ACI design raises immediately interesting questions around how such systems can scrutinize and iterate upon their own designs. Ensuring such horizon leads to incremental performance improvements across a longer horizon is also challenging question.
Finally, the scope of SWE-agent is exclusively focused on programmatic tasks like software en- gincering and code generation. We're curious to see whether the same principles of ACI and our observations of agent behavior are transferable to different domains. Recent work around applying LM agents to a variety of digital work applications have proliferated, such as use cases in education technology, data analysis, and enterprise workflows. We hope that thinking about improving perfor- mance of agentic workflows on these domains through the lens of ACI design can be symbiotic process. For instance, for a task such shopping on the web, in place tool, could agents benefit from additional information beyond a list of each page's ' title and snippet? Would the design vary the nature of the downstream task were to change slightly? For a completely different task, such navigating internal company knowledge base to help recently on-boarded employee, how might the search interface be best adjusted to the agent?
typical Google-style search
Similar to the progression of the field of User Experience (UX) and Human Computer Interaction (HCT) research, applying ACI to other domains could not only yield improvements downstream task performance, but also further expand the list of ACI principles. We believe that the fundamental motivations for ACI, the foundational principles we put forth, and our case study SWE-agent as an instantiation of implementing and improving ACI can motivate such work.
118

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide a codebase that reflects the technical, conceptual, and any contribu- tions of this paper. The codebase includes the data, infrastructure, and analysis that allows one to reproduce the work fully. The abstract/itrroduction claims are discussed thoroughly in Sections 3, and 5. Our conceptual claims that ACI beneficial to agents are backed both thorough ablations of the final design along with extensive empirical experiments in both the main paper and appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have dedicated portions of the paper to discuss limitations specifically in $E.3 and throughout the rest of the appendix. We also discuss common failure modes in Section 5.2 and $B.3.2, where we propose potential next steps to addressing obstacles that SWE-agent consistently runs into.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
119

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not provide any theoretical results. We do not include any novel theorems, formulas, or proofs as part of our work.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We release a codebase containing all data, code, and results that allows the analysis and experiments of this paper be fully reproduced. To help make this process easier, we also improve extensive documentation both within the code and as separate markdown files within the repository. Since this work does not introduce a new LM or any model architecture changes, we do not include any checkpoints or code for model training. The SWE-agent codebase fully integrated with the APIs of the closed source, proprietary LMs that we rely on for our experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
120

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The evaluation datasets used from prior work are entirely sourced from HuggingFace, and the data has permissive licenses that allow use without charge. Along with the instructions, we push any and all code related to our work to the provided repository. The code includes the infrastructure for running SWE-agent a multi-turn environment with many proprietary and open source models. We also include reproducibility statement in B.7 that fully describes the lengths we went to in order to ensure our findings can be recovered by the community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/ /tts:://npspsss cc/
public/ guides/ CodeeSummi
onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.o.... cc/
bttaps.cc/uublc..o httpp//lllcccccc gu ligiuddee dess Cadees/CodeSaeee /et//CoodSubbaa Submis ssi onPol icyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full experiment details are included in Section 4. Since experiments rely mostly on closed source models, we provide details of the model-specific parameters, such temperature and top_k k in this section. In addition, we also discuss the hyperparameter sweep we ran to arrive at the final SWE-agent configuration, which is discussed in both the aforementioned section and in B.. All these choices are reflected in the codebase well.
Guidelines:
121

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run SWE-agent with the best configuration on the SWE-bench dataset multiple times to develop pass@k metric that we report Section and $B.5. The pass@k metric is used often in contributions around using LMs for code generation, first defined for this particular setting Chen al. [5]. Running a model or system multiple times shows the variance of the % Resolved metrics from one to the next, and also gives insight into the distribution of problems that are solved. We find that average performance variance is relatively low across runs, but per-instance resolution can change considerably.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eтrоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Since our experiments rely mostly closed source, proprietary models, we mention the specific checkpoints of the GPT-4 and Claude Opus models we use. We also provide estimations of cost and time it takes for trajectories to run both in Section 5 and throughout $B. Our project does not involve the meaningful use of any open source models novel LMs, do not have any details about compute for model training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
122

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps::/neurrin
Answer: [Yes]
Justification: Yes, the research process and results conform with the code of ethics. Our project does not involve human subjects or participants, and both the benchmarks used and the data generated by SWE-agent (e.g., predictions, trajectories) is licensed be free for the greater community. Since data is model generated, there is no personally identifiable information that would be from human task worker. The evaluation datasets released in the past several months with active contributions and leaderboards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In $E.1, included dedicated discussion of the broader impacts, particularly the implications that a system like SWE-agent has on the future LM agents' relationship with software engineering and code interaction. In this section, we have discussion about how SWE-agent could be applied towards offensive security measures and code. We provide several mitigation techniques. In Section 7, we include some brief discussion about next steps that can be taken after SWE-agent.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
123

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: In ŞE.1, we include discussions on the mitigation techniques we include for the SWE-agent infrastructure, such as execution sandboxes with Docker containers, to reduce any probability of unintended side effects that might arise from executing generated code on device. For usage of SWE-agent and evaluation on SWE-bench, we also include discussions in the aforementioned section of how we attempt to ensure no one is misled into installing unofficial repository by having well-maintained sites for pointing to official assets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the benchmarks that we evaluate SWE-agent on Sections and 5. We also mention that the data has licenses allowing evaluation and publication of results in the Experimental Setup section. We provide the sources of data and assets in $B. All data that we use for our project is available for use in the evaluation-oriented manner that we apply it. We also provide information about each dataset in $B.8.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper soperswitte thoode com. dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
124

---

The main new asset introduced by our work beyond the code is the trajectory data from runs of task episodes on SWE-bench. They are discussed extensively throughout the paper, both in Section 5 and ŞB. The trajectories are fully available in the anonymous GitHub repository, and we also plan to provide more infrastructure for easy viewing and use of trajectories.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our contribution does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our contribution does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
125


Reviews:
[{"review_id": "6KmZ6czvlq", "reviewer": "Reviewer_N4zh", "summary": "The paper presents SWE-agent, a system designed to enhance language model (LM) agents' performance in software engineering tasks through a specialized agent-computer interface (ACI). The ACI allows LMs to efficiently navigate, edit, and execute code within repositories, significantly improving performance over traditional non-interactive LMs. SWE-agent demonstrates good results on benchmarks such as SWE-bench and HumanEvalFix, showcasing a substantial increase in task resolution rates. The authors highlight the importance of designing interfaces tailored specifically for LMs, drawing parallels to human-computer interaction studies. The paper provides detailed experimental results and ablation studies to validate the efficacy of the ACI.", "questions": "1. How does the ACI manage context history to maintain essential information while avoiding unnecessary context? Note that the context length would be a bottleneck for many open-source LLMs.\n2. How does the system handle scenarios where the LM encounters repeated failed edits despite the linter's intervention?\n3. It seems that SWE-Agent is not a SOTA framework for SWE-bench. I am curious what the authors think about the recent SOTA approaches, and what would be the most valuable and promising directions for the next-gen of SWE-agent?", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The key idea of designing a specialized interface for LMs to interact with software environments is novel and well-executed. By abstracting and simplifying the interaction process, the authors address a significant gap in current LM capabilities. The experimental validation is thorough, with SWE-agent outperforming existing systems on established benchmarks, showing the practical significance of the approach. The writing is clear, making complex ideas accessible, and the data contribution is valuable, with detailed ablation studies shedding light on the importance of various design choices. Additionally, the open-sourcing of SWE-agent and its dataset contributions provide a useful resource for the research community, encouraging further exploration and validation of the proposed approach.", "weaknesses": "1. The process of managing context history within the ACI is not clearly explained, leaving ambiguity about how the system maintains essential information while avoiding unnecessary context.\n2. The method for optimizing the ACI configuration through manual inspection and grid search lacks transparency and could be elaborated further.\n3. The system's error recovery mechanisms, while improved with linting, may still struggle with repeated failed edits, indicating a potential area for further enhancement.\n4. The ACI seems to only support a limited number of tools -- how to expand it to other more advanced and useful tools are unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your time and consideration. You’ve brought up some excellent points in your feedback that we try to address below.\n\nRegarding your questions / weaknesses:\n\n**W1, Q1: How does the ACI manage context?**\n\n> The main mechanism by which SWE-agent manages to keep memory short, is by “collapsing” old environment responses into a placeholder message, keeping most of the visible history at any given time either related to the system setup and initial problem, or to recent environment observations. For example, if the agent has had 20 turns of interaction with the computer, the content of the first 15 environment responses will not be shown in full, while the last 5 observations will be shown in their entirety. This practice keeps the total context used for each turn shorter and cheaper.\n\n> We’ve clarified the language regarding the ACI context management and expanded discussion around Figure 9, which shows a diagram of system inputs and outputs. We’ve also added a figure showing the exact contents of a “collapsed” environment response, and now indicate this in Section 3 of the main paper.\n\n**W2: The ACI configuration choices lack transparency**\n\n> Due to the time and expense of developing on state-of-the-art proprietary models, such as GPT-4 (1106) and Claude 2, and the complexity of designing ACIs for challenging tasks, such as software engineering, the development process involved substantial qualitative analysis based on manual evaluations on the SWE-bench development split. We provide extensive ablations, showing the individual impact of many design decisions. We’ve expanded Section A.2 to provide further discussion on this process.\n\n**W3, Q2: How does the system manage repeated failed edits?**\n\n> This is a good insight, and currently, the system does not specially handle repeated failed edits beyond automatic linting. However, your suggestion is sound and could have clear benefits for future research on SWE agents.\n\n**W4, Q3: How can SWE-agent and ACIs improve further, provided results since submission?**\n\n> At the time of submission, SWE-agent was the SOTA system for approaching SWE-bench tasks. Since then, we’ve seen numerous papers design their own ACI’s with new tools, building on SWE-agent’s design philosophy. While your suggestions and incorporating the improvements proposed by others is definitely possible for further improving SWE-agent, we see the primary contribution of this paper to be the introduction of the ACI concept and our design principles. Future research has great potential to improve further upon these principles and the design of interfaces, tools, and model designs for agents.\n\nWe appreciate your feedback and we’ve made sure to incorporate your points to refine our paper. Thank you so much for your time and effort.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C0RLTpaqVD", "reviewer": "Reviewer_BuFF", "summary": "The paper introduces SWE-agent, a system designed to enable language model (LM) agents to autonomously perform software engineering tasks through a custom agent-computer interface (ACI). The study posits that LM agents can benefit from interfaces tailored to their specific needs, similar to how human software engineers use integrated development environments (IDEs). The SWE-agent system demonstrates significant improvements in task performance on the SWE-bench and HumanEvalFix benchmarks compared to non-interactive LMs, achieving state-of-the-art results.", "questions": "- What are the specific computational and resource overheads introduced by the ACI, and how do they impact overall system efficiency?\n- How generalizable are the ACI design principles to other domains beyond software engineering, and what modifications would be necessary?\n- Can the ACI be adapted for use with other language models with smaller context windows, and what would be the expected impact on performance?", "limitations": "n/a", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Novel Concept:** The introduction of the ACI tailored specifically for LM agents represents a significant innovation, addressing the unique needs of LMs in software engineering tasks.\n\n**Empirical Results:** The system achieves impressive performance metrics on established benchmarks, indicating the effectiveness of the ACI design.\n\n**Comprehensive Evaluation:** The paper includes a thorough analysis of the system's performance, including ablation studies and comparisons with baseline models.\n\n**Open-Source Contribution:** The authors provide anonymized code and data, contributing to the research community and facilitating reproducibility.", "weaknesses": "**Complexity and Generalizability:** The system's reliance on specific design choices and configurations might limit its generalizability to other types of LMs or software engineering tasks.\n\n**Overhead and Efficiency:** While the ACI improves performance, it introduces additional layers of complexity and potential overhead that are not fully quantified in terms of computational resources and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your interest in our research. We greatly appreciate your feedback and insights.\nWe’re especially happy that you see the novelty of the concept of ACI and the potential to impact future work in LM agents.\n\nWe’ve tried to address your particular concerns below:\n\n**What’s the overhead introduced by the ACI / SWE-agent?**\n\n> Safely running SWE-agent interactively, requires using a containerized environment, which will considerably increase the CPU resources required compared to most non-interactive systems.\nAdditionally, since SWE-agent provides an interactive environment for agents solving problems, each inference call for generating an action cumulatively consumes more GPU usage than typical non-interactive approaches.\n\n> However, as was demonstrated in SWE-bench, even using a near-perfect retrieval system (the oracle retriever in SWE-bench), non-interactive performance may have a lower practical upper-bound compared to interactive systems that can provide real-time execution and testing feedback.\n\n> Thus, at face-value, SWE-agent is a more expensive approach on both the CPU and GPU resources counts, but provided the increase in performance on SWE tasks, it is unclear what computational resources may be necessary for a non-interactive system to achieve comparable performance.\n\n**How generalizable are the ACI design principles?**\n\n> The short version of our design principles for the ACI is that interfaces should be: simple, efficient, concise, and help models recover from mistakes. While the SWE-agent ACI was built with the software development task in mind, the general design principles on the ACI should have a lot of transferability to other tasks and domains.\n\n> In SWE-agent, we show that despite LMs having been trained extensively on bash commands and scripts, they are less effective at using shells directly, compared to our simplified ACI which it had never seen before. Thus, we think that many of the lessons learned from the SWE-agent ACI design process can be applied to other domains and researchers building ACIs for different tasks.\nBased on your feedback, we’ve added some discussion about this in Appendix E.\n\n**Can the ACI be adapted for LMs with smaller context windows?**\n\n> For shorter context LMs, the ideal ACI would likely be slightly different than the default chosen for SWE-agent. At each step, we would need to minimize the number of tokens exchanged. The simplest approach to doing this would involve reducing the size of the fileviewer and performing more context compression as the interaction trajectory grows longer. We might expect a shorter context LM to still achieve non-trivial, albeit lower, performance on SWE-bench Lite, assuming that the LM still has the strong reasoning abilities of frontier models like GPT-4 or Claude.\n\nThank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lXaaNqoxho", "reviewer": "Reviewer_KbN4", "summary": "This paper introduces SWE-agent, a system that enables language models to autonomously perform software engineering tasks by interacting with computers through a specially designed agent-computer interface (ACI). The authors argue that LM agents represent a new category of end users with unique requirements, necessitating interfaces tailored to their strengths and limitations. SWE-agent's ACI is designed to enhance an agent's ability to create and edit code files, navigate repositories, and execute tests and programs.\n\nThe paper evaluates SWE-agent on two benchmarks: SWE-bench and HumanEvalFix. On SWE-bench, SWE-agent achieves state-of-the-art performance with a pass@1 rate of 12.5%, significantly outperforming previous non-interactive approaches. On HumanEvalFix, it achieves an impressive 87.7% pass@1 rate. The authors provide detailed analyses of how different ACI design choices impact agent behavior and performance.", "questions": "How might the principles of ACI design extend to other domains beyond software engineering? Are there specific challenges you anticipate in adapting this approach to different types of tasks?", "limitations": "Limitations are thoroughly discussed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* Strong empirical results: SWE-agent demonstrates significant improvements over previous approaches on challenging benchmarks, particularly on SWE-bench where it more than triples the previous best published performance.\n* Novel and important concept: The introduction of ACIs as a distinct category from human-computer interfaces is an important contribution that could shape future research in AI agent systems.\n* Detailed analysis: The paper provides extensive ablation studies and analyses of agent behavior, offering valuable insights into how different interface design choices affect performance.", "weaknesses": "* Generalizability: The paper focuses primarily on software engineering tasks. It's unclear how well the ACI design principles would generalize to other domains and how ACI could be implemented more broadly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments, all of which are very helpful for improving our work! We greatly appreciate your conclusion that ACI is a novel and important concept.\n\n**Q1, W1: Generalizability, Applying ACI to Other Domains**\n\n> Building a good ACI for a specific domain is tough, as we showed in the paper, and so here our focus was on the software engineering domain.\nOur decision to focus on coding is due to\n(1) Its practicality and popularity in current LM agent work (Sections [1,5]).\n(2) Software engineering as a task (e.g. SWE-bench) has a lot of room for improvement compared to other agentic settings, where performance is higher. Before SWE-agent, SWE-bench SOTA was 1.96%, and SWE-agent took that up to 12.5%.\n\n> Adapting an ACI to different domains is a very interesting research direction we hope the community is inspired to pursue.\nFor example, when considering a good ACI for web navigation, similar to our design process for SWE-agent’s localization tools, it may be worth considering the effect of redesigning search to be more agent friendly.\nFor instance, instead of Google-style search results, could agents benefit from additional information beyond a list of each result’s title and snippet?\nWould the design vary with the specific nature of the downstream digital task (e.g. shopping on the web vs. searching a company’s internal knowledge base)?\n\n> With future works, we hope that the study of ACI might progress similarly to HCI - with more ACI-oriented explorations in different domains, we’ll better understand how to design interactive task environments conducive to downstream performance.\nWe have expanded **Appendix E.3** with more content discussing this question.\n\nWe greatly appreciate this question for the opportunity to expand on such thoughts! Thanks again for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "acAEFbsK1B", "reviewer": "Reviewer_3ebt", "summary": "Recently, Large Language Models (LLMs) based agents have revealed huge potential in automating tasks. In this paper,", "questions": "1. Can you highlight the differences between SWE-agent and MetaGPT? MetaGPT is a multi-agent programming framework designed from software engineering. It also incorporates many unique designs in terms of software engineering.\n2. Can the proposed SWE-agent to other benchmarks? It seems like the design of SWE-agent is specialized for SWE-bench.", "limitations": "Please see my comments on weaknesses and questions.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work has presented a powerful AI Agent designed for software engineering, which configured agents with multiple functions like file viewer, file editor and so on. Such a system can significantly improve the capability of agent in programming and doing some software engineering works. Experimental results also demonstrate the capability of SWE-agent in solving tasks in SWE-bench.", "weaknesses": "1. This paper is a good engineering work but lack some insights from the perspective of methodology. For example, why we must choose search, navigation, file viewer / editor? Is it possible to add some other operations? It will be better to add some observations or ablation studies to support the motivation of this design, not just some system claims (e.g., Line 85 - 101). \n2. Compared with previous works like ReAct, SWE-agent more likes a version that extended with multiple sophisticated functions.\n3. Authors claim an agent-computer interface, but actually, it focused on coding or like a software engineering. I don't think that the proposed method can be considered as an agent-computer interface, as computer usually involves more complex functions (e.g., drawing, gaming, web search, and so on). I think such a claim is a little over-claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review of the paper and suggestions - the feedback has been helpful to clarify several details.\n\n**W1: Why these tools?**: \n\n\n> We discuss the design of SWE-agent’s interface in _Section 3_ and our motivation for this design based on principles laid out in _Section 2_.\nThe ablation studies discussed in _Table 3_ and _Section 5.1_ largely address how we settled on the final version of this interface.\nThese tools are necessary because without them, SWE-agent cannot carry out key steps to solving a software engineering task (e.g. edit code, navigate file system, execute programs).\nFurthermore, we observed that the Shell-only agent typically invokes commands that do exactly these types of operations (e.g. grep, find, cat, sed).\n\n> It is possible to add additional tools. SWE-agent’s codebase is designed to make configuring an interface easy (Technical disc. in _Appendix A.3_ + Ample documentation in SWE-agent codebase). Since its introduction, several projects have built upon SWE-agent, adding features like improved software analysis and editing.\n\n**W2: ReAct vs. SWE-agent**\n\n> While SWE-agent’s fundamental interactive loop is inspired by ReAct, it is _not_ simply ReAct + tool use / programming language, which has been explored by other works (e.g. TroVE [1], InterCode [2], CodeAct [3]).\nWhile tools are a significant facet of an Agent Computer Interface, we discuss and show the qualitative (_Section [3,5.1]_, _Appendix [A,C]_) and quantitative (_Table [1, 2]_, _Appendix B_) impacts of an agent-friendly ACI, which encompasses far more elements such as context management, i/o format, guardrails, etc.\n\n> Prior works tend to focus on how providing a tool helps performance.\nWe show that a tool can be configured (in terms of input/output, documentation, usage, implementation) in many ways, and that these choices have tangible impact on performance.\nAs expanded upon in responses [1, 5], SWE-agent is certainly extensible and allows easy addition of more functions.\n\n> [1] https://arxiv.org/pdf/2401.12869\n> [2] https://arxiv.org/abs/2306.14898\n> [3] https://arxiv.org/abs/2402.01030\n\n**W3: ACI is Not General Purpose**\n\n> While SWE-agent was designed with the domain of software engineering in mind, our goal with ACI is to generally convey that developing specialized interfaces for LM agents (ACIs) leads to tangible impact on downstream performance. The structure of our paper reflects this. Our motivations for studying interactive environment design (_Section 1_) inspire our preliminary definition of ACI (_Section 2_). SWE-agent’s design process is a case study on implementing and iterating ACI in practice (_Sections 3, 5_).\n\n> Demonstrating ACI’s efficacy on a large number of agentic tasks (e.g. web browsing, theorem proving), would likely be an effort that would far exceed the breadth of what a single paper can cover. As alluded to in Response 2, we hope and have seen some evidence of how ACI principles have led to the recent flurry of SOTA improvements on SWE-bench.\n\n> There is no concrete work that applies ACI methods to non-code domains, but we hope that the concrete takeaways and SWE-bench performance improvement will inspire such explorations.\n\n**Q1: SWE-agent vs. MetaGPT**\n\n> MetaGPT puts forth a sophisticated network of multiple agents with different roles that follow a Standard Operating Procedure. Their work focuses more on coordinating multiple agents and communication interfaces between agents. MetaGPT also has not been evaluated on SWE-bench or other software engineering or repository-level coding benchmarks (e.g. RepoBench). They primarily run experiments on HumanEval and MBPP.\n\n> We do not study multiagent systems with SWE-agent. Our focus is fundamentally different - we identify pain points in how an LM agent interacts with a computer, then upgrade the interface that sits between (e.g. tools, I/O format, context management) to reduce common pitfalls and improve actions’ usability and feedback. We will add this comparison between MetaGPT and SWE-agent to our related work section.\n\n**Q2: Extensibility to other benchmarks**\n\n> SWE-agent _can be used_ for other coding benchmarks. Our paper presents results for SWE-agent on HumanEvalFix (see _Table 2_), achieving a 87.7% fix rate. SWE-agent also works for traditional code completion benchmarks (e.g. HumanEval, MBPP). However, we did not release such results as (1) performance on these benchmarks is near saturated, and (2) Such benchmarks only test for a minor subset of SWE-agent’s capabilities (e.g. localize error, reproduce bug, navigate file system are _not_ tested), drawing focus away from our work’s main deliverables.\n\n> While SWE-agent cannot be directly applied in its current form to non-coding agentic tasks such as web browsing (e.g. WebShop, WebArena), we believe and hope that future LM agent works can further study and build upon ACI design principles across different domains. This was briefly discussed in Appendix E, and in the latest draft, we have extended this discussion to provide more concrete inspiration for the community.\n\nThank you again for your feedback! We’ve uploaded a new version of the paper which addresses your concerns. If you have any remaining questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method, SWE-agent, which is built upon a large language model to help with software engineering tasks. The paper develops an agent-computer interface (ACI) to help facilitate interaction with a coding environment. The paper evaluates SWE-agent on two benchmarks, showing positive results.

The reviewers were all positive. One reviewer raised a score after the rebuttal while others did not. Some concerns remain regarding the over-focus on SWE Bench, as it calls into question the generalizability of ACI. Further, the proposed method could have been studied more exhaustively with better/additional ablation studies. However, the reviewers appreciated the importance of the problem and the potential impact of the paper.
Decision: Accept (poster)