RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code
Authors: Dhruv Gautam
Spandan Garg
Jinu Jang
Neel Sundaresan
Roshanak Zilouchian Moghaddam
Venue: ICLR 2025 Poster (2025)
Keywords: Language Agents
Benchmarks
Code Generation
Reasoning
State-Awareness
Refactoring
Long-Horizon Tasks
Knowledge Representation
Primary area: datasets and benchmarks
TL;DR: New benchmark of multi-file code refactors to test language agent reasoning and explorations into state updates to improve language agent performance.

Abstract: Recent advances in language model (LM) agents and function calling have enabled autonomous, feedback-driven systems to solve problems across various digital domains. To better understand the unique limitations of LM agents, we introduce RefactorBench, a benchmark consisting of 100 large handcrafted multi-file refactoring tasks in popular open-source repositories. Solving tasks within RefactorBench requires thorough exploration of dependencies across multiple files and strong adherence to relevant instructions. Every task is defined by 3 natural language instructions of varying specificity and is mutually exclusive, allowing for the creation of longer combined tasks on the same repository. Baselines on RefactorBench reveal that current LM agents struggle with simple compositional tasks, solving only 22\% of tasks with base instructions, in contrast to a human developer with short time constraints solving 87\%. Through trajectory analysis, we identify various unique failure modes of LM agents, and further explore the failure mode of tracking past actions. By adapting a baseline agent to condition on representations of state, we achieve a 43.9\% improvement in solving RefactorBench tasks. We further extend our state-aware approach to encompass entire digital environments and outline potential directions for future research. RefactorBench aims to support the study of LM agents by providing a set of real-world, multi-hop tasks within the realm of code.

Published as a conference paper at ICLR 2025
REFACTORBENCH: EVALUATING STATEFUL REASON- ING IN LANGUAGE AGENTS THROUGH CODE
Dhruv Gautam" Spandan Garg² Roshanak Zilouchian Moghaddam
Jinu Jang?
Neel Sundaresan
'CC Berkeley Microsoft
ABSTRACT
Recent advances in language model (LM) agents and function calling have en- abled autonomous, feedback-driven systems to solve problems across various dig- ital domains. better understand the unique limitations LM agents, we intro- duce RefactorBench, benchmark consisting of 100 large handerafted multi-file file refactoring tasks in popular open-source repositories. Solving tasks within Refac- torBench requires thorough exploration of dependencies across multiple files and strong adherence to relevant instructions. Every task is defined by 3 natural lan- guage instructions of varying specificity and is mutually exclusive, allowing for the creation of longer combined tasks on the same repository. Baselines on Refac- torBench reveal that current LM agents struggle with simple compositional tasks, solving only 22% of tasks with base instructions, in contrast to human developer with short time constraints solving 87%. Through trajectory analysis, we identify various unique failure modes of LM agents, and further explore the failure mode of tracking past actions. By adapting baseline agent to condition on representa- tions of state, achieve 43.9% improvement in solving RefactorBench tasks. We further extend our state-aware approach to encompass entire digital environ- ments and outline potential directions for future research. RefactorBench aims support the study of LM agents by providing a set real-world, multi-hop tasks within the realm of code.
INTRODUCTION
"Repetition is the root of all software evil" - Martin Fowler
Large language models (LLMs) have been quickly acquiring new capabilities (Bubeck et al., 2023), leading towards adoption of Al-powered systems in various formats and domains. The increasing usage of LLM powered tools like Github Copilot have greatly improved the capability of develop- in software development tasks (Peng et al., 2023). More recently, emphasis multi-step execution through LLM feedback loops has unlocked the ability to solve harder problems within a variety of fields (Reed al., 2022; Sumers al., 2024; Yao & Narasimhan, 2023), including parts of software engineering.
This new paradigm of solving larger software tasks has led the construction of a variety of new automated software engincering (ASE) systems, most being structured as LM agents (Wang et al., 2024c; Cognition.ai, 2024; AWS Q Developer, 2024: Gauthier, 2024; Aide. dev, 2024; Örwall, 2024; Yang et al., 2024; Tufano et al., 2024; Wang et al., 2024d; Chen et al., 2024a; Zhang et al., 2024b; Arora et al., 2024; Xia al., 2024; Zhang al., 2024a). Evaluations for such systems are currently largely comprised from real world data on Github (Jimenez et al., 2024; LaBash et al., 2024). While being the strongest open-source signal for software engineering tasks at scale, Github inherently noisy through its snapshot nature, also requiring strong filtration and validation testing for reliable evaluations (Chowdhury et al., 2024; Bowman & Dahl, 2021). We find that the filtration causes skewed task styles, creating necessity for new data to diversify coding benchmarks.
Wook done during internship at Microsoft. Correspondence to thruvgautam@hertee
Data available http://hhhh...mmmmmccooo///fffee

---

Published as a conference paper at ICLR 2025
RefactorBench
Generated Patch
axe autonomoue programmer
graph paranetace dapandency. Use this class in all calls to add dependency through the zepo.
OO
tcon .bsapacs iapoet haodle, feom chelpere ixport hastle_int
gat a/arc/tient/ a/arr/fien//__mtt__ Init__. 77
<0s.7
refoctoring....
Reference Solution
Abstract Syntax (AST) Unit Testing
Repository
-dass -Jllseza sepost
+39.7
test, from directory helprr.eeeeerrrrrrrrrs
Assarstreal...
hawper,
scatble refectee selery_rutaccie <jaago rafacsor testepLratecter
-drue 'elpees
handin_int int
tast send faso dicestary call, Ln, test (ead)ce
colf Nasor 1 maleeo
salt refecter
refector
main
tornado refactes
Figure I: RefactorBench task instances are multi-file refactors verified by custom AST unit testing. Tasks are split over popular open-source Python repositories and reference solutions are withheld to prevent overfitting to the task.
To address these challenges, we present RefactorBench, a benchmark designed to evaluate the largely undocumented task of multi-file code refactoring in large codebases. Unlike isolated function-level edits, multi file refactoring requires comprchensive reasoning and composition of multiple smaller changes. Our benchmark, RefactorBench, also allows for controlled analysis into instruction-following through multiple instruction sets with specified and unspecified objectives. As LLMs have been extremely proficient in function level editing 2024), we find it important to evaluate the abilities of general LM agents given that they can reliably perform core subtasks, which we verify in RefactorBench' 's thorough filtering process. With unique abstract syntax tree (AST) based unit testing, the evaluation suite checks for a comprehensive variety of subtasks necessitated by the core refactor without dependence on exact line match.
model generations (Jiang al.,
Through evaluations of a baseline agent on RefactorBench, we find overfitting and poor perfor- mance, solving a maximum of 35% of tasks with our easiest instruction set. We also see a variety of unique failure modes, many centering around LM agents struggling to track and reason about previous actions. Similarly, extensive work policy learning has commonly faced issues long horizon execution (Piterbarg al., 2023; Chen et al., 2024b; Hejna et al., 2023). By editing agent interfaces, we explore introducing conditioning over state updates, a common tactic neural agent design, to our real world language agents and see 71 % increases in subtask completion rates.
Overall, our contributions in this work are threefold:
1. We introduce RefactorBench, a benchmark of code refactoring tasks that necessitate edits in multiple files and reasoning based on previous actions taken.
2. We evaluate multiple open-source systems on RefactorBench and analyze three novel fail- ure modes isolated through differing baseline runs.
3. We construct state-awar interfaces and show improvement in the reasoning capabilities of a modified baseline agent.
BACKGROUND
2.1
RELATED WORK
Benchmarks SWE-hench, a benchmark consisting of GitHub issues, is the community standard for evaluating open-ended problem-solving in code environments (Jimenez et al., 2024). Our work, comparably, focuses on handcrafted and underrepresented multi-file refactoring tasks, isolating unique language agent behaviors. Unlike existing function-level code benchmarks (Chen et al., 2021; Austin et al., 2021), which also include refactors, we concentrate on the challenges posed by multi-fil file edits. Through lazy and descriptive instruction set to accompany base instructions, we also build on previous works that scale evaluations of LLMs' instruction-follovii ties with lazier instructions (Cassano et al., 2024). Recent works have also focused evaluating
capabili-
2

---

Published as a conference paper at ICLR 2025
repository-level code completion systems (Liu et al., 2024a; Agarwal et al., 2024; Bairi al., 2023), but our work differs by evaluating larger actions than exact line match, using generalist evaluations through AST testing. New works have also started to benchmark LLMs across the life cycles of var- ious engineering tasks (Li et al., 2024; Huang et al., 2024; Xie et al., 2024c), and recent LM agent benchmarks have also started to evaluate for planning, reasoning, and decision-making abilities in multi-turn generation settings (Liu et al., 2023b; Xie et al., 2024b). We combine these two threads by evaluating on engincering tasks that necessitate multi-turn actions. Moreover, newer benchmarks have shown that emulating differing environments can help identify unique failure modes (Ruan al., 2024; Yao et al., 2024). Similarly, in RefactorBench, we find that multi- dependencies in code provide strong test bed for previously unseen failure modes in LM agents.
Compositional Tasks and Memory Some benchmarks have identified modeling long-term de- pendencies as a difficult task for core LMs (Tay al., 2021; Xie al., 2024b; Wang al., 2023b; Lee et al., 2024a). To combat this issue, many works have targeted changes in model architecture and training (Gu et al., 2022; Liu et al., 2023a; Gu & Dao, 2024; Xiong al., 2024). Other works have tackled this style of problem through augmenting LM agents to have external memory in order to learn longer term skills after large sequences of actions (Sarch et al., 2023; Shinn et al., 2023; Wang et al., 2023a; Sumers et al., 2024). Differing from long term memory in language agents, we largely focus on enabling concurrent reasoning and state-aware behaviors in language agents. Comparable to this focus, many works in embodied control and small neural agents have previously explored training and conditioning over observations about current state in multi-tum situations (Chen et al., 2024b: Wang et al., 2024a; Piterbarg et al., 2024; Li et al., 2022; Moreno al., 2021). We explore extending these concepts to real-world LM agents to improve their performance.
2.2 DEFINITIONS
We generalize varied perspectives in previous literature to construct our definition of language/LM agent and related terms: a core L.M receives an user instruction u and executes actions on using set of tools f.. receiving partial observations win. This follows a structure most similar to partially observable Markov decision process (POMDP) (Kaelbling et al., 1998), where the trajectory TN = (a1, .......; AN, WN). This largely matches the formalization of LM agents articulated in ToolEmu (Ruan et al., 2024). We also define and use the words state or stateful in the context of LM agents as the nature of being dependent necessarily all generated from the LM. Importantly, stateful reasoning focuses on making decisions based on the current state, which is partially observable and can change dynamically. Previous works in building LM agents have recognized the importance of designing interfaces that allow the core LMs to make better decisions for variety of tasks (Yang et al., 2024; Liu al., 2024b; Wang et al., 2024d; Lu et al., 2024; Shang et al., 2024). We reference the design choices behind tm and it's impact on Wn, On as interface design.
the accumulation of actions and observations w, though not
3 BENCH
RefactorBench is benchmark of handcrafted multi-file refactoring tasks. The goal for each task is to generate a patch that changes the repository follow the rules of a specified refactor, In this section, we describe our end-to-end process of constructing the refactoring tasks and highlight some important features of RefactorBench.
3.1 TASK CONSTRUCTION
To design a benchmark capturing the common practice of code refactoring, we focus on including diversity of styles of tasks, using Fowler et al. (2018) a reference point for different styles of refactors. As the test beds for all tasks, we first select popular Python repositories that have differing overall file structures (Table 1). We then run the below four step process each repository:
Step 1: Localization and Filtering. We leverage LLMs to identify potential refactoring opportuni- 4o (OpenAI, 2024) with complete files from target repository, along with examples of various refactor types from Fowler et al. (2018), requesting line numbers and suggestions for potential refactors. We then filter through the returned sites manually
ties in repositories. We iteratively prompt got-4o
3

---

Published as a conference paper at ICLR 2025
to verify if corresponding suggestions can be made and if the changes would affect multiple files. This process yields a list of refactoring suggestions and their corresponding edit locations in each repository.
Step 2: Construction of Reference Solutions. To generate a prospective reference solution for each refactor, group of experienced Python programmers handeraft unique, related edits to the refactoring suggestions generated the previous step. These edits are made based on the design principle (Fowler al., 2018), while concurrently using gpt verify that each core refactor is tractable by the language model. Tractability verification is done through prompting gpt- 4o with the file to edit, the design principle, and summary of the change needed be made. We define this process in depth in Section 3.2.
Step 3: Development of Testing Files. Once the tractability of the subtasks are verified, the de- velopers then create relevant unit tests for each overarching task. At minimum, for every core edit verified in Step 2, new unit test generated that parses through the respective file's AST and verifies that changes have the correct broad code structure and syntax necessitated by that sub- task, removing dependence on exact-match testing (Appendix D). This iterative approach creates a breadth of tests that comprises necessary minimum for the total refactor. At test time, LM agent's generated solution is applied to the codebase and the associated tests crafted for the task instance then executed. generated patch is considered successful, all of the relevant AST tests pass.
Step 4: Generation Relevant Task Instructions. After reference solution and AST test creation, the developers write short, but comprehensive task summary to help in the instruction design phase, what we refer to as the base instruction. In order to evaluate different degrees instruction following with specified and unspecified objectives, we generate instruction and descriptive instruction. These instruction sets are generated through a few-shot leamiing prompt with the respective base instruction and the related unit tests (Appendix A.1).
other instruction sets: lazy
The above four step process yielded 100 large overarching multi-file refactoring tasks and corre- sponding tests in 9 different Python repositories. Throughout this work, we report the success on run based on passing all tests for task. Table I and Figure 2 show a breakdown by repository and other statisties related the tasks.
Django (18.0%
Scrapy (13.0%)
Table 1:
Statistics RefactorBench tasks, repositories, and AST-based unit tests.
Tamaao (9.0%)
Mean
Max
Ansible (11.0%)
Lazy Instruction
Length (Words)
16.0
38
Requests (10.0%)
Base Instruction
Length (Words)
20.6
44
Desc. Instruction
Length (Words)
68.8
116
Celery (12.0%)
Tlesk (6.0%)
Files
2327.6
6815
Codebase
Lines
304K
1.8M
FastAPI (6.0%)
Salt (15.056)
Reference Solution
Files edited
4.3
31
Length (Functions)
6.5
27
Figure 2:
Distribution RefactorBench tasks across 9 open source Python repositories.
AST Tests
Length (Lines)
131.1
378
3.2 IMPORTANT FEATURES
Multi-File By filtering out single-file refactors as part of our task construction process, all tasks in RefactorBench involve multi-file edits. Our tasks edit between 2 31 files, with files edited in our reference solutions on average. This feature, by definition, detracts the ability of single-shot LLMs of solving the tasks, and forces feedhack-based editing systems to reason over multiple files.
Varying Instruction Sets RefactorBench offers three sets of instructions with varying degree of de- scriptiveness. With multiple instruction sets, we are able to test for a breadth types instruction- following and provide a way to effectively scale the difficulty of RefactorBench. The lazy instruc- tions match the styles of real users, where objectives are often unspecific. We also include the base instruction which describes the task completely in succinct manner. And through the descriptive instruction, we are able to evaluate on an exhaustive instruction where systems are given insights on what they will be tested on, theoretical upper bound on performance.
4

---

Published as a conference paper at ICLR 2025
Subtask AST Testing In RefactorBench, unit tests for each task are designed to cover various subtasks the LM agent needs to accomplish. During the test construction process, we separate the unit tests to break apart the behavior of subtasks within tasks. This makes understanding the failures within patches an interpretable and quick process. For instance, one can evaluate which files the agent makes edits in, giving more comprehensive understanding of the order of tasks and proximity to correct solution. RefactorBench's 's unit tests comprise of to 27 subtests, with an average of 6.51 tests per task. See Appendix for an example test file and Appendix E for multiple test outputs seen through the lens of this subtask testing format on RefactorBench.
Tractability Through verification steps during task construction, we also make sure that all the core edits are feasible by frontier models at the time of writing. Due to this, our refactors have stronger signal on evaluating the reasoning behaviors between function calls of L.LM feedback loops rather than the broad open-ended task of generating passing code changes. Similarly, previous work has also recognized the importance of focusing agent benchmarks to interpretable subtasks (Côté al., 2019; Xie et al., 2024b; Shridhar al., 2021; Chowdhury et al., 2024). Overall, this tractability requirement allows for a more dedicated focus on evaluating the stateful reasoning abilities of LM agents.
EXPERIMENTS
In this section, we explain our approaches to evaluate language agents on RefactorBench. All main studies done on SWE-agent, which is the highest performing open-source agent framework on the full SWE-bench split at the time of writing. SWE-agent also structurally follows our earlier definition of POMDP-based LM agent (Yang et al., 2024: Jimenez et al., 2024), while other agents sample from multiple agents (Wang et al., 2024d), weakening ablation studies. Often, due to costs and rate limits on model endpoints impacting efficient ablation studies, we opt to use gpt-4 4 in experiments, but find that our results scale similarly across models.
Update:
These baselines are outdated compared to current SWE-agent implementations. We share more updated baselines in Section 7.3.
4.1 PRELIMINARIES
Current systems have overfit to solving reproducible bugs. As a prior, we observed poor per- formance for some LM agents when running them on simpler tasks in RefactorBench. Upon inves- tigating internal code of a few open-source LM agents, find that their internal prompting and in-context examples steer towards solving Github issues. This task specific prompting causes these language agents to treat refactoring problems bug-fixes. For instance, many systems will attempt to create a bug reproduction script for simple renaming task. We causate this initial finding as result of having a lack of benchmarks: it is hard robustify LM agent systems without ways to eval- uate on diverse styles of tasks (Kapoor et al., 2024; Dehghani et al., 2021). For the rest of this work, to better understand the frontier of capabilities within current systems, we alter internal prompts to focus on the task of refactoring. We therefore consider these baselines as an upper bound on perfor- mance, and hope for future systems to be designed accordance to and evaluate over diverse styles of problems. We discuss directions for future systems and generalist performance in Section 7.
4.2 BASELINES
Using containerized framework that emulates a user file system with the target repository, run a baseline of SWE-agent on all RefactorBench tasks with a per instance cost limit of $10.00?. We report the percentage of completely successful task instances on each run. On the lazy, base, and de- scriptive instruction sets, SWE-agent with gpt 4 solves 12%, 18% and 27% respectively. To verify generalization across models, we also run the descriptive baseline with cl claude-3 5-sonnet, which solves 35% of the test cases completely. To contextualize this performance, we have profi- cient human developer attempt all the tasks within the benchmark, with a limit of 5 minutes per task
Thii cost is due large taken counts being necessitated with multi-file reasoning.
5

---

Published as a conference paper at ICLR 2025
Performance Comparison Instruction Sets
100
Lazy ons
Tasks Solved in Refactor serrtennch
Baxe Instructions
Table 2:
Baseline task performance relative to instruction type. Through the varied categories, we find that lan- guage agents are sensitive to unspeci- fied objectives (Lazy) and improve in performance greatly when given infor- mation on which files make edits in (Descriptive).
Descriptive Instructions
60-
40-
20-
Instruction Type
Resolution Rate
Promot Adjusted SWE-agart
Haman Developer <5 Minutes
Baselines
Lazy
12.0%
Figure Baselines of prompt adjusted SWE-agent with gpt-4 and human developer given IDE access and time limit of 5 minutes.
Base
22.0%
Descriptive
27.0% %
using the base instructions, and they solve 87% of the test cases. The average length of successful trajectory using gpt- is about 45.8 actions and the overall average length about 58.5 actions.
Additionally, we sample random solved RefactorBench instances in repositories that have 3 or more solved, and combine their descriptive prompts to run as singular instances. We find that SWE- agent, although able to solve the singular tasks, is unable to solve any of these longer pseudotasks. We further discuss related results and tackle long horizon planning in later sections of this work.
ANALYSIS
From manual review of trajectories on RefactorBench, we find repeating general behaviors language agents perform. Many prior works have outlined some strengths and failures of current LM agents in different scenarios (Yang et al., 2024; Wang et al., 2024d; Xie et al., 2024b). As such, we focus on three novel failure modes isolated through our baseline experiments. After large-scale human review of trajectories and developing an understanding of failures, to confirm their prevalence on a larger scale, we use gpt-4 with reference solution diffs to analyze unresolved trajectories and the respective test outputs following one of the failure modes in this section. Through this, we classify about 58% of failed trajectories are corresponding to one of these failure modes, and in held out validation, human reviewer agreed with the classifications about 74% of the time.
Agents fail find relevant locations and make applicable changes. Through our task construc- tion, our descriptive instructions provide information all files that need to be edited. However, still observe through about 44% of the tests checking for some change, agents initialized with descriptive instruction did not edit the target files, although being prompted to. These results differ from previous results that firmly found that most LM agent coding systems create patches at the correct location, and mainly fail through incorrect implementations (Chen al., 2024a; Yang al., 2024). Moreover, none the tasks that require changes in more than 6 files are solved in any of our baselines. These results complement previous work evaluating planning capabilities of LLMs, where increases in constraints correlate with decreases in performance (Xie et al., 2024b; Huang al., 2022). We hypothesize that the increase in files serves as a proxy constraint and LM agents fail in both efficient exploration and composing previous actions. We formalize the related problems of action tracking and stateful reasoning in-epph and tackle through state updates Section 6.
Agents fail due to interactions that necessitate erroneous intermediate states, Our classifica- tions also show that 78.4% of trajectories error in a code editing step. Through analyzing these tra- jectories, we commonly encounter cases where making change that temporarily introduces errors a necessary step to solve the task. This often because subsequent modifications, either within the same file across multiple files, are concurrently required resolve these issues. Consequently, the practice of automatically enforcing strict linting rules and rejecting edits based on errors proves to be an impractical approach for scaling real world agents, even though most open-source systems have previously found en-t linting to significantly boost performance for bug fixing (Örwall, 2024; Yang et al., 2024; Wang et al., 2024d). This identified scenario demonstrates that LM agents often

---

Published as a conference paper at ICLR 2025
imitate human forms of interaction, and removing innate capabilities through guardrails can backfire in unintended manners. We further discuss unobstructed LM agent interaction in Section 7.1.
Agents fail due to context flooding and losing sight of objectives. We find that agents struggle in decision making after having commands that are rejected due to formatting issues or unexpected output (Figure 8). In recent work defining in-context reward hacking (ICRH) (Pan et al., 2024), LLMs, through feedback loops in small synthetic tasks, have been shown to model proxy objectives when optimizing over some larger objective. We find, in our real world task of refactoring, that the negative effects from ICRH are also accentuated by extensive context space being taken up by the handling of constraint violations, deprioritizing the initial objective in a form few-shot learning (Brown et al., 2020). Specifically, a common linting error edit in our tasks shows the model an average of about 1,466 tokens à: comprising of two blocks of code, error handling prompts, and the flagged errors. We find that this lengthy repetition for error handling function calling weakens trajectory structure. For instance, in some trajectories, we see the agent run end after an intensive function level feedback loop is resolved, a form of prioritizing the objective. Language mod- els losing sight of initial goals has recently been tackled within single-shot code generation tasks through attention dilution (Tian & Zhang, 2024), but find this new issue is prevalent in LM agent trajectories, and is exacerbated through the context-expensive feedback loops. We further discuss ways approach robust trajectory reasoning in Section 7.1.
TOWARDS STATE-AWARE LANGUAGE AGENTS
In our analysis, we found a general issue with LM agents struggling to plan edits in the right lo- cations. We hypothesize that an innate limitation of the POMDP setup of LM agents is that after sufficiently many timesteps n, due to partial observability, the LM's understanding of the current state at such action becomes weaker, through divergence from the initial state before a1 In this section, we first explore this claim through synthetic setup and then attempt conditioning over state updates to improve the failure mode. We later generalize environments and discuss the implications of state-awareness for agent interaction.
approach to entire digital
Computation of state almost linearly decreases with respect number of actions taken. Sim- ilar to previous work in testing entity tracking in language models (Kim Schuster, 2023), we test the existence of the earlier divergence hypothesis in synthetic setup through prompting LM with 15 categories of preferences, emulating a web agent. We iteratively give an increasing list of updates the preferences (i.e. Dogs to Dislikes) and prompt the LM to output the updated list of preferences. Based on 125 randomly initialized runs. we find that failures in cumulative state construction linearly scale with the amount of actions taken (Figure 4). We show the exact setup for reproducibility in Appendix E.
Accuracy Step
Performance Comparison Base State Aware
130
Inatr rntructions
Taska Solved Bencttrrenchh
80-
Descriptive Instructions
Indt
AccurccyIN
50
60
30
10 Step
Pro not Adjuttd SiE-ageeit
aware SWE agent
Agents
Figure 4: GPT-4 Turbo accuracy in end state construction drops consistently over in- creased iterations of actions. Prompts and examples are available in Appendix E.
Figure 5: Comparison of prompt adjusted SWE- agent and a state-aware SWE-agent implementa- tion, both using gpt-4. Sample implementation code available in Appendix G.
Using the o200k base tokenizer task instances within the ansibl aeeeaaaaiblll repository

---

Published as a conference paper at ICLR 2025
General Sware-Aware Language Agent
Agent Trajectory Timestep
Trunal Starz
External Changes
Function Call
Action
Foodback
State
Function Call
State Update Policy
Furcttion Call
Action N
Foodbbakk
Foedback N+1
State N
Figure 6: Example flow of language agent some timestep N interacting with state update policy to generate the natural language state summary appended to the agent trajectory. The new updated trajectory passed to the core LM generate the function call and execute it.
6.1 STATE-AWARE INTERFACES
In our baseline runs, we find that agent trajectories extend long (60+), necessitating actions across multiple files. However, real world implementations of LM agents often restrict the amount of previ- ous information Count(w) in TN to a controlled number of steps* to avoid flooding context windows. Being able to model long term changes with limited context has been problem space in neural policy learning (Piterbarg et al., 2023). To tackle this, a recent SOTA approach in NetHack, a long horizon video game requiring continual learning (Hambro al., 2022), used unix di ff on previous observations in order to keep base models on track (Piterbarg et al., 2024). Their results confirm the importance of continuous and efficient modeling of state changes environments, but also demon- strate that diff history exploits structure that is present apriori in observations. Other works using LMS plan for embodied systems have found computation of state observations alongside baseline observations important for long-horizon task planning (Chen et al., 2024b). We combine the idea of efficient modeling of state observations with previous proven results with feedback- design (Yang et al., 2024; Shang et al., 2024), to motivate our approach: state-aware interfaces.
interface
Our implementation for a state-aware interface for interacting with code focuses on succinetly rep- resenting the divergence from initialization state, which is represented through previous edit tions. As such, before every function call, we have a cached and updated section with informa- tion related to all previous edits, prompting the model with understanding of the accumula- tion of its own changes (Appendix G). Formally, we add a recurring externally computed state variable
POMDP, where NN is the state timestep N, and trajectory now follows TN (a1, W1, UI,..,, aN,WN, WN,NN)
Agents with state updates have stronger performance in RefactorBench tasks. We modify SWF-agent with gpt-4o to track and display representations of state (Appendix G). This change boosts the agent's overall performance on RefactorBench: an average of 43,9% relative increase over the instruction sets compared to baseline agents (Figure 5). We also find strong upwards trend in subtask completion: an average of 71% increase over the instruction sets. our abstract syntax tree testing isolates unique subtasks in different files and functions, we find that passing more subtasks is correlated with stronger stateful reasoning, the intended goal state-aware interface.
FFive step observational window in the case of SWF-agent.
8

---

Published as a conference paper at ICLR 2025
6.2 STATE UPDATE POLICIES
Having precomputation from the state update allows the LM to ignore computing the reconstruction subtask when generating the next function call. We extend state-updates generalize entire digital environments, not just an agent, through the construction of a state update policy.
We define our state update policy stane as function that manages and provides the cumulative state information to the agent within environment. Formally, let n represent the current state at timestep N derived from all prior interactions. The state update policy Tstate can be expressed as conditional function:
sstate (TN, NN 00) + ON,
where Ty = (01, .I....,. 1N-1, 11 0--1; ON, W is the trajectory up to the generation of NN including all actions a,, observations wi, and prior states og.
State update policies can lead to deviations from typical sequential agent reasoning. We find states expressed in natural language be natural approach to facilitate concurrent interactions between language agents in open digital environments. Through our initial implementation of a state update policy, we are also able to model simple external changes from a concurrent state-awand aware user (Figure 6). In simple example, we concurrently make a change with an external agent to rename function that the LM agent has already edited to complete it's refactoring task. Through the state update policy, we are able to propagate this edit information and agent is able to decide to later view the new edit for more context (Appendix H). However, as all the tasks in RefactorBench are mutually exclusive, we do not further evaluate on modeling conflicting objectives between agents at a larger scale, but expand on similar directions for future work in Section 7.1.
7 DISCUSSION
We introduce RefactorBench, a benchmark that isolates unique failure modes of LM agents through code. Through our experiments, we find that most agents struggle at composing simple actions, and a diverse set of task evaluations is necessary for understanding and designing generalist language agent systems. We also show improvement on baselines through natural language representations of state and hope that further studies within stateful reasoning in differing scenarios can aid in the larger understanding of the limitations of language agents.
7.1 FUTURE DIRECTIONS
Although there are many avenues to take for improving LM agents, we generalize our analysis from evaluations RefactorBench tasks into two main categories.
Reasoning Through the synthetic state construction experiment, we formalize that language models innately lose state understanding with respect actions taken. As such, alongside introduction of state update policies, we hypothesize that constructing smarter ways to generalize context rather than having the LM condition problem. Various recent works on gist-based memory systems within agents, collaboration through optimizable graphs, exploration methods, skill learning, and mitigating partial observability seem promising (Zhuge et al., 2024; Nayak et al., 2024; Lee et al., 2024b; Wang et al., 2023a; Bruce et al., 2023; Xie et al., 2024a; Allen et al., 2024), but works have tackled concrete methods scale concurrent state-awarenes, for simple agent tasks. Many new approaches to improve agent perfor- mance have also been shown to scale up inference compute and score higher on various agent related benchmarks (Zhang et al., 2024a; Kapoor et al., 2024; Brown et al., 2024; Wang et al., 2024b). As real world refactoring results are not immediately verifiable, we find this style of repeated sampling to be insufficient without robust critic models. We encourage future works to scale inference time compute language agents with open-ended tasks like those in RefactorBench.
the full trajectory is an important direction for tackling this
Interaction In regards to interaction with the real world, find that LM agents edit code in inefficient manners and have low success rates per single edit. Many agents have switched to diff- based editing (Örwall, 2024; Gauthier, 2024), which has empirically shown to be more scalable solution. However, these systems do not get around the issues that come with temporary error states (Section 5) and format restrictions. The natural approach of full-file edits has its distinct

---

Published as a conference paper at ICLR 2025
issues: such generalizing for files longer than token limits, inference speed, token cost, and context flooding. Future approaches could attempt to intertwine full-file rewrites with speculative decoding (Cursor, 2024) and custom trajectory truncation schemes to limit context window flooding. Overall, even outside of code generation, we predict this interaction problem for language agents to be of importance in varying digital domains, and expect interaction to be large focus in generalist agent interface constructions, especially in multi-agent scenarios. Our state update policy demonstrates a primitive case of agents being aware of other actions, and we hope for future works to generalize the mnvironment-specinnn policy approach (Figure 6) in variety of digital tasks.
7.2 LIMITATIONS
RefactorBench's 's task instances are all in Python, and we hope to expand the benchmark to various languages that statically and dynamically typed, allowing for evaluations on more styles of refac- tors. We also focus on highly used open-source Python repositories, and language models may have better understanding of the repositories due their prevalence in training data. RefactorBench also has limited amount of task instances due to the intensive process to create singular end-to- end task and the necessity for quick evaluations (evaluations still takes hours with Bench). In our evaluations, we also raise the cost limit much past limits in previous works in software en- gineering agents, due to the inability for agents to solve multi-file tasks quickly and cheaply. We also find, similar to previous works, that agent runs are not deterministic and can solve differing tasks in different runs. RefactorBench is step forward in evaluating LM agents robust manners through complex task construction, but like all benchmarks, is still plagued by the possible issues of over-fit data distributions (i.e. only refactors) (Kapoor et al., 2024). To prevent this repetitive issue, we do not release gold reference solutions (only the testing files) and we recommend eval- uating software engineering agents on multiple styles of tasks: function editing tasks, bug fixes in SWE-bench, refactors in RefactorBench, etc. to truly define robustness in general coding agent. Creating a general multi-faceted evaluation suite for language models and agents interacting with code a compelling direction for future work.
7.3 UPDATED BASELINES
We evaluate the updated base SWE-agent 1.0 on RefactorBench using got-4o. We largely find that our previous results hold, with SWE-agent solving 21% of base instances and 31% of descrip- tive instances. However, we find that the speed and cost in which SWE-agent completes tasks significantly decreases to $1.69 per successful instance, showcasing new improvements in agents localizing changes to make. We are excited see the community test their autonomous systems on RefactorBench, especially with reasoning models. Similar to recent results on SWE-bench, we have seen preliminary results of reasoning agents solving more RefactorBench tasks compared to non-reasoning model agents.
ACKNOWLEDGMENTS
We thank the entire Data& Group at Microsoft for feedback and valuable discussions throughout the work. Dhruv Gautam thanks Michele Tufano, Alexander Pan, and Joe Guan for feedback of an earlier draft of the paper and for many formative research discussions about language models, reasoning, and code generation.
REFERENCES
Anisha Agarwal, Aaron Chan, Shubham Chandel, Jinu Jang, Shaun Miller, Roshanak Zilouchian Moghaddam, Yevhen Mohylevskyy, Neel Sundaresan, and Michele Tufano. Copilot evaluation harness: Evaluating llm-guided software programming, 2024. URL https://arxxi.orrggg
abs/2402. 14261.
Aide.dev. Sota on swe bench lite, 2024. sota-on- swe- www-weece--eeeeeeeee
URL https://aide.devv///
Cameron Allen, Aaron Kirtland, Ruo Yu Tao, Sam Lobel, Daniel Scott, Nicholas Petrocelli, Omer Gottesman, Ronald Parr, Michael L. Littman, and George Konidaris. Mitigating par-
10

---

Published as a conference paper at ICLR 2025
tial observability in sequential decision processes via the lambda discrepancy, 2024. URL attps:///rxiis org/ /ttp////././.0333323
Daman Arora, Athary Sonwane, Nalin Wadhwa, Abhav Mehrotra, Saiteja Utpala, Ramakrishna Bairi, Aditya Kanade, and Nagarajan Natarajan. Masai: Modular architecture for software-
engineering ai agents, 2024. URL https:
114466.1660 138..
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: //rxiaaivvv//.////// www.riii.oog/oaaeoeeoee wwwpoogg//b///eee 2108 07732.
AWS Q Developer. developer/.
Aws developer center, 2024. URL https: //atps://wws... amazon com/cocc////oooooo ///
Ramakrishna Bairi, Athary Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. Codeplan: Repository-level coding using Ilms abs 2309. 12499.
and planning, 2023. URL https: / http///xxvvvv////
Samuel R. Bowman and George E. Dahl. What will it take to fix benchmarking in natural language understanding?. 2021. URL hwwptttt///// peattt::://axiiee wawpiaavvvvvi............./.. abs /2104. 02145.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and
Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL https: / /aaxivivv.. org/ / /2ss/ 21787.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: / arxii. o abs 2005.000000 14165.
Jake Bruce, Ankit Anand, Bogdan Mazoure, and Rob Fergus. Learning about progress from experts. In The Eleventh International Conference on Learning Representations, 2023. URL https:
// //ooperrevieee ew. www//feruumeeeeee forummiiooooo wtt.://oroom..-aseefffff..o...e.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. URL https: arxiv. org/ /tab//2/333 12712.
Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones Anton Lozhkov, Car- olyn Jane Anderson, and Arjun Guha. Can it edit? evaluating the ability of large language models to follow code editing instructions. In Conference on Language Modeling (COLM), 2024.
Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, and Qianxiang Wang. Coder: Issue resolving with multi-agent and task graphs, 2024a. URL https: / arxiv. www.iiiwgg//.eeee.. abs/2406 01304.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Mathhis Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021. URL https pactps:/AAe h/tbpiss////vvvi org/ /bbs/ /21 21.7..700 03374.
11

---

Published as a conference paper at ICLR 2025
Siwei Chen, Anxing Xiao, and David Hsu. Llm-state: Open world state representation for long- horizon task planning with large language model, 2024b. URL httpps://axii......//// abs/ 2311.17400
Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang. Kevin Liu, and Aleksander Madry. Introducing swe-bench verified. https::/openat com/bloocooooooooooooocccooooooo f Fied, August 2024.
ietroducing--sw--eeeddte
Cognition.ai. Introducing devin, 2024.
URL https::/ww..cgnniiioonaaaaaaaa
introducing-deviin
Cursor. Editing files at 1000 tokens per second. https://ww..crsor.coom instant-apply 2024.
2024.
Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworld: A learning environment for text-hased games, 2019. URL https:
//wtxxv.org//aaaaeeaeee 1106..1101111111 11532.
Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery, 2021. URL hwtpt:t/t//tiiiiie 2107.07002
abs/
Martin Fowler, Kent Beck, John Brant, William Opdyke, and Don Roberts. Refactoring: Improving the Design of Existing Code. Addison- Wesley Professional, 2nd edition, 2018
Paul Gauthier. Aider: Ai-powered coding assistant, 2024. URL https::/p/// /tte:///gtthtaa com/
paul hatl-gauthier/aidee
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024.
URL htttps://aaaatae org/ abs /2312. 00752.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured abt// 00396.
state spaces, 2022. URL https: ottps//w....///////1......
Eric Hambro, Sharada Mohanty, Dmitrii Babaev, Minwoo Byeon, Dipam Chakraborty, Edward Grefenstette, Minqi Jiang, Daejin Jo, Anssi Kanervisto, Jongmin Kim, Sungwoong Kim, Robert Kirk, Vitaly Kurin, Heinrich Küttler, Tachwon Kwon, Donghoon Lee, Vegard Mella, Nan- tas Nardelli, Ivan Nazarov, Nikita Ovsov, Jack Parker-Holder, Roberta Raileanu, Karolis Ra- manauskas, Tim Rocktäschel, Danielle Rothermel, Mikayel Samvelyan, Dmitry Sorokin, Maciej Sypetkowski, and Michal Sypetkowski. Insights from the neurips 2021 nethack challenge, 2022. URL https:tpp... / /awxxviivv////////// org/ abs /tw.b////2..... 11889.
Joey Hejna, Pieter Abbeel, and Lerrel Pinto. Improving horizon imitation through instruction prediction, 2023. URL https: htt////www...oo///////33 12555.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents machine learning experimentation, 2024. URL https://azxtii /ogg /bs/ 03302.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation, 2024. URL atttps://arxiie org/ aww.abaaaa. hat.a//4..00.11.1.. 00515.
Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https: /openreview. net/ net/forum? forum?
d=VTF 8yNQM66.
12

---

Published as a conference paper at ICLR 2025
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99-144 1998.
Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter, 2024. URL https: / /ttp//aaivvvv//..////////././.av www.iii.ooo org/ abs 2407. 01502.
Najoung Kim and Sebastian Schuster. Entity tracking in language models, 2023. URL https: //arxiv. wtw.iiiorrrgeoo wtwp://///aaaaaaaaooaa///oass ata.//2355.. 02363.
Beck LaBash, August Rosedale, Alex Reents, Lucas Negritto, and Colin Wiel. Res-q: Evaluat- ing code-editing large language model systems at the repository scale, 2024. URL https:
//arxiv. org/ /ttp/240..660000000 16801.
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Amold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, and Kelvin Guu. Can long-context language models subsume retrieval, rag, sql, and more?, 2024a. URL
Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and Ian Fischer. A human-inspired reading agent with gist memory of very long contexts, 2024b. URL pettps://rxiv.. abs/2402. 09727.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Language modeling with latent situations, 2022.
URL httphttt///// //axivaaavvv/v//a///// org/ abs 2212. 10012.
Bowen Li, Wenhan Wu, Ziwei Tang. Lin Shi, John Yang. Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, and Kai Chen. Devbench: A comprehensive benchmark for software development, 2024. URL https:
//axxiv..rggabs/225000.0 htt.a/////440... 08604.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context, 2023a. URL https: // /awxivvvvvv..///v//.aa//aaaaavva ooroorrgssss.
01889.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2024a. URL www.thtss////aatte wrtp
03091.
Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Wenmeng Zhou, Fei Wang, and Michael Shieh. Codexgraph: Bridging large language models and code repositories via code graph databases, 2024b. URL https: /ttp://aaavvvve.aa org/ /a////bs////0 2400 03910.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms agents, 2023b. URL htttps://rxxvv.org//ee/e 2308. 03688.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024. URL https: //aaiiiv../..i//a//aa
arriiv.iiivv org/ abs /400 00203.
Pol Moreno, Edward Hughes, Kevin R. McKee, Bernardo Avila Pires, and Théophane Weber. Neural recursive belief states in multi-agent reinforcement learning, 2021. URL https: arxiv.
org/abs/ 2102.01222 02274.
Siddharth Nayak, Adelmo Morrison Orozco, Marina Ten Have, Vittal Thirumalai, Jackson Zhang, Darren Chen, Aditya Kapoor, Eric Robinson, Karthik Gopalakrishnan, James Harrison, Brian Ichter, Anuj Mahajan, and Hamsa Balakrishnan. Long-horizon planning for multi-agent robots in partially observable environments, 2024. URL
20000 10031.
OpenAl. Gpt-4o system card. https: 2024.
Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. Feedback loops drive in- context reward hacking in llms. arXiv, 2024.
13

---

Published as a conference paper at ICLR 2025
Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot, 2023. httttt:////xiiit www.w.wwww..w...ooooooooooooooot abs/ abb//302. 06590.
Ulyana Piterbarg, Lemrll Pinto, and Rob Fergus. Nethack is hard hack, 2023. URL https:
//aaxiv. wwwpaiawwgg/aaa///aoaaaaaaaaaaaa aalaaaaa/// 2200 19240.
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. diff history for neural language agents, 2024. URL https://arat orgg /ttg/as//////eeaoooe//oo//ssaa.r 221222122222 07540.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Bccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Had- sell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022. URL
https://arxiy: org/ hts//ab///at 012205066722
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. Identifying the risks of Im agents with an Im-emulated sandbox. In The Twelfth International Conference on Learning Representations, 2024.
Gabriel Sarch, Yue Wu, Michael J. Tarr, and Katerina Fragkiadaki. Open-ended instructable em- bodied agents with
large language models, 2023. URL https:://aaat::t
org/abas 231.01111222
Chuyi Shang, Amos You, Sanjay Subramanian, Trevor Darrell, and Roei Herzig. Traveler: A multi- Imm agent framework for video grestion-answeree 2024. URL https://arxiv.org
abs/2404 01476.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36, 2023.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL
abss//////////aoo////////////ae/ https: / /ttp:///aavvvv/ii org/ abs 2200 03768.
Theodore R. Sumers, Shunyu Yao, Kartik Narasimhan, and Thomas L. Griffiths. Cognitive archi- tectures for language agents, 2024. URL https:///axxii org/ hts//bbs/23999 02427.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https: / //openreview.rr www.:/nte//oomdmm//iooo wemmiiii@VeeWggeeee
Yuan Tian and Tianyi Zhang, Selective prompt anchoring for code generation, 2024. URL https:
09121.
Michele Tufano, Anisha Agarwal, Jinu Jang, Roshanak Zilouchian Moghaddam, and Neel Sundare- san. Autodev: Automated ai-driven development, 2024. URL httts://aaxxiiie wwwpeiiiivoorvaaaaeeoe /bs/ 2403.08299
Boyuan Wang, Yun Qu, Yuhang Jiang, Jianzhun Shao, Chang Liu, Wenming Yang, and Xiangyang Ji. Lim-empowered state representation for reinforcement learning, 2024a. URL https:// arxiv. orw. abs absp////////// /2077 13237.
Evan Wang. Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Scan Hendryx, Summer Yue, and Hugh Zhang. Plamning in natural language improves 11m search for code generation, 2024b. URL https: /twpxv.vvvv.//////e//o/.a org abss/ / 2209. 03733.
Guanzhi Wang, Yugi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023a. URL
6291.
14

---

Published as a conference paper at ICLR 2025
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Ji- akai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science 18(6), March 2024c. ISSN 2095-2236. doi: 10070/7/ 1704-024-40231-1 URL http: //xx dot. .org/ 10.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Opendevin: An open platform for ai software developers generalist agents, 2024d. URL https:://arxt
org/ alos 220772411211 16741.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang. Xiaojian Ma, and Yitao Liang. Jarvis-l: Open- world multi-task agents with memorý-augmenteed multimodal language models, 2023b. URL
https: /awxiv..vvvvvvvv////v//..///.avv org/ http/abs/200 a211 05997.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang, Agentless: Demystifying Ilm- based software engineering agents, 2024. URL https: /aarii/aaaiivv.iiia org abs / 2407 000.00.011488 01489.
Annie Xie, Logan Mondal Bhamidipaty, Evan Zheran Liu, Joey Hong, Sergey Levine, and Chelsea Finn. Learning to explore in POMDPs with informational rewards. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 54414-54429, PMLR, 21-27 Jul 2024a. URL https: pttppeeieceg...cecoe mww.m.rreeeeee press v255/22355///2 xi 2e..ee24a.htn. html.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents, 2024b. URL
https: org/ alss 2402 01622.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao. Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024c. URL https: /arrxvv.ogg/
abs /204. 07972.
Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, and Dimitris Papailiopoulos. From artificial needles to real haystacks: Improving retrieval capabilities 1lms by finetuning on synthetic data, 2024. URL www.tttt//.cm //tp////avvv////////////eaaaaaaa
org/ abs / 2406.11.1 19292.
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024.
Shunyu Yao and Karthik Narasimhan. Language agents in the digital world: Opportunities and risks. princeton-nlp..ifhubbii Jul 2023. URL https//////innnoonnnnn/.ee language - agent- impact/.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 7-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL 1406.12045.
Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, Bo Pang, Yingbo Zhou, Shelby Heinecke, Silvio Savarese, Huan Wang, and Caiming Xiong. Diversity empowers intelligence: Integrating expertise of software engineering agents, 2024a. URL https: : / arxii http://wwwwwww..//////o wwtp://////aaaaaaa/eooae 07060.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury, Autocoderoover Autonomous program improvement, 2024b. URL https : /tt///iiivv..////// wtwp:///wwwo..bb///...... 05427.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Language agents as optimizable graphs, 2024. URL https: /ttp///xxvvv////////////..////// abs/2402. 182.3.
15

---

Published as a conference paper at ICLR 2025
Albert Örwall. Moatless tools. https://tggtteeeeeee
http:///gghhaaa...mmmaaaa////tta 2024.
16

---

Published as a conference paper at ICLR 2025
A TASK CONSTRUCTION PROMPTS
A.1 PROMPT FOR LAZY INSTRUCTION
We prompt gpt-4-toe 4-turbo with the handerafted base instruction based all the edits and this prompt to get our lazy instruction.
Please convert the following instruction be less specific. Do not change the behavior of the task, but give a short, less descriptive version of the task in human-like prose. Your final instruction should
a partial sentence and should net instruct to run any tests. It should
just describe the changes to the repository. Do not output. ANYTHING BLSS
BUT THE NEW INSTRUCTION. Here is the original instruction:
(aas instruction|
Here examples of lazy instructicns:
(few_shot_lazy)
Remember to only output the NEW LAZY INSTRUCTION CORRESPONDING TO THE BASE TASK.
A.2 PROMPT FOR DESCRIPTIVE INSTRUCTION
We prompt gpt 100---ttun 4-turbo with the handcrafted base instruction based on all the edits, the corre- sponding lesting file, and this prompt get our descriptive instruction.
Please convert the following instruction to be more specific and have specific filenames for edits (not paths) Do not change the behavior of the task, but give longer, more descriptive version of the task in human-like specifications. Reason over the AST tests provided to give more information on which files could be relevant, implementation details or anything related to what generalizations the tests are looking for. Your final instruction should be around 2-3 full sentences and should not say to run any tests anything like that. It output
do not give exact
should just describe the changes the reposi torsttyyy... ANYTHING ELSE BUT THE NEW INSTRUCTION. Here the original instruction and its related test file:
(base_instruction)
Test File Starts Here:
(inst_test_file)
End of Test File.
Here are examples of descriptive instructions:
[few_shot_des
Remenber to only output the NEW DESCRIPTIVE INSTRUCTION CCRRESPONDING TO THE BASE TASK.
17

---

Published as a conference paper at ICLR 2025
B AGENT PROMPT CHANGES
B.I UPDATED AGENT SYSTEM PROMPT
As described in Section alter the SWE-agent prompt stop the agent from creating bug reproduction scripts for refactors and focus on the style of task at hand. We also remove pytest and other testing functionality in order prevent timeout issues with the large codebases.
autonomous programmer speciali aiiiiliiin in refactoring, and you 'e working directly in the command 1ine with special interface. The special interface consists of file editor that shows you WINDOW lines of a file at a time.
SETTING: You
In addition to typical pash commands, you can also use the following commands to help you navigate and edit files.
COMMANDS:
command's command\_docs _dcss
Please note that THE EDIT COMMAND REQUIRES PROPER INDENIATION.
If you'd 'd like to add the line print (x) you must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be
RESPONSH, FORMAT:
Your shell prompt is formatted as follows:
(Open file: <path>) <owd>>
You need to fornat your output using two fields: discussien and comnand. Your output should always include \one)__ discussion and \_one\_ command field EXACTLY as in the following example:
DISCUSSION
First I'11 start by using is to see what files are in the current directory. Then maybe we can look at some relevant files to see what they look like.
(begin(verbatim)
is -a verd(verbatim)
You should only include *SINGLE* command in the command section and then wait for response from the shell before continuing with discussion and commands. Everything you include in the DISCUSSION section will be saved for future reference.
If you'd like to issuc two commands at once, PLEASE DO NOT DC TEAT!
Please instead first submit just the first command, and then after receiving response you'll be able to issue the second conmand.
You re free to use any other bash commands you want (e.g. find, grep, cat 1s, cd) in addition to the special commands listed above.
However, the .g. python, vim), , please do not invoke them.
does NOT support interactive ie session commands (e
instance templaaeet 1-
We 'r currently solving the follewing issue within our repository. Here's the issue text:
ISSUE:
INSTRUCTIONS:
Now, you're going to solve this refactoring issue your Your terminal session has started and yeu're in the repository's root directory. You can use any bash commands or special interface to help you. Edit all the files you need to and run any checks or tests that you want.
Remember, YOU CAN ONLY ENTER ONE COMMAND AT A TIME. You should always wait for feedback after every command.
18

---

Published as a conference paper at ICLR 2025
When you'ze satisfied with all of the changes you've made, you submit your changes the code base by simply running the subrit command. Note however that you cannot use any interactive session commands (e.g. python, vim) in this environment, but you can write scripts and run them. E.g. you can write a python script and then run it with "python <script _name>.py".
NOTE ABOUT THE EDIT COMMAND: Indentation really matters! When editing a file, make sure to insert appropriate indentation betore each line!
IMPORTANT TIPS:
1. Always start by chesking your working di directory, cd od'ing to the task repo, and then trying to find where to do the re refactor using the search tools. Do not into other directories like root or sys. Just to the task repo and make edits in there.
2. If you run a command and it deesn't work, try running a different command. A command that did not work once will not work the second tine unless you modify it!
3. you open file and need te get to an area around specific line that is not in the first 100 lines, say 1ine 583, don't just use the scroli\_aowe command muitiple times. Instead, use the gote 583 command. It's much quicker.
4. Always make sure to look at the currently open file and the current working directory (which appears right after the currently open file). The currently open file might be in different directory than the working directory! Note that so they might change the current open file.
commands, such 'create', open files,
5. when editing files, it is casy to accidentally specify a wrong 1ine write code with incerrect indentation. Always check the code after you issue an edit make sure that it reflects what you wanted accompl ish. If it didn't, issue ancther command to fix it.
number
6. may be necessary to install the repository from source before you can run code. Please think about how install the environment from the repository directory if you need do so.
7. Do not the repo tests, just focus comprehensively finishing the multi-file refactor to the best of your abilities.
8. If you make a change that could have dependencies, always reason and search where imports need to be updated and update them accordingly.
9. run huge grep commands that could print and flood the context with too many items. not sure, just check how many items the commana will run to begin with.
(Open file: ofell_file) \ (Current directory: workingl_dir) W
19

---

Published as a conference paper at ICLR 2025
C EXAMPLE CONDENSED SUCCESSFUL TRAJECTORY
We show portion of a successful trajectory for the refactoring task: remmmeesen-froomdddeetttor in the flask repository. Full examples are available in our repository.
Base Instruction: Rename the helper function send from directory to send from directory .ellper but keep the overall flask usage as fond_ffommdireeee
Stare SwE-agent Run
"action": "Is -F\n",
"observation": "flask http://eeeeeeeeeaaeeeeeeeererree nust/\nvar/*,
\nlibb wwwproott///uu////ae /nssinn///niinneee wtw.liinnnns////yyyeeaeeee htgp:///nnnnn//aaa/aaaaa//////// hnsrv//nsss//aang//////
"response"; "DISCUSSIONhoFiret I'll list the files in the current
directory to get everview of the project structure. This will help
locate the 'helpers' module and the 'flask' folder where the changes need to be made, \nl* -Fin"""" "state": "(\"orkiiggdiip" A", ""recent_editsl"" 111", "thought": #DISCUSSION.aFirst I'll 1ist the files in the current directory get everview the project structure. This will help me locate the 'helpers' module and the 'flask' folder where the changes need to be made.An!
\"Elass_retactor\"" \"open File\": \"n/a
(Makes changes multiple files)
"role"; "assistant",
"content": "DISCUSSIONNnie need update the import statement for send_from_dd rectory" to import fenddfrom_p rectory_helpper as rndd_frro_director
let's make the necessary edit. \n" Aniit Anedit 9:91 send from directory'nend "thought"; "DISCUSSION need to update the import statement for
nfrom blasskhhelpers import edit'n"""",
heeleer as send_ffomm_irrettorr to import rect_fr send_from__ireeton Let's make the necessary edit. "action"; *edit 9:9Anfrom flask.helpers import send_from www.ccrrreeeeeeeeeeeee helper send from faceaffro___eeeeeee dwwectorylnedd of edit
as
"role": "user",
"content"; "[Truncated File Output]
(Current State: l'Edited helpers.py lines 514:514', 'Edited
Linit_ py at lines 24:24', 'Rdited app-py at 1ines 42:42', 'Rdited blueprints.pp at lines 9.9', 'kdited test_helpers.py at lines 9:9\An ']) (Open file: /flask refactor/teststtst..tee pttp.essstt.helppr...p. (Current directory: flask refactor) bash-$",
"role": "assistant", "content"; "DISCUSSIONATTe import statement in "test ttosthhelpors.p pp"" has been updated. We have now made all the necessary changes to the relevant files. Let's submit the changes to the code base.\n""") nsubmit\n">"",
"thought"; "DISCUSSIONN1TH import statement "test_helpers.pp has been updated. We have now made all the necessary changes to the relevant files. Let's submit the changes to the code base. \,", "action": "submit\n",
20

---

Published as a conference paper at ICLR 2025
D EXAMPLE TEST FILE
Here is the parameterizz--uunzippp test file for the parameterrize-unzzip task:
Base instruction: Encapsulate the parameters in gunzip into a new class called GunzipParams and update all references within the repository to create GunzipParams object.
import unittest import import ast
class TestScrapyMigration (unittest.. (unittest.TestCase)
def test test_ gunzipparams class (rists(self): Path to the file where the class should defined file_path ".../scray/uutil/gg..pp
# Check if the file exists assf. (os wtt..ppph..eaee. (ww. exist")
(tt.e...ettttttilleeettttt f'(file_path) does not
t Check if the GunzipParams class is defined in gz.py with (pe 'r') file:
ast. pft.
False
for node in ast walk (tree):
if isinstance (node, ast. ClassDef) and node 'GunzipParams
class_found = Truc break
self asserraserr (class found, "Class 'GunzipParams' not found in gz. py")
def gest_gguzzppprrmm.... saateaamsshaaseeeee aatpppprrraa_aa___tttaaaadadade_ max Path to the file where the class should detined file_path "..//srapy/utiissgg....
Check if the file exists self asserrsserrrrue (os. wet.ppath..iiseeee exists (file_path), f"(file_path) does not exist")
$ Check if the GunzipParams class has self.date and self.max_si.e max_siie attributes
with (file_path,, 'r') file:
tree ast. (file.
class_node = None for node in ast walk (tree):
if isinstance (node, CltssDe)))) and node name 'GunzipParams
class_nods node break
self asf..asssttttotttoooo (class_ncde, "Class *GunzipParams' not found g2 gz.py")
= False
max max_size_found False node
(elasssee__
if isinstance (node, ast Assign))
for target in node. targets:
if isinstance (target, ast. Attribute) and target.attr -- data':
21

---

Published as a conference paper at ICLR 2025
data found True
isinstance (target, ast.Attribute) tärget. max_size';
_ax_size_found = True
aelf.aasert (data_ "Attribute 'self.data" not found in GunzipParams class")
are (max see___ize__oooo found, "Attribute maatmmax_size not found in GunzipParams class")
def test_gunzipp_unntt iocation_iiin onssignature (self): Path to the file where the function should be defined file_path ".../crray/util//gg..yy
Check if the file exists asserrsser (os. path. (iill ["(file_path) does not cxist")
Check if the gunzip functien has the correct signature with open(file_path,, 'r') file:
tree ast. reale
Sunction_foure = False
for node in walk (tree):
it isinstance (node, Functincno)))) and nare 'gunzip': A Check function parameters args - node.args
arrgs.argggln. if len (args =- and isinstance (args. (args(]] [0] annotation
Name) and args. annottttnnnttoooonnn id == 'GunzipParams Check return type
if isinstance (node. ast ast.Name) and node. returns.id
== 'bytes':
foattion_foud True break
self.assert ( efuncciio_fuunnnt "Function 'gunzip' with signature def gunzip GunzipParams, -> bytes' not found in g2.py")
def test_ gunzipg in istem
$ Path to the file where SitemapSpider should be defined
file_path = ht./ssrray/ppiddess//ieemsss
Check the file exists self. areertteet (os. (os.path.................... aath.cxists exist")
('(flle_path) does not
Check if the SitemapSpider class has method _get_sitemap_bor that uses gunzip with GunzipParans with (file__eepaata 'r') as file:
tree = ast.parse read))
tetmapppider__lass
for node in ast. walk ask. (tree):
if isinstance (node, ast. ClassDe))) and node. name = sitemapSpider':
sitemapspider_clan class node break
wwl.raasstttsootttooo (sitamapspidrr_clas,, "Class 'SitemapSpider' not found in sitemap.py")
method_found False
gunzip_pan params used = False for node in ast walk weitemapspider.classs.
22

---

Published as a conference paper at ICLR 2025
if isinstance (node, ast.FunctionDef) and mode quet_sitemap_boyy
method_found True
for inner_node in ast walk (node)
if isinstance (inner_node, ast.Call) and isinstance( inner_ Lomer_node.fung ast. Name) and inner_node func. gunzip':
if 1en (inner_ncdeeeargg
== 1:
= haner_ncde- argss
# Check if the argument passed gunzip is instance GunzipParams
if isinstance (arg, Name) or (isinstance (arg, ast .Atrribuee and arg.attr =- "GunzipParams')
_sed = True
break
self assfrassett (method_ found, "Method in sitcmapSpider class")
not found
assf. (gunzip_params _sed, "gunzip function inside net_sitemap_bodyy does not 'GunzipParams! object as parameter")
def test_ inst_immpots_iin in_ intemmmmapppiteeeeeeeeeeeeee (seli):
Path to the file where the imports should be defined file pathhpahhh
Check if the file exists assertsser (os. path. exists (file f'(file_path) does not exist")
# Check if the correct import statement is present with ofen 'r') as file:
tree ast paree (filc.rcad())
imports_fonnt
"GunzipParams" False, "gunzip": ralse, "grip_magiic_numbrr"" False
for node in ask. (tree):
if isinstance (node, ast. ImportFrom) and module 'scrapy. utils.gz':
for alias in node names: if alias. name in imports_ found: inports_found [alias. namee) True
for import_name, found imperts_ srmundditems ()
self asserttrue (found, I"Import ' "impprt_namee" not found in sitemap.py")
def test rest_imports_ine jr Path to the test file where the imports should be defined test_file filefiie__pahhll
assorrrse (os, /t:///sss/testttillssssss docs not exist")
Check if the test file exists
('toos_ffle_path)
Check if the correct import statement is present with tree ast. (eade
path, 'r') file:
Imports_foune = "GunzipParams": False,
23

---

Published as a conference paper at ICLR 2025
"gunzip": False, "grip_magic_number" False
for node in walk (tree):
if isinstance (node, ast. ImportFrom) and node.modull utils.gz':
sscappy.
for alias in node mode.naned nanes: it alias. name in pnvorts_foudde imports_ foundl alias, name] True
for import_name, found in imports_found items ():
self.assertIn (foundd, f" f'Import [iiportt_amee) not found in test cest_utils_g2.221
def test rest__gunzipparan used in_test_uttttteeeeeeeeeeeeeeee_t (self)
$ Path to the test file where gunzip should be used with GunzipParams httpaatttttttttttttttttttttta "../tests/cestt_uill_....
test_file_path
Check the test file exists self.as aal.aassertTrus wwt.ppat........oo.oooooooooo (tesstfiltee_eatttttt f" ([test_file_ppa aoes not exist")
Check the gunzip functien used with GunzipParams the test file with spen(test_file_path 'r') as file:
tree ast parse rea)))
gunzip_params
parpppaaaaaaeeeeeaaaaaa_aa used False
for nede in ast. ast.wakk (tree): if isinstance (node, ast. and isinstance (node. func, ast.Namt and mode.fune.ee == 'gunzip';
if len (node.arge)
1:
arg [ode Check if the argument passed to gunzip is an instance of GunzipParams
isinstance (arg, ast.Name) (isinstance (arg, ast.
Attribute) and arg.attr 'GunzipParams'). gunzip params used True break
self assertTrue (ganiipppaaaaaeeeaaaaaa pasd "gunzip function test_utils_gg.ppt does not use a 'GunzipParams' object as parameter")
def test sirt__mmorts____ _i__iii__s____________________ss test www..dowcwaaeemmddeeeeee.... http:m/mddddeeeeeeeeeeerrrrrrrre httppooppressiinnn tegarss (self): Path to the test file where the imports should defined fest_ = .../tests/
test_ test
Check the test file exists (os. wwt.ppat...ii....... (test_ ftest_file_path) ("(testffile_pah) does not exist")
Check if the correct import statement is present with (tenstest_file_paat 'r') file:
tree ast. (filc.
imports_found =
"GunzipParams": False, "gunzip": False
for node in walk (tree)
24

---

Published as a conference paper at ICLR 2025
if isinstance (node, ast. ImportFrom) and node module 'scrapy. utils.gz';
for alias in node names:
if alias name in fouud:
imports_found (alias. namias.namm] = True
for import_nama, found imports_ ():
(found, I"Import [import_name] not found in
def
eware (self):
Path to the middleware file where gunzig should be used with GunzipParams
middleware file file_pathh =
nttpcompression.py"
Check if the middleware file exists
self assertIrue (os, path.cciste miiddloarre_fill_perr E"[ micoleearre_file_ptteee does net exist")
Check the gunzip function used with GunzipParams in the middleware file with open (middleware wttple/ware__ie__eettt__eee_eeee path, 'r') as file: tree parse (file.read())
gunzip_params__uee
False
for node in ast ast.walk (tree):
if isinstance (node, cst. and isinstance (node, func, ast.Nar Name and node. node. fu == 'gunzip':
if len (node..args) == ==
arg node.
Check if the argument passed to gunzip is an instance of GunzipParams
if isinstance (arg, ast Name)) or (isinstance (arg, ast.
Attribute) and arg.attr 'GunzipParams") Sunzip_params_used True break
self assert (gunzip params used, "gunzip function in attpcompression.pp does not use a 'GunzipParams' object as a parameter")
if name == mmmin miiteest.main//
25

---

Published as a conference paper at ICLR 2025
E EXAMPLE TEST OUTPUTS
Here are the results of running a subset of the custom AST unit tests. The outputs showcase the subtask testing formats and the specificity in unit test function names.
Patch Evaluation Results
Test file: tatp:///hwg..eefttto///aaacccccp
Test results: Passed
Test file: httt://ssl...aacccc///aarrrrtttr
Error: test t
ceass__e__aatcreat_isttttt used (ct.:-ccaateetee....ettaaadadoo.
[est_ss_cancrrat_imnm__eedd]]]j ... FAIL
ok
tet__e__caatcrreee.............a ... FAIL
test_ee__cnttrraat___sssssd
FAIL
(cant-create-test.
ok
PA
PerssutfcitCceesssssssss.......d does [xmporrt_xitcodes]
ok
test a cantcrreateee (cant-create-test.
FAIL
FAIL: teet_ex_ccaccrcat___etttuuu org.apaw...aoaaoo.o............. telt:-caaaeeeee...ooaaaaaaooooe. (ett_ex_cantcrrat__ian_.eeed
Traceback (most recent call last):
File //taa//w.o//.aaaaaaa////////aaaa 155, in prst__ex_cattrrat__iase self.assertFalse (ex cantcreat_founde "aalt.eetaulss.exxtooeeee but it should
line
Ex_CANTCREAT was found in salt/chient/sh/ not be used. ")
AssertionError: True is false salt. EX_NTTCCRATT found in salt/client/shh/ init_..y. py, but it should not be used.
FAIL: test test__e__cntttteeeeeeeeeee hegs_er__caccratte__.sssss...... org.::a.....aaaaaaaaaaaaaaattarr (cant-create-test. wtt.ppat..ttte.........c..oaaaaa
Traceback (most recent call last):
File httpa//aaaaaattaaaaaaaaaaattatte
line "" not found in
39, in teststss__catcrrtteeiis_sssssss. aelf.assertTrueeex_.oo file_pathi")
AssertionE:ror: False not true CENTCREATE not found in ../salt/ http:///sssssssh.yshhihie
FAIL: het.ttte__eaatcrerreeeeeeaeaaaee tean--reeate-cee...eeatatttttt.e tett_ex_cantcreeree_____eee
Traceback File http://w..p//aaaaaaaaaaaaaaaaaaa 131, in test_er_ccatccresss_ss_uss aelffassertIrue (ex_ conorratutuuccccttacaaarttttdtd "edlt.effallee.eeeeeee.e. line
Traceback (most recent call last):
CA_NCCCREEA was not found in htw.iieenttshhiiiii__ppppp.
AssertionError: False is not true salt waa...defatts..xxttooto www.eeeetcccc.cc.ccneeee........ EX_CANTCREATE not found in salt/client/ssh init _nntt____ittt
26

---

Published as a conference paper at ICLR 2025
FAIL:
test ssh py shim does have ex_cantrreat
(cant-create-test.
Traceback (mest recent call last): File
line
107, in test_ash.py_shhm_dee
F""EX_CANTCRENT (misspelled)
found in [sile_ppthl
AsserticnError: True is not false
'EX_ CANICREAA (misspelled) Found in
PAIl:
(cant-create-test.
Traceback (most call last):
File http:///wwaaaa.eeeeoaaaaaaaeeaee
line
in
self.asscrtIruc (ta.aaccctttccccoccattttrrteaaat f" EXEEACCTTREE not used in filc_path)")
Assertiionrroo:: False client/ssh/ssh_py_shhmi
EE__CTNTCE used in /saat///
Ran tests in 0.046s
FAILED (failures=5)
Test file: ttt.a////aaa....aaaeaaaaaaaaaaar Error: https//c.........aacc....eeeeaes reerraattoonttttttttteeeeoeooorr
ck
test httppp/pw...cc.aooooooooooooooaa
FAIL
ok
ok
fest_value
atils (value-is-a- (valu--ie--
_ttpaaddeeeeeeeeeeeeeeeeeee not _et__eesttttt_eeeeeeeeeeee exist_in_atiss))
ok
FAIL: Rcteeooooc.clliooalloaa.oaaaoaaa org.rpawo....aaalaaaaaaaa.oeerar
Traceback (most File http:///wweaaaaeee..o..o..oooaaa
last):
test.py", line 75, in test Lmwpptttaaaueee__...eeeeeeeeeeee
self.assertTrue (import_ (import_found, "fastapi__compe in dependenciessittils.pp""
sttpllue_i____eeuueeeeee net imported from
AssertionError: False not true 'value_is___sequencee not imported
from peptappi_ccmpatt in dependenciessuutil..py
Ran tests in 0.026s PAILED (failures=1)
Test file: tatt://mreyyeeeeeecooo/ooooooooe Test resulta: Passed
27

---

Published as a conference paper at ICLR 2025
F STATE RECONSTRUCTION EXPERIMENT
We prompt gpt-4- -utuoo with randomly initialized and randomly changed lists preferences. We generate preferences through this script and simply iteratively prompt with subportions of the json.
import jaen
import random
def amnerate_randmm_prefereeeee (categories, productss) return
(cattgggr
category:
product: v (I (l"Likes", "Dislikes", "NA"I) for product
in products |category]
for category categories
def praerrat__trjeectoorrititttt
num_actions, categories, products)
actions ( ("SetPreference")
trajectories = []
preferences (eat: dict (Lnttial for cat in Maitial_preta
Deep copy to prevent mutation
trajectory ("actions": [1, "states": ())
for in range (1, num_actions 1):
category random. choice (categories)
product landom.choice
action_type type random choice (actiens)
olé_preference preferences [product] A Track old
preference
new_preference checmec ["Likes", "Dislikes", "NA"])
while new new_preference oid_preference: Ensure the preference
different
new_preference = random.chcice ("Likes", "Dislikes", "NA"])
details
"action"; action_type
"category";
category,
"product"; product,
#"old_preferrene"; old_preference, nee_pprfference": _ew_preference
preferences (aarenccesccatepgee ccategory] [preduct] new_preference # Update to nex preference
trajectory ("actions") append (details)
Snapshot of system state after each action
trajectory [" [f") {cat: dict (preferences [cat]] for cat in preferences)
aprend return trajectories
def m)in():
categories ("Electronics", "Bocks", "Clothing", "Garden", "Games"]
products =
["Laptop", "Smartphone", "Headphones"), ["Novel", "Biography", "Science Ficticn"), "Clothing"; ["Jeans", "T-shirt", "Jacket"],
"Electronics":
"Bocks":
"Garden": ["Shovel", "Lawn Mewer", "Gloves"),
"Games": ["Board Game", "Video Game", "Puzzle")
num_initial_statee 50
trajectories pearrriies_ppr__tteeee__eeese
28

---

Published as a conference paper at ICLR 2025
actions petplpep_________ trajeoooyyy 50 Example, number of actions per trajectory
dat_ []
for in range (mn_iiittttttallsstteee)) mntiial_prefs gemarrate_rendom_prefeeeeess (categories, products)
trajectories = []
for in range (trajectories_per____e
trajectory generate_trajecttone (initiallitiieeeteeeeeee preis, act actions_per_traaeeteee trajectories eetriiissextendde extend (trajectory) Add the generated trajectory to the list
categories, products)
all all_dataa_ae dad.aaapapadd
"initial "initial preferences": Saatial_prefss
"trajectories"; trajectories
1)
with open (' coccoomlleeaaeccccc watppplee_xcttoos__tt_eeeeee wwwooonsssttttttteeeeeeeeeeeeeee ststtessssssn,, 'w') as f: json dump (all_data, f, indent=4)
if name main ()
ma
Given these randomly generated actions in json, we prompt spt-4-turbe for 125 random ini- tializations iteratively over 0-50 actions of the generated actions, Figure shows the prompt and expected output. We are strict with format rules, and allow minor mistakes, however, our parser requires the larger category separations.
Toy Agent Reconstruction Task
Bere are your initial greserences ca dicterent categorios.
Preferences:
('Electrondes': { "Laptop"; "Likes', "Smastphona": 'Likes', 'Esadphoncs"; "Dielikes" "Bookst: "Bovel's 'Bielikee", "Biography" H8' *Science Fiction' 'Diclikes' }, rCisthing's : { Jeann' "Lakes", "T-shirt"; *takes', "Jacket"; "likes" b. 'Garden'; { "Shovel"g ; "ikks'", "tawn Nower's "BA", "Gloves" *Na }, GGameet: 'Boord Game!
"Likes', "Vidoo Gancf: 'Likas', 'Fuzzie': "Likes' actione
after Action 11 Ellctrontes Saptop to "NA".
Action Clothing T-Shirt 'XA" chaneas.
is the your response EXACILY how fornatted the inpet initial preferencos state. Pretercncas:
the of profesences on all catagorias aftez
answez: ('Beeccooncaa { "Laptop"s "HAr, "Bnactphone"s "Likes', "Headphonas's "Diallkes'
"Books": [Hoovrrs "Dislikes", "Bioesaphy's rua,, "Selence Piction': "Dislikes' }, "Clothlagr' { (Jeanars
"Täkes', (n-Shirt's INA, "dacket': "Likes" }. 'Garden's { (Shovel's (Tikeef, "lown Moverfs (hes"
"Gloves's (NA }, "Gamexfs { flowrd Ganets kes', VVideo Game'ss fhixlikes', PPoxlee's Mfikes!
Figure 7: Example of singular instance of the synthetic state construction task.
29

---

Published as a conference paper at ICLR 2025
G SIMPLE SINGLE AGENT STATE-AWARE IMPLEMENTATION
As an self-contained example, we have a simple implementation a state-aware interface contained within singular agent instance. This comtmadd tracks all it's previous edit commands and concatenates them in separate section. In practice and for results in the paper, we augment the state cache to relay more information about related edits by integrating parts of previous observations as well.
state_command:
name: state
code:
state() local Barking_dir="S50
local open sre__iile_inf==$_3 CECCURRRTTT_LLLE-/SSAS
local recont_ccits__sson= []]
if -n "SRECENT_EDITS"
then Split 3RACENT_RDITS into an array of edits
IFS-" read edits_array <<< "$RRCENT_KDITS" declare -A seen_edits
Filter out duplicate silename-line humberr filtered edits=() for edit in "((edits_array(@))
nurber pairs
filename=$ (echo "$edit" cut -d';' -f1)
line_number=$ (echo "$edit" cut -d';' -f2)
Check if this filename:line [ -z "$(seen [[ssee__eiiss filtered_edits++ ("Sedit") edits ['en_eedtss"$(iieee "$filename namberr__mbrr" "J=1
pair has been before
1 then
fi done
Convert the filtered edits into JSON array recent_edin srcent_edite_jscneee (printf 'as'n' "S jq -R (sla))
fi
state_json=$ (iq -n --arg wd "sworking_dir" --arg of "$ (rcalpath folen_file_to info"l infe")" --argison "Srecent_edits__ee '"workinggdir"" $ud, "open_file"; $ef, "recent_edits": $re)')
echo "$state_jscn"
We also, communicate all the recent edits, append the outputs of this mttee_comman in every step template through:
next_step_templatee |- (observation)
(Current State:
(Open file: copen_file))
(Current directery: (
bash-$
30

---

Published as a conference paper at ICLR 2025
H EXAMPLE STATE-AWARE OBJECTIVE UPDATE
In this example excerpt from a trajectory, we are able to see our state-aware agent reason about an external edit (propagated through the state-aware policy) made to the local repository it was generating concurrent patch for.
"role": "assistant",
"thought";
"DISCUSSIONANW" have found the "get_nncdiiggffooobaaaee function calls at 1ines 622 and 623 in the "test_utils.p" file. Let's navigate line 622 and update the function calls pass "oog=False ".n""
"action": "goto 622 "agent": "primary"
"role": "user", "content";
[File Contents Printed, Truncated Brevity]
(External Edits: ['Since your previous action, another user edited adapters. py at lines 359:359 n'
(Your Recent Edits: l'Edited utils.py py at lines 542:542', 'Edited adapters.py at lines 378:378', !kdited adapters.py at lines 618:618/|n/)
(Open file: /requests_seeeeeeeeeeee htgp:ssterrrtcteeeeessssttttttst
(Current directory: requests_refactorr bash-$",
"agent"; "primary"
"role": "assistant",
"thought": "DISCUsSTONNnlat' edit the function call line 623 to pass Log=False". Then, let's view the external edit adapters.py.",
"action";
edit 623:623
assert get_encodingg... from headers (value, log=False) == expected end_of_edit edit
31

---

Published as a conference paper at ICLR 2025
1 CONTEXT FLOODING FIGURE
Context Flooding & ICRH
Error Handling
proposed has introduced syntax
Instruction (Initial Objective)
error (5) Flease read this error message retry
oo
carefully
file.
Benane pen, Cinger to pen, fingerpeint
without changing any function names.
BRRORS:- 2999 SyntaxError: unnatched ') This is how your edit would have looked applied
Failed Function Call Loop
[after_fia)
26:48
This is the original
your Your changes have NOT been applied.
(before file)
Tast, Sype
Detura 3um
Function Call Instruction (New Objective)
your comzand tzy again. You el ther need to 1) Specify the correct stact/und line argusents or 2) Corsect your edit code.
Core agent loses sight of initial objective
00
Do NCT ze-run the sane falled odit comrand. Bunning it will lead to the
Figure 8: Visual example of a language agent having failed function call loop showcasing the context flooding and deprioritized objective failure mode.
32


Reviews:
[{"review_id": "n0TWi6mjtt", "reviewer": "Reviewer_LgvP", "summary": "This paper introduces a novel benchmark, RefactorBench, designed to evaluate the limitations of LM agents in complex, multi-file code refactoring tasks within open-source repositories. RefactorBench consists of 100 handcrafted tasks, each defined by 3 natural language instructions that vary in specificity, requiring agents to explore dependencies across multiple files to complete tasks successfully. Initial experiments demonstrate that current LM agents struggle  with these compositional tasks, solving only 18% compared to human developers, who achieve 87% under short time constraints. Through trajectory analysis, the authors identify distinct failure modes, particularly the agents' difficulty in tracking past actions. To improve this, the authors propose to adapt baseline agents to condition on state representations which improves performance by 40.4%. The study positions RefactorBench as a critical tool for advancing the  development of LM agents in code-related environments.", "questions": "See the above weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* RefactorBench is a novel benchmark to evaluate LLMs in the complex multi-file code refactoring tasks, which can isolate unique language agent behaviors compared with similar benchmarks like SWE-bench.\n* The benchmark construction process is technically sound, with unit tests included to ensure rigorous correctness assessment. The experiments provide insightful analysis of how current LM agents perform on this new task. Furthermore, rather than simply discussing the results, the authors go a step further by proposing enhancements based on their findings, achieving a 40.4% improvement in performance over baseline agents.", "weaknesses": "A key weakness of this work is its narrow focus on GPT-4o and SWE-Agent for evaluation, omitting other closed models like Claude, as well as open-source models and other LM agent frameworks. For a benchmark paper, evaluating a broader range of baselines would provide a more comprehensive view of existing LLM performance and help new models or LM agents identify their relative strengths and weaknesses. Additionally, a wider evaluation would allow for assessing whether the observed results generalize across different models, strengthening the benchmark’s robustness and relevance to the broader field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lpOTliYgL1", "reviewer": "Reviewer_VXXQ", "summary": "RefactorBench introduces a new benchmark for evaluating LM agents on performing a variety of multi-file \"refactoring\" tasks. For instance, one task may involve adding new arguments to a class with sensible default values and updating dependent / affected components throughout the code base. There seem to be other diverse tasks. The authors show that these tasks, while seemingly straightforward for human programmers (most being solvable within 5 minutes by humans), can still be a challenge for existing agents when evaluated on AST-based unit tests against reference solutions. They provide a few evaluation settings, such as having a `lazy`, `standard`, and  `descriptive` version of each task and provide analysis of the dataset and model performance. Finally, state-aware interfaces for LM agents are proposed as an improvement over a simple SWE-agent baseline agent, which includes providing LMs a summary of code changes made overtime, that is demonstrated to improve performance.", "questions": "- Line 162: can you clarify what you mean by design principle?\n- Seems to be a mistake on line 415: Perhaps you mean a 40.4% relative improvement in performance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- RefactorBench introduces a practical setting that remains relatively difficult for LM agents.\n- The data quality seems fairly high based on my manual inspection.\n- The three-tiered instruction approach (lazy, standard, descriptive) offers valuable insights into agent capabilities.\n- The proposed improvements to SWE-agent and agent architectures in general represent meaningful advances.\n- Provides thorough analysis.", "weaknesses": "- The term `pseudo-task` is used in the abstract but doesn't seem to be used or elaborated on explicitly in the main text. I recommend using a more common term in the abstract, especially if it isn't defined elsewhere.\n- Tractability is first used in Section 3.1 but is not defined until a bit later in section 3.2. I recommend pointing to this in the text in 3.1 when it is first used or cite a work that introduces this term. It doesn't seem conventional enough to use without definition.\n- Figure 1 lacks substance and could be more much informative.\n- With 100 instances the benchmark remains a bit small - however it may be understandable provided that there may a fair bit of human labor involved in creating instances.\n- The lack of a train or, more importantly, a development set may limit the useful lifetime of this benchmark. Test-split only benchmarks encourage overfitting when they require some upfront system design (like agent systems). This may also have been a problem in the development of the agent systems presented in this paper.\n- The writing is acceptable but could be slightly simplified or reorganized for greater clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ag9GeObdiY", "reviewer": "Reviewer_2pe1", "summary": "This paper presents RefactorBench, a new benchmark designed to evaluate language agents ability to perform non-bug-fixing refactoring of multi-file code repositories. The authors run baselines and present analysis of common failure modes of existing LM agents on the task. The authors then present a new approach to improving AI agent task performance by directly modeling and conditioning inference on task state.", "questions": "* how did you choose the repositories to do refactoring on?\n* Why do you think that refactoring now causes LM agents to edit the wrong files, while debugging LM agents rather consistently do? Does it have to do with the nature of refactoring being correct --> intermediately broken --> correct but refactored?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* clear explanation of generation of the benchmark + helpful metrics of the benchmark by proxies for difficulty (number of files / lines, instruction detail). Many questions can be explored with this dataset by using the different degrees of detail of NL insn, small-granularity feedback from the AST unit tests, and complexity of large repositories\n* clean formulation of the task as a POMDP well-addresses the identified failure modes and motivates the state-aware method \n* compelling analysis of the results, with proper side experiments and task-related ablations to motivate state-aware agents\n* refactoring is an interesting task, as prior works have focused on bug-fixing refactors. The insight that refactoring often necessitates \"broken\" intermediate states on the way to task completion poses an interesting research question for agentic frameworks.\n* representing the \"state\" as NL keeps the framework flexible and fairly interpretable", "weaknesses": "* the main distinction from SWE-bench is described as \"handcrafted and underrepresented multi-file refactoring tasks\", but SWE-bench does include some tasks that resemble \"refactoring\" in the form of feature requests.\n* I worry that this is a benchmark that would soon be saturated, but I would not necessarily consider that a reason for rejection. Additionally, the contributions of this paper in the form of analysis make this point less relevant.\n* I would have liked to see more analysis of why state is important specifically for refactoring. Can we get some qualitative analysis of what sorts of subtasks are done better with state modeling? What are some failure cases in the NL description of the state?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUXlPVXLRr", "reviewer": "Reviewer_vwuT", "summary": "The sources present *RefactorBench*, a novel benchmark for evaluating language models' (LMs) capabilities in multi-file code refactoring tasks. The benchmark comprises 100 handcrafted tasks requiring edits across multiple files, with varying instruction sets to assess the models' ability to follow instructions and reason over past actions. The authors demonstrate that current LMs struggle with these complex tasks, particularly due to difficulties tracking past actions and maintaining context. To address this limitation, they propose \"state-aware\" interfaces that provide LMs with a representation of the current state of the codebase, leading to significant improvements in task performance. Finally, the authors outline potential future directions for research, including developing more efficient context generalization techniques and constructing robust interaction mechanisms for language agents in digital environments.", "questions": "''Repetition is the root of all software evil'' -- Martin Fowler\n\nIt is a reference to the software engineering principle of Don't Repeat Yourself (DRY). The idea behind DRY is that repeating code or logic in multiple places can lead to problems.\n\nWhat is the relationship of this quote with the whole paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This is a well-written benchmark paper with clear motivation.\n\n1. Filling the gap between existing bug-fixing benchmarks (e.g., SWE-Bench) and the need for real-world developers (e.g., code refactoring). Repo-level code generation benchmarks [1, 2, 3] are necessary.\n2. Well-organized and comprehensive literature review for automated software engineering (ASE) systems.\n3. Clear data collection procedure. Statistics on RefactorBench tasks, repositories, and AST-based unit tests are included.\n4. Clear problem formulation as POMDP. Standardized evaluation with AST-based unit tests. The code is easy to run from anonymous GitHub.\n5. An interesting discovery of the SWE-Agent prone to fix bugs instead of refactoring. The error analysis is also comprehensive.\n6. An interesting discovery of state awareness's significance in enhancing language agents' capabilities with a synthetic experiment. The figure and flowchart are easy to understand.\n\n\n[1] Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., ... & Chen, W. (2023, December). RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 2471-2484).\n\n[2] Li, B., Wu, W., Tang, Z., Shi, L., Yang, J., Li, J., ... & Chen, K. (2024). Devbench: A comprehensive benchmark for software development. arXiv preprint arXiv:2403.08604.\n\n[3] Liu, Y., Tang, X., Cai, Z., Lu, J., Zhang, Y., Shao, Y., ... & Gerstein, M. (2023). ML-Bench: Large Language Models Leverage Open-source Libraries for Machine Learning Tasks. arXiv e-prints, arXiv-2311.\n\n[4] Huang, Q., Vora, J., Liang, P., & Leskovec, J. (2023). Benchmarking Large Language Models as AI Research Agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop.", "weaknesses": "This paper has several weaknesses regarding the evaluation methodologies.\n1. The tasks are relatively easy:\n    - For most of the tasks given in the anonymous GitHub, I've learned that LLMs cannot even move some code snippets to another file or add additional parameters to one function. Is that correct?\n       - In `problems/ansible_refactor/sort-groups-to-group-sort-task.txt`: Rename `sort_groups` to `group_sort` and update all references in the repo.\n       - In `problems/ansible_refactor/add-log-parameter-get-group-vars-task.txt`: Add a log parameter to `get_group_vars,` keep the base case as true, and update the rest of the repository calls to have `log=False.`\n       - $\\ldots$\n    - From my experience using Cursor and GitHub Copilot, the observation in this paper is not true. Most refactoring like these can be solved with one click of the mouse.\n       - For example, I can modify the original function using the Cursor and open all the relevant files (i.e., Ctrl+F for that specific function). The Cursor will devise an auto-completion suggestion to refactor all usage of this function.\n       I am pretty sure that if our language agent is equipped with some retrieval tools and employs a hierarchical structure (i.e., multi-agent) in which each agent updates one file separately, it would largely improve performance and consistency.\n\n2. Bad definition of refactoring:\n    - Martin Fowler's \"Repetition is the root of all software evil\" is a good point. However, I did not see any prompt examples of reducing repetition within the refactor tasks. From my daily experience, a good refactor should reduce the complexity and repetition of a large codebase and increase the portability of modules and functions. It is not about adding a logger to a function or moving some code to `utils.py.`\n    - The refactor should not be evaluated using unit tests only. Unlike the bug-fixes scenario in SWE-Bench, the function before refactoring should also pass the unit test. The objective should be, for example, to reduce the length/loops of the initial code after refactoring while still passing the unit test.\n    - I don't have a clear idea of how to evaluate the refactoring process, but it is obvious that using a pass rate for unit tests is insufficient. Below are some potential metrics you can choose from (maybe for your next benchmark).\n      - *Test Execution Time*: After refactoring, measure if the test suite executes faster.\n      - *Linting and Style Compliance*: Assess adherence to coding standards. (This might not be helpful because most code generated by LLM should follow linting compliance).\n      - *Lines of Code (LOC)*: While not always directly indicative of quality, a reduction in LOC often reflects simplification.\n      - *Function Size*: Track average function length; shorter, well-named functions are generally easier to understand and test.\n      - *Coupling Metrics*: Evaluate coupling using Coupling Between Objects (CBO) measures. (I hate the coupling of my code. So, if the LLMs can solve that, it will be really helpful.)\n      - *Dependency Analysis*: Check the number and nature of dependencies between modules. Reducing unnecessary dependencies enhances modularity and flexibility in the code.\n\n3. Other frameworks worthy to test on [1, 2, 3]\n     - Aider [1] also has a good way of code edits and refactoring using `git diff` [4].\n     - Papers such as [2, 5] evaluate their performance on multiple agent frameworks.\n\n[1] P. Gauthier. How aider scored sota 26.3. https://aider.chat/2024/05/22/swe-bench-lite.html. Accessed: 2024-06-05.\n\n[2] Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., ... & Neubig, G. (2024). OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv preprint arXiv:2407.16741.\n\n[3] Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024). Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489.\n\n[4] P. Gauthier. Unified diffs make GPT-4 Turbo 3X less lazy. https://aider.chat/2023/12/21/unified-diffs.html. Accessed: 2024-10-31.\n\n\n[5] Liu, Y., Tang, X., Cai, Z., Lu, J., Zhang, Y., Shao, Y., ... & Gerstein, M. (2023). ML-Bench: Large Language Models Leverage Open-source Libraries for Machine Learning Tasks. arXiv e-prints, arXiv-2311.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)