DOI: 10.5281/zenodo.17196849
title: SWE-Refactor: A Repository-Aware Benchmark for Evaluating LLMs on Real-World Code Refactoring

description: SWE-Refactor


SWE-Refactor is a new benchmark for evaluating LLM-based code refactoring. It contains 1099 real-world, pure refactorings collected from 18 Java projects.

Each refactoring instance is verified through:



Compilation

Test execution

Automated refactoring detection tools


This ensures the correctness and purity of each refactoring.

Compared to existing refactoring benchmarks such as ref-Dataset, community corpus, extended corpus, and RefactorBench,SWE-Refactor stands out in several key aspects:



Includes both atomic and compound refactorings.

Guarantees pure refactorings with no entangled changes.

Provides developer-written ground truth and test cases.

Ensures test availability for correctness validation.

Built through a fully automated pipeline from real project commits.


 



SWE-Refactor Sample Schema


Each sample in the SWE-Refactor benchmark contains the following fields:



Basic Information






type (string)Type of the applied refactoring (e.g., Inline Method).




description (string)A concise summary of the refactoring action, including involved methods and visibility changes.




projectName (string)Name of the project containing the refactoring (e.g., checkstyle).




commitId (string)Git commit hash where the refactoring was applied.




uniqueId (string)A unique identifier derived from commit and line information.





Location & Structure






diffLocations (list of dicts)Each dictionary contains:



filePath: path of the modified file.

startLine, endLine: start/end line numbers.

startColumn, endColumn: start/end column numbers.





filePathBefore (string)File path before the refactoring.




filePathAfter (string)File path after the refactoring (if moved).




moveFileExist (bool)Indicates whether the target class exists in the destination file after the method was moved.





Code Snippets






sourceCodeBeforeRefactoring (string)The method body before refactoring.




sourceCodeAfterRefactoring (string)The method body after refactoring.




sourceCodeBeforeForWhole (string)Full content of the file before refactoring.




sourceCodeAfterForWhole (string)Full content of the file after refactoring.




diffSourceCode (string)Line-level diff between the before/after versions.





Code Metadata






methodNameBefore (string)Fully qualified method name before refactoring.




classNameBefore (string)Fully qualified class name before refactoring.




classSignatureBefore (string)Declaration of the class (e.g., class SinglelineDetector).




callInfo (string)Call relationships relevant to the refactoring; "N/A" if unavailable.





Purity Validation






isPureRefactoring (bool)Whether the change is a pure refactoring (no semantic/feature change).




purityCheckResultList (list of dicts)Each dict includes:



isPure

purityComment

description

mappingState






Compilation & Testing






compileResultBefore (bool)Whether the project compiled successfully before refactoring.




compileResultCurrent (bool)Whether the project compiles successfully after refactoring.




compileJDK (int)Java version used for compilation (e.g., 11).




compileCommand (string)Maven command used for compiling the project.




hasTestC (bool)Whether the refactored method is covered by any test cases.




coverageInfo (dict)Test coverage statistics:



INSTRUCTION, LINE, COMPLEXITY, METHOD: each with missed and covered.






Experimental Results Folder


The experimental result directory contains all evaluation outputs on SWE-Refactor.

It is organized by prompting strategy:



multi-agent

rag

simple prompt

gpt-4o_result

openai-codex_result


Under each strategy, we include results from 9 widely-used LLMs, such as:



GPT-4o-mini, GPT-3.5-turbo-0125

DeepSeek Coder (6.7B & 16B), DeepSeek-Chat

CodeLlama (7B & 13B)

Qwen2.5 Coder (7B & 14B)


Each folder contains model-specific refactoring results.

At the root, the file Experiment result on SWE-Refactor.xlsx summarizes overall success rates and detailed comparisons across all strategies and models.



Code Folder


The code directory contains all scripts and configurations for constructing and evaluating SWE-Refactor.



Subdirectories




rag/: Code for building contextual Retrieval-Augmented Generation (RAG) and retrieving relevant examples.

data/: Includes static tools, prompt templates, and temporary runtime folders.

model/: Defines the core refactoring entities used throughout the pipeline.




Key Files




config.yaml: Configuration file for evaluating SWE-Refactor.

requirements.txt: Python dependencies for running the evaluation.

multiple_agent_rag_refactoring_main.py: Implementation of the RAG and multi-agent workflow.

llm_refactoring/: Implementation of simple prompt strategy.

pre_process_data/: Scripts for constructing the SWE-Refactor benchmark.

clone.sh: Script to clone target project repositories.


 



Configuration


There are four configurations in config.yaml that need to be set.



project_prefix_path: {Path to your project directory, e.g., /Users/xxx/xxx/SWE-Refactor/code}

OPENAI_API_KEY: {Your OpenAI API key}

chromadb_host: {ChromaDB host address; use "localhost" if running ChromaDB locally}

project_name: {Name of the evaluation project, e.g., "commons-io"}




How to run the code




Set up




install the requirements.

install the chromadb vector database.



the guide link: trychroma

once the installation is complete, you need to configure chromadb_host in the config.yaml.

it is recommended to use a local Docker installation, as it is more convenient.



install the jenv, a tool for switching between different Java versions.



the guide link: jenv

install Java 8, Java 11, Java 17, and Java 21 using jenv



install the build system (Maven and Gradle)

run clone.sh to clone the project code to be analyzed

configure project_prefix_path, OPENAI_API_KEY, project_name in the config.yaml.




Automatic pipeline for construction SWE-Refactor




cd ./code/data/tools/RefactoringMiner-3.0.10/bin
./RefactoringMiner -pbc {project_path} {start_commit} {end_commit} e.g. ./RefactoringMiner -pbc /RefactoringMiner/tmp/checkstyle 0ae1b19ddf4167c3d3fdc2544980a00927c9b974 b007d563c4f9da44040452a8a9de2b76bc64875e
(update param in pre_process_data.py)
python pre_process_data.py

 




Evaluation




python llm_refactoring.py
python multiple_agent_rag_refactoring_main.py