Synthesizing Software Engineering Data in a Test-Driven Manner
Authors: Lei Zhang
Jiaxi Yang
Min Yang
Jian Yang
Mouxiang Chen
Jiajun Zhang
Zeyu Cui
Binyuan Hui
Junyang Lin
Venue: ICML 2025 poster (2025)
Keywords: Large language Model
Software Engineering
Test-Driven Development
Code Agent
Primary area: applications

Abstract: We introduce **SWE-Flow**, a novel data synthesis framework grounded in Test-Driven Development (TDD).
Unlike existing software engineering data that rely on human-submitted issues, **SWE-Flow** automatically infers incremental development steps directly from unit tests, which inherently encapsulate high-level requirements.
The core of **SWE-Flow** is the construction of a Runtime Dependency Graph (RDG), which precisely captures function interactions, enabling the generation of a structured, step-by-step *development schedule*.
At each step, **SWE-Flow** produces a partial codebase, the corresponding unit tests, and the necessary code modifications, resulting in fully verifiable TDD tasks.
With this approach, we generated 16,061 training instances and 2,020 test instances from real-world GitHub projects, creating the **SWE-Flow-Eval** benchmark.
Our experiments show that fine-tuning open model on this dataset significantly improves performance in TDD-based coding.
To facilitate further research, we release all code, datasets, models, and Docker images at [Github](https://github.com/Hambaobao/SWE-Flow).

SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner
Lei Zhang' 12
Jiaxi Yang 12
Min Yang" Jian Yang
Mouxiang Chen Jiajun Zhang Junyang Lin
Zeyu Cui
Binyuan Hui
Abstract
engineering (Rozière et al., 2023; Guo al., 2024; Hui al., 2024; Huang al., 2024). As result, numerous LLM- powered code applications have emerged, including GirHub Copilor', which provides code completion and answers to programming-rclatn queries; Cursor", which enables cross- file code modifications; and Devin', an autonomous agent designed for fully automated software development. These tools are becoming essential for enhancing developer pro- ductivity and advancing intelligent software engineering.
We introduce SWE-Flow, a novel data synthe- sis framework grounded in Test-Driven Devel- opment (TDD). Unlike existing software engi- neering data that rely on human-submitted issues, SWE-Flow automatically infers incremental de- velopment steps directly from unit tests, which in- herently encapsulate high-level requirements. The core of SWE--Flow is the construction of a Run- time Dependency Graph (RDG), which precisely captures function interactions, enabling the gener- ation of structured, step-by-step development schedule. At each step. SWE-Flow produces a partial codebase, the corresponding unit tests, and the necessary code modifications, resulting in fully verifiable TDD tasks. With this approach, we generated 16,061 training instances and 2,020 test instances from real-world GitHub projects, creating the FVEE-Flow-Bench benchmark. Our experiments show that fine-tuning open model on this dataset significantly improves performance in TDD)-based coding. To facilitate further research, we release code, datasets, models, and Docker images at Github.
Despite their impressive capabilities, current LLMs still face limitations when applied to real-world software devel- opment. Existing evaluations, such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), primarily assess standalone function implementations, whereas practi- cal development involves complex dependencies, incremen- tal modifications, and multi-file interactions. Constructing datasets and evaluation methodologies that more accurately reflect real-world development challenges is thus a critical and ongoing research problem. Recent efforts, such as SWE- Bench (Jimenez al., 2023), have attempted to bridge this gap by mining Github Issues from open-source projects, capturing authentic bug fixes and feature enhancements. However, this approach heavily depends on the availability and quality of human-submitted issues, requiring extensive data cleaning and filtering. Furthermore, the reliance of SWE-Bench on human -genrratd commits derived from issue reports fails to encompass the full spectrum of de- velopment tasks and variations, thereby overlooking key aspects of the iterative and complex nature of real-world software development.
1. Introduction
In recent years, Large Language Models (LLMs) have achieved remarkable performance in code-related tasks (Chen et al., 2021). Training on large-scale code data, these models have made significant advancements in code comple- tion, generation, debugging, and refactoring within software
To address these challenges, we introduce SWE-Flow, a reverse data synthesis approach centered on Test-Driven De- velopment (TDD) (Beck, 2002). TDD is highly structured methodology in which development is driven by test cases: developers write tests first, then implement the required functionality, and finally verify correctness by executing the tests. SWE-Flow automatically infers the incremental development process directly from unit tests, thereby gen- crating high-quality training instances. The key insight is
Work done during an internship at Alibaba Qwen.
'Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China Uniiversity the Chi- nese Academy of Sciences, Beijing. China Allibaaa Group, Bei- jing. China 'Zhejiang University, Hangzhou, China Uniiverityy of Science and Technology of China, Hefei, China. Corre- spondence to: Min Yang yang siat.ac.en>>, Binyuan Hui <binyuan. hby
inc.om...
httpss//jihubb.co////ttttrs/pppp "htps://www.cursor.com http::/ddvvinaii
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SWE-Flow: Synthesizing Software Engincering Data in Test-Driven Manner
that each unit test inherently represents a high-level expres- sion of requirements. It specifies the behaviors the code must exhibit and implicitly encodes the developer's inten- tion and design considerations. Consequently, SWE-Flow eliminates the need for human commit histories by harness- ing TDD to automatically produce development tasks with clear structures and explicit goals. Concretely, SWE-Flow captures the function call relationships during unit test exe- cution to construct Runtime Dependency Graph (RDG) for the entire project. This tactic overcomes the limitations of traditional static code analysis tools, which often struggle to accurately parse the dependencies of functions and variables. Drawing on the RDG, SWE-Flow generates a project de- velopment schedule that delineates how an entire codebase can be built from scratch in an incremental manner. At each step, new functions must be implemented on top of existing functionality to pass the corresponding unit tests. For each development step, SWE-Flow produces three types of train- ing instances: (i) Partial Codebase: The codebase is stripped of the functions that need to be implemented in the current step, simulating the state of incomplete development. (ii) Requirement Document: Unit tests associated with the cur- rent step provides a high-level specification of the required functionality. (iii) Reference Solution (diff)): The difference between the complete codebase and the partial codebase, serving as guide for the development task. SWE-Flow offers three major advantages:
We present a dedicated benchmark for evaluating LLMs realistic software engineering tasks, address- ing a significant gap in existing assessment methods.
We generate 16,061 training instances and fine-tune demonstrating the ef- ficacy of SWE-Flow data in empirical studies. We publicly release all code, models, datasets, and Docker images, fostering further research in the community.
1ee.5-Cooer--22-Issttte
Additionally, data generated by SWE-Flow has the potential to support two future directions. Firstly, by scaling the SWE- Flow data in pre-training, one can further strengthen LLM capabilities in software engineering tasks. Secondly, since SWE-Flow provides verifiable correctness feedback, it can be integrated into reinforcement learning.
2. Preliminaries
To ensure clarity and consistency, we define the key nota- tions used in the SWE-Flow framework in this section.
2.1. Definition of Software Engineering Data
Definition 2.1. Software Engineering Data (SED). An SED instance is tuple (C, S, G),, where:
C (Codebase): partially implemented software;
Verifiability: All data is centered on unit tests, ensur- ing generated code is both executable and verifiable.
S (Specification): textual requirements;
G (Ground-Trutt Patch): expected implementation;
Given any codebase with unit tests, SWE- Flow can easily synthesize TDD-compliant training data, obviating the need for excessive data filtering.
Scalability:
T (Unit Test): test cases validating G.
Given a dataset {(C), S G,,, 7;)}, we train an LLM M so that (C,, + G,, with correctness verified by T,
Configurability: SWE-Flow allows tuning the diffi- culty level based on the complexity of function calls, providing various levels of LLM training and evalua- tion.
2.2. Definition of Function Node
A Function Node (FN) represents a function in a code repository, uniquely identified by the triplet: filepath, I Lineno), and (function name). [filepath] is the relative path of the file from the root of the repository, iineno] is the starting line number where the function is defined, and function name is the name of the func- tion. Based their roles and calling relationships during unit testing, Function Nodes are categorized into four dis- tinct types, as defined below:
Using SWE-Flow, we synthesized 16,061 training instances and 2,020 test instances from open-source GitHub projects, and we introduce SWE-hlow-Bench, a specialized bench- mark for evaluating LLM performance in TDD-oriented fine-tuned Qwen2.5-Codee
tasks. Furthermore, Instruct on data generated by SWE-Flow. Experimental results demonstrate that SWE-Flow data significantly en- hance the TDD development capabilities of the LLM, thus validating its effectiveness.
Definition 2.2. Target Test Function Node (TTEN). A Targel Test Function Node serves as the entry point for a unit test. It is explicitly invoked by the testing framework to initiate the testing process. For instance, in frameworks like pytest or unitiest) functions prefixed with (test_) are typical Target Test Function Nodes.
In summary, our contributions are as follows:
We propose a novel TDD-based data synthesis strategy that effectively enhances LLM performance in incre- mental development tasks.
2

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Step
Test Function
Target Function
Dependent Function
Function Call
Dependencies Resolving
Dataset
test_core Testtrrttsii 119:test states
Lib/oython//ne unt ttest/ case. py: 188 Cas---s--eeeeee Case:
Start the Program Unit Tests
Capture valid functions with custom hooks and resolve dependencies
Pull a project from GitHub and configure the dependency environment
py: Machi cked ose iomment
Docker En Unit Tests
Runtime Function Call Stack
Function Dependencies
Step
del leet wwwculateeeeece price(:
Requirement Description
Convert to human-like requests
Test Nermal discount calculation
Fundtoon "calculate_piinf ppiicee
calculato price(100, 0.2) ==
Test 2: No discount
Generate detailed requirement description
calculate calculate_price100,,
100.0
Purpose
based on content of test function
eesert calculate price(100, 1.0) 0.0
The function calculates the final
Input Parameters
calculate price( price, discount_rate, tax-Fallse,
Resolved Function Dependencles
def calculate__proot price. discount rate. apply. Fax=False, tax rato-0.1
Remove all content of dependent functions from codebase
tax, 18te=0.11
Skeletonize Function
y:
Calculate discounted price discounted price price (... # Apply tax needed apply_tax:
Save dataset
Calculate final product applying discount and optionally adding tax
Core Functions
Generate a concise doc- string based content target function
Skeletonized Codebase
rownddoinnl_noeeeeeeoo pricee
Figure 1. The framework of SWE-Flow. Step Given a codebase and its corresponding development environment, sweflow executes unit tests, constructs the project's s Runtime Dependency Graph (RDG), and generates development schedule. Step 2: Based the development schedule, sweflow removes the implementation of core functions covered by the current step's 's test functions, forming an incomplete codebase for development. Additionally, it generates development document based the content of the test functions.
Definition 2.3. Dependent Test Function Node (DTFN). Dependent Test Function Node is designed to assist the Target Test Function Node in completing its execution. These nodes typically include setup, teardown, or other functions required for test environment management.
that are invoked during execution. Each node v E V corresponds to Function Node in Section 2.2.
E' V the set of directed edges, where an edge (u, v) E indicates the function represented by node directly calls the function represented by node
Definition 2.4. Target Core Function Node (TCFN). A Target Core Function Node represents the core funetionality explicitly invoked by Target Test Function Node during the test execution. These nodes are the primary focus of the test and are integral to the functionality being verified.
3. SWE-Flow
3.1. Overview
Definition 2.5. Dependent Core Function Node (DCFN). Dependent Core Function Node supports the Target Core Function Node by providing auxiliary core functionality required for its implementation. These nodes are indirectly invoked during the execution of the test and are essential for the successful operation of the Target Core Function Node.
As illustrated Figure 1, the SWE-Flow framework con- sists of two main steps: Given GitHub project along with its corresponding development environment (e.g.. Docker container), SWE-Flow first executes unit tests to build the project's RDG; 2. Based on the constructed RDG, SWE-Flow generates a development schedule for the project and synthesizes software engineering data accord- ingly. The following sections provide detailed explanation.
2.3. Definition of Runtime Dependency Graph
Definition 2.6. Runtime Dependency Graph (RDG). A Runtime Dependency Graph is directed graph denoted as G = (V, E), where:
3.2. Runtime Dependency Graph Generation
As illustrated in the upper part Figure 1, given the source code of a GitHub project along with its corresponding devel- opment environment (e.g., Docker container), SWE--Flow
/ the set of nodes, representing the Function Nodes
3

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Algorithm 1: The procedure of SWE-Flow-Trace
Algorithm 2: The procedure of SWE-Flow-Schecule
Input: Test program test.program
Input: RDG set SRDG
Output: Function call relationship dictionary
Output: Development schedule P
Initialize empty list devSchedule, map and set
Initialize empty stack fune call stack and
dictionary func.call.relation.
developedFuncs:
Procedure TraceCallsffranc, event):
foreach item E SKDG do if item.funcNodes funcNodeTTTestaaa
Generate unique function ID from frame;
event is call and the function ID belongs to the current project then
then
Merge fiem.ttrgge/eesfuunss into
Retrieve the caller information from
func call _allssssckk
else functlooderooeesttappllii//iiiie + item.targef/estfunes
Push the current function call information onto func.eall l..ttckkkkk
Add the caller-callee relationship to
Sort FuncNodeeTTTestMaa by the size of
func call relations if is new; else if event is return and the function ID matches the top of func call stack then
funcNodes and store sortedMap
foreach entry E sortedMap do newF newF uncNodes +
L Pop the top entry from func_call _sackk
entry.funcNodes \ deve opedFuncs;
if rewFuncModes * () then Add newFuncNodes to developedPunce: Append
Procedure Cratoo---Treeeeettt.om Trace(test.program)
Monitor the function call stack in memory with
StaceCalls:
Execute the test program;
Centy.taargeteettFuns. hew.eeeeeeeccceeeeuconeeeeeeeee to devSchedule;
Stop monitoring and save the relationships in
func.call.relationn
Execute swef Towee (test. program) to collect the function call relationships;
else
Merge enty.targge:Test?unes into the last ta targetTestFuncs of devSchedule;
return devSchedule;
first collects unit test information across the entire project to identify all target test function nodes. The collected set of test functions is denoted as FrIEN. Subsequently, SWE- Flow employs customized hook program, SWF-FFlo- Trace, execute all unit tests parallel while recording the function call relationships during each execution. The following command demonstrates how SWE-Flow- Flow-Trace executes unit tests in Python project via the terminal:
mally:
SRDG [(f, RDG(f) f FrTes),
where RDG(f) denotes the Runtime Dependency Graph rooted at the Targel Test Function Node J. The algorithm detailed implementation of SWE-Flow-Trace.
sweflow-trace pytest test_case_ia
3.3. Development Schedule Generation
During the execution phase, SWE-Flow-Trace continu- ously monitors the function call stack, automatically collect- ing all function call relationships generated during runtime. As illustrated the Runtime Function Call Stack of Fig- ure I, the execution of test functions frequently invokes functions from external sources, such as system libraries or third-party dependencies. To ensure the relevance of the collected data, SWE-Flow-Trace filters out these extrane- ous calls and retains only those associated with the current project repository. The relationships retained through this filtering process naturally form the Runtime Dependency Graph (RDG). Once all unit tests have been executed, we obtain set of TTENs and their corresponding RDGs, for-
The procedure of generating the development schedule is detailed in Algorithm 2. Given the constructed set SRDG. we first merge all TTFNs that cover the same Core Func- tion Nodes (CFNs). Next, we sort the elements in SRUG in ascending order based on the number of CFNs each el- ement covers. The sorted set is denoted then iterate sequentially through Storted further merging any TTFNs whose CFNs have already been developed into the preceding element in the set. This ensures that each step in the generated Development Schedule corresponds to valid incremental development process.
Swarted We
After the iteration, we obtain development schedule that

---

SWE-Flow: Synthesizing Software Engincering Data in Test-Driven Manner
Replace Format Solution
Patch Format Solution
The string
replace: path/to/fi
patch
The scope of the context that needs to modifled.
needs to be
SEARCH This 1s file a.
path/to/file.p
*** path/to/fite.ppt
replaced in the original file.
It has some text.
@8 4 +1,4 88
Here is another Line.
This file
The lines to be deleted from the original file.
-It has some text.
The string
>322222 REPLACE
-Here is another line.
used to replace
It has updated text.
+It has updated text.
the original string.
This Line has been modified.
+This line has been modified.
The lines be added original file.
And ends here.
And ends here.
Figure 2. Examples of the Replace Format and Patch Format solution. The left side presents example of Replace Format solution, which follows the GitHub merge conflict format. The right side shows an example a Patch Format solution, which can be directly generated using diff tools.
naturally satisfies the topological dependency order. The development schedule P is formally defined as:
Representative examples of development documents gen- crated by the Qwen2.5-Coder-33 -onoddrr-2B-Innttuuu are provided in the Appendix C.
p = {(FrTe(1)), FICEN(1), FDCIN()))
This approach leverages the structured and goal-driven na- ture of unit tests to ensure the generated development tasks are well-aligned with specifications. The rationale for this method is grounded in several key considerations. Firstly, Test-Driven Development (TDD) Philosophy. Modern software methodologies, such as TDD (Beck, 2002), advo- cate writing tests prior to implementation, treating tests as documentation of intended behaviors. Generating develop- ment tasks from test functions aligns seamlessly with this by tracing requirements directly to these pre-defined spec- ifications. Secondly, Explicit Functional Context. Unit tests act as executable specifications, defining precise soft- ware behavior through concrete inputs, expected outputs, and assertions. This structured context enables us de- rive implementations directly from test cases, minimizing ambiguity. Thirdly, Precision and Minimal Ambiguity, Unit tests enforce strict, verifiable constraints, eliminating ambiguity inherent in natural language requirements. Their deterministic validation ensures alignment with functional expectations, preventing unintended deviations.
where (i), (i), and (i) represent the sets of TTFNs, TCFNs, and DCFNs at the i-th step of develop- ment.
The lower part of Figure 1 illustrates the detailed process of synthesizing software engineering data based the gener- ated development schedule P. The constructed dataset 2 is formally defined as:
D = Si, G., T:))
where C, S. G; and T; denote the awaiting codebase, de- velopment task, ground-trutt solution and corresponding unit tests at the i-th step of development, respectively. The detailed construction for each entry D is described in the following sections.
3.4. Development Document Generation
For each entry in the development schedule P, the content of the target test function nodes (TTFNs) is provided as input to an LLM alongside two-shot examples. The LLM then tasked with generating a detailed development document, also referred to task description, denoted S,, based on the content of the test function.
3.5. Development Codebase Generation
Given an entry from the development schedule P, the code- base generation process begins with the original codebase and involves systematically reducing its content through a process known skeletonization. For Target Core Function Nodes and Dependent Core Function Nodes, employ distinct skeletonization strategies modify the codebase.
To assess the quality of development documents generated by different LLMs from unit test functions, we manually re- viewed a sample of documents produced by state-of-the-ar LLMs, including gpt-4o (Hurst al., 2024), claude-3.5- sonnet (Anthropic, 2024), DeepSeek-V3 (Liu et al., 2024a), and Siwen2.5-Coder-. 32B-Iust (Hui al., 2024). Our sults indicate that, under two-shot setting, the quality of the generated requirement documents remains largely consistent across these models. Given this finding, we opted to use the Qwen2.5-Coder-23 32B-Instruct for development document generation due accessibility and cost-effectiveness.
Target Core Function Nodes. The concrete implemen- tations of functions in the original codebase are entirely removed. To retain contextual information, we utilize an LLM to generate a new docstring based on the original func- tion content. This new docstring either replaces the existing one or supplements it if none was originally present.
5

---

SWE-Flow: Synthesizing Software Engincering Data in Test-Driven Manner
Dependent Core Function Nodes. Both the function def- initions and their contents are completely removed from the codebase. This step ensures flexibility for the subsequent development process, allowing developers to build freely without constraints imposed by the original framework.
Verifiable Data Generation. After setting up the project environment, we used SWE-Flow to collect comprehensive unit test information for each project. We then executed these unit tests in parallel using SWE-Flov-Trace, record- ing detailed function call information associated with each unit test. We only keep the unit tests that pass. The output of this step is a Runtime Dependency Graph (RDG) for each unit test, capturing the functional dependencies and relationships within the project.
The skeletonized codebase C, is then used to develop the corresponding development task.
3.6. Ground-Tree Truth Solution Generation
Utilizing the RDGs generated in the previous step, we em- ployed SWE-Flow-Schedulee to produce a development plan for each project. Based on the development sched- ule, we performed skeletonization of the original codebase, a process that simplifies the codebase while retaining its structure. To construct the ground-truth solutions: First, we compared the skeletonized codebase with the original code- base using diff tool to produce Patch Format ground-truth solutions. Second, we further converted these patmh-format solutions into Replace Format ground-truth solutions for additional versatility in downstream tasks.
The differences between the original codebase and the skele- tonized codebase naturally form ground-truth solution. In real-world software development, generating the com- plete content of edited files can be both time-consuming and cost-prohibitive. Therefore, following the settings of existing evaluations, we define two formats for represent- ing the ground-truth solution: Replace Format and Patch Format. These two formats minimize overhead by focusing on localized changes rather than reconstructing entire files. Specific examples of each format are provided in Figure 2.
3.7. Dataset Construction
Following this process, we synthesized a comprehensive dataset that includes 16,061 training instances and 2,020 test instances, tailored to improve and evaluate the performance of AI systems in real-world software development scenarios.
To construct a high-quality dataset for enhancing LLMS in development tasks, we followed systematic process involving the selection, preparation, and processing of active Python projects. The key steps are detailed below.
4. SWE-Flow-Bench
Collection of Projects. We selected 150 of the most pop- ular and actively maintained Python projects from the li- braries.iof platform. Each project in our dataset meets the following criteria: 1. It has recognized open-source license (e.g., MIT, Apache-2.0). 2. It has received at least 2,000 stars. 3. It has demonstrated recent activity, with updates made within the past six months. For each selected project, we cloned the latest codebase from GitHub and recorded the corresponding commit hash to ensure reproducibility.
4.1. Evaluation Framework
Language Model Evaluation. For the evaluation of lan- guage models, we first construct a task prompt, which con- sists of two main parts: the system prompt and the user prompt. In the system prompt, we define the requirements for the language model to act as an experienced software engineer, specifying the basic expectations for completing the software development task. Additionally, we clarify the solution format and provide specific examples to ensure consistency. In the user prompt, we include relevant files from the codebase that directly related to the current development task, followed by detailed description of the task requirements. Finally, we reiterate the instructions for the solution format to maintain alignment and clarity. Figure 5 in the appendix provides an example of the task prompt.
Preparation of Test Environment. To ensure consis- tent and isolated environment for processing each project, we created a custom script for installing project dependen- cies within a Docker container with following steps: 1. Launches a Docker container to provide an isolated environ- ment. 2. Installs basic or optional dependencies specified in the project's pyproject t toml or [setup up.pyp files. 3. Iteratively parses and installs dependencies listed in any requi rements......................... txt formatted files within the project. This step ensures that the project is fully prepared for subse- quent analysis. Finally, we obtain 74 projects that can pass all the unit tests in the installed test environment. Among these projects, 12 projects are selected for testing, and the remaining 62 projects are used for training.
We then prompt the language model to generate solution for the software development task based on the constructed prompt. The response generated by the language model is parsed to extract the solution that conforms to the specified format. This extracted solution is applied to the correspond- ing codebase under development. Finally, unit tests are executed on the updated codebase to verify whether the task has been successfully completed.
thups///ibraries.io
6

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
LLM
A1
PROMPT
Apply solutions codebase
Test solution
Human Request Code Files
Generate Solutions
Results
Agent
Iterate N times
TASK
Make changes
Run test to check correctness
Send task to agent
Feedbacks
Requirements Document
Agent Framework
Container Workspace
Results
Figure evaluation framework of SWE-Flow-Benh. Upper:
A prompt containing current codebase information, development document, and output format is sent to the LLM. The LLM generates response based on the prompt's requirements. A post- cost-processing tool then extracts the solution from the L.M''s response, applies to the codebase, and executes the corresponding unit tests to verify correctness. Lower: Given incomplete codebase and development document, an agent iteratively performs development until the task is completed or a preset iteration limit reached, After the agent terminates, the corresponding unit tests are executed to assess the correctness of the development.
Agent Evaluation. For the evaluation of agents, we mount the codebase under development to the agent's workspace directory and send the task requirement document from the test samples as the task to be completed. The agent then be- gins iteratively working on the development task. Complex software engineering tasks often require numerous itera- tions to be completed, and in some cases, the development may remain incomplete even after multiple iterations. To minimize testing overhead, we set a maximum number of iterations. the agent exceeds this limit but continues run- ning normally, manually terminate the task execution. After the agent's task execution is terminated, we run the unit tests corresponding to the task to verify whether the development task has been successfully completed.
value measures the efficiency of completing given devel- opment task. Its formal definition is as follows:
Pass Rate
Efficiency Value
log
(> tasks
Iters (tesk)tskk;)
==1
where (terk(task,) is the number of iterations the agent used to complete the i-th task or the max number of iterations set by the user if the agent exceeds the max number of iterations.
5. Experiments
5.1. Fine-tuning Language Models
To validate the effectiveness of the synthesized training data, we fine-tuned the Dwven2.-CCoder32BBBB Instruct model (Hui et al., 2024) using the generated dataset. The entire training process was completed within two hours on 128 H800 GPUs using Megatron-LMM (Shoeybi et al., 2019). For detailed description of the training process and parameters, please refer to Appendix D.
4.2. Evaluation Metrics
Pass Rate. Execution pass rate is the most direct metric to assess the correctness of the generated code. Following previous research on code evaluation (Chen et al., 2021), we adopt the pass rate as the primary metric to evaluate the performance of language models and agents. The pass rate defined the ratio of successfully completed tasks (those that pass unit tests) to the total number of development tasks. The pass rate is calculated as follows:
5.2. Experimental Results
Large Language Models. We conducted a comprehen- sive evaluation of 11 mainstream LLMs on the SWE-Flow- Bench (Lite) benchmark. Furthermore, compared their performance with our UF-Coder-333 Co--Cooder----ttrruuue Instruct, which was fine-tuned from 2024) using our synthesized training data. Figure 4 illus- trates the ability of these LLMs to generate solutions in both Replace and Patch formats. For more detailed evaluation results, please refer to Table the Appendix.
Σ
tasks
(task))
Pass Rate
1=1
tasks
where isPass(task) is a binary variable indicating whether the i-th task passes unit tests.
(Hui et al.,
Efficiency Value addition, for the evaluation of agents, we also use the efficiency value as a metric. The efficiency

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Replace Format
Patch Format
DeepSeek-V33
clande- Sande--.--sennee-1--. 10-22
UF-Coder-322
UF-Coder-322
DeepSeck-R1
21-preiiew2222-0--1
DeepSeek-Coode Coder 2-instrrcc
xpe 46 2024 18
Qwen2 Been2.--Code--BB 32B-Instrc
DecpSeck-RI
ol-presionn-200 20--200--022-09.12
DeepSecl-Cude--N V2-lsrr--l
Jamaa S.-0S---iiit
01-mmm1-2002--0-12
civnde- 3.5-ooneff-10-2
QwenZ55 5-C Coder- Soder--28---ane Stut-rstt
Ovreni--Coder--ee
Liama 3.3-708--88880 Instruct
405B Instruct
4p1-44-2022405-66
DropSeok-V33
Quem2.5-7BB-nssteee
Qwen25--78-4notrurr
110
147
183
220
1.5
187
200
Figure 4. The overview of evaluation results of large language models SWE-Flow- Bench (Lite). The x-axis represents the number of development tasks for which the solutions generated by LLMs successfully pass lests.
As shown in Figure 4, the SF-Coder-32B: Instructt model achieved significant performance improvement over the Qwen2.5-Couee Matterr-2B-hie 32B- Instrucct.. Specifically, its performance in the Replace Format generation was second only to that of DeepSeek- V3 (Liu et al., 2024a), while in the Patch Format generation, it ranked just below claude3.5-some (Anthropic, 2024), outperforming all other models sub- stantial margin. The experimental results demonstrate that the training data synthesized using the SWE-Flow frame- work can significantly enhance the code generation capabili- ties of LLMs in real-world software development scenarios.
1.aud=-..-voamet-10-22
opt-40-20224008-06 (ween2.5codder-32--nntttt
deepseek-chat
pandas
cryptography
16
pylint
10
Birosa
marshmallow
transitions
Agents. SWE-Flow-Bend can be integrated with any Agent framework for evaluation. In this study, we selected OpenHands (Wang et al., 2024a), the most widely used Code Agent framework, for testing. Specifically, we evaluated Blaude-3.5-somee gpt-4o, deepseek-chat and Qwen2.5- Coder-32B-Inne IIntructt on the SWE-Flow-Bencd (Lite) test set. The evaluation results are presented in Figure and Table 6 in the appendix.
waborn
Jinja
py pydantic
Pillow
gensim
Figure 5. Efficiency values of various LLMs integrated with Open- Hands. Each axis represents specific software engineering task. values indicating the corresponding efficiency scores of the agent for that task.
The experimental results show that claude3.5-sonnet (An- thropic, 2024) significantly outperformed other models, in- cluding gpt-4o (Hurst et al., 2024), in real-world software development scenarios. However, even --soonttt struggled to complete those complex development tasks, highlighting the substantial limitations current large lan- guage models in handling practical software engineering challenges. This observation underscores the pressing need for further advancements in the field. Specifically, there is clear requirement for more comprehensive and domain- specific software development datasets to enhance the train- ing of large language models. Such improvements would be essential to bridge the gap between their current capabilities and the demands of real-world software development tasks.
6. Discussion and Future Work
1. Synthesizing More Challenging Data: By merging con- secutive tasks from SWE-Flow-Schedle more complex development scenarios. In the extreme case, combining all tasks into one forces an entire project to be built from scratch. 2. Enhancing Reinforcement Learning: Recent reasoning- aware models need large-scale, verifiable data. SWE-Flow ggnnrrtedd tasks are inherently testable in containerized environments, making them ideal for training
we can create
8

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Table Comparison of existing software engineering benchmarks.
Automated Synthesis
TDD Based
Configurable Tasks
Cross-File Editing
Complex Dependencies
Dataset
SWE--Bench
CommitO- Bennctt-Benchh
X
X
X
SWE-Flow-Bench
RL-baaeed code models. 3. Enhancing Pre-training: Pre- training data determines an LLM's core capabilities. Cur- rent open-source models lack sufficient high-quality, verifi- able software engineering data. By leveraging Cl-enabled GitHub projects, SWE-Flow synthesize large-scale veri- fied corpora, potentially boosting LLM performance in code generation and software development.
the Natural Science Foundation of Guangdong Province of China (2024A151503016) 2025B1515020032), Shen- zhen Science and Technology Innovation Program
(KQTD2019999917233666))
Impact Statement
This study introduces the SWE-Flow framework, designed to synthesize test-driven, iiocuutio-verifiii software engi- neering data to enhance the software engineering capabili- ties of LLMs. Additionally, we propose SWE-Flow-Bench benchmarking framework for evaluating the performance of LLMs and LLM-based agents software engineering tasks. All datasets and models used in this study are open- source and comply with their respective licenses. We hope that our findings and contributions will facilitate future re- search in this field and further advance AI technologies, particularly in the domain of software engineering.
7. Related Work
SWE-Bench (Jimenez et al., 2023) evaluates agents on GitHub issue resolution, emphasizing patch-level fixes rather than broader development capabilities. CommitO- Bench (Zhao al., 2024) requires models generate full implementations in one attempt using all unit tests, which does not align with real-world iterative coding practices. In contrast, WVF-Flow-Bench follows a Test- Devel- opment approach, breaking projects into incremental steps guided by minimal test cases. This enables a finer-graine evaluation of code organization, architecture construction, and functionality expansion while improving interpretability and real-world relevance. For more discussion of related work, Appendix A.
References
Anthropic. Claude 3 family, 2024. URL https: /www. anthropic. com/ news/eww/eeee /tews/clle claude 3-family. Ac- cessed: 2025-01-30.
8. Conclusion
We propose the SWE-Flow framework for generating ver- ifiable software engineering data, along with the SWE- Flow-Bench framework for evaluating the performance of large language models (LLMss) and Al agents real- world software development tasks. Using synthetic software engineering data, we fine-tuned the Qwen2.5-Codee 1ree2.5-Coder-33 -2BB Instruct, resulting in the ST-Coder-32B-Instire which demonstrated significant performance improvements on SWE-Flow -encchh (Lite), thereby validating the effec- tiveness of synthetic data generated by SWE-Flow holds potential for pre-training and post-training, further enhancing the Al cod- ing applications.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint urXiv:2108.07732 2021.
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of language models to fill the middle. arXiv preprint arXhv:2207.1425,,, 2022.
model,
syythsstteeeeeed data. Moreover, the
Beck, K. Test Driven Development: By Example. Addison- Wesley Longman Publishing Co., Inc., 2002.
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-
Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., and Jangda, A. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. JEEE Transactions on Software Engineering, 49(7):3675-366 2023. doi: 10. 10.1109/TSF.2023332644444
Acknowledgments
Min Yang supported by National Key Research and Development Program of China (2022YF00001000, Na- tional Natural Science Foundation of China (62376262),

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Chai, L.., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436. 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.11276 2024.
Chaudhary, S. Code alpaca: An instruction-following Ilama model for code generation. https: /tt.bbb..tt..tggg thub. com/ sahi 1280114 wewoodaaapacaao 2023.
Husain, H., Wu, H., Gazit, T., Allamanis, M., and Brockschmidt, M. Codesearchnet challenge: Evaluat- ing the state of semantic code search. arXiv preprint arXiv: 09436, //arxiv. org. abs / 90..9999.49..
2019, URL http:
Chen, M., Tworek. J., Jun, H.. Yuan, Q.. Pinto, H. O.D. Kaplan, J., Edwards, H., Burda, Y.. Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint trxXvv:200700332 2021.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Cheng. Y., Chen, J., Chen, J., Chen, L.,, Chen, L.., Chen, W., Chen, Z., Geng, S., Li, A., Li, B., et al. Fullstack bench: Evaluating 1lms full stack coders. arXiv e-prints, pp. arXiv-2412, 2024.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.07999 2024.
Gu, A., Rozière, B., Leather, H., Solar-Lezama, A., Syn- naeve, G., and Wang, S. I. Cruxeval: A benchmark for code reasoning, understanding and execution. 2024.
Jimenez, C. E., Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Guo, D., Zhu, Q. Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv:2401.14199. 2024. URL https: //rxiv. org/abs/2401 14196.
Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-T., Fried, D., Wang. S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/221 1.5501,, 2022.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability 1lms via reinforcement learning, arXiv preprint arXiv:2501.12941 12948, 2025.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E.,, Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M., Uma- pathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M.. Wang, Z. V, R. M., Stillerman, J., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., Zhang, Z., Moustafa-Fahmyy N., Bhattacharyya, U., Yu, W. Singh, S., Luccioni, S.. Villegas, P., Kunakov, M.. Zhdanov,
Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al. Metagpt: Meta programming for multi-agent collabora- tive framework. arXiv preprint 2rXiv:2308.003::
2023.
Huang, J., Tang, D., Shou, L., Gong, M., Xu, K., Jiang, D., Zhou, M., and Duan, N. Cosqa: 20,000+ web queries for code search and question answering. 2021.
Romero, M.. Lee, T., Timor, N., Ding, J., Schlesinger, C., Schoelkopf, H., Ebert, J., Dao, T. Mishra, M., Gu, A., Robinson, J., Anderson, C. J., Dolan-Gavitt, B., Contractor, D., Reddy, S., Fried, D., Bahdanau, D., Jernite, Y., Ferrandis, C. M., Hughes, S., Wolf, T., Guha, A., von Werra, L., and de Vries, H. StarCoder: May the source be with you! arXiv preprint arXiv:2305. 06100, 0ads/235..6166) 2023. doi: 10.48550/aaxiiv.2305.000 /
Huang, S., Cheng, T., Liu, J. K., Hao, J., Song, L., Xu, Y., Yang. J., Liu, J., Zhang, C., Chai, L., et al. Opencoder: The open cookbook for top-tier code large language mod- els. arXiv preprint arXiv:241 11.e22/11.444 1,04905, 2024.
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.1 12186, 2024.
URL https: 100 48550/8r5iii 2305 061 61.
Li, Z., Zang, Q., Ma, D.,, Guo, J., Zheng, T., Niu, X., Yue, X., Wang, Y., Yang, J., Liu, J., et al. Autokaggle: A multi- agent framework for autonomous data science com- petitions, arXiv preprint 10.iv:2410.200 20.22444..4... 2024.
Huq, F. Hasan, M., Haque, M. M. A., Mahbub, S., Iqbal, A., and Ahmed, T. Review4repair: Code review aided automatic program repairing. 143:106765, 2022.
10

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C.,, al. Deepseek-
Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and in Neural Information Processing
v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Liu, J., Xia, C. S., Wang, Y., and Zhang. L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.001220 abs22305.012000 2023. URL https: / /tt./aixvvv//./////// www./oggg/s..o.oo.o 2305. 01210.
Oh, A. (eds.). Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans,
LA, USA, November 28 December 9, 2022, 2022b.
Pan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., and Zhang, Y. Training software engineering agents and verifiers with swe-gym. arXiv preprint urXiv:2412.21139, 2024.
Liu, J., Deng, K., Liu, C., Yang. J., Liu, S., Zhu, H., Zhao, P. Chai, L., Wu, Y., Jin, K., al. M2rc-eval: Massively multilingual repository-level code completion evaluation. arXiv preprint arXiv:
2024b.
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I. Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code Llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023. URL attps://arxiv.ort wwwp:/vv.rrg/bb///.o.ooo 12950.
Liu, S., Chai, L., Yang, J. Shi, J., Zhu, H., Wang, L.,, Jin, K., Zhang, W., Zhu, H., Guo, S., et al. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02311 2024c.
Liu, S., Chai, L., Yang. J., Shi, J., Zhu, H., Wang, L., Jin, K., Zhang, W., Zhu, H., Guo, S., et al. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310. 2024d.
Schick, T., Dwivedi- Yu,eedi-YY,,,. J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539-68551, 2023.
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. Starcoder and the stack v2: The next generation. arXiv preprint ar Xiv:22002. 19173, 2024.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint 1909::9909.0 08053, 2019.
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A.,
Blanco, A., Clement, C., Drain, D., Jiang, D... Tang, D., Li, G., Zhou, I.., Shou, I.., Zhou, I.., Tufano, M., GONG, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and LIU, S. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Thitty-ffth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https:://// wttp//wwneeeeeeeeeeeeeee www.reeeieweeeeeeeeeeeeeeeeeeeee forum? id 11611114XXaaa.
Su, A., Wang, A., Ye, C., Zhou, C., Zhang, G., Zhu, G., Wang, H., Xu, H., Chen, H., Li, H., et al. Tablegpt2: A large multimodal model with tabular data integration. arXiv preprint arXiv:241 1.02059, 2024.
Talebirad, Y. and Nadiri, A. Multi-agent collaboration:
Harnessing the power of intelligent Ilm agents. arXiv preprint arXiv:2306.088. 0.3006003144 2023.
Tian, R. Ye, Y., Qin, Y., Cong, X., Lin, Y., Liu, Z., and Sun, M. Debughench: Evaluating debugging capability of large language models. 2024.
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Em- powering code large language models with vvol-instruct. CoRR, abs/2306.08555
Wang. X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407. 16741, 2024a.
2023. doi: 10.48550/ARXIV. 2306.08568. URL https: ///o:///oii.. org/10..0111. 48550/ arXiv 2306. 08568.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022a.
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407. 16741, 2024b.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In
Ouyang, L., WU, J., Jiang, X., Almeida, D.,, Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K.,

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Rogers, A., Boyd-Graber, L.,, and Okazaki, N. (eds.), Proceedings of the 6lst Annual Meeting of the Associ- ation for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 13484-13508. Association for Computational Lin- guistics, 2023. doi: 10. 10.18653/V1/202222 10.16665///000233L--LLOCG 754. URL https:/ /toiiii..o/// or01....110.11655/1111 2023. a223.acllong 754.
Zhang, F., Chen, B., Zhang, Y., Liu, J., Zan, D., Mao, Y.,
Lou, J., and Chen, W. RepoCoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arxin::333. abs/2303.125700 2023a. doi: 10.488500/a1Xiv2233..2222 org 48550 2tt.aaxww.......171...... 12570.
URL ht.tttt::// /doi.
Zhang. L., Wang. J., He, S., Zhang, C., Kang, Y., Li, B., Wen, J., Xie, C., Wang, M.. Huang, Y., et al. Di-bench: Benchmarking large language models on dependency in- ference with testable repositories at scale. arXiv preprint arXiv:2501 13699, 2025.
Wei, Y., Wang. Z., Liu, J., Ding, Y., and Zhang, L. Magicoder: Source code is all you need.
CoRR, abs/2312.02120) 2023. doi: 10.48550/ARXIV. 2312.02120. URL https:
Zhang. R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., and Qiao, Y. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. CoRR, abs/2303, 16199, 2023b. doi: 10.48550/ARXIV, 2303.16199 URL https: : htt////oii.oo//// //t.i.oor//....1// 48550/ a arXiv 2303.1611111 16199.
ar axXiv.2312 02120.
Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Liang, D., Shu, D., Cheng, X., Sun, T. et al. Tablebench: A comprehensive and complex benchmark for table ques- tion answering. arXiv preprint arXiv:2408.00 09174, 2024.
Zhang, S., Zhao, H., Liu, X., Zheng. Q. Qi. Z., Gu, X., Dong, Y., and Tang. J. Naturalcodebench. Examining coding performance mismatch on humaneval and natural user queries. In Findings of the Association for Com- putational Linguistics, ACL 2024. Bangkok, Thailand and virtual meeting, August I 11-16, 2024, pp. 7907-7928, 2024.
Yan, W., Tian, Y., Li, Y., Chen, Q. and Wang, W. Code- transocean: comprehensive multilingual benchmark for code translation. 2023.
Yang, A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint
2024a.
Zhao, W., Jiang, N., Lee, C., Chiu, J. T., Cardie, C., Gallé, M., and Rush, A. M. CommitO: Library generation from scratch. arXiv preprint arXivv:2412.0 24/2.//769 2024.
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engineer- ing. arXiv preprint arXiv:2405. 15793, 2024b.
Zhuo, Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari,
R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406 2024.
Yang. J., Yang, J., Jin, K., Miao, Y., Zhang, L., Yang, L., Cui, Z.. Zhang, Y., Hui, B., and Lin. J. Evaluating and aligning codellms on human preference. arXiv preprint arXiv:2412.052100 2024c.
Yang. J., Zhang, J., Yang. J., Jin, K., Zhang, L.. Peng, Q., Deng, K., Miao, Y., Liu, T., Cui, 7., et al. Execrepobench: levei-level executable code completion evaluation. arXiv preprint arXiv:2412.1 1990, 2024d.
Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q. Ma, Y., Liang, G., Li, Y., Wang, Q. and Xie, T. Codereval: A benchmark of pragmatic code generation with gen- erative pre-trainee me-ttrined models. In Proceedings of the 46th IEEF/ACM International Conference Software Engi- neering, pp. 1-12,2024 2024.
Yu, Z., Zhang. X., Shang, N., Huang, Y., Xu, C., Zhao, Y., Hu, W., and Yin, Q. Wavecoder: Widespread and versatile enhanced instruction tuming with refined data generation. CoRR, abs/2312.14187, 2023. doi: 10.48550/ ARX1V.2312.44187. URL https: http//oi..ogg/o//////o 48550/arxiv 11555//aXXxv.221.11433333 2312.14187
12

---

SWE-Flow: Synthesizing Software Engincering Data in Test-Driven Manner
A. Additional Related Work
A.1. Code Large Language Models
The advent of large language models (LLMs) tailored for code-centric tasks, such as CodeLlama (Rozière et al., 2023), DeepSeek- (Guo et al., 2024), OpenCoder (Huang et al., 2024), and Qwen2.5-Coder (Hui al., 2024), has rev- olutionized software engineering by automating repetitive tasks, proposing code improvements, and facilitating natural language-to-code conversion. These models, trained vast corpora of billions of code snippets, have significantly enhanced the development process. Notable contributions include Starcoder (Li et al., 2023; Lozhkov et al., 2024), CodeLlama (Rozière et al., 2023), each advancing coding assistance tools with unique innovations. Inspired by the success of grammar- based parsed trees in various domains, we leverage the abstract syntax tree to augment code completion training, further promising greater efficiency and intuitiveness in software creation.
A.2. Code Agents
Recent research highlights the pivotal role of large language models (LLMs) in the development of Al agents, showcasing their capability in facilitating complex task execution through tool utilization (Schick et al., 2023; Talebirad & Nadiri, 2023; Hong et al., 2023). Notable examples include ToolFormer, which enables tools to be used more effectively by LLMs; Meta-GPT and BabyAGI, which demonstrate advancements in autonomous task management. Studies on self-edit and self-dehug have further illustrated the capacity of code models engage in multi-round interactions for code correction and improvement. Contemporary work also underscores the efficacy of agent systems like OpenDevin (Wang al., 2024b) and SWE-Agent (Yang et al., 2024b) in handling complex programming tasks at the repository level, such as SWE-Bench (Jimenez et al., 2023; Pan et al., 2024)
A.3. Code Instruction Tuning with Synthetic Data
Instruction tuning represents a significant advancement in the field of large language models (LLMs) by refining these models with specifically designed instruction datasets, thereby improving their ability to follow instructions more accurately and generalize better (Ouyang et al., 2022b; Zhang et al., 2023b; Wang et al., 2023). This method involves using a foundational LLM to generate initial instruction data, which is then used to fine-tune the model, enhancing its performance through synthetic data (Wang et al., 2023: Chaudhary, 2023; Yang et al., 2024c). To further this approach, WizardCoder (Luo et al., 2023) introduced code Evol-Instruct, utilizing heuristic prompts to increase the complexity and diversity of the synthetic dataset, thus producing higher-quality data. More recently, initiatives such as OSS- (Wei et al., 2023) and CodeOcean (Yu et al., 2023) have leveraged real-world code snippets to guide LLMs in generating more controllable and realistic instruction corpora.
A.4. Code Benchmarks
Code edit and generation is a basic task for code language models (LLMs), requiring them to interpret natural language
descriptions and generate corresponding code snippets that fulfill user requirements (Gu et al., 2024; Lai et al., 2022; Liu et al., 2023; Yu et al., 2024; Ii et al., 2024). To thoroughly evaluate the diverse capabilities L.LMs, numerous benchmarks have been proposed, including code translation (Yan et al., 2023), code retrieval (Huang et al., 2021; Husain et al., 2019; Lu et al., 2021),, code completion (Bavarian al., 2022: Liu et al., 2024b; Zhang et al., 2023a; Yang al., 2024d), code debugging (Huq et al., 2022; Tian et al., 2024: Liu et al., 2024d), and structured data understanding (Wu et al., 2024; Su et al., 2024). Further, multilingual benchmarks like MultiPl-E, McEval, and MdEval (Cassano al., 2023; Chai et al., 2024; Liu et al., 2024c) have been proposed to evaluate the multilingual capabilities of code LLMs, ensuring their effectiveness across various languages and applications. Recent studies explore more diverse scenarios (Jain et al., 2024; Zhang et al., 2024; Cheng et al., 2024; Zhuo et al., 2024) to evaluate the model performance across a variety of real-world coding scenarios, such& as LiveCodeBench and NaduralCodeBench.
13

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
B. SWE-Flow-Bench
Table 2. Information of projects used SWE-Flow-Bench.
Project
Functionality
Stars
License
Last Commit
Commit Hash
Date and Time
8.8k
Apache-2.0
2024.11.22
1d70d00
cryptography
Cryptography
6,8k
Apache-2.0 & BSD-3-Clause
2025.01.00 ...600000
4a31bc3
gensim
Language Processing
15.8k
LGPL-2.1
2024.12.05
8b6b69c
jinja
Templating Engine
10.5k
BSD-3-Clause
2024.12.22
6aeab5d
librosa
Audio Processing
7.3k
2024. 11.27
24270be
marshmallow
Serialization
7.1k
MIT
2025. 10.00..06
7lab95a
pandas
Data Analysis
44.2k
BSD-3-Clause
2025 01 .04
8fbe6ac
Pillow
Image Processing
12.4k
MIT-CMU
2025.01.00
dfb368a
pydantic
Data Validation
21.9k
MIT
2025.01
59b35de
pylint
Code Analysis
5.4k
GPL-2.0
2025.01 3025.01.04
c21276f
seaborn
Data Visualization
12.7k
BSD-3-Clause
2024.12.16
8fc4051
transitions
Design Patterns
5.9k
MIT
2024.08.13
4d8d103
In this section, we present SWE-Flow-Bench. the dataset used in the SWE-Flow-Bench benchmarking framework.
B.1. Statistics of SWE-Flow-Bench
Table 3. Statistics of WEEE-Floo-Bence
Steps
# Files
# Functions
# Context Tokens
Patch Tokens
Dep. Depth
Difficulty
Project
Functionality
Full
Lite
Full
Lite
Full
Lite
Full
Lite
Full
Lite
Full
Lite
Easy
cryptography
arow
Date and Time
Cryptography
474
124
50
50
1.00
1.00
1.00
1.00
1.1
1.4
1.0
1.0
20,633
8,130
21,728
3,951
274
392
303
306
2.2
1.6
1.0
1.0
librosa
Audio Processing
90
50
1.01
1,00
2.1
1.2
12,642
12.727
1,832
1,274
2.3
1.4
Medium
marshmallow
Serialization
70
50
1.01
1.02
2.1
1.8
8,160
6,546
454
363
2.8
1.8
scaborn
Data Visualization
235
50
1.00
1.00
2.1
1.1
9,703
8,425
846
753
3.5
1.1
Pillow
Image Processing
259
50
1.02
1.00
2.3
1.2
11,261
8,017
627
430
4.2
1.3
gensim
Language Processing
223
50
1.06
1.08
3.0
1.5
9,154
9,359
1,032
626
4.1
1.6
pydantic
Data Validation
160
50
1.01
1.00
3.0
1.5
8,087
8,018
589
525
1.6
jinja
Templating Engine
50
1.09
1.10
4.2
4.7
7,234
7,243
642
698
5.3
Hard
transitions
Design Patterns
50
1.02
1.02
4.6
5.0
7.761
7,794
906
944
7.9
pylint
Code Analysis
1.05
23,76
pandas
Data Analysis
223
50
1.17
1.16
5.3
3.8
27,076
11,663
1090
813
2.8
SWE-Flow-Bende consists 2,020 development tasks spanning most popular Python software engineering projects, covering a diverse range of software engineering domains, including: Date and Time, Cryptography, Language Process- ing, Templating Engines, Audio Processing. Serialization, Data Analysis. Image Processing. Data Validation, Code Analysis, Data Visualization, Design Patterns. Table 3 provides detailed statistics these 12 software engineering tasks, and the metrics in the table are defined as follows:
Steps: The total number of development steps required complete the project.
# Files: The average number of files modified per development step.
# Functions: The average number of functions that need to be implemented per development step.
# Context Tokens: The average number of tokens in the contextual code files relevant to each development step.
Patch Tokens: The average number of tokens in the solution (patch) for each development step.
Dep. Depth: The average dependency depth, representing the number of function calls development step relies on.
14

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
We categorize the 12 software engineering projects into three difficulty levels,easy, medium, and hard, based the average number of functions that need to be implemented per development step:
Easy: Projects where the average number of functions per step is less than 2.
Medium: Projects where the average number of functions per step is between 2 and 3.
Hard: Projects where the average number of functions per step greater than
To facilitate efficient validation, SWE-Flow-Bende is divided into two splits: Full and Lite.
The Full split includes all 2,020 development tasks.
The Lite split contains only the first 50 development steps from each software project (or all available steps a project has fewer than 50 development steps), resulting in a total of 589 development tasks.
Since earlier steps in the development process tend to have shallower dependency depths, the Lite split presents lower level of difficulty compared to the Full split.
B.2. Comparison Between SWE-Flow- -enohh and Existing Software Engineering Benchmarks
Table 1 presents a comparison between SWE-Flow-Bench and existing software engineering benchmarks (Jimenez et al., 2023; Zhao et al., 2024), highlighting their similarities and differences.
SWE-Bench (Jimenez et al., 2023) primarily evaluates an agent's ability to resolve GitHub issues, focusing on patch-level code fixes rather than the entire development process. As result, it cannot systematically assess a code generation model's capabilities in architectural design, functionality expansion, and incremental development.
CommitO-Bench (Zhao et al., 2024), on the other hand, requires models to generate complete implementation in single attempt based on all available unit tests. This approach does not align with real-world software engineering practices, as it fails to measure a model's step-by-step code construction performance. Additionally, lacks interpretability, making difficult to analyze failure cases.
In contrast, SWE-Flow-Benn adopts a Test-Driven Development (TDD) approach, decomposing projects into multiple incremental steps. Each step guides code generation through minimal test cases, enabling more fine-grained evaluation of model's ss ability in code organization, architecture construction, and functionality expansion. Moreover, SWE-Flow-Beee -Bnchh better reflects real-world development workflows while providing greater interpretability, allowing for deeper analysis of model failures.
Overall, SWE-Flow- Flon-Benw-Bh offers more rigorous and systematic evaluation of code generation models, providing more precise insights for model optimization.
15

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
C. Content of SWE-Flow-synthesizee Data
C.1. Large Language Model Genearted Development Document
We conducted comparative analysis of gpt-4o (Hurst et al., 2024), claudc-3.5-somie (Anthropic, 2024), DeepSeek- (Liu et al., 2024a), and Qwen2.5- (ene::5-Codrr32---mmeeee Instruct (Hui et al., 2024) in terms of the quality of requirement documents generated based a given unit function. Our findings indicate that, under two-shot setting, the quality of the generated requirement documents remains largely consistent across these models. Given this observation, open-source model, for requirement document generation due to its accessibility and cost-effectiveness. The upper section of Figure 6 presents the content of target test function from the tiktoken project, while the lower section displays the requirement specification document generated by Ooven..--Coder-321B-- on the test function.
selected
based
Content of Target Test Function
def test tiktoken. gett get_eecodieeng (p50k "apggggpaea"""""""""""
s_rr__aai______ prssiccencode()>
assert enc. encode ("hello 'hello world') == [31373, 995] enc tiktoken, get net__encoing(("" (c1100k ct11kkk1_____
base')
assert enc. encode) 'hello world') == [15339, 1917] assert enc. enc.eec \x850
[220, 126, 227, 15]
LLM Generated Development Document
"Development Requirements"
*##
'GGoal 1: Provide Encoding Functionality for Multiple Models
#### "Feature Description" The system should provide encoding functionality that supports multiple tokenization models, including pp50__ base', and cc100k_baaal base'.
#*** "Required Functionality"
1. "EEncoding with 'p50k base' Model Implement functionality to encode the string 'hello world' using the 'p50k__ base' model and ensure the output matches [31373, 995].
2. "EEcooding with 'cl100k_ base' Model""
Implement functionality to encode the string 'hello world' using the 'cl100k_ base' model and ensure the output matches [15339, 1917J.
Implement functionality to encode the string 'xx850' using the 'cl100k base' model and ensure the output matches [220, 126, 227, 15).
'Methoods Functionality Overview"*
Method Name
Input Parameters OutpuutFunctionalit
tikloken.get gec.odinnnnnngggggeeoeeo..ee model name: str Return encoding model instance for the specified model name Return list of encoded token IDs based the text and the model instance
encode text: str
## * "Exeectd Outcomes**
1. The '500k_ base' encoding model should produce [31373, 995] when encoding the string 'hello world'.
base''
2. The cc1000 base' encoding model should produce [15339, 1917] when encoding the string hello world',
3. The tc1100k base' encoding model should produce [220, 126, 227, 15] when encading the string 'x8850'.
Figure Content target test function and the requirement document generated by Qwen2.5 5-Cooerr--BBBBB. 32B-e---eB-Instuue based
C.2. Large Language Model Genearted Doc-string
For doc-string generation, we adopt the same strategy as used for development document generation. Specifically, we employ 2-shot setting, prompting the on the content the Core Function. The upper part of Figure presents the content of a Core Function, while the lower part displays the doc-string generated by Qwen22.5Coddr-3222 32B-tnstuuct. This doc-string will be utilized for code skeletonization.
Coder-32B-Instruce model to generate the corresponding doc-string based
16

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Content of Target Core Function
@c aclassmethod
_iter_ from wwwfrfrooouorrroeeerlrrroooooole aaam__rrpen www.tthsphhhccccc (cl, urlpaths, down download_conffl ioneeng Generator[str, None, None]; if not isinstance(urllpatss list):
urlpaths = [urlpaths]
for urlpath
urlpaths:
if leriieelu kisfilelurlpath, sisdirrlpath elif kisdir(urlpath,
for dirpath, dirnames, filenames xwalk(urlpath, download config=down gttesddwwload_confffgg
dirnames [:] sorted [dirname for dirname in dirnames if not
dirname. wernamm.startswttttte. (ttrrtswit((".'' "))1)
(_))
continue
for filename in sorted(
if filename. "itrameestarrswith((..... "));
continue
yield xjoin(dirpath, filename)
else:
raise FileklotFoundError((rlltprtt (ur
LLM Generated Doc-string
Generates file paths from given URL paths, downloading and iterating through directories as necessary.
Args:
uripaths (Union[str. List(str]]]: A single URL path or list of URL paths to iterate over. download_come (DownioadConfig. optionaly: Configuration for downloading files. Defaults to None.
Yields:
file path from URL paths.
Raises:
FileNotFoundError: If a URL path is neither a file nor directory.
Figure Content of target core function the doc-string generated by Qween2.5-Coder-22--nnesse based on
stt/dattsett//eeturre/feeeut Synthesized Reference Patch
+++ 3rc/datasets/ httss///etuuuuteterrereeeeeeeeee @B -954,7 7 +954,21 21 @e
bool: 'True" if the object and its first elements (if it sequence) not 'None'
empty,
otherwise FFalse"
if obj is None:
return False
elif isinstance(obj, (list, tuple)) and (schema is isinstance(schema, (list, tuple, LargeList, Sequence))):
if len(ob)) 0:
if schema is None:
pass elif isinstance(schema, (list, tuple)):
schema schema [0]
else:
schema = schema. feature
return htwareooooonnnooooooeeeeeeeeeeee [0], schema)
else:
return False
else:
return
def get get_nested www__nettee_tppeeschheeeeee FeatureType) -> DatDayepype:
Musal
Figure An example a reference patch in synthetic data from sweflow.
17

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
D. Fine-tuning Language Models
D.I. Training Dataset
Patch Format
Replace Format
7.5
Percentage (%)
6.0
4.5
3.0
10k
11k
126
12k
21k
22k
21
Sequence length
Figure 9. The distribution of total tokens
training data generation. Here, we utilize the synthesized dataset to fine-tune the Qwen2.5-Coderr -o---oddd-----sssrueee Instruct (Hui et al., 2024) model. The training dataset consists of two data formats: Patch and Replace To ensure efficient training, we filter out samples exceeding 32k tokens. The final sequence length distribution of the training dataset is illustrated Figure 9.
In Section 3.7, we described the process of SWWE-Flo-ssyttesst
D.2, Training Parameters
Table 4. Fine-tuning parameters.
Parameter
Max Seq-len
Batch Size
Training Steps
Warmup Steps
Learning Rate
Min LR
LR Decay
Value
32,768
1024
32
6
7e-6
7e-7
Linear
For the training framework, we employ Megatron- (Shoeybi et al., 2019), with detailed training parameters provided in Table D.2. The entire training process can be completed within hours using 128 H800 GPUs.
18

---

SWE-Flow: Synthesizing Software Engincering Data in Test-Driven Manner
E. Detailed Evaluation Results
E.1. Evaluation Results of Large Language Models
Table Evaluation results of language models on SWE-Flow-Bench (Lite).
cryptography
librosa
marshmallow
seaborn
Pillow
Model
replace
patch
replace
patch
replace
patch
replace
patch
replace
patch
replace
patch
01 11-previeww2222--9--4
40.0%
32.0%
54.0%
22.0%
24.0%
24.0%
36.0%
18.0%
26.0%
16.0%
36.0%.
20.0%
o1 01-mini-20244-09-1
20.0%
8.0%
8.0%
28.0%
8.0%
14.0%
14.0%
14.0%
2.0%
6.0%
16.0%
16.0%
4p1-40-2022408-066
6.0%
26.0%
6.0%
28.0%
6.0%
14.0%
2.0%
18.0%
2.0%
18.0%
6.0%
24.0%
(auub--.5--0mme--222400000
66.0%
48.0%
24.0%
58.0%
22.0%
34.0%
16.0%
58.0%
12.0%
26.0%
26.0%
60.0%
1amma-3114405-Imm
38.0%
12.0%
34.0%
14.0%
30.0%
8.0%
30.0%%
18.0%
34.0%
6.0%
40.0%
8.0%
-Jumm-3.3-700-Insttm
10.0%
18.0%
30.0%
10.0%
20.0%
12.0%
12.0%
8.0%
18.0%
8.0%
28.0%
6.0%
DeepSeek-RI
70.0%
14.0%
48.0%
20.0%
36.0%
20.0%
44.0%
30.0%
34.0%
12.0%
58.0%
38.0%
DeepSeek-V3
68.0%
0.0%
62.0%
6.0%
42.0%
10.0%
48.0%
4.0%
42.0%
2.0%
60.0%
18.0%
DeepSeek-Coode Coder-VV.5-Inssttu
60.0%
14.0%
62.0%
16.0%
36.0%
28.0%
38.0%
16.0%
34.0%
8.0%
52.0%
18.0%
Qwen-2.5 725-street
22.0%
6.0%
2.0%
8.0%
0.0%
8.0%
0.0%
4.0%
0.0%
6.0%
2.0%
6.0%
Owen-2.5-Coderr. Inntstrctt
46.0%
30.0%
56.0%
14.0%
22.0%
8.0%
44.0%
16.0%
28.0%
14.0%
50.0%
12.0%
WV-Coder-32B-IIstree
76.0%
64.0%
78.0%
44.0%
34.0%
26.0%
54.0%
38.0%
34.0%
22.0%
50.0%
42.0%
gensim
pydantic
jinja
transitions
pylint
pandas
Model
replace
palch
replace
patch
replace
patch
replace
patch
replace
patch
replace
palch
ol-previcw-22024-09112
32.0%
24.0%
26.0%
18.0%
24.0%
14.0%
10.0%
10.0%
20.5%
17.9%
12.2%
10.2%
ol-mini-20244009-12
8.0%
8.0%
8.0%
8.0%
2.0%
12.0%
6.0%
6.0%
10.3%
15.4%
8.2%
12.2%
gpt-4o-2024408-066
4.0%
16.0%
6.0%
26.0%
2.0%
18.0%
14.0%
6.0%
2.6%
15.4%
0.0%
10.2%
22.0%
36.0%
20.0%
42.0%
12.0%
30,0%
26.0%
20.0%
7.7%
30.8%
14.3%
28.6
1:mm--3.-405B-Immse
44.0%
10.0%
34.0%
16.0%
16.0%
8.0%
16.0%
6.0%
23.11%
7.7%
10.2%
6.1%
JJama-3.3-70B-nnstuu
22.0%
10.0%
20.0%
10.0%
6.0%
2.0%
12.0%
6.0%
17.9%
7.7%
8.2%
6.1%
DeepSeek-RI
50.0%
24.0%
42.0%
24.0%
28.0%
12.0%
18.0%
10.0%
15.4%
10.3%
24.5
8.2%
DeepSeck-V3
42.0%
4.0%
42.0%
10.0%
38.0%
2.0%
14.0%
6.0%
17.9%
10.3%
24.5%
6.1%
40.0%
14.0%
38.0%
18.0%
32.0%
10.0%
14.0%
6.0%
23.1%
20.5%
22.4%
10.2%
Dwen-22.-72B--Isttree
4.0%
4.0%
2.0%
12.0%
2.0%
2.0%
8.0%
6.0%
2.6%
2.6%
4.1%
6.1%
Qwen-2.5-Coder-22B-mme -nnstruucttt
38.0%
12.0%
42.0%
14.0%
26.0%
10.0%
4.0%
4.0%
15.4%
10.3%
16.3%
8.2%
3V-Conler32B-IImttrree
50.0%
34.0%
50.0%
36.0%
38.0%
24.0%
20.0%
28.0%
25.6%
17.9%
16.3%
14.3%
During the evaluation of language models, we observed that the generated patches were almost entirely incompatible with system tools such as Linux's 's patch utility, making impossible to apply them directly to the codebase. While the generated patches generally contain the correct modifications, they often fail to accurately define the contextual modification range. Due to this limitation, we apply post-processing step, converting the generated patches into the replace format. We then use this replace-based approach to modify the codebase more effectively.
Table 5 presents the execution pass rates of 12 mainstream LLMs on SWE-Flow-Bench (Lite). The results indicate that, with the exception of -..ude-3.--oonet, all other models perform poorly on this benchmark. Furthermore, -laude-3.5-sonne demonstrates significantly higher accuracy in the patch format compared to the replace format. This suggests that its training data likely contains a substantial amount of patch format software engineering data. In contrast, SF-Coder-32B. Instruct, data, achieves significant performance improvements on these software
which is fine-tuned on SWE-Flow-sytthen engineering tasks. It consistently outperforms other models in both the replace and patch formats, reaching -tate-of-the-aat performance levels.
E.2. Evaluation Results of Agents
For the evaluation of agents, we employed the OpenHands (Wang et al., 2024a) framework with its default configuration, setting the maximum iteration limit to 30. Table 6 presents the performance of different language models integrated with OpenHands on SWE-Flow-Bench (Lite). The results indicate that 3auude-3.5-some consistently outperforms all other struggles to successfully complete complex software engineering development
LLMs. However, even tasks, as shown in the lower section of the table.
19

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
Table Evaluation results of OpenHaadds with language models SWE-Flow-Bench (Lite).
Model
arrow
cryptography
librosa
marshmallow
seaborn
Pillow
Acc.
EV
Acc.
EV
Acc.
Acc.
EV
Acc.
EV
Acc.
EV
Sauude-3.5-onnnee222241000
80,0%
12.12
98.0%
15.29
72.0%
10.72
92.0%
13.95
82.0%
12.4
90.0%
14.34
gpt-4o-2024-08-00
42.0%
6.03
82.0%
12.31
42.0%
6.18
30.0%
4.41
42.0%
6.07
70.0%
10.72
deepseek-chat
14.0%
2.34
22.0%
3,65
32.0%
4.96
10.0%
1.65
76.0%
11.42
12.0%
2.05
Owen2.5-Coder-330 32s-Instruct
14.0%
1.99
18.0%
2.52
26.0%
3.81
10.0%
1.44
8.0%
1.14
12.0%
1.72
gensim
pydantic
jinja
transitions
pylint
pandas
Model
Acc.
EV
Acc.
FV
Acc.
EV
Acc.
EV
Acc.
FV
Acc.
FV
Souude-..5-soneet-..
68.0%
10.37
50.0 %
7.41
58.0 %
8.51
8.0%
1.14
49.0%
7.52
48.0%
7.0
2pt-4o-2024-08-06
30.0%
4.4
12.0%
1.73
28.0%
4.1
8.0%
1.12
15.0%
2.18
6.0%
0.86
doepseck-chat
12.0%
1.9
12.0%
1,89
12.0%
1.88
2.0%
0.29
31.0%
4.62
6.0%
1.1
2oan225-Cooder332-Immttte
18.0%
2.56
12.0%
1.7
4.0%
0.57
0.0%
0.0
8.0%
1.18
4.0%
0.57
20

---

SWE-Flow: Synthesizing Software Engincering Data in Test-Driven Manner
F. Limitations and Future Work
F.. Limitations of sweflow
While SWE-Flow provides highly accurate and comprehensive function dependency analysis for synchronous programs, has limitations in handling asynchronous execution and multi pallti-proeessseee applications.
Challenges in Asynchronous Program Analysis. SWE-Flow is designed to track function dependencies sequential ex- ecution flows. However, in asynchronous programs that rely on event loops, coroutine scheduling, and callback mechanisms Python's 's asyncio), exccution order is dynamic and non-deterministic. This makes it difficult to precisely reconstruct function call relationships, potentially leading to incomplete or imprecise dependency graphs in such cases.
Incomplete Dependency Tracking in Multi-Process Applications. In multi-process applications, function calls occur across independent memory spaces and communicate via inter-process communication (IPC) mechanisms such as message queues, shared memory, or sockets. Since SWE-Flow currently operates within a single process scope, it cannot fully capture function dependencies that span multiple processes, limiting its effectiveness for distributed execution analysis.
E.2. Future Work
Synthesizing More Challenging Data. In this study, we employ the SWE-Flow framework to decompose a complex software engineering task into multiple simpler subtasks, each requiring only a minimal amount of incremental code development. Owing to the flexibility of SWE-Flow, we can also synthesize more challenging software engineering tasks, For instance, by merging multiple consecutive tasks within the development schedule generated by SWE-Flow-Schedune into single new task and subsequently applying the same post-processing steps for data synthesis, demanding task. In the extreme case, where all development tasks are merged into single task, this results in a scenario where the entire project must be developed from scratch.
construct more
Enhancing Reinforcement Learning. Recently, a surge of reasoning awwaee aware models, such as ol (Jaech et al., 2024), DeepSeek-R1 (Guo et al., 2025), and QwQ (Yang et al., 2024a), have emerged, all of which are trained using reinforcement learning. Reward feedback a crucial component of reinforcement learning (Ouyang et al., 2022a); however, there remains a significant lack of large-scale, verifiable software engineering datasets. contrast, the data synthesized by our SWE- Flow framework inherently possesses verifiability, as the correctness of model-generated content can be directly assessed by executing the generated code within the corresponding containerized environment. We believe that the SWE-Flow framework will significantly facilitate the future reinforcement learning training of code generation models.
Enhancing Pre-training. The pretraining dataset directly determines the fundamental capabilities of a large language model (LLM). However, due to the scarcity of verifiable software engincering data, current open-source LLMs suffer from severe lack of high-quality software engineering data in their pre-training corpora. By leveraging a vast number of GitHub projects with continuous integration (CI) enabled environments, we can already synthesize an extensive amount of verifiable training data. Furthermore, ongoing research (Zhang et al., 2025) is exploring automated dependency environment construction for software projects, which will further expand the scale of data that can be synthesized using the SWE-Flow framework. We believe that incorporating large volume of synthetic, verifiable software engineering data into the pre-training corpus of open-source models will significantly enhance the foundational capabilities of LLMs in the domain of code generation and software development.
21

---

SWE-Flow: Synthesizing Software Engincering Data in Test-Driven Manner
G. Runtime Dependency Graph
Figures 10, 11, 12, and 13 illustrate examples Runtime Dependency Graphs (RDGs) constructed by SWE-Flow from Hugging Face Datasets library.
testa/test til.py-33-tess datasels mode enabled
19gwtest Isspec offline
arodottasntstutllsst lio utils. stils..y=-313 31-fsss pec_get
smoldntasets/utilat de utils po-1ls-p pec
testshutils.p pyttshttils.p--35500
tests/atiis py-401
sToddotasssshtilst utils. 2y-285m offline mode is enabled
Figure 10. An Runtime Dependency Graph (RDG) instance form datasets project.
Jenaller Bay
Hetiuted Besotia
mndpoondron d-lenal
Pr. JM
colamondiimn belesce ceiree
moknneddemn Ieretnit Jien
Figure 11. An Runtime Dependency Graph (RDG) instance form datasets project.
22

---

SWE-Flow: Synthesizing Software Engineering Data in Test-Driven Manner
toko
intratatiie
Seetra I 43
1.00 de 14--b
waltoridbsiim
yollh-ar EnC
Madlna pedtin
Figure 12. An Runtime Dependency Graph (RDG) instance form datasets project.
QEUR AIR
OUjIWINNE prlcle
pelteiatiee
minaniaalko bittle casl lick
yells aroi
Figure 13, An Runtime Dependency Graph (RDG) instance form datasets project.
23


Reviews:
[{"review_id": "PHPuCDOktJ", "reviewer": "Reviewer_qTpi", "summary": "The paper introduces UnitFlow, a novel framework for synthesizing test-driven software engineering data. Unlike prior datasets that rely on human-submitted issues, UnitFlow generates incremental development steps directly from unit tests. The framework constructs a Runtime Dependency Graph (RDG) to capture function interactions, enabling structured development tasks that mirror real-world iterative coding practices. Evaluation of state-of-the-art LLMs and agents, demonstrating that current models struggle with complex software engineering workflows.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe deeply appreciate your thoughtful review and your recognition of our contributions. Below, we provide point-by-point responses to your concerns and suggestions.\n\n---\n\n> **Concern A1:** The evaluation does not compare against human-written commit sequences, which could serve as a valuable real-world baseline.\n\nGiven that the unitflow-bench benchmark comprises more than 2,000 test cases, conducting experiments with human-written commits is quite time-intensive. Regrettably, we were unable to complete these experiments within the rebuttal timeframe. Nonetheless, we are committed to performing the experiments and will include the results in the revised version of our paper.\n\n\n> **Concern A2:** The study does not report how often models generate nearly correct but slightly misformatted code, which may impact pass rates.\n\nWe calculated the accuracy of the solution format generated by each model and present the results in the table below.\n\n|Model|Empty Patch|Empty Replace|\n|-|:-:|:-:|\n|claude3.5-sonnet-1022|0.00%|0.00%|\n|DeepSeek-Coder-V2-Instruct-0724|2.00%|1.67%|\n|DeepSeek-R1|1.67%|1.67%|\n|DeepSeek-V3|4.01%|6.66%|\n|gpt-4o-2024-08-06|0.00%|0.00%|\n|Llama-3.1-405B-Instruct|6.67%|8.03%|\n|Llama-3.3-70B-Instruct|2.04%|1.04%|\n|o1-mini-2024-09-12|5.34%|3.33%|\n|Qwen2.5-72B-Instruct|3.51%|0.59%|\n|Qwen2.5-Coder-32B-Instruct|2.71%|0.33%|\n|UF-Coder-32B-Instruct|4.17%|1.38%|\n\n\n> **Concern A3:** The paper lacks statistical significance tests for performance differences, are observed differences robust under variations in prompts or datasets?\n\nIn the evaluation, we standardized only the format of the prompts, which contained solely the essential formatting instructions. Since the actual content of the prompts changed significantly based on the contextual information of each test instance, the robustness of the experimental results is preserved.\n\n\n> **Concern A4:** No qualitative analysis of whether models learn good coding practices or just pattern-match training data.\n\nOur UF-Coder-32B-Instruct model is trained based on Qwen2.5-Coder-32B-Instruct. As shown in the error format statistics table above, the proportion of formatting errors generated by UF-Coder-32B-Instruct is comparable to that of Qwen2.5-Coder-32B-Instruct — in fact, the latter even exhibits a slightly lower error rate. However, the accuracy of UF-Coder-32B-Instruct is significantly higher than that of Qwen2.5-Coder-32B-Instruct. This suggests that UF-Coder-32B-Instruct has not merely memorized patterns from the training data, but has indeed improved its capability in software development tasks.\n\n\n> **Concern A5:** How does UnitFlow compare to human-written commit sequences in terms of task structure and complexity?\n\nCompared to human-written commit sequences, which often vary in granularity and mix different types of changes, UnitFlow provides a more structured and goal-driven development trajectory. By aligning each commit with the satisfaction of a specific unit test, UnitFlow enforces a clear functional decomposition and promotes modular, incremental progress. Despite being automatically generated, the commit sequences can capture meaningful task dependencies and span multiple components, reflecting a level of complexity comparable to real-world development. This makes UnitFlow a valuable tool for generating realistic and reproducible development workflows, particularly in settings where systematic task structure and functional clarity are desired.\n\n\n> **Concern A6:** Typos and Suggestions\n\nWe will incorporate these improvements in the revised version.\n\n---\nWe sincerely appreciate your thoughtful and positive review. Should you have any further questions or suggestions, we would be glad to provide additional clarification.", "claims_and_evidence": "The core claims are well-supported by experimental results.\n\nPotential Limitations:\n\n- Comparison with other benchmarks is limited. The paper discusses SWE-Bench but does not provide a direct side-by-side comparison in terms of realism or task difficulty.\n- The robustness of UnitFlow-generated data against real-world code review practices is unclear.", "methods_and_evaluation_criteria": "++ The Runtime Dependency Graph (RDG) effectively captures function dependencies.\n\n++ Using unit test execution traces instead of static analysis improves accuracy in identifying function interactions.\n\n++ Benchmarking both LLMs and agents provides a systematic and well-defined evaluation.\n\n-- The evaluation does not compare against human-written commit sequences, which could serve as a valuable real-world baseline.\n\n-- The study does not report how often models generate nearly correct but slightly misformatted code, which may impact pass rates.", "theoretical_claims": "Not applicable (no formal proofs or theoretical claims in the paper).", "experimental_designs_or_analyses": "++ The choice of models is reasonable, covering OpenAI, Google, and specialized models.\n++ The benchmarking methodology is sound.\n-- The paper lacks statistical significance tests for performance differences—are observed differences robust under variations in prompts or datasets?", "relation_to_broader_scientific_literature": "- Prior benchmarks like HumanEval, MBPP, and SWE-Bench focus on single-step or static QA, whereas UnitFlow emphasizes incremental development.\n- Unlike SWE-Bench, which extracts tasks from GitHub issues, UnitFlow synthesizes development tasks, making it more scalable and controllable.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n- Automated and scalable dataset generation.\n- Benchmark evaluates LLMs in an iterative development setting, making it more realistic than static function-based tasks.\n- Fine-tuned LLM outperforms baselines, validating the effectiveness of the dataset.\n\nWeaknesses:\n\n- No qualitative analysis of whether models learn good coding practices or just pattern-match training data.", "other_comments_or_suggestions": "- Section 4.1, line 319: Reference to Figure 5 appears incorrect.\n- Section 5.2, line 370: The term \"unitflow-bench-lite\" should be introduced earlier for clarity.", "questions_for_authors": "1. Have you analyzed how often models generate near-correct solutions with minor formatting issues (e.g., missing imports, minor syntax mistakes)?\n2. How does UnitFlow compare to human-written commit sequences in terms of task structure and complexity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DehOeKHms8", "reviewer": "Reviewer_YdKw", "summary": "This paper proposes a data synthesis framework UnitFlow that leverages Test-Driven Development to automatically generate high-quality, structured, and verifiable training data for LLMs in software engineering. It constructs a Runtime Dependency Graph (RDG) from unit tests to capture function interactions and generates a step-by-step development schedule. For each step, UnitFlow produces a partial codebase, a requirement document based on unit tests, and a reference solution. Using UnitFlow, the authors synthesized a dataset of 16k training instances and 2k test instances, which they used to fine-tune the Qwen2.5-Coder-32B-Instruct model, resulting in the UF-Coder-32B-Instruct model. Experiments on the UnitFlow-Eval benchmark demonstrated significant improvements in the model's ability to perform TDD-based coding tasks.\n\n## update after rebuttal\nI appreciate the rebuttal by authors. My concern has been addressed and I would like to keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe are grateful for your valuable suggestions. Our detailed responses to the concerns are provided below.\n\n---\n\n> **Concern B1:** The evaluation of the proposed method is limited to the UnitFlow-Eval benchmark, which shares the same distribution as the training data. The authors did not conduct experiments on other software development benchmarks such as SWE-Bench. This limitation may affect the assessment of the method's generalizability across different software development contexts and tasks.\n\nWe agree that evaluating the generalizability of the proposed method beyond the UnitFlow-Eval benchmark is important. In addition to our main evaluation on UnitFlow-Eval, we conducted further experiments using the **SWE-Bench-Verified** benchmark, which differs significantly from our training data in both distribution and task format. Specifically, we fine-tuned **Qwen2.5-Coder-32B-Instruct** with UnitFlow synthetic data, and evaluated it using the **Agentless** framework. The results are as follows, reported as the average over three runs with standard deviation:\n\n| **Model** | **SWE-Bench-Verified Accuracy** |\n| - | :-: |\n| Qwen2.5-Coder-32B-Instruct | 33.79% ± 0.32% |\n| +Ours (fine-tuned with UnitFlow data) | **35.27%** ± 0.26% |\n\nThis improvement of **+1.48%** indicates that our method not only improves performance on in-distribution tasks but also enhances the model’s ability to generalize to real-world software engineering tasks under different settings.\n\nWe appreciate the reviewer’s suggestion and will include these results in the revised version to strengthen the generalizability evaluation of our method.\n\n\n> **Concern B2:** While the approach demonstrates theoretical scalability, the volume of synthetic data and the experimental evaluation are confined to supervised fine-tuning.\n\nSince the initial submission, we have significantly scaled up the synthetic data generation process. Specifically, we have automated the construction of 6,008 executable repository images and used *UnitFlow* to synthesize over **200k** task instances. \n\nOur analysis suggests that supervised fine-tuning (SFT) benefits saturate  around **20k–30k** instances. Therefore, we plan to leverage the additional data in the **continued pretraining** phase, which contributes to improving general code understanding and task generalization.\n\nTo support further research and demonstrate practical scalability, **we will release all synthesized data and executable environments**.\n\n---\nPlease feel free to reach out if you have any further questions — we’d be happy to clarify. If you find our response addresses your concerns, we sincerely hope you would consider raising your score accordingly.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, but could be further improved.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The evaluation of the proposed method is limited to the UnitFlow-Eval benchmark, which shares the same distribution as the training data. The authors did not conduct experiments on other software development benchmarks such as SWE-Bench. This limitation may affect the assessment of the method's generalizability across different software development contexts and tasks.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Other Strengths: \n- Clear writing and organization.\n- Scalablility of the proposed method.\n\nOther Weaknesses: \n- While the approach demonstrates theoretical scalability, the volume of synthetic data and the experimental evaluation are confined to supervised fine-tuning.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please refer to Experimental Designs and above Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D63yfIIdjg", "reviewer": "Reviewer_ffV1", "summary": "This paper introduces UnitFlow, a novel data synthesis framework based on Test-Driven Development (TDD), which automatically generates software engineering data by inferring incremental development steps directly from unit tests. The framework constructs a Runtime Dependency Graph (RDG) to capture function interactions, enabling the generation of structured development schedules and verifiable TDD tasks. In addition, the authors created the UnitFlow-Eval benchmark by generating 16,061 training instances and 2,020 test instances from real-world GitHub projects. The paper demonstrates that fine-tuning a large language model (LLM) on this synthetic dataset significantly enhances performance in TDD-based programming.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe sincerely appreciate your thoughtful and constructive feedback. Please find our responses to the concerns below.\n\n---\n\n> **Concern C1:** Is the synthesized data valid and of high quality?\n\nWe fine-tuned the **Qwen2.5-Coder-32B-Instruct** model using data synthesized by UnitFlow, resulting in a substantial accuracy improvement on the UnitFlow-Bench benchmark. To further validate the effectiveness of UnitFlow-generated data, we conducted evaluations using the **Agentless** framework on the **SWE-Bench-Verified** dataset, reporting results as the average over three runs with standard deviation.\n\n| **Model** | **SWE-Bench-Verified Accuracy** |\n| - | :-: |\n| Qwen2.5-Coder-32B-Instruct | 33.79% ± 0.32% |\n| +Ours (fine-tuned with UnitFlow data) | **35.27%** ± 0.26% |\n\nThe consistent improvements observed on both **UnitFlow-Bench** and **SWE-Bench-Verified** demonstrate the practical utility and generalizability of the synthetic data produced by UnitFlow.\n\n> **Concern C2:** The model UF-Coder-32B-Instruct is actually a fine-tuned version via synthesized training data, so I am not sure if the performance improvement on UnitFlow-Bench-Lite can demonstrate the effectiveness of UnitFlow-synthesized data.\n\nAs mentioned above, we also fine-tuned **Qwen2.5-Coder-32B-Instruct** and evaluated it on **SWE-Bench-Verified**. The results provide further evidence of the effectiveness of the synthetic data generated by **UnitFlow**.\n\n> **Concern C3:** The paper did not consider the scenario of software evolution, where software keeps changing.\n\nWe are exploring ways to incorporate software evolution into our setting, but we have found it to be a challenging problem.\n\nWe constructed executable and verifiable environments for 1,276 repositories related to SWE tasks. However, since the environments were built based on the latest commit on the main branch, some environments became invalid when checking out historical commits—due to changes in the software’s structure or dependencies over time. As a result, out of the 24,961 instances covered by these repositories, only 1,242 instances could be successfully validated using the prebuilt environments.\n\n> **Concern C4:** Are there any differences between the criteria in the ‘Preparation of Test Environment and Verifiable Data Generation’?\n\nDuring the construction of executable verification environments, we use the **exit code of pytest** as an indicator of environment status. Some key exit codes are interpreted as follows:\n\n- **Exit code 0**: pytest ran successfully and all unit tests passed.\n- **Exit code 1**: pytest ran successfully but some unit tests failed.\n- **Exit code 3**: pytest failed to run, typically due to import errors caused by an incomplete or broken environment.\n\nIn the environment construction phase, we treat both **exit code 0 and 1** as indications of a **successfully built environment**. However, during the data synthesis phase, we retain **only instances where unit tests pass** (exit code 0), to ensure that each synthesized data point can be **verified within the current environment**. Instances that fail unit tests are discarded to maintain data quality.\n\n> **Concern C5:** “Among these projects, 12 projects are selected for testing, and the remaining 62 projects are used for training”. Did you select these projects for testing and training randomly or not? Any criteria for projects selection?\n\nThe **test projects** were selected based on **functional diversity**. Specifically, we chose 12 projects that serve **distinct purposes** (e.g., visiualization, data processing) to ensure broader coverage. The selection criteria also included **popularity (GitHub star count)** and **recent activity** (i.e., whether there were commits within the past six months).\n\nWe will clarify the project selection criteria more explicitly in the revised paper.\n\n> **Concern C6:** Does these “Core Functions” refer to both Target Core Function and Dependent Core Function?\n\nYes, your understanding is correct. We will clarify this in the revised version of the paper.\n\n> **Concern C7:** The word \"task\" is confusing. Does it mean an unit test? What is the granularity of a task?\n\nYes, by \"task\" here, we are referring to whether the given development task can be completed successfully and pass the corresponding unit tests.\n\n> **Concern C8:** Does the CFNs refer to both Target Core Function Node and Dependent Core Function Node? Any clarification?\n\nYes, your understanding is correct. We will clarify this in the revised version of the paper.\n\n> **Concern C9:** Typos\n\nWe sincerely appreciate your careful reading and for identifying the typos in **Figure 3** and **Figure 5**. These will be fixed in the revised version.\n\n---\nPlease feel free to reach out. We would be happy to clarify any remaining questions you may have. If you find our response addresses your concerns, we sincerely hope you would consider raising your score accordingly.", "claims_and_evidence": "Some claims are not supported by clear and convincing evidence. The paper could show/prove the validity and quality of the synthesized software engineering data. Is the synthesized data valid and of high quality?", "methods_and_evaluation_criteria": "Pass Rate and Efficiency Value are used to measure the performance of several different LLM on UnitFlow-Bench-Lite. The paper shows that the limitations of current large language models in handling practical software engineering challenges could be addressed by fine-tuning LLMs with the synthesized data. However, the authors only fine-tuned the Qwen2.5-Coder-32B-Instruct model and evaluated it on the UnitFlow-Bench-Lite test set. More experiments could be conducted.", "theoretical_claims": "The paper does not contain theoretical proofs and claims.", "experimental_designs_or_analyses": "The model UF-Coder-32B-Instruct is actually a fine-tuned version via synthesized training data, so I am not sure if the performance improvement on UnitFlow-Bench-Lite can demonstrate the effectiveness of UnitFlow-synthesized data.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "The paper did not consider the scenario of software evolution, where software (including its structure/dependency graph) keeps changing, such as those discussed in the following paper:\nDewu Zheng et al., Towards more realistic evaluation of LLM-based code generation: an experimental study and beyond\nhttps://arxiv.org/pdf/2406.06918.", "other_strengths_and_weaknesses": "The paper is generally well written.\n\nThe data and code are not publicly available (data link cannot be found in the paper)", "other_comments_or_suggestions": ". Figure 3 never get referred in the whole paper. \n\n. Page 8, line 398. “The evaluation results are presented in Figure 5 and Table 6 in the appendix”, Figure 5 is actually not in the appendix.\n\n. Page 6, line 308 and line 318. Line 308 “Finally, we obtain 74 projects that can pass all the unit tests in the installed test environment.” Line 318 “We only keep the unit tests that pass.” Line 318 implies that there are unit tests that cannot pass while line 308 claims that all unit tests can pass. Are there any differences between the criteria in the ‘Preparation of Test Environment’ and ‘Verifiable Data Generation’?\n\n. “Among these projects, 12 projects are selected for testing, and the remaining 62 projects are used for training”. Did you select these projects for testing and training randomly or not? Any criteria for projects selection?\n\n. Page 3, line 3 “UnitFlow removes the implementation of core functions covered by the current…” and Page 4, line 205 “that cover the same Core Function Nodes (CFNs).” Does these “Core Functions” refer to both Target Core Function and Dependent Core Function?\n\n. Page 7, line 317, “The pass rate is defined as the ratio of successfully completed tasks (those that pass unit tests)…” The word \"task\" is confusing. Does it mean an unit test? What is the granularity of a task?\n\n. Page 4, line 205 “we first merge all TTFNs that cover the same Core Function Nodes (CFNs)”. Does the CFNs refer to both Target Core Function Node and Dependent Core Function Node? Any clarification?", "questions_for_authors": "As the model UF-Coder-32B-Instruct is actually fine-tuned using synthesized training data, can the performance improvement on UnitFlow-Bench-Lite demonstrate the effectiveness of UnitFlow-synthesized data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7TtmdiL6ja", "reviewer": "Reviewer_Ep2m", "summary": "This paper proposes a TDD-driven data synthesis framework `UnitFlow`, which can\ngenerate data samples for incremental development tasks based on real-world\nGitHub project and unit tests. Based on this framework, this paper constructs a\npromising benchmark `UnitFlow-Eval`, which contains data samples from 74\nreal-world Python projects. Fine-tuning code models on this dataset can\neffectively enhance LLM performance in incremental development tasks. This paper\nhas practical value for the development of `AI for SE` and code models but still\nhas issues to be clarified.\n\n## update after rebuttal\n\nI have updated my review after the rebuttal. Main concerns are addressed so I raised the score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe are grateful for the reviewer’s valuable suggestions. Our detailed responses to the concerns are provided below.\n\n---\n> **Concern D1:** The paper would benefit from additional empirical evidence to substantiate the claimed improvements in LLM performance on incremental development tasks, such as comparisons with other fine-tuned models or evaluations on other benchmarks.\n\nWe fine-tuned **Qwen2.5-Coder-32B-Instruct** using the **UnitFlow synthetic dataset** and evaluated its performance on **SWE-Bench-Verified** using the **Agentless** framework. The results are as follows, reported as the average over three runs with standard deviation:\n\n| **Model** | **SWE-Bench-Verified Accuracy** |\n| - | :-: |\n| Qwen2.5-Coder-32B-Instruct | 33.79% ± 0.32% |\n| +Ours (fine-tuned with UnitFlow data) | **35.27%** ± 0.26% |\n\n**UnitFlow-Bench** evaluates models on incremental software engineering tasks, whereas **SWE-Bench-Verified** targets automated bug fixing. Improvements observed on both benchmarks demonstrate the robustness and effectiveness of our synthetic data generation approach.\n\n> **Concern D2:** It is still hard to demonstrate that the generated data aligns with real-world data.\n\nUnitFlow is explicitly designed to simulate real-world incremental software development, based on a detailed analysis of unit tests from real open-source projects.\n\nSpecifically, we:\n- Use existing unit tests to define realistic development goals,\n- Synthesize code changes that mirror typical developer behavior,\n- Ensure the process is traceable, reproducible, and guided by real testing practices.\n\nAll instances are validated in executable environments, ensuring both correctness and practical feasibility. In summary, the design, source, and validation process of UnitFlow strongly support its alignment with real-world development workflows.\n\n> **Concern D3:** User studies were not conducted to get a sense of the quality and practicality of the generated dataset.\n\nWhile user studies can be valuable for assessing perceived quality, our focus is on systematic, executable validation: each UnitFlow-generated instance is tested in a execution verifiable environment to ensure that:\n- the code runs correctly, and\n- the intended behavior is preserved or modified as expected.\n\nThis process provides a scalable, objective, and reproducible measure of data quality, which we believe is more precise and reliable than subjective user feedback, especially for software engineering tasks\n\n> **Concern D4:** The experiment cannot provide sufficient evidence for the claimed contributions. Only involving 74 projects in the Python programming language, limits the practical value of the benchmark.\n\nWe would like to clarify that the current scale of our dataset has significantly expanded.\n\nSince the initial submission, we have successfully constructed over 6,000 executable environments across diverse real-world Python projects. Based on these, we have synthesized over 200k validated data instances, making UnitFlow-Bench a substantially larger and more practical benchmark than initially reported.\n\nMoreover, models fine-tuned on UnitFlow-synthesized data demonstrate consistent performance gains on UnitFlow-Bench-Lite and SWE-Bench-Verified. These results highlight the effectiveness and generalizability of UnitFlow-generated data, further supporting the practical value and impact of the benchmark.\n\nTo support further research and demonstrate practical scalability, **we will release all synthetic data and executable environments**.\n\n> **Concern D5:** This paper uses LLMs to generate development documents and docstring in Lines 212 and 233. I wonder how authors deal with the hallucinations in the LLM generation.\n\nAs described in Section 3.4, we manually inspected samples from several models before scaling up generation. For tasks such as generating development instructions and docstrings, Qwen2.5-Coder-32B-Instruct performed consistently well, and we did not observe hallucinations in the sampled outputs, they were semantically aligned with the code or tests.\n\nAdditionally, all generated instances undergo executable validation: if the synthetic data fail runtime checks, they are automatically discarded. This ensures that only accurate and functionally correct data are retained, effectively filtering out hallucinated or inconsistent outputs.\n\n> **Concern D6:** Figure 5 not found in appendix.\n\nThe prompt format figure referenced in line 319 was accidentally left out during compilation. We will make sure it is included in the revised manuscript.\n\n---\nWe’re happy to clarify any remaining questions you may have. If you feel our response resolves your concerns, we would be grateful if you could consider raising your score.", "claims_and_evidence": "The authors provide the necessary preliminaries and definitions in Section 2 and\nexhibit detailed experimental results and benchmark information in the Appendix.\n\nHowever, the authors fail to provide sufficient evidence to support the claimed\ncontribution, namely, `effectively enhances LLM performance in incremental\ndevelopment tasks'. Specifically, the author only shows the improvement of the\nPass Rate after finetuning one single model on their dataset. It will better\nillustrate the practical significance and contribution of this benchmark if the\nauthors can provide some comparisons of the fine-tuning results of other models\n(even small models of 7b and 14b) or show the performance of the fine-tuned LLM\non other development tasks/datasets.", "methods_and_evaluation_criteria": "The paper itself proposes to generate a benchmark, which I think the synthesis method is promising and should be studied. It is still hard to demonstrate that the generated data aligns with real-world data.\n\nUser studies were not conducted to get a sense of the quality and practicality of the generated dataset.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The experiment cannot provide sufficient evidence for the claimed contributions.\n\nOnly involving 74 projects in the Python programming language, limits the practical value of the benchmark.", "relation_to_broader_scientific_literature": "This paper proposes a data synthesis framework, which is guided by the concept\nof TDD to synthesize code samples for incremental development tasks from\nopen-source repositories. It is related to the prior SWE-benchmark but is\ndifferent. This paper splits one project into incremental steps, in line with\nthe process of real-world software development. This paper has cited appropriate\nrelated works in Section 7 and Appendix and discussed differences between this\nwork and prior works.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Cannot find `Figure 5 in appendix` mentioned in the right part of line 319.  Figure 5 is on page 8 and shows the efficiency.", "other_comments_or_suggestions": "The axes in Figure 5 are not suitable. Its caption claims that each axis is a software engineering task, but it actually refers to a repository. Such a design seems to follow the design of the SWE benchmark (Figure 4 in their paper).\n\nHowever, UnitFlow-Eval and SWEbenchmark are different, and this paper breaks up a project into several incremental development tasks. In this case, simply evaluating the effectiveness of LLM code generation on a project is not appropriate. I think a more appropriate approach might be to summarize the types of tasks involved in incremental development. For example, some incremental development solves bugs in the code, while some supplement new functions. They are different tasks with different targets.\n\nThe authors may consider summarizing these incremental development tasks and then designing new axes (each one is an incremental development task). Based on such classification, the evaluation of the pass rate and efficiency value of different LLMs will be more in line with the needs of incremental development scenarios. New axes can also allow this benchmark to measure the capabilities of LLM in incremental development.", "questions_for_authors": "This paper uses LLMs to generate development documents and docstring in Lines 212 and 233. I wonder how authors deal with the hallucinations in the LLM generation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Meta-Review of “UnitFlow: Synthesizing Software Engineering Data in a Test-Driven Manner”**

**Summary of the Paper**  
This paper introduces **UnitFlow**, a novel framework designed to synthesize step-by-step software development tasks based on test-driven development (TDD). The approach departs from previous task creation approaches that rely on human-submitted issues or single-function tasks. Instead, UnitFlow automatically constructs a **Runtime Dependency Graph (RDG)** by running unit tests and capturing function interactions, then “skeletonizes” partial code bases to reflect incremental development steps. From these steps, the authors create the **UnitFlow-Eval** benchmark—over 16,000 training instances and 2,000 test instances—aimed at measuring and improving LLM performance in TDD-style software engineering. The experiments show that fine-tuning an open-source large language model (Qwen2.5-Coder-32B-Instruct) on the synthesized dataset yields notable gains in pass rates for test-driven development tasks.

---

## Strengths

1. **TDD-Based Data Synthesis**  
   Reviewers generally agreed that using unit tests to guide data generation is a worthwhile idea. By using *executed* tests as the driver for incremental code construction, the paper sidesteps the noise and variability of human commits and issue reports, while still maintaining verifiability and clarity on intended functionality.

2. **Executable, Verifiable Tasks**  
   Each synthetic task is validated in a containerized environment, ensuring correctness checks through real unit tests. This explicit requirement that tasks pass or fail in a fully functional environment strengthens the benchmark.

3. **Demonstrated Performance Gains**  
   Fine-tuning a code LLM on the UnitFlow dataset results in improved performance on the in-domain **UnitFlow-Eval** task. Authors rebuttals show small accuracy improvement in the out-of-domain **SWE-Bench-Verified** task as well. This indicates that UnitFlow data may transfer beyond its own test distribution, possibly addressing a common reviewer request about generalizability.

4. **Scalable Data Generation**  
   The authors show that, after the initial submission, they were able to scale up UnitFlow to cover over 6,000 Python repositories, synthesizing more than 200,000 validated tasks. This indicates the framework’s potential to produce large-scale training corpora for code-centric LLMs.

---

## Weaknesses and Limitations

1. **Clarity on Node Removal and Typographical Issues**  
   The initial manuscript confused TTFNs (test-related nodes) and TCFNs (core function nodes) in a few places, causing reviewers and the area chair to question exactly which functions are preserved or stripped from the code base. The authors clarified that only core function implementations (TCFNs, DCFNs) are removed, while test functions remain intact. A revision is needed to fix the naming/terminology errors and ensure consistency between Figure 1 and the text.

2. **Experimental Scope and Real-World Comparison**  
   Multiple reviewers noted the limited experiments:  
   - The paper primarily focuses on **UnitFlow-Eval** which shares the same distribution as the training data. There are limited side-by-side comparisons. The rebuttal added data on SWE-Bench-Verified.  
   - No direct human-commit or human-developer “baselines” are measured.
   While the rebuttal mitigates this somewhat by testing on SWE-Bench-Verified, additional data points or real commit sequences would strengthen claims about real-world applicability.

3. **Focus on Python Projects**  
   Although the authors mention ongoing scaling efforts, the current benchmark remains Python-centric (74 initial repositories in the main experiments, or 6,000 in their scaled version). Cross-language generalization and broader coverage would be needed to show approach’s applicability to other languages.

4. **Incremental Development Complexity**  
   Several reviewers suggested that real-world incremental coding can be more convoluted than the TDD-inspired steps extracted by UnitFlow. For instance, code can shift across directories, library versions can break older commits, and developer commit habits can be ad hoc. While the authors describe “skeletonization” plus function removal, the real-world unpredictability is missing. Addressing genuine software evolution scenarios remains a future challenge.

5. **Limited Agent Evaluations**  
   Though the paper shows how UnitFlow-Eval can be used with agents, only OpenHands is tested. More comprehensive agent evaluations might be necessary.

---

## Recommendation

All four reviewers converged on acceptance or borderline-acceptance.
I recommend acceptance.
Decision: Accept (poster)