EpiCoder: Encompassing Diversity and Complexity in Code Generation
Authors: Yaoxiang Wang
Haoling Li
Xin Zhang
Jie Wu
Xiao Liu
Wenxiang Hu
Zhongxin Guo
Yangyu Huang
Ying Xin
Yujiu Yang
Jinsong Su
Qi Chen
Scarlett Li
Venue: ICML 2025 poster (2025)
Keywords: large language model
code generation
code
data synthesis
Primary area: deep_learning->large_language_models
TL;DR: This paper introduces a novel feature tree-based synthesis framework for generating diverse and complex code instruction data, and presents the EpiCoder series that achieve SOTA in both function and file-level code generation tasks.

Abstract: Existing methods for code generation use code snippets as seed data, restricting the complexity and diversity of the synthesized data. In this paper,  we introduce a novel feature tree-based synthesis framework, which revolves around hierarchical code features derived from high-level abstractions of code. The feature tree is constructed from raw data and refined iteratively to increase the quantity and diversity of the extracted features, which captures and recognizes more complex patterns and relationships within the code. By adjusting the depth and breadth of the sampled subtrees, our framework provides precise control over the complexity of the generated code, enabling functionalities that range from function-level operations to multi-file scenarios. We fine-tuned widely-used base models to obtain EpiCoder series, achieving state-of-the-art performance on multiple benchmarks at both the function and file levels. In particular, empirical evidence indicates that our approach shows significant potential in the synthesizing of repository-level code data. Our code and data are publicly available.

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Yaoxiang Wang" 12*1 Haoling Li3 Xin Zhang Jie Wu Xiao Liu" Wenxiang Hu Zhongxin Guo
Yangyu Huang* Ying Xin Yujiu Yang²
Jinsong Su
Qi Chen' Scarlett Li*
Abstract
MBPP (-)
Existing methods for code generation use code snippets as seed data, restricting the complexity and diversity of the synthesized data. In this paper, introduce novel feature tree-based synthesis framework, which revolves around hierarchical code features derived from high-level abstractions of code. The feature tree constructed from raw data and refined iteratively to increase the quan- tity and diversity of the extracted features, which captures and recognizes more complex patterns and relationships within the code. By adjusting the depth and breadth of the sampled subtrees, our framework provides precise control over the com- plexity of the generated code, enabling functional- ities that range from function-level operations to multi-file scenarios. We fine-tuned widely-used base models obtain EpiCoder series, achieving state-of-the-ane art performance multiple bench- marks at both the function and file levels. In particular, empirical evidence indicates that our approach shows significant potential in the syn- thesizing of repository-level code data. Our code and data are publicly available.
77.6
17.9
68.6
70.3
HumanEval (4)
XFileDep
72.9
$1.6
85.7
58.8
5012
BigCoeeBench
EvoEval
3
41.1 440.9
59.2
47.5
FullStackBench
LplCoder-Qwwn--B
OpenCoden 68-Instruct DeepSesk-Coder-6. Coder-6. 70-Instruct
Owen2.5 5ennni55-ooor---iiin -B-isst-ti
Figure 1: Benchmark performance of EpiCoder-Qwen-T7 Coder--B-BBsee BB-Base) and its counter-
(fine-tuned on parts. XFileDep is file-level code generation benehmark, all others function-level.
1. Introduction
Large Language Models (LLMs) (OpenAI, 2023; Zhang et al., 2022) have demonstrated significant potential in the field of code understanding and generation, particularly
through pre-training on large-scale code data (Sun al., 2024; Zhu et al., 2024). However, the latent code knowl- edge in these models is often underutilized without fine- tuning high-quality instruction data. Instruction tuning has emerged as a critical step in unlocking the full capa- bilities of code L.MMs (Wei et al., 2022), enabling a more precise alignment with user intent and improving the usabil- ity the model real-world scenarios.
Equal contribution 'This work is done during their in- ternships Microsoft 'School Informatics, Xiamen Uni- versity Keey Laboratory of Digital Protection and Intelli- gent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China 'Tsinghua University *Microsoft. Cor- respondence lo: Xin Zhang <xinzhang3e microsoft.com.. Yujiu Yang syang.yujuuu...tiinnnu....lll sunninanaaa.,,, Jinsong Su <jssu xmu.edu.cn>.
Despite recent advances, most existing methods for synthe- sizing high-quality instruction data rely code snippets as seed data (Chaudhary, 2023; Yu et al., 2023b; Wei et al., 2024b; Muennighoff et al., 2023). While code snippets demonstrate specific functionalities, they fail to capture the full range programming constructs, patterns, and interac- tions that are common in real-world programming scenarios.
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 hy the author(s).
thub. httpstttw..ccoiiccccsooo////pcoo and https://
gi thub.

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Additionally, due to the inherent rigidity of code snippets, it is difficult to rearrange recombine them generate new and diverse combinations. These limitations restrict the overall complexity and diversity of the synthesized data, highlighting the critical need for more structured representa- tions as seed data to overcome these constraints and address real-world programming challenges.
plexity. Building on these results, our approach further demonstrates strong potential in synthesizing highly com- plex repository-lveel data, as illustrated in Figure 5. To assess data complexity, we incorporated definitions from software engineering principles and employed the LLMM-ss- a-judge methodology. Moreover, we examined the diversity of datasets from a feature-based perspective to underscore the versatility and robustness of our framework.
Inspired by Abstract Syntax Tree (AST), we propose a fea- ture tree-based code data synthesis framework that revolves around hierarchical code features derived from high-level abstractions such as variable types function structures, and control flow. While AST captures the syntactic structure of code, our approach extends this idea by organizing features into a tree structure that captures the semantic relationships between code elements.
Our contributions are summarized as follows:
We propose a feature tree- based code synthesis frame- work that enables controllable synthesis of diverse and complex instruction data, ranging from function-level to file-level tasks.
We synthesize 433k instruction data and train EpiCoder. Our BereCoder-ween--e 7B achieves state-of-the-art per- formance among comparably sized models in multiple function-leved and file-level benchmarks, demonstrat- ing its capability to tackle programming problems of varying complexity.
Specifically, we extract features from the seed data and use hierarchical clustering to generate a tree structure demon- stration, starting from a feature set and proceeding bottom- up. This demonstration serves as a guideline for the LLM to directly extract tree structures from raw code data. To ensure comprehensive coverage of real-world scenarios, we enhance the diversity of features by iteratively expanding the tree structure both in breadth and in depth. Compared to methods that evolve based on individual code snippets or single instructions (Luo et al., 2023), our approach is more efficient and achieves broader coverage, as the tree struc- ture provides clear and organized directions for evolution. The resulting feature tree a large and hierarchical struc- ture, from which we can sample subtrees to generate code data. Our feature tree-based synthesis method offers two advantages: (1) Controllable Complexity: Our framework allows for adjusting the complexity of synthesized data by modifying the depth and breadth of subtrees. This enables the creation of code ranging from simple function-level tasks to comprehensive. multi-file solutions, (2) Targeted Learning: By adjusting the probability of sampling fea- tures, we can prioritize specific knowledge areas that are underrepresented, ensuring a more tailored and effective learning process for the L.MMs.
By conducting further analysis, we showcase the ad- vantages of our data in terms of complexity and diver- sity, as well as its potential for synthesizing large-scale repositories.
2. Methodology
In this section, we present our feature tree-based code gen- eration framework, which consists of three key steps: (1) Feature Tree Extraction (Section 2.1), where we construct the tree demonstration and extract feature trees from seed data; (2) Feature Tree Evolution (Section 2.2), where we iteratively expand the feature tree to enhance diversity and coverage; and (3) Feature Free-Based Code Generation (Section 2.3), where we use the evolved feature tree to gen- erate diverse code instruction data with varying complexity. An overview of the framework is illustrated in Figure 2.
We conduct extensive experiments to validate our fea- ture tree-based code data synthesis framework, training and DeepSeek-Codee Coder-6.7B-Ban to derive the EpiCoder series model. Our EpiCoder- Qwen- 7B trained on 380k function-level and 53k file-level data, respectively. Regarding the five function-level code gener- ation benchmarks, it has achieved state-of state-of-the-ard (SOTA) performance on average among models of similar scales. Notably, in the completion tasks of BigCodeBench- Hard, it outperforms --oee---oode--7--nnnn
2.1. Feature Tree Extraction
Qwen2.5-Coder-7B-
Inspired by Abstract Syntax Trees (AST), which represent the syntactic relationships in code, we use the LLM to ex- tract hierarchical representation that organizes key ele- ments of code into a tree structure to capture more funda- mental semantic relationships.
Raw Code Collection To ensure data diversity and com- prehensiveness, we obtain seed data from The Stack v2 (Lozhkov et al., 2024), a publicly available large-scale dataset widely used for pre- code LLMs. Follow- ing (Yu et al., 2023b), we apply the KCenterGreedy al- gorithm (Sener & Savarese, 2018) to select core set of
by a margin of 7.4%. Furthermore, on file-level benchmark XeileDep, Epi- Coder exhibits superior performance, underscoring its ca- pability to address programming problems of varying com-
2

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Feabure Tree Extraction
Code Generation
extract
merge
Il
reweight
pre-extract
feature trees
feature tree with frequency
contect
hierarchical cluster
Computotion
Arithmetic operations:
sample
Addition: 21,
extractdd feature
Multiplication:
cusstred feature
Subtroction: 13,
Computation
generate
Arithmetic Operation
Logical Opeatioo
feature set
tree demonstration
XOR Division
task
X-] code
Implementing on opplication
Project!
Feature Tree Evolution
README.mo
srcl
evolve
merge
COT env
main.py
utils.py
sample
python matplotiib scikit -earn
configi
settings.jon
lests!
test.py
refine
dopte
evolve breadtth
docker
Figure 2: Overview of our feature tree-based code generation framework, which consists of three steps: (a) Feature Tree Extraction, where we first extract the feature set to construct the tree structure demonstration and then extract the feature trees; (b) Feature Tree Evolution, where the feature tree is iteratively expanded in depth and breadth; and (c) Feature Tree-Based Code Generation, where the evolved feature tree is used to generate diverse code instruction data. A detailed example of feature evolution and code generation is shown in Appendix
diverse samples based on the code embeddings encoded roberta-large-vi (Liu al., 2019).
LLM perform hierarchical clustering, producing structure that represents the relationships among the features. We iteratively refine the tree structure demonstration by adjusting the clustering of features, ensuring a well-organized hierarchical relationship.
Tree Demonstration Construction To extract features from the seed data, we leverage a powerful large language model (LLM), specifically GPT-4o. Since the initial prompt provided to the LLM can significantly impact its response, we therefore propose an iterative method to opti- mize the demonstration tree structure within the prompt. As shown Figure (a), the construction the demonstration consists of the following two steps:
Feature Tree Extraction Using the refined demonstration of the tree structure, the LLM is tasked with extracting a tree-structured feature representation for each code snippet. These individual feature trees then merged into compre- hensive strueture that unifies the extracted features across all code snippets. During this process, we record the frequency of each node to reflect the distribution of features in the seed data. Since the seed data is derived from the pre-training data, this frequency can serve as an approximate measure of the knowledge distribution within the pre-trained model.
1. Feature Pre-Extraction With draft version of the prompt, the LLM is used to extract an initial set of features from the seed data, producing a collection of feature keywords.
2. Iterative Demonstration Generation: For each step, we take subset of the feature and prompt the
2.2. Feature Tree Evolution
To overcome the limitations in both diversity and quantity of features in the seed data, we expand the feature tree through
Unless otherwise specified, the strong LLM refers GPT-4o.
3

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
an evolutionary process. Compared to evolving directly from the seed code or instructions, this approach ensures broader feature coverage and improves efficiency by leverag- ing the tree's s structured representation, which provides clear and systematic directions for evolution. As illustrated in Fig- ure 2 (b), at each iteration, a subtree is sampled from the full feature tree. This subtree is then evolved by the LLM along two dimensions, depth and breadth, by adding finer-grained child nodes to existing nodes well as sibling nodes at the same hierarchical level. These newly evolved subtrees are then merged back into the overall structure, significantly enriching the feature space and facilitating the generation of more diverse and complex code. An example of the evolution of single subtree is provided in Appendix A.3.
Algorithm Feature Sampling for Task Generation
1: Input: Current root node R. frequeney library F, tem-
perature t, sample size S
2: Output: Selected feature set
3: selected_sed + 0
4: for s = 1 to do
C < get Sis_cciildeennFR R)
if = @ then
7: break
8: end if
9:
f: + F(i] for all
10:
Ps 5- Ejec for all i E
11: Compute P. using Equation (1) for all E C
cur_ 13: selected_set.addd set.add (cur_node)
12: cur node + sample Secplle__neee___eeeeee [p p2........))
One key challenge in feature evolution is to estimate the frequency of newly generated features. Unlike the feature extraction stage, where frequencies are calculated directly, the frequency of an evolved feature is estimated as the av- erage frequency of its siblings. This approach reasonably estimates the frequency of new features according to the existing feature distribution, ensuring that evolved features integrate seamlessly into the broader feature tree.
14: end for
15: Return selected_ set
feature tree according to the adjusted probability distribution. The sampling process is guided by a predefined shape of the subtree to sample, where we recursively apply Algorithm I to get the subtree. The LLM then uses this sampled subtree to generate code instruction data. By adjusting the depth and breadth of the subtree to sample, we can flexibly create tasks with varying complexity, thereby providing broad spectrum of coding challenges.
2.3. Feature Tree-Based Code Generation
Traditional code generation methods that rely solely on code snippets often produce outputs that closely resemble the original seed data, limiting diversity and complexity. We mitigate this issue by generating code based on feature trees and the process is outlined as follows.
Distribution Reweighting The previously recorded fea- ture frequencies partially reflect the distribution of natural data, helping to simulate real-world scenarios. However, some high-frequency but easy features, such as config and ini miaaiizee, do not require strong focus during instruction tuning. To address this issue, we adjust the probability distribution of a node's child features:
Content Generation Based the sampled subtree, the LLM proceeds to select compatible subset and then gener- ate a task and the corresponding code and execution envi- ronment. The solution code can range from a single func- tion to comprehensive multi-file project, depending on the task's complexity. By supporting multi-file solutions, this approach enables the generation of code that reflects realistic project structure and effectively captures cross- file information for problems requiring interactions across different components. As illustrated in Figure 6 of Ap- pendix 1, different files implement distinct functionalities and collectively form an integrated system through their interdependencies.
exp(logp:// p1 Ejec exc((eoolgg///
where ps represents the normalized original frequency of the child feature and p is the adjusted probability. As detailed in Algorithm 1, the summation applies to all child features j in the set C, which denotes the set of child nodes for the current parent node. A higher temperature value leads to a smoother distribution, allowing less dominant features a higher probability of being selected. To further enhance the diversity of the generated data, employed multiple temperature values during the data synthesis process for wider range of feature distributions.
Iterative Refinement To improve the quality of the gener- ated data, the solution code is accompanied by relevant test files, which are also generated by the LLM. These tests executed in an isolated environment, allowing us to identify and filter out incorrect solutions. Through an iterative de- bugging process, information from the execution results is used to guide the LLM to refine the solution, ensuring the correctness of the generated code.
Feature Sampling To generate diverse code instruction data, we sample a subtree of candidate features from the

---

Among models of the EpiCoder: Encompassing Diversity and Complexity in Code Generation
3. Experiment
Overall performance of the LLM on FullstackBench (önglish. Python).
61:28
208+
138+
68+
In this section, we introduce the details of the synthetic data (3.1) and evaluate the model's performance in code generation different levels. Specifically, Section 3.2, evaluate the model's ability using five function-level code generation benchmarks. In Section 3.3, we employ our meticulously crafted XFileDep benchmark to evaluate the
47.33
43.02
43.25
Performance (%)
41.1140:4
model's file-level code generation capabilities. We demon- strate the potential ability to generate particularly complex
Dasp www.eeteeeeeeeeeooo moon
Owmn Coder Indruct Areeee
St g Codeee BiBinrrrtttn 10
Own Sem0-5000 Cooe pemmrr m
Deppt Coweeoooeeeee....... wooteeeTinatttttte
COOLUN 238 intruet
code repositories in Section 3.4.
Yn Cotee w0we-9--oo Cha
EaColle 05 6:75
CoorOwnl
Despfree:
3.1. Implementation Details
Ownn
We utilized our pipeline to extract 5k features from the core set of 150k Python language files in The Stack V2. These features were then expanded through evolutionary meth- ods to 140k features. Then synthesized 380k function- level data samples and 53k file-level data samples based the features. We choose DeepSeek- SeepSSek-Codde-Baale Coder-Base-6, 6.7B (Guo et al., 2024) and Dwwen255-Coder-7B (Hui et al., 2024) as the base LLMs and obtain the EpiCoder-DS 6.7B and EpiCoder- Qwen-7B models after training. To ensure a fair comparison with other baselines, we incorporated the evol-codealpaca- v1 (applied the same filtering criteria as described in (Yu et al., 2023b)) dataset into the training of only the DeepSeek- Coder-Base Base-6.7B model. Additionally, for models trained solely on file-level data, we employed additional notation. We evaluated the models on benchmarks corresponding to their respective training levels.
Figure 3: Model performance across domains of Python in the English Subset of FullStackBench.
same size, EpiCoder--Qwen-B achieves the state-of-the-art (SOTA) average performance. The evaluation of the Epi- Coder series models on these benchmarks highlights their capability to solve challenge and complex programming problems. Moreover, the results also demonstrate that the feature tree-based code generation method can provide high- quality and diverse synthetic data tailored function-level programming problems.
3.3. File-level Generation
Deepseak-
Magicoder:s.DS D5-6.78
61.2
3.2. Function-level Generation
VasveCader-Ultra-633 6.77
59.4
OpensCodelnteppeeter..
EptCoder-DS:6.78
Many previous code LLMs have exhibited overfitting to specific benchmarks after fine-tuning, which somewhat con- strains their ability to generalize to other benchmarks. To prevent this, we employed five benchmarks for evaluation, with HumanEval (Chen et al., 2021), MBPP (Austin al., 2021), BigCodeBench (Zhuo et al., 2025), EvoEval (Xia et al., 2024) and FullStackBench (Liu et al., 2024), ensur- ing that they are generally broad, comprehensive, reliable, and decontaminated. For further clarity, all benchmarks are described in detail in Appendix B.I.
Sede@wen1.S-78.CMM
54,5
61.2
Model
EpiCoder-Qwen-70
OpenCCddr-SBBttnttuuus
57.1
Qwen2.5 Qwen2:5-Code-114B 1tt.---cooor11--nnnrrrre
68.5
starcoder2 15b 35b-tnstructt-00.
DeepSeek-Coder//V
Qwen2 S28-iiiiinn
66.3
WizardCeder (o:teerr-------
62.9
Coder 398-errr
gwwn22..728--mmtroe
68.7
6
Passip1 (96)
Figure 4: Pass (%) results of different LLMs on XFileDep computed with greedy decoding.
Table and Figure 3 illustrate the performance of model in these programming tasks at the func- tion level. Except for the results underlined, which are sourced from their respective papers, all other re- sults are obtained from the EvalPlus-Leaderboard and BigCodeBench-LLeddrbbar
XFileDep Benchmark Many existing code benchmarks focus on function-level code generation and lack evalua- tions of file-level generation capabilities. To address this limitation, we have developed Cross-File Dependency Benchmark (XFileDep) specifically designed to assess the file-level code generation capabilities of LLMs while con- sidering cross-file dependencies. XFileDep provides a com- prehensive framework by treating multiple interdependent code files as context and testing the model's ability to gen- crate missing files. This benchmark not only measures the model's ability to generate isolated files, but also evaluates
evol evol-codealpacca-v1
"nttps://evalplus.e itb..too/eeedeebbrrrrrr html
"nttps://huggingffaa co/ spaces/bigcooe//
5

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Table I: Pass@ (%) results of different LLMs on HvmanEval (+), MBPP (+) and BigCodeBench computed with greedy decoding. We conducted the evaluation on the Full and Hard subsets of BigCodeBench (BCB), including the Complete (Comp.) and Instruct (Ins.) tasks.
HumanEval Base
MBPP
BCB-Full Comp.
BCB-Hard Comp.
Model
Base
EvoEval
Average
Plus
Base
Plus
Ins.
Ins.
source Model
GPT-4-Turbo (April 2024)
90.2
86.6
85.7
73.3
58.2
48.2
35.1
29.1
61.4
63.1
Claude-3.5-Sonnet (June 2024)
87.2
81.7
89.4
74.3
46.8
33.1
25.7
7B+ Scale
Owen2.5-Coder-33 Codeer33BB Instruct
92.1
87.2
90.5
77.0
58.0
49.0
33.8
27.7
67.6
64.8
OpenCoder-85 8B-Isrrrutt
81.7
77.4
82.0
71.4
50,9
43.2
18.9
18.2
54.8
DeepSeek-Codee -oeee--oooor-----iieeeee
81.1
75.0
80.4
70.1
51.1
42.0
20.9
17.6
51.6
54.4
Codestral-22B-vD..
79.9
73.8
72.5
61.9
52.5
41.8
24.3
16.9
57.2
53.4
7B Scale
Y DSCoder-6.7B
47.6
72.0
58.7
41.8
13.5
30.4
DeepSeekCaler-677---1neee
74.4
71.3
74.9
65.6
43.8
35.5
15.5
10.1
41.4
48.1
Magicoder-S-DD
76.8
71.3
79.4
69.0
47.6
36.2
12.8
13.5
44.6
50.1
WaveCoder-Ultra-66
75.0
69.5
74.9
63.5
43.7
33.9
16.9
12.8
42.0
48.0
OpenCCodelteeppeeer-DS----7
77.4
72.0
76.5
66.4
44.6
37.1
16.9
13.5
44.2
49.8
EpiCoder--DS-6.7B
80.5
76.8
81.5
68.3
50.6
37.9
19.6
12.8
50.0
53.1
v (wwen2.-CCoder-7B-BBB
61.6
76.9
62.9
45.8
16.2
35.6
(ron2.5-Coder-7B-Big
88.4
84.1
83.5
71.7
48.8
40.4
20.3
20.9
55.2
57.0
EpiCoder-Qwen-7B
89.0
82.3
84.1
71.4
51.9
43.8
27.7
22.3
58.8
59.0
its proficiency in understanding and managing cross-file dependencies. The detailed process for constructing the benchmark is described in Appendix B.2.
explore its limit by attempting to synthesize much more complex real-world code repository data than the file-level data that contains only several files. Specifically, utilizing a feature tree extracted from the popular open-source GitHub repository ALaMA-Factoryy, we successfully generate a repository mirroring the structure with over 50 files. The example files within this repository demonstrate the feasi- bility and scalability of our approach, as illustrated in Fig- ure 5. This example highlights the potential of feature-based code generation to produce complex and structured reposi- tories, offering promising direction for future research in repository- code synthesis.
Evaluation Details The XFIleDep benchmark comprises total of 466 problems. For problems, we employed pass@1 I as the evaluation metric, we believe that only code passing all test cases be considered to be function- ing correctly. To ensure stable evaluation, we used docker to standardize the running environment for all problems. During the testing of different models, we consistently ap- plied their default prompts and greedy decoding, with a maximum token length 8192. All open-source models were accelerated using vLLM (Kwon al., 2023). Figure 4 shows the results of different LLMs this benchmark. The experimental results indicate that the EpiCoder, which have been specifically trained on a dataset consisting of over 53k multi-file code samples, significantly outperform the base- line models. This demonstrates the EpiCoder's exceptional capability in generating file-level code while simultancously considering cross- file dependencies. Furthermore, it vali- dates approach's advantage in synthesizing multi-file code data of higher complexity.
4. Further Analysis
In this section, we first analyze the complexity of the gener- ated code (Section 4.1), and then we evaluate the diversity of the code based on feature analysis (Section 4.2). Addi- tionally, we investigate the scaling effect of code instruction data and assess potential data leakage issues, which are detailed in Appendix C.
4.1. Complexity Evaluation
3.4. Potential Repo-level Generation
We emphasize that our ability to generate more complex code data stems from its hierarchical feature tree structure. By flexibly adjusting the depth and width of the feature tree,
Our approach benefits from hierarchical structure of the feature tree, enabling the synthesis of instructions and cor- responding outputs of varying complexities. We further
httpp::/ httb..////hhoougggaaaaaaaooooo
6

---

data EpiCoder: Encompassing Diversity and Complexity in Code Generation
LLaMA-Factory
LLMTune
1lntune/ httpa///eeppreeets//ellutttt
>
config
inport argparse
inport logging
data
docs
fron 1intune modells models.nodel. loader inport load nodel from 1intune.data.de data.data loader inport DataLoader
evaluation
scripts
scripts
limtune
fron 11mtune models nodel wwweenoddddllaeeeeee eevaluator inport ModelEvaluator
src
def parse_angs( () +> argparse. Nanespace:
llamafactory chat
dataloaderpy
Parses command- 1ine argunents.
preprocess.pp py
data
validatorpy
:return: Nanespace of parsed argunents.
processors
[...]
loader.py
experiments
collatorpy
logging
def ()in():
[..]]
models
args (a)gsaas
>
eval
evaluatorpy
logging. www.ccanaisggeeeeeeeooooo (levell-anseeeeeee Lew..eeee...peerrr
model
exporterpy
Load model
adaptorpy
trainespy
logging.info inggiinginfo(f"ooddig nodel (args.nodel_patee
loader.y
[..]]
model load_nodel pardel(argg... (args.nodel, nodel_path)
[..]]
server
train
api.py
Baaating.ino(F""ooraa evaluation data fron (angs.data_patth))
[..]
modelppyyyy
dataloader = (arpadderaangdada_ppppo.o
aplipy
uipy
nodel
train.py
[...]
evaluator HodelEvaluator (nodel, dataloader)
webuil.y py
utils
netrics evaluator. evaluaton.evaluatee]
examples
[...]
Save evaluation results
accelerate
tests
with open(args outpotutppttteee path, 'w") as f:
logging, info evaluation results to prrps.ouupuu_ppah""
inference
test rest_arguments.p
(.urite(str(metriss)
[...]
logging. Braging.info(() inniiinnonaaaaaa (Madel evaluation completed successfully.")
requirements.tol
requirements.bx
setup.py
setup.py
name main()
main
An example of our repo-level code generation. The left part shows the original LLaMA-Factory repository structure, the middle part presents the structure of LLMTune, which we generated based on the extracted feature tree, and the right part illustrates an example file from the generated repository.
Figure 5:
dynamically control the complexity of the synthetic data, ensuring adaptability diverse requirements. Code complexity is evaluated through two approaches: 1) using various software engineering principles; 2) leveraging an external LLM judge to evaluate from multiple perspectives.
Similarly, Table 12 in Appendix C.3.1 demonstrates that our dataset exhibits greater strictness and cyclomatic com- plexity at both the function and file levels. These analyses collectively demonstrate that feature tree-based code syn- thesis can create code that both more complex and more sophisticated than existing methods.
4. USING SOFTWARE ENGINBERING PRINCIPLES
4.1.2. EVALUATING CODE COMPLEXITY USING LLMS
We first adopt software engineering principles to compare the complexity of our generated code (at both the function and file levels) with existing code datasets. This compari- son is based on three well-established software engineering metrics: Halstead Complexity (Halstead, 1977), Strictness Complexity (Ray et al., 2016), and Cyclomatic Complex- ity (McCabe, 1976). Halstead measures logical complexity via operands and operators, Strictness evaluates execution path strictness, and Cyclomatic quantifies control flow com- plexity through branching analysis.
We evaluate complexity using GPT-4o a judge, comparing to existing codebases across four dimensions: Error Han- dling, Modularity, Data Structure Complexity, and Third-
Table 2: Comparison of Halstead complexity between and existing codebase. UO: Unique Operators, UP: Unique Operands, TO: Total Operators, TP: Total Operands
Dataset
UO
UP
TO
TP
Table 2 compares Halstead complexity, with detailed results in Appendix C. Our function-level data outperforms the runner-up by 2.55 and 20.99 for unique operators and operands, and by 56.98 and 100.36 for total counts, nearly doubling the baseline. File-level results further amplify this trend, significantly exceeding function-level metries.
Code Alpaca
4.83
8.22
10.66
15.89
CodeFeedBack
8.11
20.42
30.98
50.05
Evol CodeAlpaca
7.94
18.97
29.91
46.70
OSS Instruct
7.44
20.99
28.05
47.55
Ours (func-level)
10.66
44.32
56.98
100.36
Ours (file-level)
11.64
72.87
100.24
179.98

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Table 3: Distribution of unique features across different datasets.
Datasets
Workflow
Implementation Style
Functionality
Resource Usage
Computation Operation
Security
User Interaction
Data Processing
Code Alpaca
994
6
393
7
282
8
82
221
CodeFeedback
2079
535
689
143
Eval CodeAlpaca
2163
11
591
21
783
60
134
1401
OSS Instructt
2254
5
669
39
413
49
192
Ours (func-level)
2422
6
657
37
819
156
363
2533
Ours (lle-level)
2475
11
812
43
536
103
800
2196
File
Error Handling
Dependency Relations
Data Structures
Implementation Logic
Advanced Techniques
Dutasets
Operation
Logging
Algorithm
Avg.
Code Alpaca
54
43
232
67
10
2.48
CodeFeedback
39
229
10
121
427
100
49
63
5,45
Evol CodeAlpaca
212
15
226
414
130
74
OSS-Instruct
102
211
62
238
150
140
82
5.54
Ours (func-level)
203
357
96
305
316
116
40
100
8.53
Ours (file-level)
218
293
Party Dependencies (criteria in Appendix C.3.2). Table summarizes the average scores of 5k samples, consistently showing significant improvements. Our function-lewe code improves by 32.6% over OSS- ISS-Instruct, while file-level per- formance surpasses it by 52.5%, demonstrating our ability to synthesize more complex code.
4.3. Comparison under Comparable Data Size
further investigate the impact of data size and data qual- ity on model performance, we conducted two additional experiments comparing EpiCoder with existing baselines under similar data size conditions.
Comparison with Magicoder and WaveCoder. We sam- pled 75k function-level data from EpiCoder's full dataset and fine-tuned the SeepSeek-Coder-Barl 7B model, re- ferred to compared it with Magicoder-DS (75k) and WaveCoder- Ultra-6.7B (130k). As shown in Table 5, despite us- ing the same or fewer data samples, EpiCoder-DSS6.. 7B- Sample75k achieves higher average performance, demon- strating a 5.4 and 3.0% improvement over Magicoder- DS and WaveCoder-Uhrres
Table 4: Comparison of complexity EH: Error Handling,
MD: Modularity, DP: Dependency, DS: Data Structure.
BilCouee-DSS-677--Sapplee
We then
Dataset
EH
MD
DP
DS
Avg.
Code Alpaca
2.04
2.10
2.09
2.38
2.15
CodeFeedBack
2.71
3.47
2.23
3.75
3.04
Evol CodeAlpaca
2.53
3.32
2.66
3.58
3.02
OSS Instruct
2.74
3.79
2.78
3.92
3.31
respectively.
Ours (func-level)
4.11
4.71
3.83
4.90
4.39
Ours (file-level)
4.23
5.94
4.62
5.41
5.05
Comparison with SelfCodeAlign. Following the setting in (Wei et al., 2024a), we fine-tuned CodeQwen-7B using 74k subset of EpiCoder function-level data, denoted as Soneddr-Coaeeeeee---S-pppeeeee
The performance is compared against SellCoddeAlign-CQ--T5 (74k) in Ta- ble 6. Cereoder--ooeeeee--- 7B-mpm7e75 achieves 4% higher average score, indicating that the evolutionary data enhancement strategy effectively improves data quality and model generalization.
4.2. Diversity Evaluation
To assess the feature diversity, we sample 1k instances from ours, CodeFeedhack, and other relevant datasets for compar- ison. Features are extracted using GPT-4o (with the prompt provided in Appendix C.4.1) 1), and the number of unique features in each dataset is reported.
5. Related Work
Table shows that our dataset surpasses others in feature diversity, achieving an average 8.53 unique features at the function level and 8.95 the file level. outperforming the nearest competitor by 2.15 and 2.57, respectively.
5.1. Code LLMs
Following the success of general LLMs (Brown et al., 2020; Chowdhery et al., 2023), models like CodeX (Chen et al., 2021) have catalyzed a new surge of research in code in- telligence (Sun et al., 2024). The applications of code in- telligence have gradually encompass broader real-world scenarios (Wang et al., 2023b; Zhu et al., 2024; Shao et al., 2025). Li et al. (2024) explore the use L.LMS for rewriting
Our function-level data significantly improves in areas like data processing, егтоr handling, dependency relations, and user interaction, all 2-3 times higher than in existing code- bases. Additionally, our data also surpasses existing code- bases total feature count, as detailed in Appendix C.4.2.
8

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Table 5: Comparison of Serpder-D--6..B-Samppp with Magicoder-DS (75k) and WaveCoder-Ultra-6777 (130k) on
function-level benchmarks. All values are Pass (%).
HumanEval
MBPP
BCB-Full
BCB-Hard
Model
Data Size
EvoEval
Average
Base
Plus
Base
Plus
Comp.
Ins.
Comp.
Ins.
Magicoder-DS
66.5
60.4
75.4
61.9
46.8
34.8
13.5
11.5
41.2
45.8
WaveCodde--Ultra-6677
130k
75.0
69.5
74.9
63.5
43.7
33.9
16.9
12.8
43.6
48.2
FipiCoder-DS-6-77--755
75k
78.0
73.2
79.4
68.8
48.2
35.6
18.4
12.8
46.2
51.2
Table Comparison of SepiiCaler-Coaeeewaa-------e marks. All values represent Pass @1 (%).
with SelfCodeAAlig-CQQ7B3 (74k) on function-level bench-
Model
Data Size
EvoEval
MBPP+
HumanEval+
Average
SelfCoodeAlignn-Q--7B FpilCoder-CCddeewwen--4
74k
43.6
65.2
67.1 73.2
58.6
74k
47.4
67.2
62.6
code enhance code search performance, while Koziolek al. (2024) propose retrieval-augmented method for con- trolled code generation. Currently, code LLMs typically developed through continual pre-training (Roziere et al., 2023) and supervised fine-tuning (Yu et al., 2023b) based on general LLMss. Given that general LLMs have already extensively utilized real-workd data during their pre-training phases, the construction of data for post-training stages re- mains a critical issue that requires urgent attention (Ding et al., 2024).
6. Conclusion
In this work, we introduce a feature tree- breebased synthesis framework for generating diverse and complex code instruc- tion data. Inspired by Abstract Syntax Trees (AST), our approach constructs hierarchical feature trees capture semantic relationships within code, enabling scalable syn- thesis of instruction data with controllable complexity. The experimental results demonstrate that our synthesized data excels in both diversity and complexity, and the trained Epi- Coder achieves outstanding performance in tasks of varying complexity, from function-level to file-level benchmarks. Moreover, our approach shows strong potential for scaling to repository-level code synthesis and advancing the usabil- ity of code LLMs in complex programming environments.
5.2. Data Synthesis for Code
Current research indicates that using LLMs generate syn- thetic instruction pairs is an effective strategy to address the scarcity of instruction data (Wang et al., 2023a; Yu et al., 2023a). WizardCoder (Luo et al., 2023) employs the Evol- Instruct method to synthesize more complex and diverse in- structional data. Similarly, Magicoder (Wei et al., 2024b) uti- lizes code snippets to guide L.MSs in generating high-quality programming problems and solutions. WaveCoder (Yu et al., 2023b) proposed a generator- distattirodssemiittooooo framework based on LLMs to produce diverse and high-quality instruction data. Inteereeenerrppreer (Zheng et al., 2024) constructs datasets through interactions of users, LLMs, and compilers, aiming to meet specific criteria such as diversity, challenge, and multi-turn dialogue structure. Genetic-Instruet (Majum- dar al., 2024) simulates the evolutionary processes of nat- ural selection and genetic algorithms, employing crossover and mutation operations to generate new instructional sam- ples. SelfCodeAlign (Wei et al., 2024a) extracts code con- cepts to generate new data. These methods collectively demonstrate the efficacy of leveraging L.LMs to synthesize instruction data, significantly enhancing the coding capabil- ities of language models.
Acknowledgment
Yaoxiang Wang and Jinsong Su are supported by National Natural Science Foundation of China (No. 62036004 and No. 62276219). Natural Science Foundation of Fu- jian Province of China (No. 2024J011001), and the Public Technology Service Platform Project of Xiamen (No. NN..3502Z2023 1043). Haoling Li, Jie Wu and Yujiu Yang are supported by the Shenzhen Science and Technology Pro- gram (JCYJ202208 18101001004) and the research grant No. CT2024090512600 of the Doubao Large Model Fund. We also thank the reviewers for their insightful comments.
Impact Statement
In this work, we propose a feature tree-based code synthe- sis framework that enables the generation of diverse and complex code instruction data. Our approach improves the performance of code LLMs across tasks of varying complex- ity, enhancing their applicability in real-world programming scenarios. This, in turn, has the potential to significantly
9

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
improve the efficiency of professionals working in software development and related fields.
Kwon, W., Li, Z., Zhuang. S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E.,, Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
References
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q. et al. Program synthesis with large language models. arXiv preprint
Li, H., Zhou, X., and Shen, Z. Rewriting the code: A simple method for large language model augmented code search. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1371-1389 2024.
2021.
Brown, T. Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Chaudhary, S. Code alpaca: An instruction-followinng Ilama model for code generation. https://githuu com/ sahi 12801 c/coeaalpaca 2023.
Liu, S., Zhu, H., Liu, J., Xin, S., Li, A., Long, R., Chen, L., Yang, J., Xia, J., Peng, Z., et al. Fullstack bench: Evaluating Ilms as full stack coder. arXiv preprint arXiv:2412.00535, 2024.
Chen, M., Tworek. J., Jun, H., Yuan, Q., Pinto, H. O.D Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.,, and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 7.0. 1669, 2019.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research,
Lozhkov, A., Li, R., Allal, L. B., Cassano, F,, Lamy-Poirier,
J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402. 19173, 2024.
241240):1-113,000 2023.
Ding, B., Qin, C., Zhao, R., Luo, T., Li, X., Chen, G., Xia, W., Hu, J., Luu, A. T., and Joty, S. Data augmentation using LLMs: Data perspectives, learning paradigms and challenges. In Findings of the Association for Compu- tational Linguistics: ACL 2024, pp. 1679-1705, August 2024.
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma. Lin, Q. and Jiang, D. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08366 2023.
Majumdar, S., Noroozi, V., Narenthiran, S., Ficek, A., Balam, J., and Ginsburg, B. Genetic instruct: Scaling up synthetic generation of coding instructions for large lan- guage models. arXiv preprint arXiv:2407.21007 2024.
Guo, D., Zhu, Q. Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deopseek-coder: When the large language model meets programming-ther the rise of code intelligence. arXiv preprint arXiv:2401. 14196, 2024.
McCabe, J. complexity measure. In Proceedings of the 2nd International Conference on Software Engineering, ICSE '76, pp. 407, Washington, DC, USA, 1976. IEEE Computer Society Press.
Halstead, M. H. Elements of Software Science (Operating and programming systems series). Elsevier Science Inc., USA, 1977. ISBN 0444002057.
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q.. Hui, B., Zhuo, T. Y.. Singh, S., Tang, X., Von Werra, L., and Longpre, S. Octopack: Instruction tuning code large lan- guage models. In NewrlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Hui, B., Yang, J., Cui, Z., Yang. J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Koziolek, H., Grüner, S., Hark, R., Ashiwal, V., Linsbauer, and Eskandani, N. Llm-based and retrieval- augmented control code generation. In Proceedings of the 1st Inter- national Workshop on Large Language Models for Code, pp. 22-29, 2024.
OpenAL. GPT-4 Technical Report. arXiv e-prints, 2023.
Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., and Devanbu, On the "naturalness" of buggy code.
10

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
In Proceedings of the 38th International Conference on Software Engineering, ICSE' 16, pp. 428-439, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978 178145333001.
Yu, L., Jiang, W., Shi, H., Yu, Liu, Z., Zhang, Y., Kwok, J. T., Li, Z.,, Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 1 12284, 2023a.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12t.300...99,,, 2023.
Yu, Z.,, Zhang, X., Shang, N., Huang, Y., Xu, C., Zhao, Y., Hu, W., and Yin, Q. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. arXiv preprint
2023b.
Zhang, S., Roller, S., Goyal, N., and Artetxe. OPT: Open Pre-trained Transformer Language Models. arXiv prints, 2022.
Sener, O. and Savarese, S. Active learning for convolutional neural networks: A core-set approach. International Conference on Learning Representations (ICLR), 2018.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Fu, J., Chen, W., and Yue, X. Opencodeinterpreeer: Integrating code generation with execution and refinement, In Find- ings of the Association for Computational Linguistics: ACI. 2024, 2024.
Shao, I.., Yan, Y., Poshyvanyk, D., and Su, J. Unigencoder: Merging seq2seq and seq2tree paradigms for unified code generation. arXiv preprint arXiv:2502. 12490, 2025.
Sun, Q.. Chen, Z., Xu, F., Cheng, K., Ma, C., Yin, Z., Wang, J., Han, C., Zhu, R., Yuan, S., al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403. 2024.
Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R.,
Wu, Y., Li, Y., Gao, H., Ma, S., et al. Deepseek-coder- v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2400. 1931, 2024.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct; Aligning language models with self-generated instructions. In Pro- ceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 13484-13508, 2023a.
Zhuo, T. Y., Chien, V. M., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., Brunner, S., GONG, C., Hoang, J., Zebaze, A. R., Hong, X., Li, W.-D., Kaddour, J., Xu, M., Zhang, Z., Ya- dav, P., Jain, N., Gu, A., Cheng, Z., Liu, J., Liu, Q. Wang, Z., Lo, D., Hui, B., Muennighoff, N., Fried, D., Du, X., de Vries, H., and Werra, L. V. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, 2025.
Wang, Z., He. X., Chen, K., Lin, C., and Su, Code-aware cross-program transfer hyperparameter optimization. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 37, pp. 10297-10305, 2023b.
Wei, J., Bosma, M., Zhao, V. Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. In International Conference on Learning Representations, 2022.
Wei, Y., Cassano, F., Liu, J., Ding, Y., Jain, N., Mueller, Z., de Vries, H., von Werra, L., Guha, A., and Zhang, L. Selfcodealign: Self-alignment for code generation. In Advances in Neural Information Processing Systems, volume 37, pp. 62787-62874. Curran Associates, Inc., 2024a.
Wei, Y., Wang, 7., Liu, J., Ding, Y., and Zhang, I.. Magi- coder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learn- ing, 2024b.
Xia, C. S., Deng, Y., and ZHANG, L. Top leaderboard rank- ing top coding proficiency, always? evoeval: Evolving coding benchmarks via LLM. In First Conference on Language Modeling, 2024.

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
A. Appendix of Data Synthesis Framework
In this section, we first present an example of file-level code synthesized by our framework (Section A.I). Then, we provide detailed implementations for several key components of the framework, including Feature Tree Extraction (Section A.2), Feature Tree Evolution (Section A.3), Task Generation (Section A.4), and Code Generation (Section A.5).
A.1. Case of generated File-level Code
This is an example of the file-level code we generated.
mothhay
scraper iy
tests/test_ree rain.py
fron scraper teport Scraper
Ieport recuests
Ioport unittest
parser tmpert Parser
teport Beaut LfulSoup
frox iegort Scraper
opcintzer teport Optintzor
starage LepOrt Storage
Enpert Search
Scraperz
....nt....seefr unts3s
frox search teport Search
frox storage leport Starage
perser ieport Parser
ma
frox unittost. nock taport patcs
detth_
//tops///ooopl con/preducts',
class Testsabbcraapinn htgp:/bicrroogggnnnnnnneee hat.ppptnnnttett...sssssa. TecrCase):
URLs neaded
postchl reques get")
parser:
tast, pert_scrapert (salf, nock_get)::
fron b54 Ieportt EesuttrulSoup
pock_pet-retum_ve wtt.uuumsssuu.sttssoooo.
293
Data
nock_ www.ueet.rttumveeee www.nnssleeeeeeeeeee tex: =
Scrapetturte)
class Parsert
htal_ scrapor.fetch datal)
getaricmerae
scraper Scrapert( httpps////espppeee ///ssmpll..o//prrocctt conppptcoctss 1)
def parso(3t1)_071c))
htel data scraper-fetch data()
parser = Parser()
Simp 2: Perau Bats
self asserttuuu (/ L,etta,,
parsed data
storage Iyy Lepert Jaen Irport sqli
persercseel22
Stop St Store Bata storuge Staruget)
data parser Parsart)
storage.store, datal patsod__aaa)
Storages
persed data parser persehhorll data)
Isel, dounases data
self asssertuuu F (Tentoar sed dadad 1)
Opttonat: Save to J50N
self asserttquu Mparsed data[[]][]][ nene'e) 'Test Product')
storage-Ssve to parrlperaed date)
code:
Step Optintzo (Exonpto usage)
free col Tections import dirque
search.
test_ stttsstoroosssl1))
storage Storaget uxesory:
dsta
class Searcht
storage, store datafdats)
Sstatiomethed
fetched_ data storage. data))
def brewdrs. first resrchcrists, keyaere) 3
self. asserttqqu bytent satched_dtta),,
opriatzer.op codefcods)
se
St Search Sata data 4atal)
opttalicer.pp
keyvord 'exarple'
Irport ast
Impert compttestt
search Searchh)
results search seercshhhheeds,s "Test"s
resalty wwwrchabraad www.ethhhfffiiieeeeeee First, amarchfdnts, leywors) pries(("SSaaree resultss (rasultu)')
worchh Smarchi()
class Optieizzor
@stattowchoo
self helf.asserter dEqui 1/ (Lan(resuitshh 13
salf.assertEqual (aaguuu((rrstt (resul tsfo]l mase 'Test Product:))
_ame _
__ne matn...
unittest.cainf
Figure 6: An illustrative example of file-level code generation (including the corresponding test code file). Different files contain distinct functional modules, with complex dependencies existing across multiple files.
A.2. Feature Tree Extraction
Here are our draft prompts for pre-extraction and the refined prompts for feature tree extraction. For brevity, only a portion is shown here. The complete prompts can be found in our released code.
Draft Prompts for Pre-extraction
Extract high-level information from code snippet using keywords separated by "##". For example:
1. Function Description: Describe the main functionality of code. Use keywords such as list sorting input parsing
## data storage ## image processing.
2. Algorithm Details: Describe the specific algorithm used and its characteristics. Use keywords such as dynamic programming #ttt greedy ahgorithm it divide and conquer #it backtracking ## graph traversal.
...
Please use this code as input and extract as much of the specified information as possible hased on the content the code.
Input: [source_code) Output: <your answer>
12

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Algorithm 2 Feature Evolution with Frequency Estimation
1: Input: Feature tree T, frequency library F containing the frequency of each node in T, maximum steps N
2: Output: Updated frequency library F
3: for step=1 to N do
4: 1 + sample(T)
{Sample subtree / from T)
5: expanded t + LLM.evolve(t))
(Evolve t along depth and breadth)
6: for each node expanded do
< find siblings(node, expanded 8: if then
(Find siblings node in expanded
9:
S < find_ siblings(node,1)
(If no siblings in expanded t, find siblings in T')
10:
end if
if S = 0 then F(node) <- 13: else
11:
12:
(Still no siblings found)
F(node) [3] Eves F(s)
(Uddtee frequency)
15: end if
16:
end for
end for
Part of Refined Prompts for Feature Tree Extraction
Extract features from the provided code snippets, following the requirements for each category below, formatted JSON structure.
Categories to extract:
1. Programming Language: Note the specific programming language used. Example: ["Python", "Java"].
2. Workflow: Outline the main steps and operations the code performs. Example: ["data loading", "preprocessing", "model training", "evaluation", "results saving"].
3. Implementation Style: What programming paradigm the code follows. Example: ["procedural", "object-oriented" "functional"J.
4. Functionality: Explain the function of the code. Example: ["data processing", "user interaction", "system control"].
Resource Usage: Analyze how the code utilizes system resources. Example: ["CPU Cycles", "GPU ComputeOperations",
"Network Bandwidth"].
6. Data Processing: Describe how the data is processed. This category can include the following subcategories:
6.1 Data Preparation:
Steps taken to prepare data for further processing. Example: ["validate units", "strip
whi thippace""
6.2 Data Retrieval: Methods for obtaining data. Example: ["fetching records", "retrieve top-level items").
6.3 Data Transformation:
Describe data transformation operations. Example: ["convert to numpy array",
"jsonschema"
Other relevant subcategorics...
7. Computation Operation: What computation operations are used. This category can include following subcategories:
7.1 Mathematical Operation: Specify mathematical computations, such calculations involving statistics algebra. Example: ["standard deviation calculation", "compute power flow"].
7.2 Algorithmic Operation: Identify algorithm based operations, such those in optimization data sorting. Example: ["simulated annealing", "Best-Fit Decreasing"].
Other relevant subcategories...
8. More content is omitted here:
The demonstration tree for extracting additional categories has been truncated for brevity. For the full list of categories and detailed instructions, please refer to our code. Input: [source_ccom code) Output: syour answer>
13

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
A.3. Feature Tree Evolution
Figure 7 presents an example of the feature evolution. In the experiment, after 9000 steps of evolution, the number of features increased from 5000 to 1,40,000. The estimated frequencies of evolved features calculated as Algorithm 2.
weighted random choice
randomly choose first player
round-robin selection
random index within range
random operation
random index selection
random index from list
random seed initialization
random shuffle
mathematical operations
genetic algorithm
computation operation
optimization algorithms
simulated annealing
algorithmic operations
Dijkstra's algorithm
graph algorithms
A search
mean calculation
statistical operations
variance calculation
standard deviation calculation
Original features
Depth evolution
Breadth evolution
Figure 7: An example of feature evolution.
Prompts for Feature Evolution
You are provided with a feature tree represented as nested ISON structure. Each node this tree represents feature or sub-fcature of a software system, with the leaves being the most specific features. Your task is to expand this feature tree both in depth and breadth. Depth expansion features at the current levels.
Feature Tree Evolution Task:
adding
specific sub-features existing leaves. Breadth expansion adding sibling
Here are some explanations of the features: (explanations)
The input feature tree will be provided in JSON format, and your output should be JSON structure that represents the expanded feature tree.
Output Format:
Expanded Feature Tree: Provide the expanded feature tree as JSON structure.
(example)
Constraints:
1. For breadth expansion, add least 2 sibling features each existing node.
For deep expansion, add new sub-features to any leaf node that could have more fine-grained feature.
3. Focus on generating new and innovative features that are not present in the provided examples.
Please follow the above constraints and expand the feature tree accordingly. Input: [feature_tree )
Output: <begin>expanded feature treecend>
14

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
A.4. Task Generation
To ensure that the language model (LLM) does consistently default to familiar common content, introduced strategy guide the selection of features. From the sampled optional features, certain features are designated mandatory, and the LLM is directed to incorporate them into the scenario and task. Below is example of how this approach is applied.
Prompts for Task Generation
You provided with set of features/keywords, and specific programming language. Your task is use these inputs to conceive specific real-world application scenario that effectively integrates of these features. Then, based the scenario, formulate task or problem that needs be addressed with code.
Procedures:
1. Receive Imputs: These can range from technical specifics like data processing to broader concepts like system interaction.
2. Select Features: Choose combination features from the provided set can realistically integrated into cohesive scenario.
3. Conceive Scenario: Think practical application where the selected features play a critical role.
4. Formulate Task Description: Based the scenario, formulate task that needs be addressed with code. The task should have certain level of difficulty and test the programmer's coding skills through logical reasoning. Specific details, such as numerical values or cuvironmental conditions, should be included to create a well-defined setting for the task, ensuring the programmer doesn't I need to guess any missing information. The task description should not include code or detailed guidance code implementation.
5. Generate an Instruction: Generate high-level instruction one two sentence that describes the overall goal or problem be solved, without diving into the specific implementation details.
Enclose the selected features with <f> and </ʃ>. Enclose the scenario with 
 and 
. Enclose the task with <t> and </t> Enclose the instruction with <i> and </i>.
Inputs:
Optional Features: Aoptional_featuree
Mandatory Features: [mandatory_featurcs)
Output:
Features: <f>your answer</f>
Scenario: 
your answer

Task Description: <t>your answer</t
Instruction: <i>your answer</i>
Optional Features:
dependencyy relations": {
"time": { "time zones handling": []
}. cv2": { "cvtColor": []
"file operation":
"read configuration file":
"read YAML configuration file"
],
"write data file": "write file"
15

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
},
"workflow": {
"data augmentation": [
"audio augmentation"
1, valiiation": [ "check data integrity"
"progr ammiing language": "Python"
}
Mandatory Features:
["read YAML configuration file"]
Selected Features:
["read YAML configuration file, write to CSV file, check data integrity"]
Scenario: A company wants to automate the generation of daily status report for its distributed servers. The configuration for each server (such as IP address, location, and operational status) is stored in YAML file. The system needs to read this configuration file, validate the data to ensure there are no missing or malformed entries, and then write the validated data to a CSV file that can be used for further analysis.
Task: Create a function in Python that reads server configuration data from YAML file, validates integrity of the data (checking for missing or malformed entries), and writes the validated data CSV file. The YAML file will contain a list of servers with their respective details such as IP address, location, and operational status. The function should ensure that all required fields are present and correctly formatted before writing the data to the CSV file. The CSV file should have columns for each server detail and only include entries that have passed validation.
Instruction: Develop a Python function that reads, validates, and exports server config data from a YAML to a CSV file.
A.5. Code Generation
To demonstrate the process of generating code based on the previously outlined task, we present the prompts used, the generated code, its test cases, and the debugging process leading to the final implementation.
1. PROMPT FOR CODE GENERATION
The prompt provided specifies a detailed structure for code generation, including the expected outputs and format.
16

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Code Generation Prompt
Now that you are code expert, I have provided you with the QUESTION. Complete the problem with awesome code logic and give richly commented analysis in the code of your answer. Include the necessary packages and test cases.
QUEESTIN (task)
Full code implementation with test cases
Enclose the python code with "python and and enclose the file name with <file> and </file>, For example:
<file>add. sisepaddppssfile..
ppthon
# add.py
# Code implementation here def add(x, y):
return x
..- The test code should be in single file. <file>test. .yreesttp//fll>>>>
Note the following code executed directly, so only the test cases that can he executed directly need retained. only need to test some simple functions the code. Tests that depend on external files cannot be executed because these files do not exist.
ppyhon
from add import add
def t)st():
assert add(3, assert add(4, 6) == 10
te
File names in order and packages required Answer file names and packages JSON format, wrapped in <json> and </json> tags. For example: <json>
{
["add. py", "test.py"] "packages": : ["packagel"; "package2"]
}
</ <sjoo>>
A.5.2. INITIAL CODE IMPLEMENTATION
# server_config.pp
import yaml import csv
import re
def validate_ippip):
"""Validate the address format""*
pattern = re. compile(rr (?: [0-9] 33111,,,,,))30000 (300
return pattern.match(ipg is not None
def validate_serraeeeeeeaeeeess Letiaate_seveer_dddaee dateserrrr::::::: "*"Validate the server data""*
required_fields [' ['ip_address", 'location', 'operational_status"
Check for missing fields for field required_fiell fields:
if field not in server:
return False
Validate IP address format if not validate_ip(server_ iip_address" ")):
17

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
return False
Check operational status (assuming it should be string, non-empty) if not isinstance "emnsaanceeervee"""praatonnnodd operational_status]) str) or not server[ operational_stats"" J. return False
return True
def read_and wadd_aaaalaeelllaCCCCC (ym__aaalm_llle)))
"*"Read validate the YAML file"*M with open(yaml_file 'r') as file:
try:
data =
if not (daa,, list):
raise ValueError file format incorrect, expected a list servers.
valid_servers = [server for server in data if return valid_servers
except yaml. YAMLError as
print(f"Error in file: (exc}")
return
except ValueError as ve:
"rint(f"Validatinn error: (ve]")
return
def
csv_file):
"""Export validated server data to CSV"*"
if not servers:
p"int("No valid server data export.") ")
return
keys servers[0]. (e)yss((()))))))))))
with (pev 'w', newline='!) file: writer csv. Dictwriterffile, fieldnames=keys) writer.wr writtheader() writer.writeronn triteroossserverrs
def process_seee server_.onfig(yaml_f. csv_file):
**"Main function process server configuration valid_servers read_and_valicee validate_yami (yaml_file)
cSV
csv_file)
File I:
# (est_server_config.n
import unittest
import os
from server_config import read_and__allidate__ee export_to_csv, validate_ip
class TestServerConfiggnne (unttttstttttettttatae
def setUp(self))
"""Set the test environment""'
self.valid gelffwwalid_yaml_commeee conmt_tttteettt...m
ip_address: 192. 168.16.1.1
location: Data Center 1
aperatiinal_status:: operational
ip_address: 10.0.0.2
location: Data Center 2 operational_status:: maintenance *."
self. inw.alid_yaml_contee
ip_address: 192. 168. 1.999
location: Data Center
18

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
operational_statee operational
ip_address: 10. 0.0.2
location: Data Center 2
operational_status:
ip_address: 192. 168. 1.1
location: Data Center 1
operational_status operational ip_address: 10.0.0.999
location: Data Center 2
sttratiooallssttuss maintenance
with open ('valid_servers saaereers.yll,,,,.... 'w") as file: sellewrite(self yalid__yml__ooneen
with open ('invaliddsseeeeeeeeeeeeeee servers. yaml', 'w') file:
file. wel.e.writesself... invalid invallid_ymm__onteen
with open serxxe_seerers..aam 'w') as file:
file.write weeeeweieesseeee yixee__ymm__ooneeee))
def tearDown(self):
Clean up after tests"""
remove (aaooveevvlid_serveee yaml')
remove eomooe('invalle ('invalid_serveer yaml
os. remove mnnixe__seveere yaml')
if os path. exists (etteeesssssuuppt..sss remove output.csvv)
def test_ vastvidddaate_p(ee1)))
"""ts IP validation "unction"""
aasserrrruealllidteee.iiloo 1. 1)))
htt.ssssrtttllllllllllteeeeeeooe 168. 1. 9)
self oat.asrrrflllllllllllccc.iiiiiia
def aagaa__and_allidte_eaama
*""Test reading and validating YAML data"""
valid_servers read_and wadddaaaddldddatee___. ('walid_servers.yyal
self asseertaul [sserttruulllmmmaalidd.eerrr)))
invalid_sena servers = read_and__allidtee__eee (in((aaallid__erees
self aelf.asser (Lnasrr
mixed_servers read_and_walidate__ yaml ('mixed_servers. yeml yaml')
self
1)
def "oss__expor__o__cs(sse1=))
*""Test exporting data CSV
valid_ read_and. waw.waallleeeeeeeeeeeeeeeeeeeeee
servers.yaml yaml ')
'output.csv')
with opupttouupu..csv' 'r') file:
content read()
self allf.asseettnn(ip__ddrrr adddesss,, location, wtt.:aoonooeeaaiooooiiooo operational_status content) sel aelf.assertIn(119 1 168. 1. Data Center 1, oooeerational' content)
self.assertIn("11 10.0 0.0.0...................... Data Center 2, 2,maintenance', content)
if name___ "__main__": unittest. mnittest.main()
File 2: test_servercconfg..py
19

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
A.5.3. DEBUGGING AND OPTIMIZATION
Initial execution of the tests revealed several issues:
The IP validation function incorrectly accepted some invalid IP addresses, such as 192. 168. 19999.
Edge cases in operational status validation not handled properly.
Given the error message and the original code, the LLM gives a corrected implementation.
# server_config.3 py
import yaml import csv import re
def valiidteeiipipp:
"""Validate the IP address format""r
pattern co.commlle(r" (?: [0-91(1, (1001111 ().. 3N))1330--91111333
if not pattern.match(ip)
return False
parts = split(".".
for part in parts:
if not <= (part) <= 255: return False
return True
rest remains unchanged
File 3: corrected server_config.py
20

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
B. Appendix of Evaluation
In this section, we introduce the function-level code benches and provide results on EvolEval and FullStackBench (sec- tion B.1), then detail the construction of XFileDep (section B.2) and provide case of file-level code generation (section B.3).
B.1. Function-level Code Generation Benchmark
detail the individual function-level code generation benchmarks this subsection, as well as more detailed results.
HumanEval and MBPP HumanEval (Chen al., 2021) and MBPP (Austin et al., 2021) are popular benchmarks for assessing code generation. Considering the limited test cases in these benchmarks, followed previous work (Wei et al., 2024b; Zheng et al., 2024) and utilized the EvalPlus (Liu et al., 2023) framework evaluate model robustness across broader range of test cases. To ensure fair comparison, we used version 0.2.0 of MBPP+ provided by EvalPlus, which removes some broken tasks (399 + 378 tasks). Table shows the results of different LLMs on these benchmarks.
BigCodeBench BigCodeBench (Zhuo et al., 2025) a comprehensive benchmark designed to assess a model's 's ability to handle real-world programming tasks, particularly its effectiveness in utilizing various function calls as tools. Our model's ability adeptly manage these high-complexity scenarios underscores its suitability for BigCodcBench.
EvoEval Many benchmarks prone data leakage. To mitigate this, comprchensively evaluate LLM coding capabilities on EvoEval (Xia et al., 2024), constructed by evolving HumanEval to different target domains (Difficult, Creative, Subtle, Combine, and Tool Use). Table shows results of different LLMs. We obtained results from (Xia et al., 2024) and, for models without reported results, conducted tests using their default prompts.
Table 7: Pass (%) results of different LLMs on EvoEval computed with greedy decoding.
Model
Difficult
Creative
Subtle
Combine
Tool Use
Avg
Closed Model
GPT-4-Turbo
50.0
61.0
82.0
45.0
69.0
61.4
GPT-4
$2.0
76.0
53.0
68.0
63.0
Claude-3
50.0
53.0
81.0
42.0
69.0
59.0
ChatGPT
33.0
42.0
70.0
33,0
64.0
48.4
Claude-3-haike
40.0
47.0
65.0
17.0
56.0
45.0
7B+ Scale
57.0
58.0
90.0
58.0
75.0
67.6
Codestral-22B-v0.1
52.0
53.0
69.0
41.0
71.0
57.2
DeepSeekCColer:333--nsttte
47.0
47,0
67.0
31.0
66.0
51.6
WizardCoder-333--J.
48.0
66.0
64.0
49.2
CodeL
31.0
41.0
65.0
18.0
65.0
44.0
MmenCoder-8B-Instteee
45.0
50.0
73.0
28.0
50.0
49.2
7B Scale
DeepSeek-Cooler-6.755
21.0
47.0
5.0
$5.0
30.4
40.0
37.0
61.0
18.0
51.0
41.4
Magicodee---DSS-ee
40.0
34.0
67.0
21.0
61.0
44.6
WaveCoder-Ultra-
38.0
42,0
71.0
24.0
35.0
42.0
OpenCCodeInterprtterr
DS.6.7
43.0
37.0
65.0
25.0
51.0
44.2
EpiCoder-DDS66778
40.0
45.0
70.0
30.0
65.0
50.0
Swen2.5-Coder-7il
35.0
20.0
55.0
27.0
41.0
35.6
Bren2.5-Coder-7B..
48.0
77.0
37.0
65.0
55.2
EpiCoder--Qwe-77B
53.0
48.0
78.0
47.0
68.0
58.8
FullStackBench Most existing code evaluation datasets cover only limited application areas, such as basic programming and data analysis, lacking comprehensive and rigorous assessment of code L.MMs capabilities across broader and more complex computer science domains. To convincingly demonstrate our model's strong performance across a wide and diverse range of areas, we utilize FullStackBench (Liu et al., 2024). This benchmark encompasses 16 programming languages and various computer science domains, aiming to thoroughly and systematically evaluate the abilities of large models in diverse real-world coding scenarios. Table shows the results different LLMSs on the FullStackBench.
21

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Table 8: Model performance across domains of Python in the English Subset of FullStackBench.
Model
BP
AP
DP
MA
ML
SC
DB
MM
Others
Overall
Close- Sourced API Model
OpenAl -reevvee
55.56
78.61
64,29
76.80
79.14
18.75
51.28
61.76
40.00
47.37
100.00
74.47
66.47
OpenAT o1-mini
72.22
75.62
50.00
76.00
80.58
28.75
56.41
56.62
40.00
57.89
100.00
72.34
66.23
Claude-35-Sonnct
50.00
75.62
71.43
76.00
76.26
13.75
51.28
61.76
50.00
63,16
100.00
78.72
65.52
GPT 4o-0806
72.22
72.14
53.57
78.40
76.98
21.25
66.67
55.15
40.00
68.42
100.00
72.34
65.05
Doubao-Codee Coder-Preview
55.56
69.65
50.00
77.60
75.54
27.50
51.28
60.29
20.00
63.16
50.00
55.32
62.91
DeepSeck-v2.5
55.56
68.16
50.00
76.00
76.26
20.00
48.72
56.62
40.00
63.16
50.00
65,96
61.85
Qwen-Max
50.00
70.15
39.29
77.60
72.66
13.75
56.41
57.35
30.00
47.37
50.00
63.83
60.78
GLM----Puus
55.56
65.67
39.29
76.80
74.82
13.75
58.97
50.00
40.00
52.63
100.00
53.19
58.77
20B+ Instruction Tuned Coder
DeepScckKooder<2-Instuee
55.56
68.66
35.71
81.60
79.14
16.25
48.72
53.68
40.00
52.63
50,00
57.45
61.26
Bren2.5-Codcr-32BB InIstruct
50.00
70.15
50.00
77.60
66.19
17.50
61.54
43.38
30.00
47.37
100.00
61.70
58.41
50.00
59.70
21.43
71.20
48.92
18.75
48.72
40:44
30.00
42.11
50.00
44.68
49.05
Code CodeLLlama-34B-Bi Innsstrutt
5.56
22,89
14.29
40.00
17.27
16.25
15.38
18.38
30.00
26.32
0.00
23.40
22.27
13B+ Instruction Tuned Coder
Swen2.5-Coder-11e
Broodcr-----Issiii I---tstruct
55.56
62.69
32.14 14
76.00
70.50
18.75
53.85
38.97
30.00
57.89
100.00
55.32
55.57
DeepScekCoderr-v22 Instrutt
50.00
64.68
32.14
64.00
56.12
26.25
43.59
33.82
60.00
21.05
50.00
53.19
50.47
StarCoder2-151 IS--nsstrur--vO.
61.11
44.28
32.14
63.20
36.69
31,25
$3.85
28.66 68
60.00
36.84
50.00
53.19
43.01
Codel.lama- Instrutt
11.11
22.39
25.00
24.00
20.86
30.00
20.51
13.97
40.00
10.53
50.00
23.40
21.56
6B+ Instruction Tuned Coder
Swen2.5-Coder-TP1 Bodder-7B-Instruee Instrrcttt
33.33
58.21
39.29
66.40
48.92
18.75
38.46
32.35
40.00
47.37
50.00
59.57
47.51
Yi-Coder-9B-Chrr
61.11
50.25
32.14
66.40
46.76
26.25
43.59
36.76
50.00
36.84
50.00
48.94
46.56
DeepSeek-Coder--
50.00
51.74
25.00
64.80
37.41
25.00
30.77
34.56
20.00
52.63
50.00
48.94
43.60
WeenCoder-&B-Instmue
44.44
53.73
28.57
57.60
35.97
26.25
28.21
28.68
0.00
47.37
0.00
44,68
41.11
SeepSeek-Coder-6 IB-Instruct
61.11
49.75
28.57
65.60
38.13
18.75
38.46
22.79
30.00
31.58
50.00
42.55
40.88
BodeQwenl.5-7B-CCh
38.89
45.77
50.00
58.40
31.65
15.00
33.33
22.79
20.00
31.58
0.00
42.55
37.20
CodeLlama-7B-Instree
27.78
23.88
25.00
28,00
20.86
23.75
10.26
11.76
50.00
10.53
0.00
21.28
21.33
EpiCoder-DD--67B
61.11
47.26
25.00
61.60
41.01
40.00
41.03
27.21
50.00
36.84
50.00
42.55
43.25
FpiCoder-Qwen-77
44.44
61.19
17.86
72.80
61.15
28.75
51.28
27.94
20.00
47.37
50.00
40.43
50.24
B.2. Cross-File Dependency Benchmark
The Cross-File Dependency Benchmark (XFileDep) is a specialized benchmark designed to evaluate the performance of code generation models in handling cross-file dependencies. In real-world programming scenarios, there exists a complex web of dependencies between different code files. XFileDep provides comprehensive framework that tests model's ability to generate missing files by considering multiple interdependent files as context. This benchmark not only measures the model's capability to generate individual isolated files but also assesses its proficiency in understanding and managing dependencies between files. As illustrated in Figure 8, the constructing the XFileDep consists of following steps.
Step 1: Data Sample Selection. The construction of the XFileDep starts from carefully data selection. From the initial cross-file dataset of 35,000 data samples constructed using our pipeline of synthetic data based extracted features, we carefully filtered out cross-file data samples with fewer than interconnected files (excluding any test files), resulting in refined set of 3,435 high-quality samples. Figure 9 comprehensively displays the distribution of file counts, the average file length for each data sample, and the overall structural characteristics of the dataset.
Step 2: Dependency File Selection. We utilized Abstract Syntax Trees (AST) to conduct dependency analysis and structural identification of the Python code files. AST allows parsing of the syntactic structure Python files, enabling extraction of module import dependencies, function definitions, and class definitions. By parsing all code files within the data sample and identifying the collaboration between classes and functions, we documented the details of defined functions and classes along with the information imported modules. With these capabilities, analyze the dependency relationships between files, discern key files, and select representative candidate file (to serve as the target file for the code generation task) that has a substantial amount of cross-file dependencies. This approach allows us to generate a structured data analysis report. The systematic nature of this analysis allows us to efficiently handle large cross-file data, providing clear dependency graphs and detailed information on code structure. We also filtered out data samples that lacked rich cross- file dependencies, retaining 2,934 samples that met the criteria.
were able to traverse the entire data sample,
22

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Passed
Soed 3,436
Data Sumple Selection
Unsote Fitering
9,436
1,006
Dependincyy Selection
2,204
Human Checking
Flltring 2,231
Teat Case Ausmentation
2,231
Fulled 1600
Herathee Twett Code Refremmnt
284
Figure 8: The Sankey diagram for XFileDep benchmark creation, with numbers indicating the quantity of data samples.
Distribution File (w/c Files)
Distribution of Average File Characters per Sample (w/o Test Files)
5429
1329
3029
Prequercy
620
A29
219
Fic Count
Avarogo Character Court
Figure 9: the distribution of file quantities and the average file length for each data sample.
Step 3: Filtering. We analyzed the runtime environment, required libraries, and external files (such .npy, .xlsx, json, log) for each data sample. Based on this analysis, data samples that could not provide the necessary files or dependencies were filtered out. We also excluded data samples that had long runtimes or required the generation of specific output files, which made obtaining test results difficult. addition, to increase the overall difficulty of the task and ensure that cross-file -ile code generation operates an appropriate file-level length, we filtered candidate files whose length was less than 300 characters. Finally, we obtained a total of 2,231 I data samples.
Step 4: Test Case Augmentation. To enhance the test coverage of the code within each data sample, utilized GPT-4o to generate additional test cases. This process ensured that the core functional methods were robustly and comprehensively tested. In Table 9, we have compiled the statistical data before and after the augmentation of the test cases. We conducted a
Table 9: Comparison of Test Functions and Test Cases before and after augmentation for 930 data samples.
Original
Augmented
Test Functions
Total
3,837 4.13
7,933
Average per sample
8.53
Max in file
12
23
Test Cases
Total
6,010
14,305
Average per sample
6.46
15.38
Max in file
20
44
23

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
runtime check the data after augmenting the test and obtained 611 samples that successfully passed the test cases. The prompt for augmenting the test case with GPT-4o is illustrated below.
Prompts for Augmenting Test Cases
are an expert Python programming and lest-driven development. have a repository of Python code with corresponding test files aimed at verifying the correctness of my code. However, believe the current test cases are insufficient and need more comprehensive and robust test cases.
Requirements:
TTee test cases should be written using a suitable testing framework (e.g., pytest, unittest).
Maintain consistency with existing test structure and naming conventions. Ensure that all test cases pass before finalizing.
Inputs:
The file structure (including filenames) and contents of the Python code. (python_code)
'The file structure (including filenames) and contents of the current test files. (test_code)
Deliverables:
Updated test files with additional test cases.
Documentation or comments within the test files explaining each new test case.
Your output:
Returss code content only.
Prompts for Refining Test Code
are an expert Python programming and test-driven development. have a repository of Python code with corresponding test files aimed at verifying the correctness of my code. However, there are errors in the current test cases and test code that would like you to proofread and correct.
Requirements:
The test cases should be written using a suitable testing framework (e.g., pytest, unittest).
Maintain consistency with existing test structure and naming conventions.
Ensure that all new test cases pass before finalizing.
Inputs:
The file structure (including tilenames) and contents of the Python code: (python_codds
The file structure (including filenames) and contents of the current test files: [test_coo code)
Program error messages caused by running the test file:
Deliverables:
Fully correct test code file.
Documentation or comments within the test files explaining each new test case.
Your output:
Returns code content only.
Step 5: Iterative Test Code Refinement. For data samples that fail the test cases, the code content, test cases, and error information are extracted. Based on these detailed input descriptions, we utilize GPT-4o for checking and modification, and subsequently re-run the refined test code for validation. We performed a single iteration of modification on the test code, resulting in 394 successful test cases out of 1620 samples. Finally we have a sample 1,005 that pass the test cases. The prompt for refining the test code with GPT-4o shown above.
24

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Step 6: Unsafe Filtering. To ensure the validity of our test cases, we constructed unit test environment based the dependency requirements specified in each Python file. We then executed all test cases and filtered out any samples that failed the tests presented unsafe operations, such as kill, terminate, rmtree, rmdir, and rm. This approach ensures that our canonical solution is absolutely correct. Finally, we retained 930 samples of cross-file folss-iil data.
Step 7: Manual Checking and Verification To ensure the quality of the questions in the XFileDep. we manually review and verify each question. The evaluation criteria require that the Python code in the Answer accurately reflect the functionalities described in the question and produce correct outputs that meet the expected requirements. Any questions that do not meet the criteria will be filtered. All manual reviews are conducted by individuals with least 5 years of Python programming experience and master's is degree higher. Additionally, these reviewers currently employed as software engineers or in similar roles at leading internet companies. After manual review and filtering, we obtained 466 questions.
Step 8: Annotation. We selected target files with extensive cross-file dependencies (either frequently invoked by other files or frequently invoking other files). Using GPT-4o, we meticulously annotated all classes and methods in these files with detailed documentation, emphasizing their purpose, functionality, and relationships with other components. The annotation process did not alter the original code in any way, and the successful execution of the annotated files verified the correctness of the ground truth. The full prompt for annotating the target file with GPT-4o is illustrated below.
Step 9: Benchmark Construction. To maintain a high level of difficulty in the benchmark construction, we extracted all code blocks from the functions and classes within the target files, leaving only the import statements, FunctionDet ClassDef, and the corresponding docst trigss. The instruction set provided the names and contents of all other files in the cross-file data sample as context and included the target file's name and skeleton structure for the completion task.
Prompts for Annotating Target File
Your task read through provided Python code and add detailed docstrings describe purpose and functionality each class and function. Your additions should follow the PEP 257 conventions and should not alter the original code in any way. The docstrings should provide enough detail to help other developers understand what each part of the code does and how to use it appropriately.
Here the Python code: python
(target_file_codee)
Please add the necessary docstrings without changing the actual code. Ensure that output is enclosed with its corresponding tags:
python
(Your code here]
25

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
B.3. Case of File-Leevel Code Generation
This section provides comprehensive details on file-level code generation using generated case. The directory structure of the example and the detailed contents of each file follows:
-=example_root/
main.py
optimizer.pp
parser.py
scraper.py
search. py storage.
--tests/
test_main.py py
# main.py
from scraper import Scraper from parser import Parser from storage import Storage from search import Search
from optimizer import Optimizer
def main():
hhttps://example.com com/products", Add more URLs as needed
]
Step 1: Scrape Data
scraper = (url))) html data scraper. fetch_aattc))))
Step 2: Parse
parser Parser parsed_date data parser parser.parse pareereeelssetmmaaaa
Step 3: Store Data storage Storage() storage. (atrreddatt(rsse__daaa
Optional: Save to JSON storage. save_to towv__tt_jsoopprrseeeee
Step 4: Optimize Code (Example usage)
code
def sxample_function/]]
result return result""e ""w
optimized_code =
Step 5: Search Data
data storage. fetch_data()
keyword = 'example'
search Search
results search. cearch.breadth_fiee
keyword)
print (eaaerrc results: (results)")
if ___am____ = main__'n (ain()
File 4: main.py
optimizer.pp py
26

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
import
import compileall
class Optimizer:
Example of constant folding optimization
@staticmethod
def monstant_foldng((e))
tree = ast parse(code)
optimized_ trreeedd__eeeeeeeee ast. rrixmmisiin__lccttionsrrre
return compileeapttimize__trr filename="<ast>" mode="exec")
@staticmethod
def
optimized_code Optimizer constant_.olldiggcood,
compileall. poppleeal.compple..e code (optimized__com return optimized_code
File 5: optimizer.py
parser.pp .py
from bs4 import BeautifulSoup
class Parser:
@staticmethod
def parse(html_data)
parsed_dats
for html in html_data:
soup = BeautifulSoup(htei "html. parser')
products []
for product in soup. (llect((.productt)
name product. select_onee("" one prrouuppoouct text.stripg
price = welouuttseeect_.ooo. tttrrooonntppoocctprrrre productt-picc') text. text.strip()
description product. selest_oneeceeeeeeeeeeeeeeeeeeee seelect_nee..prduuttteeseerr pwoduct-descrippeeeee iitissrrriiionnn text.exttseeeee strip(
products.c p
'name': name,
'price': price, descriptiin': description
parsed _add_ddata.. wtt.ddaaa.pppa.ee... (prouuup)))
return parsed_date
File parser.py
# scraper.pp py import requests
from bs4 import fuluti
class Scraper:
def _init_ (self, urls): self.uris urls
def fetch_data :tth__dttaseef):: html_data = []] for url in self.urla try:
response = requests.get(une
response. saaeesserriis__sssttttttue
html_data. aap.daaappppeddespppsseeeeee text)
except requests. requests. RequestException as
print(f"Error fetching data from {url]: (e}")
return html_data
File 7: scraper.py
27

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
search.py
from collections import deque
class Search:
@staticmethed
def breasth_firstt_eerrhhtttt keyword):
queue deque(data)
results = []
while queue:
item queue popleft()
keyword, lower() in item[" name""n ].lower() () or keyword. lower item[" description ]. 1.lower( ():
append
return results
File 8: search.py
# storage. py import json import sqlite3
class Storage:
def _snit__self,, db_name='dat "ataaddb"""" db)): self conn (ct.eee.cooneeccccnaaceaee lelf.create_table(
def Treate_table(stt)
with self.comm self.conn.ed "oon.exxc CREATE TABLE IF NOT EXISTS products id INTEGER PRIMARY KEY, name TEXT, price TEXT, description TEXT )"
def store_tt @see_,, data):
with self. conn: self.conn:
for products data:
for product in products:
self conn. execute(*
INSERT INTO products (name, price, description)
VALUES (:name, :price, :description)"" product)
def letch_data(set
cursor self. con.. cursor()
cursor. eessorrexecute("EELL SELECT name, price, description FROM products') return cursor. feechall()))
def (sef,, data, filename=' ddtaa with open(filename, 'w') f: json.dump (data, f, indent=4)
File 9: storage.py
tests/test_main import unittest
from scraper import Scraper from parser import Parser
from storage import Storage from search import Search
from unittest. mock import patch
class TestWebScrapinggppuniteeee (esstttttttt..eeetttss)
@patch requests. . def (self mock_get):
28

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
mock_get. retuur value. wtw..aat.....uuu........ status_code 200
rettr text '<html><div "xass=" class= "las=="producct-name"TTTT
Product</span>>span p pttpppiieee..sio/pppppppppp 0</span><span class=" "rass="produc--dssrriptiooos""id (eserr/ptoook/ssppn/dd//////tbb>
scraper Scrapr((hhttpss//eaaplleccc comppoodutssss
html_data = scraper.fetch_data.
self.assertEquual (.sseerrEEuullleeee(mmmtta..... 1)
def rest_parrer
ittm__ataa = [ ['<html><div <lass="praducct>>>spnn class=prroductt-ame""es Products/span>><pan
parser Parser
parsed_data parser.parse parse(htmm__dtaa))
self asser (paeepepdaaaddaaa 1)
self.asser [atrssedddda[[]][[][[]]]] name'') 'Test Product')
def (est_storageeself)):
storage :mmmmrryy")
data = [[['name's 'Test Product', 'price": 'S10', 'description': 'Test Description }]]
fetched_data storage. storage.fetch fetch_data()
self asser tEqual (Ln((eeccd(e)))))..da)aa))))) 1)
def test jest_search(seef)
data [
('name' 'Test Product', 'price': '$10', 'description': 'Test Description')
('name': 'Another Product', 'price': "220', 'description': 'Another Description'
search Search()
results = search. breadth_firstt fir.ttsssrarccaaaaa, 'Test') self asser tEqual (sen(reuults), 1)
self asser tEqual (results[O][ name J, 'Test Product')
if _ame______ == "__main__'' unittest.maiin main()
File 10: tests/test_ Destshesttmain.ppy
29

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
C. Appendix of Analysis
this section, we first present the scaling effect of code instruction data (section C.I), then discuss the data leakage issue (section C.2), and provide detail analysis regarding complexity (section C.3) and diversity evaluation (section C.4).
C.I. Scaling of Code Instruction Data
85.7
85.1
84.8
85
83.9
83.9
82
81.7
HumanEval (Avg)
MBPP (Avg)
79.6
80
79.0
77.7
78.1
78.4
77.8
76.9
75.8
75.5
75
P
55
52.9
52.3
52.3
52.0
50.9
50
48
45
43.8
41.1
40.5
40
38.8
39.1
39.4
39,5
38.4
ACB-Complete BCB- Ins-tnstt
35
10k
40k
80k
120k
160k
200k
280k
320k
380k
Number of Training Samples
Figure 10: The scaling law code instruction data. The results obtained from randomly sampled subsets of 380k data points across the HumanEval, MBPP, and BigCodeBench benchmarks.
Although both the fields of mathematics and code are characterized by rigorous logic and precision, they exhibit different phenomena in terms of the quantity of instruction data. Motivated by previous analyses of instruction data scaling laws in the mathematical domain, we design experiments to understand the scaling laws in the code domain. We randomly sample 380k data points and set a series of data quantities for our experiments. The results on the HumanEval, MBPP, and BigCodeBench benchmarks are depicted in Figure 10. is evident that with the increase in data volume, the performance improves significantly. Moreover, even with the data size reaching 380k, there is still an upward trend, demonstrating that our dataset possesses sufficient diversity effectively combat overfitting.
30

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
C.2. Data Leakage
C.2.1. OVERALL RESULTS
BigCodeBench
evol-codealpaca-vl
Maglicoder-OSS-nnstruutt//
CodeAlpaca-20k
Codefeedback-Fiteeeee filturrddinstructiinn
Leakage Threshold
MBPP
HumanEval
Similarity Score
Figure 11: The distribution of cosine similarity scores between various benchmarks HumanEval, MBPP, and BigCodeBench.
We investigate potential data leakage issues to ensure that our synthetic data are free from such risks. Specifically, we use the jinaai/ jinaa http:/jiiaaememmiiiieeaem embeddings-v embedding model to generate embeddings for the output segments of all training data, including our synthetic data and other training datasets used for comparison. For the HumanEval, MBPP, and BigCodeBench benchmarks, we encode their test datasets and compute the cosine similarity between each test instance and all training samples. For each test instance in the benchmarks, we identify the training-test data pair with the highest similarity score and plot the distribution of these scores Figure 11. Furthermore, through case-based analysis of similarity scores, we define threshold for potential leakage iimilarity Score=0.99 9), with detailed explanations provided in Appendix C.2.2. Despite the large scale of our dataset, which puts it at disadvantage when identifying the most similar sample for each test instance, Figure II demonstrates that our 380k synthetic function-level data show minimal evidence of data lcakage, even for the HumanEval benchmark, where the risk of leakage is most pronounced. Further analysis of similarity scores across other benchmarks supports the conclusion that our synthetic data are not strongly similar to the benchmark. This confirms that our model's performance gains are not due to overfitting benchmarks but stem from data quality and diversity.
31

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
froe byping stpert Latt
1 Fron 199ing seport Los
(oel,,
interprneni Lisflel), Lotlbe)) "rarbery'
Entersperelll I O
sa Inserageneel)) 41
99%
for comcrattcssetrr
Cewter
rater resuls
reeam rarlts
LiAT
aieenics LIKE Evitieet SAVEARG 335
now and (4,
nedianl Dt,
95%
0
sedlues panced 199/2)
<lee:
15
<1se:
SrSrcimedsant D.
2011
paur Laie wgreeatt Di
SATFICARR DN CAE TERNC' 3 TEEE THERSIEE 30 MATS
In SATEE
DOTAEES TIE ENIr SYE SECEURT STATNOCK
Potse
ontre www.:/eeeill..
90%
futaa
netur
prose POLTEEITTT 3, 5,
prant printiptinn OUL PA
700
meettrrsch BE11: Elnte
LISU ELoNoEE
Ietur EaIE
85%
BONAERaaNNIR 3. MIDE
27000 P7O0ERATTEE
Fats
AwrIAoeA21 Trai
SrA.SINreoowaaIA.... s. 1081, AST T.
F7000 SrocURrEEeeS atrRes-TrantE Tras
Figure 12: Cases from HumanEval (left) and evol evol-codealpaca-v1 (right) with varying similarity. Embeddings are computed based on the "output" portions of the trainset and the "prompt canonical_solution from HvmanEval data.
32

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
C.2.2. LEARAGE THRESHOLD SETTING
We embedded multiple training datasets and different benchmark datasets and calculated the cosine similarity between them. Additionally, we analyzed the most similar training data sample for each test data point to identify potential data leakage case study, found that the
issues. Using the evol-codealppccav dataset, which exhibited the most severe leakage, training data contained extremely serious data leakage. Figure 12 presents pairs of benchmark data (left) and training data (right) with various similarity scores. The index of the data samples presented in the case study is provided in Table 10. After observing and analyzing multiple samples, we manually selected a similarity score of 0.9 as the similarity threshold.
Table 10: The index of the data samples presented in the study.
Similarity
99%
95%
90%
85%
HumanEval index
5
47
43
57
evol -oodeeecaaplc index
81260
45682
9508
51167
C.3. Complexity Evaluation
C.3.1. DETAILED METRICS FROM THE SOFTWARE ENGINEERING PERSPECTIVE.
Halstead-derived Principle. We supplement Halstead-deriven metries in Table 11, which are based on unique operators (n1), unique operands (722), total operators (N1), and total operands (N2). Among these recognized metries, our data consistently shows significant performance gains compared to the current codebase. For instance, we achieve notable complexity advantages in program volume and program difficulty. The programming effort and estimated programming time further confirm that our data requires more time and effort to achieve. While the increased complexity may suggest higher potential for bugs, we address this issue by incorporating test cases during the data systhesis.
Strictness and Cyclomatic Complexity Measures. Table 12 in demonstrates that our dataset exhibits greater strictness and cyclomatic complexity at both the function and file levels. In Table 13, we break down eyclomatic complexity to observe scores for specific operations, such as while, and boolean operations. Table shows that our gains in Cyclomatic complexity mainly due the higher occurrence of if. for, except, and return statements. This suggests that our program handles more loops and incorporates broader range of exception handling scenarios. We break down the code strictness complexity scores. Table 14 shows that our data achieves a significantly improvement in Doc Strings, indicating a more
Table 11: Derived Halstead metrics. These metrics are derived from unique operators (n1), unique operands (n2), total operators (N1), and total operands (N2).
Program Length (N)
Vocabulary (n)
Volume (V) V = N x log2(n)
Difficulty (D)
Dataset
N N₁ + N₂
ni + 12
D
N2
mg
Code Alpaca
26.55
13.05
108.39
5.07
Evol CodeAAlpaca
76.61
26.91
381.45
10.76
CodeFeedBack
81.03
28.54
416.78
10.49
OSS Instruct
75.61
28.43
381.32
8.75
Ours (func-level)
157.34
54.97
932.78
12.34
Ours (file-leve)
280.22
84.51
2035.63
13.64
Programming Effort (E)
Estimated Time (T)
Predicted Bugs (B)
Dataset
E = D x V
T
E
B
T8
3000
Code Alpaca
1043.26
57.96
0.03
Evol Codeal
5954.64
330.81
0.09
CodeFeedBack
6204.38
344.69
0.09
OSS Instruct
4528.98
251.61
0.08
Ours (funn-level)
13396.28
744.24
0.17
Ours (file-level)
67851.94
3769.55
0.28
33

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Table 12: Comparison of Strictness complexity (left) and Cyclomatic complexity (right).
Dataset
Mean
Median
Std
Dataset
Mean
Median
Std
Code Alpaca
0.18
0.00
0.52
Code Alpaca
2.10
1.00
1.66
Evol CodeAlpaca
0.82
0.00
1.63
Evol CodeAlpaca
3.76
3.00
3.48
CodeFeedBack
0.97
0.00
2.09
CodeFeedBack
3.96
3.00
3.33
OSS Instruct
1.50
1.00
2.19
OSS Instruct
3.45
3,00
2.98
Ours (func-level)
4.95
4.00
3.77
Ours (func-level)
5.14
5.00
3.01
Ours (file-level)
5.41
4.00
3,85
Ours (file-level)
14.93
14.00
6.73
Table 13: Comparison different control flow and logical operation frequencies.
Dataset
while
for
and
or
except
return
break
continue
bool_op
Code Alpaca
0.42
0.06
0.43
0.03
0.01
0.01
0.66
0.01
0.00
0.05
Evol CodeAlpaca
1.35
0.15
0.68
0.15
0.14
0.12
1.59
0.03
0.02
0.29
CodeFeedBack
1.59
0.14
0.76
0.19
0.14
0.08
1.62
0.05
0.02
0.33
OSS Instruct
1.38
0.07
0.59
0.16
0.08
0.07
1.58
0.04
0.01
0.24
Ours (func-level)
2.29
0.16
1.14
0.16
0.20
0.70
3,06
0.05
0.10
0.35
Ours (file-level)
3.60
0.38
1.77
0.25
0.21
1.11
5.24
0.08
0.10
0.45
Table 14: Detailed metrics of code strictness complexity
Dataset
Type Hints
Parameter Validation
Value Verification
Exception Handling
Doc Strings
Return Value
Assertions
Validation
Code Alpaca
0.00
0.00
0.04
0.02
0.00
0.06
0.07
Evol CodeAlpaca
0.21
0.08
0.14
0.20
0.03
0.01
0.15
CodeFeedBack
0.42
0.09
0.16
0.10
0.01
0.05
0.14
OSS Instruct
0.94
0.09
0.12
0.10
0.02
0.08
0.15
Ours (func-level)
0.94
0.10
0.29
0.81
0.02
2.45
0.34
Ours (file-level)
0.43
0.28
0.40
1.76
0.01
1.74
0.80
comprehensive and rigorous consideration to code documentation. Additionally, we demonstrate clear advantages in exception handling, return value validation, and type hints, suggesting that our data is more standardized and stringent.
These analyses upon Halstead complexity, strictness and cyclomatic complexity, collectively demonstrate that feature tree-based code synthesis create code that is both more complex and more sophisticated than current synthesis methods.
C.3.2. PROMPTS FOR EVALUATING CODE COMPLEXITY USING GPT-40.
We adopt the LLM-as-a-judge methodology, using GPT-4o to comprehensively evaluate code complexity across four key dimensions: Error Handling, Modularity, Data Structure Complexity, and Third-Party Dependencies. We define four distinct levels of standards for each dimension and strategically leverage GPT-4o to assign precise score to each sample based on these well-defined criteria. Detailed evaluation criteria, corresponding prompts, and scoring methodology are shown below.
34

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Prompts for Grading Data Structure Usage Complexity.
You are an expert in evaluating complexity levels of data structure implementations for given code. Please provide a single integer score from 2 to 8.
Criteria:
Score 2 for case:
Basic data types only; Only use primitive types (int, string, etc.); Simple arrays or lists; No custom data structures.
Score case:
Basic data structures; Uses built-in in collections (sets, maps): Simple combinations of basic structures; Basic object-oriented classes.
Score 6 for case:
Intermediate data structures; Custom data structures for specific needs; Efficient combination of multiple structures; Clear interfaces for data access.
Score 8 for case: Advanced data structures; Specialized tree/graph structures; Optimized for operation requirements; Well-designed structure relationships
Inputs:
You are judging the following code: Begin Code (code) #tf End Code #il
Output Format:
Please provide your evaluation the following format:
Grade: single integer within 2, 4, 6, and 8
Prompts for Grading Modularity Implementation Complexity.
are an expert in evaluating code architecture complexity, with special focus modularity implementation patterns. Please provide single integer score from 2 to
Criteria:
Score case:
Code without any modular designs, logic a single file with clear separation.
Score 4 for case:
Code with minimal modularization, basic separation of logic but with tight coupling.
Score for case:
Code with reasonable modularization and logical separation (e.g., some decoupling and partial adherence to design patterns, but limited scalability or reuse).
Score case:
Code implementation with clear and practical modularization (e.g.. modules have distinct responsibilities, dependencies are simple and direct, and logical layers such database, business logic, and user interface are separated. The design is easy to read, maintain, and extend for most real-world needs).
Inputs:
You are judging the following code: ## Begin Code ##
(code) ## End Code ##
Output Format:
Please provide your evaluation the following format:
Grade: single integer within 2, 4, 6, and 8
35

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
Prompts for Grading Third Party Dependency Complexity.
You are an expert in evaluating third-party dependency complexity of given code. Please evaluate the complexity of third-party library usage and dependencies in the following code and provide single integer score from 2 to 8.
Criteria:
Score 2 for case: No external library usage, only built-in modules (e.g., os, sys, json).
Score 4 for case: Uses single third-party library with basic function calls (e.g., pandas, scipy, numpy).
Score for Uses 2-3 third-party libraries.
Score case: Uses at least three third-pat partt libraries with some interaction between them.
Inputs:
You are judging the following code: ## Begin Code ## (code) ## End Code ###
Output Format:
Please provide your evaluation the following format:
Grade: single integer within 4, 6, and 8,
Prompts for Grading Error Handling Complexity.
are an expert evaluating emorr handling complexity of given code. Please provide single integer score
Criteria:
Score 2 for case: Complete lack of eиror handling.
Score case: Basic error handling that prevents crashes.
Score case: Basic error handling that prevents crashes with informative logging info.
Score 8 for case: Error handling covers major scenarios.
Inputs:
You are judging the following code: #ti Begin Code ##
(code) ## End Code ##
Output Format: Please provide your evaluation the following format: Grade: single integer within 2, 4, 6, and 8.
Table 15: Distribution of total features across 1 samples.
Implementation Style
Resource Usage
Computation Operation
User Interaction
Datasets
Workflow
Functionality
Security
Processing
Alpaca
1842
1005
324
525
331
CodeFcedback
3718
1018
1260
560
1432
50
1354
Evol-A/paca
3550
1013
1305
598
1290
60
325
1838
OSS-Instruct
3106
1015
1192
421
585
49
278
1163
Ours (lunc-level)
4004
1050
1671
831
1213
227
542
3436
Ours (file-level)
4663
1244
2629
1183
778
123
1407
3160
File
Error Handling
Dependency Relations
Data Structures
Implementation Logic
Advanced Techniques
Datascts
Logging
Algorithm
Operation
Alpaca CodeFeedhack
18
77
166
365
1309
1143
17
8.24
46
421
453
719
1624
1770
81
14.90
Evol-Alpaca
85
354
811
579
1661
1480
15.10
OSS- Ins-Iustruct
236
799
12.77
Ours (func-level)
567
887
205
2499
451
2297
1866
21.89
Ours (file-leve)
1453
1029
548
4010
454
2565
2187
212
27.65
36

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
C.4. Diversity Evaluation
C.4.1. PROMPT FOR FEATURE EXTRACTION.
Prompts for feature extraction.
Extract features from the provided code snippets, following the requirements for each category below, formatted JSON structure.
Responses in the following categories should be concise and organized in JSON format surrounded with <begin> and <end>, Categories may include nested structures if applicable. Here an example the expected format:
<hegin>{
"functionality": [ "data processing"
"computation operation": "mathematical operation": "find common divisor", "base conversion", "prime factorization' "statistical calculations" "maximum"
"data processing". "data transformation": "drop rows" J.
"data structures". "string". "list", "graph". ]tree"].
"implemennation <end>
"loop"]
Categories extract:
1. Programming Language: Note the programming language used. Example: ["Python". "Java"].
2. Workflow: Outline the main steps and operations the code performs. Example: ["data loading", "preprocessing", "model training", "evaluation". "results saving"].
3. Implementation Style: What programming paradigm the code follows. Example: ["procedural", "object-oriented"; "functional")..
4. Functionality: Explain the function of the code. Example: ("data processing", "user interaction", "system control"].
16. Advanced Techniques: Specify any sophisticated algorithms methods applicd. Example: ("Machine Learning", "Linear Regression". "Optimization Algorithms"J.
Requirements:
1. If the code snippet contains fewer than three lines, only extract the most precise and relevant single feature. For function, provide no more than five features, prioritizing the most critical and distinctive aspects. 3. Only evaluate the code snippet; disregard any natural language deseriptions or comments outside the code context.
4. Try not to let feature appear in multiple categories at the same time.
Inputs:
cource__code
Output Format:
<begin>
"workflow": l"your answer"].
"implementation style": ['your answer"],
"functionality": ['your answer"],
"resource usage": ["your answer"], "computation operation": ["your answer"], "security": ['your answer"].
"user interaction": ['your answer"].
"data processing": l'your answer"],
"file operation": I'your answer"),
"error handling": ["your answer"),
"logging" : ['your answer"].
"dependency relations": ['your answer"], ["your answer"],
"algorithm" : :
"data structures": ["your answer"].
"advanced techniques": ["your answer").
<end>
features a category cannot directly extracted from code, please set it an empty [].
C.4.2. TOTAL FEATURE DIVERSITY COMPARISON.
We compared the number of features in each category, as shown in Table 15. On average, our function-level data contains 21.89 features per sample, while the file-level data includes 27.65 features per sample. This demonstrates that our dataset not only leads unique features but also significantly surpasses others total feature count.
37

---

EpiCoder: Encompassing Diversity and Complexity in Code Generation
38


Reviews:
[{"review_id": "73XrNy2vhN", "reviewer": "Reviewer_AwKV", "summary": "This paper presents a new data synthesis method to generate complex and diverse code data. Given some seed code data, this method prompts an LLM to extract code features (e.g., functionality concepts, programming paradigm, etc.) from each code and organize them into a tree structure (i.e., feature tree). It then prompts the LLM to expand the tree by introducing new concepts in the same category or sub-categories. The authors used this method to generate 380K code functions and 53K code files. Using this new data, they finetuned base LLMs to obtain EpiCoder series and demonstrated that these models achieved better performance than existing models on five benchmarks.\n\n## update after rebuttal\nI want to thank the authors for conducting the additional experiments based on my suggestion. My major concern about the unfair comparison has been addressed by the new comparison results using datasets with comparable sizes. The new comparison between EpiCoder and SelfCodeAlign is also helpful. \n\nOn the other hand, while the authors claimed that the newly proposed benchmark is a more challenging/better benchmark than ClassEval etc., it would be more convincing if the authors could evaluate EpiCoder on some existing benchmarks since they are peer-reviewed and widely used. I also suggest the authors increase the number of manually analyzed data points in the test case analysis. 30 data points seem not enough. \n\nNevertheless, my major concerns have been addressed. So I am happy to raise my score from 2 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and for recognizing the contribution of our work. Below, we address your key concerns.\nThe table link [https://anonymous.4open.science/r/epicoder_rebuttal-C619/tables.md](https://anonymous.4open.science/r/epicoder_rebuttal-C619/tables.md) contains all the tables referenced in our responses.\n## Questions\n### Q1: How do you know that the better performance of EpiCoder is due to data complexity and diversity instead of dataset size?\nA: We acknowledge that data complexity, data diversity, and dataset size all contribute to performance improvements. To isolate the effect of dataset size, we randomly sampling 75K data from EpiCoder and comparing the results with MagiCoder (75K) and WaveCoder (20K + 110K). The results, presented in Table 1 of the table link, show 5.4% and 3.0% improvements respectively.  \n\n### Q2: How many generated code functions/files have a reasonable combination of features?\nA: Our data generation process incorporates constraints to ensure feature compatibility, as detailed in Appendix C.4 (line 1663-1664).  When generating tasks, the LLM selects a subset of features that are mutually compatible.  \n\n### Q3: Why not use existing class-level or repo-level code generation benchmarks?\n\nA: Existing benchmarks present several limitations:\n\n- Class-Level Benchmarks (e.g., ClassEval)\n  - These are essentially function-level generation or completion tasks, a simplification of our file-level code generation.\n\n- Repo-Level Benchmarks face two major issues:\n  - Metrics such as Exact Match (EM) and Edit Similarity (ES) are only suitable for base models and not for content-rich instruct models (e.g., CrossCodeEval, RepoBench).\n  - EvoCodeBench's codebase showing focuses mainly on the local context (in the same file) and does not take into account the dependencies of the whole repository.\n\nOur benchmark  fully encompasses complete tasks and generation, covering various instruction formats, making further elaboration unnecessary. Instead of focusing on function-level tasks, we prefer to evaluate the LLM’s ability to generate complete files based on natural language instructions.  \n\n### Q4: How many generated test cases are valid?\n\n- Manual Evaluation:  We manually examined 30 data samples and found that all the generated test cases correctly reflected the task requirements. However, we observed that these test cases tend to be relatively simple and may not cover all edge cases. A concrete example is provided in Appendix C5.2 and C5.3.\n\n- Pass Rate Improvement: According to our statistics, before the first refinement iteration, only 32% of the generated code passed all test cases. After three iterations, the pass rate increased to 61%, demonstrating that the test cases effectively filter out low-quality data and improve overall data quality, even if they do not guarantee 100% correctness.  \n\n### Q5: What is the accuracy of GPT-4o in estimating code complexity?\n\nA: To estimate the accuracy of GPT-4o in assessing code complexity, we conducted pairwise comparisons on a dataset using both human evaluation and GPT-4o, showing an average win rate of 84.4% and 74% for our data and a consistency of 79.375% between human evaluation and GPT-4o.\n\nIn section 4.1, we use both GPT-4o and the sofware engineering method to estimate code complexity, and both approaches consistently indicate the better complexity of our data. We also assessed complexity using DeepSeek-V3-0324 and Llama3.1-70B-Instruct, following the same metrics as in Section 4.1.2. The results align with our original conclusions. Detailed results are in Table 3-6 of the table link.  \n\n## Other Comments  \n\n### Comparison with SelfCodeAlign  \nThank you for acknowledging our contribution. We will discuss SelfCodeAlign in our paper. As you mentioned, the key difference between EpiCoder and SelfCodeAlign is that EpiCoder organizes features into a tree structure and uses evolution to enrich features.  To make a fair comparison, we first sample a subset of our data with same size of SelfCodeAlign and then finetune CodeQwen-Base for comparing with the results in their paper. Table 2 of the table link shows that our data has a 4% improvement.  \n\n### Performance Improvement\nAs you mentioned, we have achieved a 2-3% improvement compared to the second-best baseline, which is already a notable gain at the instruction tuning stage. Additionally, the second-best baseline Qwen2.5-Coder-7B-Instruct utilizes tens of millions of synthetic instruction samples as stated in section 4.2 of their technical report, which demonstrate the effectiveness of our data.  \n\n### Manual Check\nDuring the whole pipeline, we incorporated manual inspection and optimization to ensure that the generated code is reasonable and coherent. We provide concrete examples in Appendix C.  \n\n### Order of Appendices\nThanks for your valuable suggestions and we will adjust it accordingly.   \n\nWe hope our responses effectively resolve your concerns.", "claims_and_evidence": "The claims about the technical approach and novelty are largely reasonable. To the best of my knowledge, SelfCodeAlign is the only work that shares a similar idea of extracting high-level code features (or code concepts as called by the authors of SelfCodeAlign) from seed data and generating new data based on the extracted features. Compared with SelfCodeAlign, this work considers more kinds of code features, represents the extracted features in a nice and clean tree structure, and proposes a new component to enrich the features in the tree structure. So I think this work is novel enough.  \n\n- Wei, Yuxiang, et al. \"SelfCodeAlign: Self-Alignment for Code Generation.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems.\n\nMy main concern is about the claims on the effectiveness of the data generated by the proposed method. The authors compared EpiCoder with existing models like MagiCoder and WaveCoder and attributed the better performance of EpiCoder to the higher complexity and diversity of their synthetic data. However, this is a fair comparison, since the EpiCoder data is much bigger than the synthetic data used by MagiCoder and WaveCoder. Specifically, the EpiCoder data includes 380K code functions and 53K code files, while the MagiCoder data only includes 75K code functions, and the WaveCoder data only includes 111K code functions. It is likely that the better performance of EpiCoder is simply because of the significantly larger finetuning dataset. Furthermore, the performance improvement over the baseline models is not significant (2-3% compared to the second-best baseline). This makes it less convincing that the feature-based synthesis method is really effective.", "methods_and_evaluation_criteria": "The proposed method makes sense. I do have some concerns about LLM hallucinations since this method makes heavy use of LLMs for feature extraction, clustering, and evolution. Yet other methods like OSS-Instruct also makes use of LLMs and it also seems LLMs are able to learn meaningful patterns from noisy data. So I don't think this is a major issue. But some analysis or manual validation would be helpful. \n\nA more significant issue is about the feature sampling and code generation steps in the proposed method. Currently, this method performs weighted sampling over the features in the tree and then prompts the LLM to generate code based on the sampled features. It does not consider whether it makes sense to put some features together when generating code. So it may generate code with potentially irrelevant or even contradicting features. The generated code may be syntactically correct but doesn't make much sense in practice. This may be a potential reason why the synthetic dataset does not gain a significant performance improvement even though its size is 3-4 times bigger than other datasets. I suggest the authors sample some generated code functions/files and conduct a manual analysis to check whether they make sense. \n\nThe last step of the data synthesis method is about iterative refinement. However, it is questionable whether the test files generated by the proposed methods are indeed effective. It is likely that the LLM generates some weak or even invalid test files. While these tests are executable, they do not really examine the generated code properly.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "As mentioned above, a major issue of the evaluation is the unfair comparison to the baseline models like MagiCoder and WaveCoder. The authors need to make their synthetic dataset the same size as the baselines to eliminate this confounding factor. Otherwise, it is not convincing that EpiCoder's better performance is really due to the diversity and complexity of the synthetic dataset. \n\nSince SelfCodeAlign also leverages code concepts to generate data and achieves good performance, it is a more related and state-of-the-art method to compare with. \n\nThe authors constructed a new benchmark called XFileDep for file-level code generation. Since there are many known class-level or repo-level code generation benchmarks, it is questionable why the authors chose to create a new benchmark instead of using existing benchmarks, especially given that XFileDep is not carefully evaluated. The authors should evaluate EpiCoder on at least one known and peer-reviewed class-level or repo-level benchmark. \n\nIn Section 4.1.2, the authors used GPT-4o to estimate code complexity in four dimensions. There is no evaluation of the accuracies of GPT-4o. If GPT-4o is not very accurate, it doesn't make much sense to analyze or interpret the results generated by GPT-4o.", "relation_to_broader_scientific_literature": "The proposed method can be inspirational to data synthesis research beyond code generation.", "essential_references_not_discussed": "As discussed in my comments to claims, a very related method is SelfCodeAlign. The authors should discuss this work and compare EpiCoder with SelfCodeAlign.\n\n- Wei, Yuxiang, et al. \"SelfCodeAlign: Self-Alignment for Code Generation.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "The order of the appendencies is confusing. It would be easier to follow if the authors first showed the prompts for each component in the data synthesis method, followed by additional experiments and results.", "questions_for_authors": "1. How do you know that the better performance of EpiCoder is because of data complexity and diversity instead of data size?\n\n2. How many generated code functions/files have a reasonable combination of features?\n\n3. Why not use existing class-level or repo-level code generation benchmarks?\n\n4. How many generated test cases are valid?\n\n5. What is the accuracy of GPT-4o in estimating code complexity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQblhxD5ir", "reviewer": "Reviewer_n7np", "summary": "This paper introduces EpiCoder, a feature tree-based framework for code generation that addresses diversity and complexity in generated code. The authors propose a hierarchical feature to represent features like concepts used in the code. The framework consists of three components: (1) Feature Tree Extraction, where features are extracted from seed data and organized into a tree structure, (2) Feature Tree Evolution, which iteratively expands the tree to increase diversity beyond the seed data, and (3) Feature Tree-Based Code Generation, which samples from the tree to create code conditioned on a sampled subtree. The generated code can range from single function to file levels. They not only show good performance on function-level benchmarks including HumanEval, MBPP, and BigCodeBench, but they also created a new file-level benchmark XFileDep to evaluate the performance of their method, showing great performance compared to other open models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and for recognizing the novelty and comprehensiveness of our work. We address your concerns below.\n\n## 1. Train/Test Leakage in Function-Level Benchmarks  \nWe address train/test leakage analysis in Appendix B.2 (Figure 9), demonstrates that EpiCoder has a low risk of data leakage. We further validates our low potential leakage in EvoEval and you can view the updated figure at the following anonymous link:  \n[`https://anonymous.4open.science/r/epicoder_rebuttal-C619/leakage_analysis.png`](https://anonymous.4open.science/r/epicoder_rebuttal-C619/leakage_analysis.png)\n\n## 2. Potential Bias in the XFileDep Benchmark  \nThe details of the XFileDep construction process are in Appendix A.2. To mitigate potential bias, we have implemented the following measures:\n\n- **Pipeline Difference**: More filters and human checks make the distribution differ.  \n- **Data Isolation**: The benchmark data is strictly separated from the training data to prevent direct overlap.  \n- **Similarity Filtering**: We apply similarity-based filtering to remove benchmark data that exceeds the leakage threshold according to embedding similarity, reducing potential bias.  \n- **Data Format Difference**:  \n  - In Supervised Fine-Tuning (SFT), our model generates the entire code given the task.  \n  - In the benchmark, the model generates code based on the key class/function name/docstring in the file.\n\nWe believe these measures help ensure that XFileDep serves as a fair and valuable benchmark for evaluating file-level code generation.", "claims_and_evidence": "Their claim that hierarchical feature trees can enable the generation of more complex and diverse code is supported by the experimental results. The authors demonstrate performance improvements on multiple standard benchmarks and provide analysis of complexity metrics compared to existing approaches. They also show that their approach can handle multiple levels of complexity, from function-level to file-level generation with the newly proposed XFileDep benchmark.", "methods_and_evaluation_criteria": "They address the train/test leakage problem by using the EvoEval benchmark to show that their method is not overfitting the benchmark data. Although the score on EvoEval showing the model has a similar score to closed models like Claude 3 is a bit strange.\nThe proposed XFileDep benchmark seems like a good benchmark for file-level code generation. There are details about the benchmark design in the appendix. One issue is that since it uses a similar synthetic generation pipeline, it may unfairly favor their own model as the finetune data may be more aligned with the distribution of this benchmark dataset.", "theoretical_claims": "N/A: The paper is an empirical studies on synthetic data generation for code and finetuning LLMs.", "experimental_designs_or_analyses": "The experiments for the function-level code follow the standard coding benchmarks. The experiments for the file-level code are run on their own proposed XFileDep benchmark.", "relation_to_broader_scientific_literature": "The work is related to many code synthetic data generation methods and finetune work like WaveCoder and Evol-Instruct, etc.", "essential_references_not_discussed": "Most of the related work is discussed.", "other_strengths_and_weaknesses": "The paper is well written and the method is novel and simple. The experimental results are comprehensive. The main weakness is that the function-level benchmark may include a train/test leakage problem, and I don't think the EvoEval as a synthetic benchmark is a gold standard to fully address this issue. The file-level benchmark is very interesting, but it being synthetic data coming from a very similar pipeline may give unfair advantage to the proposed method.", "other_comments_or_suggestions": "Please see the above sections for some comments and suggestions.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WejjEtr3nJ", "reviewer": "Reviewer_eNxJ", "summary": "This paper presents EpiCoder,  a novel framework designed for code generation, addressing the limitations of existing methods that rely on code snippets as seed data. It introduces a feature tree-based synthesis approach that captures hierarchical code features, enhancing complexity and diversity in generated code. By refining a structured feature tree, EpiCoder allows precise control over code synthesis, supporting function-level and multi-file scenarios. Extensive experiments demonstrate that EpiCoder-trained models achieve state-of-the-art performance on multiple benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We address your concerns below and hope these clarifications help resolve them.\n\n## 1. Missing Supplementary Material  \nOur paper includes a 26-page appendix at the end, which provides extensive details on our methodology, implementation, and experiments.\n\n## 2. Implementation Details  \nAppendix C includes a step-by-step breakdown with detailed prompts and execution examples for better understanding. Additionally, we plan to **open-source our code and data** as soon as possible (in a month) while adhering to anonymity policies, ensuring full reproducibility. We hope this addresses your concern.\n\n## 3. Rationality of Feature Combination  \nAs detailed in Appendix C.4 (line 1663-1664), our approach ensures that the LLM selects mutually compatible feature subsets when generating tasks. This guarantees that all generated data samples maintain a reasonable and coherent combination of features.\n\n## 4. Advantages of the Feature Tree  \nOur feature tree is constructed based on real code, leveraging structured modeling of the hierarchical relationships between code features to explicitly capture semantic associations among code elements. Furthermore, by utilizing the hierarchical topology of the feature tree, we can synthesize new data that has not appeared in real-world scenarios (seed code corpus), thereby expanding the model's generalization boundary for complex code patterns.  \n\nAs mentioned in the introduction (line 70-84), the key advantages are:  \n\n### (1) Controllable Complexity  \nWe control complexity by adjusting the tree's shape, such as depth and width. In contrast, using independent features relies solely on increasing feature count, which often leads to incompatible features or unnatural combinations that do not reflect real-world scenarios.  \nSection 3.4 (Figure 5) and Appendix A.3 show the effectiveness of feature tree for generating complex file-level and repo-level code data.\n\n### (2) Targeted Learning  \nFor example, if we need to generate some data focused on data processing, we can adjust the distribution to increase the probability of sampling the node \"data processing\" and its subnodes. This structured relationship is not possible with independent features, where it is unclear which features are related to \"data processing\".\n\n### (3) Evolution Efficiency  \nThe tree structure provides clear and organized directions (depth and breadth) for evolution, making the process more efficient and achieving broader coverage.  An example is shown in Appendix C.3.\n\nBesides, in our response to Reviewer AwKV, we have added the comparison with SelfCodeAlign [1], which follows an approach closer to independent feature-based methods. The result is shown in Table 2 of [`https://anonymous.4open.science/r/epicoder_rebuttal-C619/tables.md`](https://anonymous.4open.science/r/epicoder_rebuttal-C619/tables.md), and the 4% improvement highlights the effectiveness of our approach.\n\n---\n\n## References  \n[1] Wei Y, Cassano F, Liu J, et al. \"SelfCodeAlign: Self-Alignment for Code Generation.\" *NeurIPS*, 2024.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.  Epi-coders are evaluated on several common code benchmarks.", "relation_to_broader_scientific_literature": "This work is related to code generation for large language models and fine-tuning code instructions for llms.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**\n1. The authors propose a novel method to synthesize code instruction fine-tuning data. \n2. This method could generate diverse Instruction data, furthermore, it could be adopted to repo-level code generation\n\n**Weaknesses**\n1. The method of constructing a feature tree is very complicated, and the description in the main text is not detailed enough and difficult to understand. \n2. Many key parameters for building the tree are missing.  It’s difficult to reproduce or follow without code.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. What do you think are the advantages of building features through trees compared to directly extracting independent features?\n2. When generating task data, how to ensure the rationality of the provided feature combination, for example, there are some contradictory features.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hYdWsDW7k", "reviewer": "Reviewer_Sgyp", "summary": "The paper introduces *EpiCoder*, a novel approach to enhancing code-generation performance through hierarchical feature trees extracted from seed code. Empirical results demonstrate that EpiCoder surpasses similarly sized baselines in functional correctness (measured via pass@k) and complexity (measured using Halstead complexity and LLM-as-a-Judge). Additionally, the authors explore EpiCoder's potential to handle code generation tasks from function-level generation to multi-file and repository-level code synthesis.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback on our paper.\n\n## 1. Additional References\nWe appreciate your suggestion and will include discussions on them in our paper. Li et al. (2024) explore the use of LLMs for rewriting code to enhance code search performance, while Koziolek et al. (2024) propose a retrieval-augmented method for controlled code generation.\n\n## 2. Necessity of Synthetic Data\nCertain types of data are extremely scarce in real-world scenarios, making synthetic data an essential and widely adopted approach in both academia and industry. For example, Qwen2.5-Coder [1] utilizes tens of millions of synthetic instruction samples, and models like DeepSeek-V3 [2] and R1 [3] also incorporate synthetic data during training. For code instruction data, while collecting raw code is relatively easy, obtaining well-structured tasks along with their corresponding solution code is significantly more challenging. Other works also adopt synthetic code instruction data, such as WaveCoder [4], MagiCoder [5], SelfCodeAlign [6], etc.\n\nWhile potential of hallucinations are inherent challenges in any synthetic data approach, we have implemented several measures to mitigate these issues, including frequency-based distribution adjustment, verification through test cases, and enhancing complexity and diversity. Therefore, we believe these concerns are not specific weaknesses of our method, but rather common challenges faced by all synthetic data approaches. In addition, recently there is an interesting work by Stanford University [6] on studying which is the critical factor of model’s benefitting from synthesized data. The main conclusion shows that the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor, which suggests that although hallucinations do exist in the synthesized data, these data may still improve the models because of some other factors we must not neglect.\n\nRegarding the domain shifting risk of feature tree evolution, we must admit that every coin has two sides.\n\nFor the positive side, we curate feature trees from seed data, which has been under certain preprocessing pipeline and will be unable to capture the entire domain distribution of “real word data”. Thanks to the evolution of feature trees, such innate problems can be alleviated.\n\nFor the negative side, evolution on feature tree will involve noises.\n\nHowever, to alleviate the problem of evolution noises, we apply filtering in feature combination when generating new questions. Furthermore, the results of training on our synthesized data can present consistent improvements across multiple benchmarks, which proves that the benefits outweigh the drawbacks.\n\n## 3. Definition of Feature and Feature Tree\nFeatures refers to abstractions of code. We organize features into a tree structure based on their logical relationships, where parent and child nodes represent a hierarchical containment relationship. To aid understanding, we provide illustrative examples in Figure 1(a) and Appendix C.\n\n## 4. Clarifications on Figure 2\n### Example of Raw Code Sample\nYou can refer to the dataset at bigcode/the-stack-v2 on Hugging Face. https://huggingface.co/datasets/bigcode/the-stack-v2\n\n### Difference Between Clustered Features and Extracted Features\nThe key difference lies in how they are generated. Extracted features are directly obtained during the extraction process, while clustered features are introduced during clustering to ensure structural completeness. For example, in Figure 1(a), we extract features like \"computation,\" \"XOR,\" and \"AND.\" During clustering, we introduce the clustered feature \"Logical Operation\" to connect them in a meaningful hierarchy.\n\n### Evolution Process\nWe use an LLM to guide this process. Relevant prompts and detailed examples can be found in Appendix C.3.\n\n### Code Generation\nAn example of the full generation process is provided in Appendix C.4 and C.5.\n\n### Docker\nThe Docker is an execution sandbox. We will open-source our code along with relevant configuration details.\n\n## 5. Advantages of the Tree Structure\nWe first clarify a misunderstanding: our method does not sample individual features from subtrees but instead samples entire subtrees, ensuring compatibility during code generation. And please refer to our response to reviewer eNxJ to see the clarification on the advantages of the tree structure.\n\n## References \n[1] \"Qwen2.5-Coder Technical Report.\" arXiv 2409.\n\n[2] \"DeepSeek-V3 Technical Report.\" arXiv 2412.\n\n[3] \"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.\" arXiv 2501.\n\n[4] Yu Z, et al. \"WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning.\" ACL2024.\n\n[5] Wei Y, et al. \"Magicoder: Empowering Code Generation with OSS-Instruct.\" ICML2024.\n\n[6] Kanishk Gandhi, et al. \"Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs\" arXiv 2503.", "claims_and_evidence": "Claims made in this submission are supported by experimental evidence.", "methods_and_evaluation_criteria": "Correct to me.", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "Correct to me.", "relation_to_broader_scientific_literature": "This paper builds on existing research in **LLM-driven code generation** by introducing a feature tree–based framework that systematically captures hierarchical semantic relationships. It is Inspired by **abstract syntax trees (ASTs)** yet extends beyond snippet-based approaches. By clustering features and enabling control over depth and breadth, their method moves beyond simple function-level tasks to complex multi-file or repository-level code.", "essential_references_not_discussed": "The key contribution of this paper is a feature tree-based code generation algorithm. It only cites common code LLMs and code synthesis datasets. There are several more relevant works [1, 2] should be discussed.\n\n[1] Li, H., Zhou, X., & Shen, Z. (2024). Rewriting the code: A simple method for large language model augmented code search. arXiv preprint arXiv:2401.04514.\n\n[2] Koziolek, H., Grüner, S., Hark, R., Ashiwal, V., Linsbauer, S., & Eskandani, N. (2024, April). LLM-based and retrieval-augmented control code generation. In Proceedings of the 1st International Workshop on Large Language Models for Code (pp. 22-29).", "other_strengths_and_weaknesses": "## Strengths\n- **Feature Tree–Based Code Synthesis**: \nThe hierarchical “feature tree” approach captures fine-grained code features (e.g., data structures, control flows), enabling adjustable levels of complexity in code generation.\n\n- **Promising Performance**:\nThe authors synthesize 433k instruction data and train EpiCoder, which achieves state-of-the-art performance among comparably sized models in multiple function-level and file-level benchmarks.\n\n- **Extensive Experimental Validation**:\nThe authors conduct extensive experiments to compare their proposed method with baselines, demonstrating their approach outperforms others in terms of Complexity and Diversity.\n\n## Weaknesses\n- **Complex and Resource-Intensive Pipeline**:\nConstructing and evolving a feature tree, then repeatedly refining the generated code through test-and-debug cycles, can be intricate and computationally expensive. An alternative approach—retrieving and filtering real-world code—may offer greater efficiency and control to provide high-quality training data.\n\n- **Potential for Hallucinations or Overfitting**:\nAlthough feature tree–based synthesis can increase the diversity and complexity of generated code, the reliance on evolved features may introduce distribution shifts from the 'real-world' code. This can lead to hallucinations or overfitting, potentially compromising real-world code generation quality.", "other_comments_or_suggestions": "- **Definition of 'Feature' and 'Feature Tree':** Since this is a feature tree-based code generation framework, providing a clear, upfront definition of “feature” and “feature tree” would greatly improve the paper’s clarity and readability.\n\n* **Overview illustration (Figure 2)**: \n    * **A) Feature Tree Extraction**: Could you provide what the raw code samples look like in your code set? Also, please clarify what the 'blue box' represents and how 'clustered features' differ from 'extracted features'.\n    * **B) Feature Tree Evolution**: Since the notion of “feature” remains somewhat unclear, please specify if any restrictions or guidelines govern how new feature nodes are added during evolution.\n    * **C) Code Generation**: If the two histograms represent feature distribution or frequency, it would be helpful to show the exact steps from the feature tree (Section B) to the final code (Section C). Additionally, please explain Docker’s role—is it an execution sandbox? If so, you should include some configuration details. The folder tree in the code block is also somewhat confusing; more explanation would be beneficial.\n    * **Overall**: The overview could be made clearer by indicating what each stage takes as input and produces as output, along with more explicit navigation through the framework.", "questions_for_authors": "- My main concern is about the purpose of the feature tree. In the code generation pipeline, you sample features from the subtrees into a set and feed it to an LLM. This raises the question of whether the tree structure is necessary. Could you clarify why you need such a complex pipeline to construct a tree rather than a simple frequency-based representation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents EpiCoder, which uses a novel feature-tree based synthesis framework to generate more diverse and complex synthetic  data as well as allows for targeted learning by adjusting probabilities of features. The fine-tuned models on these synthetic datasets achieve better performance on many benchmarks. All reviewers appreciated the overall simple and novel idea of using feature tree to represent and generate synthetic code examples for better diversity and complexity, and the experiments show promising improvements. However, there were also some concerns around the complexity of the approach, unfair comparisons with baselines, potential of contradictory features during generation, XFileDep being a bit more biased benchmark, hallucinations during data generation and potential data leakage. The author response helped alleviate many of these concerns. It would be great to incorporate the discussions and detailed feedback from reviews in the final version of the paper.
Decision: Accept (poster)