MaintainCoder: Maintainable Code Generation Under Dynamic Requirements
Authors: Zhengren Wang
Rui ling
Chufan Wang
Yongan Yu
Sizhe Wang
Zhiyu li
Feiyu Xiong
Wentao Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Code Generation
Maintainability
Design Patterns
Requirement Evolution
Software Engineering
Primary area: applications

Abstract: Modern code generation has made significant strides in functional correctness and execution efficiency. However, these systems often overlook a critical dimension in real-world software development: \textit{maintainability}. To handle dynamic requirements with minimal rework, we propose \textbf{MaintainCoder} as a pioneering solution. It integrates the Waterfall model, design patterns, and multi-agent collaboration to systematically enhance cohesion, reduce coupling, achieving clear responsibility boundaries and better maintainability. We also introduce \textbf{MaintainBench}, a benchmark comprising requirement changes and novel dynamic metrics on maintenance efforts. Experiments demonstrate that existing code generation methods struggle to meet maintainability standards when requirements evolve. In contrast, MaintainCoder improves dynamic maintainability metrics by more than 60\% with even higher correctness of initial codes. Furthermore, while static metrics fail to accurately reflect maintainability and even contradict each other, our proposed dynamic metrics exhibit high consistency. Our work not only provides the foundation for maintainable code generation, but also highlights the need for more realistic and comprehensive code generation research. Resources: https://github.com/IAAR-Shanghai/MaintainCoder.

Maintain Maintainable Code Generation Under Dynamic Requirements
Zhengren Wang 1.33 Rui Ling!,, Chufan Wang Yongan Yu
Sizhe Wang', Zhiyu Li Feiyu Xiong', Wentao Zhang" "Center for Data Science, Peking University "McGill University Center for LLM, Institute for Advanced Algorithms Research, Shanghai
wzrostu, pku, edu. en, {lizy, xiongfy 1 Biaar ac. cn, wentao. zhang8pku. edu. cn
Abstract
Modern code generation has made significant strides in functional correctness and execution efficiency. However, these systems often overlook a critical dimen- sion in real-world software development: maintainability. To handle dynamic requirements with minimal rework, we propose MaintainCoder as pioneering solution. It integrates the Waterfall model, design patterns, and multi-agent col- laboration to systematically enhance cohesion, reduce coupling, achieving clear responsibility boundaries and better maintainability. We also introduce Maintain- Bench, a benchmark comprising requirement changes and novel dynamic metrics maintenance efforts. Experiments demonstrate that existing code generation methods struggle to meet maintainability standards when requirements evolve. In contrast, MaintainCoder improves dynamic maintainability metrics by more than 60% with even higher correctness of initial codes. Furthermore, while static metrics fail accurately reflect maintainability and contradict each other, our proposed dynamic metrics exhibit high consistency. Our work not only pro- vides the foundation for maintainable code generation, but also highlights the need for more realistic and comprehensive code generation research. Resources: com IAAR Shanghai/Maa http//MMaiaaninCCocc...
Introduction
"The Only Constant in Life is Change."
Heraclitus
The advent of large language models (LLMs) has revolutionized code generation [20] 44 [1], with modern tools like GitHub Copilot and ChatGPT demonstrating remarkable capabilities in synthesizing functionally correct code. Benchmarks like HumanEval (7). MBPP (28), and SWE-Bench [21] have driven these advancements by measuring correctness through static test cases. However, this narrow focus overlooks critical dimension of real-world software engineering: maintainability- the ability to adapt code to evolving requirements with minimal rework.
Maintainability Crisis Heraclitus' philosophy, "The only constant in life is change," " reflects a truth painfully evident in software evolution: as user needs evolve, markets shift, and technologies advance, software undergo perpetual changes. This oversight on maintainability leads to the dangerous reality-code that works today but becomes prohibitively expensive adapt tomorrow [6]. For instance, the collapse of Knight Capital exemplifies this crisis-$440 million lost minutes due to
Equal contribution; * Corresponding author.
The first author completed this work during an internship at IAAR.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

unmaintained legacy code [33]; the S100 billion remediation cost for Year 2000 Problem revealed how short-sighted design decisions create exponential future costs B3. Industry studies also quantifies the severity: 60-80% of software lifecycle costs stem from post-deploymen maintenance (29) A] due to structural deficiencies like high coupling and low cohesion that amplify rework effort. Another example is online multi-round code generation. Due to ambiguous specifications and human-AI communication barriers, the initial codes usually deviate from final requirements, while subsequent refinements either are prone to errors or inadvertently discard previously satisfied requirements [27]
Motivation Despite taking up the lion's share of software development resources, maintenance is the least studied phase of software development [3S 9 Although recent benchmarks have expanded evaluations to diverse dimensions like readability and efficiency [43] [88, yet retain static test cases, The static test cases fails capture the iterative nature of software evolution, and ill-suited for maintainability evaluation. Moreover, traditional software metrics like cyclomatic complexity and maintainabilityy index, still remain limited to static code analysis, failing to capture the dynamic scenarios. Meanwhile, code generation systems from foundation models like CodeLLaMA [32) to multi-agent frameworks like SWE-agent [41] optimize for task completion but neglect structural qualities critical for maintainability. Therefore, fundamental gaps challenges persist: (1) No benchmark quantifies maintainability under requirement evolution cycles; (2) No method system- atically applies software engineering principles (e.g.. design patterns) to enhance maintainability; (3) No discussion on the interaction between correctness and maintainability.
In contrast to naive implementation, structured approaches like the Waterfall model and design patterns are proven be effective to improve maintainability during decades. For example, Dong et al. [10] confirms the benefits of the waterfall model in initial code generation, with 29.9-47.1 relative improvement on correctness; Hegedus al. [ reveals a very high Pearson correlation of 0.89 between design patterns and software maintainability, via the analysis of more than 300 revisions of JHotDraw system. Given these challenges and opportunities, we propose MaintainBench and MaintainCoder as a paradigm shift toward maintainable code generation, bridging the gap between static correctness and dynamic maintainability.
Contributions
Our main contributions are threefold:
Dynamic Benchmark:
We introduce MaintainBench., the first benchmark assessing code maintainability through requirement evolution cycles. Constructed through systematic exten- sion of well-established benchmarks (HumanEval, MBPP, APPS, CodeContest, xCodeEval), it incorporates diverse requirement changes with expert-curated test cases. Novel metrics quantifyy modification efforts and structural adaptability dynamically.
Systematic Generation: We introduce MaintainCoder as a pioneering solution. It integrates the waterfall model with multi-agent collaboration and classical design patterns. Specialized agents conduct requirements analysis, modular decomposition, and pattern application to enforce high cohesion, low coupling, single responsibility, and maintainability.
Extensive experiments reveal that (1) Previous methods suffer from significant maintainability degradation under dynamic requirements (low pass and high code changes), even for multi-agent systems such as AgentCoder and MapCoder. (2) MaintainCoder not only improves maintainability metrics by up to +60%, but also enhances the correctness of initial codes robustly. (3) Static metrics not only fail to accurately reflect maintenance efforts, but also exhibit contradictory trends among different metrics. In contrast, the dynamic metrics proposed by MaintainBench are more effective and consistent.
Empirical Insights:
2 Related Work
Code Generation Benchmarks Existing research on code generation has been propelled by
benchmarks emphasizing functional correctness. Pioneering works span multiple levels of complexity: function-level tasks, such as HumanEval [8] and MBPP [3; competition-leww problems, as addressed
by APPS [14 and CodeContests (33: and repository-scale challenges tackled by SWE-Bench [21]
and RepoBench 25 Despite these advancements, these benchmarks predominantly use static test cases measuring correctness through execution pass rates. Although recent efforts, such as RACE 43]
and EffiBench [18), have expanded the evaluation include readability and efficiency, yet remain

---

constrained by static evaluation, failing to capture the dynamic nature of software development. Even SWE-Bench, which evaluates dynamic problem-solving via GitHub issue resolution, focuses on challenging task completion capabilities rather than assessing code maintainability itself-i.e. i.e., generating inherently maintainable code from the outset. MaintainBench bridges this critical gap with metricization of code maintainability through requirement evolution cycles, where 80% of software costs occur in real-world applications 229 5]
Code Generation Systems Modern code generation systems have evolved from three key aspects: foundation models, instruction-tume variants, and multi-agent frameworks. Foundation models such as AlphaCode [23] and CodeLLaMA [2]. established fundamental code synthesis capabilities through large-scale pretraining on code corpora. Building on these foundations, instruction-tunet variants like WizardCoder 26 and Magicoder (88] improved context understanding through data
distillation and complex instruction augmentation. The frontier now shifts to multi-agent frameworks:
MetaGPT [IS] and ChatDev (30 employ standardized outputs to coordinate multiple agents, while
SWE-agent [41] and OpenDevin 37 focus on repository-level task decomposition. AgentCoder [17] and MapCoder [19] introduce specialized testing agents and retrieval process respectively. EvoMAC
[6 proposes sell-evolving multi-agen collaboration networks to optimize code through multiple iterations. However, all of them overlook maintainability, and prone generate brittle code that difficult to maintain. MaintainCoder bridges this gap by integrating principles like the waterfall model and classical design patterns.
3 Method
3.1 Problem Formulation
Task Definition Previous code generators optimize for immediate correctness but neglect software engineering's 's long-term challenge: evolving problem series PP.,P Pa]
drive iterative code adaptations (Co, C1,... ..... Cz which incurs maintenance cost MMC; + C++1). Current approaches treat each P. independently. and ignore structural deficiencies that accrue technical debt. We investigate code generator & producing Co- ((P)) with both high correctness and maintainability.
Maintainability Measurement Previous works measure maintainability through static analysis indexes, such as Halstead volume [12], cyclomatic complexity maintainability index. However, ultimate goal of maintainability is to reduce maintenance efforts. We thus formalize dynamic maintainability through cumulative maintenance efforts with discount factor
n-1
M(C))
Σ MCC
Given maintainability's s intrinsic nature, we posit that limited probing can substitute unlimited requirement evolution simulations. We employ Monte Carlo estimation and first-order truncation as the proxy to address infinite expectation horizons and persistent requirement changes, which reduces both computational cost and estimation variance, yielding M(Co) s N E M(C. + Ct).
3.2 MaintainBench: A Dynamic Benchmark for Code Maintainability
In this section, we introduce MaintainBench, a novel benchmark designed to evaluate code maintain- ability in response to evolving software requirements. Unlike traditional code generation benchmarks that focus solely on correctness, MaintainBench evaluates how effectively models can adapt existing code to meet changing requirements. MaintainBench consists of five carefully curated datasets: HumanEval-Dyn. MBPP-Dyn, APPS-Dyn, CodeContests-Dyre and xCodeEval-Dyn, comprising over 500 Python programming data of diverse difficulty levels. Each extends established benchmarks with systematic requirement changes. The overview of construction process is shown in Fig. [
3.2.1 Data Selection and Preprocessing
MaintainBench comprises code samples of varying difficulty levels, roughly entry-level, mixture- level, and competition-level. Specifically, it extends five widely-used code generation benchmarks: HumanEval 77, MBPP (28), APPS [4]. xCodeEval [2]. and CodeContests (23].

---

Data prepecesinng
Requieeeet Chorge Groendtion
Seluion Text Caecolation
Pinal Chipps
Misteral
Compelttte
Myr Kleer
Requiienunt chango patterns
Prakkan Chuck
Find Cleck
Falled
Marually
Tau Coees
feo
LIMs
Standantic Sermal
Figure 1: The systematic construction of MaintainBench. 1) Data preprocessing selects and curates problems from existing benchmarks into entry-level, mixture-level, and competition-level difficulties. 2) Requirement change generation applies four systematic patterns (functional extensions, interface modifications, data structure transformations, and error handling enhancements) to create evolved problem variants. 3) Solution-test co-evolution refines solutions and test cases together through iterative execution and refinement. 4) Quality check employs both automated test and multi-stage expert review to ensure reliability, correctness, and alignment with intended requirement changes.
Entry Level For entry-level difficulty, we select problems from the HumanEval and MBPP datasets, which are designed to be solvable by newbie programmers. These datasets are widely used to evaluate the code generation capabilities of LLMs, covering topics such as language comprehension, algorithms, basic mathematics, programming fundamentals, and standard library functionality, Each problem includes task description, a reference solution, and set of automated test cases. We sample 30 problems from each dataset randomly, and extend them to 120 new problems by systematically modifying their requirements.
Mixture Level For mixture-leve difficulty, we extend the APPS dataset, which contains problems from introductory to collegiate competition levels. For example, problems from Kattis Katth&TTith with difficulty <3 are introductory, 3-5 are interview-level, and >5 are competition-level We start with a random subset of 50 problems and expand them to over 200 new problems. We standardize the dataset by converting original solutions to functional form and test cases to Python assertions. E.g., to avoid the keyboard input, utilize GPT-4o to generate functional implementations with manual review.
Competition Level The competition-level dataset includes problems from CodeContests and XvadeEval. CodeContests focuses on competitive programming tasks, while xCodeEval provides a large-scale executable dataset with varying levels of difficulty. To distinguish this level from entry-level and mixture-level tasks, we select 30 high-difficllty problems from each dataset and extend them to over I 120 new problems. As with the mixture-level dataset, we standardize the data format using -.e--geee solutions, followed by manual verification to ensure correctness.
3.2.2 Extended Data Generation
Building our preprocessed datasets, the core innovation of MaintainBench is the systematic application of four requirement change patterns, formalized as:
P1, S Ti = Modify (Po, So, To, mz), VIT E II Si, Ti = Refine St, TY)
where Po. Sy, and To, represent the original problem, reference solution, and test cases, respectively. The Modify Po, Su. To, n,) applies one of four requirements change patterns from our pattern set:
m { Text: Tint; Tdst; Terr
and generates an initial extended versions. This transformation process is designed to simulate realistic software evolution scenarios where developers must modify existing code to accommodate
https://open.katti.coom/

---

changing requirements rather than implementing solutions from scratch. The refinement process co-evolves both new solution S and test cases Ti with automatic evaluation and manual curation loops, ultimately yielding the final reference solution and test cases for the modified requirement n.
Requirement Change Generation According to the ISOЛEС/ЛЕЕ 14764:2022 specification [34, software maintenance can be roughly classified into four types: corrective maintenance, preventive maintenance, adaptive maintenance, perfective maintenance, where the latter two categories comprise about 80 percent of software maintenance [36]. Hence. we craft four distinct requirement change patterns capture different software evolution scenarios:
Functional Extensions[ Teul: Functional extensions increase complexity by introducing realistic additional requirements while preserving the relevance with the original problem. Effective solutions must both comprehend the original code and appropriately extend its problem-solving capabilities. For example, the extended problem should involve self- invoking 42), i.c. calls to the existing functions, modeling the interactions between functions or classes. Functional extensions cover both adaptive and perfective maintenance.
Interface Modifications( Interface modifications further enhance the diversity of gen- erated problems, and examine the adaptability of original solutions in protocol dimension. E.g., the API changes caused by the update of the external dependency library. The modified problem remains closely related to the original, but introduces changes to input parame- ters, return types, or other interface components assess the robustness and flexibility of solutions. Interface modifications correspond to adaptive maintenance.
Data Structure Transformationns Data structure transformations require the generated problem adopt different data structures where the description explicitly specifies the modifications of the data representation. This transformation mimics real-world scenarios in which software systems evolve accommodate efficiency, scalability, or compatibility constraints. Data structure transformations primarily align with perfective maintenance.
Error Handling Enhuncements : Error handling enhancements explicitly intro- duce required error-handling mechanisms, which capture specific error types such as ZeroDi onError. IndexErrox and other problem-specific exceptions. The en- hanced new problem better reflects the fundamental aspect of reliability and maintainability. Error handling enhancements encompass corrective, preventive, and perfective maintenance.
Solution-Tes Test Co-evolution We utilize GPT-4o to generate initial versions of each transformation, including preliminary solution S and test cases Ty, followed by an iterative refinement process as validation. That is, we execute S against T using Python interpreter to assess correctness. For any failing tests, we conduct manual expert reviews to diagnose and resolve issues in both the solution and test cases, refining Si into Si and i into Ti. If discrepancies persist, the execution-review cycle is repeated until all test cases pass successfully.
More details on solution-test co-evolution and quality check are left in Appendix B for space reason.
3.3 MaintainCoder: A Multi-Agent System for Maintainable Code Generation
As shown in Fig. MaintainCoder delivers highly maintainable codes via novel multi multi-agent system that mirrors the human software development lifecycle. It designs an orchestrated pipeline of LLM agents, each specializing in distinct phases of software development while maintaining contextual awareness through inter-agent communication empowered by AutoGen framework 39
3.3.1 Code Framework Module
This module transforms user requirements into maintainable architectural blueprints.
Requirements Analysis Agent The requirements analysis agent is tasked with in-depth analysis of software requirements. It decomposes the problem step by step through chain-of- thouughss prioritizing user goals and rethinking practical constraints. The agent receives user requirements, extracts pivotal goals, identifies core functions, highlights key challenges, and proposes high-level solutions. The output analysis report provides concise guidance for follow-up agents. This analysis process avoids unnecessary complexity and focuses on high high-priority tasks and high-level computational logic.
5

---

Bedgg Podiee. rrgert
Tectbock Talk
pridam ad
Meetback
Ten
hes codl
Yel unflect das Tiy
Modak hace
Oer
Reqainment
Framanock
Flcal Code Framsnork
Optinied Code Sobaiii
cover makes
comeuctty
Modde
Figure 2: Overview of MaintainCoder. MaintainCoder features best practices like the Waterfall model, classical design patterns, and iterative refinement mechanisms for critical stages. 1) The Code Framework Module systematically transforms user requirements into an optimized, maintainable architectural blueprint; 2) The Code Generation Module implements the blueprint into production- ready code, rigorously adhering to both the framework specifications and initial user requirements.
Design Pattern Selection Agent The design pattern selection agent serves as the software architect, focusing on selecting appropriate design patterns for each functional module. It receives functional modules and key challenges from the requirements analysis agent, evaluates the overall architecture and module interactions, and selects the most suitable design pattern. This selection prioritizes patterns that promote modularity, reduce coupling, and enhance scalability and reusability. The agent also suggests alternative patterns with corresponding applicable scenarios. The output a structured list of modules, each including the module name, main design pattern, reasons for selection, and alternative patterns with applicable scenarios.
Framework Design Agent The framework design agent agent constructs a modular and robust code framework based on functional modules and selected design patterns. This agent not only designs a preliminary class structure adhering to single-respnnsibiitty principles, but also clarifies dependencies enhances modularity, reusability, and maintainability. It will revise the framework in response to feedback from the framework evaluation agent.
Framework Evaluation Agent Given user requirements, analysis reports, and class structures/re- lationships as inputs, it reviews design clarity, scalability, performance, and conformance to best practices. This agent identifies issues related to coupling, cohesion, and reusability, avoids overly fine-grained modules, i.e. over-fragmentation. Finally, it delivers actionable suggestions focused on major improvements that impact performance and scalability, which will be returned to the framework design agent for improvement loop.
3.3.2
Code Generation Module
This module implements blueprints into executable codes, also guided by requirement analysis.
Code Generation Agent The code generation agent converts detailed framework designs into complete, functional Python code. ensures readability and maintainability with clear comments and documentation. By carefully analyzing the original requirements, requirements analysis and final framework design, it creates code structures conforming to PEP and PEP 257 specifications. Comments will focus on code purpose, design choice, and especially non-obvious logic, rather than redundantly describing the obvious content. The generated code then includes appropriate class structures, methods, and uniformly named interface functions for testing. After the initial generation, a test sample selected from the test set is inserted as assertion for iterative debugging. The overall goal is to produce fully functional, understandable, and expandable code.
Code Optimization Agent The code optimization agent refines the generated code to meet expected functional and performance requirements. By analyzigg the problem requirements, framework design,

---

original code and test cases, this agent can identify potential problems or room for improvement. First, the agent is forced to thoroughly understand the user requirements and framework design to ensure modifications align with the overall intent. Next, it reviews the original code for logic correctness and interface functions, creating any missing ones. Then, the code is executed to gather feedback on syntax errors, test failures, unexpected behaviors, etc. Through analysis in chain-of-thougee agent diagnoses the root causes, makes necessary modifications (fixing syntax errors, adjusting logic, handling boundaries, optimizing performance), and retests. This iterative process continues until the code meets requirements and passes tests, ensuring functional, efficient, and reliable solutions.
the
4 Experiments
Experimental Setup For a holistic assessment of maintainable code generation capabilities, we primarily utilized the MaintainBench benchmark introduced in this work, well as the original datasets with various metrics. Experiments were conducted in two phases. In phase I, we generate initial code Co for the original problem Po using MaintainCoder or baseline methods. During this phase, static maintainability metrics were recorded; In phase II, we keep the fixed generator, e.g. GPT--4o-min, to dynamically probe the maintainability of C.. Specifically, we generate modified codes reflecting different types of changes: Co - (Cezt; Cint, Cast, Cerr These variants were then evaluated to compute the dynamic maintainability metrics. For more reliability and statistical significance, we report @ k up to k=5. Please refer to the Appendix D for more details.
Baselines & Metries For comprchensive evaluation, we conduct experiments across diverse baselines including GPT-4o-mini, DeepSeek- V3, Faaas--Preeviee, GPT-4o, and Qwen-Plus 24 40 as well as multi-agent frameworks such as AgentCoder [17] and MapCoder [19]. We consider GPT-4o-mini and DeepSeek- V3 as the backbones, to highlight the benefit of MaintainCoder and explore the performance boundaries respectively. We also investigate Chain-of-Thoughee (CoT) and Self-Planning (Plan), which incentivize reasoning or planning capabilities through explicit prompting engineering. We measure static maintainability with the Maintainability Index (MI) and Cyclomatic Complexity (CC). We proposed several dynamic metrics, including post-mooe cost-modification functional correctness (Pass @ k). the likelihood of generating correct solution after modifications; code change volume (Codeai )) the percentage absolute value of modified lines; and syntax tree similarity (AST sim), the structural similarity between the original and modified abstract syntax trees.
Claude-3.7-Somme Gemini-
4.1 Main Experiments
Generally, MaintainCoder exhibits superior performance across different difficulty levels, with its advantage becoming more pronounced as the complexity increases. Due to page limit, the results on entry-level datasets and case studies are left in the Appendix E for interested readers.
Performance on Mixture-Level Dataset Table both static and dynamics metrics on the mixture-level APPS-Dyn benchmark. For static structure, MaintainCoder achieves higher MI scores than all baselines, while cyclomatic complexities are around 3-3x lower than competing models. For dynamic metrics, it also shows significant advantages:
attains 50.5% P accuracy (15-30 points higher than other variants), 0.797 AST similarity (outperforming the second-best by +28%), and minimizes the relative code differences (29.4%). These results highlight MaintainCoder's 's unique strength, surpassing advanced models like GPT-4o and Claude-3.7 Soauude3.7-Somme 7SSnnee.
Performance on Competition-Leeel Dataset As shown in Tab. the conclusions on mixture-level V3)
or competition-level datasets are largely consistent. For example. MaintainCoderr(Depp) attains 56.7% Pass@5 accuracy (outperforming the second-best by +60%), 0.785 AST similarity, and the lowest relative code changes 33.0%. Notably, MaintainCoder retains low code complexity even on challenging problems, i.e. keep CC scores around 3 while baselines inflate code com- plexity significantly. Surprisingly, multi-agent systems like AgentCoder and MapCoder, despite optimizing single-round correctness, could impair long-term maintainability and degrade second-turn Pass @k Moreover, the impact of prompt engineering methods, such as CoT and Plan, appears to be inconsistent or even random, which highlights the robustness and superiority of our MaintainCoder. Finally, advocate that absolute code changes are less indicative of maintenance costs than relative
7

---

Table 1: Maintainability evaluation on mixture-level APPS-Dyn problem set. The best performance indicated in bold, while the second-best performance is in underline.
Static metrics
Dynamic metrics
Model
MI1
CCC
Pasc (%)
AST
Code (%)
Code 25f
APPS-Dyn
GPT-lo-mini
63.3
5.10
35.5
0.589
140
17.0
GPT-4o-minicaT
5.10
17.1
GPT-4o-miniee
59.2
5.01
34.0
0.618
93.3
17.3
AgeniCoder (GPT-4o-mini)
63.3
5.81
21.0
0.510
66.3
20.1
MapCoder (GITT-o-mmii)
30.5
0.583
73.8
MaintainCoder (GPT-4o-mini)
69,5
2.75
50.5
0.797
29.4
19.0
DeepSeck-V3
61.8
7.59
59.5
0.598
131
22.6
DeepSeek-V3con
7.28
52.0
0.634
119
DeepSeek-V
59.2
6.06
54.5
0.577
130
20.6
Agen (DeepSeek-V3)
60.5
6.68
48.5
0.6111
1144
20.5
MapCoder (DeepSeck-V3)
8.76
48.0
0.659
83.0
19.1
MaintainCoder (DeepSeek-V3)
62.4
3.21
62.5
0.828
29.2
18.6
GPT-4o
63.0
4.58
39.5
0.556
140
Owea-Plus
61.2
5.61
43.5
0.638
137
17.3
Gemini-2.5 S-Flush-Preview
9.00
Claude-3.5 3.5 Sonnet
60.8
4,63
47.0
0.650
103
17.3
Claude-3.7-8 7-Snnnet
59.3
6.65
48.5
0.620
85.2
18.6
Table 2: Maintainability evaluation competition-leve CodeContests Dyn and xCodeEval-DDi problem sets. The best performance bold, while the second-bes one in underline.
Static metries
Dynamic metries
Model
MI
CC
Pass@5 5 (%)
AST
Codeber (%) 1
Code 21.7
CodeContests-DDy
GPT-4o-mini
57.8
6.06
24.2
0.661
90.1
16.6
GPT-to-minicer
5.85
23.5
0.639
96.2
17.0
GPT-4o-minig
54.6
6.62
28.8
0.716
65.6
16.7
AgentCoder (GPT-4o-mini)
62.5
7.28
18.2
0.629
44.9
18.5
MapCoder (GPT-4o-mini)
7.32
0.689
47.5
MaintainCoder (GPT-40-mimi)
65.8
2.68
32.6
0.833
23.2
17.4
DeepSeek-V V3
$5.7
6.80
26.5
0.718
87.2
17.5
DeepSeek-V3ess
53.1
11.7
20.5
0.690
$2.3
21.7
DeepSeek- V3
12.2
17.4
0.614
67.6
26.3
AgentCoder (DecpSeck-V3)
43.8
19.6
16.7
0.670
48.4
28.0
MapCoder (DeepSeck-V3)
49.0
14.6
11.4
0.655
53.0
24.8
MaintainCoder (DeepSeek
3.64
0.788
GPT-4o
57.2
5.28
25.0
0.622
94.2
Owen-Plus
52.1
6.85
28.8
0.738
72.9
18.3
Gemini-2.5 2.5--Flahh-Previi
8.48
18.9
0.714
61.0
19.3
Claudo-3.5-Somne
55.3
6.74
23.5
0.739
51.3
17.5
Claude-3,7 3,7-oonett
53.9
8.74
24.2
0.620
85.2
18.6
XCodeEval-Dyy
GPT-to-mini
56.1
6.44
23.4
81.5
GPT-4o-minica
55.9
6.44
T8.8
0.644
82.4
18.4
GPT-to-minissn
51.1
6.25
22.7
0.705
$3.0
18.0
AgentCoder (GPT-to-mini)
60.9
8.34
18.0
0.624
42.3
22.3
MapCoder (GPPT-4-mmni)
63.6
8.15
15..6
0.702
43.2
21.2
MaintainCoder (GPT-4o-mini)
60.5
2.72
27.3
0.837
21.6
19.8
DecpScok-V3
54.4
7.03
21.9
0.636
92.2
21.2
DeepScek-V3ae V3
11.6
22.7
0.613
$2.0
24.6
DeepSeck-V3ne V3
56.9
11.8
21.1
0,626
69.3
27.6
AgentCoder (DeepScek-V2) V3)
42.2
15.2
11.7
0.690
60.0
32.8
MapCoder (DeepScek-V3)
11.7
0.655
52.8
MaintainCoder (DeepSeck-V3)
57.9
3,47
36.7
0.785
33,0
21.8
GPT-4o
53.7
4.33
32,8
0.680
67.2
17.8
Owen-Plus
7.24
23.4
0.684
70.3
20.4
Gemini-2.5 2.5imi--..--lushhh 5-Flush-Previens
48.6
7.64
22.7
0.750
49.2
20.7
Claude-3, 3,5 Sonnet
511.4
6.43
17.2
0.779
41.7
18.4
Claude-3, 3,7- 7-Sonnet
54.0
8.64
21.9
0.734
50.1
21.4

---

code changes. For example, the software refactoring often inflate code lines but improves long term maintainability. Previous methods compress lines but led to tangled and unmaintainable code.
4.2 Analysis and Discussion
Correctness versus Maintainability Ideally,
Table 3: Pass on the original datasets. Maintain-
effective code design paradigms can enhance not only maintainability but also the correctness of initial code. We evaluate MaintainCoder's Pass on the original problems. The results presented Table validate our hypothesis:
Coder improves the correctness of initial codes.
Model
APPS
CodeContests
XCodeEval
GPT-Ao-mini
MaintaiCCwder (GPT--o-min))
DeepSeck-V V3
66%
48%
75%
MaimainnCoder (DespSeck-3)
69%
51%
77%
MaintainCoder demonstrates improvements in functional correctness, with this benefit becom- ing more pronounced as problem complexity increases, For example, MaintainCoder (GPT-40-mini) achieves an 11% increase on XCodeEval
Qwen-Plus
5616
30%
63%
Gemini- 2.5.22.asssh--eeeeeee
6.%
53%
74%
Claude-3
(from 46% to 57%). which is over double the improvement seen APPS (44% to 48%). Notably, these gains occur even on already high-pue gegh-performing models like DeepSeek-V
Static Metries versus Dynamic Metries For static metrics, CC score measures program con- trol flow, Halstead Volume measures arithmetic logic operations, and MI score integrates them. However, these metrics fail reflect high-level maintainability. For example, AgentCoder and MapCoder achieve high MI metrics, while the second-round correctness was reduced. More confusingly, the changes in MI and CC met- ries are contradictory, For instance, the MI and CC metrics of MapCoder(GPT-4o-nm and 7.32 respectively. both increasing GPT-4o- mini's 57.8 8 and 6.06. This inconsistency is also evident on AgentCoder. In contrast, dynamic metrics such as Pass @k, Code and AST sim demonstrate significant consistency. This mu- tual corroboration advocates that dynamic metrics reflect the maintainability more accurately. Thanks to the construction of WaintainBench the calculation of dynamic metrics has become possible.
Enoiigt
Fil
are 66.1
d81 (00)
1ST
Figure 3: Scatter plot of P and AST eim.
Ablation Studies We conduct ablation studies to investigate the impact of framework evaluation and code optimization. As shown in Table 4 both modules significantly enhance the second-turn correctness, with the code optimization contributing more substantially. For instance, omitting framework evaluation reduces @ass 5 by 25.7 1% relative points while code optimization incurs more pronounced 37. 37.13% performance degradation. Notably, framework evaluation typically involves only one iteration, which serves as lightweight yet impactful component.
Table 4: Ablations on Framework Evaluation and Code Optimization. Both agentic modules con- tribute to final performance. "Perf. Drop" denotes relative performance degradation of Pass@k
Base Model
Framework Evaluation
Code Optimization
APPS.Dyn
CodeContests-Drr
xCodeEval-Dyy
Pussti5 (%)
Perf. Drop (%)
Pass (%)
Perf. Drop (%)
Pass (%)
Perf. Drup (%)
31.82
25.71
GPT-fe-mini
40.00
20.79
28.03
13.97
17.19
37,13
50.50
32.58
27.34
61.00
2.10
33.33
11,99
33.59
8.52
DeepSeck-V3
X
48.50
25.76
23.39
62.50
37.87
36.72
Human Baseline, Reasoning Models and Multi-Agent Systems We benchmarked MaintainCoder against skilled human programmers, the 03-mini reasoning model, and other multi-agent systems such as MetaGPT and EvoMAC. For human bascline, we have recruited participants with competitive

---

programming experience (Codeforces ratings 1700-2300) to complete tasks from the CodeContests- Dyn benchmark in 30 minutes per problem. As shown in Table the code produced by human programmers demonstrated poorer maintainability metrics compared to Al-generated code. Even skilled developers, particularly under the pressure problem-solving common real-world scenarios, can produce less maintainable code. This underscores the value of MaintainCoder in systematically enhancing this vital software quality attribute. Furthermore, MaintainCoder provides significant maintainability enhancements outperforms other multi-agent systems. While frameworks like AgentCoder and EvoMAC focus on solving the immediate problem, MaintainCoder's proactive architectural design, which explicitly embeds software engineering principles, leads to superior long-term maintainability.
when applied to strong reasoning model. It consistently
Table 5: Comparison with human programmers, reasoning models and more multi-agent systems on CodeContest-Dyys MaintainCoder demonstrates superior maintainability. MetaGPT requires a stronger base model like GPT-4. 1-mini for its tool-calling capabilities
Stutic metries
Dynamic metries
Model
MI
CC1
P (%) f AST sim
Code (%)
Code diff
Human Programmers
53.2
8.17
23.5
0.541
112.3
23.3
GPT-4o-mini
57.8
6.06
24.2
0.661
90.1
16.6
AgentCoder (GPT-4o-mini)
62.5
7.28
18.2
0.629
44,9
18.5
MapCoder (GPT-4o-mini)
66.1
7.32
0.689
47.5
19.0
EvoMAC (GPT-4o-mini)
62.6
5.18
0.685
60.1
20.0
MctaGPT (GPT-4.) 1--mi-)
55.2
7.63
30.3
0.760
44.3
18.6
MaintainCoder (GPT-4o-mini)
65.8
2.68
23.2
17.4
03-mini
52.1
11.3
30.3
0.661
101.6
32.9
MaintainCoder (o3-mini)
62.3
3.85
36.4
0.794
27.8
21.3
Computational Cost Analysis We acknowledge that multi-agent pipeline incurs higher initial computational cost. However, we argue this is a justified trade-off for substantially improved long- term maintainability and higher initial correctness. As shown in Table 6 MaintainCoder'ss token consumption is comparable to other multi-agent frameworks like MapCoder and even the reasoning model o3-mini, while being significantly lower than systems like MetaGPT and ChatDev. This initial investment leads to drastic reduction in subsequent human effort required for debugging and refactoring. We provide breakdown of token usage in Appendix for interested readers.
Table 6: Total token usage. The cost of MaintainCoder is comparable to other multi-agent systems.
Dataset
MaintainCoder
MapCoder
03-mini
MetaGPT/ChatDev
GPT-4o-mini
CodeContests
33.1k 29.6k
38.7k
20.8k 21.2k
50k+
2.5k
xCodeEval
23.5k
50k+
2.3k
Robustness w.r.t Pass@k & Phase II generator For a more reliable evaluation, we examine MaintainCoder's robustness with respect to varying k values of @ass@k and different Phase II generators that conduct modifications on initial codes. We leave the experiments in the Appendix
Conclusion
In this paper, introduced MaintainCoder and MaintainBench for maintainable code generation. MaintainBench, the first benchmark dedicated to dynamic maintainability evaluation, covers diverse and systematic modification scenarios and provides standardized testing platform for future research. MaintainCoder pioneers this research line, utilizing multi-agent collaboration with classical design patterns. Extensive experiments demonstrate that MaintainCoder significantly outperforms previous methods on both maintainability and correctness. E.g. more than 60% post-modification Pass @ improvements with even higher initial correctness. The discussed flaws of static metrics also provide insights for the unique value of MaintainBench. Future work should not only expand Maintain into larger scale and more diverse modification types, also include evaluation on more complex and repository-level tasks like SWE-Bench
10

---

Acknowledgements
This work is supported by the National Key R&D Program of China (2024YFA1010003) National Natural Science Foundation of China (92470121, 62402016), CAAI-Ant Group Research Fund, and High-performance Computing Platform of Peking University.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088744 2023.
[2] Anthropic. The claude model family: Opus, sonnet, haiku. 2024. URL https://api Semanticscho.ie htt.schco......oooooooooo.......
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv: 1at::20080077553 2021.
[4] Rajiv D Banker, Srikant M Datar, Chris F Kemerer, and Dani Zweig. Software complexity and maintenance costs. Communications of the ACM, 36(11):81-94 1993.
[5] Keith H Bennett and Václav T Rajlich. Software maintenance and evolution: roadmap. In Proceedings of the Conference on the Future of Software Engineering, pages 73-87, 2000.
[6] Wallace R Blischke, DN Prabhakar Murthy, et al. Case sludies in reliability and maintenance, volume 480. Wiley Online Library, 2003.
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.00 03374,, 2021.
[9] Sharon Christa, V Madhusudhan, V Suma, and Jawahar J Rao. Software maintenance: From the perspective of effort and cost requirement. In Proceedings of the International Conference on Data Engineering and Communication Technology: ICDECT 2016, Volume 2, pages 759-768. Springer, 2017.
[10] Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engineering and Methodology, 33(7):1-38, 2024.
[11] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.035:3 2023.
[12] T Hariprasad, G Vidhyagaran, K Seenu, and Chandrasegar Thirumalai. Software complexity analysis using halstead metrics. In 2017 international conference on trends in electronics and informatics (ICEI), pages 109-1113, IEEE, 2017.
[13] Péter Hegedús, Dénes Bán, Rudolf Ferenc, and Tibor Gyimóthy. Myth or reality? analyzing the effect of design patterns on software maintainability. In Computer Applications for Software Engineering, Disaster Recovery. and Business Continuity: International Conferences, ASEA and DRBC 2012, Held in Conjunction with GST 2012, Jeju Island, Korea, November 28-December 2, 2012. Proceedings, pages 138-145. Springer, 2012.
[14] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0388 2021.

---

[15] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin. Liyang Zhou, et Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308 00352, 3(4):6, 2023.
[16] Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Siheng Chen. Self-evolving multi-agent collaboration networks for software development. arXiv preprint arXiv:2410. 16946, 2024.
[17] Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent- code generation with iterative testing and optimisation. arXiv preprint arXiv: 13010, 2023.
[18] Dong Huang, Jie M Zhang, Yuhao Qing, and Heming Cui. Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv:2402.02037, 2024.
[19] Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405.11401 2024.
[20] Juyong Jiang. Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. survey on large language models for code generation. arXiv preprint S.Kiv::240.0055111 2024.
[21] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan, Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310006770, 2023.
[22] Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint arXiv:2303. 030004 2023.
[23] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-leve code generation with alphacode. Science, 97866624):1092-1097 2022.
[24] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
[25] Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023.
[26] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023.
[27] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingho Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2022.
[28) Augustus Odena, Charles Sutton, David Martin Dohan, Ellen Jiang, Henryk Michalewski, Jacob Austin, Maarten Paul Bosma, Maxwell Nye, Michael Terry, and Quoe V. Le. Program synthesis with large language models. In n/a, page n/a, n/a, 2021. n/a.
[29] Edward E Ogheneovo al. On the relationship between software complexity and maintenance costs. Journal of Computer and Communications, 2(14):1, 2014.
[30] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang. Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. arXiv preprint arXiv:2307.07992 07994, 2023.
[31] Thomas Reps, Thomas Ball, Manuvir Das, and James Larus. The use of program profiling for software maintenance with applications to the year 2000 problem. In Proceedings of the 6th European SOFTWARE ENGINEERING conference held jointly with the 5th ACM SIGSOFT international symposium Foundations of software engineering, pages 432-449, 1997.
12

---

[32] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
[33] Christos Saltapidas and Ramin Maghsood. Financial risk the fall of knight capital group. University of Gothenburg, pages 1-8, 2018.
[34] I Standard. Software engineering- software life cycle pooesses-maiinteannce ISO Standard, 14764:2006, 2006.
[35] Bayarbuyan Ulzii, Zeeshan Akhtar Warraich, Cigdem Gencel, and Kai Petersen. A conceptual framework of challenges and solutions for managing global software maintenance. Journal of Software: Evolution and Process, 27(10):763-792, 2015.
[36] Ervin Varga. Unraveling Software Maintenance and Evolution. Springer, 2018.
[37] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024.
[38] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with oss-instruct. In Forty-first International Conference Machine Learning, 2024.
[39] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. arXiv preprint arXiv:3308.08155 2023.
[40] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[41] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-5088 2024.
[42] Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. arXiv preprint arXiv:2412.21199 2024.
[43] Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. Beyond correctness: Benchmarking multi-dimensional code generation for large language models. arXiv preprint arXiv:2407. 14400 2024.
[44] Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311 10372, 2023.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are clearly presented in the abstract and introduction, and are further elaborated and substantiated throughout the subsequent sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the Section in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer the Section 3 and Appendix in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide well-documented code repositories the supplementary materials. Additionally, we commit to making the code publicly available by the time of the camera- ready submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (ht.tps://ni.s.. cc/
public/ htlc/guiiee desidd//// CoodeSubme ss Moooooolliooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 httppsscc/ppbblccuu.ccooo /Coeseoodeeuuiie
ss ceiiinnooliyy Stt for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to the Section 4 and Appendix D in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer the Section and Appendix D in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer the Section and Appendix Din the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpttt:tttptrstppeeee httpa//ppp....ooooooooo///aacaaa
Answer: [Yes]
Justification: This paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the Appendix H in the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators and original owners of all assets used in the paper are properly credited, and the licenses and terms of use are explicitly mentioned and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswis wwwoosocittcceeeeoo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper involves human subjects in the form of interns who were recruited for data annotation. Please refer the Appendix B.3 and Appendix Clin in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The paper involves research with human subjects in the form interns for data annotation. However, there evident risks in annotation process.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Please refer to the Section 4 and Appendix D in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy (https: /ttp///errppppppp/ee cataacc/Cooo LLM) for what should or should not be described.
http://reep..c////eeeeeeeeeeeeee Conc///eeeereess.s...... 2025/
20

---

Appendix
Instruction Templates
22
A.I Instructions for MaintainBench
22
A.2 Instructions for MaintainCoder
24
Instructions Phase Generator
25
B
Benchmark Construction)
26
B. Details of Original Datasets
26
Solution- Test Co-evolution
26
B.3
Mannually Quality Check
26
Annotation Process
28
Annotation Qualification
28
Annotation Requirements
28
Implementation Details
30
Extra Experiments
31
Case Study: Proactive Architectural Design for 2048 Game
33
Architectural Foresight: The MVC Pattern
33
Maintainability Under Dynamic Requirements
33
Conclusion of the Case Study
34
Case Study: A Toy Example
35
Impact Statements
38
21

---

A Instruction Templates
A.1 Instructions for MaintainBench
Instruction for Functional Extensions [ext
Given the raw problem and solution, generate new problem that requires using the original function as component of larger system in real-world scenario. To solve new problem. new solution should include the multiple function calls of raw question. So the new problem will be not only related problem hut also more complex problem than raw problem. raw problem:
(raw [aw_problem)
raw solution:
(raw [w__solution]
The new problem should be:
Must utilize the original function a helper/core component
2. Must extend functionality to handle more complex cases
3. Must include clear input/output specifications
4. Should demonstrate practical application in specific domain (e.g., finance, hcalthcare, e-commerce)
5. Must maintain original function's core purpose
Please return with json format as follows: ((
"prompt :yomptttpe": "PROMPT_SEEI INVOKING".
"input format": "input_format".. "<input format>".
"output_fonne format": "coutput format>", "new_prohlen" "<problem description>",
nnew_solution". "ccomplete solution code including original function>", "test input": "<at least five test ussertions in Python's 's ussert format and presented in list>"
Note: Only output the solution functions new solution, no other code be included. Also test input should be in Python's assert format. Ensure that all code in the JSON is properly escaped and the entire response is a valid JSON object. I'll use Joads) to transform it to diet type.
Instruction for Interface Modifications
[m]]]]]]]]]]]]]]]]]]
Given the raw problem its solution, generate a new problem that requires enhancing the interface while maintaining backward compatibility in a more real-world scenario.
To solve new problem, new_ so__solution should require the use of the raw question's solution but with adjusted interfaces. the prohlem will be related problem but including modifications the prohlem parameters, return types, or other interfaces. raw problem:
(raw_ [ew_problem]
raw solution:
(raw_solution)
The new problem should be:
1. Must maintains backward compatibility with existing implementations
2. Must support original function parameters
3. Must include type hints
Must add new optional parameters
Please return with json format as follows:
(1
"prompt_typee type": "PROMPT INTERFACE".
"input_ format": "<input format>".
"output format": "output_format": "contput formats".
"new. hww__problemm "<problem description>".
"new. solution": "<complete solution code including original function>".
"test input": "cat least five test assertions in Python's 's assert format and presented in lists" }}
Note: Only output the solution functions in new sew_solution, no other code should be included. Also the test input
should be in Python's 's assert format. Ensure that all code in the ISON is properly escaped and the entire response is a valid JSON object. I'll use loads) to transform to dict type.
22

---

object. use Instruction for Data Structure Transformations [Tast]]
Given the raw problem and solution, generate new problem that requires optimizing or changing the underlying data structures in real-world scenario.
To solve new__ prooleeo,em,, new new_solutine solution should require the use of the raw question's 's solution but with different data structures. So the new problem will be related problem but including changing from arrays to dictionaries, from lists trees,
raw problem:
(raw_pp [iw_problem)
raw solution:
rrw_solution!
The new problem should
1. Must use different data structure than original
2. Must handle more complex data relationships
3. Must include type hints
4. Must add additional Constraints
Please return with json format as follows: (1
"prompt.type" "PROMPT TTT__AATTTSSTUTTUTT
"input_ formaft""""" "<input format>",
"output_format": "coutput format>",
format":
"new_ nee__prohlem": "<prohlem description>",
"new solution": "<complete solution code including original function>",
"test iust_input:: "<at least five test assertions in Python's 's assert format and presented in list>" }}
Note: Only output the solution functions new_solution, no other code should be included. Also input
should be in Python's 's assert format.
Ensure that all code in the JSON is properly escaped and the entire response is valid JSON object. I'll use json.louds() transform it to dict type.
Instruction for Error Handling Enhancements [Trr
Given the raw problem and its solution, generate a new problem that requires error handling a real-world scenario.
To solve new pew__eeee__leeeeeeeeee problem, new solution should require the use of the raw question's 's solution but with error handling.
So the new problem will be related problem but including adding handling the solution. raw problem:
(raw_problem)
raw_solution:
(raw_solution
The new problem should
1. Must define specific error types (e.g., Custom exceptions for domain, specific errors, Hierarchical error structure, Mcaningful error messages)
2. Must include more real-world error scenarios
3. Must handle error propagation
maintain type hints
5. MUST clearly state what errors need to be handled
The test input should trigger errors that need to be addressed in the 'new problem' much as possible. Please return with json format follows: (1
ppromptttype": "PROMPT ERROR HANDLING", "input_format":
PEROMPT_ERROR
format": "<input format>".
"output format": "coutput format>".
"new_pree hww__probemm: "<problem description>".
"new solution": "<complete solution code including original functions".
"test "test_inpee i "<at least five test assertions in Python's 's assert format and presented in lists"
II
Note: Only output the solution functions in new_solution, other code should be included. Also the test inpur format.
should be Python's
Ensure all code in JSON is properly escaped and the entire response loon.loads() to transform it to dict type.
23

---

A.2 Instructions for MaintainCoder
Requirement Analysis Agent
Design Pattem Selection Agest
System
Rote: Requinemet Asaiyada Agent.
Rola: Pastern Selection
Sepet: Plain text saftwrre reull rewstts.
Trput: lodeles ard dalleges free requiresent Asalysis agent. Task:
Taak:
Bresk down the arabien, feessing on user geela and contraccts,
Chease apprep Lati desagn petterro for each mesele.
Step-typsteen waalyste:
Adtrece bey malleegge aftertively.
Extruct requrreen
slize mobluritty law saaplang, scalabilaty, resudullity.
Saggeet
higheraaratt
Eatput:
Hodzle Name: Functional modale
Structured breaddters
Desigg Pattern: Privary satters selected.
1. Core fuerttosstitiee tree key operatisse or feanures,
auctoficartee: Pochles estlnd and citsbility.
2. Ray Gulllengga: Speralic hurdics and sosertiel impact. Aetterable, clasr, corciss, No 19de.
Alternatanca: Diher patterns and ther ure ens (iptisnat),
Figure 4: Prompts for Requirements Analysis Agent and Design Pattern Selection Agent.
Framewook Design Agent
Framework Evaluation Agent
Systen Message:
System Nessage:
Rodules
structure
Fraewars
Teak:
Develap pretisirary clas structure (cles attribeiss, sethots).
tralsete
claraty, scalability, performucce, beat
pracricee.
Dellan class re releiiirolip luneratarse, cxpoattonn depandenctes).
yy ly issus in cosplang. colesian, remsdbulity.
forse in logie asgh-level
Justity desige choices assumstions.
Sptsnize signtiicnnt esdales, Igrone atnor changee.
Retpolt
Prapsse
Class Stractures
Ortert:
Belationsh"35 Elanses, rxplarations. code satzal.
Revinues apttained deeign
If resdal, "TEN2OATT"
Figure 5: Prompts for Framework Design Agent and Framework Evaluation Agent.
Code Generation
Code Optimizazion Agent
Systen Hessage:
System Nossage:
Bote: Geveration Agert.
Bota: Optinization Agent.
Traseorrs wits sccile strictures, claeres, Interfaces, pultemn, jolifacutioo.
Task:
Underatard the problce and franoork dasign
Tack:
Beview the original oade for sesses e- Iaprovenent areas.
Write Python code vitt corcise comeatss.
Essare "appa, nust interface furction acists; create atsstng.
the RP and fot 25) guicelines
Pravide code to code cencotite Apent for esecation and feedback
Ircluce bosting arterfece function vaned 'ssparrent", Satpon:
Hodiry cade on foodback.
unlid sytter
outgut: Hodtetee code Pytton steck..
Figure 6: Prompts for Code Generation Agent and Code Optimization Agent.
24

---

A.3
Instructions for Phase II Generator
Instructions for Phase II Generator
*<Role**:
You are Python development assistant specializing in efficiently integrating new features into existing codehases while balancing the retention of original functionality and implementing new requirements.
**Input**:
existing code's purpose functionality. Original Code: The current implementation of the code.
Original Requirements: description
New Requirements: A description of the additional features changes that need to be implemented.
Test case: test case written assertion format.
**Task**;
Understand the original code, original requirements and new requirements.
Modify the original code to correctly realize the requirements.
The complete code contain interface function named [test_ interface name).
Output the complete code that integrates the new requirements the of original code.
**Notic
Do not modify the reusable code block.
Make few changes as possible.
OOnly generate functional codes, and not include test cases.
**Output**:
-Tee full updated Python code that fulfills the new requirements in format:
'python
<your code here>
25

---

B Benchmark Construction
B.1 Details of Original Datasets
To provide further context on the foundation of MaintainBench, we include additional details on the five original datasets that extends. These datasets are widely adopted in code generation research and serve as standard benchmarks for evaluating the performance and generalization ability of LLMs.
HvaanEval is a benchmark proposed by OpenAl, consisting of 164 hand-writter programming problems designed to evaluate the functional correctness of code generated by language models. Each problem includes a natural language prompt, a canoni- cal reference solution, and unit tests. The tasks are concise and focus on ceneral-purposs programming topics such as list manipulation, string processing, and simple algorithmic challenges.
HumanEval 77:
MBPP (Mostly Basic Programming Problems) [28]: The MBPP dataset contains 974 Python problems that were manually filtered from introductory programming tasks across educational and online coding resources. Each problem includes concise description, a reference solution, and at least three test cases. MBPP targets beginner-level problem-solving skills, such as basic data types, arithmetic, loops, conditionals, and string manipulation.
APPS (Automated Programming Progress Standard) [14]: APPS is large-scale dataset containing over 10,000 problems gathered from competitive programming websites, online judges, and coding tutorials. The problems span wide range of difficulty, categorized as introductory, interview, and competition level. Each instance includes a problem statement, input/output formats, and sample test cases.
xCo [22]: XCo is multilingual and executable benchmark with over 12,000 problems spanning 20 programming languages. It is designed to assess large language models in diverse and realistic programming environments, with emphasis on execution- based correctness. Each problem contains a natural language description, optional starter code, and structured test cases.
CodeContests [23]: CodeContests is benchmark created from real-world competitive programming tasks, collected from platforms such as Codeforces, AtCoder, and Google Code Jam. The dataset emphasizes algorithmic problem-solving with problems often requiring advanced logic, data structure manipulation, and mathematical insight. Each problem is accompanied by multiple human-written solutions and test cases, enabling evaluation via code execution.
B.2 Solution-Tess Co-evolution
Our refinement method maintains coherence between solutions and test cases through orches- trated co-generation process. Instead of generating these components independently, which risks misalignment, we first derive extended solutions by adapting original code according to specific requirement change patterns. Then, we generate corresponding test cases to verify both preserved and newly introduced functionality. This integrated generation process ensures traceable evolutionary path, where changes are minimal, intentional, and directly responsive to dynamic requirements. Furthermore, we wisely focus on test case representativeness and coverage where we further look at consistency between problem descriptions, solution implementations, and test assertions. i.c., standardize all test cases using the Python assertion format; incorporate holistic edge cases to verify robustness; and error handling variants, add specific tests that trigger exception handling mechanisms. For each problem variant, our test suites verify both the preservation of original functionality and the correct implementation of new requirements, achieving high code coverage and thoroughly exercising the modified components.
B.3 Mannually Quality Check
Our approach combines automated validation with expert human review across multiple stages:
In the first stage, we verify that L.M--geeeraed descriptions accurately correspond to our specified extension criteria. Once validated, we derive extended solutions by adapting
26

---

original code according to specific requirement change patterns, then generate corresponding test cases to verify both preserved and new functionality.
In the second stage, programming experts review and correct identified issues after generat- ing the whole benchmark, focusing on aligning problem descriptions, test cases, and the intended requirement changes. Particularly, they verify that test cases effectively exercise all implementation aspects (c.g., exception handlers in eπrоr handling cases) and that solutions properly reuse original code rather than introducing unnecessary rewrites, which aims to maintain the semantic integrity of functional extensions for modified components.
In the third stage, a separate team reviewers performs cross-validation between original and modified implementations to verify that: (1) core functionality is preserved across all variants, (2) new requirements are correctly implemented, (3) code modifications are appropriately scoped and minimal, and (4) test cases provide comprehensive coverage of both original and new functionality.
27

---

C
Annotation Process
C.1 Annotation Qualification
Application requirements
You must have Python3 installed.
2. Personnel requirements
You must have obtained an undergraduate degree or above in computer science or related fields. Having taken software engineering courses is plus.
You must master the basic syntax of Python, such as functions, classes, and assertions.
You must be able understand the following code:
def Dount_batik. cetiiiik_ombnnnnninnom (n, K, I, e, axx, a_y, b_x,
b_y):
MOD 10++9
# Caiculate the overlap zows and columns
overlap_rows max r abs (a_x b_x))
overlap_ccls max max c abs (a_y - b_y)]
i Calculate total number of combinations without overlap
total_ www.instiinsnsssecccc.eccocciiie pow
c, MCD)
+ Calculate the number overlapping combinations
pow pow(k, cverlap_rows
overlap_cols, MOD)
Result is total combinations squared divided by
overlap combinations
result (total_combination
tetal_ total_combinations) MDD
result = (result pow weveerlap_cmbinateet MOD - 2,
MOD) ) MCD
return result
C.2 Annotation Requirements
1. Annotation Method
Completed online, submitted in jsonl file.
2. Dataset Introduction
Each line in jsonl file represents piece of data. The data consists of the original problem (raw_ how_pprobem)...mm the original solution code (raw_sod how_ssoution), the original code test data (raw_test how__est_iiee _e__tesr__ppt),, the newly generated problem (new_ nee_ppoblem), the problem solution code (new_solution), and the new problem code test data (test_ jeetl_inutt.. Among them, saw_problem, raw sw__souutonn, and raw @w__est_inpur are completely correct. But the new new_problem may not meet the specific requirements of the requirement change. Further- more, according the solution code of new pew_problem, there may be issues with new sr__oouution and test input. The annotator needs to modify new_solwte sww_souuionn and test input according to the requirements of new pew_pero
3. annotation principles
Our task is to ensure that the new problem meets the requirements and modify the generated code and test cases based on the given problem. Here are the annotation principles:
(a) Ensure that the new_problem comply with specific change requirements, such as Functional Extensions, Interface Modifications, Data Structure Transformations, Error Handling Enhancements
(b) Ensure that the new_solution correctly meets the requirements of the new new_problem. (c) Ensure that the test_input provide comprehensive coverage of both original and new functionality.
(d) Ensure that the new new_solution can pass all test jist_input.
(c) Make sure that all new test cases given in the form of Python assertions.
28

---

4. Annotation Quality Requirements
We will check the annotated data, and unqualified data need to be re-annotated. The final annotation pass rate cannot be lower than 90%.
29

---

D Implementation Details
For all experiments, we the generation temperature 0.3 and topp 0.95.
For main experiment, it consists of two phases:
In Phase I, for MaintainCoder, the maximum number of framework evaluation is set to 3 and the maximum number of code optimization is set to 5. We test the static metrics, including maintainability index (MI) and cyclomatic complexity (CC), where the formula for calculating MI is:
= max 0. 100 (1--5- 12 (FACI
171
where V is Halstead Volume, G is Cyclomatic Complexity, L is Source Lines of Code (SLOOC), and C' is the percentage of comment lines. All methods generate five rounds of code for average static metrics.
In Phase II, given a predefined Phase II generator (GPT-4o-mini is adopted in the main experiment), the modification operation is performed as probe to calculate dynamic metrics, including Pass @ 5, abstract syntax tree structure similarity (AST and code similarity Codeaiff- The dynamic metries like AST sim and Code are averaged over five rounds. Both AST sim and Codedif are directly calculated by calling the Python library difflib 1b.
For ablation studies, MaintainCoder adopts the same settings as in the main experiment, but only the framework evaluation and the code optimization components are respectively canceled. It is still divided into phase and phase II, and @ of the code phase II is reported.
30

---

E
Extra Experiments
Table 7: Maintainability evaluation on entry-level Humannval-Dyn and MBPP-Dyn problem
Static metrics
Dynamic metrics
Model
MI
CC
Pass (%)
AST
Code (%) L Code
HumanEval-Dyr
GPT-4o-mini
76.7
2.69
89.4
0.556
140
9.64
GPT-to-minicat
88.6
0.561
141
9.58
GPT-4o-minipue
67.2
3.66
86.4
0,580
97.2
11.0
A AgenCCoder (G-ii---mmii)
87.1
3.33
90,9
0,561
90.5
19.7
MapCoder (GPT-4o-mimi)
80.7
3.42
89.4
0.567
102
15.2
MaintainCode (GPT-4o-mini)
2.39
92.4
0.850
43.2
DeepSeek-VV -3
2.95
90.2
178
DeepSeek -
78.8
3.10
90.9
0.517
320
17.2
DeepSeek- V3run
DeepSeek-V3pun
75.4
3.41
91.7
0.567
210
16.1
AgentCoder (DeepScck-V) V3)
86.2
3.18
90.2
0.589
132
15.6
MapCoder (DeepSeck-V3)
77.8
3.48
89.4
0,566
145
14.6
MaintainCoder (DeepSeck-V3)
64.5
1.92
95.5
0,686
106
16.7
GPT-4o
74.3
2.55
89.4
0,460
236
17.0
Qwen-Plus
79.2
3.07
87.9
0.588
172
10.0
2ocmimi-25--lash--Pee
3.17
91.7
15.7
Claude-3.5 3.5 3.5-Sonnet
79.6
3,07
91.7
0.510
330
17.8
S.nude-3.7-Sorte
77.2
3.44
90,2
0.598
137
9.91
MBPP-Dyn
GPT-4o-mini
79.6
1.97
76.7
0.465
233
11.4
GPT-4o-minngar
79.9
1.99
83.3
0.459
239
GPT-4o-minipiu
69.4
3.03
89.2
0.555
134
11.8
AgentCoder (GPT-to-mini)
92.2
2.55
83.3
0.462
124
16.0
MapCoder (GPT-4o-mini)
88.3
2.77
76.0
0,528
84.3
MaintainCoder (GPT-4o-mini)
72.0
2.71
85.0
0.793
39.8
17.5
DeepSeek-V3
81.7
2.11
87.5
0.512
246
DeepSeck-V3an V3pw
79.9
2.29
87.5
0.500
234
11.1
DeepSeek-Ver Vppu.
77.5
2.70
89.2
0.542
204
11.4
AgentCoder (DeepSeck-VD)
2.53
86.7
0,511
194
MapCoder (DeepSeek-V) V3)
78.3
3.00
89.2
0.549
124
10.2
MaintainCoder (DeepScck-VV V3)
64.3
1.93
89,2
0,868
23.0
15.7
GPT-4o
76.7
2.05
88.3
0.468
210
11.8
Qwen- Plus
81.3
2.32
85.8
0,513
253
11.0
Gemini-2.5 2.55-Flash-Prars Preview
87.5
0,575
168
Claude-3, 3.7-Sonnet
79.5
2.58
84.2
0.514
195
10.2
Claude 3.5 S-S-nntt
82.4
2.13
85.8
0.491
269
10.7
Dyn
CodeContests Dyn
XCoceEvval
0. R5
60
Paooa
Piocolte
0.21
0. 46
40
6.1
0.25
6.2
0.20
vallu
valum
vallue
Figure 7: Pass@k on APPS-Dyn, CodeContests-Dyy and XCodeEval-Dyn. MaintainCoder consis- tently outperforms with varying k values (1-5), demonstrating the robustness war.. Pass @k
31

---

Table 8: Performance on mixture-level dataset with Siiiini--.--Flas-PPPee as Phase II generator. Conclusions are consistent with GPT-4o-mini, demonstrating robustness w.r.f. Phase II generator.
Static metries
Dynamic metrics
Model
MI
CCL
Pass@ 5 (%)
AST
Code (%) 1 Code dify
APPS-Dyn
GPT-4o-mini
5.10
0.456
194
26.8
GPT-to-minica7
63.2
5.10
60.5
0.464
184
25.4
GPT-4o-minigu
59.2
5.01
61.0
0,496
124
26.5
AgentCoder (GPT-to-mini)
5.81
53.5
0,350
97.6
32.0
MapCoder (GPTT4o-mini)
67.8
5.98
62.0
0.471
91.0
27.7
MaintainCoder (GPT-4o-mini
69.5
2.75
63.5
0,724
55.3
37.0
DecpSeek-V3
61.8
7.59
66.5
0.569
144
26.8
DeepSeek-V3eur
7.28
72.0
26.5
PeepSeck-33gum
59.2
6.06
70.0
0.537
169
28.3
Agen(Coder (DeepSeek-V3)
60.5
6.68
68.5
0.498
141
30.0
MapCoder (DeepSeek-V)
MaintainCoder (DeepSeck-V3)
62.4
3.21
75.0
0,650
87.1
34.7
GPT-4o
63.0
4.58
66.5
0.452
173
24.5
Qwen-Plus
61.2
5.61
68.0
0.517
185
24.9
Gemini-2. Semmini-.5--laab-- S-Flasb-Preview
9.00
70.5
0.591
126
23.1
Claude-3.5 3.nude3,,-Sommee
60.8
4.63
69.0
0,502
141
29.0
Claude 3.7 Sonnet
59.3
6.65
67.5
0.504
120
31.6
Table 9: Fine-grained token usage for MaintainCoder on the CodeContests dataset.
Requirement Analysis
Patter Selection
Framework Design
Code Implementation
Code Modification
Code Extraction
Stage
Supervisor
Total
Tokens (k)
1.7
2.3
6.7
6.7
3.1
9.0
3.6
33.1
32

---

Case Study: Proactive Architectural Design for a 2048 Game
To provide a concrete, end-to-end illustration of Maint ainCoder's capabilities, we present case study on generating a complete, functional, and maintainable 2048 game. This example serves to fulfill the request from Reviewer 9Ho3 for more verbose demonstration of our agentic pipeline. highlights MaintainCoder 's philosophy: shifting from reactive problem-solving to proactive architectural design for long-term software resilience.
E1
Architectural Foresight: The MVC Pattern
Given the high-level requirement "create a 2048 game," MaintainCoder functions not merely as a code generator but as software architect. The Design Pattern Selection ion Agent identifies that the game's 's logic (Model), user interface (View), and input handling (Controller) are distinct concerns that are likely to evolve independently. Consequently, it selects the Model- View- Controller (MVC) design pattern to enforce a clear separation of concerns, enhance modularity, and reduce coupling.
This architectural decision, made proactively during the initial design phase, results in well- organized and intuitive repository structure, as shown below:
2048 D048-pychon-game/
-- main. py -- README. md
s src/
game 2048/ aeme_2048/
init py
common/
1-- _init_ py
enums.py PY
controller/
init .y
game _ooooollleerrpy
model/
init py game_states state..ppp..
view/
init init_ .py game_view.py ui mi_constants.py
utils/
init init_.py matrix_utils.ny
F22 Maintainability Under Dynamic Requirements
The true value of this proactive design is revealed when the requirements evolve. We simulate two common types of maintenance tasks that mirror real-world software evolution:
1. Scenario 1 (View Modification): A requirement is introduced to "change the color scheme of the game tiles to a high-contrast mode for better accessibility."
2. Scenario 2 (Model Modification): A feature request asks to "add new '512' tile to the game, appearing after the '256' tile."
In a monolithic, poorly structured codebase, these seemingly simple changes could necessitate complex, widespread modifications, risking the introduction of new bugs. However, with the MVC architecture generated by MaintainCoder, the maintenance effort is minimized and localized:
For Scenario 1, the required changes are confined entirely to the view/ directory, specifically within ui_constants, py where colors defined, and potentially
33

---

name_view.re py to adjust rendering logic. The core game logic in the model/ and controller/ directories remains untouched, ensuring that this cosmetic update does not affect the game's functionality.
For Scenario 2, the logic for the new tile is encapsulated within the model/ / directory, primarily affecting the game_state, -py file where the game board and tile values managed. The view/ directory might require a minor, corresponding change to add color for the new 512'' tile, but the core responsibility for the feature enhancement is correctly isolated within the model layer.
F.3
Conclusion of the Case Study
This case study demonstrates that Ma i a nCoder's architectural foresight provides persistent benefits over multiple maintenance cycles. By embedding proven software engineering principles like design patterns directly into the generation process, it produces code that is not only functionally correct for the initial requirement but is also inherently structured for future evolution. The resulting changes localized and simple, drastically reducing the long-term of maintenance and aligning with the core goals of our work.
34

---

G
Case Study: A Toy Example
Question
Denote cyclic sequence of size n. as an array such that 5n is adjacent to The segment s/r, 4] where r is the concatenation s(r, ni and s[1,
You are given an array a consisting n integers. Define as the cyclic sequence obtained from concatenating copies of Note that b has size
You are given an integer where k = 1 or in where the sum of elements in the segment is divisible by k.
prime number. Find the number of different segments
Two segments
considered different if the set of indices of the segments different. For example, when n. = 3 and m = 2, the set of indices for segment s|2, 5] is {2, 3, 4, 5), and for segment s|5, 2] is (5, 6, 1, 2) In particular, the segments s[1, s), s|2, ....................... s 5] are considered the same segment.
Output the answer modulo 10 + 7.
Input
The first line contains three integers n, m, and k[1 $ n, m, S 10 k or is is a prime number).
The second line contains integers an, 02, an an S a; 2 10010..
Output
Output an integer denoting the number of different segments in b where the sum of elements in the segment is divisible by k modulo 10
MaintainCoder's response:
from typing import List
class CyclicArray;
***
class to represent a cyclic array derived from a given original.
array.
It allews access to elements in cyclic manner and provides prefix sum array.
def Linit_ (self, original_array List [intl, copies: int):
Initializes the CyclicArray with the original array and number of copies.
:aram original_arrann List [int] - The original array.
param copics: int Number of copies to create the cyclic array. ***
orlf.original self.size = len (en(original_aarry))
original_array
self.copies copies
self.prefix_sum self.get__refiix_ee
def get element (self, index: int) -> int:
Returns the element the given index considering cyclic behavior.
:param index: int The index to access.
:return: int The element at the cyclic index.
return self self..orginaalarrayylooo
self.size]
def stw.errreexxxxmsslllllesees -> List [int]:
Generates and returns prefix array for the cyclic array.
35

---

:return: List/ints The prefix array.
.*.
total_length self. sizc * self, copies prefix_sum = [0] (total_Length + 1)
for in range (1, total_length 1): prefix_sum[i] [i]] prefix_sum(i 1] + self. eeeemete (i 1)
return prefix_sum
class SegmentsuunClloulatao
class to calculate the number of valid segments in cyclic array whose divisible by given integer
def iniit_tsslff, cyclic_array: CyclicArray, medulus: int):
Initializes the Segmentsumnalcuuattr with cyclic array and modulus.
param cyclis_array: ccrrryy An instance of the Cyclickrray class. :prra modulus: int The modulus value for output.
self.cyclic_arce - cyclic_array
modulus
def count_valioe segments (self, kt int) -> int:
Counts the number of segments whose sums are divisible by k.
:param int The diviser for checking segment suns, :return: int The count of valid segments modulo 'nodulus'.
count
prefix_count [0] To count occurrences each rndainder trefi__count [0] 1 There's one way to have sun of (empty
segment)
total_length eylf. size self.cyclic
for 1 in range total_length 1); current_suu = www..yyclcccaaaa.rooff.... [im]] remainder current_sum $
# Count how many times this remainder has occurred count += prefix_count remainder] count $= self modulus
Update the count of this remainder prefix_count (remainders 4- 1
return count
class MainProcossor:
class to orchestrate the process of counting valid segments in cyclic array.
det innt (self, original array: List lint], copies: int, k: int):
36

---

Initializes the MainProcessor with the necessary components.
:param original_array: dist[int] - The original array. param copies: int Number of copics to ereate the cyclic array. :para k: int The diviser for checking segment sums.
self.cyclic_array oyclickkrayylorignaaaaaaaa copies) self wew..segmet__aaauue
10**9 7)
self.k
def process(seel) -> int:
Orchestrates the overall process to calculate the number of valid segments.
:return: int The result of valid segments count.
return self www.ettccccclllllttrrrcocc.eoooe count ott.::aeeeeoaaaaaaaaa...oooooooa (sclf.k)
def codéContest_ruul (n: int, m: int, k: int, a: [ist[int]) -> int:
Interface function execute the segment counting process.
:param n: int Size of the original array.
:param m: int Number of copies to create the cyclic array.
:param k: int The divisor for checking segment sums.
:param a: List [int] The original array.
:return: int The count of valid segments modulo 1099 + 7.
processor = MainProcessor (a, k)
return processor (roccsss))
GPT-4o-min's response:
def codeContest cunnstt_rullll (n, k, a);
MOD 10++9 total_sum = sum (a) m
prefix_sum [0] (n m
for in range (1, pum (prefix_suml sum 11 all1 1) al) &
1):
count = [1
for in range (n mmm. 1): mod_value prefix_sur if mod_value in count:
count [mod_ value] += else:
count [mod value]
result =
for mod_value in count:
count (mod_value,
result (s // result $- MOD
return result
37

---

H Impact Statements
This paper introduces MaintainCoder and MaintainBench, which have significant potential impacts on both research and industry. For research, MaintainBench provides novel benchmark for evaluating the maintainabilityy of code generation models under evolving requirements. This will enable more comprehensive and realistic assessments of code generation systems, driving the development of new methods that not only focus on initial correctness but also on long-term maintainability. MaintainCoder, pioneering solution integrating the Waterfall model, design patterns, and multi- agent collaboration, offers new approach for maintainable codes. Its success in experiments demonstrates the feasibility and effectiveness of this approach, inspiring future research combining traditional software engincering principles with modern Al techniques. In industry, the ability to generate code that is casier to maintain and adapt to changing requirements can significantly reduce software lifecycle costs. This is particularly important for large-scale software projects where maintenance often accounts for substantial portion of the total cost. By improving maintainability. MaintainCoder can help companies reduce technical debt, enhance software quality, and accelerate development cycles. Additionally, the insights gained from this research can inform best practices for human developers, promoting more sustainable and efficient software engineering processes. There also other potential societal consequences of our work, yet none of which think must be specifically highlighted here.
38


Reviews:
[{"review_id": "IljFME3dAx", "reviewer": "Reviewer_AvR7", "summary": "This paper introduces MaintainCoder, a framework for generating maintainable code under dynamic requirements, and MaintainBench, the first benchmark for evaluating code maintainability. Key contributions include:\n\n1. MaintainBench: Extends established benchmarks (e.g., HumanEval, MBPP) with 500+ Python tasks, covering four requirement change patterns (*functional extensions, interface modifications, data structure transformations, error handling enhancements*). Novel dynamic metrics (e.g., post-modification correctness `Pass@k`, code change volume `Code_diff`, AST similarity `AST_sim`) are proposed.\n2. MaintainCoder: A multi-agent system integrating the Waterfall model and classical design patterns. It enforces modularity, low coupling, and high cohesion through specialized agents (e.g., requirements analysis, pattern selection, framework evaluation).\n3. Key Findings:\n   - Existing methods (GPT-4o, AgentCoder, etc.) suffer significant maintainability degradation under requirement changes (e.g., `Pass@5` drops 15–30%).\n   - MaintainCoder improves dynamic metrics by >60% while boosting initial correctness.\n   - Traditional static metrics (e.g., MI, cyclomatic complexity) poorly correlate with maintainability, while dynamic metrics show high consistency.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Problem Significance:\n   - Addresses the critical yet underexplored challenge of maintainability in code generation, aligning with real-world pain points (80% of software costs stem from maintenance).\n   - Highlights limitations of static benchmarks (e.g., HumanEval) and advocates for dynamic evaluation.\n2. Novel Contributions:\n   - MaintainBench:\n     - Systematically models requirement evolution via four change types (`Π_B`), covering 80% of maintenance scenarios (adaptive/perfective).\n     - Ensures quality via solution-test co-evolution (GPT-4o + human refinement).\n   - MaintainCoder:\n     - Uniquely combines software engineering principles (Waterfall model, design patterns) with multi-agent collaboration.\n     - Ablation studies validate the necessity of framework evaluation and code optimization modules (25–37% performance drop if removed).\n3. Rigorous Evaluation:\n   - Comprehensive Coverage: Tests 3 difficulty levels (entry/mixture/competition), 7 base models (GPT-4o, DeepSeek-V3, etc.), and 2 multi-agent frameworks (AgentCoder, MapCoder).\n   - Multi-dimensional Metrics: Exposes discrepancies between static metrics (MI/CC) and dynamic maintainability (e.g., high MI but low `Pass@5`).\n   - Robustness Checks: Consistent results across varying `Pass@k` (k=1–5) and Phase II generators.\n\nWeaknesses\n\n1. Theoretical Gaps in Dynamic Metrics:\n   - The Monte Carlo estimation for maintenance cost (`ℳ(C₀)`) lacks justification for first-order truncation. Discount factor `γ` is not discussed.\n   - `AST_sim` and `Code_diff` rely on syntactic similarity (via `difflib`), ignoring semantic equivalence (e.g., refactored but functionally identical code).\n2. Limited Experimental Comparisons:\n   - Missing Human Baseline: No comparison against code written/adapted by developers, leaving industrial relevance unclear.\n   - Language Restriction: Only Python is evaluated; generalization to stricter-typed languages (e.g., Java) is untested.\n3. Methodological Limitations:\n   - Black-box Pattern Selection: Design pattern choices rely on LLM reasoning without causal analysis (only cites correlation in [13]).\n   - Computational Cost: Multi-agent pipeline requires multiple LLM calls (≤3 framework evaluations + ≤5 optimizations), but overhead vs. single-model inference is unreported.\n4. Dataset Issues:\n   - Annotation Process: Appendix B.3 mentions \"multi-stage expert review\" but omits reviewer count, disagreement resolution, or inter-rater agreement (e.g., Kappa).\n   - Class Balance: Distribution of change types (`Π_B`) across difficulty levels is unanalyzed.", "was_revised": true, "final_justification": "Regarding the responses to W1–W4, with the exception of the Human Baseline issue, the authors have provided fairly comprehensive experimental evidence addressing my concerns. Considering my earlier score of 4 (“Borderline Accept”) and their response, I am now inclined to support this paper and get the final acceptance decision.", "rebuttal": "Dear Reviewer AvR7:\n\nThank you so much for your constructive feedback and efforts on our research. \n\nWe supplement several experiments to address your concerns.\n\n## Response to W1:\n\n> **(Theoretical Gaps in Dynamic Metrics)** \n>\n> - The Monte Carlo estimation for maintenance cost lacks justification for first-order truncation. \n> - AST_sim and Code_diff rely on syntactic similarity (via difflib), ignoring semantic equivalence (e.g., refactored but functionally identical code).\n\nWe appreciate your rigorous feedback on our dynamic metrics.\n\n**First-Order Truncation for** **Maintenance** **Cost:** \n\nWe agree that real-world maintenance is a long-term, multi-round process. However, simulating multi-round maintenance for evaluation is computationally prohibitive. We believe that **code maintainability is an intrinsic property**, and thus the single-round maintenance cost should be highly correlated with the cumulative cost of multiple rounds. **A single round, therefore, can serve as an effective and efficient proxy.**\n\nTo validate this, we followed your suggestion and extended our benchmark. We synthesized a second round of requirement changes for MaintainBench. As shown below, the costs from the **second round** **exhibit a high correlation with the first-round costs**, and are slightly more than the first ones. This suggests that the first-order approximation provides a good balance between evaluation accuracy and computational feasibility. We will supplement this discussion in the revision.\n\n*Tab1: First-round and Second-round* *Maintenance* *Costs (AST_sim & Code_diff) on CodeContests-Dyn.*\n\n| Model                       | 1st Round Cost |                 | 2nd Round Cost |                 |\n| --------------------------- | -------------- | --------------- | -------------- | --------------- |\n|                             | AST_sim ↑      | Code_diff (%) ↓ | AST_sim ↑      | Code_diff (%) ↓ |\n| GPT-4o-mini                 | 0.661          | 90.1            | 0.645          | 96.2            |\n| MaintainCoder (GPT-4o-mini) | 0.833          | 23.2            | 0.811          | 29.4            |\n| DeepSeek-V3                 | 0.718          | 87.2            | 0.694          | 92.5            |\n| MaintainCoder (DeepSeek-V3) | 0.788          | 43.2            | 0.765          | 48.9            |\n\n**Syntactic vs. Semantic Metrics:** \n\nWe acknowledge that `AST_sim` and `Code_diff` do not capture the semantic equivalence of refactored code. However, we argue that this is a feature, not a bug. **The effort required for refactoring functionally identical code is a genuine part of the maintenance cost that our metrics are designed to capture**. A developer still needs to spend time and cognitive energy to perform such refactoring. While more semantically-aware metrics are a valuable direction for future research, syntax-based metrics like `difflib` offer the unique advantages of being fast and easily verifiable, which is more friendly for methods involving reinforcement learning (e.g., RLVR).\n\n## Response to W2:\n\n> **(Limited Experimental Comparisons)** \n>\n> - Human Baseline: No comparison against code written/adapted by developers.\n> - Language Restriction: Only Python is evaluated; stricter-typed languages (e.g., Java) is untested.\n\nThanks for important suggestions to broaden our evaluation scope.\n\n**Human Baseline:** Based on the suggestion, we have already begun the process of recruiting experienced developers to perform coding. Given the time required for recruitment, coding, and data analysis, we commit to providing these results during the discussion period as soon as possible. Thanks!\n\n**Language Generalization (Java):** For industry, design patterns are successful and widely adopted in Java software development. For experiments, we translated MaintainBench from Python to Java, and evaluated MaintainCoder against the baseline on Java version. The results are consistent with Python: **MaintainCoder improves maintainability metrics, indicating our principles are not language-specific**. We also find Java performance is generally slightly lower than Python, due to imbalance training data [1,2]. We will supplement these discussion in the revision.\n\n*Tab2: Java version performance on CodeContests-Dyn.*\n\n| Model                       | MI ↑ | CC ↓ | Pass@5 (%) ↑ | AST_sim ↑ | Code_diff (%) ↓ | Code_diff (abs) ↓ |\n| --------------------------- | ---- | ---- | ------------ | --------- | --------------- | ----------------- |\n| GPT-4o-mini                 | 56.5 | 6.52 | 22.5         | 0.640     | 95.3            | 21.2              |\n| MaintainCoder (GPT-4o-mini) | 64.1 | 3.11 | 30.5         | 0.812     | 28.4            | 20.5              |\n| DeepSeek-V3                 | 54.2 | 7.21 | 25.0         | 0.703     | 91.8            | 23.8              |\n| MaintainCoder (DeepSeek-V3) | 62.5 | 3.84 | 36.0         | 0.795     | 45.1            | 22.7              |\n\n## Response to W3:\n\n> **(Methodological Limitations)** \n>\n> - Black-box Pattern Selection: Design pattern choices rely on LLM reasoning without causal analysis.\n> - Computational Cost: Multi-agent pipeline requires multiple LLM calls (≤3 framework evaluations + ≤5 optimizations). The overhead vs. single-model inference is needed.\n\n**\"Black-box\" Pattern Selection:** We clarity that our pattern selection is not purely \"black-box.\" Unlike monolithic, end-to-end models, our multi-agent pipeline makes this reasoning an **explicit and auditable** step. The Pattern Selection Agent is required to first propose a design pattern and then provide a textual justification for its choice based on the requirements. This output is reflected and logged, offering a degree of transparency and interpretability. While if LLM reasoning meets causal analysis is a big open-question beyond the scope of this paper, our method makes the *process* more transparent, mimicking auditable teamwork.\n\n**Computational Cost:** We have performed the requested analysis and agree that multi-agent pipelines incur higher computational costs. But for initial code generation, we believe this is a **worthwhile trade-off**. Specifically, the token usage of MaintainCoder is on par with MapCoder or reasoning model o3-mini, and much lower than MetaGPT/ChatDev [3,4]. But **MaintainCoder achieves a >60% improvement in maintainability and higher correctness**. The initial investment is justified by the drastic reduction in subsequent human effort required for debugging and refactoring.\n\n*Tab3: Token Usage Comparison.*\n\n| **Token Usage** | **MaintainCoder** | **MapCoder** | **o3-mini** | **MetaGPT/ChatDev** | **GPT-4o-mini** |\n| --------------- | ----------------- | ------------ | ----------- | ------------------- | --------------- |\n| CodeContests    | 33.1k             | 38.7k        | 20.8k       | 50k+                | 2.5k            |\n| xCodeEval       | 29.6k             | 23.5k        | 21.2k       | 50k+                | 2.3k            |\n\n## Response to W4:\n\n> **(Dataset Issues)** \n>\n> - Annotation Process: Appendix B.3 mentions \"multi-stage expert review\" but omits reviewer count, disagreement resolution, or inter-rater agreement.\n> - Class Balance: Distribution of change types across difficulty levels is unanalyzed.\n\n**Annotation Process Details:** Appendix B.3 mentions a “multi-stage expert review,” which refers to our sequential quality control pipeline. In this process, each problem was assigned to a qualified annotator (32 Solution-Test Co-evolution per annotator) responsible for correcting test inputs and ensuring compliance with the four evolution patterns. Subsequently, two of the authors independently reviewed the annotations. Disagreements, when present, were resolved through discussion to reach consensus. After annotation and review, the authors conducted comprehensive validation to ensure that all test cases execute correctly within our evaluation framework, the requirement evolution patterns are properly implemented, and the new problems represent meaningful software maintenance scenarios. To further assess annotation reliability, we computed inter-annotator agreement using Fleiss’ Kappa between the two reviewing authors from our team, obtaining a score of 0.899 [5]. This value indicates almost perfect agreement, suggesting that our annotation process is consistent and reliable across different contexts. We will include these clarifications in our final revision.\n\n**Class Balance Analysis:** In MaintainBench, every original problem has four requirement changes. The distribution of change types is fully balanced, ensuring that our evaluation is not biased towards a specific type of change or difficulty. For some fine-grained analysis, we provide the distribution of error changes across difficulty levels.\n\n*Tab3: The distribution of error changes across difficulty levels.*\n\n| Difficulty  | Functional Extension | Interface Mod. | Data Struct. Trans. | Error Handling |\n| ----------- | -------------------- | -------------- | ------------------- | -------------- |\n| Entry       | 28%                  | 23%            | 26%                 | 23%            |\n| Mixture     | 26%                  | 24%            | 25%                 | 25%            |\n| Competition | 24%                  | 27%            | 22%                 | 27%            |\n\nFinally, thanks again for the valuable suggestions, which have helped us improve the quality of our work.  Hope our responses address your concerns. Please feel free to contact us if you have more questions or suggestions!\n\n## Reference\n\n[1] ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation\n\n[2] JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models\n\n[3] MapCoder: Multi-Agent Code Generation for Competitive Problem Solving\n\n[4] AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation\n\n[5] Fleiss' Kappa: Measuring Agreement Among Multiple Raters"}, {"review_id": "8877kug7wg", "reviewer": "Reviewer_kobS", "summary": "This paper introduces a benchmark and evaluation criteria to assess code generation taking into maintainability account. Their proposed approach includes an orchestrated pipeline of LLM agents. The agents are specialized in four software development phases and takes into account contextual awareness in inter-agent communication via AutoGen.", "questions": "Please refer to the weakness Section. I a happy to raise my score if my questions are addressed.", "limitations": "Please see my comments on weakness.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength\n\n1.\tProposes a maintainability metrics to measure\n\n2.\tExtensive evaluation on correctness and maintainability\n\n3.\tPerformance boost over high performing models in optimizing second round of code generation with modified requirements (Section 4.1). Previous models optimized for number of lines of codes which may lead to unmaintainable codes in the long run. The proposed method generates code with minimized inflated lines.\n\nWeakness\n\n1.\tIn Section 3.2.1, “We sample 30 problems from each dataset randomly, and extend them to 120 new problems by systematically modifying their requirements”\nCould the authors elaborate on their generation scheme of requirements?\n\n2.\tSection 3.2.2: how the authors evaluate solution-test co-evaluation?\n\n3. The code generation and code optimization agents need detailed description.n. Please see details below.\n\n3a. The code generation agent. \n“After the initial generation, a test sample selected from the test set is inserted as insertion for iterative debugging”.\nGenerating high quality test cases is a challenging software engineering problem. Did the authors generate new test cases for each four-requirement generation in Section 3.2.2? \n\n3b. The code optimization agent contributes to most performance boost, as demonstrated in ablation test in Table 4. It would be helpful to have detailed analysis of the agent.\nIn the code optimization agent in Section 3.2.2, the authors mentioned “…iterative process continues until the code meets requirements and pass tests, ensuring functional, efficient, and reliable solutions”\n Could the authors elaborate on \n1)\tNumber of test cases\n2)\tNumber of iterations\n3)\tHow they measured efficiency", "was_revised": true, "final_justification": "I have read all the authors' rebuttal. I would like to keep my score unchanged.", "rebuttal": "Dear Reviewer kobS:\n\nThank you for your careful review and insightful feedback. We greatly appreciate the time and effort you have invested. Let us address your questions one by one.\n\n## Response to W1:\n\n> **(Requirement Generation Details)** In Section 3.2.1, \"We sample 30 problems from each dataset randomly, and extend them to 120 new problems by systematically modifying their requirements\" Elaborate more on generation scheme of requirements.\n\nTo ensure both diversity and quality of MaintainBench, we implemented a structured, multi-stage process for requirement changes:\n\n- For each initial problem, we systematically applied **four distinct requirement change patterns** using carefully designed prompts to simulate diverse real-world software maintenance scenarios.\n- To guarantee the quality of the resulting dataset, we followed a rigorous filtering protocol. We initially sampled over 50 problems from the source datasets. We then filtered out any problem that failed to produce a valid and meaningful new problem for all four change patterns. \n- Subsequently, these generated problems underwent a **manual evaluation** by human annotators to ensure their correctness, clarity, and relevance. This stringent process resulted in the final set of 30 high-quality base problems, which were extended to the 120 new problems used in our benchmark.\n\nWe have provided a detailed description of the change patterns and the generation pipeline in Appendix A/B/C. As suggested, we will add a more detailed scheme summary to Section 3.2.1 in the revision. Thanks!\n\n## Response to W2:\n\n> **(Solution-Test Co-evolution Details)** In Section 3.2.2, how to evaluate solution-test co-evolution?\n\nThanks for your question. We clarify that solution-test co-evolution is fully integrated and evaluated.\n\n- Firstly, we adapt the original solution code according to a specific requirement change pattern and perform a first-round check to ensure that the adapted code satisfies our formatting and structural conventions.\n- Secondly, based on the same requirement change, we simultaneously generate a new problem `new_problem`, a new solution `new_solution`, and a corresponding test case `new_test_case`.\n- Thirdly, human annotators manually inspect the `new_problem` to ensure that it is appropriate and that it faithfully reflects the intended requirement change; any problem that fails to meet this criterion is discarded.\n- Fourthly, for the remaining valid cases, annotators not oly execute `new_solution` on `new_test_case`, but also manually verify the correctness of both the `new_solution` and `new_test_case`, making necessary corrections when issues are identified.\n\nThis multi-stage pipeline ensures coherence between the problem, solution, and test case, with both automated checks and human validation in this solution-test co-evolution process. Please refer to Appendix B.2 for more details. As suggested, we will make more clarifications in the revision.\n\n## Response to W3:\n\n> **(Code Generation and** **Optimization** **Details)** For iterative debugging, did the authors generate new test cases for each four-requirement generation in Section 3.2.2? For code optimization, please elaborate on 1) Number of test cases 2) Number of iterations 3) How to measure efficiency.\n\nThank you for these important questions about our experimental setup. We will address them sequentially.\n\n**Test Cases for Iterative Debugging:** Yes. For each new problem, we generated a corresponding new set of test cases. This process involved both model-based generation and meticulous human verification. The goal was to ensure that the new test suite comprehensively covers both the newly introduced functionalities and the original functionalities.\n\n**Details of Code** **Optimization****:** For the iterative code optimization and debugging phase, our setup is as follows:\n\n- **Number of Test Cases:** The model uses **one** test case, randomly sampled from the newly generated test suite for that specific problem. This test case provides the feedback (i.e., pass or fail signal) that guides the model's subsequent refinement attempts. This setup is consistent with the evaluation protocols in prior works on code generation, such as AgentCoder and MapCoder  [1,2].\n- **Number of Iterations & Efficiency:** We set the maximum number of optimization iterations to **five**. This decision was based on empirical analysis and the goal of balancing performance improvement against computational cost (efficiency). As table shown, performance generally improves with more iterations. We observe a significant boost within the first three iterations, with performance gains starting to saturate around the fifth iteration. Limiting the process to five steps prevents excessive resource consumption while capturing the majority of the potential performance gains, a trade-off that aligns with previous research [1,2].\n\n*Tab 1: Performance vs. Number of* *Optimization* *Iterations (**opt**).*\n\n| Dataset      | Model                      | opt=0 | opt=1 | opt=2 | opt=3 | opt=4 | opt=5 |\n| ------------ | -------------------------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| CodeContests | MaintainCoder(gpt-4o-mini) | 28.0  | 28.6  | 31.7  | 30.4  | 32.6  | 32.6  |\n|              | MaintainCoder(deepseek-v3) | 25.7  | 28.6  | 27.6  | 30.4  | 32.6  | 37.9  |\n| xCodeEval    | MaintainCoder(gpt-4o-mini) | 17.2  | 18.0  | 19.2  | 23.4  | 25.0  | 27.3  |\n|              | MaintainCoder(deepseek-v3) | 25.0  | 24.1  | 27.9  | 30.8  | 32.6  | 36.7  |\n\nWe hope these responses have adequately addressed your concerns. Let us know if there are still any concerns!\n\n## Reference\n\n[1] MapCoder: Multi-Agent Code Generation for Competitive Problem Solving\n\n[2] AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation"}, {"review_id": "9aziQ6p3tg", "reviewer": "Reviewer_9Ho3", "summary": "Taking into consideration of software maintainability, which is not very broadly discussed in previous benchmarks, the author proposed a **MaintainBench** which transforms coding questions into different yet relevant tasks, aided with human labelling. The authors further proposed **MaintainCoder** that also features an agentic pipeline that follows current human developer workflows. Results are that **MaintainCoder** outperforms baseline and certain previous methods in both *correctness* and *maintainability metrics*.\n\nThe maintainability metrics are:\n\n  - Maintainability Index\n  - Cyclomatic Complexity (CC)\n  - Code Change Volume ($Code_{\\textit{diff}}$)\n  - Syntax Tree Similarity ($AST_\\textit{sim}$)", "questions": "1. This paper has made a wide comparison between coding LLMs, ranging from open-source models to closed-source models and in model sizes. Reasoning models are however not included in the comparison. Could you please clarify why is the experiment lacking this category?\n2. Number of tokens used are not reported in this paper. We'd like to know if the token cost is comparable to other code-gen methods or at least on a same magnitude.\n3. May RL training and other non inference-time techniques benefit from your benchmark and method?\n4. Can the model used for generating MaintainBench affect the results? If so, at what extent?\n5. Why is there a gap between Pass@5 on APPS-Dyn and APPS?\n\nI'm willing to increase my rating if you can provide more thorough analyses on (why your benchmark and method work), and explain why at least part of the weaknesses are so.", "limitations": "The authors have not thoroughly discussed the limitations of their work. We advise that the authors should address at least some of the following points (not exhaustive and open to discussions):\n\n- (w.r.t. Weaknesses) Can methods in this benchmark be likewise applied to longer tasks (e.g. a repository's whole lifespan)?\n- How would code maintainability improvements from LLM-generated code affect the overall software development process?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nThe authors identified that maintainability is not widely considered in current coding benchmarks and set out to resolve this issue. They leverage a custom agentic pipeline to transform existing coding problem sets into equivalent problems and *has included human-in-the-loop* to assure benchmark quality. The number of problems sum to to around 1000 questions in total.\n\n**Weaknesses:**\n\nThe author has included a short case study w.r.t. to a MaintainBench problem, but we expect an end-to-end case study such that the entire prompts and outputs are shown, both in the 'benchmark generation' process (the problem does not necessarily has to be in the actual benchmark) and in the 'MaintainCoder' process. This is crucial to grasping 'how does your agentic pipelines work' in a more fast and concrete way. The current case study is too un-verbose and provide few insights into the details of your method.\n\nWhile it may seem reasonable that, human practices, when followed by LLMs can lead to better performance in either correctness or maintainability, there is no assurance that prebuilt pipelines are the best way to achieve (large software) coding. Currently, reasoning models have large likelihoods in surpassing non-reasoning language models in multiple aspects. It is for this reason we strongly suggest the authors to include reasoning models like *o1* in the comparison, and we'd like to see that your method would still bring about some improvement atop the base models.\n\nSince this benchmark is aimed at improving or at least measuring maintainability, we suggest that problem sets like APPS, MBPP may not suffice in terms of problem complexity. The authors should consider including more complex tasks like SWE-Bench. The method coverage is also not very broad, as only *AgentCoder* and *MapCoder* are included. We suggest that whenever time and bandwidth permits, more methods should be also included in the comparison.", "was_revised": true, "final_justification": "I appreciate the efforts made by the authors during the rebuttal process, especially in providing a detailed case study and detailed experiment results, and have fully addressed my concerns on the efficacy of the proposed method and token consumption. It should be worth noting that should this paper be accepted, the authors should make sure that the tables be modified to perform fair and equivalent comparisons between methods and methods, between models and models, and between methods and non-methods.\n\nSpecifically, my suggestion is that results from plain GPT, results from the very same GPT applied with your methods should appear in the same table side-by-side at least somewhere in your paper, and have it referenced in the Analysis section. With that properly addressed, I'm pretty confident that these results could support the maintainability of *MaintainCoder* at least on simple tasks.\n\nHowever, one must take into consideration that the capabilities of base models are already strong enough to cover an abundance of coding tasks, and that agent-based approaches have emerged while this paper is being reviewed, it may appear necessary that *MaintainCoder* be compared with newer models over more sophisticated tasks like SWE-bench. These tasks are more challenging in terms of software complexity and maintainability and deserve more attention in potential future work.\n\nThus, I have changed my score from 3 to 4 (borderline accept) for this paper given its contribution to preliminary results in maintainability for LLM coding.", "rebuttal": "Dear Reviewer 9Ho3:\n\nThank you for your thorough review and insightful feedback. We appreciate the opportunity to clarify our contributions and address your concerns. Below are our responses to your comments.\n\n## Response to W1:\n\n> **(End-to-End Case Study)** The author has included a short case study w.r.t. to a MaintainBench problem, but we expect an end-to-end case study such that the entire prompts and outputs are shown, both in the 'benchmark generation' process and in the 'MaintainCoder' process.\n\nThank you for this excellent suggestion. We are happy to provide a case study of MaintainCoder.\n\nMaintainCoder is designed to function not just as a code generator but as a software architect, creating a forward-looking architecture. **For demonstration of a 2048 game. MaintainCoder adopts a Model-View-Controller (MVC) architecture**, producing the following well-organized file structure:\n\n```Plain\n2048-python-game/\n├── README.md\n├── main.py\n└── src/\n    └── game_2048/\n        ├── __init__.py\n        │\n        ├── common/\n        │   ├── __init__.py\n        │   └── enums.py\n        │\n        ├── controller/\n        │   ├── __init__.py\n        │   └── game_controller.py\n        │\n        ├── model/\n        │   ├── __init__.py\n        │   └── game_state.py\n        │\n        ├── view/\n        │   ├── __init__.py\n        │   ├── game_view.py\n        │   └── ui_constants.py\n        │\n        └── utils/\n            ├── __init__.py\n            └── matrix_utils.py\n```\n\nIf the first requirement is \"change the color of the tiles\" (a View modification), and the second is \"add a new '512' tile\" (a Model modification), the changes are localized and simple. **This architectural foresight provides persistent benefits over multiple maintenance cycles.**\n\n## Response to W2:\n\n> **(Reasoning Models)** We strongly suggest the authors to include reasoning models like o1 in the comparison, and we'd like to see that your method would still bring about some improvement atop the base models.\n\nOur initial experiments did not include reasoning models primarily due to high computational cost.\n\nAs suggested, we conducted experiments with o3-mini. As table shown, MaintainCoder significantly **decreases maintenance costs, and improves the maintenance success rate** (Pass@5). This demonstrates the robustness and general applicability of our approach. Thank you for this suggestion.\n\n| Model                       | MI ↑ | CC ↓ | Pass@5 (%) ↑ | AST_sim ↑ | Code_diff (%) ↓ | Code_diff (abs) ↓ |\n| --------------------------- | ---- | ---- | ------------ | --------- | --------------- | ----------------- |\n| GPT-4o-mini                 | 56.5 | 6.52 | 22.5         | 0.640     | 95.3            | 21.2              |\n| MaintainCoder (GPT-4o-mini) | 64.1 | 3.11 | 30.5         | 0.812     | 28.4            | 20.5              |\n| o3-mini                     | 52.1 | 11.3 | 30.3         | 0.661     | 101.6           | 32.9              |\n| MaintainCoder (o3-mini)     | 62.3 | 3.85 | 36.4         | 0.794     | 27.8            | 21.3              |\n\n## Response to W3:\n\n> **(More benchmarks and methods)** The authors should consider including more complex tasks like SWE-Bench. The method coverage is also not very broad, as only AgentCoder and MapCoder are included. We suggest that whenever time and bandwidth permits, more methods should be also included in the comparison.\n\n**Regarding Benchmarks:** We agree that evaluating on repository-level tasks like SWE-Bench is an important future direction. As demonstrated in Response to W1, **MaintainCoder is capable of generating repository-level code**. However, building a *maintainability* benchmark at that scale presents significant challenges in constructing and verifying the dynamic requirements and test cases. The field of code maintainability evaluation is still in its early stages. We believe **MaintainBench serves as a crucial first step, analogous to how simpler benchmarks like HumanEval and MBPP catalyzed research in code correctness**.\n\n**Regarding Methods:** As you and Reviewer wgB7 suggested, we have included more multi-agent systems. The results are shown below.\n\n*Tab2: More multi-agent baselines (MetaGPT needs gpt-4.1-mini for strong tool call capability, ChatDev has different output format and is not listed).*\n\n| CodeContest-Dyn             | MI↑  | CC↓  | pass@5↑ | AST_sim↑ | Code_diff^per↓ | Code_diff^abs↓ |\n| --------------------------- | ---- | ---- | ------- | -------- | -------------- | -------------- |\n| MaintainCoder (gpt-4o-mini) | 65.8 | 2.68 | 32.6    | 0.833    | 23.2           | 17.4           |\n| MetaGPT (gpt-4.1-mini)      | 55.2 | 7.63 | 30.3    | 0.760    | 44.3           | 18.6           |\n| EvoMAC (gpt-4o-mini)        | 62.6 | 5.18 | 26.5    | 0.685    | 60.1           | 20.0           |\n\nThank you for pushing us to broaden the scope of our evaluation. We will add these results and a detailed discussion of related works [3, 4, 5, 6].\n\n## Response to Q1:\n\n> **(Reasoning Models)** This paper has made a wide comparison between coding LLMs, ranging from open-source models to closed-source models and in model sizes. Reasoning models are however not included in the comparison. Could you please clarify why is the experiment lacking this category?\n\nThank you for this question. We have provided a detailed answer in our **Response** **to W2**.\n\n## Response to Q2:\n\n> **(Token Cost)** If the token cost is comparable to other code-gen methods or at least on a same magnitude.\n\n**Yes, the token cost is comparable to other methods like MapCoder**. We have performed the cost analysis and find that multi-agent pipelines incur higher computational costs. But for initial code generation, we believe this is a worthwhile trade-off.\n\n*Tab4: Computational overhead analysis (token usage).*\n\n| **Token Usage** | **MaintainCoder** | **MapCoder** | **o3-mini** | **MetaGPT/ChatDev** | **GPT-4o-mini** |\n| --------------- | ----------------- | ------------ | ----------- | ------------------- | --------------- |\n| CodeContests    | 33.1k             | 38.7k        | 20.8k       | 50k+                | 2.5k            |\n| xCodeEval       | 29.6k             | 23.5k        | 21.2k       | 50k+                | 2.3k            |\n\nSpecifically, the token usage of MaintainCoder is on par with MapCoder or reasoning model o3-mini, and much lower than MetaGPT/ChatDev [5,6]. But **MaintainCoder achieves a >60% improvement in maintainability and higher correctness**. The initial investment is justified by the drastic reduction in subsequent human effort required for debugging and refactoring. The table below details the token consumption per stage in our pipeline.\n\n*Tab5: More fine-grained token usage.*\n\n| CodeContests | Requirement Analysis | Design Pattern Selection | Framework Design | Supervisor | Code Implementation | Code Modification | Code Extraction | Total |\n| ------------ | -------------------- | ------------------------ | ---------------- | ---------- | ------------------- | ----------------- | --------------- | ----- |\n| tokens(k)    | 1.7                  | 2.3                      | 6.7              | 6.7        | 3.1                 | 9.0               | 3.6             | 33.1  |\n\n## Response to Q3:\n\n> **(Potential Application)** May RL training and other non inference-time techniques benefit from your benchmark and method?\n\nThank you for this insightful question. **Yes, our work can indeed benefit RL training**. As we also discuss in our response to Reviewer AvR7, the metrics in MaintainBench (both the first-order estimates and the dynamic indicators) are designed to achieve an effective balance between **evaluation accuracy and computational efficiency**. This makes them highly suitable to be used as a reward signal for RL training, aiming at optimizing code generation models for maintainability.\n\n## Response to Q4:\n\n> **(Model Sensitivity)** Can the model used for generating MaintainBench affect the results? If so, at what extent?\n\nWe acknowledge that using GPT-4o for generating problem variations could potentially introduce a model bias. We tried our best to mitigate this, in line with best practices for synthetic data generation [1, 2]:\n\n1. **Human Expert Validation:** Each problem variation and its corresponding solution were reviewed and refined by human experts to ensure quality and correctness.\n2. **Execution-Based Verification:** All solutions and their evolutions are verified through execution against a comprehensive test suite.\n3. **Traceable Co-evolution:** Solutions and test cases co-evolve, maintaining logical consistency.\n\nMost importantly, the core conclusions of our paper are drawn from **vertical comparisons** (e.g., comparing MaintainCoder (DeepSeek-V3) to the base DeepSeek-V3), not from horizontal comparisons between different model families. **These vertical experiments are robust and independent of potential benchmark bias. Therefore, this does not affect the validity of experimental conclusions.**\n\n## Response to Q5:\n\n> Why is there a gap between Pass@5 on APPS-Dyn and APPS?\n\nThank you for the sharp observation. The pass@5 on APPS-Dyn measures the correctness of the code *after* it has been modified to meet a new requirement based on old solution. **The evaluation target is different** from the initial code generation measured by the original APPS benchmark. Thus, it is normal to have different scores.\n\nThanks again for the detailed comments. Your feedback is valuable for improving our paper. Let us know if there are still any concerns.\n\n[1] On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey.\n\n[2] Best Practices and Lessons Learned on Synthetic Data.\n\n[3] Metagpt: Meta programming for multi-agent collaborative framework\n\n[4] Self-evolving multi-agent collaboration networks for software development\n\n[5] Chatdev: Communicative agents for software development\n\n[6] SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development"}, {"review_id": "EzYGxJBsRY", "reviewer": "Reviewer_wgB7", "summary": "This paper addresses a critical gap in code generation by focusing on maintainability under evolving requirements rather than just initial correctness. The authors introduce two main contributions: (1) **MaintainBench**, the first benchmark for evaluating code maintainability through requirement evolution cycles, extending five established datasets (HumanEval, MBPP, APPS, CodeContests, xCodeEval) with systematic requirement changes; and (2) **MaintainCoder**, a multi-agent system that integrates the Waterfall model, design patterns, and multi-agent collaboration to generate maintainable code. The work proposes novel dynamic metrics (Pass@k, ASTsim, Codediff) to evaluate maintenance efforts.", "questions": "- Given the significant reliance on GPT-4o for MaintainBench construction, how do you ensure benchmark validity and avoid favoring GPT-based systems? Could you provide inter-annotator agreement scores and more details on the manual validation process to demonstrate the quality of human oversight?\n- Could you provide detailed computational overhead analysis comparing MaintainCoder to baseline methods?\n- How would MaintainCoder perform across multiple sequential requirement changes rather than single cycles? Have you considered evaluating cumulative effects of iterative maintenance on code quality, which would better reflect real-world software evolution?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The paper addresses a critical and underexplored gap in code generation by focusing on maintainability under evolving requirements rather than just initial correctness.\n- MaintainBench represents a novel and valuable contribution as the first benchmark for evaluating dynamic maintainability through requirement evolution cycles, with systematic extension of five established datasets and four well-motivated requirement change patterns.\n- The work provides both immediate practical value and opens promising research directions, potentially influencing how the community evaluates code generation systems.\n\n### Weaknesses\n\n- The mathematical formulation, this is, the Monte Carlo approximation with first-order truncation may oversimplify the complexity of long-term maintenance scenarios. Need more explanation about the motivation and relationship between the formulation and the real-world unlimited requirement evolution scenarios. It would make this idea more clear by going through a real-world coding case and the examples about each notation in this dynamic evolving software development case.\n- MaintainCoder's design lacks novelty. It applies a fixed conventional software development workflow (Waterfall model + design patterns) within a multi-agent framework. This type of workflow-fixed multi-agent system mimicking real development scenarios has been extensively discussed and explored in prior work[1,2,3]. Could you compare with these methods and highlight your novelty? Why this framework could handle the dynamic requirements while others could not?\n- Given that maintainability principles should be language-agnostic, MaintainBench's limitation to Python represents weakness where the benchmark should ideally support different statically-typed languages and diverse programming paradigms. Could this method be extend to more general language scenarios?\n\n[1] Hong S, Zheng X, Chen J, et al. Metagpt: Meta programming for multi-agent collaborative framework[J]. arXiv preprint arXiv:2308.00352, 2023, 3(4): 6.\n\n[2] Qian C, Liu W, Liu H, et al. Chatdev: Communicative agents for software development[J]. arXiv preprint arXiv:2307.07924, 2023.\n\n[3] Hu Y, Cai Y, Du Y, et al. Self-evolving multi-agent collaboration networks for software development[J]. arXiv preprint arXiv:2410.16946, 2024.", "was_revised": true, "final_justification": "The author's rebuttal addresses some of the concerns. However, the comparison with previous works such as MetaGPT and EvoMAC is inconsistent in terms of LLM settings, alternating between gpt-4o-mini and gpt-4.1-mini. Additionally, the claimed novelty of the Reactive vs. Proactive distinction does not account for EvoMAC, which also follows a proactive evolution approach. Overall, while the paper still requires further improvements, it has its merits and contributions. I will maintain a positive score.", "rebuttal": "Dear Reviewer wgB7:\n\nWe sincerely appreciate your valuable feedback and suggestions, which are helpful in improving our paper.\n\nBelow, we address each comment in detail.\n\n## Response to W1:\n\n> **(Long-term Maintenance)** Monte Carlo approximation with first-order truncation may oversimplify the complexity of long-term maintenance scenarios. Need more explanation about the motivation and relationship between the formulation and the real-world unlimited requirement evolution scenarios. A real-world coding case would make this idea more clear.\n\nWe agree that real-world maintenance is a long-term, multi-round process. However, simulating multi-round maintenance for evaluation is computationally prohibitive. We believe that **code maintainability is an intrinsic property**, and thus the single-round maintenance cost should be highly correlated with the cumulative cost of multiple rounds. **A single round, therefore, can serve as an effective and efficient proxy.**\n\nTo validate this, we synthesized a second round of requirement changes for MaintainBench. As shown below, the costs from the **second round** **exhibit a high correlation with the first-round costs**, and are slightly more than the first ones. This suggests that the first-order approximation provides a good balance between evaluation accuracy and computational feasibility. \n\n*Tab1: First-round and Second-round Maintenance Costs (AST_sim & Code_diff) on CodeContests-Dyn.*\n\n| Model  | 1st Round Cost |  | 2nd Round Cost |  |\n|  - |  - |  - |  - |  - |\n|  | AST_sim ↑  | Code_diff (%) ↓ | AST_sim ↑  | Code_diff (%) ↓ |\n| GPT-4o-mini  | 0.661  | 90.1  | 0.645  | 96.2  |\n| MaintainCoder (GPT-4o-mini) | 0.833  | 23.2  | 0.811  | 29.4  |\n| DeepSeek-V3  | 0.718  | 87.2  | 0.694  | 92.5  |\n| MaintainCoder (DeepSeek-V3) | 0.788  | 43.2  | 0.765  | 48.9  |\n\nGiven word count limit, we provide 2048 game for a coding case.\n\n```Plain\n2048-python-game/\n├── README.md\n├── main.py\n└── src/\n  └── game_2048/\n  ├── __init__.py\n  │\n  ├── common/\n  │  ├── __init__.py\n  │  └── enums.py\n  │\n  ├── controller/\n  │  ├── __init__.py\n  │  └── game_controller.py\n  │\n  ├── model/\n  │  ├── __init__.py\n  │  └── game_state.py\n  │\n  ├── view/\n  │  ├── __init__.py\n  │  ├── game_view.py\n  │  └── ui_constants.py\n  │\n  └── utils/\n  ├── __init__.py\n  └── matrix_utils.py\n```\n\nMaintainCoder proactively adopt an MVC (Model-View-Controller) architecture. This separation of concerns is an intrinsic quality of the code. If the first requirement is \"change the color of the tiles\" (a View modification), and the second is \"add a new '512' tile\" (a Model modification), the changes are localized and simple. **This architectural foresight provides persistent benefits over multiple maintenance cycles.** \n\nWe will add the discussions in the revision.\n\n## Response to W2:\n\n> **(Design Novelty)** Could you compare with prior work that mimics real development scenarios and highlight your novelty? Why could this framework handle the dynamic requirements while others could not?\n\nThe fundamental distinction and novelty of our work lie in a paradigm shift: from reactive problem-solving to proactive architectural design.\n\n- **Reactive vs. Proactive:** Prior systems like MetaGPT and ChatDev [1, 2] are *reactive*; they are optimized to produce a functionally correct solution for the *current* requirements. In contrast, MaintainCoder is *proactive*; it is designed to generate a solution that is not only correct now but also robust and easy to adapt.\n- **Focus on Architectural Quality:** Our framework's novelty is not merely in its multi-agent structure but in **explicitly embedding proven** **software engineering** **principles** (e.g., design patterns, separation of concerns) into the agents' core logic. This ensures the code is designed for evolution, not just for initial generation.\n\nThe supplemented experiments are below.  We will add these results and a detailed discussion of related works [1, 2, 3, 4] to better position our contribution.\n\n*Tab2: More multi-agent baselines (MetaGPT needs gpt-4.1-mini for strong tool call capability, ChatDev has different output format and is not listed).*\n\n| CodeContest-Dyn  | MI↑  | CC↓  | pass@5↑ | AST_sim↑ | Code_diff^per↓ | Code_diff^abs↓ |\n|  - |  - |  - |  - |  - |  - |  - |\n| MaintainCoder (gpt-4o-mini) | 65.8 | 2.68 | 32.6    | 0.833    | 23.2           | 17.4           |\n| MetaGPT (gpt-4.1-mini)  | 55.2 | 7.63 | 30.3  | 0.760  | 44.3  | 18.6  |\n| EvoMAC (gpt-4o-mini)  | 62.6 | 5.18 | 26.5  | 0.685  | 60.1  | 20.0  |\n\n## Response to W3:\n\n> **(Language Restriction)** MaintainBench ideally should support different statically-typed languages and diverse programming paradigms. Could this method be extended to more general languages?\n\nYes! For industry, design patterns are successful and widely adopted in Java/C++ software development. For experiments, we translated MaintainBench from Python to Java, and evaluated MaintainCoder against the baseline on Java version. The results are consistent with Python: **MaintainCoder improves maintainability metrics, indicating our principles are not language-specific**. The Java performance is generally slightly lower than Python, due to imbalanced training data. We will supplement these discussion in the revision.\n\n*Tab3: Java version performance on CodeContests-Dyn.*\n\n| Model  | MI ↑ | CC ↓ | Pass@5 (%) ↑ | AST_sim ↑ | Code_diff (%) ↓ | Code_diff (abs) ↓ |\n|  - |  - |  - |  - |  - |  - |  - |\n| GPT-4o-mini  | 56.5 | 6.52 | 22.5  | 0.640  | 95.3  | 21.2  |\n| MaintainCoder (GPT-4o-mini) | 64.1 | 3.11 | 30.5  | 0.812  | 28.4  | 20.5  |\n| DeepSeek-V3  | 54.2 | 7.21 | 25.0  | 0.703  | 91.8  | 23.8  |\n| MaintainCoder (DeepSeek-V3) | 62.5 | 3.84 | 36.0  | 0.795  | 45.1  | 22.7  |\n\n## Response to Q1:\n\n> **(Benchmark Construction)** How do you ensure benchmark validity and avoid favoring GPT-based systems? Provide inter-annotator agreement scores and more details on the manual validation process to demonstrate the quality of human oversight.\n\n**Model Bias:** Despite GPT-4o was used for the initial generation, the benchmark validity ultimately relies on human expert validation rather than just the LLMs. **All final test cases and solutions underwent execution-based verification and expert review,** ensuring that the ground truth is robust to the generating model biases. \n\n**Annotation details:** Each new generated solution and test input were annotated through recruited annotator and independently validated by two of authors. To further assess annotation reliability, **we obtained inter-annotator agreement using Fleiss’ Kappa from two of our authors, obtaining a score of 0.899, indicating high reliability and vadility** [7]. These details will be included in the revision.\n\n## Response to Q2:\n\n> **(Token Cost)** Could you provide detailed computational overhead analysis.\n\n*Tab4: Computational overhead analysis (token usage).*\n\n| **Token Usage** | **MaintainCoder** | **MapCoder** | **o3-mini** | **MetaGPT/ChatDev** | **GPT-4o-mini** |\n|  - |  - |  - |  - |  - |  - |\n| CodeContests  | 33.1k  | 38.7k  | 20.8k  | 50k+  | 2.5k  |\n| xCodeEval  | 29.6k  | 23.5k  | 21.2k  | 50k+  | 2.3k  |\n\nWe have performed the requested analysis and find that multi-agent pipelines incur higher computational costs. But for initial code generation, we believe this is a **worthwhile trade-off**. Specifically, the token usage of MaintainCoder is on par with MapCoder or reasoning model o3-mini, and much lower than MetaGPT/ChatDev [5,6]. But **MaintainCoder achieves a >60% improvement in maintainability and higher correctness**. The initial investment is justified by the drastic reduction in subsequent human effort required for debugging and refactoring. Table below details the token consumption per stage in our pipeline.\n\n*Tab5: More fine-grained computational overhead analysis (token usage).*\n\n| CodeContests | Requirement Analysis | Design Pattern Selection | Framework Design | Supervisor | Code Implementation | Code Modification | Code Extraction | Total |\n|  - |  - |  - |  - |  - |  - |  - |  - |  - |\n| tokens(k)  | 1.7  | 2.3  | 6.7  | 6.7  | 3.1  | 9..0  | 3.6  | 33  |\n\n## Response to Q3:\n\n> **(Long-term Maintenance)** How would MaintainCoder perform across multiple sequential requirement changes rather than single cycles? Have you considered evaluating cumulative effects of iterative maintenance on code quality, which would better reflect real-world software evolution?\n\nFor multiple sequential requirement changes, please refer to **Response to W1**.\n\nWe agree that the evaluation of \"cumulative effects of iterative maintenance on code quality\" is a valuable research direction. **Such a study would need to measure multiple facets of quality, including not only correctness and efficiency but also the very concept of maintainability that our work focuses on**. Indeed, a benchmark like MaintainBench would be a necessary component for maintainability measure. We will add this point to our \"Future Work\" section, highlighting it as a promising avenue for research on code generation for long-term software evolution.\n\nThank you again for your thoughtful review and for helping improve our work. If you have any further questions or suggestions, please let us know at any time.\n\n## Reference\n\n[1] Metagpt: Meta programming for multi-agent collaborative framework\n\n[2] Chatdev: Communicative agents for software development\n\n[3] Self-evolving multi-agent collaboration networks for software development\n\n[4] SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development\n\n[5] MapCoder: Multi-Agent Code Generation for Competitive Problem Solving\n\n[6] AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation\n\n[7] Fleiss' Kappa: Measuring Agreement Among Multiple Raters"}]

Author rebuttal:

Meta-review: - Summary: This paper focuses on maintainability in code generation and proposes MaintainCoder as a multi-agent system to generate maintainable code under dynamic requirements integrating Waterfall model and classical design patterns. The paper also introduces MaintainBench, extension of common benchmarks for code generation by subjecting the code generation task to different requirements. They also propose novel metrics capturing various aspects of maintenance efforts. 
- Strengths:
  - The paper explores a significant problem in code generation, motivated by real world scenarios and would be of interest to the Neurips community.
  - The experiments conducted are rigorous, spanning various base models, difficulty levels, agent frameworks.
  - The paper’s contributions on the benchmark and the agent system are novel.
- Weaknesses:
  - The reviewers mentioned the lack of evaluations with respect to human written code. Even though the authors tried to address this during the rebuttal, I feel like the design of this would benefit from a round of peer review. The reviewers mentioned other limitations of evaluations like generalization to different languages, reasoning models and the authors addressed these concerns during the rebuttal phase.
  - Application of the approach to SWE-bench like benchmarks would be interesting.
- Suggestions:
  - It is suggested by reviewer 9Ho3 and wgB7 to make the comparisons with raw models and models paired with agentic approaches more clear as part of the results.
- Recommendation:
  - All of the four reviewers recommended acceptance of the paper, three of whom gave borderline acceptance scores.
  - This paper tackles an underexplored area in code generation and provides benchmarks to motivate further research in the area. I believe it will be of interest to the Neurips community and there is why I recommend acceptance.
Decision: Accept (poster)