Title: MaintainCoder: Maintainable Code Generation Under Dynamic Requirements
Venue: NeurIPS 2025
Accepted: yes
Keywords: Code Generation

Maintainability

Design Patterns

Requirement Evolution

Software Engineering
Abstract: Modern code generation has made significant strides in functional correctness and execution efficiency. However, these systems often overlook a critical dimension in real-world software development: \textit{maintainability}. To handle dynamic requirements with minimal rework, we propose \textbf{MaintainCoder} as a pioneering solution. It integrates the Waterfall model, design patterns, and multi-agent collaboration to systematically enhance cohesion, reduce coupling, achieving clear responsibility boundaries and better maintainability. We also introduce \textbf{MaintainBench}, a benchmark comprising requirement changes and novel dynamic metrics on maintenance efforts. Experiments demonstrate that existing code generation methods struggle to meet maintainability standards when requirements evolve. In contrast, MaintainCoder improves dynamic maintainability metrics by more than 60\% with even higher correctness of initial codes. Furthermore, while static metrics fail to accurately reflect maintainability and even contradict each other, our proposed dynamic metrics exhibit high consistency. Our work not only provides the foundation for maintainable code generation, but also highlights the need for more realistic and comprehensive code generation research. Resources: https://github.com/IAAR-Shanghai/MaintainCoder.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a critical gap in code generation by focusing on maintainability under evolving requirements rather than just initial correctness. The authors introduce two main contributions: (1) **MaintainBench**, the first benchmark for evaluating code maintainability through requirement evolution cycles, extending five established datasets (HumanEval, MBPP, APPS, CodeContests, xCodeEval) with systematic requirement changes; and (2) **MaintainCoder**, a multi-agent system that integrates the Waterfall model, design patterns, and multi-agent collaboration to generate maintainable code. The work proposes novel dynamic metrics (Pass@k, ASTsim, Codediff) to evaluate maintenance efforts.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper addresses a critical and underexplored gap in code generation by focusing on maintainability under evolving requirements rather than just initial correctness.\n- MaintainBench represents a novel and valuable contribution as the first benchmark for evaluating dynamic maintainability through requirement evolution cycles, with systematic extension of five established datasets and four well-motivated requirement change patterns.\n- The work provides both immediate practical value and opens promising research directions, potentially influencing how the community evaluates code generation systems.\n\n### Weaknesses\n\n- The mathematical formulation, this is, the Monte Carlo approximation with first-order truncation may oversimplify the complexity of long-term maintenance scenarios. Need more explanation about the motivation and relationship between the formulation and the real-world unlimited requirement evolution scenarios. It would make this idea more clear by going through a real-world coding case and the examples about each notation in this dynamic evolving software development case.\n- MaintainCoder's design lacks novelty. It applies a fixed conventional software development workflow (Waterfall model + design patterns) within a multi-agent framework. This type of workflow-fixed multi-agent system mimicking real development scenarios has been extensively discussed and explored in prior work[1,2,3]. Could you compare with these methods and highlight your novelty? Why this framework could handle the dynamic requirements while others could not?\n- Given that maintainability principles should be language-agnostic, MaintainBench's limitation to Python represents weakness where the benchmark should ideally support different statically-typed languages and diverse programming paradigms. Could this method be extend to more general language scenarios?\n\n[1] Hong S, Zheng X, Chen J, et al. Metagpt: Meta programming for multi-agent collaborative framework[J]. arXiv preprint arXiv:2308.00352, 2023, 3(4): 6.\n\n[2] Qian C, Liu W, Liu H, et al. Chatdev: Communicative agents for software development[J]. arXiv preprint arXiv:2307.07924, 2023.\n\n[3] Hu Y, Cai Y, Du Y, et al. Self-evolving multi-agent collaboration networks for software development[J]. arXiv preprint arXiv:2410.16946, 2024.\n\nQUESTIONS:\n- Given the significant reliance on GPT-4o for MaintainBench construction, how do you ensure benchmark validity and avoid favoring GPT-based systems? Could you provide inter-annotator agreement scores and more details on the manual validation process to demonstrate the quality of human oversight?\n- Could you provide detailed computational overhead analysis comparing MaintainCoder to baseline methods?\n- How would MaintainCoder perform across multiple sequential requirement changes rather than single cycles? Have you considered evaluating cumulative effects of iterative maintenance on code quality, which would better reflect real-world software evolution?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTaking into consideration of software maintainability, which is not very broadly discussed in previous benchmarks, the author proposed a **MaintainBench** which transforms coding questions into different yet relevant tasks, aided with human labelling. The authors further proposed **MaintainCoder** that also features an agentic pipeline that follows current human developer workflows. Results are that **MaintainCoder** outperforms baseline and certain previous methods in both *correctness* and *maintainability metrics*.\n\nThe maintainability metrics are:\n\n  - Maintainability Index\n  - Cyclomatic Complexity (CC)\n  - Code Change Volume ($Code_{\\textit{diff}}$)\n  - Syntax Tree Similarity ($AST_\\textit{sim}$)\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe authors identified that maintainability is not widely considered in current coding benchmarks and set out to resolve this issue. They leverage a custom agentic pipeline to transform existing coding problem sets into equivalent problems and *has included human-in-the-loop* to assure benchmark quality. The number of problems sum to to around 1000 questions in total.\n\n**Weaknesses:**\n\nThe author has included a short case study w.r.t. to a MaintainBench problem, but we expect an end-to-end case study such that the entire prompts and outputs are shown, both in the 'benchmark generation' process (the problem does not necessarily has to be in the actual benchmark) and in the 'MaintainCoder' process. This is crucial to grasping 'how does your agentic pipelines work' in a more fast and concrete way. The current case study is too un-verbose and provide few insights into the details of your method.\n\nWhile it may seem reasonable that, human practices, when followed by LLMs can lead to better performance in either correctness or maintainability, there is no assurance that prebuilt pipelines are the best way to achieve (large software) coding. Currently, reasoning models have large likelihoods in surpassing non-reasoning language models in multiple aspects. It is for this reason we strongly suggest the authors to include reasoning models like *o1* in the comparison, and we'd like to see that your method would still bring about some improvement atop the base models.\n\nSince this benchmark is aimed at improving or at least measuring maintainability, we suggest that problem sets like APPS, MBPP may not suffice in terms of problem complexity. The authors should consider including more complex tasks like SWE-Bench. The method coverage is also not very broad, as only *AgentCoder* and *MapCoder* are included. We suggest that whenever time and bandwidth permits, more methods should be also included in the comparison.\n\nQUESTIONS:\n1. This paper has made a wide comparison between coding LLMs, ranging from open-source models to closed-source models and in model sizes. Reasoning models are however not included in the comparison. Could you please clarify why is the experiment lacking this category?\n2. Number of tokens used are not reported in this paper. We'd like to know if the token cost is comparable to other code-gen methods or at least on a same magnitude.\n3. May RL training and other non inference-time techniques benefit from your benchmark and method?\n4. Can the model used for generating MaintainBench affect the results? If so, at what extent?\n5. Why is there a gap between Pass@5 on APPS-Dyn and APPS?\n\nI'm willing to increase my rating if you can provide more thorough analyses on (why your benchmark and method work), and explain why at least part of the weaknesses are so.\n\nLIMITATIONS:\nThe authors have not thoroughly discussed the limitations of their work. We advise that the authors should address at least some of the following points (not exhaustive and open to discussions):\n\n- (w.r.t. Weaknesses) Can methods in this benchmark be likewise applied to longer tasks (e.g. a repository's whole lifespan)?\n- How would code maintainability improvements from LLM-generated code affect the overall software development process?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a benchmark and evaluation criteria to assess code generation taking into maintainability account. Their proposed approach includes an orchestrated pipeline of LLM agents. The agents are specialized in four software development phases and takes into account contextual awareness in inter-agent communication via AutoGen.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1.\tProposes a maintainability metrics to measure\n\n2.\tExtensive evaluation on correctness and maintainability\n\n3.\tPerformance boost over high performing models in optimizing second round of code generation with modified requirements (Section 4.1). Previous models optimized for number of lines of codes which may lead to unmaintainable codes in the long run. The proposed method generates code with minimized inflated lines.\n\nWeakness\n\n1.\tIn Section 3.2.1, “We sample 30 problems from each dataset randomly, and extend them to 120 new problems by systematically modifying their requirements”\nCould the authors elaborate on their generation scheme of requirements?\n\n2.\tSection 3.2.2: how the authors evaluate solution-test co-evaluation?\n\n3. The code generation and code optimization agents need detailed description.n. Please see details below.\n\n3a. The code generation agent. \n“After the initial generation, a test sample selected from the test set is inserted as insertion for iterative debugging”.\nGenerating high quality test cases is a challenging software engineering problem. Did the authors generate new test cases for each four-requirement generation in Section 3.2.2? \n\n3b. The code optimization agent contributes to most performance boost, as demonstrated in ablation test in Table 4. It would be helpful to have detailed analysis of the agent.\nIn the code optimization agent in Section 3.2.2, the authors mentioned “…iterative process continues until the code meets requirements and pass tests, ensuring functional, efficient, and reliable solutions”\n Could the authors elaborate on \n1)\tNumber of test cases\n2)\tNumber of iterations\n3)\tHow they measured efficiency\n\nQUESTIONS:\nPlease refer to the weakness Section. I a happy to raise my score if my questions are addressed.\n\nLIMITATIONS:\nPlease see my comments on weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MaintainCoder, a framework for generating maintainable code under dynamic requirements, and MaintainBench, the first benchmark for evaluating code maintainability. Key contributions include:\n\n1. MaintainBench: Extends established benchmarks (e.g., HumanEval, MBPP) with 500+ Python tasks, covering four requirement change patterns (*functional extensions, interface modifications, data structure transformations, error handling enhancements*). Novel dynamic metrics (e.g., post-modification correctness `Pass@k`, code change volume `Code_diff`, AST similarity `AST_sim`) are proposed.\n2. MaintainCoder: A multi-agent system integrating the Waterfall model and classical design patterns. It enforces modularity, low coupling, and high cohesion through specialized agents (e.g., requirements analysis, pattern selection, framework evaluation).\n3. Key Findings:\n   - Existing methods (GPT-4o, AgentCoder, etc.) suffer significant maintainability degradation under requirement changes (e.g., `Pass@5` drops 15–30%).\n   - MaintainCoder improves dynamic metrics by >60% while boosting initial correctness.\n   - Traditional static metrics (e.g., MI, cyclomatic complexity) poorly correlate with maintainability, while dynamic metrics show high consistency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Problem Significance:\n   - Addresses the critical yet underexplored challenge of maintainability in code generation, aligning with real-world pain points (80% of software costs stem from maintenance).\n   - Highlights limitations of static benchmarks (e.g., HumanEval) and advocates for dynamic evaluation.\n2. Novel Contributions:\n   - MaintainBench:\n     - Systematically models requirement evolution via four change types (`Π_B`), covering 80% of maintenance scenarios (adaptive/perfective).\n     - Ensures quality via solution-test co-evolution (GPT-4o + human refinement).\n   - MaintainCoder:\n     - Uniquely combines software engineering principles (Waterfall model, design patterns) with multi-agent collaboration.\n     - Ablation studies validate the necessity of framework evaluation and code optimization modules (25–37% performance drop if removed).\n3. Rigorous Evaluation:\n   - Comprehensive Coverage: Tests 3 difficulty levels (entry/mixture/competition), 7 base models (GPT-4o, DeepSeek-V3, etc.), and 2 multi-agent frameworks (AgentCoder, MapCoder).\n   - Multi-dimensional Metrics: Exposes discrepancies between static metrics (MI/CC) and dynamic maintainability (e.g., high MI but low `Pass@5`).\n   - Robustness Checks: Consistent results across varying `Pass@k` (k=1–5) and Phase II generators.\n\nWeaknesses\n\n1. Theoretical Gaps in Dynamic Metrics:\n   - The Monte Carlo estimation for maintenance cost (`ℳ(C₀)`) lacks justification for first-order truncation. Discount factor `γ` is not discussed.\n   - `AST_sim` and `Code_diff` rely on syntactic similarity (via `difflib`), ignoring semantic equivalence (e.g., refactored but functionally identical code).\n2. Limited Experimental Comparisons:\n   - Missing Human Baseline: No comparison against code written/adapted by developers, leaving industrial relevance unclear.\n   - Language Restriction: Only Python is evaluated; generalization to stricter-typed languages (e.g., Java) is untested.\n3. Methodological Limitations:\n   - Black-box Pattern Selection: Design pattern choices rely on LLM reasoning without causal analysis (only cites correlation in [13]).\n   - Computational Cost: Multi-agent pipeline requires multiple LLM calls (≤3 framework evaluations + ≤5 optimizations), but overhead vs. single-model inference is unreported.\n4. Dataset Issues:\n   - Annotation Process: Appendix B.3 mentions \"multi-stage expert review\" but omits reviewer count, disagreement resolution, or inter-rater agreement (e.g., Kappa).\n   - Class Balance: Distribution of change types (`Π_B`) across difficulty levels is unanalyzed.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}