ARCHCODE: Incorporating Software Requirements in Code Generation with Large Language Models (2024)

Abstract: This paper aims to extend the code generation capability of large language models (LLMs) to automatically manage comprehensive software requirements from given textual descriptions. Such requirements include both functional (i.e. achieving expected behavior for inputs) and non-functional (e.g., time/space performance, robustness, maintainability) requirements. However, textual descriptions can either express requirements verbosely or may even omit some of them. We introduce ARCHCODE, a novel framework that leverages in-context learning to organize requirements observed in descriptions and to extrapolate unexpressed requirements from them. ARCHCODE generates requirements from given descriptions, conditioning them to produce code snippets and test cases. Each test case is tailored to one of the requirements, allowing for the ranking of code snippets based on the compliance of their execution results with the requirements. Public benchmarks show that ARCHCODE enhances to satisfy functional requirements, significantly improving Pass@k scores. Furthermore, we introduce HumanEval-NFR, the first evaluation of LLMs’ non-functional requirements in code generation, demonstrating ARCHCODE’s superiority over baseline methods. The implementation of ARCHCODE and the HumanEval-NFR benchmark are both publicly accessible.1

heading: 1 Introduction
text: Recent advancements in large language models (LLMs) have significantly improved code generation capabilities (Chen et al., 2021; Li et al., 2022; OpenAI, 2023). Although the primary goal for LLMs in this domain is to generate functionally correct code based on textual descriptions (Hendrycks et al., 2021; Austin et al., 2021; Chen et al., 2021; Li et al., 2022), real-world software development encompasses more than just functionality.
˚ Corresponding author. 1https://github.com/ldilab/ArchCode
In software engineering, software requirements provide a detailed framework describing what a software system is intended to achieve (Chung et al., 2012), divided into two categories (Glinz, 2007):
• Functional Requirements (FRs) dictate the behavior and functionality, e.g., input/output conditions, desired behavior of code, etc.
• Non-Functional Requirements (NFRs) are attributes or constraints beyond functionality, e.g., time and space performance, robustness, maintainability, reliability, etc.
Despite the critical role of software requirements, considering these criteria has not been studied actively in previous code generation works, merely
13520
directly from problem descriptions, often missing key requirements. ARCHCODE, in contrast, 1 reformulates (underlined) and extrapolates (not underlined) requirements from these descriptions, then 2 generates code and test cases to meet them comprehensively. Best viewed in color.
generating code directly from textual descriptions. However, textual descriptions might express requirements verbosely or even omit them. As illustrated in Figure 1a and 2 (upper right), this may result in code that neglects many desirable requirements. Code filtering based on generated test cases (Li et al., 2022; Chen et al., 2023; Huang et al., 2023) shares the same problem, as test cases often fail to cover a broader range of requirements. Consequently, the generated code might exhibit unexpected behaviors for valid inputs, ignoring FRs. Similarly, overlooking NFRs can result in time/space inefficiencies, potential system failures, or challenges in maintenance. Nevertheless, achieving conciseness in the textual descriptions of software requirements necessitates significant human effort (Perry and Wolf, 1992; Bass et al., 2003).
We introduce ARCHCODE, a novel framework that automatically incorporates software requirements from textual descriptions, then directs LLMs to align code and test case generation with those requirements, as illustrated in Figure 1b. Specifically, ARCHCODE leverages In-Context Learning
(ICL; Kojima et al., 2022; Shao et al., 2023; Zhang et al., 2023c) for adaptability, utilizing LLMs’ extensive reasoning abilities to learn within context, thereby avoiding costly parameter updates. For code generation, each in-context example comprises a triplet—a textual description, a list of software requirements (including both those expressed and unexpressed in the description), and corresponding code that satisfies all these requirements. For test case generation, we simply switch from code to test cases, each of which verifies a specific requirement. ARCHCODE prepends in-context examples to test descriptions, guiding LLMs to: 1) reformulate explicit requirements in descriptions, 2) deduce implicit requirements from their parametric knowledge, 3) generate code that fulfills these requirements, and 4) produce test cases for verifying each requirement, as shown in Figure 2.
We integrate ARCHCODE with WizardCoder (Luo et al., 2023) and GPT-3.5-Turbo (OpenAI, 2022), and assess the performance on HumanEval (Chen et al., 2021) and CodeContests (Li et al., 2022). The results confirm
that ARCHCODE notably outperforms existing techniques in terms of the satisfaction of FRs—surpassing GPT-4’s Pass@1 score on both benchmarks and achieving new state-of-the-art on CodeContests. Moreover, we introduce HumanEval-NFR based on HumanEval, the first benchmark to evaluate NFRs alongside FRs, to confirm that ARCHCODE is also effective in pursuing NFRs.
Our main contributions are as follows:
• We propose ARCHCODE, a novel framework that leverages ICL to incorporate software requirements in code generation.
• ARCHCODE with GPT-3.5-Turbo surpasses GPT-4’s Pass@1 scores on both HumanEval and CodeContests by 4.81%p and 10.45%p, while requiring 50ˆ smaller number of test cases to be generated compared to existing methods.
• We introduce HumanEval-NFR, the first code generation benchmark for NFR evaluation to confirm the effectiveness of ARCHCODE for NFR satisfaction.

heading: 2 Related Work
text: Despite the fact that LLMs recently have shown impressive capabilities in code generation, the majority of evaluations have focused solely on functional requirements (FRs; Hendrycks et al., 2021; Austin et al., 2021; Chen et al., 2021; Li et al., 2022).

heading: Solely Targeting Functional Requirements
text: Early research such as Feng et al. (2020), Chen et al. (2021), Brown et al. (2020), and Li et al. (2022) directly generates code from natural language descriptions, which may not fully capture all software requirements due to their vagueness or imperfections. Later studies (Jiang et al., 2023; Li et al., 2023; Zhang et al., 2023a) have targeted to better capture functional requirements by generating code-like outlines via in-context learning (ICL; Kojima et al., 2022; Shao et al., 2023; Zhang et al., 2023c). More recent methods enhance FR satisfaction through self-verification of the generated code: On one hand, code filtering utilizes ‘over-generatethen-filter’ strategies, where the filtering can be achieved either by predicting functional correctness without code execution (Inala et al., 2022; Ni et al., 2023; Zhang et al., 2023b), or execution with given (Shi et al., 2022b) or generated test cases (Li
et al., 2022; Chen et al., 2023; Huang et al., 2023). On the other hand, code refinement iteratively reflects and refines the generated code to improve its functionality via execution results of current version of code with generated test cases (Shinn et al., 2023).
Targeting Narrow Scope of Non-Functional Requirements While much research covers FRs, few studies have addressed specific attributes of non-functional requirements (NFRs) such as reliability and robustness (Deng et al., 2023; Xia et al., 2023), or time/space performance (Madaan et al., 2023; Luo et al., 2023).
Our Distinction Table 1 summarizes the distinction of our framework compared with existing code generation approaches. To the best of our knowledge, ARCHCODE is the first study that employs ICL to systematically extract and interpret software requirements from descriptions, ensuring the generated code and test cases closely aligns with these requirements. In addition, we introduce HumanEvalNFR, a variant version of the HumanEval (Chen et al., 2021) benchmark that can assess the fulfillment of NFRs.

heading: Problem Description

heading: 3 The ARCHCODE Framework
text: We propose ARCHCODE, a novel code generation framework that employs In-Context Learning (ICL) to LLMs, incorporating software requirements in code generation. As shown in Figure 3, ARCHCODE delineates software requirements from textual descriptions, generates code, then verifies it using custom test cases.2
Formally, given a problem space P and a code space C, the code generation task is to build a function F : P Ñ C that maps each textual problem description p P P into its corresponding code implementation c P C. ARCHCODE decomposes F by F “ g ˝ f . g : P Ñ P ˆ R maps problems to problem-requirements pairs, and f : P ˆ R Ñ C generates code from the problemrequirements pairs, where R is a space of software requirements. The test case generation function H : P Ñ T is also decomposed by ARCHCODE into H “ g ˝ h, where T is the space of test cases and h : P ˆ R Ñ T .

heading: 3.1 Delineating Software Requirements
text: ARCHCODE leverages ICL to let an LLM generate a set of software requirements, either reformulated from a given textual description, or extrapolated from the description by the LLM’s learnt parametric knowledge. Formally, given in-context examples of description-requirements pairs and the
2More details on input/output formats, in-context examples and hyperparameters are provided in Appendix A.
target description p, the LLM returns the list of software requirements
r̂ “ gprp11, r11s; rp12, r12s; ...; rpsq, (1)
where p1i and r1i “ rr1i , r2i , ...s is the description and its corresponding requirement list of i-th example pair, each rji in r 1 i is a requirement, and r̂ is the list of generated requirements. Specifically, based on the established classifications by Glinz (2007), we further break down FRs and NFRs into distinct categories that our study focuses on.
Target FRs Our approach narrows down FRs into three subtypes:
• Input/Output Conditions: Analogous to the preconditions and the postconditions in Design by Contract (Meyer, 1992), these define the desired functionality of the code by specifying valid inputs and expected outputs.
• Expected Behavior: Along with Input/Output Conditions, it explains the functionality of the target code by reformulating the description into a series of operations for valid inputs that are applicable in most general scenarios.
• Edge Cases: While this term generally comprises an array of corner cases, we restrict the scope to only consider valid inputs that necessitate distinct treatment. These include, for example, processing an empty list when the
valid input type is a list, or considering ‘0’ for non-negative integer inputs.
Target NFRs ARCHCODE considers NFRs that are both pivotal in real-world applications and feasible for assessment either through code execution or using existing metrics.
• Time Performance: Pertains to time-centric aspects like algorithmic time complexity or stipulated timeout conditions.
• Robustness: Ensures that code is resilient to invalid inputs (McConnell, 2004). For instance, a function designed for integer addition must prevent unforeseen or undesirable outcomes from the ‘+’ operation, like mistakenly returning the concatenated string when given two strings.
• Maintainability: Considers factors that contribute to the ease of maintenance, such as reducing code complexity via code modularization (Magel et al., 1982), measured by cyclomatic complexity (McCabe, 1976).
• Reliability: Ensures that the code can handle errors gracefully, without causing system failures, thereby increasing the mean time between failures (McConnell, 2004).

heading: 3.2 Requirements-aware Generation
text: Upon obtaining software requirements r̂, ARCHCODE conditions r̂ with the given description p to generate code samples and test cases.3 Specifically, ARCHCODE generates code ĉ and test cases t̂ in a parallel manner:
ĉ “fprp11, r11, c11s; ...; rp, r̂sq, t̂ “hprp11, r11, t11s; ...; rp, r̂sq, (2)
where c1i and t1i “ rt1i , t2i , ...s are the code and the list of test cases of i-th example, and each tji in t 1 i is a test case corresponding to rji in r̂. 4 We choose this parallel generation due to the potential pitfalls when these processes condition each other. We further discuss such pitfalls in Section 5.2.
3For the reliability category, we uniquely assess code reliability by checking for runtime errors with various test cases, instead of generating specific ones.
4For an intuitive explanation, we describe how a single test case is tailored to a requirement. However, in real implementation, ARCHCODE utilizes multiple generated test cases to confirm each requirement, as explained in Appendix E.

heading: 3.3 Pursuing Requirements Satisfaction
text: To ensure the conformance of the generated code snippet ĉ with the specified requirements r̂, ARCHCODE executes ĉ against the generated test cases t̂ tailored to one of the requirements in r̂: s “ EXEC `ĉ, t̂˘ , (3) where s P t0, 1u is a binary result from a code execution function EXEC, and t̂ is one of the generated test cases in t̂, matching r̂ in r̂. To return the satisfactory code towards r̂, ARCHCODE conducts code filtering. To rank each code in relation to r̂, our framework calculates a weighted sum of the scores s from each t̂, with the option to assign higher weights to preferred requirements. Adjusting those weights to tailor the scoring process is discussed in more detail in Section 5.3.

heading: 4 Experiments
text: We evaluate ARCHCODE’s effectiveness using three benchmarks, categorized into two types: 1) A novel benchmark for assessing both FR and NFR satisfaction; 2) Two public benchmarks aimed at FR evaluation, facilitating comparison of ARCHCODE with existing baselines. For the former, we introduce HumanEval-NFR for comprehensive NFR assessment, overcoming the conventional focus on FR alone. For the latter, we explore two code modalities: 1) function-level and 2) competitionlevel code generation.

heading: 4.1 Experimental Setup
text: We evaluate the effectiveness of ARCHCODE on code generation with LLMs. Throughout the experiments, we used GPT-3.5-Turbo-16k (OpenAI, 2022) as the backbone LLMs for generating code, software requirements, test cases, etc. More details can be found in Appendix A.
Evaluation Metrics We mainly consider the widely used Pass@k :“ EProblemsr1´ p
n´c k qpnkq s (Chen
et al., 2021) metric for evaluation, which is the unbiased estimator of the probability that the code generation system would have passed a problem if it were given k chances to sample c correct code snippets among n samples. Adhering to Chen et al. (2023), when applying code filtering, we denote the existence of passed code among the k filtered samples.5
5While this metric is sometimes referred to as n@k—the pass ratio of filtered n samples from k—we avoid this notation
Baselines Throughout the benchmarks, we consider three baselines: GPT-3.5-Turbo and its CoT prompting applied version of Self-planning (Jiang et al., 2023), and CODET (Chen et al., 2023). For both HumanEval and CodeContests, we further use three code filtering methods—CODERANKER (Inala et al., 2022), MBR-EXEC (Shi et al., 2022b), and MPSC (Huang et al., 2023)—along with WizardCoder 34B (Luo et al., 2023) and GPT-4 (OpenAI, 2023). For CodeContests, we additionally compare with two CoT methods: BRAINSTORM (Li et al., 2023) and ALGO (Zhang et al., 2023a).

heading: 4.2 HumanEval-NFR: Embracing NFR Evaluation
text: We introduce HumanEval-NFR benchmark, which is specifically designed to assess NFR satisfaction. It is an extension of HumanEval that additionally covers four NFR categories, chosen for their suit-
as the interpretation of k differs from that in Pass@k.
ability for evaluation, through code execution using annotated test cases or automated assessment using existing metrics. Details on the annotation process and metrics we used are provided in Appendix B.
Table 2 presents that ARCHCODE outperforms all baseline methods across various NFR categories except for maintainability. Our conjecture is that, as which NFR categories to prioritize is uninformed in this experiment, ARCHCODE’s consideration of all NFRs could potentially impede maintainability due to the influence of other categories. We study the informed case of optimizing specific categories in Section 5.3. Across all approaches, satisfying the robustness category appears to be more difficult compared to other NFR categories, for which we provide further discussion in Appendix G.
Notably, ARCHCODE is desirable for evaluating all NFRs at once (i.e. All), outperforming CODET with 22.16% of Pass@1.

heading: 4.3 HumanEval and CodeContests: Public Benchmarks for FR Evaluation
text: We additionally report results on two popular code generation benchmarks targetting functional correctness. HumanEval (Chen et al., 2021) is a hand-crafted test benchmark with 164 programming problems along with public and hidden test cases. CodeContests (Li et al., 2022) consists of 13k/113/165 instances of train/valid/test data collected from multiple code competition websites. While HumanEval tests the model’s capability to implement rather simpler functions without errors, the competitive programming oriented nature of CodeContests often requires more complex form of reasoning such as algorithmic reasoning. Each of these addresses different aspect of industrial software: the former is related to solving each of the simpler tasks composing larger and complex projects while the latter focuses on the logical and algorithmic perspective of software development.
In Table 3, ARCHCODE consistently outperforms the baseline methods. Specifically, on both benchmarks, ARCHCODE leveraging GPT3.5-Turbo, exceeds GPT-4’s performance by a substantial margin of 4.81%p and 10.45%p in terms of Pass@1. In comparison with WizardCoder 34B—a baseline that partially incorporates NFR considerations during the finetuning phase— ARCHCODE, which covers NFRs more comprehensively, achieves significantly higher performance. In CodeContests, while our custom GPT-3.5-Turbo + CoT prompting baseline is outdone by the stateof-the-art CoT methods BRAINSTORM and ALGO, the application of ARCHCODE outperforms both approaches, setting new state-of-the-art of Pass@1. We also compare ARCHCODE with MPSC, a very recent baseline. Notably, ARCHCODE surpasses MPSC in all Pass@k metrics on HumanEval and Pass@1 on CodeContests, while ARCHCODE is much more cost-efficient. We provide further discussion on computational costs in Section 5.1.

heading: 5 Analysis and Discussion

heading: 5.1 Efficiency and Effectiveness of Requirement-aware Test Case Generation
text: Efficiency In code filtering, a crucial step involves minimizing the number of generated test cases to reduce computational and time costs for code execution. As shown in Figure 4, existing approaches such as MPSC and CODET requires to generate hundreds of test cases for performance.
78
79
80
81
82
83
84
85
86
87
0100200300400500 78
79
80
81
82
83
84
85
86
87
0100200300400500
78
79
0
1
2
83
84
85
86
87
0100200300400500 78
79
0
1
2
83
84
85
86
87
0100200300400500
ARCHCODE (this work)
MPSC
CODET
HumanEval Pass@1 (↑)
# gen. TCs (↓)
CODET focuses on general test cases which are likely to have limited coverage, ARCHCODE distinctly promotes a diverse set of test cases targeting various requirement (sub)types.

heading: 5.2 Conditioning Code Generation on Test Cases
text: In contrast to our approach of generating code and test cases in parallel and then applying subsequent postprocess mechanisms such as filtering, one can also consider conditioning the code generation on test cases,6 taking inspiration from the TestDriven Development (TDD; Beck, 2022) methodology. Table 7 shows results consistent with those reported in Chen et al. (2023), indicating marginal improvement in performance is observed when conditioning code generation on the ground-truth and generated test cases, while incorporating software requirements through ARCHCODE effectively boosts the score, even without code filtering. This suggests the overhead from introducing new sequential dependency in the generation process might not be worth the additional costs incurred.
6One may also consider the opposite direction of generating the test cases conditioned on the generated code, which we do not visit in this paper.

heading: 5.3 Preference over Requirements
text: As mentioned before, ARCHCODE can be informed of any user preferences over the software requirements at code filtering time—after several code candidates have been generated and awaiting to be ranked. Figure 5 presents the Pass@1 scores
for each NFR category in the HumanEval-NFR benchmark, with different code filtering strategies applied. Using targeted test cases for reranking yields higher pass rates for that specific software requirement than using all test cases does.
Another approach to incorporate user preference over the requirements is to consider a subset of requirements when generating code or test cases, or to put emphasis on a subset of requirements by editing the prompt while presenting all requirements to the model. We present detailed analyses for such scenarios in Appendix F.

heading: 5.4 ARCHCODE under Diverse Settings
text: Here we provide empirical results suggesting that ARCHCODE generalizes well to other models, datasets, etc. than those considered in the main experiments.
Open-source LLMs First, we showcase ARCHCODE combined with a relatively smaller model, namely WizardCoder 7B. Table 8 indicates that applying ARCHCODE with the said backbone model leads to a notable 15.67%p improvement in Pass@1 on HumanEval, while incorporating in-context learning directly into WizardCoder 7B itself has negative impacts. Note that this observation is consistent with prior findings such as that in Yuan et al. (2023), that instruction tuning might compromise in-context learning capabilities of LLMs; WizardCoder 7B is an instruction-tuned model based on CODELLAMA 7B.
Meanwhile, in practical settings, diverse LLMs offer complementary benefits in terms of costperformance trade-off, and thus mixing two models has been a conventional aproach to explore costperformance design space (Sun et al., 2023; Wang et al., 2023). ARCHCODEMIX shown in Tables 8 and 9 similarly capitalizes on this space by directing most of the generation calls to affordable LLMs, while selectively delegating the part requiring the most of the reasoning capabilities to stronger ones.
Other Programming Languages We also extend the evaluation of ARCHCODE to the task of Java code generation, using the MultiPL-E (Cassano et al., 2022) benchmark and the backbone model SantaCoder 1B (Allal et al., 2023). To address the rather limited capacity of a smaller model, we further applied sparse fine-tuning (Ansell et al. (2022); SFT) on a public Java train set. We provide more details in Appendix A.1. The results in Table 9 demonstrate the effectiveness of the proposed
method in generating Java code, supporting that our method is generally applicable to programming languages other than Python.

heading: 6 Conclusion
text: We proposed ARCHCODE, a framework incorporating software requirements from textual descriptions for LLM-based code generation. This systematic approach not only identifies these requirements but also harnesses them to guide the code generation process. The verification of code snippets with the generated test cases tailored to each requirement provides a robust validation layer for the alignment with detected requirements. On HumanEval and CodeContests, ARCHCODE with GPT-3.5-Turbo exceeded GPT-4’s performance by 4.81%p and 10.45%p of Pass@1. ARCHCODE requires 50x less generated test cases compared to MPSC and CODET, while outperforming them. In addition, we introduced a new benchmark named HumanEvalNFR for evaluating how well LLMs can pursue nonfunctional requirements in code generation task. Further analysis shows the pertinence of parallel generation of code and test case, and the efficiency and the effectiveness of ARCHCODE’s requirementaware test case generation.

heading: Acknowledgment
text: This work was supported by LG AI Research, and partly supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by ICT R&D program of MSIT/IITP (2022-0-00995, Automated reliable source code generation from natural language descriptions).

heading: Limitations
text: ARCHCODE leverages in-context learning as a tool to integrate both functional and non-functional requirements in the processes of code and test case generation. We did not studied prompt engineering and devising more sophisticated in-context examples which is beyond the scope of this work.
ARCHCODE encompassed three functional and four non-functional requirements, aligning with the established taxonomy within software engineering literature (Glinz, 2007). However, the potential for future work lies in addressing more complex and varied requirements involving larger pieces of code, as well as accommodating changes in software requirements over time.
Lastly, as ARCHCODE relies on generated requirements to guide subsequent code and test case generation process, although qualitative analysis suggests its impact could be limited in practice, additional measures to mitigate cascading errors via human intervention or self-correction by LLMs, etc. (Shinn et al., 2023; Wang et al., 2023; Yao et al., 2023; Chen et al., 2024) can be necessitated.

heading: Ethical and Social Implications
text: ARCHCODE leverages LLMs to automatically generate software requirements, code, and test cases, thereby enhancing productivity and reducing manual labor for developers. However, to maximize these advantages while addressing potential risks, such as the creation of code with safety or security vulnerabilities as discussed in Chen et al. (2021), careful consideration is essential. Strategies to mitigate these risks include establishing default requirements for desired outcomes, delineating the permissible scope of generated code, and ensuring that the code remains within its authorized boundaries.

heading: B HumanEval-NFR Construction

heading: A.1 Open-sourced Backbone Models and Java Language
text: and NFRs of code. HumanEval-NFR comprises the same 164 problems as in the original HumanEval suite. While encompassing all the problem descriptions and ground truth test cases from the original HumanEval benchmark for FR verification, it introduces additional test cases for FR and NFR verification. The statistics of HumanEval-NFR’s ground truth test cases are shown in Table 11.
Writing new ground truth test cases involved a two-step process. First, we generated candidate test cases based on the existing HumanEval problems using ARCHCODE based on GPT-3.5-Turbo. Second, we revised those test cases both in automatic or manual manner to ensure the quality of the test suite, based on the following protocols.

heading: B.1 Quality Control for FR Test Cases
text: For candidate test cases evaluating FRs, we executed the ground truth code from the original HumanEval benchmark against each test case tailored to functional requirements. Those that the ground truth code does not pass were discarded.

heading: B.2 Quality Control for NFR Test Cases
text: For candidate test cases verifying NFRs, three authors manually validated the quality of generated test cases. During validation, the authors adhered to the following principles:
• Misclassified test cases should be rectified, and any duplicates should be eliminated.
• Test cases should compatible to the original ground truth code. If any discrepancy is found
in the code, or if a test case is deemed impractical or overly complex, adjustments should be made to ensure it aligns with the original problem description.
In addition, the authors consider guidelines specific to each NFR category:
Time Performance As Rice’s Theorem (Rice, 1953) states, all non-trivial properties of Turingrecognizable languages are undecidable, which in essence means that there could be no ‘timecomplexity checkers.’ Therefore, HumanEval-NFR follows conventional strategies used in competitive programming contests, such as Codeforces,11 where code is executed with relatively larger inputs to ensure that inefficient implementations cannot complete within the specified timeout. Specifically, we set the timeout as 5 seconds for all problems.
Robustness Test cases for this category verify whether the implementation gracefully handles diverse types of invalid inputs, such as a string passed as an argument where an integer is expected. For technical reasons, we expect the code to return values like None, an empty list, or False—all of which are logically evaluated as False in the Python language—rather than forcing it to raise exceptions or using any other means to indicate it has detected an abnormal input.
Maintainability To validate maintainability, we consider code complexity, which affects the ease of understanding and updating the code (Magel et al., 1982). Specifically, HumanEval-NFR computes the Cyclomatic Complexity (CC; McCabe, 1976) of code, which evaluates code complexity by accounting for the depth of nested indented blocks, then checks whether the observed CC score is lower than the threshold. The threshold is set to 5 if the ground truth code from the original HumanEval benchmark has a CC value below 5; if the CC value exceeds 5, we set the threshold as 10 (Watson et al., 1996).
Reliability Rather than generating dedicated test cases, HumanEval-NFR assesses code reliability by executing all the ground truth test cases for the problem and checks if any runtime errors are raised, without verifying if the outputs are correct. This approach aligns with the category’s focus on minimizing system failures and extending the meantime-to-failure.
11https://codeforces.com
# Code must satisfy not only functional requirements but also the following non-functional requirements. # Non-functional Requirements ## Performance: Pertains to time-centric aspects such as algorithmic time complexity or stipulated timeout conditions. ## Robustness: Ensures that code is resilient to invalid inputs. ## Maintainability: Considers factors that contribute to the ease of maintenance. ## Reliability: Ensures that code can handle errors gracefully without causing system failures over an extended period. Write a code for the problem.
Table 13: Engineered prompt which further specified the details of each NFR, used in Table 12.

heading: C Gains from Prompt Engineering
text: In this study, we did not focus on devising sophisticated prompts, as our main contribution does not rely heavily on using prompt-engineered instructions. Therefore, we can expect even more performance gains when the prompt is further engineered as in Table 13, as we intentionally kept prompt simple in our main experiments.
Table 12 shows that ARCHCODE is scalable to requirement instruction prompts, showing the best performance on HumanEval-NFR (All) when both are applied. Unlike CoT and NFR Instruction that improve Robustness and Reliability only, ARCHCODE contributes to all NFR types. Notably, time performance and maintainability are enhanced
solely by ARCHCODE’s code filtering, highlighting the unique contribution over prompt engineering.

heading: D Correctness of Generated Requirements
text: Format As presented in Figure 6, we organized the structure of software requirements into two parts: problem-agnostic and problem-specific. The former describes general guidelines throughout problems related to reliability, performance, and maintainability. The latter includes more specific instructions depending on the problem description, including all three subtypes of functional requirements, the target time complexity for time performance, the invalid conditions for robustness, and the target Cyclomatic Complexity for maintainability.
Validation To confirm the correctness of the requirements generated by ARCHCODE with GPT3.5-Turbo, we randomly selected three problems, one for each of the following categories: (1) all, (2) some or (3) none of the generated code samples passed the tests. We manually verified validity of each set of requirements for each case, of which the results are summarized in Table 17. Surprisingly, all the generated requirements were correct, regardless of the corresponding generated code’s correctness.

heading: E Analysis of Generated Test Cases by
text: ARCHCODE
Table 14 shows the average number of generated test cases by ARCHCODE for each requirement category. Table 15 reports the accuracy of gener-
ated test cases tailored to functional requirements. Although the accuracy of generated edge cases is relatively low, they still play a key role in code filtering as evidentiated by the performance discrepancy between ARCHCODE and CODET (w/o clustering) presented in Tables 4. It is noteworthy that CODET generates ‘general’ test cases, and the results for ARCHCODE and CODET are comparable, given that both methods are based on the same GPT-3.5-Turbo architecture. We conjecture that the relatively low accuracy of generated edge cases does not prevent them from being substantially useful in code filtering, for that wrongful test cases tend to accept or to reject both the correct and incorrect code, rather than selectively passing incorrect ones. In other words, the overall ranking of the generated code samples is hardly affected by the wrongly generated test cases.
For the validation of non-functional requirements, we can implicitly confirm through Figure 5 as using targeted test cases (filled green) yielded
Requirements Subtype # gen. TCs Functional General 3.1 Edge 3.1 Non-Functional Time Perf. 1.9 Robustness 2.0 Maintainability 1.0

heading: F NFR Preference Control
text: clude all NFRs in prompts just as before, while explicitly expressing the preference at the end (e.g. “Consider the time performance requirement to be the most important.”).
• Preference Control by Plug-and-Play: We only present the preferred NFRs in prompts, without an explicit description of the preference. All included NFRs are considered equally important, with no prioritization among them.
Table 16 shows that the plug-and-play approach inflicts a larger impact on Pass@k on HumanEvalNFR (All), compared to the instruction-based method. Notably, the plug-and-play approach considering all but time performance showed the best Pass@k scores, which we attribute to the trade-off between time performance and the rest of the NFRs: focusing on the other requirements is relatively free of negative interference that would hurt the performance. In the categories of time performance and robustness, both instruction and plug-and-play preference settings showed improvements when each of them was targeted. For the maintainability category, the best results were observed when only time performance was preferred in the plug-andplay approach. This is likely due to the omission of code lines for handling exceptions related to invalid inputs, as the robustness category was not considered. In the case of reliability, which is assessed by the error-free execution of code across all test cases (without dedicated test cases), performance improvement was observed irrespective of preference in the instruction-based approach. As
demonstrated in Table 12, this suggests that prompt engineering can reduce error ratios; we leave exploration towards this direction to future work.
G Varying Difficulty Levels of NFRs in HumanEval-NFR
Orthogonal to ARCHCODE’s contribution towards satisfying every requirement category, we observe a general trend of relatively low performance in the robustness category compared to others in HumanEval-NFR, as shown in Table 2 and Figure 5. One conjecture is that the difficulty lies among the NFRs as inferred from the original HumanEval benchmark. As the ground truth code snippets are generally not complex, handling large input size (time performance), managing small Cyclomatic Complexity (maintainability), and avoiding runtime error while running other test cases (reliability) might be easier than correctly handling every possible invalid input (robustness).
Index Generated Requirements Generated Code (manually validated) (validated by Ground Truth Test Cases)
Sub-requirement Generated Requirement Human
Explanation Validation
HumanEval-NFR/51 Description: remove_vowels is a function that takes string and returns string without vowels. General The function takes a string ‘text’ as input. Correct The function takes string with or without
vowels as input.
General It returns a string. Correct The function should return a string without vowels.
General The function should remove all vowels from the input ‘text’ and return the modified string.
Correct The function should return a string without vowels from the input string.
Edge Handle an empty ‘text’ string, returning an empty string.
Correct Input can be empty string. In this case, the function should return empty string.
Edge Handle the case where ‘text’ contains only vowels, returning an empty string.
Correct Input can contain only vowels. In this case, all of vowels should be removed and the function should return empty string.
Edge Handle the case where ‘text’ contains no vowels, returning the original string.
Correct Input can contain no vowels. In this case, there are nothing to be removed and the function should return original string.
Edge Handle both lowercase and uppercase vowels.
Correct Input can be string in lowercase and uppercase. Both of them should be handled.
Performance Time complexity: O(n) where n is the length of the ‘text’ string.
Correct To traverse the string once and check whether each character is a vowel or not, and then add non-vowel characters to the result string, it requires Opnq time, where n is the length of the string.
Performance Space complexity: Opnq where n is the length of the ‘text’ string.
Correct Creating a new string to store the non-vowel characters requires additional space. If the length of the input string is n, the length of the resulting string can be at most n, leading to an Opnq space complexity.
Performance Ensure efficiency even for extremely large inputs, providing results within 5 seconds.
Correct The inputs should be handled in predetermined testing time.
Robustness If a non-string input is provided, print an error message to ‘stderr’ and return None.
Correct Invalid inputs should be handled and return None.
Maintainability Target Cyclomatic Complexity: ≤ 5. Correct 5 indicates low risk, where the code mainly consists of simple blocks.
Table 18: Detailed explanation for the human validation of generated requirements by ARCHCODE (Index: 0 in Table 17) on HumanEval-NFR/51.
Sub-requirement Generated Requirement Human
Explanation Validation
HumanEval-NFR/11 Description: Input are two strings a and b consisting only of 1s and 0s. Perform binary XOR on these inputs and return result also as a string. General The function takes two strings ‘a’ and ‘b’ as
input. Correct The function should take two strings consisting only of 1s and 0s.
General It returns a string. Correct The function should return string having result of a binary XOR operation on these inputs.
General The function should perform a binary XOR operation on the input strings ‘a’ and ‘b’ and return the result as a string.
Correct The function takes two binary strings and return a result of a binary XOR operation.
Edge Handle empty strings as input, returning an empty string.
Correct The function should handle empty strings and return empty string.
Edge Handle strings with different lengths, returning None.
Correct When the two inputs have different length, a binary XOR operation is impossible.
Edge Handle strings with characters other than ‘0’ and ‘1’, returning None.
Correct When the two inputs are not valid binary string, a binary XOR operation is impossible.
Performance Time complexity: Opnq where n is the length of the longer input string.
Correct The time complexity of performing binary XOR on two strings is proportional to the length of the strings.
Performance Space complexity: Opnq where n is the length of the longer input string.
Correct If a new string is created to store the result, its length would be equal to the length of the input strings.
Performance Ensure efficiency even for extremely large inputs, providing results within 5 seconds.
Correct The inputs should be handled in predetermined testing time.
Robustness If non-string inputs are provided, print an error message to ‘stderr’ and return None.
Correct Invalid inputs should be handled and return None.
Robustness If the input strings contain characters other than ‘0’ and ‘1’, print an error message to ‘stderr’ and return None.
Correct Invalid inputs should be handled and return None.
Maintainability Target Cyclomatic Complexity: ≤ 10. Correct 10 indicates low risk, where the code consists of well-structured and stable blocks.
Table 19: Detailed explanation for the human validation of generated requirements by ARCHCODE (Index: 0 in Table 17) on HumanEval-NFR/11.
Sub-requirement Generated Requirement Human
Explanation Validation
HumanEval-NFR/0 Description: Check if in given list of numbers, are any two numbers closer to each other than given threshold. General The function takes a list of floating-point
numbers ‘numbers’ and a threshold value ‘threshold’ as input.
Correct The function takes two inputs. The first is a list of floating-point numbers and the second is a threshold value.
General It returns a boolean value. Correct The function should return True if there are any two numbers in the ‘numbers’ list that are closer to each other than the given ‘threshold’. Otherwise, it should return False.
General The function should return ‘True’ if there are any two numbers in the ‘numbers’ list that are closer to each other than the given ‘threshold’. Otherwise, it should return ‘False’.
Correct The function checks if there are any two numbers closer to each other than given threshold in given list of numbers.
Edge Handle an empty ‘numbers’ list, returning ‘False’.
Correct The function should return False if there are not any numbers in the ‘numbers’ list.
Edge Handle the case where no two numbers in ‘numbers’ are closer to each other than the ‘threshold’, returning ‘False’.
Correct The function should return True if there are any two numbers in the ‘numbers’ list that are closer to each other than the given ‘threshold’. Otherwise, it should return False.
Edge Handle the case where all numbers in ‘numbers’ are closer to each other than the ‘threshold’, returning ‘True’.
Correct The function should return True if there are any two numbers in the ‘numbers’ list that are closer to each other than the given ‘threshold’. Otherwise, it should return False.
Performance Time complexity: Opn2q where n is the length of the ‘numbers’ list.
Correct All elements of the given list need to be iterated through, and the difference between each pair of numbers needs to be calculated. This process is proportional to the size of the input list. Therefore, if we denote the length of the input list as n, the time complexity is Opn2q.
Performance Space complexity: Op1q. Correct If the problem is solved without using additional memory, the space complexity is Op1q.
Performance Ensure efficiency even for extremely large inputs, providing results within 5 seconds.
Correct The inputs should be handled in predetermined testing time.
Robustness If non-list ‘numbers’ input or a non-float ‘threshold’ input is provided, print an error message to ‘stderr’ and return None.
Correct Invalid inputs should be handled and return None.
Robustness If non-float elements in the ‘numbers’ list or a negative ‘threshold’ is provided, print an error message to ‘stderr’ and return None.
Correct Invalid inputs should be handled and return None.
Maintainability Target Cyclomatic Complexity: ≤ 10. Correct 10 indicates low risk, where the code consists of well-structured and stable blocks.
Table 20: Detailed explanation for the human validation of generated requirements by ARCHCODE (Index: 0 in Table 17) on HumanEval-NFR/0.
Index Generated Test Cases Sub-requirement # Passed Human
Type Cands. Validation
HumanEval-NFR/51
0 assert remove_vowels(‘’) == ‘’, ‘Failed to handle an empty input string.’
General 10/10 Correct
1 assert remove_vowels("abcdef\nghijklm") == ‘bcdf\nghjklm’, ‘Failed to remove all vowels from the input string.’
General 10/10 Correct
2 assert remove_vowels(‘abcdef’) == ‘bcdf’, ‘Failed to remove vowels from the input string.’
General 10/10 Correct
3 assert remove_vowels(‘aaaaa’) == ‘’, ‘Failed to handle case where the input string contains only vowels.’
Edge 10/10 Correct
4 assert remove_vowels(‘aaBAA’) == ‘B’, ‘Failed to remove only lowercase vowels from the input string.’
Edge 10/10 Correct
5 assert remove_vowels(‘zbcd’) == ‘zbcd’, ‘Failed to handle case where the input string contains no vowels.’
Edge 10/10 Correct
6 assert remove_vowels(‘a’ * 10**6) == ‘’, ‘Failed to handle large input size.’
Performance 10/10 Correct
7 assert remove_vowels(‘z’ * 10**6) == ‘z’ * 10**6, ‘Failed to handle large input size.’
Performance 10/10 Correct
8 assert remove_vowels(123) == None, ‘Failed to handle case where the input is not a string.’
Robustness 10/10 Correct
9 assert result.total_complexity <= 5, ‘Failed to have a Cyclomatic Complexity less than or equal to 5 by Radon.’ Maintainability 10/10 Correct
HumanEval-NFR/83
0 assert starts_one_ends(1) == 9, ‘Failed to count the numbers of 1-digit positive integers that start or end with 1.’
General 0/10 Incorrect
1 assert starts_one_ends(2) == 19, ‘Failed to count the numbers of 2- digit positive integers that start or end with 1.’
General 0/10 Correct
2 assert starts_one_ends(3) == 271, ‘Failed to count the numbers of 3-digit positive integers that start or end with 1.’
General 0/10 Incorrect
3 assert starts_one_ends(0) == 0, ‘Failed to handle the case where n is 0.’
Edge 0/10 Incorrect
4 assert starts_one_ends(-5) == 0, ‘Failed to handle the case where n is negative.’
Edge 0/10 Incorrect
5 assert starts_one_ends(10**6) == 900000, ‘Failed to handle large input size.’
Performance 0/10 Incorrect
6 assert starts_one_ends(‘invalid’) == None, ‘Failed to handle case where the input n is not an integer.’
Robustness 10/10 Correct
7 assert result.total_complexity <= 5, ‘Failed to have a Cyclomatic Complexity less than or equal to 5 by Radon.’ Maintainability 2/10 Correct
HumanEval-NFR/100
0 assert make_a_pile(3) == [3, 5, 7], ‘Failed to create the pile correctly.’ General 5/10 Correct
1 assert make_a_pile(5) == [5, 7, 9, 11, 13], ‘Failed to create the pile correctly.’
General 5/10 Correct
2 assert make_a_pile(0) == [], ‘Failed to handle the case where n is 0.’ Edge 5/10 Correct
3 assert make_a_pile(-5) == None, ‘Failed to handle the case where n is negative.’
Edge 5/10 Correct
4 assert make_a_pile(10**6) == list(range(10**6, 10**6 + 2 * 10**6, 2)), ‘Failed to handle large input size.’
Performance 0/10 Correct
5 assert make_a_pile(‘invalid’) == None, ‘Failed to handle the case where the input n is not an integer.’
Robustness 5/10 Correct
6 assert make_a_pile(0.5) == None, ‘Failed to handle the case where the input n is not a positive integer.’
Robustness 5/10 Correct
7 assert result.total_complexity <= 10, ‘Failed to have a Cyclomatic Complexity less than or equal to 10 by Radon.’ Maintainability 10/10 Correct
Table 21: Validation of generated test cases by ARCHCODE on HumanEval-NFR.

heading: Time Perf. All - Time Perf.

heading: Time Perf. All - Time Perf.
text: H In-Context Learning Examples

heading: Problem Description Example
text: from typing import List
def longest_subarray_with_sum_limit(nums: List[int], target: int) -> List[int]: """Find the longest subarray of the given array ‘nums’ such that the sum of its elements is less than or equal to the given ‘target’. >>> longest_subarray_with_sum_limit([3, 1, 5, 2, 1, 9, 7, 4], 10) [3, 1, 5] >>> longest_subarray_with_sum_limit([1, 2, 3, 4, 5], 3) [1, 2]
""" Requirement Generation Prompt Write requirements for the problem. Requirement Generation Example # Problem Agnostic Requirements - Ensure error-free execution, avoiding type errors, index errors, arithmetic operation errors (e.g., division by zero, overflow),
and so on. - Demonstrate efficiency in the algorithmic approach to problem-solving. - Code should be clear, well-commented, and easy to understand and maintain.
# Functional Requirements ## Input-output Conditions - The function takes a list of integers ‘nums’ and a target integer ‘target’ as input. - It returns a list of integers.
## Expected Behavior - The function should return the longest subarray of ‘nums’ whose sum is less than or equal to ‘target’.
## Edge Cases - Handle an empty ‘nums’ list, returning an empty list. - Handle the case where no subarray in ‘nums’ satisfies the condition, returning an empty list. - Handle multiple subarrays with the same maximum length, returning any of them.
# Non-functional Requirements ## Performance - Time complexity: O(n) where n is the length of the ‘nums’ list. - Space complexity: O(1). - Ensure efficiency even for extremely large inputs, providing results within 5 seconds.
## Specific Quality Requirements ### Robustness - If non-list ‘nums’ input or a non-integer ‘target’ input is provided, print an error message to ‘stderr’ and return None. - If non-integer elements in the ‘nums’ list or a negative ‘target’ is provided, print an error message to ‘stderr’ and return
None.
### Reliability - Avoid index errors while accessing ‘nums’ elements.
### Maintainability - Target Cyclomatic Complexity≤ 10.
Table 24: The in-context example for requirement generation by ARCHCODE. The problem description example is from Shinn et al. (2023).

heading: Problem Description Example
text: from typing import List ... Requirement Generation Prompt Write requirements for the problem. Requirement Generation Example # Problem Agnostic Requirements ... Code Generation Prompt Write the code for the problem. Code Generation Example from typing import List, Optional import sys
def longest_subarray_with_sum_limit(nums: Optional[List[int]], target: Optional[int]) -> List[int]: # Validate inputs if not isinstance(nums, list) or not all(isinstance(num, int) for num in nums):

heading: Problem Description Example
text: from typing import List ... Requirement Generation Prompt Write requirements for the problem. Requirement Generation Example # Problem Agnostic Requirements ... Test Case Generation Prompt Write test cases for the problem. Test Case Generation Example # Test Cases Regarding Functional Requirements ## General Cases ### The longest subarray with sum less than or equal to 10 is [1, 5, 2, 1] assert longest_subarray_with_sum_limit([3, 1, 5, 2, 1, 9, 7, 4], 10) == [1, 5, 2, 1], ‘Failed to find the longest subarray.’
### None of the subarrays have a sum less than or equal to 3 ### The function should return an empty list assert longest_subarray_with_sum_limit([2, 3, 4, 5, 6], 1) == [], ‘Failed to handle case where no subarray satisfies the condition.’
## Edge Cases ### The input list is empty, so the function should return an empty list assert longest_subarray_with_sum_limit([], 5) == [], ‘Failed to handle an empty input list.’
### The longest subarray with sum less than or equal to 10 is [5, 5] assert longest_subarray_with_sum_limit([5, 5, 5, 5], 10) == [5, 5], ‘Failed to find the longest subarray when all elements have the same value.’
### The entire nums array is a valid subarray with sum less than or equal to 15 assert longest_subarray_with_sum_limit([1, 2, 3, 4, 5], 15) == [1, 2, 3, 4, 5], ‘Failed to handle case where the entire array is a valid subarray.’
# Test Cases Regarding Non-functional Requirements ## Performance Requirements ### The nums list contains 106 elements with increasing values from 1 to 106 ### The longest subarray with sum less than or equal to 105 is [1, 2, 3, ..., 105] assert longest_subarray_with_sum_limit(list(range(1, 10**6 + 1)), 10**5) == list(range(1, 446 + 1)), ‘Failed to handle large input size.’
### The nums list contains 106 elements with increasing values from 1 to 106 ### The longest subarray with sum less than or equal to 2 is [1] assert longest_subarray_with_sum_limit(list(range(1, 10**6 + 1)), 2) == [1], ‘Failed to handle case where the subarray length is 1.’
### The nums list contains 106 elements with increasing values from 1 to 106 ### The longest subarray with sum less than or equal to 5 is [1, 2] assert longest_subarray_with_sum_limit(list(range(1, 10**6 + 1)), 5) == [1, 2], ‘Failed to handle case where the subarray length is 2.’
### The nums list contains 106 elements, all of which are 106 ### The sum of all the elements in nums is 1012, which is larger than the target ### Therefore, there is no subarray that satisfies the condition and the function should return an empty list assert longest_subarray_with_sum_limit([10**8] * 10**6, 10**7) == [], ‘Failed to handle case where no subarray satisfies the condition.’
### The nums list contains 106 elements with increasing values from 1 to 106 ### The target is the sum of all the elements in nums, so the entire array is a valid subarray assert longest_subarray_with_sum_limit(list(range(1, 10**5 + 1)), 10**10) == list(range(1, 10**5 + 1)), ‘Failed to handle case where the entire array is a valid subarray.’
## Specific Quality Requirements ### Robustness #### The sum of the elements in nums is larger than the target, #### so the function should return None assert longest_subarray_with_sum_limit([10**20, 10**20], 10**19) == None, ‘Failed to handle case where the sum of the elements in nums is larger than the target.’
#### The nums input is not a list of integers, so the function should return None assert longest_subarray_with_sum_limit(’invalid’, 10) == None, ‘Failed to handle case where the input nums is not a list.’
#### The target input is not an integer, so the function should return None assert longest_subarray_with_sum_limit([1, 2, 3], ‘invalid’) == None, ‘Failed to handle case where the input target is not an integer.’
#### The nums list contains elements that are not integers, so the function should return None assert longest_subarray_with_sum_limit([1, 2, ‘invalid’, 4], 5) == None, ‘Failed to handle case where the input nums contains non-integer elements.’
#### The target is a negative number, so the function should return None assert longest_subarray_with_sum_limit([1, 2, 3, 4, 5], -10) == None, ‘Failed to handle case where the input target is negative.’
### Reliability #### Satisfied if no errors occur across all test cases
### Maintainability #### Calculate Cyclomatic Complexity using Radon #### Check if the Cyclomatic Complexity is less than or equal to 10 from radon.visitors import ComplexityVisitor result = ComplexityVisitor.from_code("""${Generated Code}""") assert result.total_complexity <= 10, ‘Failed to have a Cyclomatic Complexity less than or equal to 10 by Radon.’
Table 26: The in-context example for test case generation by ARCHCODE. The skipped contents—Problem Description Example and Requirement Generation Example—are available in Table 24. ${Generated Code} denotes the string text of the code that is to be checked.
I In-Context Learning Prompt Templates
{Problem Description Example} Write requirements for the problem. {Requirement Generation Example}
{Test Problem Description} Write requirements for the problem.
Table 27: The in-context learning prompt template for requirement generation by ARCHCODE.
{Problem Description Example} Write requirements for the problem. {Requirement Generation Example} Write the code for the problem. {Code Generation Example}
{Test Problem Description} Write requirements for the problem. {Generated Requirements by ARCHCODE} Write the code for the problem.
Table 28: The in-context learning prompt template for code generation by ARCHCODE.
{Problem Description Example} Write requirements for the problem. {Requirement Generation Example} Write test cases for the problem. {Test Case Generation Example}
{Test Problem Description} Write requirements for the problem. {Generated Requirements by ARCHCODE} Write test cases for the problem.
Table 29: The in-context learning prompt template for test case generation by ARCHCODE.
{Problem Description Example} Write the code for the problem. {Code Generation Example}
{Test Problem Description} Write the code for the problem.
Table 30: The in-context learning prompt template for code generation by GPT-3.5-Turbo baseline.

heading: J Case Study

heading: Generated Requirements
text: # Problem Agnostic Requirements - Ensure error-free execution, avoiding type errors, index errors, arithmetic operation errors (e.g., division by zero, overflow),
and so on. - Demonstrate efficiency in the algorithmic approach to problem-solving. - Code should be clear, well-commented, and easy to understand and maintain.
# Functional Requirements ## Input-output Conditions - The function takes two integers ‘x‘ and ‘base‘ as input. - It returns a string.
## Expected Behavior - The function should convert the number ‘x‘ to the given ‘base‘ and return the string representation of the converted number.
## Edge Cases - Handle the case where ‘x‘ is 0, returning ’0’. - Handle the case where ‘base‘ is 1, returning a string of ’1’ repeated ‘x‘ times. - Handle the case where ‘base‘ is less than 2 or greater than 9, returning None. - Handle the case where ‘x‘ is negative, returning None.
# Non-functional Requirements ## Performance - Time complexity: O(log(x)) where x is the value of ‘x‘. - Space complexity: O(log(x)) to store the result string. - Ensure efficiency even for extremely large inputs, providing results within 5 seconds.
## Specific Quality Requirements ### Robustness - If a non-integer ‘x‘ or ‘base‘ input is provided, print an error message to ‘stderr‘ and return None. - If a negative ‘base‘ is provided, print an error message to ‘stderr‘ and return None.
### Maintainability - Target Cyclomatic Complexity:≤ 10.
Table 32: A real-life example of generated requirements for HumanEval-NFR/44 by ARCHCODE.

heading: Generated Code
text: def change_base(x: int, base: int) -> str:

heading: Generated Test Cases
text: # Test Cases Regarding Functional Requirements ## General Cases ### The number 8 in base 3 is represented as ’22’ assert change_base(8, 3) == ’22’, ’Failed to convert number to the desired base.’
### The number 8 in base 2 is represented as ’1000’ assert change_base(8, 2) == ’1000’, ’Failed to convert number to the desired base.’
### The number 7 in base 2 is represented as ’111’ assert change_base(7, 2) == ’111’, ’Failed to convert number to the desired base.’
## Edge Cases ### The input number is 0, so the function should return ’0’ in any base assert change_base(0, 3) == ’0’, ’Failed to handle case where the input number is 0.’
### The input base is 1, which is not a valid base, so the function should return None assert change_base(8, 1) is None, ’Failed to handle case where the base is not a valid base.’
### The input number is negative, so the function should return None assert change_base(-8, 2) is None, ’Failed to handle case where the input number is negative.’
### The input base is greater than 9, which is not a valid base, so the function should return None assert change_base(8, 10) is None, ’Failed to handle case where the base is not a valid base.’
# Test Cases Regarding Non-functional Requirements ## Performance Requirements ### The input number is 106, and the base is 2 ### The function should return a string representation of the number in base 2 within a reasonable time frame assert change_base(10**6, 2) == ’11110100001001000000’, ’Failed to handle large input size.’
### The input number is 106, and the base is 9 ### The function should return a string representation of the number in base 9 within a reasonable time frame assert change_base(10**6, 9) == ’1783661’, ’Failed to handle large input size.’
### The input number is 1018, and the base is 2 ### The function should return a string representation of the number in base 2 within a reasonable time frame assert change_base(10**18, 2) == ’110111100000101101101011001110100111011001000000000000000000’, ’Failed to
handle large input size.’
## Specific Quality Requirements ### Robustness #### The input number is not an integer, so the function should return None assert change_base(’invalid’, 2) is None, ’Failed to handle case where the input number is not an integer.’
#### The input base is not an integer, so the function should return None assert change_base(8, ’invalid’) is None, ’Failed to handle case where the input base is not an integer.’
### Reliability #### Satisfied if no errors occur across all test cases
### Maintainability #### Calculate Cyclomatic Complexity using Radon #### Check if the Cyclomatic Complexity is less than or equal to 10 import inspect from radon.visitors import ComplexityVisitor result = ComplexityVisitor.from_code("""${Generated Code}""") assert result.total_complexity <= 10, ‘Failed to have a Cyclomatic Complexity less than or equal to 10 by Radon.’
Table 34: A real-life example of generated test cases for HumanEval-NFR/44 by ARCHCODE. ${Generated Code} denotes the string text of the code that is to be checked.

heading: Generated Requirements
text: # Problem Agnostic Requirements - Ensure execution is error-free, mitigating type errors, index errors, and arithmetic operation errors (e.g., division by zero, overflow) among others. - Showcase efficiency in the algorithmic approach to problem-solving. - Ensure code is clear, well-commented, and both easy to understand and maintain. # Functional Requirements ## Input-output Conditions ### Inputs - Initial input values: n and m - Must be positioned in the first line of input - Adhere to the format: "tnutmu" - Integer n range: (0 ď n ă 2000) - Integer m range: (0 ď m ă 2000) - Subsequent input values: grid denoting the positions of telephone poles - Each input line format: "tai,1utai,2u...tai,m`1u" - Integer ai,j range: (0 ď ai,j ď 1) - Each line represents a row in the grid - There is at least one telephone pole in the given grid. ### Outputs - Output must be a single integer denoting the value of řn x“0t řm y“0tSpx, yquu ## Expected Behavior - The city is represented as a plane. - The plane is represented by a grid of size pn ` 1q ˚ pm ` 1q. - Each point px, yq on the plane has a square of the Euclidean distance between the nearest pole and px, yq denoted as Spx, yq. - Spx, yq is defined as px2 ´ x1q2 ` py2 ´ y1q2, where px1, y1q is the nearest pole and px2, y2q is px, yq. - The code must calculate the sum of all Spx, yq for each 0 ď x ď n and 0 ď y ď m. - The code must output the value of řn x“0t řm y“0tSpx, yquu as a single integer. ## Edge Cases Ensure the code functions accurately, even if: - Inputs reach the condition boundaries - n or m is 0 - n or m is the maximum value - There is only one telephone pole in the grid # Non-functional Requirements ## Performance - Test time limit: 2 seconds - Test memory limit: 256 megabytes ## Specific Quality Requirements ### Robustness Ensure the code operates correctly even if: - Input contains redundant spaces or newlines Enforce the code to print an appropriate error message if: - Input format is incorrect - Input data type is invalid - Input is out of the specified range - The grid does not contain at least one telephone pole. ### Reliability - Mitigate type errors during input and output stream management. - Prevent index errors while managing grid values. ### Maintainability - Aim for a Cyclomatic Complexity of: ď 10.
Table 35: A real-life example of generated requirements for CodeContests/2 (1575_M. Managing Telephone Poles) by ARCHCODE.
Generated Code import sys import math
def read_line(): input_str = "" while not input_str:
input_str = input().strip() return input_str
line = read_line() n, m = map(int, line.split()) if not (0 <= n < 2000 and 0 <= m < 2000):
sys.stderr.write("error at the boundary of n or m") exit()
grid = [[0] * (m + 1) for _ in range(n + 1)] for i in range(n + 1):
line = read_line() if len(line) != m + 1:
sys.stderr.write("error at the boundary of the grid") exit()
for j in range(m + 1): if not (0 <= int(line[j]) <= 1):
sys.stderr.write("error at the boundary of the grid") exit()
grid[i][j] = int(line[j])
result = 0 for i in range(n + 1):
for j in range(m + 1): if grid[i][j] == 1:
continue minDistance = math.inf for x in range(n + 1):
for y in range(m + 1): if grid[x][y] == 1:
distance = (x - i) ** 2 + (y - j) ** 2 minDistance = min(minDistance, distance)
result += minDistance
print(result)
Table 36: A real-life example of generated code for CodeContests/2 (1575_M. Managing Telephone Poles) by ARCHCODE.

heading: Generated Test Cases
text: # Test Cases Regarding Functional Requirements ## General Cases ERROR_MESSAGE = "general case error" INPUT = "2 2\n101\n000\n000" OUTPUT = "18"

heading: ERROR_MESSAGE = "general case error"
text: INPUT = "5 4\n10010\n00000\n01000\n00001\n00100\n00010" OUTPUT = "36" ## Edge Cases ERROR_MESSAGE = "error at the lower boundary of n" INPUT = "0 2\n00\n00\n00" OUTPUT = "0"

heading: ERROR_MESSAGE = "error at the upper boundary of n"
text: INPUT = "1999 2\n" + "0" * 2000 + "\n" + "0" * 2000 OUTPUT = "0"

heading: ERROR_MESSAGE = "error at the lower boundary of m"
text: INPUT = "2 0\n0\n0\n0" OUTPUT = "0"

heading: ERROR_MESSAGE = "error at the upper boundary of m"
text: INPUT = "2 1999\n" + "0\n" * 2000 OUTPUT = "0"
ERROR_MESSAGE = "error when a_{i, j} is not in the correct format" INPUT = "2 2\n101\n000\n00" STD_ERROR = "a_{i, j} is not in the correct format"
ERROR_MESSAGE = "error when a_{i, j} is not in the correct range" INPUT = "2 2\n101\n000\n002" STD_ERROR = "a_{i, j} is not in the correct range"
ERROR_MESSAGE = "error when there is no telephone pole in the given grid" INPUT = "2 2\n000\n000\n000" STD_ERROR = "there is no telephone pole in the given grid"
# Test Cases Regarding Non-functional Requirements ## Performance Requirements ERROR_MESSAGE = "execution failed while running exceptionally large input" INPUT = "1999 1999\n" + "1" * 2000 + "\n" * 1999 + "1" * 2000 OUTPUT = "0" ## Specific Quality Requirements ### Robustness ERROR_MESSAGE = "error when redundant spaces or newlines are added to the input" INPUT = "2 2\n101\n000\n\n\n" OUTPUT = "18"
ERROR_MESSAGE = "error when the input is not in the correct format" INPUT = "2 2\n101\n000\n00" STD_ERROR = "the input is not in the correct format"
ERROR_MESSAGE = "error when the data type of the input is not correct" INPUT = "2 2\n101\n000\n00a" STD_ERROR = "the data type of the input is not correct"
ERROR_MESSAGE = "error when the input is not in the correct range" INPUT = "2 2\n101\n000\n003" STD_ERROR = "the input is not in the correct range"
ERROR_MESSAGE = "error when there is no telephone pole in the given grid" INPUT = "2 2\n000\n000\n000" STD_ERROR = "there is no telephone pole in the given grid" ## Reliability Satisfied if no errors occur across all test cases ## Maintainability ERROR_MESSAGE = "error when cyclomatic complexity is more than the limit" COMPLEXITY_LIMIT = 10
Table 37: A real-life example of generated test cases for CodeContests/2/1575_M. Managing Telephone Poles by ARCHCODE. ${Generated Code} denotes the string text of the code that is to be checked.

heading: Future Work
text: As
demonstrated in Table 12, this suggests that prompt engineering can reduce error ratios; we leave exploration towards this direction to future work. G Varying Difficulty Levels of NFRs in HumanEval-NFR
Orthogonal to ARCHCODE’s contribution towards satisfying every requirement category, we observe a general trend of relatively low performance in the robustness category compared to others in HumanEval-NFR, as shown in Table 2 and Figure 5. One conjecture is that the difficulty lies among the NFRs as inferred from the original HumanEval benchmark. As the ground truth code snippets are generally not complex, handling large input size (time performance), managing small Cyclomatic Complexity (maintainability), and avoiding runtime error while running other test cases (reliability) might be easier than correctly handling every possible invalid input (robustness). Index Generated Requirements Generated Code (manually validated) (validated by Ground Truth Test Cases)
Sub-requirement Generated Requirement Human
Explanation Validation
HumanEval-NFR/51 Description: remove_vowels is a function that takes string and returns string without vowels. General The function takes a string ‘text’ as input. Correct The function takes string with or without
vowels as input. General It returns a string. Correct The function should return a string without vowels. General The function should remove all vowels from the input ‘text’ and return the modified string. Correct The function should return a string without vowels from the input string. Edge Handle an empty ‘text’ string, returning an empty string. Correct Input can be empty string. In this case, the function should return empty string. Edge Handle the case where ‘text’ contains only vowels, returning an empty string. Correct Input can contain only vowels. In this case, all of vowels should be removed and the function should return empty string. Edge Handle the case where ‘text’ contains no vowels, returning the original string. Correct Input can contain no vowels. In this case, there are nothing to be removed and the function should return original string. Edge Handle both lowercase and uppercase vowels. Correct Input can be string in lowercase and uppercase. Both of them should be handled. Performance Time complexity: O(n) where n is the length of the ‘text’ string. Correct To traverse the string once and check whether each character is a vowel or not, and then add non-vowel characters to the result string, it requires Opnq time, where n is the length of the string. Performance Space complexity: Opnq where n is the length of the ‘text’ string. Correct Creating a new string to store the non-vowel characters requires additional space. If the length of the input string is n, the length of the resulting string can be at most n, leading to an Opnq space complexity. Performance Ensure efficiency even for extremely large inputs, providing results within 5 seconds. Correct The inputs should be handled in predetermined testing time. Robustness If a non-string input is provided, print an error message to ‘stderr’ and return None. Correct Invalid inputs should be handled and return None. Maintainability Target Cyclomatic Complexity: ≤ 5. Correct 5 indicates low risk, where the code mainly consists of simple blocks. Table 18: Detailed explanation for the human validation of generated requirements by ARCHCODE (Index: 0 in Table 17) on HumanEval-NFR/51. Sub-requirement Generated Requirement Human
Explanation Validation
HumanEval-NFR/11 Description: Input are two strings a and b consisting only of 1s and 0s. Perform binary XOR on these inputs and return result also as a string. General The function takes two strings ‘a’ and ‘b’ as
input. Correct The function should take two strings consisting only of 1s and 0s. General It returns a string. Correct The function should return string having result of a binary XOR operation on these inputs. General The function should perform a binary XOR operation on the input strings ‘a’ and ‘b’ and return the result as a string. Correct The function takes two binary strings and return a result of a binary XOR operation. Edge Handle empty strings as input, returning an empty string. Correct The function should handle empty strings and return empty string. Edge Handle strings with different lengths, returning None. Correct When the two inputs have different length, a binary XOR operation is impossible. Edge Handle strings with characters other than ‘0’ and ‘1’, returning None. Correct When the two inputs are not valid binary string, a binary XOR operation is impossible. Performance Time complexity: Opnq where n is the length of the longer input string. Correct The time complexity of performing binary XOR on two strings is proportional to the length of the strings. Performance Space complexity: Opnq where n is the length of the longer input string. Correct If a new string is created to store the result, its length would be equal to the length of the input strings. Performance Ensure efficiency even for extremely large inputs, providing results within 5 seconds. Correct The inputs should be handled in predetermined testing time. Robustness If non-string inputs are provided, print an error message to ‘stderr’ and return None. Correct Invalid inputs should be handled and return None. Robustness If the input strings contain characters other than ‘0’ and ‘1’, print an error message to ‘stderr’ and return None. Correct Invalid inputs should be handled and return None. Maintainability Target Cyclomatic Complexity: ≤ 10. Correct 10 indicates low risk, where the code consists of well-structured and stable blocks. Table 19: Detailed explanation for the human validation of generated requirements by ARCHCODE (Index: 0 in Table 17) on HumanEval-NFR/11. Sub-requirement Generated Requirement Human
Explanation Validation
HumanEval-NFR/0 Description: Check if in given list of numbers, are any two numbers closer to each other than given threshold. General The function takes a list of floating-point
numbers ‘numbers’ and a threshold value ‘threshold’ as input. Correct The function takes two inputs. The first is a list of floating-point numbers and the second is a threshold value. General It returns a boolean value. Correct The function should return True if there are any two numbers in the ‘numbers’ list that are closer to each other than the given ‘threshold’. Otherwise, it should return False. General The function should return ‘True’ if there are any two numbers in the ‘numbers’ list that are closer to each other than the given ‘threshold’. Otherwise, it should return ‘False’. Correct The function checks if there are any two numbers closer to each other than given threshold in given list of numbers. Edge Handle an empty ‘numbers’ list, returning ‘False’. Correct The function should return False if there are not any numbers in the ‘numbers’ list. Edge Handle the case where no two numbers in ‘numbers’ are closer to each other than the ‘threshold’, returning ‘False’. Correct The function should return True if there are any two numbers in the ‘numbers’ list that are closer to each other than the given ‘threshold’. Otherwise, it should return False. Edge Handle the case where all numbers in ‘numbers’ are closer to each other than the ‘threshold’, returning ‘True’. Correct The function should return True if there are any two numbers in the ‘numbers’ list that are closer to each other than the given ‘threshold’. Otherwise, it should return False. Performance Time complexity: Opn2q where n is the length of the ‘numbers’ list. Correct All elements of the given list need to be iterated through, and the difference between each pair of numbers needs to be calculated. This process is proportional to the size of the input list. Therefore, if we denote the length of the input list as n, the time complexity is Opn2q. Performance Space complexity: Op1q. Correct If the problem is solved without using additional memory, the space complexity is Op1q. Performance Ensure efficiency even for extremely large inputs, providing results within 5 seconds. Correct The inputs should be handled in predetermined testing time. Robustness If non-list ‘numbers’ input or a non-float ‘threshold’ input is provided, print an error message to ‘stderr’ and return None. Correct Invalid inputs should be handled and return None. Robustness If non-float elements in the ‘numbers’ list or a negative ‘threshold’ is provided, print an error message to ‘stderr’ and return None. Correct Invalid inputs should be handled and return None. Maintainability Target Cyclomatic Complexity: ≤ 10. Correct 10 indicates low risk, where the code consists of well-structured and stable blocks. Table 20: Detailed explanation for the human validation of generated requirements by ARCHCODE (Index: 0 in Table 17) on HumanEval-NFR/0. Index Generated Test Cases Sub-requirement # Passed Human
Type Cands. Validation
HumanEval-NFR/51
0 assert remove_vowels(‘’) == ‘’, ‘Failed to handle an empty input string. ’
General 10/10 Correct
1 assert remove_vowels("abcdef\nghijklm") == ‘bcdf\nghjklm’, ‘Failed to remove all vowels from the input string. ’
General 10/10 Correct
2 assert remove_vowels(‘abcdef’) == ‘bcdf’, ‘Failed to remove vowels from the input string. ’
General 10/10 Correct
3 assert remove_vowels(‘aaaaa’) == ‘’, ‘Failed to handle case where the input string contains only vowels. ’
Edge 10/10 Correct
4 assert remove_vowels(‘aaBAA’) == ‘B’, ‘Failed to remove only lowercase vowels from the input string. ’
Edge 10/10 Correct
5 assert remove_vowels(‘zbcd’) == ‘zbcd’, ‘Failed to handle case where the input string contains no vowels. ’
Edge 10/10 Correct
6 assert remove_vowels(‘a’ * 10**6) == ‘’, ‘Failed to handle large input size. ’
Performance 10/10 Correct
7 assert remove_vowels(‘z’ * 10**6) == ‘z’ * 10**6, ‘Failed to handle large input size. ’
Performance 10/10 Correct
8 assert remove_vowels(123) == None, ‘Failed to handle case where the input is not a string. ’
Robustness 10/10 Correct
9 assert result. total_complexity <= 5, ‘Failed to have a Cyclomatic Complexity less than or equal to 5 by Radon. ’ Maintainability 10/10 Correct
HumanEval-NFR/83
0 assert starts_one_ends(1) == 9, ‘Failed to count the numbers of 1-digit positive integers that start or end with 1. ’
General 0/10 Incorrect
1 assert starts_one_ends(2) == 19, ‘Failed to count the numbers of 2- digit positive integers that start or end with 1. ’
General 0/10 Correct
2 assert starts_one_ends(3) == 271, ‘Failed to count the numbers of 3-digit positive integers that start or end with 1. ’
General 0/10 Incorrect
3 assert starts_one_ends(0) == 0, ‘Failed to handle the case where n is 0. ’
Edge 0/10 Incorrect
4 assert starts_one_ends(-5) == 0, ‘Failed to handle the case where n is negative. ’
Edge 0/10 Incorrect
5 assert starts_one_ends(10**6) == 900000, ‘Failed to handle large input size. ’
Performance 0/10 Incorrect
6 assert starts_one_ends(‘invalid’) == None, ‘Failed to handle case where the input n is not an integer. ’
Robustness 10/10 Correct
7 assert result. total_complexity <= 5, ‘Failed to have a Cyclomatic Complexity less than or equal to 5 by Radon. ’ Maintainability 2/10 Correct
HumanEval-NFR/100
0 assert make_a_pile(3) == [3, 5, 7], ‘Failed to create the pile correctly. ’ General 5/10 Correct
1 assert make_a_pile(5) == [5, 7, 9, 11, 13], ‘Failed to create the pile correctly. ’
General 5/10 Correct
2 assert make_a_pile(0) == [], ‘Failed to handle the case where n is 0. ’ Edge 5/10 Correct
3 assert make_a_pile(-5) == None, ‘Failed to handle the case where n is negative. ’
Edge 5/10 Correct
4 assert make_a_pile(10**6) == list(range(10**6, 10**6 + 2 * 10**6, 2)), ‘Failed to handle large input size. ’
Performance 0/10 Correct
5 assert make_a_pile(‘invalid’) == None, ‘Failed to handle the case where the input n is not an integer. ’
Robustness 5/10 Correct
6 assert make_a_pile(0. 5) == None, ‘Failed to handle the case where the input n is not a positive integer. ’
Robustness 5/10 Correct
7 assert result. total_complexity <= 10, ‘Failed to have a Cyclomatic Complexity less than or equal to 10 by Radon. ’ Maintainability 10/10 Correct
Table 21: Validation of generated test cases by ARCHCODE on HumanEval-NFR