SelfCodeAlign: Self-Alignment for Code Generation
Authors: Yuxiang Wei
Federico Cassano
Jiawei Liu
Yifeng Ding
Naman Jain
Zachary Mueller
Harm de Vries
Leandro Von Werra
Arjun Guha
LINGMING ZHANG
Venue: NeurIPS 2024 poster (2024)
Keywords: Large language models
Code generation
Instruction tuning
Self-Alignment
Primary area: generative_models
TL;DR: Self-alignment for code LLMs without human annotations

Abstract: Instruction tuning is a supervised fine-tuning approach that significantly improves the ability of large language models (LLMs) to follow human instructions. For programming tasks, most models are finetuned with costly human-annotated instruction-response pairs or those generated by large, proprietary LLMs, which may not be permitted. We propose SelfCodeAlign, the first fully transparent and permissive pipeline for self-aligning code LLMs without extensive human annotations or distillation. SelfCodeAlign employs the same base model for inference throughout the data generation process. It first extracts diverse coding concepts from high-quality seed snippets to generate new tasks. It then samples multiple responses per task, pairs each with test cases, and validates them in a sandbox environment. Finally, passing examples are selected for instruction tuning. In our primary experiments, we use SelfCodeAlign with CodeQwen1.5-7B to generate a dataset of 74k instruction-response pairs. Finetuning on this dataset leads to a model that achieves a 67.1 pass@1 on HumanEval+, surpassing CodeLlama-70B-Instruct despite being ten times smaller. Across all benchmarks, this finetuned model consistently outperforms the original version trained with OctoPack, the previous state-of-the-art method for instruction tuning without human annotations or distillation. Additionally, we show that SelfCodeAlign is effective across LLMs of various sizes, from 3B to 33B, and that the base models can benefit more from alignment with their own data distribution. We further validate each component’s effectiveness in our pipeline, showing that SelfCodeAlign outperforms both direct distillation from GPT-4o and leading GPT-3.5-based distillation methods, such as OSS-Instruct and Evol-Instruct. SelfCodeAlign has also led to the creation of StarCoder2-Instruct, the first fully transparent, permissively licensed, and self-aligned code LLM that achieves state-of-the-art coding performance. Overall, SelfCodeAlign shows for the first time that a strong instruction-tuned code LLM can result from self-alignment rather than distillation.

SelfCodeAlign: Self-Alignment for Code Generation
Yuxiang Wei! Federico Cassano Jiawei Liu' Yifeng Ding' Naman Jain' Zachary Mueller
Harm de Vries* Leandro Werra' Arjun Guha Lingming Zhang'
University of Illinois Jibana-Champaign "Northeastern University
University of California, Berkeley *ServiceNow Research 'Hugging Face RRoblox "Cursor Al
(ywei40, Jingminggeeillinniss....m edu X {cassano.. a aaaaoaahahttorrrhettttnen.
O httpa//o/ooooooee.oooooo//oooooa
Abstract
Instruction tuning supervised fine-tuning approach that significantly improves the ability of large language models (LLMs) to follow human instructions. For programming tasks, most models are finetuned with costly numan-annotated instructionrespoons pairs or those generated by large, proprietary LLMs, which may be permitted. We propose SelfCode-llign, the first fully transparent and permissive pipeline for self-aligning code LLMs without extensive human annota- tions or distillation. SelfCodeAlign employs the same base model for inference throughout the data generation process. It first extracts diverse coding concepts from high-quality seed snippets to generate new tasks. It then samples multiple responses per task, pairs each with test cases, and validates them in sandbox environment. Finally, passing examples are selected for instruction tuning. our primary experiments, we use SelfCodeAlign with CodeQwenl. 1.5-7B generate a dataset of 74k instruction-respone model that achieves 67.1 pass@ @ HumanEval+, surpassing LadeLlama-70BB Instruct despite being ten times smaller. Across all benchmarks, this finetuned model consistently outperforms the original version trained with OctoPack the previous state-of-the-at method for instruction tuning without human annotations distillation. Additionally, we show that SelfCodeAlign is effective across LLMS of various sizes, from 3B to 33B, and that the base models can benefit more from alignment with their own data distribution. We further validate each component's effectiveness in our pipeline, showing that SelfCodeAlign outperforms both direct distillation from GPT-4o and leading GPT-3.5-based distillation methods, such as OSS- Instructt and Evol-Instruct. SelfCode has also led to the creation StarCoder2 Instructt, the first fully transparent, permissively licensed, and self- aligned code LLM that achieves state-of the-the coding performance. Overall, SelfCode Align shows for the first time that strong instruction-tund code LLM can result from self-alignment rather than distillation.
pairs. Finetuning on this dataset leads to a
Introduction
Recent studies have demonstrated the outstanding performance of large language models (LLMs) [33, 40, 19, 57, 45, 69, 8, 70] in various code-related tasks, e.g. program synthesis [8, 3]. program repair [78, 27, 24, 79, 73], code optimization [59, 9], code completion [11, 40, 19], code transla- tion [56. 1, 51], software testing [32, 10, 42, 77]. and software agents [80, 67, 75, 37]. The reason is that modern LLMs are pre-trained over trillions of code tokens in the wild using various training ob- jectives (as such next-token prediction [52]), making the base models natively good at understanding and generating code snippets. Furthermore, fully unleash the power of LLMs, the base models are
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

typically further fine-tuned on nigh-quality following natural language instructions and solving more general software engineering tasks [25]. This step is known as instruction tuning [50].
data to boost their performance in
Curating high-quality data for instruction tuning is crucial yet challenging. One source of acquiring instruction data is to employ human annotation [50]. For example, Llama-3 [14] uses a corpus of 10 million human-annotated examples in instruction tuning. Due the high cost of human annotation, knowledge distillation is widely adopted to train a weaker LLM with outputs generated by stronger LLMs [18]. However, distillation may violate the terms of service [48. 17, 2] of proprietary LLMs and the prerequisite of using stronger LLM limits its generalizability. Therefore, recent proposals focus on instruction tuning without relying on human annotation or distillation [34, 60, 82]. One cornerstone work along this direction is SELF- INSTRUT [68], which finetunes GPT-3 with self-generated instruction data using in in-context learning.
There is a growing number of instruction-tune open-source LLMs in the code domain. However, some models, such as DeepSeck- [19], Llama-3 [14], and CodeQwenl.3 [64], either use propri- etary data or do not disclose their instruction-tuning strategies. Others, including WizardCoder [41], Magicoder [72], WaveCoder [8 and OpenCodeInterpreter [83], rely on knowledge distillation. The only exception is OctoCoder [43], which is instruction-tuned over heavily filtered GitHub commits, with commit messages as instructions and the changed code as responses, as well as data from Ope- nAssistant, a human--genrrann generated corpus of user-assistam conversations [29]. Despite its transparency and permissive licensing, OctoCoder's performance, at 32.9 HumanEval+ pass @ 1, lags behind other mainstream code L.M.s. Meanwhile, previous attempts at applying IELF-INSTRUCT for code generation have resulted in performance degradation over training on natural instruction-respone pairs [43]. Our findings imply that effective self eeff-alinmmen requires a combination of data diversity approach.
control and response validation, which is not present in the traditional
In this paper, we propose SelfCodeAlign, the first fully transparent pipeline to successfully self-align base code LLMs with purely self-generated instruction data. First, SelfCodeAlign extracts diverse coding concepts from high-quality seed functions in The Stack V1 [28]. a large corpus of permissively licensed code. Next, using these concepts, we prompt the base model to generate new coding tasks through in-context learning. We then instruct the base model to produce multiple responses for each task, each paired with test cases for self-validation. Finally, we select only the instruction-respone pairs that pass the test cases. This method ensures the model practices various coding concepts and validates the consistency between instructions and responses.
To evaluate our method, we train CodeQwen .5-7B, a state-of-the-art open-source base LLM for code, on both dataset generated with Allg and OctoPack, naturally-generated and meticulously-filtered dataset used for training OctoCoder [43]. We benchmark both, along with OctoCoder and other models, on a series of tasks: code generation (both function- and class- level) [38, 21, 76, 13], data science programming [30], and code editing [6]. On all tasks, training CodeQwen with SelfCodeAlign significantly improves performance over the base model and over training it on OctoPack. For instance, on HumanEyal+, our model achieves a pass@) score of 67.1, 21.4 points higher than CodeQwen 1.57BB and 16.5 points higher than CodeQwen! 1.5.enn1.7770 1.5-7B-On BB--ctoPacck This highlights the effectiveness of our synthetic data generation method compared to natural data in enhancing the capabilities of code LLMs.
In the component analysis, we justify the different components of the pipeline. We demonstrate that Aelg is general different LLMs whose sizes range from 3B to 33B. In particular, we find that a base LLM could learn more effectively from data within its own distribution than a shifted distribution from a teacher LLM. Additionally, we show that seed selection, concept generation. and exccution filtering all contribute positively to the pipeline. Furthermore, on HumanEval+, Self- CodeAlign (67.1 pass @1 1) outperforms oftte--t-th--rt, GPT-3.5-Turbo -ur-----Tuub- distillation methods, including OSS-Instruct [72] (61.6) and Evol-Instruct [65] (59.1), as well as direct output distillation from GPT-4o [49] (65.9).
SelfCodeAlign has also led to the creation of StarCoder2-Instruct, the first fully transparent, permis- sively licensed, and self-aligned code LLM that achieves state-of-the-art coding performance. We discuss StarCoder2-Instrucce in Appendix A.
Overall, make the following main contributions: (i) We introduce SelfCodeAlign the first fully transparent and permissive pipeline for aelf-aligning code LLMs to follow instructions. Our method

---

does not rely on extensive human annotations or distillation from larger models. (ii) We generate series of datasets using SeliCodeAlign and train multiple models on these datasets, which will all be released to the public. (iii) We thoroughly evaluate our method on a multitude of tasks, showing strong performance across all the evaluated models. (iv) Our experiments demonstrate that training models on their own data can be more effective than using data from stronger, but distributionally different, teacher models when they don't have a huge performance gap. (v) Finally, we run comprehensive component analysis that verifies the positive contribution of each component in SeliCode
2 SelfCodeAlign: Self-Alignment for Code Generation
Figure iillustrates an overview of our SeliCodeAlign technique. first generates diverse in- structions by extracting coding concepts from high-quality seed snippets. This process resembles OSS-Instruct [72], which employs GPT-3.5-Turbo convert random snippets into instructions. However, our method uses the base model exclusively and incorporates a separate concept generation phase that we prove beneficial in $4.3. SelfCode then generates several responses for each task, pairing each with test cases for sandbox execution, and finally chooses passing examples for instruction tuning. Example outputs from each step are listed in Appendix D.1. In the following sections, we provide detailed explanations of these steps.
Category
Response
Tests
Mining
Base model
Base model
Execution
Concepts
Instruction
Response
Response
Code corpus
Seed snippet
Diffiaiiyy
Response
Figure 1: Overview of SelfCodeAlign.
2.1 Seed Snippets Collection
SelfCodeAlign starts by collecting set of seed code snippets from The Stack VI. In this step, it's crucial ensure that the seed snippets are diverse and nigh-quality, as they will be used as the starting point for generating instructions and responses. To collect the seed snippets, we extract all Python functions with doestrings from The Stack V1, and then apply series of filtering rules to ensure the quality of the seed snippets. In total, we collect 250k Python functions from 5M functions with docstrings in The Stack VI, which were filtered by running the Pyright type checker, removing benchmark items, filtering out functions with poor documentation, and removing near-duplicates. Appendix B details this process in depth.
2.2 Diverse Instruction Generation
After collecting the seed functions, we perform Self-OSS-Instrune our adaptation of OSS-Instruct [72] for self-alignment, to generate diverse instructions. In detail, employ in-context learning to let the base model self-generate instructions from the given seed code snippets. This process utilizes 21 carefully designed few-shot examples listed in Appendix The instruction generation procedure is divided into the following two steps:
Concepts extraction: For each seed function, prompt the base model to produce list of code concepts present within the function. Code concepts refer to the foundational principles and techniques used in programming, such as pattern matching and data type conversion.
Instruction generation: We then prompt the base model to self-generate a coding task conditioned on the identified code concepts and two additional attributes. difficulty (casy/medium/hard and category dluntiionclass/poorran implementation). which we randomly sample to enrich the diversity of the generated instructions.
2.3 Response Generation and Self- Validation
Given the instructions generated from Self-OSS-Instrune our next step is to match each instruction with high-quality response. Prior practices commonly rely distilling responses from stronger
3

---

teacher models, such GPT-4, which hopefully exhibit higher quality. However, distilling proprietary models leads to permissiiie licensing and stronger teacher model might not always be available. More importantly, teacher models can be wrong as well, and the distribution gap between teacher and student can be detrimental.
We propose to self-align the base model by explicitly instructing the model to generate tests for self-validation after it produces a response interleaved with natural language. This process is similar to how developers test their code implementations. Specifically, for each instruction, the base model samples multiple outputs of the format (response, tests), and filter out those responses falsified by the test execution under sandbox environment. We then randomly select one passing response per instruction the final instruction tuning dataset.
Main Evaluation
In this section, we comprehensively evaluate SelfCodeAlign over a diverse set of coding tasks:
Function generation ($3.1): Given a natural-language description, LLMs are asked to generate a self-contained function whose correctness and efficiency is checked through test execution [8, 3, 38, 21, 76, 39].
Class generation ($3.2): Given a code skeleton with both class- and method-level information, LLMs are asked to generate the class and its methods [13].
Data science programming ($3.3): Given description of data science task and partial code snippet, LLMs are asked to complete the code snippet to pass corresponding tests [30].
File-level code editing ($3.4): Provided with the contents of file, the model is asked to edit the program following natural language instruction [6].
3.1
Function-level Code Generation
Table 1: Pass @1
(%) of different LLMs on EvalPlus computed using greedy decoding.
Benchmark
Artifact
Model
Instruction data
HumanEval+
MBPP Transparent
Non-proprictary Non-distilled
GPT-4-Turb
Not disclosed
81.7
70.7
Mistral Lurge [22]
Not disclosed
62.8
56.6
Gemini Pro [63]
Proprictary
55.5
$7.9
Llama3-70B-Instruet [14]
Proprietary
70.7
66.4
O
CodeLlama-70B-Imnn [57)
Proprictury
65.2
61.7
WizardCoder:3B-vv.. [41]
GPT distillation
73.2
66.9
OpenCCoakIInerppeeee-S--.. [83]
GPT distillation
67.7
Magicodar-S-DS-6677BB [72]
GPT distillation
70.7
65.4
DeepSeekiCoder:33B-nnnttcu [19]
Not disclosed
75.0
66.7
CodeQwwell5-7BB-Ch. [64]
Not disclosed
77.7
67.2
Snowflake Arctic (480B) [55]
Not disclosed
64.3
64.3
Mixtral-8x222--Instcu---- [23]
Not disclosed
70.1
62.9
Command-Re R+ (104B) [16]
disclosed
56.7
58.6
Mixtral-&x7B-Instruce [23]
disclosed
49.7
OctoCoder-16B [43]
Publicly available
32,9
49.1
StarCoder2-15p 15B [40]
37.8
53.1
CodeQwenl.5-2 5-7B-Basc [64]
45.7
60.2
CodeQwen 15.7--7-000 P.oc-OkkPPackk
Publicly available
50.6
63.2
SellCodeAllin-CQ-77B
celf-generated
67.1
65.2
HumanEval+ and MBPP+. HumanEval [8] and MBPP [3] are the two most widely-used benchmarks for function-leved code generation. We their test augmented versions, i.e., HumanEval+ and MBPP+, with x/35 more test cases for rigorous evaluation [38].
As baselines, we consider a diverse set of state-of-the-art instruction-tuned models over various dimensions, including weight openness, data openness, transparency, and performance. Table compares the pass @ I of the self-aligned SelfCodeAllin-CQ. against other baseline models

---

HumanEval+ and MBPP+. Among those trained using fully transparent pipeline without any proprietary data or distillation, SelfCodeAlign-CQ--7 stands out as the best LLM by drastically outperforming the base model, OctoCoder- 16B, StarCoder2- 15B, and CodeQwen w.t.een1.5-7701 Meanwhile, compared to much larger models, SelfnooeeAiig-CCCO R+, and Baraal--xB--nnstttee
outperforms Arctic, Command- while closely matching Mixtral-8x225 Bx22BBinnttruct Even compared to LLMs trained using proprietary data (e.g., manually annotated), Aelin-odAlignn-Q--- remains competitive, surpassing Gemini Pro, Mistral Large, and CodeLlama-7000 -onss-rr-- Additionally, SeliCodeAAign--Q--BB fine-tuned on purely self-generated data, closely rivals models finetuned with distillation-baseed or transsaaptr synthetic data.
LiveCodeBench. In subsequent evaluations, we benchmark our model against state-of oftte-of-the-rt open- open-source LLMs of similar sizes for a fair comparison. LiveCodeBench [21] is benchmark for contamination-free evaluation. features 400 recent Python algorithm challenges from May 2023 to February 2024. These tasks are curated from online judge websites such as Codeforce and LeetCode, each with over 20 test cases on average. While LiveCodeBench a holistic benchmark covering four problem types, we focus the code generation task for assessing LLM function generation.
Table reports the pass results for problem subsets created after three specific start dates. shows that Selffodee:linn-CC--7 consistently outperforms most baseline models and closely matches CodeQwenl.5-7B-CHE In addition, moving the start date forward has minimal impact on the pass of SelfCodeeAlig-CC-773 indicating that our pipeline is less likely to suffer from contamination.
Table 2: Pass@1 (%) of LLMS on LiveCodeBench. Newer start dates imply lower contamination risk.
Start date
Model
2023-09-01
2023-07-01 01
2023-05-01
DeepSeck-Coder<a 6.B.- 7B-Instruct
19.2
20.8
21.6
CodeGemma-7B-IT
15.2
13.6
Llama-3-8B-Instre
18.3
18.4
17.3
CodeQwen1.5-
19.3
20.7
21.8
CodeQwen1.5-7B-CC
23.2
25.0
OctoCoder-166
12.6
11.2
9.8
StarCoder2-15B
14.5
14.7
15.4
CodeQwen1.55 1.5- 1.t.-wmn..57B-OcotPPc
19.3
21.8
22.5
SeliCodeAlign-CQQ-BB
22.4
22.8
23.4
EvoEval. To mitigate the impact of potential data contamination, EvoEval [76] includes 828 programming problems created by prompting GPT-4 evolve original HumanEval tasks across semantic-altering and semantic-preserving benchmarks. Following the leaderboard of EvoEval, we use the 5 semantic-altering benchmarks, each of which has 100 problems.
Table 3 shows that BeifcodcAlign-CQ-77 achieves the best pass@ score among all transparently finetuned models. Meanwhile, it also surpasses most open LLMs (except CodeQwenl 1.5-7B-Chal trained on unknown instruction-tuning data.
Table 3: Pass @ (%) of code LLMs on EvoEval.
Model
Average
Difficult
Creative
Subtle
Combine
Tool use
DecpSeek-Codee Coder-6.7 Middrr6.7BB Instruct
41.4
40
37
61
18
51
CodeGemma-7BBII
35.4
31
32
49
9
56
Llama-3-8PP 3-88--Instruct
40.6
34
39
57
58
CodeQwen 1. 5-7B-Baas
36.2
26
30
46
18
61
Codee@wenl.5-7BCChhe
48.0
39
38
71
31
61
OctoCoder-16B
30.6
19
26
43
11
54
StarCoder2-155
25.8
16
19
41
5
48
CodeQwenl.5- 1.5.enll.5-BB.... 7BB-cooPakkkk
42.2
35
36
59
22
59
SelfCodeAAlinn-CQTT
43.6
33
40
60
20
65
5

---

EvalPerf. While the earlier benchmarks focus on code correctness, we use EvalPerf [39] to evaluate the efficiency of LLM-generater code. EvalPerf includes 1 118 ceaprmmmann--eeeeiieeee tasks with computation-intensivee test. inputs to fully exercise the efficiency of L.L.-generated code.
Since code efficiency only matters when the generated code is correct, in Table 4 only evaluate baselines that can achieve a decent pass@ @ (i.e., over 50%) on HumanEval+. Specifically, we run EvalPerf by following its default settings: (i) Each model generates 100 samples per task at the temperature of 1.0; (ii) We evaluate the efficiency of up to 20 correct samples per model for tasks where can at least generate 10 passing samples; and (iii) Finally we rank the models based on their win rates, where each model pair compares their differential performance score (DPS) over the common of passing tasks. Notably, DPS is LeetCode-insppel efficiency ranking of submissions. For example, Table 4 shows that SelfCode Align-CC--- CQ-7B achieves a DPS of 79.9, indicating that its correctly generated solutions can overall outperform or match the efficiency 79.9% of reference solutions across various efficiency levels.
metric that indicates the overall
Table 4 shows that SelfCodeAAign-CC--77 ranks second among the evaluated models of comparable size. Specifically, SellCodeAlign-CQ-7BB is only next to DeepSeek-Codee -oder-6. -.w.e-o..---nssstieee. 7B-Instruct whose training data not disclosed. Surprisingly, the efficiency of SeliCodeAlign-CQ-77B --ggeeegeee code surpasses many recent open models trained using private data, including the latest Llama-3. 1--BB- Instruct.
Table 4: Ranking of model code efficiency based on the EvalPerf win rates, which are computed over the common set of passing tasks for each model pair. Each model generates 100 samples per task at a temperature 1.0. To exemplify differential performance score (DPS) with SeliCCodeAlinn-CQ-BBB it means its generations if correct can match the efficiency of 79.9% LLM samples.
Model
DPS (%)
pass @ (%)
Win-rate (%)
DeepSeek-Coder-6.77 Cowee--.77--nsstneeeeee BB-Instr
83.6
73.6
63.9
-lama-3.1-8B-Instmnc
80.9
64.3
52.1
--stttuuttt
77.0
43.7
51.5
80.7
74.1
51.2
CodeQwenl. 7B-occoPtak
74.0
49.1
26.9
SelfCodeAlign-CQ-7B
79.9
65.2
54.0
3.2 Class-level Code Generation
We evaluate code LLMs on class-level code generation using ClassEval [13], a collection of 100 class-level Python code generation tasks, covering 100 classes and 410 methods, with average of 33 tests per class and 8 tests per method.
Following the ClassEval paper [13], we set the maximum model context size 2048 tokens and report the best class-level pass@) (and corresponding method-level pass@ 1) of each model among three generation strategies: (i) Holistic Generation; generating the entire class given a class skeleton, (ii) Incremental Generation; generating class methods iteratively by putting earlier generated methods in the prompt, and (iii) Compositional Generation: generating each class method independently without looking at other methods. Specifically, class-level pass@) in Table 5 refers to the pass rate of generated classes given both the method- and class-level tests. In contrast, method-level pass @ is computed by only checking if the generated methods can pass the method-level tests. Table shows, in terms of class-level performance, SelfCode CecCoddAAlignCCCe Align-CQ-7B is the best transparently finetuned model, surpassing the second-best transparent model (i.e., CodeQwent 1.000/eell.--B-Be-e while performing no worse than those using unknown proprietary instruction-tune data. For method generation, SelfCodcAAlin-CQ- also stands out in transparently finetuned models.
by 28%,
3.3 Data Science Programming
DS-0 [30] is benchmark of 1000 realistic data science challenges across 7 popular Python data science libraries. In D000 model must complete a partial code snippet to solve the problem. The solution is then evaluated through test execution. Table 6 shows that SelfCode despite being trained on limited data science code, stands out as the best in the transparent model category, while remaining competitive among the other evaluated baselines.

---

Table 5: Pass @lss@1ia (%) of code LLMs on ClassEval using greedy decoding.
Model
Class-level
Method
DeepScck-Coder-6 M.lier--.BBIInttmac
57.2
CodeGemma-7B-I
21.0
44,8
Llama-3-8B-Instruct
23.0
52.4
CodeQwenl.. -.-BBBae Bass
23.0
52,8
CodeQwenl.5-7B-CUl
27.0
54.6
OctoCoder-
19.0
38.0
StarCoder2-15 15B
9.0
24.9
SoddoQwen11.-7--Oettooee
21.0
45.2
SellCodeAAlggnCQ--BB
27.0
52.6
Table 6: Pass@) (%) on DS-1000 with temperature 0.2 and top-p 0.95 over 40 samples, following the same hyperparameter setting used in StarCoder2 [40].
Model
Avg.
Pandas
NumPy
Matplotlib
TensorFlow
SciPy
Skleam
Py Torch
PeopSeek-Coder- Instruct
44,6
34.0
51.1
58.4
45.9
34.2
45.8
50.6
ColeGGema--BBIIT
30.8
21.9
34.4
54.7
25.1
21.8
22.6
34.5
Llama-3-8B-Instrue
21.5
33.1
$1.9
34.4
25.2
23.8
37.2
CodeQwenl (ateeQwenl.5-7B---ee ---eeBBase
32.4
21.6
35.9
56.7
28.8
28.2
30.9
23.8
Coleewenl.5-7B-Chhe
47.1
34.4
51.7
67.2
46.0
38.9
47.9
52.8
OctoCoder- 16B
13.1
34.0
53.8
22.4
30.0
25.9
StarCoder2-15BB
38.9
26.2
45.8
61.4
38.1
36.0
40.5
22.5
CodeQwenl 1.--1110--77---P--5
38.2
26.7
42.6
61.8
37.7
32.7
36.6
31.4
SellCodeAlign-CQ-..
28.2
42.6
57.2
35.6
42.8
33.3
3.4 Code Editing
We further evaluate LLMs on code editing tasks using the CanltEdit benchmark [6], comprised of 210 code editing tasks from three change kinds (70 tasks each): corrective (fixing bugs), adaptive (adding new features), and perfective (improving existing features). The tasks are evaluated based on the correetness of the generated code changes, according set of hidden test cases. For each task, the model is given as input the original code snippet and natural- -atural-language instruction describing the desired code change; then it expected produce updated code snippet that satisfies the instruction. We follow the setting from the original benchmark [6] to generate 20 completions per task temperature of 0.2. Table 7 reports the pass @ for each change kind and the average pass @ across all tasks. Despite not being specifically tuned for code editing, SelfCode SignnodeeAlgnnCCO-
exhibits strong performance on CanItEdit, achieving pass @ of 39.0%, outperforming all other models except CodeQwenl,5-Chal whose instruction tuning details are not disclosed.
Table 7: @ass I (%) of code LLMs on CanltEdit.
Model
Average
Corrective
Adaptive
Perfective
TrepSeek-Coder-6e Booule--67B-Instruee
34.9
38.8
CodeGemma-7BB.
34.2
30,9
39.3
32.5
Llama-3-8B-Instree
36.0
34.9
39.1
34.0
LodeQwenl.557 1--7B-BassBee
34.7
45.6
CodeQwenl 1.5.0www..--7-C-he.
39,9
38.1
46.6
35.1
OctoCoder-16B
38.4
31.6
StarCoder2-155
36.7
32.1
43.8
34.2
CaleeQwen1.577B-Oettae
36.5
36.9
40.6
31.9
BBlfCodeAlign-CQ-1
39,0
37.4
42.4
37.2

---

Component Analysis
In this section, extensively study how different components contribute the SelfCode pipeline. To make the comparison tractable, we fix a subset of seed code snippets by randomly sampling 37k examples from the 250k corpus and evaluate finetuned models HumanEval+ [38].
4.1 Self-Alignment with Different Models
To assess whether SeliCodeAlign generalizable and how performance varies with finetuning data generated by different models, we run the same data generation pipeline end to end with different L.Mss. We include four diverse state-of-the-art model architectures and sizes ranging from 3B to 33B to observe how SelfCodeAlign performs across small, medium, and large-scale L.Mss.
Table 8 shows the comparison and guides us reach the following findings. Looking at the diagonal cells, SelfCodeAlign consistently improves the performance of the base models with varying sizes, from 3B 33B, Comparing each diagonal cell and the cell immediately its right (i.e., using base models with slightly better HumanEval+ performance model may benefit more from self geferaeedd data than stronger teacher model, when they don't have a large performance gap. However, when the teacher model is clearly stronger, the base model learns better by distilling the teacher's knowledge. For example, StarCoder2-3B achieves higher pass@1 trained on its own data (35.4) compared to Llama-3-8B data (34.1), but when tuned with stronger models, StarCoder2-3B further improves (e.g,, 42.1 with SeepSeek-Coder-1 data). Also, the last row shows that a stronger model can still learn from a weaker model, but less effectively. We provide qualitative examples in Appendix D.2.
the teacher models), we can see that a base
Table 8: HumanEval+ pass@) when finetuning the base models on different data (37k seeds).
Data-generation model
Base model (pass@)
StarCoder2-3B Llama-3-8B StarCoder2-15B DeepSock-Coder-338 CodeQwenl.555 1.5--
StarCoder2-3B (27.4)
35.4
34.1
39.0
42.1
40.2
Llama-3-8B (29.3)
42.7
40.2
41.5
43,3
StarCoder2-15B (37.8)
55.5
53.0
57.3
DeepScck-Coder-311 (44.5)
65.9
62.2
CodeQwenl.5-4 1.5-7 (45.7)
48.8
54.9
56.1
59.1
65.2
4.2 Effectiveness of Execution-based Filtering
The SelfCodeAlign pipeline samples multiple responses for an instruction and each response is equipped with self-generate test cases. Responses with failing tests are filtered out and each instruction will be paired with a randomly selected passing response. To answer the question of "to what extent is execution information helpful", in Table 9, we conduct 4 controlled experiments by varying how responses are selected while keeping the other components unchanged:
Random selection (all): pair each instruction with random response without response filtering.
Random selection (subset): 15.66 subset of "Random selection (all)" for a consistent data amount.
Failures only: pair each instruction with failing response.
Passes only: pair each instruction with a passing response.
Table 9: Pass@ 1 HumanEval+ with different response selection strategies.
Selection strategy
Data size
Execution pass rate
Pass @
Random selection (all)
27.7k
24.15 %
61.6
Random selection (subset)
15.6k
24.2%
61.6
Failures only
15.6k
0%
57.9
Passes only
15.6k
100.0%
65.2
First, we can observe that random pairing performs worse than using only passing examples, both when data sizes are aligned and when they scale up by 1.8x. Mcanwhile, the "Failure only" setting
8

---

results in the worst performance where we deliberately use failing responses for each instruction. These results suggest the importance of execution filtering and code correctness for self-alignment
4.3 Importance of Seed Selection and Concepts Generation
For instruction generation, SeliCodeAlign applies Self-OSS-Instruct that first selects high-quality seed code snippets, then mines code concepts from the seeds, and finally generates the instructions. To validate the usefulness of concept generation and high-quality seeds, we compare two variants of SeliCodeAlign in Table 10: 1) directly generating instructions from seeds, where the model produces seed snippet, and 2) using the default pipeline except for the initial seeds, where random snippets are sampled from different code documents in The Stack VI.
an instruction based solely
Table 10: Pass @ on HumanEval+ using different seeds and pipelines.
Source of seeds
Pipeline
Pass @
Filtered functions
Seed + instruction
59.8
Random snippets
Seed concepts - instruction
64.0
Filtered functions
Seed + concepts -> instruction
65.2
is shown that directly generating instructions from seeds leads to the poorest performance. This is because a direct generation from seeds requires the seed snippet to be presented in the context, whose format is not well represented in the wild and may not be distribution for the model. The generated instructions will then be distracted and thus be of lower quality. Concept generation neutralizes this effect and produces more realistic and natural instructions. Using random snippets produces a more diverse but less coherent set of concepts, leading to slightly worse performance compared using high-quality seeds. Appendices D.3 and D.4 illustrate some qualitative examples.
4.4 Comparing Self-Alignment to Distillation
Table 11: SelfCodeAlign versus distillation using CodeQwen1.55 1.5-7B as the base model.
Method
Dataset size
Teacher model
Execution filtering
Pass @
Evol-Instruct
74k
GPT-3.5-Turbo
59.1
OSS-Instruct
74k
GPT-3.5-Turbo
O
61.6
Direct distillation
74k
GPT-4o
65.9
SelfCodeAlign
74k
.5d-Qwenl.5-7B
67.1
To compare self-alignment with distillation, we evaluate SelfCodeAlign against two state-of-the-art distillation methods for code instruction tuning: OSS-Instruct [72] and Code Evol-Instruct [65]. We use the official OSS-OSSSSUS Instruct dataset. As the official implementation of Code Evol-Instruct is unavailable, we opt for the most popular open-source version [44] on Hugging Face. Both datasets are generated using GPT-3.5- Turbo [46] and we randomly select their subsets to match the 74k samples generated by SelfCodeAlign. Table 11 shows that SelfCodeAllin substantially outperforms both methods, indicating the strength and promising future of self-alignment for code. Additionally, SelfCode outperforms direct distillation, where we use the same set of SelfCodeAlige instruc- tions but rely on GPT-4o [49] to generate each response at temperature O. This suggests that weaker models, combined with more post-validation compute, can produce higher-quality responses.
Related Work
Instruction tuning for code. To build more powerful code assistants, pre-trained code models are fine-tuned over a small amount of high-quality instruction-responee respoonse pairs that are either collected from real-world [43] or synthetically generated [7, 57, 41, 72). This step is known as instruction tuning. OctoPack [43] compiles large set of real-world Git commits which are partially used for code fine-tuning. Code Alpaca [7] applies SELF-INSTRUS to the code domain, which prompts ChatGPT to generate synthetic instruction data for code. Similarly, the instruction data for CODELLAMA [57]

---

includes coding problems generated by prompting LLAMA 2 [66] and solutions and tests by prompting base CODELLAMA. Code Evol-Instruct [41] uses harder programming challenges as instruction data to fine-tune more capable models. Specifically, Code Evol-Instruct prompts ChatGPT with heuristies to evolve existing instruction data to more challenging and complex ones. Besides data complexity, the widely-adopted [14, 62, 71] quality dimension. Specifically, given a source code snippet,
[72] looks at the data diversity and
prompts ChatGPT
to get inspired and imagine potential instruction-response pairs, which inherit the diversity and quality of sampled code snippets. Besides instruction tuning, recent work on training code LLMs for performance improvement also explores multi-turn code generation [83], model merging [12], preference tuning [74, 36], and reinforcement learning [15]. Recently, various strong instruction- tuned code models have been released by major organizations [19, 64]. However, their instruction-tunm tunnnng recipes (e.g., data and strategies) are not fully disclosed. This lack of transparency underscores the need for fully transparent and permissive instruction-tuning methods to advance the field.
Self-alignment. Self-alignment an approach to instruction tuning that utilizes an LLM to learn from its own output without depending is one of the first endeavors that allow GPT-3 to improve itself by generating new instructions and responses for instruction-tunin using its in-context learning capability. SELF-ALICN [61], based on in-context learning, utilizes topic-guided principles to steer the LLM towards desired responses. These mnstruction-respnns pairs are used to fine-tune the base model, followed by a final refinement stage to ensure the model produces in in-depth and detailed responses. Instruction backtranslation [35] offers alternative self-alignment method by initially training a backward model to generate instructions from unlabeled web documents using limited seed data. It then iteratively produces new instructions from new web documents and selects nigh-quality data for self-training. Most code LLM work targets knowledge distillation. Haluptzok et al. [20] share a relevant idea our work but only consider program puzzles specified in symbolic forms. This setting cannot be generalized to real-world tasks with natural language involved.
existing well-aligned teacher LLM.
[68]
for instruction generation and pre-defines
6 Limitations and Future Work
We limit our data generation within a ~3000 window, skewing our distribution towards medium- sized samples. Therefore, generating and training on long-context promising avenue [4]. Second, we gather several negative samples during response generation, which currently filtered out. These negatives could be used in reinforcementtleammin loop to steer the model away from incorrect responses [31, 533. Furthermore, the good responses are labeled by test execution, while the generated unit tests might be erroneous, calling for research to study and improve the generation of valid test cases. Finally, we plan to apply SelfCode to more challenging domains such as complex program generation [84] and agentic software engineering [26].
pairs be
Conclusion
We introduce SelfCodeAlign, the first fully transparent and permissive pipeline for self-aligning code LLMs without extensive human annotations or distillation. SelfCodeAlignn-C-777, finetuned from CodeQwenl.5. 5-BB using SelfCodeAlign, outperforms the 10x larger 7odeLlama-70B- Instrruttt on HumanEval+ and consistently surpasses CodeQwen 1.5 trained with OctoPack on all studied benchmarks. We evaluate SelfCodeAlign across various model sizes, illustrating that stronger base models benefit more from self-alignment than distillation. We also examine the effectiveness of different components in the pipeline, showing that SelfCodeAlign is better than GPT-3.55 and GPT-4o distillation. Overall, we demonstrate for the first time that a strong instruction -unodd code LLM can be created through self-align aligmnnig,,n,, without expensive human annotations or distillation.
Acknowledgements
We thank all the reviewers for their insightful comments and suggestions for our paper. This work was partially supported by NSF grants CCF-2131943, SES-2326173, and Kwai Inc, as well as API credits from the OpenAl Researcher Access Program.
10

---

References
[1] W. U. Ahmad, M. G. R. Tushar, S. Chakraborty, and K.-W. Chang. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2 /1590, 2021.
[2] Anthropic. Terms of service, 7 2023. Accessed: August 17, 2023.
[3] Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton. Program synthesis with large language models. CoRR, 0as/2108.07332 2021.
[4] Y. Bai, X. Lv, J. Zhang, Y. He, J. Qi, L. Hou, J. Tang. Y. Dong, and J. Li. Longalign: A recipe for long context alignment of large language models, 2024.
[5] F. Cassano, J. Gouwar, F. Lucchetti, C. Schlesinger, A. Freeman, C. J. Anderson, M. Q. Feldman, M. Greenberg, A. Jangda, and A. Guha. Knowledge transfer from high-resource to low-resource programming languages for Code LLMs, 2024.
[6] Cassano, L. Li, A. Sethi, N. Shinn. A. Brennan-Jones, A. Lozhkov, C. J. Anderson, and
A. Guha. Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions. In The First International Workshop on Large Language Model for Code, 2024.
[7] S. Chaudhary. Code alpaca: An instruction-followe following Ilama model for code generation. https: 14///////llll//// codeal paca, 2023.
//github cwwc sahi 1200 14/
[8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter,
P. Tillet, F. P. Such, D. Cummings, M. Plappert. F. Chantzis, E. Barnes, A. Herbert- Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021.
[9] C. Cummins, V. Seeker, D. Grubisic, M. Elhoushi, Y. Liang, B. Roziere, J. Gehring, F. Gloeckle, K. Hazelwood, G. Synnaeve, et al. Large language models for compiler optimization. arXiv preprint (tt.:2309.07622 2023.
[10] Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, 2023.
[11] Y. Ding, H. Ding, S. Wang, Q Sun, V. Kumar, and Z. Wang. Horizon-length prediction: Advancing all-in-the-middee capabilities for code generation with lookahead planning. arXiv 2024.
preprint trXiv:2410.031100
[12] Y. Ding, J. Liu, Y. Wei, and L. Zhang, XFT: Unlocking the power of code instruction tuning by simply merging upcycled mixture-of-experts In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 12941- Bangkok, Thailand, Aug. 2024. Association for Computational Linguisties.
[13] X. Du, M. Liu, K. Wang, H. Wang. J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou. Classeval: A canually-crafter benchmark for evaluating llms on class-level code generation. arXiv preprini 20Xv::230.00186 2023.
[14) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron,
B. Tang, B. Chern, C. Caucheteux, C. Nayak. C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Copet, Lee, J. Geffert, J. Vranes, Park, J. Mahadeokar, Shah, van der Linde, J. Bil- lock, J. Hong, J. Lee, J. Fu, J. Chi. J. Huang, J. Liu, J. Wang. J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, Jia, K. V. Alwala, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, L. Rantala- Hantala-Yeary, L. van der
11

---

Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterjj. O. Duchenne, O. Çelebi, P. Alrassy, Zhang. P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q He, Q Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Gird- har, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang. S. Hosseini, S. Chennabasappa. S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang. S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Geor- giou, T. Scialom, T. Speckbacher, T. Mihaylov. T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang. X. E. Tan, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song. Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Grattafiori, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Vaughan, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Franco, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman. A. Yazdan, B. James, B. Maurer, B. Leonhardi.
Huang. B. Loyd, D. Paola,
B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence. B. Stojkovic. B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Wang, C. Kim, C. Zhou, C. Hu, C.-H. Chu, C. Cai, Tindal, C. Feichtenhofer, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Wyatt. D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang. D. Le, Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Brinkman. E. Ar- caute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Ozgenel, F. Caggioni, F. Guzmán, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. That- tai, G. Herman, G. Sizov, Guangyi. Zhang, G. Lakshminarayanan, H. Shojanazeri, H. Zou, H. Wang. H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, I. Damlaj, I. Molybog, I. Tufanov, L.-E. Veliche, I. Gat. J. Weissman. J. Geboski. J. Kohli, J. Asher, J.-B. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong. J. Jin, J. Yang. J. Cummings, J. Carvill, J. Shepard. J. McPhie, J. Torres, J. Ginsburg. J. Wang, K. Wu, K. H. U, K. Saxena, K. Prasad, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Huang, K. Chawla, K. Lakhotia, K. Huang, L. Chen, L. Garg. L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Tsimpoukelli, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Nau- mov, M. Lathi, M. Keneally, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. P. Laptev, N. Dong, N. Zhang, N. Cheng, O. Chermoguz, O. Hart, O. Salpekar, O. Kalinli, Kent, P. Parekh, P. Saab, Balaji. P. Rittner, P. Bontrager, P. Roux, Dollar, Zvyagina, P. Ratanchandani, P. Yuvraj. Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Li, R. Hogan, R. Battey, R. Wang, R. Maheswari, R. Howes, R. Rinott, S. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Verma, S. Yamamoto, S. Ra- maswamy, S. Lindsay, S. Lindsay, S. Feng. S. Lin, S. C. Zha, S. Shankar, S. Zhang, Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max. S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Cho, S. Virk. S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Kohler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Albiero, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang. W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wang. X. Wu, X. Wang. X. Xia, X. Wu, X. Gao, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Hao, Y. Qian, Y. He, Z. Rait, Z. Detito, Z. Rosnbrick, Z. Wen, Z. Yang, and Z. Zhao. The llama 3 herd of models, 2024.
[15] Gehring, K. Zheng, J. Copet, V. Mella, T. Cohen, and G. Synnaeve. Rlef: Grounding code Ilms in execution feedback with reinforcement learning, 2024.
[16] A. Gomez. Introducing command r+: A scalable Ilm built for business, April 4 2024. Accessed: 2024-05-22.
[17] Google. Generative ai terms of service, 8 2023. Accessed: August 17, 2023.
12

---

[18] S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. D. Giorno, S. Gopi. M. Javaheripi,
Kauffmann, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, H. S. Behl, X. Wang, S. Bubeck, R. Eldan, A. T. Kalai, Y. T. Lee, and Y. Li. Textbooks are all you need, 2023.
[19] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, Luo,
Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence, 2024.
[20] P. Haluptzok, M. Bowers, and A. T. Kalai. Language models can teach themselves to program better. In The Eleventh International Conference Learning Representations, 2023.
[21] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang. A. Solar-L K. Sen, and
I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403, 07974, 2024.
[22] A. Jiang, A. Sablayrolles, A. Mensch, C. Bamford. S. Chaplot, D. de las Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, Stock, T. L. Scao, T. Lavril,
T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023.
[23] A. Q- Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot,
D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L., R. Lavaud,
L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao,
T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts, 2024.
[24] N. Jiang, K. LIu, T. Lutellier, and L. Tan. Impact of code language models on automated program repair. In 2023 JEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1430-1442. IEEE, 2023.
[25] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 2023.
[26] C. E. Jimenez, Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2023.
[27] M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy. Inferfix: End-to-end program repair with llms. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1646-1656, 2023.
[28] D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, Wolf, D. Bahdanau, L. von Werra, and H. de Vries. The stack: tb of permissively licensed source code, 2022.
[29] A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. M. Nguyen, O. Stanley, R. Nagyfi, S. ES, S. Suri, D. Glushkov, A. V. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. J. Mattick. Openassistant conversations democratizing large language model alignment. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[30] Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, I.. Zettlemoyer, S. W. tau Yih, D. Fried, S. Wang, and T. Yu. Ds-1000: A natural and reliable benchmark for data science code generation, 2022.
[31] H. Le. Y. Wang, A. D. Gotmare, S. Savarese, and S. Hoi. CodeRL: Mastering code genera- tion through pretrained models and deep reinforcement Icarning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[32] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference Software Engineering (ICSE), pages 919-931. IEBE, 2023.
[33] R. Li, L. B. Allal Y. Zi. N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy- Poirier, Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.-H. Yee, L. K. Umapathi,
J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, Stillerman, S. Patel, D. Abulkhanov,
M. Zocca, M. Dey, Z. Zhang, N. Fahmy, U. Bhattacharyya, W. Yu, S. Singh, S. Luccioni,
Villegas, M. Kunakov, F. Zhdanov, M. Romero, T. Lee, N. Timor, J. Ding, C. Schlesinger, H. Schoelkopf, J. Ebert, T. Dao, M. Mishra, A. Gu, J. Robinson, C. Anderson, B. Dolan-Gavitt, D. Contractor, S. Reddy, D. Fried, D. Bahdanau, Y. Jernite, C. M. Ferrandis, S. Hughes, T. Wolf,
A. Guha, L. von Werra, and H. de Vries. Starcoder: may the source be with you!, 2023.
13

---

[34] X. P., Yu, C. Zhou, T. Schick, O. Levy, L. Zettlemoyer. J. E. Weston, and M. Lewis. Self- alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024.
[35] X. L. P. Yu, C. Zhou, T. Schick, O. Levy, L. Zettlemoyer, J. E. Weston, and M. Lewis. Self- alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024.
[36] J. Liu, T. Nguyen, M. Shang, H. Ding, X. Li, Y. Yu, V. Kumar, and Z. Wang. Learning code preference via synthetic evolution. arXiv preprint arXiv:2410.0388 2024.
[37] J. Liu, K. Wang, Y. Chen. X. Peng, Z. Chen, L. Zhang, and Y. Lou. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409. 02000, 2024.
[38] J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[39] J. Liu, S. Xie, J. Wang, Y. Wei, Y. Ding, and L. Zhang. Evaluating language models for efficient code generation. In First Conference on Language Modeling, 2024.
[40] A. Lozhkov, R. Li. L. B. Allal, Cassano, LLLL--mmmriiii N. Tazi, A. Tang, D. Pykhtar,
J. Liu, Y. Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint
arXiv:2402.191 19173, 2024.
[41] Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma. Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023.
[42] R. Meng, M. Mirchev, M. Böhme, and A. Roychoudhury. Large language model guided protocol fuzzing. In Proceedings of the 31 st Annual Network and Distributed System Security Symposium (NDSS), 2024.
[43] N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, S. Singh, X. Tang. L. von Werra, and S. Longpre. Octopack: Instruction tuning code large language models, 2023,
[44] nickrosh. Open Source Implementation of Evol-IInstuuc--ooe. datasets/ ni wwtessttsiiikooss///oooooo wwwkkross//ollo.eeee Instruct Code- 80k- 2023.
co/
[45] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang. Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023.
[46] OpenAI. Chatgpt: Optimizing language models for dialogue. chatgpt/, 2022.
[47] OpenAl. Gpt-4 technical report, 2023.
[48] OpenAl. Terms of service, 2023. Accessed: August 17, 2023.
[49] OpenAl. Gpt-4o system card. 2024.
[50] L. Ouyang, J. Wu, X. Jiang. Almeida, C. L. Wainwright, P. Mishkin, Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, Christiano, J. Leike, and Lowe. Training language models to follow instructions with human feedback, 2022.
[51] R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavu- luri, S. Sinha, and R. Jabbarvand. Understanding the effectiveness of large language models in code translation. arXiv preprint arXiv:2308.08 109, 2023.
[52] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre-training. 2018.
[53] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[54] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020.
[55] S. A. Research. Snowflake aretic: The best I1m for enterprise ai - efficiently intelligent, truly open, April 24 2024. Accessed: 2024-05-22.
14

---

[56] B. Roziere. M.-A. Lachaux, L. Chanussot, and G. Lample. Unsupervised translation of pro- gramming languages. Advances in neural information processing systems, 33:20601-20611 2020.
[57] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défosse7, J. Coper, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve. Code Ilama: Open foundation models for code, 2023.
[58] N. Shazeer and M. Stem. Adafactor: Adaptive learning rates with sublinear memory cost, 2018.
[59] A. Shypula, A. Madaan, Y. Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan,
O. Bastani, and A. Yazdanbakhsh. Learning performance-improoing code edits. arXiv preprint
arXiv:2302.07866 2023.
[60] Z. Sun, Shen, Zhang, Q. Zhou, Z. Chen, D. D. Cox, Y. Yang, and C. Gan. SALMON: Self-alignment with instructable reward models. In The Twelfth International Conference on Learning Representations, 2024.
[61] Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. D. Cox, Y. Yang, and C. Gan. Principle- driven self-alignment of language models from scratch with minimal human supervision. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[62] C. Team, H. Zhao, J. Hui, J. Howland, N. Nguyen, S. Zuo. A. Hu, C. A. Choquette-Choo
J. Shen, J. Kelley, K. Bansal, L. Vilnis, M. Wirth, P. Michel, P. Choy, P. Joshi, R. Kumar,
S. Hashmi, S. Agrawal, Z. Gong. J. Fine, T. Warkentin, A. J. Hartman, B. Ni, K. Korevec,
K. Schaefer, and S. Huffman. Codegemma: Open code models based on gemma, 2024.
[63] G. Team. Gemini: A family of highly capable multimodal models, 2024.
[64] Q. Team. Code with codeqwen April 16 2024. Accessed: 2024-05-20.
[65] theblackcat102. The evolved code alpaca dataset. betpss:/huugingen cttpaaasetc//taas/s/
theblackcatt00//orkkk eww.:///eoollooooo pac.aaalpaaaaaalpaaaaaaaaaaaaaaa
2023.
[66] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao. V. Goswami, N. Goyal, A. Hartshom, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[67] X. Wang. B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song. B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. Openhands: An open platform for ai software developers as generalist agents, 2024.
[68] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self- instruct: Aligning language models with self ceff-generated instructions. In A. Rogers, J. Boyd- Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13488-13508, Toronto, Canada, July 2023. Association for Computational Linguistics.
[69] Y. Wang, H. Le, A. D. Gotmare, N. D. Q. Bui, J. Li, and S. C. H. Hoi. Codet5+: Open code large language models for code understanding and generation, 2023.
[70] Y. Wang, W. Wang, S. Joty, and S. C. Hoi. CodeT5: Identifier aware unified pre-trained encoder- decoder models for code understanding and generation. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
[71] Y. Wei, H. Han, and R. Samdani. Aretic-snoweoder. Demystifying high-quality data in code pretraining. arXiv preprint 240i:22400902226.. 2024.
[72] Y. Wei, Z. Wang, J. Liu, Y. Ding, and I.. Zhang. Magicoder: Source code is you need, arXiv
preprint 23X.:23312.021000 2023.
15

---

[73] Y. Wei, C. S. Xia, and L. Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE, 2023, page 172-184, New York, NY, USA, 2023. Association for Computing Machinery.
[74] M. Weyssow, A. Kamanda, and H. Sahraoui. Codeultrufeedback: An lim-as-a-judge dataset for aligning large language models coding preferences. arXiv preprint arXiv:2403. 000000 2024.
[75] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying Ilm-based software engineering agents. arXiv preprint, 2024.
[76] C. Xia, Deng, and I.. Zhang. Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via Ilm. arXiv preprini arXiv:2403. /9/ 14, 2024.
[77] C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models, 2023.
[78] C. S. Xia, Y. Wei, and L. Zhang. Automated program repair in the era of large pre-trained language models. In 2023 IEEF/ACM 45th International Conference on Software Engineering (ICSE), pages 1482-1494, 2023.
[79] C. S. Xia and I.. Zhang. Less training, more repairing please: Revisiting automated program repair via zero-shot learning, 2022.
[80] Yang, C. Jimenez, A. Wettig, Lieret, Yao, K. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024.
[81] Z. Yu, X. Zhang, N. Shang, Y. Huang, C. Xu, Y. Zhao, W. Hu, and Q. Yin. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation, 2024.
[82] W. Yuan, R. Y. Pang. K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston. Self-rewarding
language models, 2024.
[83] Zheng, G. Zhang, Shen, X. Liu. B. Y. Lin, Fu, W. Chen, and X. Yue. Opencodeinterpreeeer: Integrating code generation with execution and refinement. arXiv preprint 24xXi::2402116655 2024.
[84] T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, T. Hoang, A. R. Zebaze, X. Hong, W.-D. Li, J. Kaddour, M. Xu, Z. Zhang, Yadav, N. Jain, A. Gu, Z. Cheng, Liu, Q. Liu, Z. Wang. D. Lo, B. Hui, N. Muennighoff, D. Fried, X. Du, H. de Vries, and L. V. Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024.
16

---

A StarCoder2-Isttruct Fully Transparent and Permissive Self-Alignment for
Code Generation
</>
StarCoder2 Instruct
built by BigCode
StarCoder2-Instruct is the very first entirely self-aligned code LLM created with earlier version of SelfCode/lign. The pipeline uses StarCoder2- 15BB to generate thousands of instruction-respons pairs, which are then used finetune StarCoder2-15B itself without any human annotations or distilled data from huge and proprietary LLMss. StarCoder2-1 S---Codrr-11B-hee Ins-rrttuett achieves 72.6 HumanEval score, surpassing the 72.0 score of Codel Ion-ttsrutt. More details are explained the blog: https://huggingfant ct/blcc/ccccccccc..c sc2- instruct.
B Seed Data Curation
import torch
def one_hot (y, num_dim=10):
One Hot Encoding, similar to 'torch. eyee eye (num_dim). index_ select (dim -0, index-y)
:param y: N-dim tenser
:aara num num_dim: do one-hot labeling from '0 to 'num_ dim 11m-1 :return: shape (batch_aize, num_dim)
one one_hot_y = torch. zeros (y s0)e num_dim) if cuda: one_hot_y one_hot_y c)da
return one_hot_y scatter_ (1, y. view (-1, 1). 1.)
Listing An example Python seed function from final dataset.
In this section we describe in detail our seed gathering process, which is used to collect the initial set of seed Python functions for the concept extraction phase.
provide an example seed function in Listing 1. All seeds our dataset take on the format of: imports (if any), signature, docstring, implementation.
B.1 Python Function Extraction
(function_defil definition
name: (identifier)
body: (block
17

---

(expression_statemett (string
(string_start) @docstring start
(string_content
ofunction. def (#eq? @docstring start "I "111"111"\\1"""
Istring_endl @docstring.endi
(#eq? @docstring.. end "[11"111"111"")
Listing 2: The Tree-sitter query utilized for extracting Python functions with docstrings.
Our seed gathering process starts off by extracting all Python functions with docstrings from The Stack vl, which is large dataset of code from GitHub. We accomplish this by utilizing the Tree-sitter parser and the query language provides. Listing 2 provides the query utilized for matching each function, assuring that the functions live at the top-level of the program, and that they indeed contain a docstring. Utilizing this query, we extracted total of 5,359,051 Python functions with doestrings. Interestingly, we found that roughly only 20% of Python functions contain any doestrrig.
B.2 Quality Filtering and Transformations
After gathering 5M Python functions, apply series of filtering and transformations steps. After all of our filtering rules, we are left with a dataset of 248,934 high-quality Python functions. These steps generalization of the dataset building pipeline in MultiPL-T [5], which only manages to produce half as many functions (those without imports). We detail each step of this process below.
Import prediction By naively extracting the functions from Python files, we may have lost import statements external libraries that are utilized inside the function. To remedy this, we utilize the autoimport library to infer potential import statements for unbound identifiers in the function.
Removing benchmark data To enable fair evaluation of our method, we decontaminate our seed dataset from examples that resemble prompts and solutions of items in the benchmarks on which we evaluate on. We accomplish this by checking if either the substring of the solution or prompt each benchmark item exists in any function in the dataset.
Return filtering To aid in our self self-validation step, we aim to include only functions that return value, such that potential responses will contain test cases with complex expected values. We utilize Tree-sitter to filter any function that does not contain at least one return statement with an argument value.
Type-checking To further ensure the quality of our Python functions, we apply Pyright, a heuristic type-checker for Python, on all of our functions. and keep only ones passing the check. This step also ensures that no unbound identifiers are referenced inside the function.
Docstring quality filtering We find that several Python functions, while having defined doc- string, contain poor or misleading documentation. In aims of removing such functions, we employ StarCoder2-15B with simple binary classification prompt, tasking the model to detect functions with poor docstrings. We remove all functions that were deemed poor quality by this classifier.
Near-deddullicttin As final step, we wish to increase the diversity of seeds in our dataset. To accomplish this, we utilize MinHash with Aocality-Sensittee Hashing and a Jaccard Similarity threshold of 0.5 to identify duplicate groups of functions in our seed dataset. We then only pick single function from each group, and add our final dataset. We note that this is the same process utilized to deduplicate the pre- pre-training dataset of StarCoder and StarCoder2 [33, 40].
C
Implementation Details
C.1 Data Generation
We implement few-shot examples of the form (seed, property, instruction, response, tests), where coding concepts are encoded in the property of each example. Besides coding concepts and
18

---

programming language, a property includes task category and difficulty level that are randomly sampled during data generation. We use eight-shot for concept and instruction generation, and one- shot for response generation. During response generation, we explicitly guide the model to generate tests by concatenating the response and tests in the one-shot example. For the main experiment, the test case follows specified format, we additionally include it in the instruction body with fifty percent chance to boost diversity. Table 12 shows the estimated cost for end-to-end data generation with different models.
Throughout the data generation pipeline, we follow [61] and choose a temperature at 0.7 to strike balance between diversity and quality.
Table 12: End-to-end data generation time cost on 4x A00
Base model
Seed data
Produced data
Generation cost
Finetuning cost
Llama-3-8B
37k
11k
7h
20min
CodeQwenl.5-7B
37k
16k
7h
30min
StarCoder2-15B
37k
15k
12h
2.5h
StarCoder2-3B
37k
12k
6h
16min
DeepSeck-Coder-33
37k
15k
83h
3.5h
C.2
Execution
We implement Docker-based execution server for code execution. This sandbox environment includes widely-used Python packages such Numpy, PyTorch, and Pandas, allowing us safely execute arbitrary Python code. Additionally, the server supports parallel requests to speed up validation.
C.3
Training
Our overall hyperparameter choices are derived from existing good practices [41, 35, 72, 83). We set the initial learning rate at 1e-5 for training on self-generated data and 2e-5 for training data generated from other models. Empirically, we find this to be the optimal setting for both cases. We adopt a 0.05 warmup ratio and linear scheduler. We use Adafactor [58] as our optimizer and choose a batch size of 64 with a sequence truncation length of 1280.
C.4 Computater Resources
We primarily conduct data generation, training, and evaluation node equipped with 4 NVIDIA A100 PCI-E GPUs, 128 cores, and 512 GB of memory. For experiments involving DeepSeek- Coder, node with 8 NVIDIA H100 GPUs. For DeepSeek-Code we utilize DeepSpeed ZeRO-3 [54] for training. For StarCoder2-15B we use one A100 for training since otherwise it cannot fit the GPU memory due to the extra overhead caused inter-GPU communication. For all the other experiments, we do a 4-GPU training using PyTorch's Distributed Data Parallel (DDP) module.
C.5 License
As indicated in Table 13, each datasct's license inherits the license of the model that produces it. Additionally, each model's license combines both the base model's license and the data generation model's license. Our source code is licensed under Apache-2.0.
D Examples
D.1 Seeds, Concepts, Instructions, Responses, and Tests
[Seed]
def get wemaaiiiiiaiaaaaammmemmmmaaaaiaa (email):
"**Generates email domain from given email address. "** return "иии. "temail. split ("0") [-1]
19

---

Table 13: License of models and datasets
Asset
Type
License
StarCoder2-3B
Model
BigCode OpenRAIL-M vl
Llama-3-8B
Model
Ilama3
CodeQwenl.5-7
Model
congyigjiawwen-rsseacch
StarCoder2-15B
Model
BigCode OpenRAlL-M vl
DeepSeek-Coder-333B
Model
deepseek-license
$ (DATASET) generated by (MODEL)
Dataset
License of $ 8-(MODEL>
$<MODEL_AA A} finetuned on
Model
License of $ 8-(MODEL_ _] +
((DATASET) generated by
License of $ ((MODEL_B) B)
$-MMDEEL_BB
Source code
Code
Apache-2.0
[Concepts]
[' ('domain extraction from email addresses', URL manipulation', 'string
mamipulation']
[Instruction]
Use Python to create class called 'EmailProcessor'' The class should have methods to extract the domain name from an email address, split URL into its components (protocol, host, port, path), , and perform simple reger search string. The class should also is valid can be opened a
have a method to check web browser.
[Response]
Here 's how you can create the 'EmailProcessor' class:
python
impprrt re
from urllib.pars import urlparse
class EmailProcessor:
@staticmethod
def extract_domain (email: str) str: return email. spli(( [i]
@staticmethod
def split_url str) >> dict: parsed_url urlparse(url)
return {
'protocol': url hostname, host': parsed_url
freeeduur..schenr
'port': parsed_url urlddurl.ppll ureeot,
'path"; parsed_url. path,
}
@staticmethod
def regex_search stex:: str, pattern: str) bool: return re. search (pattern, text) is not None
@staticmethod
def is is_valid_ura url (url: str) bool:
try:
result urlparse(uri)) (url)
return ([resutt scheme, result. reslot)
except ValueBrror:
20

---

return False
Ostaticmethod def open_url (url: str):
import webbrouser webbrouser (uen(url)
The cextract_domaine method extracts the domain name from an email address by splitting on the ' symbol. The 'slii_urr'' method urlparsee to split URL into its components and returns method uses 're. search"
them as dictionary. The "regex_search to perform a regex search on a string and returns True if match is found, otherwise False. The 'is_valid_urs urrl method uses urlparse' to validate URL and returns True if it's valid, "webbrowser open
otherwise False. The 'open_url" method open a URL in web browser.
[Tests]
python
Test 1: Test decract_domaii method
assert EmailProcessor extract_domatt (rapcttttommittttttttteeeeaeee ("testt@exaple..om 'example. com
Test 2: Test split_url method
assert imailProcessor.ppli split_urllt ('ittttls:://///aaaaa http://wwwwwampple..///. com :8080/ path /t/ resource') ==
'protocol'; 'https',
'osst' 'www. www.exxapee.com.
'port': 8080, 'path'; //path/to/resourcee,
Test 3: Test regex_search method assert EmailProcessor rogex_search<HHell world!', r'world')
Test 4: Test ta_valid_uri method
assert EmailProcessor is valid url ('hhtips // //www//wwww//////w//////////w example.......ee...e..eeeeeeeeee com aasert EmailProcessor.. is va_aaiid_urlll ('not -at-l-)
Test 5: Test open_url method (skipped, as it's not testable in Python)
[Seed]
def gget_composite_nue (compssite_terms ebm_term_namest ".*Returns composite name format "term_name_1, term_name_2, ....
Args: composite_terie A list of ebm_term_ list of all ebm term names
indices
Returns:
str: The composite name
name
for term in composite_terms if (tintannce(term, str) and term in abm_term_namest
if len (name) 0 else " + temm__amess elif isinstance (term, int) and term len (ebm_
name +- names [term] if len (name) - else ", ebm. term [term]
21

---

else:
raise ValueError ("Term '0 is string valid integer format (term))
return name
[Concepts]
[ Pstrrng matching for names', 'integer indexing for indices eeror handling for invalid inputs
[Instruction]
Design Python class "WordNet" that stores collection of word terms their corresponding definitions. You can assume term is a string and its definition is a list of strings. You should implement the following methods:
init.. Initializes definitions.
empty collection of terms and their
add_ term, definition)'; Adds a new term and its definition to the collection.
get_definition (self, term)": Retrieves the definition of a given term from the collection.
got_terms (self, definition)'; Retrieves terms that have the given definition.
get__trm__contt (self)' Returns the total number of terms in the collection.
get_definiiiinneee count (self)"; Returns the total number of definitions the collection.
The class should handle input validation, such as ensuring terms are non non-empty strings and definitions lists of strings, and raise appropriate errors invalid inputs are provided.
[Response]
To implement such class, can define the class 'Wodllet" with the required methods. We'll
dictionary store the terms and their definitions. The "add_term" method will check the input term and definition are valid and raise appropriate errors if not. The get get_definition'. get_terma', get_term_count" and set_definition. count" methods will retrieve the corresponding data from the dictionary.
python class Wordliet:
def (self_ self terms
def ted_ter term, definition):
if not term not isinstance (term, str):
raise ValueError ("Invalid term. Term be non eoptympty string.")
if not definition or not isinstance (definition, list) or not definition:
raise ValueError (" ("Invalid definition. Definition must be non empty list of strings. self.tere [term] definition
def get_definition (self, term): if term not in self.terms. raise ValueError (T"e '{term) not found.")
return self terms (term]
def ger_terms (self, definition):
terms [term for term, defs in self items if definition in defs] return terms
22

---

def get_tern_count (self): return len (self. terms) def return sum (len (defs) for defs in self terms, values ())
count :self):
This implementation handles input validation and raises appropriate errore if invalid inputa are provided. The "et_ttema" method returns list of terms that have the given definition, which can be empty if such terms exist.
[Tests]
We can write tests to validate the correctness of the 'WordNet" class:
python
import unittest
class TestWordNet (unittest. TestCase):
def setUp (self):
self wordnet WordNet ()
add_term ('apple', ['iruit', 'red']) wordnet. add_tere ('banana', ['fruit', 'yellow'])
waffwwordnet add_term (t ('carrot', ['vegetable', 'orange '])
ter__ae)
self. wordnet.......mmm add_ters fodd_term''grappee ('grape', ['fruit', 'green'l) self. assertEquull (self wordnet 'green'])
['fruit
self assertEqul (self. wordnet get_definition ('apple))) [fruiit 'red'])
test_get_termsse11)) self assertEgelllll (sorted (self wordnet get_terms ('fruit ?)), [> apple', 'banana'])
self assertEqual (aelf wordnet get_ terms green "), [ D'grape '])
def test_gee..orr_.cott(teellll
self assertEquu (self wordnet get_teree tert_tter_cccutt (), 3)
def gess_ge__deiintiionccmmmm
self assertEqual (self (self wordnet. get_ ret__eeinnitonncouns 6)
def test (es__hmallld_lmptts(sss...:
with self seserrrtiiieesss
self welf.vordnet. www.nerdde...mm... wordnet. ("), [ ['fruit]])
with self aesertteeeeeee((llelbrrr)))) ['fruit', ?))
self. aord with self self wordnet get get_definition. oorangee)
if name..... '__main_." unittest main()
[Instruction]
Write Python function "comt_.ubttringiterrtivee that takes two argumenta: a string "text and string "subatring" The function should iterate through the string 'text' count the number of occurrences of the string "substring" The function should return the count of non-overlapping occurrences.
23

---

For example, "ount_substringgttrraat ("abababa", "aba")" should return "3", since "abababa" contains three overraaaaaiin occurrences aaba: "ababa", baba",, and "aba".
[Response]
The goal is to implement Python function "count_ "ount_suustringgttrrea
that takes arguments: string "text' and string "subatring
The function should iterate through the string "text" count
the number of occurrences of the string aubstring', The function should return the count of non non-overlapping occurrences.
Here are the key steps to implement this function:
Initialize a variable 'count' to keep track of the number of occurrences.
Loop through the string "text" using sliding window the size of
the string "subatring':
Check if the current substring of the same size as "aubstring" matches the string "substring'.
If it matches, increment the count.
Move the window the next position by advancing by one character.
Return the final count of non-overlapping occurrences.
Here 's how you implement the ccunt conn_ssubstrnng function:
"python
def count soutt_subtring (tex:: str, substring: str) -> int:
count =
sub_len len (eubsstring) i range (len (text) sub_les 1): if text text(i:iiaub_len == subatring:
count +=
return count
[Tests]
To test the function, you use typical test cases such as count www.sabttin.teeratttiiteeeeeeeee ("abababa" "aba")" and count_sustrinng.teertiee ("hello", "11")".
python
assert couttsubstrinng.toortvvveeeeeeee ("abababa", "aba") == 3 assert countssubttrnngltterrtiee ("hello", "I1")
assert count_substti Sountssubttinng_tttrrtttt (ew.rrtiivveniisssssssppp ("mississippi", "iss") 2
[Seed]
def splitLine (text))
"""split line of text the first space character and return strings, the first word and the remaining string. This is used for parsing the incoming messages from left to right since the command and its arguments are all delimited by spaces and the command may not contain spaces "** sp = text.split(" ")
try:
a =sp[0]
b= b (sp
except:
text
n"
return a, b
[Concepts]
24

---

('string manipulation', ', 'string delimiting', "list slicing', exception handling")
[Instruction]
Create a Python function that splits a string into list of non empty substrings using specified delimiter. The function should raise exception if the imput string delimiter empty. This implementation should use string manipulation techniques to perform the split operation and should not use the 'split()* method directly.
[Response]
can implement a Python function that splits a string into a list of non non-empty substrings using a specified delimiter. The function should raise exception if the input string delimiter is empty. Here is the Python program that implementa this functionality:
python
def split_string crrprigggccstoonnnuuttttttti (input_string, delimiter):
***
Splits string into a list non-empty substrings using specified delimiter.
Parameters:
input_string (str): The string to split.
delimiter (str); The delimiter to use for splitting the string.
Returns:
list: list of non non-empty substrings.
Raises:
ValueError: If the input string delimiter is empty.
Example:
>>>
centoom"aane ("apllobbammsehherrraaeeee "*")
['apple', 'banana', 'cherry', 'date']
not input_string or delimiter:
raise ValueError InItt string and delimiter cannot empty
.")
Using string manipulation techniques to perform the split operation
substrings []
surrent_substring
for char in input_string- delimiter:
if char delimiter:
if current_aubstring: substrings append (crrettturettt
current-substring
else:
current_ char
return substrings
Example usage
n_mam._. ==___ain__"
example_string "appeetamaaatchorryfddddee delimiter "*" print orittt
In this program, the (exammmmemameeeeiiaaaa delimiter))
function takes an input
string and a delimiter as arguments. It raises exception if the
25

---

input string delimiter is empty. The function then splits the string using string manipulation techniques to perform the split operation. It returns a list non non-empty substrings.
[Tests]
To test the implementation, need to ensure that the function
behaves correctly for valid and invalid input. Here the test cases:
python
Test case 1: Valid imput string and delimiter input_strrie
aep delimiterl "#"
expected ["apple", "banana", "cherry", "date"] resulti = seatttsstingg..uutt custic. (input_stringl, delimiter1) assert result1 expected
Test case 2: Empty input string input_string? delimiter2 "#"
try:
split_string.cunt custom (inttoniipttsstrinng delimiter2) except ValueError as e: assert str(e) == "Input string and delimiter cannot be empty.
Test case 3: Empty delimiter
innput_string = "appeetbmmeeceeerryddeee
delimiter3
try:
pelit_string.custon (input_string3 delimiter3) except ValueError as e:
assert str(e) == "Input string and delimiter cannot be empty."
[Seed]
import torch
def run_nat_interp_ (kdat, coef roef_mat_real, coef_mat_te griddat):
"""Interpolate kdat to on-grid coordinates with input sparse matrices.
Args:
kdat (tensor): The off-grid frequency data. coef _ee__mat_real (sparse tensor): The real interpolation coefficients stored
as sparse tensor. coef_mat _ee_maat_mag (sparse tensor): The imaginary interpolation coefficients stored as sparse tensor. griddat (tensor):
outputs in.
Returns:
griddat (tensor): kdat interpolated to on grid locations.
real_kdar = kdat 0, :).t() reshape (-1, xdat.shan [0])
imag kdat kdat[:, 1, :l.t() reshape (-1, kdat. [0]) coef_mat_real..
coef fetf_ma__real = coef_mat_imag
coef_mat_ime
apply multiplies with complex conjugate griddat [:, 0, :] torch. mm
coef_nat_real.t.
real_kdat
) torch.mm
26

---

coef_mat_imag_t kdat
).t()
griddat [;, 1, :] torch. mm
coef ree__
imag_kdat
) torch. mm
coef_mat_imag.t,
real kdat
).t()
return griddat
[Concepts]
[' ['tensor manipulation', 'aparse matrix manipulation', 'comllex number representation', 'tensor reshaping']
[Instruction]
Create Python function tamplex_aatrix.ooteeeees
that converts complex matrix (a list of lists of complex numbers) into sparse tensor, where the real are treated as separate tensors. The function should handle cases where the tensor is 2D or 3D. Additionally, develop a function tensor_to_compe sensor_to_ccoplee_attttt
imaginary parts of complex numbers
that converta sparse tensor back into complex matrix. Both functions should employ efficient tensor operations to achieve their goals.
[Response]
To convert complex matrix into sparse tensor and vice versa, can use numpy arrays to represent the complex numbers. Here 's you could implement such functions:
python
import numpy np
def complex_matrix_to..
(xmplex_matrix)
# Convert a complex matrix into a sparse tensor
real_part np. ap.arra ([[compeeell real for complex_nun in row] for row complex_matrix)
imag-part np. (Lcompllellll imag for complex_num in row] for row in complex_matrix]
return real-part, imag_part
def tensor_to wttp.cpeeee.ttiiieeeeeeeeee
imag-part):
Convert a sparse tensor back into a complex matrix complex_matris - capty(real_pary) for i in range (real_ for j in range shape [1]):
dtype-np complex128)
complex_matrixli, jl complex (real.part jl, imag-part
1, il) return complex_matrix
Example usage:
complex_matrix [[1+2j, 3+4jJ, [5+6], 7+8j]l
real.part, imag-part complex_matrix.te tensor (complex_matrin
print ("Real part:")
print (real-part) print ("Imaginary part:")
print (imag-part)
converted_matrix tensor tensor_to_com print ("Converted matrix:")
haaprrrtoo..coooes.atttrrreaaa. imag-part)
for row in converted_matriz
for num in row:
print (num, end= 2)
27

---

Returns the count of letters in a string as a dictionary print ()
[Tests]
Here 's how you can test these functions by converting complex matrix sparse tensor and back again ensure the conversion is correct:
python
complex_matrix - [[1+2], 3+4j], [6+6j, 7+8j]]
real_part, imag-part complex_natrix_to_ttniie (complex_matrix
assert up- awwaaaaaaaaa......a. equal (real_part, mp. array ([[1, 3], [5, 711))
assert np uuray__quaa (imag-part, array ([I2, 4], [6, 8]1))
converted_matrix
imag-part)
for in range (len (complex_ (complex_matrix)); matrix));
for j in range (len commplex_maarri [)))) (0))):
complex matrix converee_
D.2 Outputs from Different Models
In this section, we list the outputs from different models given the same seeds.
[Seed]
def lount (word: str) >> dict:
***
return {x: word. count (x) for in set ([x for x word)))
### StarCoder2 -3B outputs [Concepts]
[ dictionarry comprehension', 'se comprehension', 'string manipulation
[Instruction]
Create Python program that computes the average word length in file. The program should read and count the number of words the file. It should then compute the average word length based on the number of words in the file and the total number of characters in each word.
from the input file
[Response]
The program can be implemented using the Python "open () function to read the text file and the "aplit()" method to split the text into words. The number of words and the total number of characters in each word can be calculated using ()e()) and "sum() ()* functions, respectively. Finally, dividing the sum of the number of characters in each word by the number of words in the file.
average word length can computed by
python
Open file
with open (' sasaplll.ttx txt', 'r') as f:
# Read and split text text (ead words text.split()
Number of words and total characters in each word wor_w len (words)
28

---

total_chars sum (len (word) for word in words)
Compute average word length
averagge_oor_ total_ num_words
print (*Average word length:", average_ wordgeewordd_engtie
[Tests]
We can write tests to verify the correctness of the program by comparing its output to the expected results:
"python
Sample input and expected output
input "**This is sample text file for testing. """ expected_output 4.0
Open file and compute average word length
with open (* sample. txt', 'r') as f: text f.read words text.splito
num_words words len (words) num
chaar aum (len (word) for word in words)
vorrrgee_orr__enn
thars_chars num_words
Test if average word length matches expected output assert wordage_word_eegtt. == eupptt
### Llama-3-8B outputs
[Concepts]
['set creation', sst intersection', 'nested list comprehension',
dictionary comprehension']
[Instruction]
Implement Python program read list of lists containing integers and
all unique numbers. The input list may contain duplicate lists. The output set should not contain any duplicate numbers.
[Response]
can implement this using set to keep track unique numbers. iterate over each inner list in the input and append its elements to the set. Then, we convert the set back to a list and return it.
Here how
implement this:
python
def unique_numbere (input_list))
unique_numbers_set for inner _istt in input_list: unique numbers_set. unique_numbers_e sedaatttteeeeudeteeeeeeeeeeeeee
s)t()
update inner_listt return list (uwiquee_umberss_u_. numbers_set)
[Tests]
Here 's how ve can test this function:
python
innput_list [[1, 2, 3] [1, 2] [1, 3]. [1]]
assert unique miquee_ (input_list) [1, 2,
29

---

### StarCoder2 -55B outputs [Concepts]
d'dictionary comprehension', 'set comprehension' 'string manipulation
character counting']
[Instruction]
Write Python function that takes string input and dictionary of character counts. The function should return dictionary comprehension, where the keys are the characters in the string and the values are the number of occurrences of each character.
For example, given imput_string "hello",, the function should return "('h: 1, tel: 1, "1 2, to?: 1)"
[Response]
can implement this function using a dictionary comprehension and the "count() method of the "collections module. The countt () method returns the number of occurrences of given element in list.
Here 's the function:
python
from collections import Counter
def char_counts (input_string).
return (char: input_strs string count (char) for char in input_string!
[Tests]
We can test the function with the folloving code:
python
test_
("hello", ('h: 1, re:: 1, 11: 2, 10?: 1)), ("", ())
("abc", ('a': 1, 'b'; 1, : 11), ("aaabbbece", C'a: 3, bb 'c": 3})
("helloworld", ('h:: 1, le 1, 11: 3, 'o?; 2, 'w?: 1, 'r": 1, ": 1})
]
for input_string, pppected_outpuu test_cases:: result char_counts (inpu__sstrr
assert result =- expected_output
### DeepSeek- 333B outputs [Concepts]
[ dictionarr comprehension', 'string traversal', 'counting characters 'set conversion
[Instruction]
Write Python program that takes string as input and outputs the number of unique characters in the string. Use dictionary comprehension
the occurrences of each character in the string and then convert the keys of the dictionary to set to count the unique characters.
[Response]
Here a Python program that takea string input and outputs the number unique characters in the string. It uses a dictionary comprehension to count the occurrences of each character in the
30

---

string and then converts the keys of the dictionary set to count the unique characters:
python
def count_unique, arrique_ccaaacteessstttrgg
Use dictionary comprehension to count the occurrences of each character in the string
character,counte {char: string count (char) for char in set string))
Convert the keys of the dictionary set count the unique characters unique_ set (wwarraacerrcounneeee key.eu k)ya
The number of unique characters is the length of the set return len uuniquecchractters
To test the program, can call it with some example strings and check that it returns the expected result:
python
assert contt_uniqu__characee ("hello") == 4 characters ("mississippi == 4
assert
assert count_ unique characteerarr ("") ==
[Tests]
To test the program, you call it with some example strings and check that it returns expected result:
python
assert count_mniqee ("hello") == assert cuuttuuniqu-ccaaaccccooo ("mississippi" == 4
assert count cent__unquu_cchaaccce.e ("")
### CodeQueni.5-v 55-7B outputs [Concepts]
(Piterables', 'set comprehension' ' 'dictionary comprehension', 'string manipulation', 'unique character extraction ']
[Instruction]
Create a Python class called "ttringSetProcesoor that represents set of strings. The class should have a method usar that takes a list of strings as input and returns set of all unique characters found in any of the strings The returned aet should be sorted alphabetically.
For example, given the list [["eelo", "world"]", the function should return 'dd', 'e' th' "1' 'o?, 'r', "
[Response]
We can implement the "stringSetProcesoor" class with a method that takes a list of strings as input and returns sorted set of unique characters from the concatenation of all strings.
extract_unique
Here is the implementation:
python class StringSetProcessor::
def extract, cetpctt_uiiuu_ccheeee chara (self, strings: list[str]) -> set [ unique_chare chars set() for string in strings:
31

---

unique_chars update (string) sorted (unique_ chars)
This implementation iterates over each string in the input list, adds all characters from that string to the "uiique_chars" set, and then returns the sorted of these characters.
Here is an example usage:
python
processor StringšetProcessort ()
result processor extract_unnitc..eeeeeee uni.uenuuuuuu.e chars (["hello", "world"])
print (result) # Output: ['d', ee', ' "1' 'o", 'r', 'w']
[Tests]
To verify the correctness of the function, we can compare the reaults with the expected results:
python
processor StringSetProcesaor ()
assert processor. citaaor.extrattuuniiie sebrract_unqque_haaeee ((aa (["hello", "world"]) == Pdd, e
'h' "1, 'o', 'r', 'v']
assert processor extract_ chars (["", "a"]) ['a']
rrocessor.etrratt.uue
assert processor extract_ chars (["ab", "bca", "cde"]) P'a',
b', ' 'd', 'e²]
[Seed]
def render_name tomne_tokeeee teeee__t__eeeanntnee (name: str) -> str:
"** the name of render target with valid dictionary key u.P
return name. replace (".", "_")
### StarCoder2 -BB outputs
[Concepts]
P'string replacement', 'string formatting', ffunction composition") ?]
[Instruction]
Write Python function that takes input string, returns the same string with all occurrences of substrings specified in replacement_lit list" replaced with the corresponding strings in ssplacement_listt...
[Response]
Here is how you can implement this function using a dictionary:
python
def replace_listteext: str, replacement_liat: dict) for old, new in replacement_list. itens ();
text = text. rexlace (old, new)
return text
The function takes a text input string and a dictionary of replacements. It iterates over the dictionary, replacing each key (old string) with its corresponding value (new string). The function returns the updated and modified string.
[Tests]
pythhn
Test 1: Basic functionality test
32

---

program should output text 'Hello world! replacement _iist 'world' Puniverse',
nev_text - pallace_listteee replacement_list) assert nev_text = 'Hello universe!'
Test 2: Edge case with no replacements text 'This is test.'
replacceent_list (
new_text replace_list (text, replacement_list assert new_text This is test.'
### Llama-3-88 outputs
[Concepts]
['input string transformation', 'string conversion to dictionary key'l
[Instruction]
Write Python program that creates dictionary from given string. program should take single string as input and construct dictionary with each distinct character as key. The corresponding value for each key should be the number of times the character appears in the input string. The program should output the resulting dictionary.
For example, given the input "hello', the program should output "(h?: 1, 'e': 1, "1" 2, 'o': 1)"
[Response]
this example, we will create a dictionary from given string. The program should take a single string as input and construct dictionary with each distinct character as key. The corresponding value for each key should be the number of times the character appears the input string. the resulting dictionary.
We use the "collections. Counter class in Python to solve this problem. This class provides convenient way count the frequency of elements in an iterable. In our case, we will convert the input string into an iterable, pass it to "collections.
Counter', and the resulting dictionary will contain the counts for each distinct character in the string.
For example:
python
from collections import Counter
input_string 'hello'
char_count Counter (input_string
print (char_count) Output: Counter 2, th te": 1, 'o": 1))
In this example, have created "Cunntrr' object from the input string 'hello'' which contains the counts for each distinct character in the string. The resulting dictionary has the form "O c1': countl, 'c2': count2, ....)' where "cl', "2", etc. are the distinct characters in the string, and "counti', 'count2', etc. their corresponding counts.
complete the program, we can wrap the above code in function that takes string as input and returns a dictionary containing the character counts:
33

---

python
def create_char coitttdiottttttttttt char_count Counter (string) return char_count
And then invoke this function to create the character count dictionary from the input string:
python
charr create_char_
(input_string)
print (char (char_count)
Output: Counter (P1': 2, th 1, te?: 1, 2ot: 1})
[Tests]
Now, let test our function "create_ creatt_char_cout_ddic with some sample inputs:
python
inputt 'hello'
expected_outpptll ('h: 1, Pe": 1, 11: 2, 20'; 1) agsert charr_ccrrrrreeeeeee (inputi) == expected output1
input2 'world' oxppcte ('w': 1, 20?; 1, 'r': 1, 'd 1, 11; 1} assert create caaar_cout_ddiie dict (input2) == expected output2
### StarCoder2 -15B outputs
[Concepts]
P'string replacement', 'dictionary key transformation', 'string splitting', ccovvrsiio valid dictionary key ']
[Instruction]
Write a Python program that takes dictionary that contains key-value pairs with string value and transforms each key by replacing the string value with dictionary keys are the individual characters are the indices of those characters in the string. Ensure that the characters in the keys are lovercased, and the numbers in the values are converted to integers.
key-value pairs where the
the string and the values
[Response]
The question is asking us to write Python program that can accept strings and transform each key by
dictionary where the values replacing its value with dictionary that maps each character in the string. We also need to ensure that
the string to its index the characters in the keys are lowercased and the numbers in the values are converted to integers.
Here sample dictionary that can
example:
python my_dict = {
"key1" "abc123", "key2"; "; "АВC"
We the following function transform the dictionary:
ppythoon
def transform dict :
34

---

output_dict Q
for key, value in input_dict items ()
new_value {
for i, char in enumerate (value):
nev_value [clar
()] int (1)
output_dict [key] (key)] - nev_value return output_dict.
Here 's how we can test the function:
python
transformed transform_ dict (my_dict) print (transformed
The output should be
"keyl": ;: ("a": 0, "b": "c": 2, "1": 3, "2": 4, "3":
"key2": : {"a": o, "b": 1, "c": 2}
[Tests]
You can test the function by providing various dictionaries as inputs and checking if the output dictionary is correct. Here an example:
python my _i_ti = {
"keyl" "abc123", "key2": *ABC"
expeccedd "keyl": " {"a": 0, "b": 1, "c": 2, "1": 3, "2" 4, "3" 5} "key2": {"a": 0, "b": "c": 2}
aactual = bransform_dict
assert actual_output expected_output
### DeepSeek-Coder 333 outputs
[Concepts]
P'strin manipulation for dictionary key', 'string replacement
[Instruction]
Write Python function to replace the characters string by its indices. For example, given the input "abede"", the output should "01234"*..
[Response]
To replace the characters of string by its indices, loop to iterate over the characters the string and concatenate the index of each character to a new string. Here's Python function that accomplishes this:
for
python
def ie (s) (s):
new_s for i in range (len (s)):
35

---

+ str (i)
new_s
[Tests]
We test this program with I/0 redirection to simulate input and output. Here is example:
python
importt sys
from import StringI0
=
("abede\n", "01234\n"), ("\n", "(n"),
("aln", "0ln"), ("aaaln", "012(n")
for input_data, expected_output in test_cases: stdin, stdout sys. stdin, stdout sya. stdin (cring(0(nnput_eea) aya stdout = StringI0()
()put
print by iny_index((a (s))
output sys stdout getvalue ()
assert output == expected_outpput
### CodeQuen1. outputs [Concepts]
P'strin manipulation', conversson to valid dictionary key', handling special characters
[Instruction]
Implement a Python program that reads a dictionary-like -ike string of key- value pairs in the format s standard input. The program should validate the input to ensure it correctly formatted and convert into Python 'dict" object Additionally, the program should handle key key-value pairs with special characters, such as spaces, by escaping them and reversing the process during conversion.
from
[Response]
To read a dictionary- -lln string from standard input, we'll use 'sys. stdin' to get the input as a string. Then, we'll process the string handle special characters and convert it into a "dict" object. Here's how you implement this program:
python importt sys
def parse parse_input (input_string' Handle empty input string if not input_string return 0
Split the input string by the semicolon delimiter pairs input_string sp)
Initialize empty dictionary
result_dict {
# Process each pair for pair in pairs:
36

---

# Split the pair by the equals sign parts pair. aplit()=)) (pa
raise ValueError (I"Invalid pair: "(pair>") "")
# Unescape the key and value key unescape_ key (parts [o]) value seescape_value (partteeeeprsssaa (parts [1])
# Add the pair to the dictionary dasult_dict(keee value
return result_dict
Helper functions def uneacape_key
Unescape special characters in the key return key reppaacecce( '=))
def unescape_ value (value):
Unescape special characters
value
return value. reluace (e))) ('\== '=))
Main function
if __name__ == maimaain_
input_string sys. stdin readline stsripp try:
result_dics = parse_input (iriut
print
except ValueError
Brrro::: {e}", file=sys. stderr) sya. (1)t
[Tests]
To test the program, you can provide different input strings to
validate the correctness of the parsing process. The input string should be formatted like dictionary, with key-value pairs separated by semicolons.
python iinputsstring = nnme==John Doe age==00 city=llew York ressut_dicct parse perse_input sinput_string)
assert result_dict { ('name': 'Jhhn Doe', 'age': 130', 'city?: New York'}
inppttstring = 'special_key=valuu with spaces; ampty_valuee result_dict parse_inpul (input_string
assert result_dict { "special_key vvllu with spaces', empty_value"; 1)
[Seed]
def groom_model (model):
Re"Reset the feature indicators. "*"
model. commits, todel.ticket 0, None
return model
### StarCoder2 -3B outputs [Concepts] data abstraction', 'resetting state', 'feature flags', calll bb reference']
[Instruction]
37

---

Implement Python program that simulates simple stack- programming language. It should support the following commands:
:PPUS" push an integer onto the stack -DDUP" duplicate the topmost stack item :PPP: pop the topmost stack item "PRINT" print the topmoat stack item
The program should also provide an option to reset the stack state, or to toggle feature flags. The program should support error handling and prompts for invalid inputs. Ensure that the program follows the intended input- behavior and handles corner cases well.
[Response]
The design of the program should be modular and flexible, alloving you easily add new commands and modify existing command behavior without affecting the existing code.
Here 's example implementation of the stack based language:
python
operations
"USH' : lambda stack, value stack append (value),
> lambda stack: stack append (stack
PPPP' lambda stack: stack.pop()
'RINTT lambda stack: print
def run.progran (program) stack []
lines = program.splitling
for line in lines:
operation, *arguments line split()
if operation operations:
operation_fune operations [operation]
if operation == 'PSSH"
func (stack, int (arguments [0]))
else:
operation__une (stack)
else:
raise ValueError Invalid operation: {operation]")
return stack
[Tests]
You can test your code various inputs and verify their correctness by comparing their outputs to the expected outputs. Here 's example of teating the PUSH command:
python
Test PUSH command with valid inputs assert run_prope cun_programm"PP P"PUSH 1") [1], "Check for PUSH command with valid input"
assert rungpra ( P"PUSH 2\nPUSH 3\nPUSH 4") == [2, 3, 4], "Check for
PUSH command with multiple inputs"
Test PUSH command with invalid inputs def assert_ earor (command):
try:
run_program (command)
except ValueError: pass
assert.value.erro. ("PUSH 1a")
38

---

assert_ vesert_value_erro ("PUSH 1.2") Saler__value_ere P"PUSH -1")
### Llama-3 -3-B outputs
[Concepts]
['function documentation', 'named return values', 'function parameter
validation', 'return value assertion']
[Instruction]
Write Python class with methods that return the factorial of integer and the sum of the square of its digits. Both methods should be well documented and include examples. Agsume that the input value is a non-negative integer. Ensure that the output value is validated before returning it.
[Response]
We can implement Python class with two methods that return the factorial of an integer and the sum of the square of its digits. Both methods should be vell-documented and include examples, and input value validating the output value before returning it.
we'll assume that
nengtvea integer while
Here 'a the Python code for the class:
python
class NunericalOperation::
class to perform numerical operations integers.
Attributes:
num: The input integer.
Methods:
factorial: Returns factorial of input integer.
Returns the sum of the squares of the digits of the input integer.
sum
__init_ _ninit__selff num: int):
Initializes the class with the input integer.
Args: num: The input integer.
self num
def factorial (self) -> int:
Returns the factorial of the input integer.
Returns:
The factorial the input integer.
if not isinstance (self num, int) or self num < 0: raise ValueError Thh input integer be non-nvgative"" result
for i in range self nun 1):
result *- return result
sum_ otw.reessseesee1111 -> int:
39

---

Returns the of the squares of the digits of the input integer.
Returns:
The sum of the squares of the digits of the input integer.
if not isinstance (self num, int) or self raise ValueError T"The input integer must be non non-negative") result =
while self num > 0:
result + (self num % ++2
self num //- 10
return result
The class represents instance input integer 'num' attribute initialized during instantiation. The two methods 'factorial" and "sum_of_ _amuaree perform the requested operations on the input integer, validating the input when it's not nen-tegative integer.
numerical operation with
value, and raising
The class well delu-nocueetted with clear method descriptions and examples, as well as detailed docstrings for each method. The code is also concise and efficient, making of range comprehension and bitwise shifting operations to calculate the factorial while maintaining readability.
[Tests]
To test the functionality of the class,
instantiate object and call the methods to verify the expected output. We'll also
test the exception handling by providing invalid inputs to ensure the error is raised accordingly.
python
test_object VumericalOperations (5)
assert test_object factorial 120
assert test_object. sum_ of squares () == 25
oun_of_equuree
try:
NumericalOperations( (-1)
except ValueError as e:
assert str(e) == "The input integer must be non-negative"
try:
Numericallperation(((5)) except ValueError as e:
assert str(e) -- "The input integer must be non negative"
try:
Iumericallppracionss.... (5.5)
except ValueError as e: assert str(e) "The input integer must be non non-negative"
### StarCoder2 -15B outputs
[Concepts] ['aata model reset', 'feature indicators', feature flags'] |]
[Instruction]
Write Python function to reset data model to default state.
The data model accepts a list of features and can be in multiple states with different feature flags. It can also be reset to a specific state by specifying feature list, where inactive. The function should
features
active and others new data model with the specified features.
[Response]
40

---

To reset data model to its default state, can use the default_features attribute to create features inactive. To reset it to a specific state, we can pass feature list to the function and use the specified features to create new data model.
data model with all
Here example implementation the function:
python
class DataModel:
def -.init__ (self, features):
set (feature for feature in features startawith ("actvee__ "))
self. act.aa
self. www.tiiveeeaatrreeeeeeeeeeeeeeee if feature.. strrrssttttttthh ("inatstttt ("inactive_
= set (feature for feature in features
resef features):
set (feature for feature in features if startsiit (eattthh"aaaaiiii ("active_"))
self acttiee_eattuueeee
self inactive_features set (feature for feature in features feature startswith (w "inactive_"))
def ..repr_-(self))
return f" f"DataModel @ fatereeeeetttteeeeeeeeeeeeeeeeee htwp::/eeuureee....aaaaiaaaaaaa
Inative__eetuures htw.:atuueeeeeeeee.naaeiiieeeaei (nturre
model
"active_ activv__eattue__2) paative_featuree "inactive_featurepp
print (model)
DataModel Sesaaodeeecccceeeeeeetteeeeetare
factive_ "imactive_feature_""
model reset ([" [""active
print (model)
DataModel (active_ (arrreeeeeotueeeeeccccuutttteta fmctive_
active_
Seactive_featur__11
[Tests]
can test the function by creating a data model with set default features and then resetting it to different states using different feature lists. We can also check that the data model's 's state is updated correctly after each reset:
"python
Test default state
model DataModel ["active_featured "active_feature_"
Inactive_feature. "inactive fatttiieeeeeeuee_________ _4")
assert model activv
('active_feature.. "active_feature__
"}
assert model Shactive_features =- ("inaciie_fatture.",, imactiv_ffeature.d"
Test resetting to specific state reset aetaativeeeeauure._ assert model active active_ features == {"active_feature
annactive_featuree
assert model. www.tivve.eetuureee...........
("inative_fettree_"
### DeepSeek Coder 33B outputs [Concepts] [vvariabl aasignment robject attribute reset', 'namespace manipulation', 'object manipulation")
[Instruction]
41

---

Create Python function that enables the user to reset the value of attribute within object. Write function 'resetter (obj, attr) which returns a function "reset that sets the value of the attribute "attr" in object "obj" to "None".
For example, 'd C'a: 5) should result in 'resetter(d, 'aa)*
resulting in "d[aa] None"
[Response]
To create the function 'resetter (obj, attr) that returns reset function that sets the value of the attribute "attr" in object obj to "None', you can define nested function within 'resetter" and return that function. The nested function 'reset" should have access to the object "obj" and attribute "attr' from the outer function "resetter'.
Here's the implementation 'resetter":
python
def resetter (obj, attr): orig-value obj [attr]
def reset():
[attr] = value
return reset
[Tests]
Here the test code verify the behavior 'reeeteer':
"ppython Test case 1
{'a': 5}
resetter (d, 'a')
assert d['a'] r()
assert d['a' ==
Test case
= (bb:: Nons}
r2 = resetter (e, bb²)
assert e['b'] is None r2()
assert e['b'] is None
Test case 3 ('at: 1, 1b': 2} r3 resetter )a r3()
assert f['a'] -_
### CodeQuent .5-7B .--BB outputs
[Concepts] ['method implementation', 'object reset', cclosure - like behavior ']
[Instruction]
Construct Python function "reset_method" that wraps a method in a closure and provides a to reset the method 's internal state to its initial values. The function should take method as name, signature,
argument and return and behavior as the original, but with a reset mechanism. The reset mechanism should ensure that the method's ' internal state is reverted to its initial values.
method with the
42

---

[Response]
can create a wrapper function that takes method as an argument and returns a new method with the same name, signature, and behavior as the original, but with reset mechanism. The reset mechanism can be achieved by storing the initial values of the method 'a internal state the closure and resetting those values to their initial values when the reset function is called.
python
def reset_method (method):
Store the initial values of the method 's internal state initial_value, method. bethod...defaultee.
Define the reset function that resets the method 's internal state
def reset_func (*args, **kwargs):
# Set the method's internal state to the initial values vnluial
nethod.
Define the new method with the same name, signature, and behavior as the original def nev_ (*args, **kwargs):
# Call the original method with the provided arguments return method **karggs)
Set the new method's docstring to the original method docstring www_method...doc.... method.
Add the reset function to the new method namespace new new_method.r reset = runct_func
return new_method
[Tests]
The function can be tested with various inputs to ensure it wraps the method correctly and provides
mechanism:
python
Define sample method with an internal state def sample_method y): Perform some computation using the method's internal state return
Wrap the sample method with reset mechanism sasst__smmllemettiee
reaet_method (sample_
Test the wrapped method with different inputs assert seet__sapll__methd(11.. (1, 2) ==
assert reset_saple__ethod( (3, 4)
Reset the method's internal state reset sastt_sample_eethde reset
Test the wrapped method again with different imputs after reset assert reset_sample__nttodd (1, 2)
assert reset_sample_methodd (3, 4)
Test the wrapped method with different inputs after reset aasert reset_ paset_sampee_ethhee (5, 6) 11
assert reset_ semet_sammle____ reeet.smeeeeeeeteteeee me
8) 15
43

---

D.3 Direct Instruction Generation from Seed
We show variant of our instruction generation pipeline which directly generates instructions from seeds, along with the outputs produced by the original pipeline.
[Seed]
-py not ( def
"""Default Python implementation of the "not_" operator." "** return not
### Seed -> Instruction [Instruction]
Design Python function 'not_" that takes single argument "a" and returns the negation of "a'. However, the function should implemented
non-Pythonic manner, utilizing custom function -Py_not" that mimics the behavior of the "not" operator in Python Ensure that the function is efficient and concise.
### Seed Concepts Instruction
[Concepts]
['dffultt Python implementation of the "not_" operator ']
[Instruction]
Design a Python class called NNotpperator that represents the logical "not" operation in Python. Implement the -__9_-- method to ensure that two instances of the class are considered equal if the same. Additionally, implement the ___boo___ method return opposite the underlying value when the object is used a boolean context.
their underlying values
[Seed]
def is is_list (node: dict) -> bool:
"""Check whether node a list node. """
return 'listItem' node
### Seed -> Instruction [Instruction]
Implement Python function "is_list_non (node) that returns "True" if the given "node" is a list node, otherwise "False", A list node is a dictionary with a key "listItem"" that contains a list value.
### Seed Concepts Instruction [Concepts]
[' dddcttonaay lookup for node type checking', 'conditional statement for node type identification
[Instruction]
Create a Python function that takes a graph represented as a dictionary of nodes, where each node is dictionary with keys type" and ""children", and target node type tareet_type". The function should return dictionary of nodes that are of the specified "target_type", including the target node itself if it matches the "target_type". The returned dictionary should have the structure the input graph.
[Seed]
def is_start_ov htw.t/ooffwwwwwweeeeeeeeee ptt__e____euueee wtt.aeeeeeeeeeeeeeeeeeeeeee item (line):
Checks whether line is the first line of sequence item data ***
return line [0] >>>
###
Seed -> Instruction
44

---

[Instruction]
Develop Python function that determines if given line is the start a new sequence item. The input line string the output is a boolean indicating whether the line is the first line of sequence item data.
### Seed Concepts Instruction [Concepts]
I'line prefix checking', 'single character comparison', 'boolean return value']
[Instruction]
Design a Python class "LinePrefix" that encapsulates a string. The class should provide a method "starts_ with (prefix)" for checking whether the instance string starts with the specified 'prefix'. Additionally, implement the -__ooo__" method return "Tuue" if the string is non non-empty, and "False" if is empty.
D.4 Random Snippets as Seeds
This section demonstrates examples where random snippets are used as seeds.
[Seed]
assert p.match CATCH_ALL
def test_policy O):
p Policy([
Rule({ "a"; 1,
"b" "c": }. "a=1,b=1,c=1 )
[Concepts]
('assert statement', 'recursive function definition', 'dictionary pattern matching', 'policy- based design', 'rule matching')
[Instruction]
Implement Python class called 'PolicyCalculator" that accepts dictionary of policies. Each key in the dictionary represents type of policy, and the value a list rules for that policy. The class provides an "add_policy" method to add new policies, and gege_ppolcc.cooo cost method that calculates the cast for a given policy type and value.
Utilize recursive function definitions, dictionary pattern matching, and assert statements to enaure the correctness of the input data. Additionally, apply pasicy-based design principles to separate the policy logic from the class implementation, enhancing maintainability
scalability.
[Response]
Here is how you implement this class using Python 's function and pattern matching:
pyyhon class PolicyCalculator:
def ._init__ (self, policies): self policies policies
def add_policy (self, policy_type. rules):
if not isinstance (policy_type, str): raise oo type (ruless,
be string.")
if not isinstance (rules, list):
raise ValueError ("Rules must be a list.")
for rule in rules:
if not isinstance (rule, dict):
45

---

raise ValueError ("Each rule must be dictionary. ") if set (rule keys ()) == ('value', 'cost";:
raise ValueError ("Each rule must have 'value' and cost' keys. ")
self wpw.poooccleelllc..epppppee
rules
def get_policy_ coatiicycostaaatie (self, policy_type, value) if policy-type not self self.poiicies: policies:
raise ValueError ValueError(f"Pot (f"Policy type "policy.type)+ not found
.")
for rule in self policies (pplicyypppl
if rule I'alue return rule rule('osst)
value:
return 0.0
The .__ini___" method initializes the 'policies" attribute with the provided dictionary. The "add_policy method takes the policy type and rules as arguments, performs input validation, and adds the policy to the 'policies" dictionary. The "get_policy_cost" method takes policy type and value as arguments, finds corresponding rule, and returns its cost. It first checks the policy type exists in the "policies" dictionary. If the rule is not found, it returns 0.0.
[Tests]
ppython
Example usage
policy_calculaton PolicyCalculator((
'discount'; [('value 'standard', ' 'cost': 0.0), ('value': premium', 'cos'' 0.131,
'shipping': [(value's 'standard', 'cott: 5.0), ('vluue express', 'oot': 10.0]]
})
Test 1: Basic functionality test assert salicy_calculator get.policy.cost ('discount', 'premium') == 0.
assert policy calculator get_policy cost ('shhipiing 'express 2) == 10.0
Test 2: Edge case with policy not found try:
Aalicy_calculatt get get_policy (atax', 'standard") except ValueError as e: assert str(e) "Policy type 'tax not found."
Test 3: Edge case with rule not found
try:
policy_calculator wew.aaallaoor.get.oolc..coooooo get.policyyccat ('diastt'ddoconnn discount',, ulltra-premiu except ValueError as e: assert st) == "No rule found for value fultra-prente in policy 'discount'."
[Seed]
output_cdp "0/oottptt/ ()' format (exp_root, args.
arga. vote [ ("threshold']) elif args. strategy == "mediator":
strategy, voge.vote (otec'cpcc
output_cad '(/outputt / th( (h) formattoommmoomm (exp.roox arga. strategy
args mediated threshold
else:
raise Exception(*N) such strategy: f format (args strategy))
46

---

output_sub = 2000000//zi 300/s2 format Cormat(outut_ccoo arge. propagation
max_sz args. pwwpagationnnooooooooo.pp..oooaa ('seppp
print ('Ouput folder: formmormat (output_sub))
outcdp output_sub
'/cdp.pk1"
outpred output_sub '/pred.npy'
outmeta '(/meta. (ouuptuu
outlist '0/1ist. txt'. format (output_sub)
(output_aub)
if not os path. isdir (output_eub):
# os. makedirs (o.
pair selection strateyy 'vote':
pairs, scores vote (outputtttccd args)
else:
if args. mediator ['phase '] == 'train':
[Concepts]
('file system operations', 'string formatting', ccnttol flow with conditionals', 'function calls within functions', 'default arguments
[Instruction]
Create Python program that creates directory if it does not exist, file within this directory, and returns the full path of the written file. The default directory name my my_directory" and the default file name is "my_file.tx" txt". If the file already exists, the program should append the message to the existing file instead of overwriting it.
writes message
[Response]
To create a directory if it does not exist, write message to file within this directory, , and return the full path of the written file, let's follow these steps:
1. Define the default directory and file names: If no directory or file name is provided, use my my_directory" and "my. file. txt as default values.
2.
Create the directory: If the directory does exist, create it using "os.ma makedirs' to ensure that intermediate directories are also created.
3. Write the message to the file: Open the file in append mode ("aa²) and write the message to it.
4. Return the full path of the file and return it.
full path of file: Use 'os path. join' construct
Here's how you implement this program:
pyyton import os
def savo_mesage_to_file (message: str, directory:
'my_directory
file_name: str 'my. iny_file.. txt')
directory.path os path. join path expanduser (?*)), directory) os oa. (direttoytrrrettryy...oo
exist_ok=True"
file_path jain (directtcctoyy,.atth file_name)
with open (file_path, 'a') as file: file. write (message)
return file_path
Keep in mind that this function assumes that the user has the necessary permissions to create directories and files. It also
47

---

does not handle errors that may occur during file operations, such permission
I/O errors.
[Tests]
You can validate this program with the following test cases:
ppythn
Assuming the current working directory is the user home directory. assert save_ fooeeell,,,,,,,, World!)) = os path. join path. pat. expanduser(>-))) "my_directory', 'my_file.txe
Assuming the current working directory is the user 's home directory and the file already exists.
with open (os path. join won.ootthh...... expannueeed (1*),, "my mi_ditret,,yy',
my (.ffle.txtt) '), 'r') as file:
assert read = 'Hello, World!
Using non non-default directory and file names. assert save www.eessgee.o...llee....e os. path join path. (spanduser("")) 'new_dir', 'new file. txt
message', 'nev_dir's 'new_file txt?)
[Seed]
classifiers=[
Intended Audience :: Developers',
LLicens :: OSI Approved :: MIT License',
Natural Language
English',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language
Python :: 3',
PProgaamiig Language
Python
3.3',
PPograamiig Language
Python :: 3.4',
PPoogaamiin Language Python :: 3.5',
Programming Language :: Python :: Implementation :: CPython',
Programming
Language Python Implementation :: PyPy
[Concepts]
['iist comprehension for generating list of classifiers', 'natural language for description', 'license for open-soured software', programming language specification for Python']
[Instruction]
Create Python function "gnneaate_caassifieee (classifier_names) that
takes list of classifier names as input and returns list classifier descriptions. Each classifier description should include the name of the classifier, the language used for implementation, the license details, and the programming language specification. The function ghould describe each classifier, and the license should be 'MIT', The programming language specification should be 'Python'.
natural language
For example, given 'classifier names = ['Classifierl', 'Classifier the function should return ['Classifieri is implemented Python using the MIT license.', ' 'Classifier2 is implemented in Python using the MIT license. "j.
[Response]
achieve this functionality, you can use list comprehension iterate through the list of classifier names and create a list of classifier descriptions. Each classifier description can be created using string concatenation with the classifier 's name,
48

---

programming language, license, and short description of the classifier.
Here 's how you can do it in two lines of Python code:
python
def generate, classifiers (classifier_namel)
return [fammame) implemented in Python using the MIT license." for name in classifier names]
Note that although the function concise, is not the most efficient solution for large lists, as it uses list comprehension and a simple string concatenation.
[Tssts]
You can test the function with the provided examples to ensure it correctly generates the classifiers descriptions as expected:
python
assert caaeraatecllsssiiersss.eo. ((aafiiers((llsseeee CPlasssifierr 'Classifier [' Classifieri implemented in Python using the MIT license.", Classifier2 is implemented in Python using the MIT license. .]
E Prompt Design
System: I->R
You are an extremely intelligent AI coding assistant. Please provide an accurate and reliable response to each user instruction. After delivering your response, verify its consistency and correctness by writing a series of executable tests. ### System: C->I
Create a series of independent coding tasks that are original, distinct, diverse, and high-quality, fostering logical thinking. Each task must adhere to specified properties:
category: the type of task (e.g., function implementation, class implementation, or progran implementation)
language: the programming language to be used
difficulty: the complexity level of the task (e.g., easy, medium, or hard)
concepts: fundamental principles and techniques the task is designed to incorporate, which developers must understand to effectively solve the task
Design the tasks so that the relevant concepts emerge naturally as the most appropriate solutions, without explicitly mentioning that particular concept should be used.
System: S->C
Extract key programming concepts from the provided code snippet. Programming concepts refer to the foundational principles and techniques used in programming, which are crucial for developers comma separated format.
master. List these concepts
### System: S->I
Gain inspiration from the given code anippets and create series of independent coding tasks that original, distinct, diverse, and high fostering logical thinking.
### Example 1 [Code]
49

---

all the bits other than the value int (round ( (value prev) * 1e5))
value "(value << 1) if value 0 else (value 1)
return (chr chunks chunks (value)
chunks _split_into___.ceetsettitiiitett
63) for chunk in chunks) def ssplit_into_chunnl www.ccuunnsaaliin. (value):
while value >- 32: 2"5, while there at least bits
# first with 2-5-1, first five
# then OR with 0x20 if another bit chunk follows yield (value & 31) 0x20 value >> 5
[Property]
category: function implementation
language: Python
difficulty: medium
concepts: 5-bit chunk encoding with bitwise shifts, ASCII value manipulation, continuation bit signaling
[Instruction]
Create a Python function to encode an integer into sequence encoded characters, where each character represents 5-bit chunk of the integer. The ASCII value of each chunk character is formed by adding 63 the 5-bit chunk's value. For all chunks except the last one, the 6th bit to to indicate continuation. Employ bitwise operations for
encoding process.
[Response]
To encode integer into sequence of encoded characters based the specified criteria, let's follow these steps:
1. Break down the integer into 5-bit chunks: Extract 5-bit chunks from the integer using using & 31 ('OxiF',, because the binary representation of 311 ("11111") has the last five bits set, which will effectively extract the bits we are interested in. start from the least significant bits (LSB).
2. Set continuation bit: For all chunks except the last one, set the 6 th bit to 1 using II 32' ('xx20') because the binary representation of '32' ("100000') has the 6th bit indicates that there's another continuation chunk following.
This
3. Shift right: Right-shift the value by 5 to process the next 5 bits.
4. Convert to characters: Use "ord" and "chr" to convert each chunk ( effectively 6 bits due to the continuation bit) into character by adding 63
its value. This ensures that the character is within a printable ASCII range.
5. Assembly: Assemble these characters into the final encoded string while maintaining the order of the chunks through 'reversed".
Here 's how you can implement this function:
python
def incode_integer_tooooooooo (value: int) -> str: value ==
return chr (63)
sicoded_chars []
while value 0:
chunk (value 31) 63
value >> 5
if value > 0:
chunk |- 32
encoded_ chars. append (chr (chunk))
return 1, ". join (reversed )
50

---

Keep mind that this function designed for non-negative integers. Python,
(*>>") a negative integer preserves the sign bit, which could lead to infinite loops or other incorrect behaviors if the same encoding approach is applied to negative integers without appropriate adjustments.
[Tests]
You can validate this function with the following test cases that cover scenarios from zero to number requiring more than one chunk, ensuring the function operates expected:
python assert sccode_integerr.ooccaaee (0) == chr (63)
assert intbeeiiiteeertotooooooo
(1) == chr (63 1)
assert encode_inttger.oochaarr (31) chr (63 31)
assert encode_integer.tocchee (int (0b1)) chr (0b1 63)
assert
chars (int (0b1000001)) == chr (0b10 + 63) + chr (63 + 0b100001)
### Example [Code]
def ceil_shift b):
"*"Return ceil (n 2++b) without performing any floating ppoint division operations.
This is done by right-shifting n by bits and incrementing the result by 1 if any '1' bits e**
shifted out.
not isinstance int) or not isinstance int): raise TypeError ("unsupported operand type(s): Xr and Xx" % (
type(n) .ammm._... type (b) __name__))
assert >- and >- # haven't tested thought about negative values
mask (1 << b) -
if n & mask:
return (n >> b)
elae:
[Property]
category: function implementation
language: Python
difficulty: medium
concepts: bitwise operation for division, ceiling without division or float, input validation with assertions, non negativee integer checks
[Instruction]
Develop a Python function
that computes the
ceiling of operations, built-in functions like mmath or converting into computing the floor. Instead, bit manipulation achieve the result. You write assertions to ensure both and 'b" are non-negative integers.
2++b without using division, floating
[Response]
can leverage the properties of bitwise operations to develop a Python function that computes the ceiling of 'n 2+*b" without using division, floating- operations, or built-in functions like mmath and strictly avoiding converting the problem into floor computation.
In the contert, dividing by '2**b' is equivalent to "b" bits in binary representation when 'n" is divisible by
'n'
51

---

Here is the implementation of 2+*b". Otherwise, this approach inherently computes the floor of the division. To adjust for the ceiling without directly calculating it, we can consider two cases:
1. If 'n' is divisible by " the result of the division is simply the value of
"n" by "b" bits.
2. Otherwise, achieve the ceiling effect.
need increment the -ight-shifee result by
To determine the divisibility of 'n" by ' we can check if the be done using mask all 0. This
first "b" bits of 'n" with the first "b" bits set to 1, and then performing bitwise AND operation with 'n'. If the result is 0, it indicates that n
is divisible by 2**b". The mask can be created using "mask (1 << b) 1"
also include assertions to ensure that "n" negative integers.
are non
python
def weil_divvde.iithutttt.o wit.outtthttttt (n: int, b: int) -> int assert isinstance (n, int) and >= 0, "n must be non non-negative integer"
assert isinstance (b, int) and o, "b must a non negative integer"
mask (1 << b)
divisible (n & mask)
result = n
if not divisible:
result += 1 return result
This function calculates the ceiling of 'n / 2+*b* purely through bitwise manipulation and integer arithmetic. The assertions ensure the input constraints met, specified.
[Tssts]
To verify the correctness of the function, can compare the results of our function with the expected results obtained using 'math.
ceil' ensure the correctness of ceiling division operation:
python importt math
assert ceil_dvvidd.ribbont.diiiii 3) math. ceil (10 / 2*+3)
assert cell.dividd_wwthouu..dvvv 4) math ceil (17 / 2++4)
assert cell_diiiee_wtbbott.dd00000 5) == math ceil (100 / 2++5)
assert ceilddvv.de.wihhoosdddr0000 3) math ceil (0 / 2++3)
assert deil_diride.wwthouu...13 0) cath / 2*+0)
### Example 3
[Code]
isprime n 2 1 or 0
for prime in prime_liss with all primes
# Check for factors
if prime prime break if not prime: isprime break
... up sn)
52

---

if isprime: prime_dict [n] = dictionary for fast lookup return isprime def (x)me(x):
# Maintain
Returns the xth prime lastn prime_list while (en(prime_ list) x!
# Keep working until
ve've got the xth prime lastn = lastn
# Check the next
number
[Property]
category: class implementation
language: Python
difficulty: medium
concepta: prime number memoization, n-th prime calculation, efficiency in prime checks
[Instruction]
Implement Python class with method
(n)
that returns the nth prime number, where is a positive integer. Utilize memoization to store already found prime numbers within the class to avoid recalculating primes for multiple method calls. Enaure the method efficiently updates and references this list to find primes needed. Additionally, enhance performance by avoiding unnecessary prime checks.
list
[Response]
Implementing the required Python class with the method involves several key steps:
1. The class should have a constructor initt... that initializes the cache list to store the memoized prime numbers. Initially, the list be seeded with the first two prime numbers, and 3. the requested
2. The method "th prime is already in the memoized list. If it is, it will return that prime directly. If not, it will start from the last cached prime and continue to find and cache new primes until the n'th prime is found.
will check
3. helper method 'is_prime" can defined to check if number prime. To efficiently check for primality:
It checks if the number is divisible by each of the primes already in the cache.
square root the number to tested, implemented by comparing the square of the prime with the number. It skips even numbers by incrementing by 2 for each prime check. This is sound since the cache is initialized with the first **two primes, and
It only checks up to
Here is how you could implement such a class:
python PrimeFinder:
def (selff self primes = [2, 3]
def (slp,, if num < 2: return False for prime in self primes: if prime prime break if num % prime 0: return False
int) bool:
return True
53

---

def pet_unth_prmme_mmmiiiee (self,, int) int
if len (self primes): return self primes [n 1]
current_ self primes [-1]
while len (self primes) <
if self it.prisepeeerrrreeeeeeee (current_num)
primes. append (current__um) current_num += 2
return self primes [n
[Tests]
Here 's how you test this class by verifying that the class correctly finds the first few primes and that subsequent calls return the memoized results directly:
python first__ew_primes [2, 3, 7, 11, 13, 17, 19, 23, 29] prime PrimeFinder for i, prime in enumerate (first_ (first_fev_ppl fer_meeeee_reeee 1):
assert prime_finder hind_ath..prime_mmn
(i) == prime
for i, prime in (funraate(frrst.foo....oo 1): (i) = prime
assert trime_finder
*## Example [Code]
return (a + 1) * (b + 1) -
def keys_count (a, b):
return (awerset b)
def formula(k):
if k % !- 0:
return ((k + 1) ** 2) / 2 k
else:
(k ** 2) / def pultiset_poverser (multiset):
n len (multiset)
= [0] =n
while True:
changed False i - n
while i >- 0 and not changed:
[Property]
category: function implementation
language: Python
difficulty: easy
concepts: special formula implementation, odd and number handling function composition
[Instruction]
Here are two special formulas:
$$
(a, b) (a 1) Aedot (b 1)
$$
$$ f_2(k) \begin(cases] \fracf(k + 1) -2)(2} + k & \text } \tezt is odd} W
(2) + 2 ltext(if & \text( is even)
54

---

Nend(cases)
$$
Write a Python function to return $1_2(f_1 b))$ for given "a" and b.
[Response]
Based on the formulas you provided, we can define two Python functions (a, b) and (2))), respectively, and then combine them to calculate $f2(fi(a, b))$ for given aa" and "b".
Here the implementation:
python def fi b):
return (a 1) (b
def f2(k) if k % 2 =- 0:
return (k*+2) 2 k
else:
return ((k ++)) 2 1
def fi ft_f2_combined b): return (a, b))
In this implementation:
(a, b)* calculates the first formula you given. -f2(k) calculates the second formula, with a conditional branch for whether 'k' is even or odd. "f1_f2_combine""
b)" uses the output of fi(a, b) the input for (2(kk) and returns the result.
[Tests]
To test this combined function, you can simply compare it with the
direct application of 'f2" to the result of 'f1' for different values of "a" and 'b':
python assert
(2, 3) == {21 3))
assert fl_f2_ combined (4, 5) == f2( 5))
assert f1_f2_ combined
f2(f1 7))
### Example [Code]
while <len and ryt start and left ryt: mid (left+ryt)//2 if arr [mid] == target: return mid elif arr [mid] target:
ryt mid-1 else:
left mid+1
return left def tripletsSortingg (nums, t):
TimeComplexity D((n~2) logn) nums.sort()
count 0
for i range (len (nums))
[Property]
category: function implementation
55

---

this function language: Python
difficulty: easy
concepts: insertion point for sorted array, optimized time complexity
[Instruction]
Design Python function that takea sorted array and target value, and return a valid index where target can be inserted to maintain the array's sorted order. Optimize the function to run in logarithmic time complexity.
For example, given arrra [1, 3, 5, 5, 6] and 'target 5", the function should return either or because 5 presents at both indices 2 and 3.
[Response]
solve this problem efficiently and ensure logarithmic time complexity, we can use a binary search algorithm. Compared with standard binary search that looks for an exact match, we can modify the algorithm such that when the target is not found, return 'left" bound, which represents the index where target can be inserted to maintain the array's sorted order. Since any valid index for insertion is acceptable, we can direclty return the index if there is exact match.
Here a Python function that implements this approach:
"python
from typing import List
def search_insert (nums: List [int], target: int) int:
Finds the index to insert 'target" into "nums" to maintain sorted order. For example,
given 'nums [1, 3, 5, 5, and 'aaret returns either 2 or 3, as "target" can be inserted at index 2 or 3 to maintain the sorted order.
left, right 0, len (nums)
while left <= right:
mid (left + right) //
if nums [mid] == target:
return mid
elif nums [mid] target:
left mid
else:
right mid
return left
This implementation ensures that when 'target" is not found, "left" insertion index. This is because the
represents the the loop, all elements to the left of "left' are smaller than target', , all elements to the right of 'left" are equal to or larger than ttargt", and "left right'. Therefore, "left" is the correct index for insertion maintain the sorted order.
[Tssts]
You can test the function with scenarios where the target is found at multiple indices and where the target is not present in the array:
python
assert search
([1, 3, 5, 5, 6] 5) in [2, 3]
assert
([1, 3, 5. 6, 6] 2) == 1
56

---

assert sarch_ 3, 5, 6, 6] 7) = 5
assert search ([1, 3, 5. 61 0) 0
### Example 6
[Code]
def decompress (self)
source self compressed
if isinstance (source, (bytes, bytearray)):
return self www.mpdess._byeeee
pos
node self root root res bytearray()
while pos < len len
code int (source [pos])
child node children [code] if haild.is_es
res. append append(chiild (child) node self root
node child poa +=
return pytes(res)
[Property]
category: function implementation
language: Python
difficulty: medium
concepts: decoding compressed data, binary tree traversal, byte array processing
[Instruction]
Write Python function nufffann_eccmprees
that takes two arguments: compressed_data"" a byte array of 0/1 sequence representing Huffman compressed data, and "huffman,tred tree', a Huffman tree that stores the actual data (each leaf is associated with character). The function should return the original uncompressed data bytes". Define the "HuffmanNode" class yourself as a general tree structure with a "children" list.
[Response]
To accomplish Huffman decompression, we first need define the Huffmanlode" class, which, as required, should be a general tree structure with a "children" list and an optional 'value" attribute to store the character associated with leaf node.
Then, we'll implement the "huffman derronndccompesss" function. This function will traverse the Huffman tree according to the bits in compressed_data". When it encounters leaf node, the associated character will be appended to the result, and the traversal This process continues until restarts from the root of the processed. the entire compressed_data""
You said that compressee_data is byte array of 0/1 sequence, so assume that each byte, each bit, represents step in the Huffman tree. In practice, it more efficient
Here 's how we can implement this:
python
class HuffmanNode:
def .nint___ (self, value=None):
self value = value
57

---

self. children []
def deffman_decomprese (coppeesseccpppeeeeeedeeoo (aoppressed_data, huffman_tree)
result [] node huffman_tree
Each byte represents bit ddo
for bit in compressed_data:
if len (node. children)))) > 0:
node node. children (int
if len (node. childrenn) == 0:
result append (node. node huffman_tree
return "* join (result) encode
In this implementation, each byte in compressed_datae representa a bit, guiding the traversal of the Huffman tree. Characters are appended upon reaching leaf nodes and the result is then encoded bytts"..
[Tests]
You can test this function with sample Huffman tree and compressed data to verify that the decompression works correctly:
ppython
root HuffmanNode ()
root. children = (HuffmanNYode
HuffmanNode (*B*)]
root. children [0] children [HuffmanNode (*C²), HuffmanNode ('D*)]
root. children [1] children [HuffmanNode (EE), HuffmanNode ('F?)]
compressed_ data bytearray (1o, 1, o, 0, 1, 1l)
01 -> D, 00 -> c, 11 ->
assert huffman, decompress (compressed_ root) DCD
### Example 7
[Code]
def format Bize hatpaaaorrrrrooooocco/////99999 com/ "
size (num):
for in ['bytes', 'KB 'MB', GB *] if num < 1024.0 and num -1024.0:
return "13.1%%' (num,
num /- 1024.0
return "/3.11%s" % (num, 'TB') assert format_size (1024**2 1) == '1024.0XB3
aasert format_size (1024+512) '512.0KB'
[Property]
category: function implementation
language: Python
difficulty: easy
concepts: arithmetic operations for size conversion, rounding numbers, dynamic unit selection, string interpolation
[Instruction]
Create a Python function to format a file size in bytes into human readable string representation, using 'bytes', 'KB' (kilobytes), MB (megabytes), 'GB' (gigabytes), output should be rounded format: "[I. "Unitt"" where "Unit" the appropriate aize unit and "X.Y" is the rounded size. For example, 'format siie (1024 512) should return "KB and "format_ size (1024*+2 1) should return "1022.0KK ..B
'TB' (terabytes) units. The decimal place and follow the
58

---

[Response]
create such a function, we can use a loop to iterate through a predefined list of units ('('bytes', 'KB', 'MB', 'GB', 'TB']') and check if the given number of bytes fits into the current unit by comparing it with 1024.0 and -1024.0 (for handling negative sizes) We'l1 keep dividing the number by 1024.0 until the correct unit found where the number is less than 1024.0. Then, we'll format the result using the appropriate unit and the rounded size.
python
def units ['bytes', 'KB 'MB', 'GB', 'TB']
Directly return for 0 bytes to avoid division by zero in loop if size_bytes == 0: return "0. "0.0bytes"
num - abs (size_bbby)
unit - units [0] u units [1:]:
if num < 1024.0: break
/= 1024.0
unit formatted_size "f(num:..))uunitt)
return formatted_size if size_bytes >= 0 else ""=(formatted_size)
This implementation handles both positive and negative file sizes returns the formatted size as string, even though negative sizes are not common in real-world scenarios.
[Tests]
can write tests that cover both positive negative sizes across different units to validate the correctness of the format_size" function:
python
test data
(1024 2, "1.0MB"),
(1024 ** 2 1. "1024. .1024.0KB").
(1024 ** 3 512, "512. "512.0GB")
(1024
512, "512.0TB"),
for size, ppocted_outpun in test_data: assert format_size == expected assert format_ size (-aize) f"=[eeppecte_ooupput)
### Example 8 [Code]
def unify_stringlist(:: list):
*** Adds asteriske to strings that appear multiple times, so the resulting
list has only unique strings but still the same length, order, and meaning.
For example:
wbify_stringliss(((a,,, b., 'a') -> Pa','an" 'b', ".''''']
assert(all (Lisinstance {
for L))) return [ count count (L[i]) for in range (1en (L))]
assert unify_stringlies (list (liat("abc))) = list ("abc")
59

---

[Property]
category: function implementation
language: Python
difficulty: easy
concepts: list comprehension, unique string creation, precondition
checking for type validation
[Instruction]
Deaign a Python function to encode a list of strings into a unique list with the same length, order, and meaning. The new list is formed by appending asterisks (+) to duplicate strings. For instance, given ['a', 'a', bb', 'a', 'c',, the function should return ['a', 'ak', 'b', 'ass', 'c'l, while for ['a', 'b', 'c'] should return P'a,, 'b', 'c'l without any changes. Write assertions to ensure the input is list of strings.
Solve the problem in two lines of code, one for the precondition check and the other for the main logic.
[Response]
You can achieve this functionality by first ensuring the input meets your requirements using an "assert" statement with list comprehension. Then, use another list comprehension to iterate through the input list and append 122- to the strings. The number of "*x-- is the number occurrences of the string before the current index.
Here 's how you can do it in two lines of Python code:
python
def encode_stringe (input_list)
assert all (isinsttance (item, str) for item in input_liat). "Input must be list of strings"
return [s + "** input_list (:1). count for i, in enumerate input_list)]
Note that although the function is concise, it is not the most efficient solution for large lists, the "count" method has time complexity of ((n) for each iteration.
[Tests]
You test the function with the provided examples to ensure it correctly encodes strings expected:
python
assert encode strring (['a,, 'a', bb', 'a', 'c']) = Pa', la*,, "b' a 'c'] assert encode_ strings (['', 'b' 'cl)) Pa', "b' 'c']
### Example [Code]
def createefolds list (data, count):
Creates folds from the given data.
:param data: the data to fold
:aara count: the number of folds to create
returr: list of folds ***
fold_count = len (data) count
folds list() ()
60

---

for fold_ing index in range
low int (fold_index fold_count)
high int ((fold_index + 1) count)
fold data hat::hoh folds. append (fold)
return folds
[Property]
category: function implementation
language: Python
difficulty: medium
concepts:
data partitioning and slicing, learning from input output examples
[Instruction]
Construct Python function frratte_fldd list of folds from the given data, where each fold is subsequence the original data. length of each fold should approximately equal with at most difference of 1. The function should satisfy the folloving assertions:
count) to create a
python
assert create_ (range (12),
[range (0, 6) range 12)]
assert freate_folds frang (8), 2) [range (0, 4) range (4, 8)]
assert create_ folds (range (25), -5) == []
assert foldd (range (6), 2) == [range (0, 3), range 6)]
[Response]
To create Python function that generates list of folds from given data with each fold having approximately equal length (vith at most a difference
between any two folds), you can follow
these steps:
1. Return empty list immediately if 'count' is non-positive since it's possible
valid number of folds.
2. Divide the total length the data by "cuutt" rounding down, to get the minimum number of items each fold should have.
3. Calculate the remainder to see how many extra items need to be by to distributed among the folds. They are distributed the first "remainder' folds.
4. Use a loop to create each fold, adjusting the starting and ending indices based on the calculations above to ensure each fold has the correct number of items.
Here 's how you can implement this:
python
def create_folds (data, count);
Return an empty list count is non non-positive count 0: return []
data_length len (data)
fold_size data_length // count
remainder data_length count
folds []
start_inder
for i range (count):
# Add 1 element to each of the first 'remainder' folds
61

---

end_index start_index + fold_size (1 if i remainder else
0)
folds append (range (r (start_index, end_index))
start_ index end_index
return folds
This implementation ensures that each fold has an equal length with at most difference of 1 between any two folds. It handles edge cases, such as non-positive 'count', by returning an empty list specified.
[Tests]
You can test your code with the provided assertions to verify that the function works correctly:
python
assert create_folde (range (12), 2) == [range (0, 6). (6, 12)1
assert create_folde (range (8), 2) [range (0, 4) r4nge 8)]
assert create_folds (range (25). -5) -- []
assert create_ (range (6), 2) [range (0, 3), range (3, 6)
### Example 10 [Code]
def duration (stringg)
match re. match string) if not match:
raise ValueError ( Invalid duration: wormattttttinng
match.group (2) suffix match. group if auffix not in SUFFIX_MAP:
raise ValueError ( Invalid duration suffix: O format (string))
return int (match group (1)) SUPPIX_MAP [suffix]
[Property]
category: function implementation
language: Python
difficulty: medium
concepts: regular expression for pattern matching, dictionary lookup for suffix mapping, error handling with exceptions
[Instruction]
Create a Python function that converts string representation of a time duration into its total number of seconds.
The input string should follow a format that includes numeric part followed by time unit suffix without any spacea between.
The time unit can be seconds in the form of 's' or 'gec', minutes
the form of 'm' or 'min', hours in the form of 'h' or 'hour', and
days in the form of 'd' 'day? (units are in small case).
If the input format is invalid, the function should raise ValueError
For example, the function should pass the following assertions:
pyytonn
assert parse_duration ('15s')
assert parse_duration ('2min ") 120
assert parse_duration ( ('1h') =- 3600
assert parse_duration ('dd') -- 86400
try:
carse_duration (330')
except ValueError as e:
pass
62

---

[Response]
The desired function aims to transform string representation into its total number of seconds. The implementation can be split into two parts:
Input validation:
1, The input string includes two parts: number the prefix and the unit the suffix.
2. can use regular expression
assert the prefix must be number (d+) and suffix must be one s/sec/m/min/hhhour /a/oorr///ay..../
3. If the input string cannot the regular expression, we raise ValueError.
Result computation:
1. Based on the matches of regular expression, we capture the number and unit part.
2. convert the unit to seconds: 's'/'sec' 1 second, 'm'/mmn'' seconds, 'h'/'hour' seconds.
3600 seconds, 'd'/'dyy -> 86400
3. We multiply the seconds of the unit and the prefix number and return it
integer.
Here's 's how you can implement this:
python
import re
def dereedddraiionnuuraat
pattern re. compile (r
hhlhuur I day) $*)
match pattern match (duration_str)
not match:
raise (f duration: (duration_str)
value, auffix match groups
value = ivalve)
suffix_map =
's': 1, 'sec': 1,
'm' 60, 'min': 60,
"h": 3600, 'hour : 3600,
'd": 86400, 'day': 86400
return value auffix_map [suffix]
[Tests]
You can test your code on various valid and invalid inputs to ensure that the function behaves as expected:
python
Test additional valid inputs with different suffix representations assert parse_duration (*10sec')
10, "Check for 'sec' suffix" assert parse_duration ('3min ") == 180, "Check for 'min' suffix"
assert parse_duration ('5h') = 18000. "Check for 'h suffix"
assert parse_duration ('2hour)) ") 7200, "Check for 'hour' auffix"
assert parse_duration ('1day)) ") 86400, "Check for 'day' suffix"
Test edge cases
assert parse_nuration ('Os') -- 0, "Zero duration" aasert parsenduration ('0mi == 0, "Zero minutes"
Test inputs with leading zeroa
assert parse_duration (001s') == 1, "Leading zeros"
63

---

assert parse_duration ('010min') = 600, "Leading zeros with minutes"
Test case sensitivity and invalid suffix handling def assert_value (tr.aaaao...oooooomuaaaaaooooo)) (duration_str)
try:
parse_ durationno (duration_str)
except ValueError:
pass
assert_value_ ('108")
arrerto ('2hours") ')
assert_value_erze ('5 mins
('h1')
gasert_value_ers (100')
Test extremely large numbers assert parse_duration (>1000000s') 1000000, "Very large number of seconds"
*## Example 11
[Code]
class Restaurant ()
def -_init__ (self, name, cuisine_type):
self name name. title()
cuisine.. self nalf.numbrr_sere
cuisine_type
def describe_restaurant (self))
mag f"(self,,,. name) print(f"\n(mmg))
name]
{self seiisine_type.)
def open_rostaurant msg f"(eeff name) print (f (f")
def set uet_nuumer__o served (self, number_served self. number served number served
def cettaaett_uumer_seeeee serve additiomal_served) self number_sorvar += additional_served #
[Property]
category: class implementation
language: Python
difficulty: hard
concepts: object- programming, string formatting and manipulation, encapsulation, input validation
[Instruction]
Develop Python class "StringEnhancer" that offers advanced tools for managing a string. The class should maintain internal that is initially an empty string. The class should support
extending the string with additional prefix or suffix, converting the entire string to lovercase characters within the string. All the operations through a central method: "apply_operation (self, operation: str)
uppercase, and altering specific communicated
The method ooplatoperattin" interprets the operations specified in the form of "[[pname]: [arg]" Here the detailed formats for each operation, explained through examples:
Prefix addition: Suffix addition: 'apply_operation ("suffix: World!")
("prefix:Heil Hello, ")"
64

---

Lovercase conversion: "apply-operation ("lower:")
Uppercase conversion: apply_ ("upper:")"
Character replacement: aapply-operation (tapypoprratiooeeeeeeaceeeeee ("replacera->e")
The class should handle erroneous inputs gracefully, such as incorrect operation formats arguments. Additionally, implement the __str__ method to return formatted string showing the current value the string and list of operations that have been applied to it.
[Response]
The goal is to create Python class "StringEnhancer" capable managing a string through various operations, using central method opplatoperation' The class also needs to handle errors gracefully and record the operation history. Here are the key steps implement this class:
Initialize the class with an empty string and an empty list to store the operations.
Inside the aapply.operation method:
Parse the operation string determine the operation type and argument (if any).
Gracefully handle any errors in the input format by printing an error message without actually raising an exception.
Execute the correct string manipulation based the operation. Append the operation to the records list.
Implement the -_str__" method by concatenating the current string value and the recorded operations.
Here how you can implement the "StringEnhancer" class:
python
class StringEnhancer:
def __init__ (self); self string "
self celf.operation
[]
def (sf operation: str):
parts operation. (at.oon.sltt("))
if len (pn
print ("Invalid operation format. Expected [[opnam]] [arg
)
return
opname, arg - parts [0], [arts[1]
if оpname == "prefix":
self striigtnigg.... elif оpnаme
arg + self string
"auffix":
self self selfistring... += arg
elif opname "lover":
self string self string lower ()
elif оpname
"upper":
self string self strirtring...m upper ()
elif оpname "replace":
if ".>" not in arg
print ("Invalid replace operation format. Expected
replace:a->b">
return
char, new char arg sp1it("))))" self stringtrig = self string replace char new_char)
else:
print Unnunoo operation: {opname)")
self. onffaooprtioons.. append (operation)
65

---

def _.str_-
operation_histe history "ojon operations)
return f Cuuurrent String: {self. ( string/ { operation.history
[Tests]
To test the class, you should apply different operations and check
the string and operation history are updated correctly:
python
enhancer StringEnhancer
enhancer www.:aapppyp..praappaea : Hello")
enhancer. caaancrr.applyy.opraaiioo ("a.loopprrttiooaaafiii.. ('suffix: World!")
enhancer. wpt..aaayyyy.petiii...mm ("lover:")
enhancer ewhancer.apply.coeo. ("apppaattt.ooppppppc...aaaa))))
expectod..peraaioo...htttor
Hello"" "suffix:, World!",
lower:", "replace:o->a")
expectel.operatiooo.bissoo.r___
'ooiin (eo
expocted_str.value "hella, warld!"
"hella,
assert enhancer
operations ophraacor.oprrations
assert str == furrreett String: sxapected_str_vallee value)
nOperations:
history-atr]"
*## Example 12 [Code]
while cur num < len(ebrree_llppottacees:)
cluster_ idx (nttssoredd_.iportanee [our nomr_ [1])
filter_idz inttttored.ipptttacee [cur_ [cur_num] [2])
if amp.praaing.guutas [ccusss 0:
tmp.pruning-quotas [cluster_idx] 1
else:
cur_nus 1
continue
cluster self cluster_idx)
get.cluuter_by.
for node in cluster. elements:
[Property]
category: function implementation
language: Python
difficulty: easy
concepts: iteration with while loop, index-base data access, conditional branching, nested loops
[Instruction]
Create Python function that identifies all pairs of elements within list of integers where the second element of the pair is least greater than the first by "grouth_ and the indices the elements are within specified maximun_distant other.
from each
[Response]
You can write a Python function 'find_pairs" that takes three parameters:
"numbers's The list of integers. "growth_ratio'' A non-negative "float that specifies the ratio by which the second element should be greater than the first.
66

---

maximum_distance" non non-negative integer that indicates the marimum index distance allowed between the two elements in the pair.
The function will return a list of tuples where each tuple contains the indices of the pair of elements that satisfy the conditions.
Here how you can implement this function:
ppython
from typing import List, Tuple
def (naubere (numbers: List [int], grouth_ratio: float, taximum_distanee int) -> List (Tuplelii, int]]:
Find pairs of elements within list of integers where the second element is at least ratio greater than the first the derimum_distance from each other.
indices are within
pairs: List [int] []
i range (len (numbers)):
# For each number, look ahead up to "maximm_distance" elements
for in range + 1, min(i + maximu__istaace, len numbers))):
if numbers[j] numbers [i] grouth_ratio: pairs. append ((i, j))
return pairs
This function iterates over the pairs elements the list whose indices satisfy the distance constraint and stores the valid pairs that meet the growth ratio condition.
[Tests]
To ensure the function's 's correct behavior, you can compare your implementation against a brute-force approach that leverages list comprehension to first generate all possible pairs and then filter them based the conditions. The teats should cover both the basic functionality where pairs are found, and edge cases where no pairs are found, while also altering the rgrotth.ratio" and maxtann_distance parameters to cover more cases:
python
def brute_force_patle (numbers: List grouth_ration float, maximum distuncdistneee int) -> List [Tuple [int, int]]: return [
(i, j)
for i in range (len (numbers))
for j in range + 1, min(i + 1 maximu__dsstance len numbers)))
if numbers[i] numbers [i] >= growth_ratio
test_ cases ([30, 15, 60, 7, 77
13, 10] 8, 3)
([30, 15, 60,
13, 10] 0.2, 3),
([30, 15, 60, 7, 77
13, 10]
30, 0),
([30, 15, 60, 7, 77, 13, 10], 0, 2)
(0), 1, 3)
([8, 8, 8, 8], 1, 0)
]
for numbers, grouth_ratin maximum in test_ cases:
67

---

expected_paira = brate_force_paize (numbers, growth_ratio, aaxiuu_disstance
assert find_pairs (numbers, grosth_ratio, Daximun_distanee expected_pairs
### Example 13
[Code]
def accuracy (output, target, top)=(1,))
"o"Computes the precision0k for the specified values of k""" maxk max (topk)
batch_size target s0)
pred output topk (maxk, 1, True, True)
pred pred. t()
correct pred eq (target. view(1, -1) expand_andee (pred))
rea =
for k in topk:
correct_k correct (-ew(-1) float sum (0) res. append (correct_k mul1_ (100.0 batch_size))
mul_
return res
[Property]
category: function implementation
language: Python
difficulty: medium
concepts: top-k accuracy computation, tensor operations in PyTorch, batch processing, default arguments
[Instruction]
Create a Python function "aalculate_too___accam
using PyTorch that calculates the top-k accuracy from batch of predictions and actual labels. It should accept two tensors, "predictions" and targets', and an integer 'k' (default 3). The function should return the average top-k accuracy as a float, determining if the true label is vithin the top 'k' predictions for each input.
[Response]
The function will take the following parameters as input:
ppeddctioos' tensor of shape (batch_size, nun_classes) that contains the predicted scores for each class. 'targets's a tensor shape (batch that contains true class labels for each input.
'k': an integer representing the number of top predictions to consider for computing the accuracy. The default value should be
Here is how you can write this function using PyTorch:
python
import torch
def K, (predictions: torch. Tensor, targets: torch Tensor, k: int = 3) -> float:
Ensure this function works if is greater than the number classes
min(k, predictions (icdicttionsssze(1))) si)e
-, top_k_indices predictions. topk (k, dim-1)
- targets unsqueeze (1) expand_ass indicees
correct_predictions
correct correct_ any (dim=1)
accuracy = correct. men
mean
return accuracy. item()
68

---

The function retrieves the top k predictions' indices, aligns shapes for comparison with true labels, computes accuracy by checking matches, and returns the average accuracy as a float.
[Tests]
python
Test 1: Basic functionality test k =1
labels torch. tensor ([2, 1, 11)
preds torch. tensor ([
[0.1, 0.2, 0.9], yes
[1.0, 0.2, 0.3], # yes
[o.3, 1.1, 0.2], yes
[0.6, 0.1, 0.3],
1)
accuracy = calculate_ _ap_uk_cccuracyyc.ococcpoaaaaa_p (precss labels, k) assert accuracy -- 0.75
Test 2: Edge case with number of classes
preds torch. tensor ([ [0.5, -1.2, 0.3], yes [-1.0, 0.1, 1.2], yes
1) labels torch. tensor ([O, 2]) accuracy = calculate_ top_ accuracy (preds, labels, k)
assert accuracy == 1.0
Test 3: greater than the number of classes should still work
preds = torch. torch.tensore tensor (
[0.1, -0.2] yes
[-0.3, 0.4], yes
labels torch. tensor ([1, 0])
accuraay = calculate_ top_tpp_- ttt_t__accuueeeaeeeeappapppppeee accuracy (preds, labels, k=5)
assert accuracy == 1.0
*## Example 14 [Code] def aleermstinnhharaattee(())))
j count -
while (<len(s)
if s[j] s[il:
while j<len(s) and s[j] [i]: j+
count += j-i-1
i
j +=
return count
[Property]
category: function implementation
language: Python
difficulty: easy
concepts: 2-pointer technique, conditional loop, string traversal and character comparison, count accumulation
[Instruction]
Write a Python function called 'count_ "count_conditt conditional saattiionl_mmtchessss that takes two strings, 'stri' and "sr22", as well as boolean flag "strict"
69

---

return count arguments. The function should count how many times characters from the end of 'strl" match with characters from the beginning ssr2 when both strings are traversed simultaneously. The traversal should continue until a pair of characters do not match or until specific condition influenced by the "strict" flag is met:
If "strict" is "True, stop counting as soon as non non-matching pair is found.
If "strict" is 'Fass'' skip the non non-mant non-matching character in "srr1" and continue matching with the next character.
The function should return the total count matching characters. This implementation requires managing two pointers for the characters in "str1" and "str2", respectively, and using conditional loop handle different stopping criteria based the "strict" parameter.
[Response]
for each string, iterate We characters from the end stri and the beginning of str2. The loop continues either until the characters do not match (for both strict and non-strict cases) or until it ezhausts one of the strings. The "strict" flag controls the behavior when mismatch is encountered.
pointers,
the
python
def htw.e/oonniinnannaeeeenee
str2, strict): Initialize pointers for the end of strl and the start of str2 i, j len (str1) 1, count 0
Loop long as is non non-negative and is less than the length of str2
while
and (str2))
if stri[i] str2[j]:
count +=
i -=
j
elif strict: break
else:
[Tests]
The function be teated with various inputs
handles
both strict non-strict cases correctly:
pyyhon
Test function with matching characters
assert commt ("hello, "o" True) ==
assert conttcooddtioonal.aacceeee ("abcde", "e", False) ==
(a ("fancy", "nancy", False)
assert count_coditiioal_.attcooo assert count_coniitional_aataaae ("ciemettcchhhhhehhpttt.t.t ("helicopter", "ranges", True) == 1
Test function with no matches
assert countcoondtiinal..attee ("apple", "banana", True)
assert moant_condtioonalmaatehee (" "bacdef", "xyzab", False) --
Test when both strings are empty
assert count_coditioaal..atteo (**, "", True)
assert count count_conditioaa__eee matchaeeaaateeeeeeeeeaeeaaa ("* "", False) == 0
70

---

Test when one string is empty
assert conttconditioaal.mttahe www.aaaaahheeeeeeeeaeaeae ("nonempty", "", True) assert count_ comt
"nonempty", False) 0
### Example 15
[Code]
def
(phrase):
E="Eammlee:
'Eeeny, Meeny, Miney, and Moe' --> E'een'',, 'Meeny', 'Miney', Moe']
-'Black and White' --> ['Black', 'White'] -San Francisco and Saint Francis' --> ['San Francisco', 'Saint Francisco
items []
subphrase in Chrase.splitf 2): items extend
citem.strip(. for item in sechhrase.spltt.. and 2)])
return items
[Property]
category: function implementation
language: Python
difficulty: easy
concepts:
string manipulation and parsing, list comprehension, iterative list extension, handling text connectors
[Instruction]
Create concise Python function
transforms string into list individual items. The input string contains list of items separated by commas and the last item preceded by an optional "and
". For example:
Input: "Eeeny, Keeny, Miney, Moe" --> Output: ['Eeeny', 'Meeny,,
Miney', 'Moe']
Input: "Eeeny, Meeny, Miney, and Moe" ..> Output: [' ('Eeeny', 'Meeny', 'Miney', 'Moe']
Input: "San Francisco and San Mateo" Output: Saa Francisco',
San Mateo
Remove any leading or trailing whitespace.
[Response]
Here 's concise function "string_to_ tisst to transform comma- separated string into a list, replacing and with comma, splitting the result, and stripping whitespace:
python
def string_to string.to_listtt).
return atrte..strpp() for item in s. replace (" ", ","). split (",") if item. strip() != "*]
This approach ensures empty strings included if there excess commas or spaces.
[Tests]
We this function different input strings (e.g., with and without the "and" connector, different numbers items):
python
assert string_to cisttttteeeeeo ("Eeeny, Meeny, , Miney, Moe") ['Eeeny', 'Meeny 'Miney', 'Moe '] assert string_to_lne ("Eeeny, Meeny, Miney, and Moe") ['Eeeny', Meeny', 'Miney', , 'Moe
71

---

assert string-to_list ("San Francisco and San Mateo") == ['San
Francisco', 'San Ma
assert string_to. ("Single Item") ['Single Item
### Example 16
[Code]
def gat_e (buf):
hexdigests = namedtuple ( (Digests', 'md5 sha1 sha256 sha512 ctph 2)
if isinstance (buf, str):
buf open (buf, 'rb').
hashlib. md5 (buf). hexdigeest
shai hashlib. shal (buf) hexdigest ()
sha256 hashlib sha256 (buf). ( ()
sha512 hashlib sha512 hashlib.sha512 (buf). hexdigetst ()
ctph sadeep hash(but)
return hexdigests. _make ((nd5, shal, sha256, sha512, ctph))
[Property]
category: function implementation
language: Python
difficulty: easy
concepts: binary string hash computation, file content reading, string hashing, named tuples, conditional file content reading
[Instruction]
Implement a Python function that computes different types of hash values for the binary content of a file whose path is defined by input string. You should calculate 'md5', 'shal', 'sha256', and these fields. When
'sha512' and return "namedtuple" input string does not correspond to file, the function should directly compute the hash values of path itself.
[Response]
compute the hashes a file or string, we'll follow these steps:
1. If the input is a valid file path, read the binary content of the file.
2. Othervise, convert the input string a binary string.
3. Compute hash values of all types using the "hashlib' module.
4. Return the hash values as a "namedtuple" for easy access.
Here 's how you implement this function:
python
import hashlib
from collections import namedtuple
from pathlib import Path
HashValues namedtuple ( Hashhaalus'', ('md5', 'shal', 'sha256', sha512
def compute_hasher (file_path):
try:
with open (file_path, 'rb') as f: content = f.read ()
Handle where the file does not exist the path directory
except (FileNotFoundError, IsADirectoryError):
content
md5 hash hashlib. md5 (content). hexdigest
shai hash hashlib shal (content). hexdigest sha256
hash hash hashlib. sha256 (20256(con)) (content). hexdigest () tha512_hast hashlib. sha512 (content). hexdigest ()
72

---

return HashValues (md5_ shal_haske sha256_hash,
[Tests]
To test the function, you provide string inputs to avoid file I/0
operations and verify that the hash values are correctly computed:
python
string 'random string'
binary string. string.encode(
computed_hashes compute_hashes hashes (string) assert
md5 hashlib md5 (binary). hexdigest hexceee
assert saauteddhasheseee shal == hashlib shal shal hexdigest ()
assert computed_hashes hashes sha256 == hashlib sha256 (hanaayb hexdigest Q
assert Seaputed_hasheee sha512 hashlib sha512 (binary) herdiiget ()
### Example 17
[Code] from collections import Counter
class Solution:
def sumlfUnique (self, nums: List[int]] -> int:
return sum (k for in Counter (nums) items () if 1)
[Property]
category: function implementation
language: Python
difficulty: easy
concepts: dictionary- counting, filtering unique elements with list comprehension, summation of elements based on condition
[Instruction]
Write Python function that takes list from user input and prints the sum of counts for elements that occur more than once. Use a dictionary to count frequencies efficiently. Utilize list comprehension to extract repeated elements based on the frequency counts, and their corresponding counts from the dictionary.
[Response]
To implement this function, first need to parse the user input list of integers and then count the frequency of each element using dictionary.
can then calculate the sum of the counts for elements that occur more than once.
Here's Python function that accomplishes this:
python
def sum_repeatedd uun_reppated_couutss () nums = list (map (int, input (patttts.p0000 ())) freq_dict O
for
dreq [num] freq get (num, 0) + sum_of_repead = sum (v for k, in freq_dict. items () if v 1)
print(sum_orr (fum_of_repeats) repeats)
[Tests]
We can test this program with I/0 redirection to simulate user input and output. Here is example:
python
import ays
from is import StringI0
73

---

test cases
("1 2 3 "5(n"), ("1 1 1 A\n", "5\n"),
("1 2 3 4 5\n", "Oln"),
("An", "O\n")
]
for input_data, expected_outpur in test_casee: stdin, stdout sys. stdin, stdout stdin Stringl0(input_dare stsout StringID
sum_ run_reppated_coonts getvalue
output assert output == expected_outpur
### Example 18
[Code]
fp-open (m s'','r))
Line==fp.readlinee readlines
for i,line in enumerate (1ines
sad=line.replace ('\n,, " print i, cmd
p=os. popen (cmd)
x=p.read() close
tp. close
[Property]
category: program implementation
language: Python
difficulty: medium
concepts: iterating over file lines, command execution file input new line character removal
[Instruction]
Develop Python program that reads
file containing shell command per line and executes these commands sequentially. Each command should be sanitized by removing unnecessary newline characters before execution. Provide clear error handling and logging to manage exceptions that might occur during the execution the commands.
[Response]
To create Python program that reads and executes shell commands from text file, running the commands and 'logging' for handling logs and errors. The program will involve reading and sanitizing the commands from file, executing then sequentially, and logging the results or errors.
Python built-in module "subprocess" for
Below is the Python script that demonstrates this:
python
imporrt subprocess
importt logging
logging basi:oonf INFO, format= (.
def coa.mmmdddddsfillee...
"**Reads a text file and returns a list of commands..
try:
74

---

with open (file_path, 'r') as file:
commands f readlines
commands [cmd.stripd for cmd in commands] characters and spaces
Remove newline
return commands
except FileNotFoundBrror:
logging (f"The file (file_path) raise except Exception as e:
found.")
logging error (f"An error occurred while reading the file: {str (e))")
raise
def execute command (oommandccommaaa (command):
"**Eees single shell command. "."
try:
logging. (f command: {command}"" result = subprocess run (command, shell=True, check=True,
suduutssubpoocesss PIPE, dederr-subrooesse PIPE)
logging }")
output: (result.steee stdout decode () strrpp
except subprocess CalledProcessError as e:
logging error (f"rrror executing command "(command) (e. stder
((code()))
except Exception
logging.ers error (AnAn unexpected error occurred: (str(e)3")
def main cain(fie_ppah))
"**Main function read commands from file and execute them commands read_commands (file_path)
for command commands:
eeecute__ommatte (command)
[Tests]
To test this program, we can use Python 's 'unittest' framework along
with mock_open" to simulate file operations, 'patch" to mock
subprocess. run' for testing command execution, and "assertLoga" verify the logging output. Here's example of how you write tests for this program:
python
import unittest
from unittest.mock import mock_open, patch, XagicMock
class TuetComenddeeeuttooulittttee (ettttt)))
def tett_read.cmmmnddsuuccccsssc...,
=o m): with patch result coad_dom ("dummy-path"" self aassettEuual ['1s', 'pvd'])
def test_ heweeedcooaand_flleeeee
(self):
with ('tchh'bulltin oppnittnn..oonn, mock mock_open()) as mocked_file:
mocked_ wiw..e.eeeeeeeeee.eeeeeeeeeeeeee = FilellotFondEEror
with self comdand eeadccommaddd"toorssttttttnpppp ""ooerristent_path""
@patch ('subprocess.ruu"
def ott.orccattcccmmmu_uucccceeeeeee
mock_run):
mock_ run www.rreeuaaaaleeeeeeeeeeee
successful", stderr=b"")
execute_ "eecute_comandd"1"""
rock_run. assert cawprrt_aalled_deeeeeee ciclled_onee_veeee v' shell=True, check=True,
stdout=subprocess PIPE, stderr= suderresubprocess.... ( PIPE)
75

---

"patch("suuprooce
def (ssuppoocas..ruu
(self, mock_run):
error_ b"Command failed"
subprocess. CalledProcesEtrord (1, 'ls',
stder with self assertLoga (1evel=VERROR)
log:
execute_ ("1s")
self assertIn (" Error executing command "1s"", log. output
[o])
self assertin (" Command failed", log. output (0])
if ..name... "___ain_."
unittest main()
**# Example 19
[Code]
def busca_Iinarr
el elemento):
- return busca_recursiva (array, elemento,
def busca__ecursiva (array elemento, i):
+= 1
if len len (arraay)))
return -1 if (array [i] = elemento):
return
return busca_recursiva (array, elemento,
[Property]
category: program implementation
language: Python
difficulty: medium
concepts: linear search with recursion, default parameter initialization, multiple base cases in recursion
[Instruction]
Design Python program that only searches for target value list using recursion but also constructs trace of the indices it checks. This program should return both the index of the target if found, "-1" if not, alongside list of the indices it examined during the search. The function should handle cases where target not present in list.
[Response]
We can implement recursive linear search function that returns the index of the target element if found --1" if not. To keep track the indices examined during the search, we can pass a list as an additional parameter to store these indices.
Here the Python program that implements this functionality:
ppython
def rearch target, index, aidices_checkede
"""Reuursive function to search for the target value in the list
unn
index lan(ar)):
return -1 medices_checked (ppecckddappend(ii eppend(inde)
if arr [index] -- target:
return index return search search_recursiveearr, target, index 1, indices_checked
def recursive.search (arr, target):
76

---

""aapr function to initiate recursive search. """ indices_checkes result recursi (arr, target, 0, indices_checked) return result, indices_ checked
Example usage if n.mam... " "__main_. ": array [2, 3, 4, 8, 5, 9, 1, 6, 5, 7,
target =
result, indices - recursive_sean (array, target) (f (target) found index (result) with indices checked: (indices)")
In this program, the Search_recursie for the target element in the list and appends the indices checked the "incices_checked" list. The reeurssiv_ssaacl function acts as wrapper to initiate the recursive search and return the result along with the list of indices checked.
function recursively searches
[Tests]
To test the implementation, we need to ensure that both the index
the target element and the list of indices checked are correct.
Here are the test cases:
python
Test case 1: Target found in the list arrl [2, 3, 4, 8, 5, 9, 1, 6, 5, 7, 5]
targetl = 6
expected_resulti arr1 index (target1) expected_indicess. list (exppeeeddde_deeullee resulti, indices_checkede = setuuraivve_srrchaaeee target1) assert resulti == expected_reaulti assert cndices_checkedl -- indices
1))
Test case 2: Target not found in the list arr2 [1, 2, 3, 4, 5]
target2 =
expected_result2 expected_indices2 list (range [l
result2, aedtices_checkede = secursive_searchhaae (arr2, target2) assert result2 == expected_result2 assert indices_checked. == indicees
Test case 3: Target found at the end of the list arr3 [5, 4, 3, 2, 1]
target3 =
expected_rreuul3 len
expected_indices] list (range (expected result3 + 1)) result3, indices_checkent
recursive_searchhaan target3) assert result3 =- repscted_result3 assert aadices_checked] == expected indices3
### Example 20 [Code]
Data np. zeros (lbreak.points [-1] size size_blocks])
Data_stacked np zeros zeros ([break_points [-1] -nmm blockst,, size_blocks*
num_blocks])
cluster_point_lis - []
for counter in xrange (len (break_points))
break_pt break Bacckk_oointsseeee [counter)
cluster seg [counter]
if counter == 0:
ble_preak_pt
77

---

else:
old_ pt break_points [counter -1] for num xrange bold_break_pt. .reak_.t.bbeekk.tt) break_pt): ##generate the point from this cluster print "num is:", num if num - 0:
[Property]
category: program implementation
language: Python
difficulty: medium
concepts: NumPy array initialization, iteration over break points, cluster segmentation, neated loops
[Instruction]
Implement Python program that initializes NumPy array randomly (0--1) and segments it into clusters based on provided breakpoints Each cluster is defined by consecutive elements between two breakpoints. The output should be a list of NumPy arrays representing these clusters.
For example, given a size 10 and breakpoints [3, 6, 8] the progran should first initialize a NumPy array of 10 random elementa between and and then output three clusters: elements indices '0-2", "3-5", "6-7", and '8-9". Ensure that the program handles cases where breakpoints are the boundaries of array or if there are no breakpoints, in which case the entire array should be a single cluster. Include error handling to manage invalid inputs such negative array sizes or breakpoints that out of range.
[Response]
Here is the program that achieves what you ve deacribed:
pyyhon
import numpy as np
def segment_ (data, break.points).
"eggegent the given array into clusters based specified breakpoints. .*.
break_points [0] break_points [danad)ta)]
segmented_arrays = []
for i in range (len (brreak.poinnta
1):
start_index break.points
end_index break.points 1]
segmented_arrape append (data [start_index index: ind_inxx)))
return segmented_arraye
def create_clusters (size, break.points):
ee*Generate and segment random array of given size using breakpoints. "**
if size <= raise ValueError ArArray data np. random..... (size)
be positive.")
clusters segment_ array (data, break,points: return clusters
Example usage if name.... m_ina__..".
size - 10
break_points [3, 6, 8]
clusters create_clusters (size, break_points)
for i, cluster in enumerate (clusters): (lluterrr (i): {cluster}")
78

---

Construct Python program that processes a list of strings provided by the user. The first element is target string, the second is this implementation, regment_array transforms the input data array into a list NumPy arrays based on the provided breakpoints. The "main" function initializes a random NumPy array of a specified size and calls segment_array" to segment the array into clusters based the breakpoints.
[Tests]
To test the program, we can verify that the values in each cluster are within the expected range (0--1) and that the clusters are correctly segmented. Here
test
python
def test_ lent__gegentaiionnaee (size, break.points):
clusters = create_ create_ clusters break_points)
for cluster in clusters:
assert allal(tuluster >= 0) and np. (clustrr (cluster 1) break_points [0] break_points + [size]
for i in range (len aenrebb_pointee
1):
assert len (clusters [1]) -- break_points 1] break_points
Test cases
cases [
(20, [5, 10, 15]),
(10, [3, 6, 81),
(15, [3, 6, 12))
(30, [7, 14, 21, 27)),
(5, [1]),
(10, []),
(50, [10, 20, 30, 40]),
for size, breakpoints in test_cases: test__ogaentation (size, breakpoints)
### Example 21
[Code]
case 'auto'; if len(set [[cch_amee.._ lover () for ch_name in ch_names])) < len (set ch_names)):
match_case = True
else:
match_case False
if match_case:
picks
channels (ch_names, pick_cha, ordered=ordered)
else:
cames cam_name... lower () for ch_name in ch_names]
chsk_chs [pick_ch lover () for pick_ch in pick_chs]
picks pick_channele (ch_names, chsk ordered=ordered)
return picks
[Property]
category: program implementation
language: Python
difficulty: medium
concepts: case case- insensitiive string matching, controlled case matching based
option, list comprehension
[Instruction]
79

---

either "CaseSensitive' ?CaseInsensitive" to determine the matching approach, and the remaining elements comparison. The program should output the number of times the target appears among the candidates according to the specified matching rule. The solution should include robust error handling to validate the matching rule and check that at least two strings provided before proceeding.
candidates for
For example, given the input '['apple,, 'CaseInsemsitive', 'Apple', banana', 'APPLE']" the program should output "2" since 'Apple' and 'APPLE' match the target 'apple' in laseiinsnssitiv
[Response]
Here's a Python program that fulfills the requirements
specified. It first parses the input list to validate the presence of the target string and matching rule. Then, it compares the target with the candidate strings, adjusting for case sensitivity as specified. Fimally, it outputs the count of matches.
python
def count_ conn. not input_ or len (input_ _iitt) 2:
raise ValueError (" Input list must contain at least two items: the target string and the matching rule.")
target_string, matching_rule input_list [0], input_list if matching_rule not in ["CaseSensitive', "Caselnsensitive ']: raise ValueError ("Matching rule must be either "CaseSensitive" "CaseInsensitive
candidates input_list if matching_rule 'CaseSensitivee:
match_count = sum (1 for candidate in candidates if candidate target_string) else:
target_striaa target_string lower ()
match_count sum (1 for candidate in candidates if candidate. lover () == target_string)
return mauch_
def main();
input_list list input s)
print (count_matcheee (input_list))
if __name_. =_ "__main__".
main ()
The "cout__matcha function implements the main logic of the program, counting the occurrences of the target string among the candidate strings based the specified matching rule. The main" function reads the input list from the user, calls "count_matches" and prints the count matches.
[Tests]
You test the program with various inputs different scenarios correctly:
handles
python
assert count _attchemteeesseee ([' a *Caselnsensitive', 'Apple', 'banana', (PPLE']) ==
assert count_matchin matches [['aplle', 'CaseSensitive', 'Apple', 'banana', APPLE *]) ==
assert count_mn maant_matcheseeeee ( ('apple', * ']) ==
80

---

assert count_matchnn ([' ['apple', " "Caselnsensitive' 'apple', 'APPLE', aPpLe', 'banana ']) == 3
assert count matches count_matches (['aple', "CaseSensitive', 'apple', ?APPLE', aPpLe', 'banana") ']) 1
try:
count_matches ([ ['pappee)] ']) assert False
except ValueError as e:
assert str(e) =- "Input list must contain at least two items: the
target string and the matching rule."
try:
count matches ([' (Papple', 'InaaliRRule', Mpple'1)
assert False
except ValueError
assert str(e) "Matching rule must be either 'CaseSensitive" or Caseensensitive""
81

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We listed the core contributions and key results in the last paragraph of the introduction section ($1). We pointed out that our scope is transparent instruction tuning for code without the use of proprietary data in the first paragraphs of $1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We listed the limitations of our work in $6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
82

---

Answer: [NA]
Justification: Our work studies the curation of high-qualin instruction-tuning data for the post-training of LLMs. Therefore, theoretical results are not applicable here. Instead, we perform comprchensive set of end-to-end evaluations ($3) and component analysis ($4) in an empirical fashion.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
In addition to the general technique in $2, we detailed our experimental config- urations of data generation, code execution, model training, and evaluation in Appendix C, such as the temperature, maximum length of newly generated tokens, etc. We have included a pipeline reproduce the model training and evaluation steps the supplemental material and will also open-source it.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
83

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have included the source code, data, and corresponding instructions the artifact in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ httpc/ggiides/odeeuunnnnnn onoooicc for more details.
While encourage the release of code and data, understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: hetplli/cggidds/ooeeeeeeee
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detailed the configurations and rationales for model finetuning in Ap- pendix C, including data amounts, hyperparameters, optimizers, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Admittedly, we did not draw enror bars for all evaluations we managed align our experimental settings with prior work. However, except for DS-1000 [30] in Table 6 and CanltEdit [6] in Table 7, all other evaluations use greedy decoding compute
84

---

pass I, making the results in theory deterministic. pass is commonly used in code LLM papers assumes in code completion most users would either accept reject completion in one shot.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library functionn bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report I-sigma error bars, but one should state The authors should preferably report 2-sigma eπrоr bar than state that they have 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We reported our compute configurations in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurd: httppp:///nrrrps..cppbbiiiiiieie httpc///cc///.hhcccsiieeeee.ee ines?
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and believe our work does not violate the terms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
85

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Guidelines:
Justification:
Our technique is neutral in not implying clear positive or negative impacts on
society.
The answer NA means that there is societal impact of the work performed.
If authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We provided a sandbox implementation for code execution during synthetic data generation and self- vali-valitttion. The sandbox safeguards the LLMs such that execution --geerrrrteed code cannot impact beyond the sandbox.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use the model, for example by requiring that users adhere to usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
86

---

Justification: We cited the datasets and models for synthetic data generation and evaluations (e.g.. $3 and $4) and specified their versions our best efforts. We listed the licenses of derived assets in both Appendix C.5 and the supplementary materials. We also followed the licenses to license our derivative assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We listed the licenses of our newly created assets both Appendix C.5 and the supplementary materials, which follow and respect the license requirement of their derived work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
87

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
88


Reviews:
[{"review_id": "j9oS0cI09q", "reviewer": "Reviewer_NEpV", "summary": "This paper proposes a pipeline for generating synthetic instruction tuning data. The method consists of the following steps: 1. data filtering is applied to seed coding data to select high quality examples; 2. base LLM is used to generate a set of coding concept and category based on the seed data; 3. base LLM is used to generate coding instruction, response and test; 4. generated examples are selected based on the code execution result.", "questions": "1. Have you tried this framework using stronger LLM to generate synthetic data?\n2. Can you get even better performance by running several rounds of data generation with improved base model?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the paper focuses on using base model to generate synthetic data to self-improve, which is an interesting and useful angle for synthetic data generation\n2. the method is evaluated on several different coding LLM benchmarks which shows the effectiveness of the method\n3. there are also ablation experiments verifying the contribution of specific design choices in the framework.", "weaknesses": "While using base model to self-improve is an interesting and useful direction, synthetic data generation could be improved by using a stronger LLM than the base model. It is not clear from the paper whether the proposed framework would be effective compared to previous methods if we use a stronger LLM to synthesize the data. The synthetic data generation could also be potentially improved by having multiple rounds of data generation process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions! We put our responses to your questions as follows.\n\n> Q1: Have you tried this framework using stronger LLM to generate synthetic data?\n\nThank you for your question. We want to kindly highlight that we explored this point in Section 4.1, which examines the effectiveness of the SelfCodeAlign framework on various base models. Table 7 demonstrates that all models benefit from the framework, and weaker models (e.g., StarCoder2-3B) can achieve larger performance gains when trained on data synthesized by stronger models (e.g., DeepSeek-Coder-33B).\n\n> Q2: Can you get even better performance by running several rounds of data generation with improved base model?\n\nThank you for your insightful suggestion. We agree that an improved base model could potentially produce better synthetic data. While time constraints during the rebuttal period prevent us from conducting additional experiments, we appreciate your point and plan to incorporate this idea with relevant experiments in our final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jB0IIZAC8g", "reviewer": "Reviewer_qGfX", "summary": "This paper introduces SelfCodeAlign, an entirely transparent and permissive pipeline designed for self-aligning code large language models without the need for human annotations or distillation. By applying SelfCodeAlign to CodeQwen1.5-7B, the authors generated a dataset containing 74k instruction-response pairs. They then fine-tuned CodeQwen1.5-7B using this dataset, resulting in SelfCodeAlign-CQ-7B, which demonstrates robust performance on the HumanEval+ benchmark.", "questions": "Refer to the weakness.", "limitations": "Refer to the weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The performance is satisfactory: SelfCodeAlign-CQ-7B achieves a pass@1 score of 67.1 on HumanEval+, outperforming larger models like CodeLlama-70B-Instruct (65.2), which is a significant achievement.\n2. The process is auto-mated: This paper introduces a novel self-alignment pipeline including concept extraction from seed code, task generation, multiple response generation, and execution validation. This approach is independent of human annotations or large model distillation, making it easy to be applied.\n3. Scalability: Experiments demonstrate the method's applicability to models ranging from 3B to 33B parameters, showing good scalability across different model sizes.", "weaknesses": "1. Lack of Diversity in Generated Tasks: While the method aims to produce a variety of coding tasks, it is unclear how this diversity is achieved or measured. There is a risk that the generated tasks may be biased towards certain types of coding problems, which could limit the model's ability to generalize effectively.\n2. Overreliance on Self-Generated Tests: The method relies heavily on tests generated by the model itself to validate responses. This self-validation approach could result in a feedback loop where the model learns to create tests that are easy to pass, rather than generating truly challenging or comprehensive tests. The paper does not address how this potential issue is mitigated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and suggestions! We address your questions as follows.\n\n> Q1: Lack of Diversity in Generated Tasks: While the method aims to produce a variety of coding tasks, it is unclear how this diversity is achieved or measured…\n\nGood question. We ensure task diversity through a rigorous seed gathering pipeline (Appendix A) with quality filtering and deduplication, and 21 carefully designed, diverse few-shot examples (Appendix D) to enhance response generation variety. To measure diversity, we conducted comprehensive evaluations across 7 coding benchmarks covering 4 different problem types. The evaluation results consistently demonstrate SelfCodeAlign's effectiveness in different coding tasks.\n\n> Q2: Overreliance on Self-Generated Tests: …This self-validation approach could result in a feedback loop where the model learns to create tests that are easy to pass, rather than generating truly challenging or comprehensive tests…\n\nThis is a great point. We've implemented two key strategies to mitigate the risk of generating overly easy problems. First, during instruction generation, we have an attribute “difficulty” whose value is randomly sampled from easy/medium/hard. Second, for each instruction, we generate 10 different responses and choose any of the passing responses. This ensures that the model is exposed to a variety of problem-solving approaches and not just always the easiest path. In the table below, we analyze the 74k SelfCodeAlign dataset. The decreasing pass@1 rate with increasing difficulty shows the attribute is meaningful, while the higher and more consistent pass@10 rates help to create a balanced dataset:\n\n|Difficulty|Pass@1|Pass@10|Number of Samples|\n|-|-|-|-|\n|Easy|30.8|77.2|25.6k\n|Medium|27.9|75.9|24.6k|\n|Hard|25.5|74.6|23.7k|\n\nWe also experimented with directly distilling GPT-4o responses to the SelfCodeAlign dataset. As the table shows, our approach outperforms direct distillation from GPT-4o, despite using a weaker base model. Notably, even GPT-4o generates 14% erroneous code, highlighting the importance of execution-based validation in our method.\n\n|Teacher Model|Execution Pass Rate|HumanEval+|\n|-|-|-|\n|GPT-4o|86%|65.9|\n|CodeQwen1.5-7B|100%|67.1|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RHsipqEMyr", "reviewer": "Reviewer_pw5h", "summary": "The authors proposed SelfCodeAlign that finetunes the model based on the filtered data generated by the same model itself. The authors conduct experiments to show that SelfCodeAlign outperforms most open-sourced models that were finetuned on public code dataset.", "questions": "1. Can you properly highlight the row in table 1? \n2. I would suggest to give a brief summarization of the component analysis after line 92.\n3. It seems crucial to me to understand why SelfCodeAlign outperforms other dataset (for example, GPT-generated dataset), is there an analysis on how these datasets differ? Also, you mentioned distribution shift across different models in 4.1, is there a qualitative/quantitative comparison between the code generated by different models?", "limitations": "Limitations have been discussed.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The code generation problem is important and the results (compared to models trained on public dataset) are promising.", "weaknesses": "Compare to models that are distilled/trained on non-disclosed data, the performance of SelfCodeAlign is not as competitive. The presentation can be improved, see **questions**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions! We provide our response as follows.\n \n> Q1: Can you properly highlight the row in table 1?\n\nThanks for the feedback. We appreciate your suggestions for improving Table 1. Could you kindly provide more specific details regarding your concerns, such as which rows you believe need to be highlighted? This will help us make the necessary adjustments more effectively.\n\n> Q2: I would suggest to give a brief summarization of the component analysis after line 92.\n\nAbsolutely. We will ensure to include the summarization in our revised manuscript.\n\n> Q3: …why SelfCodeAlign outperforms other dataset (for example, GPT-generated dataset), is there an analysis on how these datasets differ? Also, you mentioned distribution shift … is there a qualitative/quantitative comparison between the code generated by different models?\n\nGreat questions. Regarding your first question, as shown in Section 4.2, execution filtering and code correctness are important to the effectiveness of self-alignment. The Evol-Instruct [1] and OSS-Instruct [2] datasets used in the paper are direct distillation from GPT-3.5-Turbo, which means they may include incorrect code that harms the model. To verify this, we use the GPT-4o model to generate responses for the 74k SelfCodeAlign dataset and compare it with the original dataset, keeping the instructions the same. The table below shows that while GPT-4o is stronger, it still generates 14% erroneous code, and using its outputs for instruction tuning is less effective than using execution-filtered outputs from CodeQwen1.5-7B:\n\n|Teacher Model|Execution Pass Rate|HumanEval+|\n|-|-|-|\n|GPT-4o|86%|65.9|\n|CodeQwen1.5-7B|100%|67.1|\n\nSecond, Section 4.1 indicates that a base model can benefit more from data within its own distribution than a shifted teacher distribution. In the table below, we compare the perplexity of our base model, CodeQwen1.5-7B, on self-generated data and two GPT-generated datasets. It shows that CodeQwen1.5-7B has a lower perplexity on self-generated data, suggesting it is easier to learn from. This observation complies with the finding from [3] that self-generated positive data with lower perplexity yields better finetuning performance.\n\n|Dataset|Perplexity|\n|-|-|\n|SelfCodeAlign|2.12|\n|OSS-Instruct|2.20|\n|Evol-Instruct|3.76|\n\nFor your second question, we compute the perplexity of CodeQwen1.5-7B on the outputs from different models to ​​quantitatively measure the distribution shift. The table below shows that self-generated data achieves the lowest perplexity:\n\n|Model that Generates Data|Perplexity|\n|-|-|\n|StarCoder2-3B|2.67|\n|Llama3-8B|2.73|\n|StarCoder2-15B|2.28|\n|DeepSeek-Coder-33B|2.25|\n|CodeQwen1.5-7B|2.13|\n\nPlease also kindly refer to Appendix C.2, which provides qualitative examples of the outputs generated by different models at each step of the SelfCodeAlign framework.\n\n[1] Luo et al. WizardCoder: Empowering Code Large Language Models with Evol-Instruct.\n\n[2] Wei et al. Magicoder: Empowering Code Generation with OSS-Instruct.\n\n[3] Setlur et al. RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold.\n\n> C1: Compare to models that are distilled/trained on non-disclosed data, the performance of SelfCodeAlign is not as competitive.\n\nWe want to kindly highlight that SelfCodeAlign still excels over many larger, proprietary models, including Gemini Pro, Mistral Large, and CodeLlama-70B-Instruct. As also mentioned by Reviewer 4fJs, “The results are highly significant, as SelfCodeAlign achieves performance improvements, notably surpassing models that are an order of magnitude larger in size”. Additionally, in Q3, we demonstrate that our approach outperforms direct distillation from the much stronger GPT-4o model.\n\nMeanwhile, SelfcodeAlign is the first fully transparent and permissive pipeline for self-aligning code LLMs without extensive human annotations or distillation. We are more than happy to further discuss this point during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v9GTLuSZZm", "reviewer": "Reviewer_4fJs", "summary": "- The paper introduces SelfCodeAlign, a fully transparent and permissive self-alignment pipeline for code generation in LLMs without relying on extensive human annotations or distillation from larger models. SelfCodeAlign generates instruction-response pairs from seed snippets, evaluates responses with test cases, and fine-tunes models based on successful executions. The approach shows superior performance over state-of-the-art methods, including GPT-3.5-Turbo-based distillation, particularly in HumanEval+ benchmark. The pipeline demonstrates effectiveness across various model sizes, emphasizing the benefits of self-generated data over teacher models with smaller performance gaps. \n- Overall, I feel that SelfCodeAlign is a very easy workflow to follow and I see much potential for such pipelines that do not depend on distillation or human annotations. I recommend an accept.", "questions": "- What about experiments/benchmarking on models that uses the GPT4 family as part of distillation? \n- Why did you only limit it to 33B, what about 70b?\n- Line 121: What is the difficulty for? It is subjective, so how does the difficulty aid the model/aid in finetuning of models?\n- Line 228-233, Table 7: Any reason why the increasing trend is not consistent, for eg for StarCoder2-15B, the score decreased when tuned with DeepSeek-Coder-33B data? \n- Line 232, Table 7: Why did you not fill up the blank cells just like the last row? This would have ensured that your statement is true for all models, because you are just basing off CodeQwen1.5-7B model only.", "limitations": "Limitations are stated. However, I think the authors miss one important point – the reliance on the seed snippets. The performance of the model is based on what it was finetuned on, so if the seed snippets are not sufficiently diverse or representative of the target tasks, then it might have resulted in a large drop in accuracy.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality\n- The paper adequately cites related work, clearly identifying gaps such as the lack of transparency in existing methods, which is a key motivation for their work. \n\n## Quality\n- The submission is technically sound with both quantitative and qualitative analysis.\n- The authors provide detailed experimental results, demonstrating significant performance improvements over baselines.\n- The inclusion of both large-scale and small-scale model evaluations further strengthens the quality of the research.\n- In terms of ethical considerations, they have considered all terms of use as well as the data in code snippets.\n\n## Clarity\n- Well organized paper, except for appendix.\n\n## Significance\n- The results are highly significant, as SelfCodeAlign achieves performance improvements, notably surpassing models that are an order of magnitude larger in size. This work addresses the challenge of instruction tuning without human annotations or distillation, offering a scalable and transparent solution that advances the state of the art in code generation.", "weaknesses": "## Originality\n- Perhaps similar to this paper [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://arxiv.org/pdf/2312.06585) ? Even if it is different, I think that this should also be part of your baseline comparison as well.\n\n## Quality\n- The qualitative examples provided in the appendix are excessively long, which may overwhelm the reader and obscure the main differences and contributions of the methodology. It would be beneficial to reduce the number of examples or to shorten them, focusing on highlighting the key differences and improvements over baseline methods. Additionally, the examples are presented in black and white with no descriptions or annotations, making it difficult to discern their significance. Providing clearer, annotated examples with concise explanations would enhance the readability and impact of this section.\n- I do not see any weaknesses discussed in this work, for example, in what scenario do you think does this methodology not work? Why is the score still not perfect? (or for eg, below 80% accuracy)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and important suggestions! Also thanks for pointing out the presentation issues in Appendix, which we will fix in the revision. We provide our responses to your questions and concerns as follows.\n\n> Q1: What about experiments/benchmarking on models that uses the GPT4 family as part of distillation?\n\nPlease kindly note that we included two GPT-4-distilled models in Table 1: OpenCodeInterpreter-DS-33B [1] and Magicoder-S-DS-6.7B [2]. We also conducted an experiment directly distilling GPT-4 responses to our 74k SelfCodeAlign dataset. As shown in the table below, GPT-4o still generates 14% erroneous code, and training on the dataset produced by SelfCodeAlign outperformed GPT-4o distillation.\n\n|Teacher Model|Execution Pass Rate|HumanEval+|\n|-|-|-|\n|GPT-4o|86%|65.9|\n|CodeQwen1.5-7B|100%|67.1|\n\n[1] Zheng et al. OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement.\n\n[2] Wei et al. Magicoder: Empowering Code Generation with OSS-Instruct.\n\n> Q2: Why did you only limit it to 33B, what about 70b?\n\nThank you for your suggestion. Due to resource constraints, we were only able to experiment with the 33B model as the largest. We plan to explore larger models in future work as we acquire additional resources.\n\n> Q3: Line 121: What is the difficulty for? It is subjective, so how does the difficulty aid the model/aid in finetuning of models?\n\nGreat question. The difficulty attribute is randomly sampled to create problems of varying complexity, which is important for code instruction-tuning datasets according to prior research [3]. To demonstrate the effectiveness of the \"difficulty\" attribute, we analyzed the 74k SelfCodeAlign dataset. As shown in the table below, the decreasing pass@1 rate as difficulty increases shows that the attribute is meaningful. SelfCodeAlign generates 10 responses for each instruction and keeps any that pass. This approach leads to higher overall success rates (pass@10) and more similar rates across difficulty levels, creating a more balanced final dataset.\n\n|Difficulty|Pass@1|Pass@10|Number of Samples|\n|-|-|-|-|\n|Easy|30.8|77.2|25.6k\n|Medium|27.9|75.9|24.6k|\n|Hard|25.5|74.6|23.7k|\n\n[3] Luo et al. WizardCoder: Empowering Code Large Language Models with Evol-Instruct.\n\n> Q4: Line 228-233, Table 7: Any reason why the increasing trend is not consistent, for eg for StarCoder2-15B, the score decreased when tuned with DeepSeek-Coder-33B data?\n\nThe inconsistent trend aligns with our observation in lines 226-228: when the performance gap between models is small, a base model may benefit more from self-generated data than from data generated by a slightly stronger teacher. While DeepSeek-Coder-33B has a higher HumanEval score, StarCoder2-15B outperforms it on math and code reasoning benchmarks [4], indicating their overall performance gap is not substantial.\n\n[4] Lozhkov et al. StarCoder 2 and The Stack v2: The Next Generation.\n\n> Q5: Line 232, Table 7: Why did you not fill up the blank cells just like the last row?...\n\nGood point. We intentionally left those cells blank. Most models in Table 7 were evaluated using both self-generated and stronger-model-generated data to demonstrate self-alignment effectiveness and show distribution shift impact. The last row is an exception, as it represents our strongest base model, which doesn't have a \"stronger\" model to generate data from.\n\n> C1: Perhaps similar to this paper Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models…\n\nGreat suggestion. We'll discuss this paper in our revision. The key difference is that SelfCodeAlign expands and improves both input and output spaces systematically, while \"Beyond Human Data\" focuses solely on gathering high-quality outputs from a fixed original dataset. This highlights our approach's broader scope in generating diverse coding problems and solutions.\n\n> C2: The qualitative examples provided in the appendix are excessively long…\n\nWe appreciate your feedback. We will improve our presentation and make all the examples concise, clear, and easy to comprehend in our revised manuscript.\n\n> C3: I do not see any weaknesses discussed in this work…for example, in what scenario do you think does this methodology not work? Why is the score still not perfect?\n\nThank you for noting this. We did discuss limitations in Section 6, but we can expand further. The main weakness of the methodology is its reliance on test execution. In practice, not all solutions can be verified through execution, and generated tests can also be erroneous. These factors contribute to imperfect scores.\n\n> C4: ...if the seed snippets are not sufficiently diverse or representative of the target tasks, then it might have resulted in a large drop in accuracy\n\nThis is an excellent comment. We ensure seed quality and diversity through rigorous mining, filtering, and deduplication, which we describe in Appendix A. We will further elaborate on this point in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We deeply appreciate all the reviewers for their insightful feedback and suggestions for our work. In our responses below, we address each primary question (denoted as Q) or comment (denoted as C) raised by the individual reviewers. Additionally, we will revise our paper to incorporate editorial suggestions. Should there be any misunderstandings of the questions, please kindly let us know; we are eager to communicate with all the reviewers throughout the discussion period.

Meta-review: The paper studies a method that generates synthetic data for instruction finetuning Code LLMs. They first generate a diverse set of data examples, convert and validate them. The authors are able to train a 7B model to match the performance of CodeLlama-70B-Instruct on HumanEval+. The authors addressed the questions and concerns from the reviewers and discussed key design choices of the method. The authors also acknowledged the limitations of generated tests and execution verifications.
Decision: Accept (poster)