CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow (2024)

Abstract: We introduce a novel dataset tailored for code 001 generation, aimed at aiding developers in com002 mon tasks. Our dataset provides examples that 003 include a clarified intent, code snippets associ004 ated, and an average of three related unit tests. 005 It encompasses a range of libraries such as 006 Pandas, Numpy, and Regex, along with more 007 than 70 standard libraries in Python code de008 rived from Stack Overflow. Comprising 3,402 009 crafted examples by Python experts, our dataset 010 is designed for both model finetuning and stan011 dalone evaluation. To complete unit tests eval012 uation, we categorize examples in order to get 013 more fine grained analysis, enhancing the un014 derstanding of models’ strengths and weak015 nesses in specific coding tasks. The examples 016 have been refined to reduce data contamina017 tion, a process confirmed by the performance 018 of three leading models: Mistral 7B, CodeL019 LAMA 13B, and Starcoder 15B. We further in020 vestigate data-contamination testing GPT-4 per021 formance on a part of our dataset. The bench022 mark can be accessed at anonymized address. 023

text: Findings of the Association for Computational Linguistics ACL 2024, pages 5935–5947 August 11-16, 2024 ©2024 Association for Computational Linguistics
We introduce a novel dataset tailored for code001 generation, aimed at aiding developers in com-002 mon tasks. Our dataset provides examples that003 include a clarified intent, code snippets associ-004 ated, and an average of three related unit tests.005 It encompasses a range of libraries such as006 Pandas, Numpy, and Regex, along with more007 than 70 standard libraries in Python code de-008 rived from Stack Overflow. Comprising 3,402009 crafted examples by Python experts, our dataset010 is designed for both model finetuning and stan-011 dalone evaluation. To complete unit tests eval-012 uation, we categorize examples in order to get013 more fine grained analysis, enhancing the un-014 derstanding of models’ strengths and weak-015 nesses in specific coding tasks. The examples016 have been refined to reduce data contamina-017 tion, a process confirmed by the performance018 of three leading models: Mistral 7B, CodeL-019 LAMA 13B, and Starcoder 15B. We further in-020 vestigate data-contamination testing GPT-4 per-021 formance on a part of our dataset. The bench-022 mark can be accessed at anonymized address.023
1 Introduction024
In the dynamic landscape of software engineering,025 developers frequently confront the challenge of026 translating conceptual ideas into functional code.027 While navigating this process, the gap between028 intention and implementation can often be a hur-029 dle, even for experienced programmers. Tradition-030 ally, developers have turned to online resources like031 Stack Overflow, searching for solutions in natural032 language to address their specific coding dilemmas.033 The emergence of large language models034 (LLMs) trained on code has heralded a new era035 in this domain. Innovations like Codex (Chen et al.,036 2021a) have revolutionized the field by providing037 real-time code suggestions in Integrated Devel-038 opment Environments (IDEs). Similarly, models039 such as ChatGPT and CodeLLAMA (Rozière et al.,040 2023) demonstrate the potential for integrating into041
IDEs, offering developers context-aware assistance 042 in initiating and refining code, thereby enhancing 043 the efficiency of the software development cycle. 044
However, the ascent of code generation through 045 LLMs underscores the heightened need for datasets 046 that emphasize precision, context-awareness, and 047 syntactic accuracy. While existing datasets have 048 propelled advancements in this arena, they are 049 not without limitations. The shift towards LLM- 050 focused datasets has led to a decreased emphasis on 051 traditional training sets, directing attention towards 052 evaluation sets. This shift challenges the training of 053 models from scratch or for specific task fine-tuning. 054 Moreover, while datasets like HumanEval (Chen 055 et al., 2021b) or APPS (Hendrycks et al., 2021) 056 provide valuable insights, they often fall short of 057 mirroring the real-world coding challenges devel- 058 opers encounter. 059
Addressing these gaps, this paper introduces the 060 CodeInsight dataset, a resource specifically tai- 061 lored for Python code generation. This focus is 062 anchored in Python’s widespread adoption in key 063 sectors like data science, machine learning, and 064 web development. The dataset, comprising 3,402 065 unique, expert-curated Python examples, spans ba- 066 sic programming to complex data science chal- 067 lenges, complete with unit tests for evaluation. The 068 CodeInsight dataset stands out in its ability to pro- 069 vide a nuanced balance between breadth and depth, 070 offering a finely-tuned resource for training and 071 evaluating LLMs in Python code generation. By 072 bridging the gap between natural language and 073 code, CodeInsight presents a tool for understand- 074 ing and enhancing the capabilities of LLMs in real- 075 world programming contexts. 076
The dataset provides three primary innovations, 077 uniquely combined within this resource: 078
• It includes a unit test based evaluation, of- 079 fering a more robust evaluation metric than 080 traditional methods such as BLEU score. 081
1 5935
• Examples are annotated to facilitate a deeper082 analysis of its strengths and weaknesses.083
• It provides a training set in addition to a test084 set, with each example being manually cu-085 rated to ensure high quality, supporting effi-086 cient fine-tuning.087
Organized as follows, this paper first details the088 dataset construction process in Section 2, includ-089 ing our sources, selection criteria, and annotation090 methods. Section 3 presents astatistical analysis091 of the dataset, highlighting its diverse applications.092 In Section 4, the dataset’s is tested through eval-093 uations using LLM baselines. Lastly, Section 5094 situates CodeInsight within the broader landscape095 of code generation datasets.096
2 Dataset Construction097
Our pipeline for building CodeInsight consists of098 three pivotal steps. Initially, we identified the099 sources to retrieve examples. Subsequently, from100 these sources, we filtered the most relevant natu-101 ral language-code pairs. The final phase involved102 annotating these pairs and crafting associated unit103 tests. This section provides a breakdown of each104 of these stages.105
2.1 Data Sources106
To develop a dataset for code generation aimed107 at aiding development, we prioritize sources that108 closely mirror real-world development challenges,109 ensuring a match between natural language and110 code. We chose Stack Overflow due to its exten-111 sive collection of real-world programming ques-112 tions and solutions, featuring balanced complexity113 and contributed by a broad and experienced com-114 munity.115 Despite Stack Overflow’s extensive collection116 of developer queries, only 36% of Python-tagged117 questions fit the how-to format essential for our118 dataset, as identified in Yin et al. (2018). A ’how-119 to’ question typically presents a clear, task-oriented120 query where the developer seeks a method to ac-121 complish a specific programming task.122 To address the challenge of identifying relevant123 examples, we utilized the CoNaLa dataset (Yin124 et al., 2018), a curated collection of Python ’how-125 to’ examples from Stack Overflow. This dataset126 features 2,379 examples that have been manually127 reviewed and corrected by annotators, alongside128 approximately 600,000 unrefined examples ranked129
by their likelihood of fitting the ’how-to’ criteria. 130 Our selection encompassed the 2,379 hand-written 131 examples and the top-ranked 3,121 unrefined ex- 132 amples. 133
To broaden the scope and applicability of our 134 dataset, we have incorporated an additional 600 135 samples from Stack Overflow, emphasizing the use 136 of packages like Pandas, Numpy, and Regex. The 137 integration of these packages is a decision to align 138 the dataset with the emergent code generation de- 139 mands in data science, both in academic research 140 and industry applications. Moreover, Regex’s in- 141 clusion enhances the dataset’s to accommodate a 142 wider range of specialized problems. 143
The sourcing procedure began with the elim- 144 ination of redundancies and the filtration of is- 145 sues based on a baseline of community engage- 146 ment—measured by votes and views—and the pres- 147 ence of accepted answers. We then prioritized the 148 problems using a weighted ranking system that ac- 149 counts for the temporal dimension, recognizing that 150 older issues may naturally garner more attention 151 over time. 152
Finally, from our selection process, we gathered 153 a total of 7,300 raw examples to serve as the foun- 154 dation for our dataset. 155
2.2 Data Filtering 156
The transition into the data filtering phase necessi- 157 tates a strategy to select examples from the source, 158 acknowledging that not all contributions from the 159 Stack Overflow community are directly amenable 160 to our goals, as underscored by Yin et al. (2018); 161 Lai et al. (2023). To illustrate, the most upvoted 162 question on pandas is ’How to iterate over rows in 163 a DataFrame in Pandas’, yet the consensus answer 164 advises against iteration, highlighting the complex- 165 ity inherent in the selection process. 166
To navigate these intricacies, we established cri- 167 teria for inclusion: 168
Authenticity of Developer Inquiries Only those 169 questions that present realistic programming sce- 170 narios are considered, ensuring the dataset’s rele- 171 vance to the actual needs of developers. 172
Direct Extractability of Code We require that 173 the code snippet can be unambiguously identified 174 and extracted from the accompanying explanatory 175 text. 176
Natural Language and Code Alignment A ro- 177 bust correspondence between the problem state- 178
2
ment and the code solution is necessary for main-179 taining semantic integrity.180
Executable Code Samples The code must be181 functionally valid, capable of running in a desig-182 nated environment, which is essential for both veri-183 fying its effectiveness and constructing unit tests.184 We decide to exclude code where we need to open185 or save a file.186
After implementing our filtering process, we re-187 fined our initial collection of 7,300 examples to188 2,707 distinct problems, constituting about 37% of189 the original raw examples. This significant reduc-190 tion is represented across various sources: from191 Stack Overflow’s CoNaLa dataset, we retained192 1,993 out of 5,500 examples; in the Pandas, Numpy,193 and Regex categories, the numbers were pruned194 down to 294, 242, and 178 from their respective195 totals of 600.196
The low retention rate in my dataset can be at-197 tributed to various factors. For instance, some198 CoNaLa dataset examples were either non-testable199 or too specialized, necessitating extensive mod-200 ification for practical use. Additionally, certain201 examples offered best practice advice or warnings202 rather than direct code solutions. The complexity of203 queries involving advanced libraries like Pandas,204
Numpy, and Regex also posed challenges. While 205 these queries provide valuable specialized advice 206 on Stack Overflow, they often require significant 207 adaptation for generalization. The second and third 208 examples on the left of Figure 1 illustrate these chal- 209 lenges: one involves extracting features from an 210 image using a library, which is relevant but difficult 211 to test due to the need for incorporating and process- 212 ing images. The other example from the Pandas 213 library focuses on best practices rather than direct 214 coding solutions, not aligning with the dataset’s 215 aim for concrete developer tasks. More examples 216 of what we consider as real-world problems, overly 217 specialized queries, or edge cases are detailed in 218 Appendix A. 219
2.3 Data Annotation 220
Our data annotation workflow is designed to pre- 221 vent model memorization and instead cultivate 222 problem-solving skills within the generated dataset. 223 Through a multi-stage annotation process presented 224 on the right of Figure 1, we curate selected ex- 225 amples from the filtering phase into delineated 226 instances, which are then tested against specially 227 crafted unit tests to ensure their correctness. By 228 refining natural language focusing on the semantic 229 relationships between functions and their descrip- 230
3
tions, we diminish the likelihood of models trained231 on massive datasets to merely replicate solutions232 seen in their training data, a concern highlighted by233 Lai et al. (2023) regarding examples sourced from234 Stack Overflow. To maintain focus and efficiency,235 annotators are allocated a strict twenty-minute win-236 dow per example to ensure timely progression and237 a broad coverage of examples. The ensuing steps238 show our annotation strategy:239
Task 1 - Code Extraction from Stack Overflow240 This initial phase entailed the extraction of code241 solutions from Stack Overflow in response to devel-242 opers’ inquiries. When the question admits more243 than one valid response, annotators are expected244 to capture alternate solutions as well, creating a245 supplementary example for the same intent. Upon246 extraction, they transform these snippets into a stan-247 dardized Python function named test, systemati-248 cally renaming arguments (e.g., vari for variables,249 arri for arrays, etc. See Appendix B for all normal-250 ized names). This normalization approach aligns251 with Yin et al. (2018), recognizing, as pointed out252 by Beau and Crabbé (2022), the significant influ-253 ence this method has on models performance.254
Task 2 - Refinement for Natural Language and255 Code Consistency During this stage, annotators256 refined the natural language descriptions to pre-257 cisely correspond with the test function created258 in Task 1. The challenge lay in harmonizing the259 language descriptions with the Python code’s logic,260 ensuring they are concise yet informative. Anno-261 tators were also tasked with incorporating normal-262 ized argument names into these descriptions to bol-263 ster the dataset’s internal coherence and force the264 alignment.265
Task 3 - Development of Function Test Cases266 The concluding annotation task involved the gener-267 ation of 3 unique test cases for each test function,268 designed to rigorously assess the function’s opera-269 tional integrity and accuracy. These test cases in-270 clude a normal scenario, an edge case, and an error271 situation, providing comprehensive coverage. This272 ensures a thorough yet time-efficient evaluation.273 Once the test cases have been passed, annotator274 can proceed the next example.275
A team of five data science professionals, each276 with a minimum of five years of experience, con-277 tributed to the labeling of the filtered examples.278 They managed to complete the annotation in an av-279 erage time of twelve minutes per example, amount-280
ing to a collective annotation effort of over 540 281 hours. 282
This process yielded a compendium of 3,402 ex- 283 amples derived from 2,702 distinct problem state- 284 ments formulated by seasoned developers. 285
3 Dataset Statistics 286
This section outlines the statistical framework of 287 our dataset, highlighting the diversity of program- 288 ming tasks and the complexity of the included code 289 samples. We approach the analysis from two an- 290 gles: the representation of code libraries and differ- 291 ent labels representing the characteristics of code. 292 Key metrics such as item count, average words per 293 natural language problem, and lines per code sam- 294 ple, alongside the depth of Abstract Syntax Tree 295 (AST) to complete code analysis, are presented to 296 give keys of the difficulty of the dataset. 297
3.1 Packages Statistics 298
The Table 1 illustrates the scope of CodeInsight, 299 which encompasses a variety of packages. 300
A key aspect of CodeInsight is its focus on con- 301 cise and precise problem descriptions, a departure 302 from datasets that retain extensive problem con- 303 texts. This approach is aimed at reducing the word 304 count in problem descriptions without sacrificing 305 clarity and specificity, a crucial factor for effective 306 code generation. 307
Code complexity is evaluated using two pri- 308 mary quantitative metrics: the mean line count 309 of code alongside the depth of ASTs. The lat- 310 ter, a measure of syntactic structure complexity, 311 serves to augment the insights gained from line 312 count data. Analysis of AST depth within our 313 dataset reveals a trend: more intricate coding struc- 314 tures, characterized by nested or conditional logic, 315 are associated with deeper ASTs, whereas sim- 316 pler, linear code correlates with shallower ASTs. 317 Notably, across different packages in our dataset, 318 the AST depth remains relatively consistent, with 319 minor variations observed in packages like Scipy 320 and Scikit-learn, potentially attributable to their 321 smaller sample sizes. 322
Our dataset is compared with the DS-1000 323 dataset (Lai et al., 2023), the latter comprising 324 1,000 evaluation instances and utilizing specialized 325 data science tools from Stack Overflow, complete 326 with extensive problem descriptions. Despite sim- 327 ilarities in code line counts and complexity, dif- 328 ferences in AST depths, particularly in Pandas, 329
4
indicate nuanced syntactic complexity variances.330 CodeInsight features a higher average number of331 unit tests per example, suggesting a more thorough332 evaluation methodology. Unlike DS-1000, which333 lacks detailed analysis of code model failures in334 unit tests, we provide a statistical breakdown of335 code categories to enhance understanding of model336 performance in the next Section.337
3.2 Labels Statistics338 In our study, we identified 10 Categories to en-339 hance our analysis and gain a better understanding340 of our dataset. These predefined categories provide341 insights into the conditions under which models342 were successful or not. These categories vary from343
basic indicators like BUILTIN denoting the use of 344 Python’s built-in functions, to ASSIGN marking 345 variable assignments. More complex categories 346 include COMPLEXTASK for codes with multiple 347 imports, and >THREEVARS for functions with over 348 three arguments. Each example in the dataset is bi- 349 nary annotated—marked as positive if it falls under 350 a category and negative otherwise. For a precise 351 definition of all Categories , refer to Appendix C. 352
Figure 2 illustrates the ratio of positive to neg- 353 ative examples for each category to highlight the 354 impact of each category. For example, we compare 355 the ASSIGN category against all examples that do 356 not include variable assignments. Our analysis pri- 357 marily focuses on the most striking ratios, namely 358 the item count, average code lines and average AST 359 depth, as we found that the unit tests and average 360 problem words exhibit minimal variation across 361 the dataset. Detailed statistical data is provided in 362 Appendix D. 363
The blue bars in the chart, representing item 364 count ratios, significantly highlight the volume and 365 distribution of data in each category. This show- 366 cases the prevalence of certain coding practices; for 367 instance, the BUILTIN category, with nearly twice 368 as many instances as its counterpart, suggests fre- 369 quent utilization of built-in functions, indicative 370 of a Pythonic approach in our dataset. In contrast, 371 labels like COND and LOOP exhibit more balanced 372 distributions, reflecting a diverse representation of 373 these elements. Notably, categories such as COM- 374 PLEXTASK and >THREEVARS are less represented, 375 aligning with the expectation of their complexity. 376
In the context of average code lines, represented 377 by green bars in the graph, specific categories 378
5
such as COMPLEXTASK, MULTIPLETASK, and379 >THREEVARS exhibit notably higher ratios. This380 finding suggests a more intricate and voluminous381 nature of code associated with these tasks. Contrary382 to initial expectations, the LOOP category does not383 show an increased number of lines. Further investi-384 gation indicates that this outcome can be attributed385 to the frequent utilization of Python list compre-386 hensions in this category, which typically reduces387 the number of code lines. In terms of AST depth, it388 remains relatively consistent for MULTILINE and389 >THREEVARS categories. This observation implies390 that longer codes or handling multiple variables do391 not necessarily correlate with increased syntactic392 complexity. However, in the cases of MULTIPLE-393 TASK and COMPLEXTASK, there is a correlation394 between the number of code lines and higher syn-395 tactic complexity. For other categories, the com-396 plexity levels remain to be stable.397
Overall, this analysis underscores the diverse na-398 ture of coding practices and the value of categoriza-399 tion in understanding code complexity and coding400 styles in a nuanced manner. This category-based401 perspective evaluation on code analysis can be used402 in general to understand better model mistakes and403 way to improve model development.404
4 Baselines405
In this section, we test our dataset using state-of-406 the-art LLMs for code generation. Considering407 the volume and nature of our dataset, we explore408 various model evaluation methodologies. Initially,409 we employ a zero-shot evaluation framework, aug-410 menting it with pre-prompts to align better with411 our specialized task. Subsequently, we experi-412 ment diverse partitioning strategies of the dataset413 for model fine-tuning, followed by evaluation on414 the remaining data. Additionally, we conduct a415 comparative performance analysis of GPT-4 on a416 subset of CoNaLa examples and their modified ver-417 sions by our annotators, to understand the impact418 of language and code remodeling relatively to data419 contamination.420
4.1 Experimental Setup421
Models We evaluate the following pre-trained422 language models: Mistral 7B (Jiang et al., 2023) ;423 CodeLLAMA 13B (Rozière et al., 2023) and Star-424 coder 15B (Li et al., 2023). These models have425 been selected to provide a perspective on the scal-426 ability of model performance in relation to their427
size and the intricacies of code understanding and 428 generation. 429
Evaluation Metrics We follow Lai et al. (2023) 430 and measure the execution accuracy using the 431 pass@1 metric i.e. we generate one code and test 432 it agains all unit tests. We also use the BLEU score 433 (Papineni et al., 2002) and the codeBLEU score 434 (Ren et al., 2020) to complete our evaluation. 435
Model input For evaluation, we give to the 436 model the intent in natural language and its as- 437 sociated function header with its arguments. Once 438 the generation is finished, we automatically detect 439 the end of the function -when it exists- to get the 440 whole code and test it. 441
Without prompt Initially, the models were eval- 443 uated using the entire dataset without any addi- 444 tional context added to the natural language intent. 445 The results, as presented in the Table 2, indicate a 446 contrast in performance. Mistral showed notably 447 lower efficiency compared to CodeLLAMA and 448 Starcoder, which both passed nearly 45% of the 449 unit tests. A key observation was the absence of a 450 return statement in a significant proportion of the 451 generated code. While Python allows for scenarios 452 where not returning an explicit value is acceptable, 453 such as actions or modifications without a return 454 value, our dataset did not align with these scenarios. 455 Mistral particularly exhibited a tendency (25% of 456 the cases) to end functions with print statements 457 instead of return statements, affecting its accuracy. 458
First prompt In an attempt to steer the models 459 towards generating return statements for develop- 460 ment aid tasks, a pre-prompt was introduced: “You 461 are a powerful code generation model. Your job 462 is to convert a given natural language prompt into 463 Python function code and return the result.”. Sur- 464 prisingly, this prompt only marginally improved 465 Mistral’s performance, with a slight increase in re- 466 turn statement generation. However, it did not sig- 467
6
nificantly affect the performance of CodeLLAMA468 and Starcoder. Notably, CodeLLAMA’s perfor-469 mance even dropped to 40%, indicating that this470 prompting method might not be optimal.471
Second prompt Aiming to further encourage the472 generation of return statements, a different prompt,473 “Return the Result.” was added to the end of the474 natural language intent. This change led to an over-475 all improvement in performance across all mod-476 els, with CodeLLAMA outperforming Starcoder.477 Mistral, although still lagging, showed an improve-478 ment, successfully passing 10.1% of the unit tests.479
4.3 Fine-Tuning Evaluation480
This segment delves into fine-tuning configurations481 to discern their impact on model efficacy.482
Splitting Method For the assembly of our test483 subset, we curated a collection of 3,094 unique484 problems, each along with at least three unit tests to485 ensure an assessment of model performance. This486 selection criterion is grounded in the necessity for487 test case coverage, which is important in evaluat-488 ing model robustness. Out of this repository, we489 allocated different subset to evaluate the need of a490 train set to perform on test set.491
Fine-Tuning Details We finetuned using Lora492 with r = 16 and α = 16. The LoRA layer incor-493 porated a dropout rate of 0.05 and was configured494 without bias adjustments. The batch size was estab-495 lished at 128, encompassing a warmup phase of 100496 steps and an overall training regimen of 400 steps.497 The learning rate was set at 3× 10−5, with the op-498 timization executed using the AdamW algorithm.499 To optimize computational efficiency, training was500 conducted using half-precision computation (FP16)501 on an a100 GPU with 40GB memory.502
We crafted four distinct training/test splits - 20-503 80, 40-60, 60-40, and 80-20 - to fine-tune the504 CodeLLaMa model. Each split was evaluated over505
five different seeds, and the results are depicted in 506 the following table. 507
In our analysis, we noticed that the performance 508 scores for CodeLLaMa exhibit minimal variation 509 when the training set ranges between 40% to 80%. 510 Interestingly, these scores surpass those achieved 511 through prompting alone. It appears that fine- 512 tuning with just 20% of the dataset approaches 513 the performance levels seen with prompting meth- 514 ods, yet it falls short by approximately 4 percentage 515 points in the pass@1 metric and at least 6 points in 516 both BLEU and codeBLEU scores. Given our ob- 517 jective to maximize the utilization of unit tests, we 518 have determined that a 40-60 split represents the 519 most optimal division for the final configuration of 520 the CodeInsight dataset. This decision is grounded 521 in achieving a balanced approach between training 522 efficacy and test coverage. 523
4.4 Results 524 Finally, we chose the 40-60 split to perform our 525 final evaluation on our baselines. We report the 526 result in Table 5. The Table highlights that fine- 527 tuning has a varied impact on different models. 528 Fine-tuning yields comparable outcomes for Star- 529 coder and CodeLLaMa, each passing slightly over 530 half of the problems. Notably, Starcoder excels in 531 complex tasks like COMPLEXTASK and >THREE- 532 VARS, though it drops to 30% in logical complex 533 tasks. Regex, being a distinct language, poses chal- 534 lenges for all models. Interestingly, Mistral shows 535 significant improvement post-finetuning, adapting 536 well to the task with 38.4% test pass rate. However, 537 Mistral struggles with complex tasks and Regex, 538 likely due to its non-code-specific pre-training, un- 539 like the other two models. 540
We provide a deeper error analysis of CodeL- 541 LaMa in Appendix E. 542
4.5 Exploring remodeling relatively to data 543 contamination 544
In Section 2.3, we detail our approach to mitigate 545 data contamination by rephrasing natural language 546 intents and converting code snippets into function 547 formats. Out of 2,379 CoNaLa handwritten exam- 548 ples, we annotated 812 for analysis. Considering 549 the possibility of these examples being included in 550 GPT-4’s training set—a model not open for fine- 551 tuning—we evaluated its zero-shot performance on 552 them, achieving a BLEU score of 58.8. 553
Further analysis was conducted on the 812 ex- 554 amples post-annotation to assess the impact of our 555
7
modifications. This evaluation resulted in a BLEU556 reduction to 47.6. Remarkably, without fine-tuning,557 GPT-4 passed 64% of unit tests for these examples,558 indicating its effectiveness in understanding natural559 language.560
The performance of GPT-4, despite a drop in561 BLEU score, suggests its coding capabilities rather562 than full data leakage from its training phase. The563 contrast in BLEU scores before and after annota-564 tion suggests our approach’s impact. Appendix F565 presents GPT-4’s predictions, illustrating potential566 memorization.567
5 Related Works568
We introduce a comparative analysis, as detailed in569 Table 4, to assess our evaluation set against prevail-570 ing code generation datasets. This analysis clusters571 HumanEval (Chen et al., 2021b), MBPP (Austin572 et al., 2021), and APPS (Hendrycks et al., 2021)573 due to their emphasis on resolving comprehensive574 programming challenges. Our dataset, however, is575 distinguished by its focus on development assis-576 tance, which is typically characterized by a lower577
average line count in the provided code examples, 578 reflecting a different use case compared to the afore- 579 mentioned datasets. It is important to note that 580 many datasets are designed primarily for the pur- 581 pose of evaluating LLMs, which aligns with the 582 prevalent trend in the field. However, the CodeIn- 583 sight dataset sets itself apart by offering an aver- 584 age number of unit tests per example that exceeds 585 those found in datasets oriented towards data sci- 586 ence, such as DSP (Chandel et al., 2022), DS-1000 587 (Lai et al., 2023), and ODEX (Wang et al., 2022). 588 This difference is largely due to the specific re- 589 quirements of data science code generation tasks, 590 which often necessitate fewer but more complex 591 test cases, dealing with sophisticated input objects 592 like square matrices, classifiers, or dataframes. 593
Featuring three distinct unit tests per example, 594 a specialized training set, and predefined labels 595 for in-depth performance analysis, the CodeInsight 596 dataset represents an unparalleled resource. It sup- 597 ports fine-tuning and a comprehensive evaluation of 598 code generation models, offering a novel approach 599 to enhance their development and assessment. 600
6 Conclusion 601
In conclusion, CodeInsight proposes a new frame- 602 work for testing code generation, specialized in as- 603 sisting developers. It adeptly links natural language 604 and code in more than 3,402 problems, providing 605 a robust platform for model training and evalua- 606 tion. The dataset’s strength lies in its diversity, 607 expert annotation, and focus on practical coding 608 scenarios, making it a valuable asset in the inter- 609 section of computational linguistics and code gen- 610 eration research. Thanks to its categories, it allows 611 a more precise comprehension of code generation 612 model on this task and is completely compatible 613 with other datasets for development aid. 614
8
Limitations615
The CodeInsight dataset, while innovative, presents616 several limitations. Firstly, its specialized nature617 in development aid may not fully represent the618 broader spectrum of coding challenges. Expert an-619 notations, while valuable, could introduce biases620 and may not capture diverse coding methodolo-621 gies. Additionally, the dataset’s current scope may622 limit its adaptability to evolving programming lan-623 guages and practices. Furthermore, its reliance on624 Python restricts its applicability across different625 programming environments. These limitations sug-626 gest areas for future expansion and improvement627 to enhance the dataset’s comprehensiveness and628 applicability in diverse coding contexts.629
References630 Jacob Austin, Augustus Odena, Maxwell I. Nye,631 Maarten Bosma, Henryk Michalewski, David Dohan,632 Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le,633 and Charles Sutton. 2021. Program synthesis with634 large language models. CoRR, abs/2108.07732.635
Nathanaël Beau and Benoît Crabbé. 2022. The impact636 of lexical and grammatical processing on generating637 code from natural language. In Findings of the As-638 sociation for Computational Linguistics: ACL 2022,639 Dublin, Ireland, May 22-27, 2022, pages 2204–2214.640 Association for Computational Linguistics.641
Shubham Chandel, Colin B. Clement, Guillermo Ser-642 rato, and Neel Sundaresan. 2022. Training and643 evaluating a jupyter notebook data science assistant.644 CoRR, abs/2201.12901.645
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan,646 Henrique Pondé de Oliveira Pinto, Jared Kaplan,647 Harrison Edwards, Yuri Burda, Nicholas Joseph,648 Greg Brockman, Alex Ray, Raul Puri, Gretchen649 Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas-650 try, Pamela Mishkin, Brooke Chan, Scott Gray,651 Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz652 Kaiser, Mohammad Bavarian, Clemens Winter,653 Philippe Tillet, Felipe Petroski Such, Dave Cum-654 mings, Matthias Plappert, Fotios Chantzis, Eliza-655 beth Barnes, Ariel Herbert-Voss, William Hebgen656 Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie657 Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,658 William Saunders, Christopher Hesse, Andrew N.659 Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan660 Morikawa, Alec Radford, Matthew Knight, Miles661 Brundage, Mira Murati, Katie Mayer, Peter Welinder,662 Bob McGrew, Dario Amodei, Sam McCandlish, Ilya663 Sutskever, and Wojciech Zaremba. 2021a. Evaluat-664 ing large language models trained on code. CoRR,665 abs/2107.03374.666
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan,667 Henrique Pondé de Oliveira Pinto, Jared Kaplan,668
Harrison Edwards, Yuri Burda, Nicholas Joseph, 669 Greg Brockman, Alex Ray, Raul Puri, Gretchen 670 Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- 671 try, Pamela Mishkin, Brooke Chan, Scott Gray, 672 Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz 673 Kaiser, Mohammad Bavarian, Clemens Winter, 674 Philippe Tillet, Felipe Petroski Such, Dave Cum- 675 mings, Matthias Plappert, Fotios Chantzis, Eliza- 676 beth Barnes, Ariel Herbert-Voss, William Hebgen 677 Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie 678 Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, 679 William Saunders, Christopher Hesse, Andrew N. 680 Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan 681 Morikawa, Alec Radford, Matthew Knight, Miles 682 Brundage, Mira Murati, Katie Mayer, Peter Welinder, 683 Bob McGrew, Dario Amodei, Sam McCandlish, Ilya 684 Sutskever, and Wojciech Zaremba. 2021b. Evaluat- 685 ing large language models trained on code. CoRR, 686 abs/2107.03374. 687
Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- 688 tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, 689 Samir Puranik, Horace He, Dawn Song, and Jacob 690 Steinhardt. 2021. Measuring coding challenge com- 691 petence with APPS. In Proceedings of the Neural 692 Information Processing Systems Track on Datasets 693 and Benchmarks 1, NeurIPS Datasets and Bench- 694 marks 2021, December 2021, virtual. 695
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- 696 sch, Chris Bamford, Devendra Singh Chaplot, Diego 697 de Las Casas, Florian Bressand, Gianna Lengyel, 698 Guillaume Lample, Lucile Saulnier, Lélio Re- 699 nard Lavaud, Marie-Anne Lachaux, Pierre Stock, 700 Teven Le Scao, Thibaut Lavril, Thomas Wang, Timo- 701 thée Lacroix, and William El Sayed. 2023. Mistral 702 7b. CoRR, abs/2310.06825. 703
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, 704 Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, 705 Daniel Fried, Sida I. Wang, and Tao Yu. 2023. DS- 706 1000: A natural and reliable benchmark for data sci- 707 ence code generation. In International Conference 708 on Machine Learning, ICML 2023, 23-29 July 2023, 709 Honolulu, Hawaii, USA, volume 202 of Proceedings 710 of Machine Learning Research, pages 18319–18345. 711 PMLR. 712
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas 713 Muennighoff, Denis Kocetkov, Chenghao Mou, 714 Marc Marone, Christopher Akiki, Jia Li, Jenny 715 Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue 716 Zhuo, Thomas Wang, Olivier Dehaene, Mishig 717 Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh 718 Shliazhko, Nicolas Gontier, Nicholas Meade, Armel 719 Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, 720 Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, 721 Zhiruo Wang, Rudra Murthy V, Jason Stillerman, 722 Siva Sankalp Patel, Dmitry Abulkhanov, Marco 723 Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa- 724 Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam 725 Singh, Sasha Luccioni, Paulo Villegas, Maxim Ku- 726 nakov, Fedor Zhdanov, Manuel Romero, Tony Lee, 727 Nadav Timor, Jennifer Ding, Claire Schlesinger, Hai- 728 ley Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, 729
9
Alex Gu, Jennifer Robinson, Carolyn Jane Ander-730 son, Brendan Dolan-Gavitt, Danish Contractor, Siva731 Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jer-732 nite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas733 Wolf, Arjun Guha, Leandro von Werra, and Harm734 de Vries. 2023. Starcoder: may the source be with735 you! CoRR, abs/2305.06161.736
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-737 Jing Zhu. 2002. Bleu: a method for automatic evalu-738 ation of machine translation. In Proceedings of the739 40th Annual Meeting of the Association for Compu-740 tational Linguistics, July 6-12, 2002, Philadelphia,741 PA, USA, pages 311–318. ACL.742
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu,743 Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio744 Blanco, and Shuai Ma. 2020. Codebleu: a method745 for automatic evaluation of code synthesis. CoRR,746 abs/2009.10297.747
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten748 Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi,749 Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom750 Kozhevnikov, Ivan Evtimov, Joanna Bitton, Man-751 ish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori,752 Wenhan Xiong, Alexandre Défossez, Jade Copet,753 Faisal Azhar, Hugo Touvron, Louis Martin, Nico-754 las Usunier, Thomas Scialom, and Gabriel Synnaeve.755 2023. Code llama: Open foundation models for code.756 CoRR, abs/2308.12950.757
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham758 Neubig. 2022. Execution-based evaluation for open-759 domain code generation. CoRR, abs/2212.10481.760
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan761 Vasilescu, and Graham Neubig. 2018. Learning to762 mine aligned code and natural language pairs from763 stack overflow. In Proceedings of the 15th Interna-764 tional Conference on Mining Software Repositories,765 MSR 2018, Gothenburg, Sweden, May 28-29, 2018,766 pages 476–486. ACM.767
A Detailed overview of filtering phase768
We include two tables that analyze the exploitabil-769 ity of examples from the CoNaLa dataset. The770 Table 6 presents the 10 examples with the high-771 est probability of exploitability, highlighting their772 votes, titles, and whether they are exploitable. The773 Table 7 displays a random selection of 10 exam-774 ples from the same dataset, also detailing their ex-775 ploitability probability, votes, and titles.776 We provide a detailed description of an accepted777 example, a rejected example and a borderline case778 for passing the filtering phase.779
Accepted example We detailed the accepted ex-780 ample which is the first one on the left of the Figure781 1. This particular example, a query about finding782 the largest values in a numpy array, demonstrates783
a typical developer’s question due to unfamiliar- 784 ity with specific numpy functions. Its solution, 785 involving the argpartition function, is directly 786 responsive to the query and easily testable, making 787 it a perfect fit for our dataset. 788
Rejected example The "List sorting with multi- 789 ple attributes and mixed order" question on Stack 790 Overflow from Table 7 presents an excessive level 791 of specificity for inclusion whereas it has a high 792 P(expl) value. This question delves into sorting 793 a list by different attributes of a particular class, 794 emphasizing the specific class’s complexity rather 795 than a broader understanding of sorting functions. 796 The high level of detail in both the problem and its 797 solution complicates the extraction of universally 798 applicable code examples. Therefore, including it 799 may not aptly represent the range of coding tasks 800 and challenges. 801
Edge example An example such as "How to con- 802 vert a Date string to a datetime object?" presented 803 in Table 6 necessitates a more specific reformu- 804 lation for a precise coding answer, like "How to 805 compare a date string in ISO format to a datetime 806 object." This demands the annotator’s understand- 807 ing of ISO format data. These are considered edge 808 cases in our dataset and are included based on the 809 annotator’s expertise, who are constrained to a max- 810 imum of 20 minutes per annotation process. 811
10
B Normalized variable names812
The Table 8 outlines the standardized variable813 names utilized in the dataset, such as vari for814 ’Variable’ and dicti for ’Dictionary’, where i cor-815 respond to the number of the element appearing.816 This approach also allows for evaluating model817 efficacy with or without these normalized names.818 Note that vari is employed universally, even when819 alternatives might be applicable, without affecting820 test outcomes.821 C Code Categories822
823
D CodeInsight Statistics824
The two tables provide a detailed statistical analy-825 sis of the CodeInsight dataset, breaking down by826 Packages and Labels. The Table 10 covers various827 Python packages like Pandas, Numpy, and Regex,828 detailing the item count, average problem words,829 code lines, and unit tests. The second Table 11 ana-830 lyzes different labels presented in Appendix C such831 as Builtin, Assign, Cond, and others, also including832 their item count and average metrics. Both tables833 gives insight on the dataset’s complexity and diver-834 sity into the typical problem structure and testing835
E Code Error Analysis 838
We present in Table 12 an error analysis of dif- 839 ferent CodeLLaMa outputs on our dataset. 840
For the first scenario, involving scikit-learn, the 841 task is to generate a textual representation of a 842 decision tree with specified feature names for read- 843 ability. Despite providing the model with import 844 statements and the function signature, the gener- 845 ated code omits the necessary argument syntax 846 feature_names=feature_names0, resulting in a 847 TypeError due to excessive positional arguments. 848 This minor but pivotal error exemplifies a scenario 849 where a developer could struggle with argument 850 specification. 851
In the second case, the task is to split a string 852 while retaining separators, such as punctuation or 853
11
special tokens like newline characters. The gener-854 ated code, however, omits an essential parenthesis855 in the re.split function, leading to a split that ex-856 cludes the separators. This highlights the dual com-857 plexity of understanding both Python and Regex858 syntaxes.859
The final example presents an annotation discrep-860 ancy. It involves resetting the index of a Pandas861 dataframe without specific instructions on handling862 the old index. The model correctly employs the863 reset_index function, typically retaining the old864 index as a new column. However, the reference865 code, and consequently the unit tests, do not pre-866 serve the old index. Thus, while the generated code867 aligns with the stated task, it fails unit tests due to868 the discrepancy in index handling. This case under-869 scores the need for nuanced dataset analysis and870 exemplifies the challenges of borderline scenarios871 in dataset construction.872
F GPT-4’s prediction873
We investigate data contamination within the874 CoNaLa and our dataset by examining GPT-4’s875 outputs on both dataset. This can offer information876 in re-writting example during data annotation877 phase to mitigate data contamination.878
879
12
Intent Reference Solution
Prediction
Convert a list of integers lst0 into a single integer. 1 def test(lst0): 2 return int (’’.join( map(str, lst0))) 1 def test(lst0): 2 return int (’’.join( map(str, lst0)))
Discussion This discussion presents an analysis 880 of GPT-4’s outputs on equivalent examples from 881 the CoNaLa dataset and our dataset, CodeInsight, 882 as detailed in Tables 13 and 14. 883
For the CoNaLa dataset, the analysis of GPT- 884 4’s predictions reveals interesting observations for 885 the first and third examples. Specifically, GPT-4 886 autonomously includes an assignment to variable 887 r in its prediction for the first example, despite the 888 absence of such instruction in the original example. 889 Similarly, in the third example, GPT-4 employs the 890 k_v variable, an uncommon choice, demonstrating 891 a potential memorization to variable naming. 892
Conversely, the second example highlights GPT- 893 4’s ability to generalize. The model infers the use 894 of date_string even though the specific datetime 895 format is not explicitly mentioned in the intent, 896 showcasing its adeptness at filling in contextual 897 gaps based on the provided intent. 898
Regarding the outputs on CodeInsight for the 899 same rewritten intents, GPT-4’s accuracy remains 900 consistent for the first example. The second exam- 901 ple further underscores GPT-4’s precision, where 902 the model’s prediction aligns exactly with the ref- 903 erence, attributed to a more clearly defined intent 904 or to a memorization from CoNaLa. 905
The third example diverges in response using 906 itemgetter instead of lambda function but notably 907 passes our unit tests, illustrating GPT-4’s capacity 908 for generating viable alternative solutions. This 909 indicates that despite potential differences, GPT- 910 4’s inherent generalization capabilities enable it to 911 offer valid code solutions, reflecting its understand- 912 ing of programming concepts without memorizing 913 data. 914
13

heading: Future Work
text: These limitations sug-626 gest areas for future expansion and improvement627 to enhance the dataset’s comprehensiveness and628 applicability in diverse coding contexts. 629
References630 Jacob Austin, Augustus Odena, Maxwell I. Nye,631 Maarten Bosma, Henryk Michalewski, David Dohan,632 Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le,633 and Charles Sutton. 2021. Program synthesis with634 large language models. CoRR, abs/2108. 07732. 635
Nathanaël Beau and Benoît Crabbé. 2022. The impact636 of lexical and grammatical processing on generating637 code from natural language. In Findings of the As-638 sociation for Computational Linguistics: ACL 2022,639 Dublin, Ireland, May 22-27, 2022, pages 2204–2214. 640 Association for Computational Linguistics. 641
Shubham Chandel, Colin B. Clement, Guillermo Ser-642 rato, and Neel Sundaresan. 2022. Training and643 evaluating a jupyter notebook data science assistant. 644 CoRR, abs/2201. 12901. 645
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan,646 Henrique Pondé de Oliveira Pinto, Jared Kaplan,647 Harrison Edwards, Yuri Burda, Nicholas Joseph,648 Greg Brockman, Alex Ray, Raul Puri, Gretchen649 Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas-650 try, Pamela Mishkin, Brooke Chan, Scott Gray,651 Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz652 Kaiser, Mohammad Bavarian, Clemens Winter,653 Philippe Tillet, Felipe Petroski Such, Dave Cum-654 mings, Matthias Plappert, Fotios Chantzis, Eliza-655 beth Barnes, Ariel Herbert-Voss, William Hebgen656 Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie657 Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,658 William Saunders, Christopher Hesse, Andrew N. 659 Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan660 Morikawa, Alec Radford, Matthew Knight, Miles661 Brundage, Mira Murati, Katie Mayer, Peter Welinder,662 Bob McGrew, Dario Amodei, Sam McCandlish, Ilya663 Sutskever, and Wojciech Zaremba. 2021a. Evaluat-664 ing large language models trained on code. CoRR,665 abs/2107. 03374. 666
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan,667 Henrique Pondé de Oliveira Pinto, Jared Kaplan,668
Harrison Edwards, Yuri Burda, Nicholas Joseph, 669 Greg Brockman, Alex Ray, Raul Puri, Gretchen 670 Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- 671 try, Pamela Mishkin, Brooke Chan, Scott Gray, 672 Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz 673 Kaiser, Mohammad Bavarian, Clemens Winter, 674 Philippe Tillet, Felipe Petroski Such, Dave Cum- 675 mings, Matthias Plappert, Fotios Chantzis, Eliza- 676 beth Barnes, Ariel Herbert-Voss, William Hebgen 677 Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie 678 Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, 679 William Saunders, Christopher Hesse, Andrew N. 680 Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan 681 Morikawa, Alec Radford, Matthew Knight, Miles 682 Brundage, Mira Murati, Katie Mayer, Peter Welinder, 683 Bob McGrew, Dario Amodei, Sam McCandlish, Ilya 684 Sutskever, and Wojciech Zaremba. 2021b. Evaluat- 685 ing large language models trained on code. CoRR, 686 abs/2107. 03374. 687
Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- 688 tas Mazeika, Akul Arora, Ethan Guo, Collin Burns, 689 Samir Puranik, Horace He, Dawn Song, and Jacob 690 Steinhardt. 2021. Measuring coding challenge com- 691 petence with APPS. In Proceedings of the Neural 692 Information Processing Systems Track on Datasets 693 and Benchmarks 1, NeurIPS Datasets and Bench- 694 marks 2021, December 2021, virtual. 695
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- 696 sch, Chris Bamford, Devendra Singh Chaplot, Diego 697 de Las Casas, Florian Bressand, Gianna Lengyel, 698 Guillaume Lample, Lucile Saulnier, Lélio Re- 699 nard Lavaud, Marie-Anne Lachaux, Pierre Stock, 700 Teven Le Scao, Thibaut Lavril, Thomas Wang, Timo- 701 thée Lacroix, and William El Sayed. 2023. Mistral 702 7b. CoRR, abs/2310. 06825. 703
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, 704 Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, 705 Daniel Fried, Sida I. Wang, and Tao Yu. 2023. DS- 706 1000: A natural and reliable benchmark for data sci- 707 ence code generation. In International Conference 708 on Machine Learning, ICML 2023, 23-29 July 2023, 709 Honolulu, Hawaii, USA, volume 202 of Proceedings 710 of Machine Learning Research, pages 18319–18345. 711 PMLR. 712
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas 713 Muennighoff, Denis Kocetkov, Chenghao Mou, 714 Marc Marone, Christopher Akiki, Jia Li, Jenny 715 Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue 716 Zhuo, Thomas Wang, Olivier Dehaene, Mishig 717 Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh 718 Shliazhko, Nicolas Gontier, Nicholas Meade, Armel 719 Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, 720 Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, 721 Zhiruo Wang, Rudra Murthy V, Jason Stillerman, 722 Siva Sankalp Patel, Dmitry Abulkhanov, Marco 723 Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa- 724 Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam 725 Singh, Sasha Luccioni, Paulo Villegas, Maxim Ku- 726 nakov, Fedor Zhdanov, Manuel Romero, Tony Lee, 727 Nadav Timor, Jennifer Ding, Claire Schlesinger, Hai- 728 ley Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, 729
9
Alex Gu, Jennifer Robinson, Carolyn Jane Ander-730 son, Brendan Dolan-Gavitt, Danish Contractor, Siva731 Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jer-732 nite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas733 Wolf, Arjun Guha, Leandro von Werra, and Harm734 de Vries. 2023. Starcoder: may the source be with735 you! CoRR, abs/2305. 06161. 736
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-737 Jing Zhu. 2002. Bleu: a method for automatic evalu-738 ation of machine translation. In Proceedings of the739 40th Annual Meeting of the Association for Compu-740 tational Linguistics, July 6-12, 2002, Philadelphia,741 PA, USA, pages 311–318. ACL. 742
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu,743 Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio744 Blanco, and Shuai Ma. 2020. Codebleu: a method745 for automatic evaluation of code synthesis. CoRR,746 abs/2009. 10297. 747
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten748 Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi,749 Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom750 Kozhevnikov, Ivan Evtimov, Joanna Bitton, Man-751 ish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori,752 Wenhan Xiong, Alexandre Défossez, Jade Copet,753 Faisal Azhar, Hugo Touvron, Louis Martin, Nico-754 las Usunier, Thomas Scialom, and Gabriel Synnaeve. 755 2023. Code llama: Open foundation models for code. 756 CoRR, abs/2308. 12950. 757
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham758 Neubig. 2022. Execution-based evaluation for open-759 domain code generation. CoRR, abs/2212. 10481. 760
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan761 Vasilescu, and Graham Neubig. 2018. Learning to762 mine aligned code and natural language pairs from763 stack overflow. In Proceedings of the 15th Interna-764 tional Conference on Mining Software Repositories,765 MSR 2018, Gothenburg, Sweden, May 28-29, 2018,766 pages 476–486. ACM. 767
A Detailed overview of filtering phase768
We include two tables that analyze the exploitabil-769 ity of examples from the CoNaLa dataset. The770 Table 6 presents the 10 examples with the high-771 est probability of exploitability, highlighting their772 votes, titles, and whether they are exploitable. The773 Table 7 displays a random selection of 10 exam-774 ples from the same dataset, also detailing their ex-775 ploitability probability, votes, and titles. 776 We provide a detailed description of an accepted777 example, a rejected example and a borderline case778 for passing the filtering phase. 779
Accepted example We detailed the accepted ex-780 ample which is the first one on the left of the Figure781 1. This particular example, a query about finding782 the largest values in a numpy array, demonstrates783
a typical developer’s question due to unfamiliar- 784 ity with specific numpy functions. Its solution, 785 involving the argpartition function, is directly 786 responsive to the query and easily testable, making 787 it a perfect fit for our dataset. 788
Rejected example The "List sorting with multi- 789 ple attributes and mixed order" question on Stack 790 Overflow from Table 7 presents an excessive level 791 of specificity for inclusion whereas it has a high 792 P(expl) value. This question delves into sorting 793 a list by different attributes of a particular class, 794 emphasizing the specific class’s complexity rather 795 than a broader understanding of sorting functions. 796 The high level of detail in both the problem and its 797 solution complicates the extraction of universally 798 applicable code examples. Therefore, including it 799 may not aptly represent the range of coding tasks 800 and challenges. 801
Edge example An example such as "How to con- 802 vert a Date string to a datetime object?" presented 803 in Table 6 necessitates a more specific reformu- 804 lation for a precise coding answer, like "How to 805 compare a date string in ISO format to a datetime 806 object. " This demands the annotator’s understand- 807 ing of ISO format data. These are considered edge 808 cases in our dataset and are included based on the 809 annotator’s expertise, who are constrained to a max- 810 imum of 20 minutes per annotation process. 811
10
B Normalized variable names812
The Table 8 outlines the standardized variable813 names utilized in the dataset, such as vari for814 ’Variable’ and dicti for ’Dictionary’, where i cor-815 respond to the number of the element appearing. 816 This approach also allows for evaluating model817 efficacy with or without these normalized names. 818 Note that vari is employed universally, even when819 alternatives might be applicable, without affecting820 test outcomes. 821 C Code Categories822
823
D CodeInsight Statistics824
The two tables provide a detailed statistical analy-825 sis of the CodeInsight dataset, breaking down by826 Packages and Labels. The Table 10 covers various827 Python packages like Pandas, Numpy, and Regex,828 detailing the item count, average problem words,829 code lines, and unit tests. The second Table 11 ana-830 lyzes different labels presented in Appendix C such831 as Builtin, Assign, Cond, and others, also including832 their item count and average metrics. Both tables833 gives insight on the dataset’s complexity and diver-834 sity into the typical problem structure and testing835
E Code Error Analysis 838
We present in Table 12 an error analysis of dif- 839 ferent CodeLLaMa outputs on our dataset. 840
For the first scenario, involving scikit-learn, the 841 task is to generate a textual representation of a 842 decision tree with specified feature names for read- 843 ability. Despite providing the model with import 844 statements and the function signature, the gener- 845 ated code omits the necessary argument syntax 846 feature_names=feature_names0, resulting in a 847 TypeError due to excessive positional arguments. 848 This minor but pivotal error exemplifies a scenario 849 where a developer could struggle with argument 850 specification. 851
In the second case, the task is to split a string 852 while retaining separators, such as punctuation or 853
11
special tokens like newline characters. The gener-854 ated code, however, omits an essential parenthesis855 in the re. split function, leading to a split that ex-856 cludes the separators. This highlights the dual com-857 plexity of understanding both Python and Regex858 syntaxes. 859
The final example presents an annotation discrep-860 ancy. It involves resetting the index of a Pandas861 dataframe without specific instructions on handling862 the old index. The model correctly employs the863 reset_index function, typically retaining the old864 index as a new column. However, the reference865 code, and consequently the unit tests, do not pre-866 serve the old index. Thus, while the generated code867 aligns with the stated task, it fails unit tests due to868 the discrepancy in index handling. This case under-869 scores the need for nuanced dataset analysis and870 exemplifies the challenges of borderline scenarios871 in dataset construction. 872
F GPT-4’s prediction873
We investigate data contamination within the874 CoNaLa and our dataset by examining GPT-4’s875 outputs on both dataset. This can offer information876 in re-writting example during data annotation877 phase to mitigate data contamination. 878
879
12
Intent Reference Solution
Prediction
Convert a list of integers lst0 into a single integer. 1 def test(lst0): 2 return int (’’. join( map(str, lst0))) 1 def test(lst0): 2 return int (’’